2024-02-05T07:47:49.792 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-05T07:47:49.792 DEBUG:teuthology.run:Teuthology command: teuthology --name gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi --verbose --description orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} --owner scheduled_gabrioux@teuthology --archive /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546773 -- /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546773/orig.config.yaml 2024-02-05T07:47:49.825 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-05T07:47:49.901 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546773 branch: wip-guits-testing-3-2024-02-04-2254 description: orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} email: gabrioux@ibm.com first_in_suite: false job_id: '7546773' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-testing-3-2024-02-04-2254 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-testing-3-2024-02-04-2254 sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 7317 sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-testing-3-2024-02-04-2254 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_164d3be4a527be775f4d3c343ea591122f1b8921/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 targets: smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKiYq7PnZVrWaRE/XchE/ec0BBUYiIqXs1TFpVXu3BjaqUFQu5fd8tuLhFhrEaQYnqpJrUYohctQwcmiVcJYk2U= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPL54Xfgje0159iEnIFNlZXin0/8cKqk211ej0lfjf0mi2xcneOS7cDg16GCqeRq7/NmyD+sGJ44J8DkWehus3Q= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-05_05:57:01 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-05T07:47:49.901 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_164d3be4a527be775f4d3c343ea591122f1b8921/qa; will attempt to use it 2024-02-05T07:47:49.902 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_164d3be4a527be775f4d3c343ea591122f1b8921/qa/tasks 2024-02-05T07:47:49.902 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-05T07:47:49.904 INFO:teuthology.task.internal:Checking packages... 2024-02-05T07:47:49.926 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '164d3be4a527be775f4d3c343ea591122f1b8921' 2024-02-05T07:47:49.926 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-05T07:47:49.926 INFO:teuthology.packaging:ref: None 2024-02-05T07:47:49.926 INFO:teuthology.packaging:tag: None 2024-02-05T07:47:49.926 INFO:teuthology.packaging:branch: wip-guits-testing-3-2024-02-04-2254 2024-02-05T07:47:49.926 INFO:teuthology.packaging:sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:47:49.926 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-testing-3-2024-02-04-2254 2024-02-05T07:47:50.143 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1241.g164d3be4 2024-02-05T07:47:50.144 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-05T07:47:50.151 INFO:teuthology.task.internal:no buildpackages task found 2024-02-05T07:47:50.152 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-05T07:47:50.180 INFO:teuthology.task.internal:Saving configuration 2024-02-05T07:47:50.193 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-05T07:47:50.206 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T07:47:50.232 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546773', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 07:42:18.792631', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKiYq7PnZVrWaRE/XchE/ec0BBUYiIqXs1TFpVXu3BjaqUFQu5fd8tuLhFhrEaQYnqpJrUYohctQwcmiVcJYk2U='} 2024-02-05T07:47:50.252 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546773', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 07:42:18.794313', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPL54Xfgje0159iEnIFNlZXin0/8cKqk211ej0lfjf0mi2xcneOS7cDg16GCqeRq7/NmyD+sGJ44J8DkWehus3Q='} 2024-02-05T07:47:50.252 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-05T07:47:50.260 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-05T07:47:50.260 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-02-05T07:47:50.260 INFO:teuthology.run_tasks:Running task console_log... 2024-02-05T07:47:50.341 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff6e4c440d0>, signals=[15]) 2024-02-05T07:47:50.341 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-05T07:47:50.349 INFO:teuthology.task.internal:Opening connections... 2024-02-05T07:47:50.349 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-02-05T07:47:50.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:47:50.445 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T07:47:50.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:47:50.532 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-05T07:47:50.540 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-02-05T07:47:50.557 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-02-05T07:47:50.557 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-02-05T07:47:50.614 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-02-05T07:47:50.614 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-02-05T07:47:50.614 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-02-05T07:47:50.614 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-02-05T07:47:50.614 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-02-05T07:47:50.614 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-02-05T07:47:50.614 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-05T07:47:50.614 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-02-05T07:47:50.614 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-02-05T07:47:50.614 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-05T07:47:50.615 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-02-05T07:47:50.615 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-05T07:47:50.615 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-05T07:47:50.615 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-05T07:47:50.615 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-02-05T07:47:50.637 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-02-05T07:47:50.655 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-02-05T07:47:50.655 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-02-05T07:47:50.711 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-02-05T07:47:50.711 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-02-05T07:47:50.711 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-02-05T07:47:50.712 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-02-05T07:47:50.712 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-02-05T07:47:50.712 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-02-05T07:47:50.712 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-05T07:47:50.712 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-02-05T07:47:50.712 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-02-05T07:47:50.712 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-05T07:47:50.712 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-02-05T07:47:50.712 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-05T07:47:50.712 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-05T07:47:50.712 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-05T07:47:50.713 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-02-05T07:47:50.737 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-05T07:47:50.750 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-05T07:47:50.775 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-05T07:47:50.775 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-02-05T07:47:50.778 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-02-05T07:47:50.794 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-05T07:47:50.805 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-05T07:47:50.806 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-02-05T07:47:50.835 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-02-05T07:47:50.933 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-05T07:47:50.992 INFO:teuthology.run_tasks:Running task kernel... 2024-02-05T07:47:51.012 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-05T07:47:51.012 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-05T07:47:51.012 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-05T07:47:51.013 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-05T07:47:51.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:47:51.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:47:51.030 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-02-05T07:47:51.031 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-02-05T07:47:51.087 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-361.el9.x86_64 2024-02-05T07:47:51.087 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.14.0-361.el9.x86_64 2024-02-05T07:47:51.087 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-02-05T07:47:51.088 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-361.el9.x86_64 2024-02-05T07:47:51.089 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-361.el9.x86_64 2024-02-05T07:47:51.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-02-05T07:47:53.719 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 6.1 MB/s | 6.6 MB 00:01 2024-02-05T07:47:54.462 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 3.9 MB/s | 6.6 MB 00:01 2024-02-05T07:47:55.128 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-02-05T07:47:56.337 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 2.5 MB/s | 2.0 MB 00:00 2024-02-05T07:47:58.935 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 3.5 MB/s | 8.0 MB 00:02 2024-02-05T07:48:01.120 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-02-05T07:48:01.717 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 1.2 MB/s | 8.0 MB 00:06 2024-02-05T07:48:05.897 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 5.9 MB 00:01 2024-02-05T07:48:06.418 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 5.2 MB/s | 19 MB 00:03 2024-02-05T07:48:07.254 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 15 kB 00:00 2024-02-05T07:48:10.156 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 20 MB 00:02 2024-02-05T07:48:11.034 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 6.0 MB/s | 5.9 MB 00:00 2024-02-05T07:48:12.335 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 15 kB 00:00 2024-02-05T07:48:14.312 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2024-02-05T07:48:15.064 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-02-05T07:48:17.279 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-05T07:48:17.279 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-05T07:48:17.398 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:48:17.407 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:48:17.407 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-05T07:48:17.407 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:48:17.407 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:48:17.407 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-416.el9 CentOS-BaseOS 6.0 M 2024-02-05T07:48:17.407 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-05T07:48:17.407 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 5.14.0-416.el9 CentOS-BaseOS 21 M 2024-02-05T07:48:17.408 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 5.14.0-416.el9 CentOS-BaseOS 39 M 2024-02-05T07:48:17.408 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core x86_64 5.14.0-416.el9 CentOS-BaseOS 33 M 2024-02-05T07:48:17.408 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:48:17.408 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:48:17.408 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:48:17.408 INFO:teuthology.orchestra.run.smithi134.stdout:Install 4 Packages 2024-02-05T07:48:17.408 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:48:17.409 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 99 M 2024-02-05T07:48:17.409 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 122 M 2024-02-05T07:48:17.409 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:48:17.940 INFO:teuthology.orchestra.run.smithi134.stdout:(1/4): kernel-5.14.0-416.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-02-05T07:48:18.399 INFO:teuthology.orchestra.run.smithi134.stdout:(2/4): kernel-core-5.14.0-416.el9.x86_64.rpm 21 MB/s | 21 MB 00:00 2024-02-05T07:48:18.924 INFO:teuthology.orchestra.run.smithi134.stdout:(3/4): kernel-modules-5.14.0-416.el9.x86_64.rpm 26 MB/s | 39 MB 00:01 2024-02-05T07:48:19.169 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-02-05T07:48:19.440 INFO:teuthology.orchestra.run.smithi134.stdout:(4/4): kernel-modules-core-5.14.0-416.el9.x86_6 22 MB/s | 33 MB 00:01 2024-02-05T07:48:19.444 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:48:19.444 INFO:teuthology.orchestra.run.smithi134.stdout:Total 49 MB/s | 99 MB 00:02 2024-02-05T07:48:19.444 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:48:19.829 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:48:19.829 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:48:20.876 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:48:20.876 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:48:21.376 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-05T07:48:21.376 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-05T07:48:21.489 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:48:21.499 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:48:21.500 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T07:48:21.500 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:48:21.500 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:48:21.500 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-416.el9 CentOS-BaseOS 6.0 M 2024-02-05T07:48:21.500 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T07:48:21.500 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-416.el9 CentOS-BaseOS 21 M 2024-02-05T07:48:21.500 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-416.el9 CentOS-BaseOS 39 M 2024-02-05T07:48:21.500 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-416.el9 CentOS-BaseOS 33 M 2024-02-05T07:48:21.500 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:48:21.500 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:48:21.500 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:48:21.500 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-02-05T07:48:21.501 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:48:21.501 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 99 M 2024-02-05T07:48:21.501 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 122 M 2024-02-05T07:48:21.501 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:48:21.937 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-416.el9.x86_64.rpm 14 MB/s | 6.0 MB 00:00 2024-02-05T07:48:22.437 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-core-5.14.0-416.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-02-05T07:48:23.285 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:48:23.338 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-core-5.14.0-416.el9.x86_6 24 MB/s | 33 MB 00:01 2024-02-05T07:48:23.671 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-modules-5.14.0-416.el9.x86_64.rpm 18 MB/s | 39 MB 00:02 2024-02-05T07:48:23.674 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:48:23.674 INFO:teuthology.orchestra.run.smithi196.stdout:Total 46 MB/s | 99 MB 00:02 2024-02-05T07:48:23.675 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:48:24.027 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:48:24.027 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:48:24.182 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-core-5.14.0-416.el9.x86_64 1/4 2024-02-05T07:48:24.332 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T07:48:25.057 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:48:25.057 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:48:27.081 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T07:48:27.260 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T07:48:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:48:28.093 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-416.el9.x86_64 1/4 2024-02-05T07:48:28.221 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T07:48:30.961 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T07:48:31.140 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T07:48:31.954 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T07:48:32.155 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:48:35.803 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T07:48:36.001 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:48:36.861 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:48:40.728 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:49:06.187 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:49:06.187 INFO:teuthology.orchestra.run.smithi134.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-05T07:49:06.188 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:49:06.256 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:49:06.629 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:49:06.629 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-416.el9.x86_64 1/4 2024-02-05T07:49:06.629 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T07:49:06.629 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T07:49:07.049 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:49:07.049 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:49:07.050 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:49:07.050 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-5.14.0-416.el9.x86_64 2024-02-05T07:49:07.050 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core-5.14.0-416.el9.x86_64 2024-02-05T07:49:07.050 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-5.14.0-416.el9.x86_64 2024-02-05T07:49:07.050 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core-5.14.0-416.el9.x86_64 2024-02-05T07:49:07.050 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:49:07.050 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:49:07.545 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T07:49:07.679 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T07:49:07.679 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T07:49:07.679 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-416.el9.x86_64 2024-02-05T07:49:07.679 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-05T07:49:07.679 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-05T07:49:07.679 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-05T07:49:07.679 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-02-05T07:49:07.680 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T07:49:07.786 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T07:49:07.787 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T07:49:07.787 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-02-05T07:49:07.820 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-02-05T07:49:07.820 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-02-05T07:49:07.820 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.06 2024-02-05T07:49:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 68.el9 2024-02-05T07:49:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-02-05T07:49:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-05T07:49:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:Group : Unspecified 2024-02-05T07:49:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 8264254 2024-02-05T07:49:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-02-05T07:49:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-05T07:49:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-05T07:49:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-05T07:49:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-05T07:49:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : builder@centos.org 2024-02-05T07:49:07.821 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-02-05T07:49:07.822 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-05T07:49:07.822 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-02-05T07:49:07.822 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-02-05T07:49:07.822 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:49:07.822 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-05T07:49:07.822 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-05T07:49:07.822 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-05T07:49:07.822 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-02-05T07:49:07.822 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:49:07.822 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-02-05T07:49:07.823 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-05T07:49:07.823 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 5.14.0-416.el9.x86_64 2024-02-05T07:49:07.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-05T07:49:08.414 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-02-05T07:49:09.706 INFO:teuthology.orchestra.run.smithi134.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-05T07:49:09.726 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-02-05T07:49:09.728 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-02-05T07:49:09.743 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.3jU7Iayt45 2024-02-05T07:49:09.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3jU7Iayt45 2024-02-05T07:49:09.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.3jU7Iayt45 2024-02-05T07:49:09.962 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.3jU7Iayt45 is 6KB 2024-02-05T07:49:10.011 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.3jU7Iayt45 2024-02-05T07:49:10.025 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-02-05T07:49:10.097 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-05T07:49:10.097 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-05T07:49:10.097 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-05T07:49:10.097 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64.conf 2024-02-05T07:49:10.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64 2024-02-05T07:49:10.119 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:49:10.119 INFO:teuthology.orchestra.run.smithi196.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-05T07:49:10.119 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:49:10.191 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:49:10.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-02-05T07:49:10.615 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:49:10.615 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-416.el9.x86_64 1/4 2024-02-05T07:49:10.616 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T07:49:10.616 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T07:49:11.044 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T07:49:11.044 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:49:11.044 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:49:11.044 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-416.el9.x86_64 2024-02-05T07:49:11.044 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-416.el9.x86_64 2024-02-05T07:49:11.044 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-416.el9.x86_64 2024-02-05T07:49:11.044 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-416.el9.x86_64 2024-02-05T07:49:11.044 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:49:11.044 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:49:11.547 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T07:49:11.675 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T07:49:11.675 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T07:49:11.676 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-416.el9.x86_64 2024-02-05T07:49:11.676 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-05T07:49:11.676 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-05T07:49:11.676 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-05T07:49:11.676 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-02-05T07:49:11.676 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T07:49:11.783 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T07:49:11.784 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T07:49:11.784 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-02-05T07:49:11.821 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-02-05T07:49:11.822 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-02-05T07:49:11.822 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-02-05T07:49:11.822 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 68.el9 2024-02-05T07:49:11.822 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-02-05T07:49:11.822 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-05T07:49:11.822 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-02-05T07:49:11.822 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8264254 2024-02-05T07:49:11.822 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-02-05T07:49:11.822 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-05T07:49:11.823 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-05T07:49:11.823 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-05T07:49:11.823 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-05T07:49:11.823 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-02-05T07:49:11.823 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-02-05T07:49:11.823 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-05T07:49:11.823 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-02-05T07:49:11.823 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-02-05T07:49:11.823 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:49:11.823 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-05T07:49:11.823 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-05T07:49:11.823 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-05T07:49:11.823 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-02-05T07:49:11.824 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:49:11.824 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-02-05T07:49:11.824 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-05T07:49:11.825 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-416.el9.x86_64 2024-02-05T07:49:11.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-05T07:49:12.423 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-02-05T07:49:13.881 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-05T07:49:13.932 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-02-05T07:49:13.935 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-02-05T07:49:13.949 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.cfPkH6zjS9 2024-02-05T07:49:13.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cfPkH6zjS9 2024-02-05T07:49:14.018 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.cfPkH6zjS9 2024-02-05T07:49:14.156 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.cfPkH6zjS9 is 6KB 2024-02-05T07:49:14.204 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.cfPkH6zjS9 2024-02-05T07:49:14.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-02-05T07:49:14.283 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-05T07:49:14.283 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-05T07:49:14.283 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-05T07:49:14.284 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64.conf 2024-02-05T07:49:14.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64 2024-02-05T07:49:14.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-02-05T07:49:40.274 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-05T07:49:40.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-02-05T07:49:40.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:49:44.425 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-05T07:49:44.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-02-05T07:49:44.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:49:58.716 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-02-05T07:50:02.816 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-02-05T07:50:07.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-02-05T07:50:07.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:50:10.780 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-02-05T07:50:11.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-02-05T07:50:11.819 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:50:14.876 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-02-05T07:50:22.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-02-05T07:50:22.790 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:50:26.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-02-05T07:50:26.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:50:54.700 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-02-05T07:50:55.174 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-02-05T07:50:55.174 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-416.el9.x86_64"... 2024-02-05T07:50:55.174 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-02-05T07:50:55.188 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-416.el9.x86_64 2024-02-05T07:50:55.188 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-416.el9.x86_64 vs 5.14.0-416.el9.x86_64 2024-02-05T07:50:55.188 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-05T07:50:55.188 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-05T07:50:56.190 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-05T07:50:56.190 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-05T07:50:56.321 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-02-05T07:50:56.401 DEBUG:teuthology.parallel:result is None 2024-02-05T07:50:58.680 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-02-05T07:50:59.060 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-02-05T07:50:59.060 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-416.el9.x86_64"... 2024-02-05T07:50:59.061 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-02-05T07:50:59.074 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-416.el9.x86_64 2024-02-05T07:50:59.074 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-416.el9.x86_64 vs 5.14.0-416.el9.x86_64 2024-02-05T07:50:59.074 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-05T07:50:59.074 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-05T07:51:00.074 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-05T07:51:00.075 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-05T07:51:00.174 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-02-05T07:51:00.210 DEBUG:teuthology.parallel:result is None 2024-02-05T07:51:00.211 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-05T07:51:00.220 INFO:teuthology.task.internal:Creating test directory... 2024-02-05T07:51:00.220 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-05T07:51:00.223 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-05T07:51:00.248 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-05T07:51:00.255 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-05T07:51:00.292 INFO:teuthology.task.internal:Creating archive directory... 2024-02-05T07:51:00.292 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-05T07:51:00.295 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-05T07:51:00.353 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-05T07:51:00.359 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-05T07:51:00.360 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-05T07:51:00.362 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-05T07:51:00.391 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:51:00.405 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:51:00.421 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:51:00.445 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T07:51:00.447 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-05T07:51:00.454 INFO:teuthology.task.internal:Configuring sudo... 2024-02-05T07:51:00.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-05T07:51:00.456 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-05T07:51:00.499 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-05T07:51:00.508 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-05T07:51:00.509 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-05T07:51:00.528 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-05T07:51:00.555 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T07:51:00.624 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T07:51:00.687 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T07:51:00.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T07:51:00.772 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T07:51:00.772 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-05T07:51:00.840 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T07:51:00.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T07:51:01.012 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T07:51:01.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T07:51:01.096 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T07:51:01.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-05T07:51:01.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-02-05T07:51:01.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-02-05T07:51:01.211 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T07:51:01.303 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T07:51:01.574 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-05T07:51:01.582 INFO:teuthology.task.internal:Starting timer... 2024-02-05T07:51:01.582 INFO:teuthology.run_tasks:Running task pcp... 2024-02-05T07:51:01.612 INFO:teuthology.run_tasks:Running task selinux... 2024-02-05T07:51:01.661 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-05T07:51:01.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-02-05T07:51:01.713 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-02-05T07:51:01.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-02-05T07:51:01.772 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-02-05T07:51:01.774 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-05T07:51:01.774 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-02-05T07:51:01.800 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-02-05T07:51:01.801 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-02-05T07:51:01.823 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-02-05T07:51:01.823 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi134.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-02-05T07:51:01.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T07:51:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:51:01.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T07:51:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:51:01.898 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-05T07:51:01.898 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-02-05T07:51:01.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-02-05T07:51:01.983 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-05T07:51:01.994 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-05T07:51:02.011 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-05T07:51:02.012 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi134.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-05T07:55:07.431 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi134.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-02-05T07:55:07.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-02-05T07:55:07.433 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:55:07.527 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-02-05T07:55:07.603 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-02-05T07:55:07.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-02-05T07:55:07.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T07:55:07.689 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-02-05T07:55:07.763 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-02-05T07:55:07.763 INFO:teuthology.run_tasks:Running task clock... 2024-02-05T07:55:07.776 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-05T07:55:07.777 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-05T07:55:07.777 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T07:55:07.779 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-05T07:55:07.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T07:55:07.807 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-05T07:55:07.824 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-05T07:55:07.834 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-05T07:55:07.851 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-05T07:55:07.853 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-02-05T07:55:07.866 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-02-05T07:55:07.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-02-05T07:55:07.883 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-05T07:55:07.894 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-02-05T07:55:07.899 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-05T07:55:07.910 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-05T07:55:07.926 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-05T07:55:07.957 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-02-05T07:55:07.960 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T07:55:07.960 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-02-05T07:55:07.960 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:55:07.960 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:55:07.960 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:55:07.960 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:55:07.981 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-02-05T07:55:07.984 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T07:55:07.984 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-02-05T07:55:07.984 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:55:07.984 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:55:07.984 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:55:07.984 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:55:07.985 INFO:teuthology.run_tasks:Running task pexec... 2024-02-05T07:55:08.004 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-05T07:55:08.004 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi134.front.sepia.ceph.com 2024-02-05T07:55:08.004 DEBUG:teuthology.orchestra.run.smithi134:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-05T07:55:08.005 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T07:55:08.005 DEBUG:teuthology.orchestra.run.smithi196:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-05T07:55:08.583 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:47 ago on Mon 05 Feb 2024 07:54:21 AM UTC. 2024-02-05T07:55:08.614 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:48 ago on Mon 05 Feb 2024 07:54:20 AM UTC. 2024-02-05T07:55:08.736 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:55:08.738 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:55:08.739 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-02-05T07:55:08.739 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:55:08.739 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:55:08.739 INFO:teuthology.orchestra.run.smithi134.stdout: runc x86_64 4:1.1.12-1.el9 CentOS-AppStream 3.1 M 2024-02-05T07:55:08.739 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:55:08.739 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:55:08.739 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:55:08.740 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-05T07:55:08.740 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:55:08.740 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.1 M 2024-02-05T07:55:08.740 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 9.8 M 2024-02-05T07:55:08.740 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:55:08.766 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:55:08.781 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:55:08.781 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-02-05T07:55:08.781 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:55:08.781 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:55:08.781 INFO:teuthology.orchestra.run.smithi196.stdout: runc x86_64 4:1.1.12-1.el9 CentOS-AppStream 3.1 M 2024-02-05T07:55:08.781 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:55:08.782 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:55:08.782 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:55:08.782 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T07:55:08.782 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:55:08.782 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 3.1 M 2024-02-05T07:55:08.782 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 9.8 M 2024-02-05T07:55:08.782 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:55:09.013 INFO:teuthology.orchestra.run.smithi134.stdout:runc-1.1.12-1.el9.x86_64.rpm 11 MB/s | 3.1 MB 00:00 2024-02-05T07:55:09.013 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:55:09.013 INFO:teuthology.orchestra.run.smithi134.stdout:Total 11 MB/s | 3.1 MB 00:00 2024-02-05T07:55:09.014 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:55:09.019 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:55:09.019 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:55:09.042 INFO:teuthology.orchestra.run.smithi196.stdout:runc-1.1.12-1.el9.x86_64.rpm 12 MB/s | 3.1 MB 00:00 2024-02-05T07:55:09.043 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:55:09.043 INFO:teuthology.orchestra.run.smithi196.stdout:Total 12 MB/s | 3.1 MB 00:00 2024-02-05T07:55:09.043 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:55:09.048 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:55:09.048 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:55:09.089 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:55:09.089 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:55:09.120 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:55:09.120 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:55:09.302 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:55:09.329 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:55:09.398 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-05T07:55:09.411 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-05T07:55:09.969 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-05T07:55:09.993 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-05T07:55:10.343 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-05T07:55:10.344 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:55:10.344 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:55:10.344 INFO:teuthology.orchestra.run.smithi196.stdout: runc-4:1.1.12-1.el9.x86_64 2024-02-05T07:55:10.344 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:55:10.344 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:55:10.348 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-05T07:55:10.348 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:55:10.348 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:55:10.349 INFO:teuthology.orchestra.run.smithi134.stdout: runc-4:1.1.12-1.el9.x86_64 2024-02-05T07:55:10.349 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:55:10.349 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:55:10.592 DEBUG:teuthology.parallel:result is None 2024-02-05T07:55:10.621 DEBUG:teuthology.parallel:result is None 2024-02-05T07:55:10.621 INFO:teuthology.run_tasks:Running task install... 2024-02-05T07:55:10.630 DEBUG:teuthology.task.install:project ceph 2024-02-05T07:55:10.631 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'}} 2024-02-05T07:55:10.631 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'} 2024-02-05T07:55:10.631 INFO:teuthology.task.install:Using flavor: default 2024-02-05T07:55:10.636 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-05T07:55:10.637 INFO:teuthology.task.install:extra packages: [] 2024-02-05T07:55:10.637 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921', 'tag': None, 'wait_for_package': False} 2024-02-05T07:55:10.637 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:55:10.640 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921', 'tag': None, 'wait_for_package': False} 2024-02-05T07:55:10.640 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:55:10.783 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/ 2024-02-05T07:55:10.783 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1241.g164d3be4 2024-02-05T07:55:10.790 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/ 2024-02-05T07:55:10.790 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1241.g164d3be4 2024-02-05T07:55:10.893 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-05T07:55:10.893 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T07:55:10.893 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-05T07:55:10.921 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-05T07:55:10.921 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/164d3be4a527be775f4d3c343ea591122f1b8921/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-05T07:55:10.960 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-05T07:55:10.960 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T07:55:10.960 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-05T07:55:10.988 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-05T07:55:10.989 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-05T07:55:10.990 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/164d3be4a527be775f4d3c343ea591122f1b8921/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-05T07:55:11.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-05T07:55:11.069 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-05T07:55:11.138 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-05T07:55:11.141 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-02-05T07:55:11.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-02-05T07:55:11.204 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-02-05T07:55:11.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-02-05T07:55:11.380 INFO:teuthology.orchestra.run.smithi134.stdout:74 files removed 2024-02-05T07:55:11.408 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2024-02-05T07:55:11.447 INFO:teuthology.orchestra.run.smithi196.stdout:74 files removed 2024-02-05T07:55:11.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw 2024-02-05T07:55:12.046 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 250 kB/s | 80 kB 00:00 2024-02-05T07:55:12.098 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 259 kB/s | 80 kB 00:00 2024-02-05T07:55:12.439 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-02-05T07:55:12.465 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-02-05T07:55:12.789 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-02-05T07:55:12.807 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-02-05T07:55:13.250 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 16 MB/s | 6.6 MB 00:00 2024-02-05T07:55:13.268 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 16 MB/s | 6.6 MB 00:00 2024-02-05T07:55:14.692 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-02-05T07:55:14.693 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-02-05T07:55:16.125 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 7.2 MB/s | 8.0 MB 00:01 2024-02-05T07:55:18.733 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-02-05T07:55:19.137 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 19 MB 00:01 2024-02-05T07:55:20.753 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-02-05T07:55:24.013 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 5.9 MB 00:01 2024-02-05T07:55:25.594 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 15 kB 00:00 2024-02-05T07:55:26.485 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 5.9 MB 00:02 2024-02-05T07:55:28.311 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2024-02-05T07:55:29.339 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 22 MB/s | 20 MB 00:00 2024-02-05T07:55:34.353 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-02-05T07:55:36.916 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:55:36.918 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:55:36.918 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-05T07:55:36.918 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:55:36.918 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:55:36.918 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 9.7 M 2024-02-05T07:55:36.919 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-02-05T07:55:36.919 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.4 M 2024-02-05T07:55:36.919 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.1 M 2024-02-05T07:55:36.919 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-05T07:55:36.919 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options 2024-02-05T07:55:36.919 INFO:teuthology.orchestra.run.smithi134.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-05T07:55:36.919 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 5.1 M 2024-02-05T07:55:36.919 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 20 M 2024-02-05T07:55:36.919 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 25 k 2024-02-05T07:55:36.919 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-05T07:55:36.919 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-05T07:55:36.919 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-05T07:55:36.919 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 743 k 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 498 k 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 5.1 M 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 45 k 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 131 k 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 162 k 2024-02-05T07:55:36.920 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-05T07:55:36.921 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 320 k 2024-02-05T07:55:36.921 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 298 k 2024-02-05T07:55:36.921 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 99 k 2024-02-05T07:55:36.921 INFO:teuthology.orchestra.run.smithi134.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-05T07:55:36.921 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-05T07:55:36.921 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:55:36.921 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:55:36.921 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:55:36.921 INFO:teuthology.orchestra.run.smithi134.stdout:Install 27 Packages 2024-02-05T07:55:36.921 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2024-02-05T07:55:36.921 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:55:36.922 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 57 M 2024-02-05T07:55:36.922 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:55:37.838 INFO:teuthology.orchestra.run.smithi134.stdout:(1/29): ceph-radosgw-19.0.0-1241.g164d3be4.el9. 13 MB/s | 9.7 MB 00:00 2024-02-05T07:55:37.871 INFO:teuthology.orchestra.run.smithi134.stdout:(2/29): ceph-selinux-19.0.0-1241.g164d3be4.el9. 765 kB/s | 25 kB 00:00 2024-02-05T07:55:37.938 INFO:teuthology.orchestra.run.smithi134.stdout:(3/29): libcephfs2-19.0.0-1241.g164d3be4.el9.x8 11 MB/s | 743 kB 00:00 2024-02-05T07:55:38.005 INFO:teuthology.orchestra.run.smithi134.stdout:(4/29): libradosstriper1-19.0.0-1241.g164d3be4. 7.3 MB/s | 498 kB 00:00 2024-02-05T07:55:38.289 INFO:teuthology.orchestra.run.smithi134.stdout:(5/29): librgw2-19.0.0-1241.g164d3be4.el9.x86_6 18 MB/s | 5.1 MB 00:00 2024-02-05T07:55:38.355 INFO:teuthology.orchestra.run.smithi134.stdout:(6/29): python3-ceph-argparse-19.0.0-1241.g164d 1.1 MB/s | 45 kB 00:00 2024-02-05T07:55:38.373 INFO:teuthology.orchestra.run.smithi134.stdout:(7/29): python3-ceph-common-19.0.0-1241.g164d3b 3.1 MB/s | 131 kB 00:00 2024-02-05T07:55:38.423 INFO:teuthology.orchestra.run.smithi134.stdout:(8/29): python3-cephfs-19.0.0-1241.g164d3be4.el 3.2 MB/s | 162 kB 00:00 2024-02-05T07:55:38.473 INFO:teuthology.orchestra.run.smithi134.stdout:(9/29): python3-rados-19.0.0-1241.g164d3be4.el9 6.2 MB/s | 320 kB 00:00 2024-02-05T07:55:38.524 INFO:teuthology.orchestra.run.smithi134.stdout:(10/29): python3-rbd-19.0.0-1241.g164d3be4.el9. 5.8 MB/s | 298 kB 00:00 2024-02-05T07:55:38.566 INFO:teuthology.orchestra.run.smithi134.stdout:(11/29): python3-rgw-19.0.0-1241.g164d3be4.el9. 2.3 MB/s | 99 kB 00:00 2024-02-05T07:55:38.799 INFO:teuthology.orchestra.run.smithi134.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 453 kB/s | 106 kB 00:00 2024-02-05T07:55:38.950 INFO:teuthology.orchestra.run.smithi134.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-02-05T07:55:39.000 INFO:teuthology.orchestra.run.smithi134.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-02-05T07:55:39.100 INFO:teuthology.orchestra.run.smithi134.stdout:(15/29): ceph-base-19.0.0-1241.g164d3be4.el9.x8 2.6 MB/s | 5.1 MB 00:01 2024-02-05T07:55:39.142 INFO:teuthology.orchestra.run.smithi134.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 661 kB 00:00 2024-02-05T07:55:39.167 INFO:teuthology.orchestra.run.smithi134.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-02-05T07:55:39.193 INFO:teuthology.orchestra.run.smithi134.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 823 kB/s | 41 kB 00:00 2024-02-05T07:55:39.377 INFO:teuthology.orchestra.run.smithi134.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.4 MB/s | 309 kB 00:00 2024-02-05T07:55:39.418 INFO:teuthology.orchestra.run.smithi134.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 627 kB/s | 26 kB 00:00 2024-02-05T07:55:39.460 INFO:teuthology.orchestra.run.smithi134.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-02-05T07:55:39.510 INFO:teuthology.orchestra.run.smithi134.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-02-05T07:55:39.619 INFO:teuthology.orchestra.run.smithi134.stdout:(23/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 838 kB 00:00 2024-02-05T07:55:39.669 INFO:teuthology.orchestra.run.smithi134.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2024-02-05T07:55:39.820 INFO:teuthology.orchestra.run.smithi134.stdout:(25/29): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-02-05T07:55:40.253 INFO:teuthology.orchestra.run.smithi134.stdout:(26/29): libarrow-9.0.0-10.el9.x86_64.rpm 4.2 MB/s | 4.4 MB 00:01 2024-02-05T07:55:40.579 INFO:teuthology.orchestra.run.smithi134.stdout:(27/29): ceph-common-19.0.0-1241.g164d3be4.el9. 5.7 MB/s | 20 MB 00:03 2024-02-05T07:55:40.638 INFO:teuthology.orchestra.run.smithi134.stdout:(28/29): librados2-19.0.0-1241.g164d3be4.el9.x8 4.1 MB/s | 3.4 MB 00:00 2024-02-05T07:55:40.704 INFO:teuthology.orchestra.run.smithi134.stdout:(29/29): librbd1-19.0.0-1241.g164d3be4.el9.x86_ 6.9 MB/s | 3.1 MB 00:00 2024-02-05T07:55:40.707 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:55:40.707 INFO:teuthology.orchestra.run.smithi134.stdout:Total 15 MB/s | 57 MB 00:03 2024-02-05T07:55:40.708 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:55:40.739 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:55:40.740 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:55:41.214 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:55:41.214 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:55:42.518 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:55:42.592 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-02-05T07:55:42.626 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-02-05T07:55:42.693 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-02-05T07:55:43.061 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-02-05T07:55:43.092 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T07:55:43.167 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T07:55:43.253 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 6/31 2024-02-05T07:55:43.302 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T07:55:43.358 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T07:55:43.385 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-02-05T07:55:43.420 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9. 9/31 2024-02-05T07:55:43.488 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 10/31 2024-02-05T07:55:43.513 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 11/31 2024-02-05T07:55:43.578 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 11/31 2024-02-05T07:55:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-02-05T07:55:43.720 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-02-05T07:55:43.761 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-02-05T07:55:43.801 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-02-05T07:55:43.830 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-02-05T07:55:43.863 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-02-05T07:55:44.173 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-02-05T07:55:44.193 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 19/31 2024-02-05T07:55:44.283 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 19/31 2024-02-05T07:55:44.324 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 20/31 2024-02-05T07:55:44.376 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-02-05T07:55:44.831 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-02-05T07:55:44.862 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 23/31 2024-02-05T07:55:44.942 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 23/31 2024-02-05T07:55:45.018 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 24/31 2024-02-05T07:55:45.048 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x8 25/31 2024-02-05T07:55:46.496 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 20 MB 00:20 2024-02-05T07:55:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T07:55:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T07:55:47.852 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T07:55:47.898 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 27/31 2024-02-05T07:55:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 27/31 2024-02-05T07:55:47.973 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-05T07:55:47.973 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-05T07:55:47.973 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:55:48.008 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T07:55:51.409 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 8.8 kB/s | 1.7 kB 00:00 2024-02-05T07:55:53.971 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:55:53.973 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:55:53.974 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T07:55:53.974 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:55:53.974 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:55:53.974 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 9.7 M 2024-02-05T07:55:53.974 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-02-05T07:55:53.974 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.4 M 2024-02-05T07:55:53.974 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.1 M 2024-02-05T07:55:53.974 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T07:55:53.974 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options 2024-02-05T07:55:53.974 INFO:teuthology.orchestra.run.smithi196.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-05T07:55:53.974 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 5.1 M 2024-02-05T07:55:53.974 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 20 M 2024-02-05T07:55:53.974 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 25 k 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 743 k 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 498 k 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 5.1 M 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse 2024-02-05T07:55:53.975 INFO:teuthology.orchestra.run.smithi196.stdout: x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 45 k 2024-02-05T07:55:53.976 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 131 k 2024-02-05T07:55:53.976 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 162 k 2024-02-05T07:55:53.976 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-05T07:55:53.976 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 320 k 2024-02-05T07:55:53.976 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 298 k 2024-02-05T07:55:53.976 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 99 k 2024-02-05T07:55:53.976 INFO:teuthology.orchestra.run.smithi196.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-05T07:55:53.976 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-05T07:55:53.976 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:55:53.976 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:55:53.976 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:55:53.976 INFO:teuthology.orchestra.run.smithi196.stdout:Install 27 Packages 2024-02-05T07:55:53.976 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 2 Packages 2024-02-05T07:55:53.977 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:55:53.977 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 57 M 2024-02-05T07:55:53.977 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:55:54.860 INFO:teuthology.orchestra.run.smithi196.stdout:(1/29): ceph-base-19.0.0-1241.g164d3be4.el9.x86 9.6 MB/s | 5.1 MB 00:00 2024-02-05T07:55:54.910 INFO:teuthology.orchestra.run.smithi196.stdout:(2/29): ceph-selinux-19.0.0-1241.g164d3be4.el9. 505 kB/s | 25 kB 00:00 2024-02-05T07:55:54.994 INFO:teuthology.orchestra.run.smithi196.stdout:(3/29): libcephfs2-19.0.0-1241.g164d3be4.el9.x8 8.7 MB/s | 743 kB 00:00 2024-02-05T07:55:55.052 INFO:teuthology.orchestra.run.smithi196.stdout:(4/29): libradosstriper1-19.0.0-1241.g164d3be4. 8.4 MB/s | 498 kB 00:00 2024-02-05T07:55:55.411 INFO:teuthology.orchestra.run.smithi196.stdout:(5/29): librgw2-19.0.0-1241.g164d3be4.el9.x86_6 14 MB/s | 5.1 MB 00:00 2024-02-05T07:55:55.453 INFO:teuthology.orchestra.run.smithi196.stdout:(6/29): python3-ceph-argparse-19.0.0-1241.g164d 1.1 MB/s | 45 kB 00:00 2024-02-05T07:55:55.511 INFO:teuthology.orchestra.run.smithi196.stdout:(7/29): python3-ceph-common-19.0.0-1241.g164d3b 2.2 MB/s | 131 kB 00:00 2024-02-05T07:55:55.570 INFO:teuthology.orchestra.run.smithi196.stdout:(8/29): python3-cephfs-19.0.0-1241.g164d3be4.el 2.7 MB/s | 162 kB 00:00 2024-02-05T07:55:55.629 INFO:teuthology.orchestra.run.smithi196.stdout:(9/29): python3-rados-19.0.0-1241.g164d3be4.el9 5.4 MB/s | 320 kB 00:00 2024-02-05T07:55:55.704 INFO:teuthology.orchestra.run.smithi196.stdout:(10/29): python3-rbd-19.0.0-1241.g164d3be4.el9. 3.9 MB/s | 298 kB 00:00 2024-02-05T07:55:55.762 INFO:teuthology.orchestra.run.smithi196.stdout:(11/29): python3-rgw-19.0.0-1241.g164d3be4.el9. 1.7 MB/s | 99 kB 00:00 2024-02-05T07:55:55.904 INFO:teuthology.orchestra.run.smithi196.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2024-02-05T07:55:56.005 INFO:teuthology.orchestra.run.smithi196.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-02-05T07:55:56.197 INFO:teuthology.orchestra.run.smithi196.stdout:(14/29): ceph-common-19.0.0-1241.g164d3be4.el9. 11 MB/s | 20 MB 00:01 2024-02-05T07:55:56.213 INFO:teuthology.orchestra.run.smithi196.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 211 kB/s | 44 kB 00:00 2024-02-05T07:55:56.281 INFO:teuthology.orchestra.run.smithi196.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-02-05T07:55:56.306 INFO:teuthology.orchestra.run.smithi196.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-02-05T07:55:56.406 INFO:teuthology.orchestra.run.smithi196.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 330 kB/s | 41 kB 00:00 2024-02-05T07:55:56.623 INFO:teuthology.orchestra.run.smithi196.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 975 kB/s | 309 kB 00:00 2024-02-05T07:55:56.665 INFO:teuthology.orchestra.run.smithi196.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-02-05T07:55:56.707 INFO:teuthology.orchestra.run.smithi196.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-02-05T07:55:56.757 INFO:teuthology.orchestra.run.smithi196.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-02-05T07:55:56.864 INFO:teuthology.orchestra.run.smithi196.stdout:(23/29): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-02-05T07:55:56.924 INFO:teuthology.orchestra.run.smithi196.stdout:(24/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.0 MB/s | 838 kB 00:00 2024-02-05T07:55:56.949 INFO:teuthology.orchestra.run.smithi196.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-02-05T07:55:57.016 INFO:teuthology.orchestra.run.smithi196.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-02-05T07:55:57.258 INFO:teuthology.orchestra.run.smithi196.stdout:(27/29): librados2-19.0.0-1241.g164d3be4.el9.x8 11 MB/s | 3.4 MB 00:00 2024-02-05T07:55:57.350 INFO:teuthology.orchestra.run.smithi196.stdout:(28/29): librbd1-19.0.0-1241.g164d3be4.el9.x86_ 9.4 MB/s | 3.1 MB 00:00 2024-02-05T07:55:57.465 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T07:55:57.465 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-02-05T07:55:57.465 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-02-05T07:55:57.465 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-02-05T07:55:57.465 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-02-05T07:55:57.465 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-02-05T07:55:57.465 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-02-05T07:55:57.465 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-02-05T07:55:57.466 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:55:58.348 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 29/31 2024-02-05T07:55:58.373 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 29/31 2024-02-05T07:55:58.373 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:55:58.373 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-05T07:55:58.373 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T07:55:58.373 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T07:55:58.373 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:55:58.375 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T07:55:58.459 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T07:55:58.459 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T07:55:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T07:55:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 1/31 2024-02-05T07:55:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2/31 2024-02-05T07:55:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 3/31 2024-02-05T07:55:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 4/31 2024-02-05T07:55:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T07:55:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 6/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9. 8/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x8 9/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 10/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 11/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 12/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 13/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-02-05T07:55:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-02-05T07:55:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-02-05T07:55:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-02-05T07:55:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-02-05T07:55:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-02-05T07:55:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-02-05T07:55:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-02-05T07:55:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T07:55:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-02-05T07:55:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 30/31 2024-02-05T07:56:00.729 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T07:56:00.729 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:00.729 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-05T07:56:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: re2-1:20211101-3.el9.x86_64 2024-02-05T07:56:00.732 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-05T07:56:00.732 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:00.732 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:56:00.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2024-02-05T07:56:01.559 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:27 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:56:01.700 INFO:teuthology.orchestra.run.smithi196.stdout:(29/29): ceph-radosgw-19.0.0-1241.g164d3be4.el9 1.3 MB/s | 9.7 MB 00:07 2024-02-05T07:56:01.706 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:56:01.706 INFO:teuthology.orchestra.run.smithi196.stdout:Total 7.4 MB/s | 57 MB 00:07 2024-02-05T07:56:01.706 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:56:01.716 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:56:01.717 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:56:01.717 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-05T07:56:01.717 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:56:01.717 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:56:01.717 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 47 M 2024-02-05T07:56:01.717 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-05T07:56:01.717 INFO:teuthology.orchestra.run.smithi134.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-05T07:56:01.717 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 163 k 2024-02-05T07:56:01.718 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-05T07:56:01.718 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-05T07:56:01.718 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-05T07:56:01.718 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:01.718 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:56:01.718 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:56:01.718 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-02-05T07:56:01.718 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:01.719 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 48 M 2024-02-05T07:56:01.719 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 204 M 2024-02-05T07:56:01.719 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:56:01.742 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:56:01.742 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:56:01.918 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-02-05T07:56:01.951 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-02-05T07:56:02.002 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-02-05T07:56:02.018 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): libcephsqlite-19.0.0-1241.g164d3be4.el9. 545 kB/s | 163 kB 00:00 2024-02-05T07:56:02.052 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-02-05T07:56:02.214 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:56:02.214 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:56:03.538 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:56:03.617 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-02-05T07:56:03.646 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-02-05T07:56:03.695 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-02-05T07:56:04.014 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-02-05T07:56:04.061 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T07:56:04.153 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T07:56:04.211 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): ceph-test-19.0.0-1241.g164d3be4.el9.x86_ 19 MB/s | 47 MB 00:02 2024-02-05T07:56:04.213 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:56:04.214 INFO:teuthology.orchestra.run.smithi134.stdout:Total 19 MB/s | 48 MB 00:02 2024-02-05T07:56:04.214 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:56:04.235 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:56:04.235 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:56:04.240 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 6/31 2024-02-05T07:56:04.272 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T07:56:04.333 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T07:56:04.362 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-02-05T07:56:04.406 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9. 9/31 2024-02-05T07:56:04.474 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 10/31 2024-02-05T07:56:04.499 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 11/31 2024-02-05T07:56:04.564 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 11/31 2024-02-05T07:56:04.616 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:56:04.616 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:56:04.678 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-02-05T07:56:04.715 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-02-05T07:56:04.754 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-02-05T07:56:04.795 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-02-05T07:56:04.824 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-02-05T07:56:04.857 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-02-05T07:56:05.163 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-02-05T07:56:05.190 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:56:05.195 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 19/31 2024-02-05T07:56:05.257 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-05T07:56:05.285 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 19/31 2024-02-05T07:56:05.293 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-05T07:56:05.316 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 20/31 2024-02-05T07:56:05.337 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-05T07:56:05.362 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-02-05T07:56:05.382 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-05T07:56:05.406 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 5/6 2024-02-05T07:56:05.815 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-02-05T07:56:05.839 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 23/31 2024-02-05T07:56:05.894 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 23/31 2024-02-05T07:56:05.969 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 24/31 2024-02-05T07:56:06.000 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x8 25/31 2024-02-05T07:56:08.230 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T07:56:08.272 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T07:56:08.800 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T07:56:08.841 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 27/31 2024-02-05T07:56:08.933 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 27/31 2024-02-05T07:56:08.940 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-05T07:56:08.940 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-05T07:56:08.940 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:08.968 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T07:56:09.902 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 5/6 2024-02-05T07:56:09.980 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 6/6 2024-02-05T07:56:12.528 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 6/6 2024-02-05T07:56:12.528 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 1/6 2024-02-05T07:56:12.529 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 2/6 2024-02-05T07:56:12.529 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-05T07:56:12.529 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-05T07:56:12.529 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-05T07:56:12.907 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-05T07:56:12.908 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:12.908 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:56:12.908 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:12.908 INFO:teuthology.orchestra.run.smithi134.stdout: jq-1.6-16.el9.x86_64 2024-02-05T07:56:12.908 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:12.908 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-05T07:56:12.908 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-05T07:56:12.908 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-05T07:56:12.908 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:12.908 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:56:13.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2024-02-05T07:56:13.749 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:39 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:56:13.908 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:56:13.911 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-02-05T07:56:13.911 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-05T07:56:13.911 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-02-05T07:56:13.911 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:56:13.911 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 6.6 k 2024-02-05T07:56:13.911 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-05T07:56:13.911 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 2.1 M 2024-02-05T07:56:13.911 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 1.5 M 2024-02-05T07:56:13.911 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 247 k 2024-02-05T07:56:13.911 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 4.7 M 2024-02-05T07:56:13.911 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 17 M 2024-02-05T07:56:13.911 INFO:teuthology.orchestra.run.smithi134.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-05T07:56:13.911 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-05T07:56:13.912 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-05T07:56:13.912 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-05T07:56:13.912 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-05T07:56:13.912 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-05T07:56:13.912 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-05T07:56:13.912 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-05T07:56:13.912 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-05T07:56:13.912 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-05T07:56:13.912 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-05T07:56:13.912 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-05T07:56:13.912 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-05T07:56:13.912 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-05T07:56:13.912 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-05T07:56:13.913 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-05T07:56:13.914 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-05T07:56:13.914 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:13.914 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:56:13.914 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-02-05T07:56:13.914 INFO:teuthology.orchestra.run.smithi134.stdout:Install 36 Packages 2024-02-05T07:56:13.914 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:13.914 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 30 M 2024-02-05T07:56:13.915 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 107 M 2024-02-05T07:56:13.915 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:56:14.752 INFO:teuthology.orchestra.run.smithi134.stdout:(1/36): ceph-19.0.0-1241.g164d3be4.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-02-05T07:56:15.711 INFO:teuthology.orchestra.run.smithi134.stdout:(2/36): ceph-mgr-19.0.0-1241.g164d3be4.el9.x86_ 1.1 MB/s | 1.5 MB 00:01 2024-02-05T07:56:16.561 INFO:teuthology.orchestra.run.smithi134.stdout:(3/36): ceph-mds-19.0.0-1241.g164d3be4.el9.x86_ 971 kB/s | 2.1 MB 00:02 2024-02-05T07:56:16.744 INFO:teuthology.orchestra.run.smithi134.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1241.g164d 1.3 MB/s | 247 kB 00:00 2024-02-05T07:56:16.970 INFO:teuthology.orchestra.run.smithi134.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 830 kB/s | 187 kB 00:00 2024-02-05T07:56:17.020 INFO:teuthology.orchestra.run.smithi134.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-02-05T07:56:17.087 INFO:teuthology.orchestra.run.smithi134.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 475 kB/s | 32 kB 00:00 2024-02-05T07:56:17.162 INFO:teuthology.orchestra.run.smithi134.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 630 kB/s | 47 kB 00:00 2024-02-05T07:56:17.221 INFO:teuthology.orchestra.run.smithi134.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 664 kB/s | 39 kB 00:00 2024-02-05T07:56:17.279 INFO:teuthology.orchestra.run.smithi134.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-02-05T07:56:17.371 INFO:teuthology.orchestra.run.smithi134.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 13 MB/s | 1.1 MB 00:00 2024-02-05T07:56:17.413 INFO:teuthology.orchestra.run.smithi134.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-02-05T07:56:17.447 INFO:teuthology.orchestra.run.smithi134.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-02-05T07:56:17.614 INFO:teuthology.orchestra.run.smithi134.stdout:(14/36): ceph-osd-19.0.0-1241.g164d3be4.el9.x86 8.8 MB/s | 17 MB 00:01 2024-02-05T07:56:17.639 INFO:teuthology.orchestra.run.smithi134.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 589 kB/s | 113 kB 00:00 2024-02-05T07:56:17.656 INFO:teuthology.orchestra.run.smithi134.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-02-05T07:56:17.739 INFO:teuthology.orchestra.run.smithi134.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 224 kB/s | 22 kB 00:00 2024-02-05T07:56:17.873 INFO:teuthology.orchestra.run.smithi134.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 698 kB/s | 151 kB 00:00 2024-02-05T07:56:17.907 INFO:teuthology.orchestra.run.smithi134.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 260 kB/s | 43 kB 00:00 2024-02-05T07:56:17.974 INFO:teuthology.orchestra.run.smithi134.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-02-05T07:56:18.024 INFO:teuthology.orchestra.run.smithi134.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 215 kB/s | 11 kB 00:00 2024-02-05T07:56:18.091 INFO:teuthology.orchestra.run.smithi134.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.9 MB/s | 358 kB 00:00 2024-02-05T07:56:18.107 INFO:teuthology.orchestra.run.smithi134.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-02-05T07:56:18.141 INFO:teuthology.orchestra.run.smithi134.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-02-05T07:56:18.183 INFO:teuthology.orchestra.run.smithi134.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-02-05T07:56:18.200 INFO:teuthology.orchestra.run.smithi134.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 335 kB/s | 20 kB 00:00 2024-02-05T07:56:18.234 INFO:teuthology.orchestra.run.smithi134.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-02-05T07:56:18.259 INFO:teuthology.orchestra.run.smithi134.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-02-05T07:56:18.301 INFO:teuthology.orchestra.run.smithi134.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-02-05T07:56:18.318 INFO:teuthology.orchestra.run.smithi134.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-02-05T07:56:18.351 INFO:teuthology.orchestra.run.smithi134.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-02-05T07:56:18.368 INFO:teuthology.orchestra.run.smithi134.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-02-05T07:56:18.429 INFO:teuthology.orchestra.run.smithi134.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.9 MB/s | 230 kB 00:00 2024-02-05T07:56:18.472 INFO:teuthology.orchestra.run.smithi134.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-02-05T07:56:18.486 INFO:teuthology.orchestra.run.smithi134.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 353 kB/s | 20 kB 00:00 2024-02-05T07:56:18.490 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T07:56:18.490 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-02-05T07:56:18.491 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-02-05T07:56:18.491 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-02-05T07:56:18.491 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-02-05T07:56:18.491 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-02-05T07:56:18.491 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-02-05T07:56:18.491 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-02-05T07:56:18.491 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:19.036 INFO:teuthology.orchestra.run.smithi134.stdout:(36/36): ceph-mon-19.0.0-1241.g164d3be4.el9.x86 1.1 MB/s | 4.7 MB 00:04 2024-02-05T07:56:19.040 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:56:19.040 INFO:teuthology.orchestra.run.smithi134.stdout:Total 5.8 MB/s | 30 MB 00:05 2024-02-05T07:56:19.061 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:56:19.105 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:56:19.105 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:56:19.408 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 29/31 2024-02-05T07:56:19.433 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 29/31 2024-02-05T07:56:19.433 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:56:19.433 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-05T07:56:19.433 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T07:56:19.433 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T07:56:19.433 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:19.435 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T07:56:19.517 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:56:19.517 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:56:19.519 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T07:56:19.519 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T07:56:20.706 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T07:56:20.706 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 1/31 2024-02-05T07:56:20.706 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2/31 2024-02-05T07:56:20.706 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 3/31 2024-02-05T07:56:20.706 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 4/31 2024-02-05T07:56:20.706 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T07:56:20.706 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 6/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9. 8/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x8 9/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 10/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 11/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 12/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 13/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-02-05T07:56:20.707 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-02-05T07:56:20.708 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-02-05T07:56:20.708 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-02-05T07:56:20.708 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-02-05T07:56:20.708 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-02-05T07:56:20.709 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-02-05T07:56:20.709 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-02-05T07:56:20.709 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T07:56:20.709 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-02-05T07:56:20.709 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 30/31 2024-02-05T07:56:20.855 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:56:20.906 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-05T07:56:20.938 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-05T07:56:20.966 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-05T07:56:21.028 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-05T07:56:21.073 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-05T07:56:21.114 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-05T07:56:21.205 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-05T07:56:21.284 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-05T07:56:21.325 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-05T07:56:21.396 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-05T07:56:21.450 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-05T07:56:21.489 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-05T07:56:21.537 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-05T07:56:21.601 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-05T07:56:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T07:56:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:21.716 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-02-05T07:56:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-05T07:56:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-05T07:56:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-05T07:56:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-05T07:56:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-05T07:56:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: re2-1:20211101-3.el9.x86_64 2024-02-05T07:56:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-05T07:56:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:21.719 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:56:21.785 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-05T07:56:21.834 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-05T07:56:21.872 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-05T07:56:21.907 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-05T07:56:21.938 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-05T07:56:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-05T07:56:22.031 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-test 2024-02-05T07:56:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T07:56:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-05T07:56:22.138 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-05T07:56:22.173 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-05T07:56:22.208 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-05T07:56:22.288 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-05T07:56:22.330 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-05T07:56:22.402 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-05T07:56:22.539 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-05T07:56:22.600 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:31 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:56:22.656 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T07:56:22.756 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:56:22.757 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:56:22.757 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T07:56:22.757 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:56:22.757 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:56:22.757 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 47 M 2024-02-05T07:56:22.757 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T07:56:22.757 INFO:teuthology.orchestra.run.smithi196.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-05T07:56:22.757 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 163 k 2024-02-05T07:56:22.757 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-05T07:56:22.757 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-05T07:56:22.758 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-05T07:56:22.758 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:22.758 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:56:22.758 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:56:22.758 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-02-05T07:56:22.758 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:22.758 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 48 M 2024-02-05T07:56:22.759 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 204 M 2024-02-05T07:56:22.759 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:56:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9. 31/36 2024-02-05T07:56:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 32/36 2024-02-05T07:56:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 32/36 2024-02-05T07:56:22.860 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:56:22.860 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-05T07:56:22.860 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T07:56:22.860 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T07:56:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:22.959 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-02-05T07:56:23.001 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-02-05T07:56:23.043 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-02-05T07:56:23.093 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-02-05T07:56:23.210 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): libcephsqlite-19.0.0-1241.g164d3be4.el9. 362 kB/s | 163 kB 00:00 2024-02-05T07:56:24.361 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 33/36 2024-02-05T07:56:24.388 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 33/36 2024-02-05T07:56:24.388 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:56:24.388 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-05T07:56:24.388 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T07:56:24.389 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T07:56:24.389 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:24.837 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 34/36 2024-02-05T07:56:24.860 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 34/36 2024-02-05T07:56:24.860 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:56:24.860 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-05T07:56:24.860 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T07:56:24.860 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T07:56:24.861 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:25.071 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 35/36 2024-02-05T07:56:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 35/36 2024-02-05T07:56:25.095 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:56:25.095 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-05T07:56:25.095 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T07:56:25.095 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T07:56:25.095 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:25.160 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 36/36 2024-02-05T07:56:27.076 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 36/36 2024-02-05T07:56:27.076 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 1/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 2/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 3/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 4/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 5/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9. 6/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-05T07:56:27.077 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-05T07:56:27.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-05T07:56:27.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-05T07:56:27.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-05T07:56:27.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-05T07:56:27.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T07:56:27.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-05T07:56:27.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-05T07:56:27.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-05T07:56:27.078 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-05T07:56:27.079 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-05T07:56:27.079 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-05T07:56:27.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-05T07:56:27.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-05T07:56:27.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T07:56:27.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-05T07:56:27.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-05T07:56:27.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-05T07:56:27.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-05T07:56:27.080 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-05T07:56:27.957 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-05T07:56:27.957 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:27.958 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:56:27.958 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:27.958 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:27.958 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:27.958 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:56:27.958 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:27.958 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:27.958 INFO:teuthology.orchestra.run.smithi134.stdout: lua-5.4.4-4.el9.x86_64 2024-02-05T07:56:27.958 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-05T07:56:27.958 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-05T07:56:27.958 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-05T07:56:27.958 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-05T07:56:27.958 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-05T07:56:27.959 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-05T07:56:27.959 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-05T07:56:27.959 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-05T07:56:27.959 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-05T07:56:27.959 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-05T07:56:27.959 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-05T07:56:27.959 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-05T07:56:27.959 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-05T07:56:27.959 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-05T07:56:27.959 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-05T07:56:27.959 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-05T07:56:27.959 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-05T07:56:27.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-02-05T07:56:27.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-05T07:56:27.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-05T07:56:27.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-05T07:56:27.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-05T07:56:27.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-05T07:56:27.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-05T07:56:27.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-05T07:56:27.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-05T07:56:27.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-05T07:56:27.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-05T07:56:27.960 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-05T07:56:27.960 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:27.961 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:56:28.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2024-02-05T07:56:28.781 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:54 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:56:28.885 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:56:28.936 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:56:28.937 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-05T07:56:28.937 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:56:29.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2024-02-05T07:56:29.566 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:55 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:56:29.723 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:56:29.723 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:56:29.724 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-05T07:56:29.724 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:56:29.724 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:56:29.724 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 748 k 2024-02-05T07:56:29.724 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:29.724 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:56:29.724 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:56:29.724 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-05T07:56:29.724 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:29.725 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 748 k 2024-02-05T07:56:29.725 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 754 k 2024-02-05T07:56:29.725 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:56:30.194 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-19.0.0-1241.g164d3be4.el9.noarch.rpm 1.6 MB/s | 748 kB 00:00 2024-02-05T07:56:30.195 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:56:30.195 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.6 MB/s | 748 kB 00:00 2024-02-05T07:56:30.195 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:56:30.200 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:56:30.200 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:56:30.206 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:56:30.206 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:56:30.283 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:56:30.635 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:56:30.706 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:56:31.124 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:56:31.471 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:56:31.471 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:31.472 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:56:31.472 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:56:31.472 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:31.472 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:56:31.717 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2024-02-05T07:56:32.287 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:58 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:56:32.447 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:56:32.448 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:56:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-02-05T07:56:32.448 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:56:32.448 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:56:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 145 k 2024-02-05T07:56:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:32.449 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:56:32.449 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:56:32.449 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-05T07:56:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:32.449 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 145 k 2024-02-05T07:56:32.449 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 436 k 2024-02-05T07:56:32.449 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:56:32.700 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-19.0.0-1241.g164d3b 580 kB/s | 145 kB 00:00 2024-02-05T07:56:32.700 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:56:32.701 INFO:teuthology.orchestra.run.smithi134.stdout:Total 577 kB/s | 145 kB 00:00 2024-02-05T07:56:32.701 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:56:32.708 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:56:32.708 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:56:32.761 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:56:32.761 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:56:32.907 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:56:32.967 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T07:56:32.991 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T07:56:32.992 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:56:32.992 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-05T07:56:32.992 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:34.019 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): ceph-test-19.0.0-1241.g164d3be4.el9.x86_ 4.2 MB/s | 47 MB 00:11 2024-02-05T07:56:34.020 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:56:34.020 INFO:teuthology.orchestra.run.smithi196.stdout:Total 4.3 MB/s | 48 MB 00:11 2024-02-05T07:56:34.020 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:56:34.038 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:56:34.038 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:56:34.063 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T07:56:34.063 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:34.063 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:56:34.063 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:34.063 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:34.063 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:56:34.336 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2024-02-05T07:56:34.421 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:56:34.421 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:56:34.913 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:00 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:56:34.989 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:56:35.016 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:56:35.066 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-05T07:56:35.067 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:56:35.068 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-05T07:56:35.068 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:56:35.100 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-05T07:56:35.141 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2024-02-05T07:56:35.143 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-05T07:56:35.188 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-05T07:56:35.212 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 5/6 2024-02-05T07:56:35.697 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:01 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:56:35.853 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:56:35.854 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-02-05T07:56:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-05T07:56:35.854 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-02-05T07:56:35.854 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 1.8 M 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 28 k 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 15 k 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-05T07:56:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:35.856 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:56:35.856 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-02-05T07:56:35.856 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-02-05T07:56:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:35.857 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 6.8 M 2024-02-05T07:56:35.857 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-02-05T07:56:35.857 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:56:36.192 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1241.g164 60 kB/s | 15 kB 00:00 2024-02-05T07:56:36.217 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1241.g16 102 kB/s | 28 kB 00:00 2024-02-05T07:56:36.309 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1241.g164d3be 4.7 MB/s | 1.8 MB 00:00 2024-02-05T07:56:36.501 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.0 MB/s | 863 kB 00:00 2024-02-05T07:56:36.535 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 749 kB/s | 238 kB 00:00 2024-02-05T07:56:36.593 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.5 MB/s | 1.0 MB 00:00 2024-02-05T07:56:37.010 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 41 kB/s | 19 kB 00:00 2024-02-05T07:56:37.352 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 423 kB/s | 144 kB 00:00 2024-02-05T07:56:37.461 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-02-05T07:56:37.661 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 473 kB/s | 548 kB 00:01 2024-02-05T07:56:37.728 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 707 kB/s | 188 kB 00:00 2024-02-05T07:56:39.659 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 5/6 2024-02-05T07:56:39.703 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 681 kB/s | 2.0 MB 00:03 2024-02-05T07:56:39.709 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:56:39.710 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.8 MB/s | 6.8 MB 00:03 2024-02-05T07:56:39.729 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 6/6 2024-02-05T07:56:39.729 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:56:39.748 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:56:39.748 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:56:39.930 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:56:39.930 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:56:40.439 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:56:40.511 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-05T07:56:40.546 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-05T07:56:40.592 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-05T07:56:40.624 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-05T07:56:40.723 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-05T07:56:40.855 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-05T07:56:40.950 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-05T07:56:40.994 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-05T07:56:41.023 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-05T07:56:41.053 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9 10/12 2024-02-05T07:56:41.292 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el 11/12 2024-02-05T07:56:41.330 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 12/12 2024-02-05T07:56:42.082 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 6/6 2024-02-05T07:56:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 1/6 2024-02-05T07:56:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 2/6 2024-02-05T07:56:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-05T07:56:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-05T07:56:42.083 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-05T07:56:42.149 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 12/12 2024-02-05T07:56:42.149 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el 1/12 2024-02-05T07:56:42.149 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 2/12 2024-02-05T07:56:42.149 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9 3/12 2024-02-05T07:56:42.149 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-05T07:56:42.150 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-05T07:56:42.150 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-05T07:56:42.150 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-05T07:56:42.150 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-05T07:56:42.150 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-05T07:56:42.150 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-05T07:56:42.150 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-05T07:56:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-05T07:56:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:42.521 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:56:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-05T07:56:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:56:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:56:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:56:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-05T07:56:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-05T07:56:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-05T07:56:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-05T07:56:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-05T07:56:42.521 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-05T07:56:42.522 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-05T07:56:42.522 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-05T07:56:42.522 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:42.522 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:56:42.527 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-05T07:56:42.527 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:42.527 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:56:42.527 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:42.527 INFO:teuthology.orchestra.run.smithi196.stdout: jq-1.6-16.el9.x86_64 2024-02-05T07:56:42.528 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:42.528 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-05T07:56:42.528 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-05T07:56:42.528 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-05T07:56:42.528 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:42.528 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:56:42.799 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-05T07:56:42.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph 2024-02-05T07:56:43.360 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:09 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:56:43.391 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:52 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:56:43.517 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:56:43.518 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-02-05T07:56:43.518 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-05T07:56:43.518 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-02-05T07:56:43.518 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:56:43.518 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 7.4 M 2024-02-05T07:56:43.518 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-05T07:56:43.518 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-05T07:56:43.518 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-05T07:56:43.518 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-05T07:56:43.518 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-05T07:56:43.519 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-05T07:56:43.519 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-05T07:56:43.519 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-05T07:56:43.519 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-05T07:56:43.519 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-05T07:56:43.519 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-05T07:56:43.519 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-05T07:56:43.519 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:43.519 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:56:43.519 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-02-05T07:56:43.519 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-02-05T07:56:43.519 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:43.520 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 37 M 2024-02-05T07:56:43.520 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 209 M 2024-02-05T07:56:43.520 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:56:43.560 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:56:43.562 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-02-05T07:56:43.562 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T07:56:43.562 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-02-05T07:56:43.562 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:56:43.562 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 6.6 k 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 2.1 M 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 1.5 M 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 247 k 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 4.7 M 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 17 M 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-05T07:56:43.563 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-05T07:56:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-05T07:56:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-05T07:56:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-05T07:56:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-05T07:56:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-05T07:56:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-05T07:56:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-05T07:56:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-05T07:56:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-05T07:56:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-05T07:56:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-05T07:56:43.564 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout:Install 36 Packages 2024-02-05T07:56:43.565 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:43.566 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 30 M 2024-02-05T07:56:43.566 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 107 M 2024-02-05T07:56:43.566 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:56:43.815 INFO:teuthology.orchestra.run.smithi196.stdout:(1/36): ceph-19.0.0-1241.g164d3be4.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-02-05T07:56:43.833 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 199 kB/s | 32 kB 00:00 2024-02-05T07:56:43.875 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 415 kB/s | 17 kB 00:00 2024-02-05T07:56:43.926 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 697 kB/s | 34 kB 00:00 2024-02-05T07:56:43.984 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.7 MB/s | 3.0 MB 00:00 2024-02-05T07:56:44.057 INFO:teuthology.orchestra.run.smithi196.stdout:(2/36): ceph-mds-19.0.0-1241.g164d3be4.el9.x86_ 5.2 MB/s | 2.1 MB 00:00 2024-02-05T07:56:44.201 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-02-05T07:56:44.268 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-02-05T07:56:44.293 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-02-05T07:56:44.382 INFO:teuthology.orchestra.run.smithi196.stdout:(3/36): ceph-mon-19.0.0-1241.g164d3be4.el9.x86_ 8.3 MB/s | 4.7 MB 00:00 2024-02-05T07:56:44.457 INFO:teuthology.orchestra.run.smithi196.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1241.g164d 3.2 MB/s | 247 kB 00:00 2024-02-05T07:56:44.469 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 802 kB 00:00 2024-02-05T07:56:44.582 INFO:teuthology.orchestra.run.smithi196.stdout:(5/36): ceph-mgr-19.0.0-1241.g164d3be4.el9.x86_ 1.6 MB/s | 1.5 MB 00:00 2024-02-05T07:56:44.636 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-02-05T07:56:44.661 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 194 kB 00:00 2024-02-05T07:56:44.666 INFO:teuthology.orchestra.run.smithi196.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 896 kB/s | 187 kB 00:00 2024-02-05T07:56:44.699 INFO:teuthology.orchestra.run.smithi196.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2024-02-05T07:56:44.724 INFO:teuthology.orchestra.run.smithi196.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 541 kB/s | 32 kB 00:00 2024-02-05T07:56:44.750 INFO:teuthology.orchestra.run.smithi196.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 940 kB/s | 47 kB 00:00 2024-02-05T07:56:44.767 INFO:teuthology.orchestra.run.smithi196.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 921 kB/s | 39 kB 00:00 2024-02-05T07:56:44.834 INFO:teuthology.orchestra.run.smithi196.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-02-05T07:56:45.034 INFO:teuthology.orchestra.run.smithi196.stdout:(12/36): ceph-osd-19.0.0-1241.g164d3be4.el9.x86 17 MB/s | 17 MB 00:00 2024-02-05T07:56:45.067 INFO:teuthology.orchestra.run.smithi196.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 3.8 MB/s | 1.1 MB 00:00 2024-02-05T07:56:45.093 INFO:teuthology.orchestra.run.smithi196.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 398 kB/s | 103 kB 00:00 2024-02-05T07:56:45.118 INFO:teuthology.orchestra.run.smithi196.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2024-02-05T07:56:45.143 INFO:teuthology.orchestra.run.smithi196.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-05T07:56:45.168 INFO:teuthology.orchestra.run.smithi196.stdout:(17/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-02-05T07:56:45.252 INFO:teuthology.orchestra.run.smithi196.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-02-05T07:56:45.478 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 292 kB/s | 245 kB 00:00 2024-02-05T07:56:45.603 INFO:teuthology.orchestra.run.smithi196.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 100 kB/s | 43 kB 00:00 2024-02-05T07:56:45.619 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.8 MB/s | 7.4 MB 00:01 2024-02-05T07:56:45.622 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:56:45.623 INFO:teuthology.orchestra.run.smithi134.stdout:Total 18 MB/s | 37 MB 00:02 2024-02-05T07:56:45.637 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:56:45.664 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:56:45.664 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:56:45.711 INFO:teuthology.orchestra.run.smithi196.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 266 kB/s | 151 kB 00:00 2024-02-05T07:56:45.845 INFO:teuthology.orchestra.run.smithi196.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 290 kB/s | 172 kB 00:00 2024-02-05T07:56:45.870 INFO:teuthology.orchestra.run.smithi196.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 67 kB/s | 11 kB 00:00 2024-02-05T07:56:46.012 INFO:teuthology.orchestra.run.smithi196.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 106 kB/s | 18 kB 00:00 2024-02-05T07:56:46.013 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:56:46.013 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:56:46.037 INFO:teuthology.orchestra.run.smithi196.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 139 kB/s | 23 kB 00:00 2024-02-05T07:56:46.062 INFO:teuthology.orchestra.run.smithi196.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 780 kB/s | 358 kB 00:00 2024-02-05T07:56:46.179 INFO:teuthology.orchestra.run.smithi196.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 116 kB/s | 19 kB 00:00 2024-02-05T07:56:46.204 INFO:teuthology.orchestra.run.smithi196.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 117 kB/s | 20 kB 00:00 2024-02-05T07:56:46.296 INFO:teuthology.orchestra.run.smithi196.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 198 kB/s | 46 kB 00:00 2024-02-05T07:56:46.347 INFO:teuthology.orchestra.run.smithi196.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 471 kB/s | 79 kB 00:00 2024-02-05T07:56:46.414 INFO:teuthology.orchestra.run.smithi196.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2024-02-05T07:56:46.464 INFO:teuthology.orchestra.run.smithi196.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.0 MB/s | 272 kB 00:00 2024-02-05T07:56:46.506 INFO:teuthology.orchestra.run.smithi196.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 566 kB/s | 90 kB 00:00 2024-02-05T07:56:46.540 INFO:teuthology.orchestra.run.smithi196.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 284 kB/s | 36 kB 00:00 2024-02-05T07:56:46.718 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:56:46.765 INFO:teuthology.orchestra.run.smithi196.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 764 kB/s | 230 kB 00:00 2024-02-05T07:56:46.784 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-05T07:56:46.807 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-05T07:56:46.824 INFO:teuthology.orchestra.run.smithi196.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 70 kB/s | 20 kB 00:00 2024-02-05T07:56:46.902 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-05T07:56:46.935 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-05T07:56:47.024 INFO:teuthology.orchestra.run.smithi196.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 826 kB/s | 427 kB 00:00 2024-02-05T07:56:47.028 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:56:47.028 INFO:teuthology.orchestra.run.smithi196.stdout:Total 8.6 MB/s | 30 MB 00:03 2024-02-05T07:56:47.048 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:56:47.092 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:56:47.092 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:56:47.113 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T07:56:47.142 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T07:56:47.219 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-05T07:56:47.500 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:56:47.500 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:56:47.891 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-05T07:56:48.013 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-05T07:56:48.876 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:56:48.935 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-05T07:56:48.975 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-05T07:56:49.004 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-05T07:56:49.065 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-05T07:56:49.103 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-05T07:56:49.135 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-05T07:56:49.228 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-05T07:56:49.273 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-05T07:56:49.306 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-05T07:56:49.347 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-05T07:56:49.419 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-05T07:56:49.472 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-05T07:56:49.511 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-05T07:56:49.560 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-05T07:56:49.624 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-05T07:56:49.806 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-05T07:56:49.857 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-05T07:56:49.890 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-05T07:56:49.921 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-05T07:56:49.951 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-05T07:56:50.035 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-05T07:56:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T07:56:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-05T07:56:50.152 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-05T07:56:50.188 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-05T07:56:50.222 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-05T07:56:50.310 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-05T07:56:50.314 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-05T07:56:50.341 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 12/12 2024-02-05T07:56:50.344 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-05T07:56:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-05T07:56:50.552 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-05T07:56:50.681 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T07:56:50.836 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9. 31/36 2024-02-05T07:56:50.866 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 32/36 2024-02-05T07:56:50.892 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 32/36 2024-02-05T07:56:50.892 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:56:50.892 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-05T07:56:50.892 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T07:56:50.892 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T07:56:50.892 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:52.334 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 33/36 2024-02-05T07:56:52.361 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 33/36 2024-02-05T07:56:52.361 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:56:52.361 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-05T07:56:52.361 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T07:56:52.361 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T07:56:52.361 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:52.793 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 34/36 2024-02-05T07:56:52.816 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 34/36 2024-02-05T07:56:52.816 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:56:52.817 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-05T07:56:52.817 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T07:56:52.817 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T07:56:52.817 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:52.858 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 12/12 2024-02-05T07:56:52.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 1/12 2024-02-05T07:56:52.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-05T07:56:52.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-05T07:56:52.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-05T07:56:52.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T07:56:52.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T07:56:52.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-05T07:56:52.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-05T07:56:52.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-05T07:56:52.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-05T07:56:52.859 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-05T07:56:53.027 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 35/36 2024-02-05T07:56:53.050 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 35/36 2024-02-05T07:56:53.050 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:56:53.050 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-05T07:56:53.051 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T07:56:53.051 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T07:56:53.051 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:53.116 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 36/36 2024-02-05T07:56:53.348 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-05T07:56:53.348 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:53.349 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:56:53.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2024-02-05T07:56:54.377 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:20 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:56:54.542 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:56:54.544 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-02-05T07:56:54.544 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-05T07:56:54.544 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-02-05T07:56:54.544 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:56:54.544 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 52 k 2024-02-05T07:56:54.544 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-05T07:56:54.544 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-05T07:56:54.544 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-05T07:56:54.544 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-05T07:56:54.544 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-05T07:56:54.544 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-05T07:56:54.545 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-05T07:56:54.545 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-05T07:56:54.545 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-05T07:56:54.545 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-05T07:56:54.545 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-05T07:56:54.545 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-05T07:56:54.545 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-05T07:56:54.545 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:54.545 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:56:54.545 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-02-05T07:56:54.545 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-02-05T07:56:54.545 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:54.546 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.0 M 2024-02-05T07:56:54.546 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-02-05T07:56:54.546 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:56:54.781 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 152 kB/s | 24 kB 00:00 2024-02-05T07:56:54.806 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 97 kB/s | 18 kB 00:00 2024-02-05T07:56:54.864 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 36/36 2024-02-05T07:56:54.864 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 1/36 2024-02-05T07:56:54.864 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 2/36 2024-02-05T07:56:54.864 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 3/36 2024-02-05T07:56:54.864 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 4/36 2024-02-05T07:56:54.864 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 5/36 2024-02-05T07:56:54.864 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9. 6/36 2024-02-05T07:56:54.864 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-05T07:56:54.864 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-05T07:56:54.864 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-05T07:56:54.865 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-05T07:56:54.866 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-05T07:56:54.866 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-05T07:56:54.866 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-05T07:56:54.866 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-05T07:56:54.866 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-05T07:56:54.866 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-05T07:56:54.866 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T07:56:54.866 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-05T07:56:54.866 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-05T07:56:54.866 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-05T07:56:54.866 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-05T07:56:54.866 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-05T07:56:54.867 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): ceph-mgr-rook-19.0.0-1241.g164d3be4.el9 214 kB/s | 52 kB 00:00 2024-02-05T07:56:54.907 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 174 kB 00:00 2024-02-05T07:56:54.941 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 989 kB/s | 132 kB 00:00 2024-02-05T07:56:54.991 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 211 kB 00:00 2024-02-05T07:56:55.016 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 393 kB/s | 43 kB 00:00 2024-02-05T07:56:55.042 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-02-05T07:56:55.075 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 179 kB/s | 15 kB 00:00 2024-02-05T07:56:55.142 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 888 kB/s | 59 kB 00:00 2024-02-05T07:56:55.167 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 1.3 MB/s | 203 kB 00:00 2024-02-05T07:56:55.201 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-02-05T07:56:55.218 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-02-05T07:56:55.220 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:56:55.220 INFO:teuthology.orchestra.run.smithi134.stdout:Total 3.0 MB/s | 2.0 MB 00:00 2024-02-05T07:56:55.221 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:56:55.241 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:56:55.241 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:56:55.350 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:56:55.350 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:56:55.677 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-05T07:56:55.677 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:55.677 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:56:55.677 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:55.677 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:55.677 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: lua-5.4.4-4.el9.x86_64 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-05T07:56:55.678 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-05T07:56:55.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-05T07:56:55.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-05T07:56:55.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-05T07:56:55.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-05T07:56:55.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-05T07:56:55.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-05T07:56:55.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-05T07:56:55.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-02-05T07:56:55.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-05T07:56:55.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-05T07:56:55.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-05T07:56:55.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-05T07:56:55.679 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-05T07:56:55.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-05T07:56:55.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-05T07:56:55.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-05T07:56:55.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-05T07:56:55.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-05T07:56:55.680 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-05T07:56:55.680 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:55.680 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:56:55.864 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:56:56.005 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base 2024-02-05T07:56:56.031 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-05T07:56:56.085 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-05T07:56:56.140 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-05T07:56:56.178 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-05T07:56:56.219 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-05T07:56:56.312 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-05T07:56:56.413 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-05T07:56:56.462 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-05T07:56:56.572 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:05 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:56:56.676 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:56:56.728 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:56:56.729 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T07:56:56.729 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:56:56.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install cephadm 2024-02-05T07:56:57.181 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-05T07:56:57.226 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-05T07:56:57.260 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-05T07:56:57.308 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-05T07:56:57.330 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 13/13 2024-02-05T07:56:57.382 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:06 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:56:57.539 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:56:57.540 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:56:57.540 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T07:56:57.540 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:56:57.540 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:56:57.540 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 748 k 2024-02-05T07:56:57.540 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:57.540 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:56:57.540 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:56:57.540 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T07:56:57.541 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:57.541 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 748 k 2024-02-05T07:56:57.541 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 754 k 2024-02-05T07:56:57.541 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:56:58.076 INFO:teuthology.orchestra.run.smithi196.stdout:cephadm-19.0.0-1241.g164d3be4.el9.noarch.rpm 1.4 MB/s | 748 kB 00:00 2024-02-05T07:56:58.077 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:56:58.077 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.4 MB/s | 748 kB 00:00 2024-02-05T07:56:58.077 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:56:58.082 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:56:58.082 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:56:58.088 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:56:58.088 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:56:58.127 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 13/13 2024-02-05T07:56:58.127 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 1/13 2024-02-05T07:56:58.127 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-05T07:56:58.127 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-05T07:56:58.127 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-05T07:56:58.127 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-05T07:56:58.127 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-05T07:56:58.127 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-05T07:56:58.127 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-05T07:56:58.127 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-05T07:56:58.128 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-05T07:56:58.128 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-05T07:56:58.128 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-05T07:56:58.165 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:56:58.510 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-05T07:56:58.510 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:58.510 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:56:58.510 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:56:58.510 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-05T07:56:58.510 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-05T07:56:58.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-05T07:56:58.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-05T07:56:58.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-05T07:56:58.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-05T07:56:58.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-05T07:56:58.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-05T07:56:58.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-05T07:56:58.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-05T07:56:58.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-05T07:56:58.511 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-05T07:56:58.511 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:58.511 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:56:58.643 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:56:58.731 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:56:58.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2024-02-05T07:56:59.136 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:56:59.352 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:25 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:56:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T07:56:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:59.465 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:56:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:56:59.465 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:56:59.465 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:56:59.508 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:56:59.508 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 138 k 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-02-05T07:56:59.509 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-02-05T07:56:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:56:59.510 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 6.8 M 2024-02-05T07:56:59.510 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 33 M 2024-02-05T07:56:59.510 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:56:59.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-immutable-object-cache 2024-02-05T07:57:00.052 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.1 MB/s | 228 kB 00:00 2024-02-05T07:57:00.219 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-02-05T07:57:00.270 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:09 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:00.302 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1241.g164d3be4.e 306 kB/s | 138 kB 00:00 2024-02-05T07:57:00.430 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:00.431 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:00.431 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-02-05T07:57:00.431 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:00.431 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:57:00.431 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 145 k 2024-02-05T07:57:00.431 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:00.431 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:57:00.431 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:00.431 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T07:57:00.431 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:00.432 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 145 k 2024-02-05T07:57:00.432 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 436 k 2024-02-05T07:57:00.432 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:57:00.553 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 173 kB/s | 58 kB 00:00 2024-02-05T07:57:00.586 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-02-05T07:57:00.603 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 170 kB/s | 51 kB 00:00 2024-02-05T07:57:00.605 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:00.605 INFO:teuthology.orchestra.run.smithi134.stdout:Total 6.2 MB/s | 6.8 MB 00:01 2024-02-05T07:57:00.606 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:57:00.615 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:57:00.615 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:57:00.657 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-immutable-object-cache-19.0.0-1241.g164d3b 644 kB/s | 145 kB 00:00 2024-02-05T07:57:00.658 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:00.658 INFO:teuthology.orchestra.run.smithi196.stdout:Total 640 kB/s | 145 kB 00:00 2024-02-05T07:57:00.658 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:57:00.666 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:57:00.666 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:57:00.705 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:57:00.707 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:57:00.718 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:57:00.718 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:57:00.866 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:57:00.933 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T07:57:00.958 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T07:57:00.958 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:57:00.958 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-05T07:57:00.958 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:01.014 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:57:01.174 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-05T07:57:01.237 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-05T07:57:01.744 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-05T07:57:01.830 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-05T07:57:01.892 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-02-05T07:57:01.917 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 6/6 2024-02-05T07:57:01.981 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T07:57:01.981 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:01.981 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:57:01.981 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:57:01.981 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:01.981 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:02.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr 2024-02-05T07:57:02.765 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 6/6 2024-02-05T07:57:02.766 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/6 2024-02-05T07:57:02.766 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-05T07:57:02.766 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-02-05T07:57:02.766 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-05T07:57:02.766 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-05T07:57:02.794 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:11 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:02.899 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:02.949 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:02.951 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T07:57:02.951 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:03.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-dashboard 2024-02-05T07:57:03.072 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-05T07:57:03.073 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:03.073 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:57:03.073 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:57:03.073 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-05T07:57:03.073 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-05T07:57:03.073 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-02-05T07:57:03.073 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-05T07:57:03.073 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-05T07:57:03.073 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:03.073 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:03.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2024-02-05T07:57:03.579 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:12 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:03.735 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:03.737 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-02-05T07:57:03.737 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T07:57:03.737 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-02-05T07:57:03.737 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:57:03.737 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 1.8 M 2024-02-05T07:57:03.737 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T07:57:03.737 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-05T07:57:03.737 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 28 k 2024-02-05T07:57:03.737 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 15 k 2024-02-05T07:57:03.737 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-05T07:57:03.738 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-05T07:57:03.738 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-05T07:57:03.738 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-05T07:57:03.738 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-05T07:57:03.738 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-05T07:57:03.738 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-05T07:57:03.738 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-05T07:57:03.738 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:03.738 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:57:03.738 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-02-05T07:57:03.738 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-02-05T07:57:03.738 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:03.739 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 6.8 M 2024-02-05T07:57:03.739 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-02-05T07:57:03.739 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:57:03.894 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:29 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:04.051 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:04.052 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-05T07:57:04.052 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:04.052 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:57:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 885 k 2024-02-05T07:57:04.052 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-05T07:57:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-05T07:57:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:04.052 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:57:04.052 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:04.052 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-02-05T07:57:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:04.053 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 964 k 2024-02-05T07:57:04.053 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.8 M 2024-02-05T07:57:04.053 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:57:04.205 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1241.g16 154 kB/s | 28 kB 00:00 2024-02-05T07:57:04.229 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 446 kB/s | 79 kB 00:00 2024-02-05T07:57:04.315 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1241.g164 52 kB/s | 15 kB 00:00 2024-02-05T07:57:04.405 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-fuse-19.0.0-1241.g164d3be4.el9.x86_ 2.5 MB/s | 885 kB 00:00 2024-02-05T07:57:04.406 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:04.406 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.7 MB/s | 964 kB 00:00 2024-02-05T07:57:04.406 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:57:04.414 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1241.g164d3be 4.5 MB/s | 1.8 MB 00:00 2024-02-05T07:57:04.426 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:57:04.427 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:57:04.504 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:57:04.504 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:57:04.506 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.4 MB/s | 1.0 MB 00:00 2024-02-05T07:57:04.532 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 238 kB 00:00 2024-02-05T07:57:04.689 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:57:04.715 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 106 kB/s | 19 kB 00:00 2024-02-05T07:57:04.749 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.5 MB/s | 863 kB 00:00 2024-02-05T07:57:04.799 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.8 MB/s | 548 kB 00:00 2024-02-05T07:57:04.831 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-05T07:57:04.883 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 12 MB/s | 2.0 MB 00:00 2024-02-05T07:57:04.896 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2/2 2024-02-05T07:57:04.908 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 910 kB/s | 144 kB 00:00 2024-02-05T07:57:04.933 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 230 kB/s | 31 kB 00:00 2024-02-05T07:57:04.959 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-02-05T07:57:04.962 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:04.963 INFO:teuthology.orchestra.run.smithi196.stdout:Total 5.6 MB/s | 6.8 MB 00:01 2024-02-05T07:57:04.980 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:57:04.997 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:57:04.997 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:57:05.177 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:57:05.177 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:57:05.650 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2/2 2024-02-05T07:57:05.650 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/2 2024-02-05T07:57:05.688 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:57:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-05T07:57:05.811 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-05T07:57:05.857 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-05T07:57:05.888 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-05T07:57:05.958 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-05T07:57:05.958 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:05.958 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:57:05.958 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-05T07:57:05.958 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:05.958 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:05.996 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-05T07:57:06.137 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-05T07:57:06.240 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-05T07:57:06.261 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-volume 2024-02-05T07:57:06.284 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-05T07:57:06.312 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-05T07:57:06.343 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9 10/12 2024-02-05T07:57:06.584 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el 11/12 2024-02-05T07:57:06.620 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 12/12 2024-02-05T07:57:06.844 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:32 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:06.999 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:07.000 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:07.000 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-02-05T07:57:07.000 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:07.000 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:57:07.000 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 263 k 2024-02-05T07:57:07.001 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-02-05T07:57:07.001 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-05T07:57:07.001 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:07.001 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:57:07.001 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:07.001 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-02-05T07:57:07.001 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:07.001 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 332 k 2024-02-05T07:57:07.001 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.5 M 2024-02-05T07:57:07.001 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:57:07.142 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 489 kB/s | 69 kB 00:00 2024-02-05T07:57:07.250 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-volume-19.0.0-1241.g164d3be4.el9.no 1.0 MB/s | 263 kB 00:00 2024-02-05T07:57:07.252 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:07.252 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-02-05T07:57:07.252 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:57:07.260 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:57:07.260 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:57:07.299 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:57:07.300 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:57:07.417 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 12/12 2024-02-05T07:57:07.417 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el 1/12 2024-02-05T07:57:07.417 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 2/12 2024-02-05T07:57:07.418 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9 3/12 2024-02-05T07:57:07.418 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-05T07:57:07.418 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-05T07:57:07.418 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-05T07:57:07.418 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-05T07:57:07.418 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-05T07:57:07.418 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-05T07:57:07.418 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-05T07:57:07.418 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-05T07:57:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:57:07.639 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-05T07:57:07.678 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2/2 2024-02-05T07:57:07.693 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2/2 2024-02-05T07:57:07.694 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:57:07.694 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-05T07:57:07.694 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:07.779 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-05T07:57:07.779 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:07.779 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:57:07.779 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-05T07:57:07.779 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:57:07.779 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:57:07.779 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:57:07.779 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-05T07:57:07.779 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-05T07:57:07.779 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-05T07:57:07.779 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-05T07:57:07.780 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-05T07:57:07.780 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-05T07:57:07.780 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-05T07:57:07.780 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-05T07:57:07.780 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:07.780 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:08.028 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-05T07:57:08.463 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 1/2 2024-02-05T07:57:08.597 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:17 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:08.754 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:08.755 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-02-05T07:57:08.755 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T07:57:08.755 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-02-05T07:57:08.755 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:57:08.755 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 7.4 M 2024-02-05T07:57:08.756 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T07:57:08.756 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-05T07:57:08.756 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-05T07:57:08.756 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-05T07:57:08.756 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-05T07:57:08.756 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-05T07:57:08.756 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-05T07:57:08.756 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-05T07:57:08.756 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-05T07:57:08.756 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-05T07:57:08.756 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-05T07:57:08.756 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-05T07:57:08.757 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:08.757 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:57:08.757 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-02-05T07:57:08.757 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-02-05T07:57:08.757 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:08.757 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 37 M 2024-02-05T07:57:08.757 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 209 M 2024-02-05T07:57:08.758 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:57:08.823 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-05T07:57:08.824 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:08.824 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:57:08.824 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:57:08.824 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-05T07:57:08.824 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:08.824 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:08.949 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 209 kB/s | 32 kB 00:00 2024-02-05T07:57:08.982 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 523 kB/s | 17 kB 00:00 2024-02-05T07:57:09.016 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-02-05T07:57:09.083 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-02-05T07:57:09.168 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2024-02-05T07:57:09.275 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-02-05T07:57:09.350 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-02-05T07:57:09.375 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-02-05T07:57:09.476 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-12 11 MB/s | 7.4 MB 00:00 2024-02-05T07:57:09.576 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 802 kB 00:00 2024-02-05T07:57:09.747 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:35 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:09.768 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-02-05T07:57:09.818 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 572 kB/s | 194 kB 00:00 2024-02-05T07:57:09.903 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:09.904 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:09.904 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-02-05T07:57:09.904 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:09.904 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:57:09.904 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 127 k 2024-02-05T07:57:09.905 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:09.905 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:57:09.905 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:09.905 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-05T07:57:09.905 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:09.905 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 127 k 2024-02-05T07:57:09.905 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 456 k 2024-02-05T07:57:09.905 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:57:10.216 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-19.0.0-1241.g164d3be4.el9.x86_64 408 kB/s | 127 kB 00:00 2024-02-05T07:57:10.216 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:10.217 INFO:teuthology.orchestra.run.smithi134.stdout:Total 406 kB/s | 127 kB 00:00 2024-02-05T07:57:10.217 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:57:10.222 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:57:10.222 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:57:10.279 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:57:10.279 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:57:10.293 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 343 kB/s | 245 kB 00:00 2024-02-05T07:57:10.297 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:10.297 INFO:teuthology.orchestra.run.smithi196.stdout:Total 24 MB/s | 37 MB 00:01 2024-02-05T07:57:10.311 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:57:10.335 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:57:10.405 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:57:10.421 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:57:10.539 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:10.690 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:57:10.691 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:57:11.016 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:11.347 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:11.348 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:11.348 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:57:11.348 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:57:11.348 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:11.348 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:11.397 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:57:11.475 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-05T07:57:11.506 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-05T07:57:11.596 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-05T07:57:11.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2024-02-05T07:57:11.643 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-05T07:57:11.819 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T07:57:11.850 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T07:57:11.926 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-05T07:57:12.217 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:38 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:12.325 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:12.377 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:12.378 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-05T07:57:12.378 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:12.451 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2024-02-05T07:57:12.585 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-05T07:57:12.685 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-05T07:57:13.013 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:39 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:13.170 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:13.170 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:13.170 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-02-05T07:57:13.170 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:13.170 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:57:13.170 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 31 k 2024-02-05T07:57:13.171 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:13.171 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:57:13.171 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:13.171 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-05T07:57:13.171 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:13.171 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 31 k 2024-02-05T07:57:13.171 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 140 k 2024-02-05T07:57:13.171 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:57:13.335 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-19.0.0-1241.g164d3be4.el9.x86_6 190 kB/s | 31 kB 00:00 2024-02-05T07:57:13.336 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:13.336 INFO:teuthology.orchestra.run.smithi134.stdout:Total 189 kB/s | 31 kB 00:00 2024-02-05T07:57:13.336 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:57:13.339 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:57:13.340 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:57:13.368 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:57:13.369 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:57:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:57:13.567 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:13.909 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-05T07:57:13.976 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:14.262 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:14.262 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:14.262 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:57:14.262 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:57:14.262 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:14.263 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:14.482 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2024-02-05T07:57:14.937 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-05T07:57:14.967 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 12/12 2024-02-05T07:57:15.050 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:41 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:15.153 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:15.206 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:15.207 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-05T07:57:15.207 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:15.284 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2024-02-05T07:57:15.856 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:41 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:15.960 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:16.012 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:16.013 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-05T07:57:16.013 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:16.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2024-02-05T07:57:16.666 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:42 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:16.770 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:16.824 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:16.825 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-05T07:57:16.825 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:16.905 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2024-02-05T07:57:17.478 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:43 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:17.569 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 12/12 2024-02-05T07:57:17.569 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 1/12 2024-02-05T07:57:17.569 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-05T07:57:17.569 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-05T07:57:17.569 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-05T07:57:17.569 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T07:57:17.569 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T07:57:17.570 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-05T07:57:17.570 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-05T07:57:17.570 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-05T07:57:17.570 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-05T07:57:17.570 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-05T07:57:17.582 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:17.635 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:17.636 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-05T07:57:17.636 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:17.722 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2024-02-05T07:57:18.067 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-05T07:57:18.067 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:18.067 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:57:18.067 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:57:18.067 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-05T07:57:18.067 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-05T07:57:18.067 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-05T07:57:18.067 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-05T07:57:18.067 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-05T07:57:18.067 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-05T07:57:18.067 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-05T07:57:18.067 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-05T07:57:18.068 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-05T07:57:18.068 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-05T07:57:18.068 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-05T07:57:18.068 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:18.068 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:18.283 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:44 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:18.388 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:18.415 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-rook 2024-02-05T07:57:18.440 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:18.441 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-05T07:57:18.441 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:18.517 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2024-02-05T07:57:18.995 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:27 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:19.085 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:45 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:19.151 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:19.153 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-02-05T07:57:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T07:57:19.153 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-02-05T07:57:19.153 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:57:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 52 k 2024-02-05T07:57:19.153 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T07:57:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-05T07:57:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-05T07:57:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-05T07:57:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-05T07:57:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-05T07:57:19.153 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-05T07:57:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-05T07:57:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-05T07:57:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-05T07:57:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-05T07:57:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-05T07:57:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-05T07:57:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:19.154 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:57:19.154 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-02-05T07:57:19.154 INFO:teuthology.orchestra.run.smithi196.stdout:Install 13 Packages 2024-02-05T07:57:19.154 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:19.155 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.0 M 2024-02-05T07:57:19.155 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-02-05T07:57:19.155 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:57:19.188 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:19.241 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:19.242 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-02-05T07:57:19.242 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:19.329 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2024-02-05T07:57:19.501 INFO:teuthology.orchestra.run.smithi196.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 175 kB/s | 24 kB 00:00 2024-02-05T07:57:19.527 INFO:teuthology.orchestra.run.smithi196.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 110 kB/s | 18 kB 00:00 2024-02-05T07:57:19.585 INFO:teuthology.orchestra.run.smithi196.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-02-05T07:57:19.611 INFO:teuthology.orchestra.run.smithi196.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-02-05T07:57:19.636 INFO:teuthology.orchestra.run.smithi196.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-05T07:57:19.661 INFO:teuthology.orchestra.run.smithi196.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-02-05T07:57:19.686 INFO:teuthology.orchestra.run.smithi196.stdout:(7/13): ceph-mgr-rook-19.0.0-1241.g164d3be4.el9 161 kB/s | 52 kB 00:00 2024-02-05T07:57:19.820 INFO:teuthology.orchestra.run.smithi196.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2024-02-05T07:57:19.845 INFO:teuthology.orchestra.run.smithi196.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 81 kB/s | 15 kB 00:00 2024-02-05T07:57:19.902 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:45 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:19.921 INFO:teuthology.orchestra.run.smithi196.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 1.1 MB/s | 203 kB 00:00 2024-02-05T07:57:19.946 INFO:teuthology.orchestra.run.smithi196.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 591 kB/s | 59 kB 00:00 2024-02-05T07:57:19.988 INFO:teuthology.orchestra.run.smithi196.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-02-05T07:57:20.013 INFO:teuthology.orchestra.run.smithi196.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 977 kB/s | 90 kB 00:00 2024-02-05T07:57:20.018 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:20.018 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-02-05T07:57:20.018 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:57:20.031 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:57:20.031 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:57:20.059 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:20.060 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:20.060 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-02-05T07:57:20.060 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:20.060 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:57:20.060 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 85 k 2024-02-05T07:57:20.060 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:20.060 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:57:20.060 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:20.060 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-05T07:57:20.060 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:20.061 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 85 k 2024-02-05T07:57:20.061 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 231 k 2024-02-05T07:57:20.061 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:57:20.130 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:57:20.130 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:57:20.271 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-19.0.0-1241.g164d3be4.el9.x86_64.rpm 406 kB/s | 85 kB 00:00 2024-02-05T07:57:20.272 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:20.272 INFO:teuthology.orchestra.run.smithi134.stdout:Total 403 kB/s | 85 kB 00:00 2024-02-05T07:57:20.272 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:57:20.279 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:57:20.279 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:57:20.334 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:57:20.335 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:57:20.529 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:57:20.637 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:20.664 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:57:20.835 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-05T07:57:20.890 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-05T07:57:20.954 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-05T07:57:20.990 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-05T07:57:21.031 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:21.031 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-05T07:57:21.126 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-05T07:57:21.233 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-05T07:57:21.283 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-05T07:57:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:21.336 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:21.337 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:57:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:57:21.337 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:21.337 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:21.603 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-02-05T07:57:22.002 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-05T07:57:22.047 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-05T07:57:22.081 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-05T07:57:22.130 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-05T07:57:22.151 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 13/13 2024-02-05T07:57:22.176 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:48 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:22.335 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:22.336 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:22.336 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-02-05T07:57:22.336 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:22.336 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:57:22.336 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.1 M 2024-02-05T07:57:22.336 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:22.336 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:57:22.336 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:22.336 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-05T07:57:22.337 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:22.337 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.1 M 2024-02-05T07:57:22.337 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 13 M 2024-02-05T07:57:22.337 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:57:22.724 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-19.0.0-1241.g164d3be4.el9.x86_64.rpm 8.0 MB/s | 3.1 MB 00:00 2024-02-05T07:57:22.724 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:22.725 INFO:teuthology.orchestra.run.smithi134.stdout:Total 7.9 MB/s | 3.1 MB 00:00 2024-02-05T07:57:22.725 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:57:22.734 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:57:22.734 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:57:22.807 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:57:22.807 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:57:23.004 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 13/13 2024-02-05T07:57:23.004 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 1/13 2024-02-05T07:57:23.004 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-05T07:57:23.004 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-05T07:57:23.004 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-05T07:57:23.005 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-05T07:57:23.005 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-05T07:57:23.005 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-05T07:57:23.005 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-05T07:57:23.005 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-05T07:57:23.005 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-05T07:57:23.005 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-05T07:57:23.005 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-05T07:57:23.234 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:57:23.300 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:23.323 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:23.323 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:57:23.323 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-05T07:57:23.323 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T07:57:23.323 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T07:57:23.323 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:23.403 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-05T07:57:23.403 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:23.404 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:57:23.404 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:57:23.404 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-05T07:57:23.404 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-05T07:57:23.404 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-05T07:57:23.404 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-05T07:57:23.404 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-05T07:57:23.404 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-05T07:57:23.404 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-05T07:57:23.404 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-05T07:57:23.404 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-05T07:57:23.405 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-05T07:57:23.405 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-05T07:57:23.405 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-05T07:57:23.405 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:23.405 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:23.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-cephadm 2024-02-05T07:57:24.217 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:33 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:24.375 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:24.375 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-02-05T07:57:24.375 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T07:57:24.375 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-02-05T07:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 138 k 2024-02-05T07:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T07:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-05T07:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-05T07:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-02-05T07:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-05T07:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-05T07:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-02-05T07:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-02-05T07:57:24.376 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:24.377 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 6.8 M 2024-02-05T07:57:24.377 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 33 M 2024-02-05T07:57:24.377 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:57:24.554 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:24.554 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:24.554 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:57:24.554 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:57:24.554 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:24.554 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:24.798 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2024-02-05T07:57:24.849 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 780 kB/s | 228 kB 00:00 2024-02-05T07:57:25.049 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2024-02-05T07:57:25.075 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1241.g164d3be4.e 266 kB/s | 138 kB 00:00 2024-02-05T07:57:25.150 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 576 kB/s | 58 kB 00:00 2024-02-05T07:57:25.175 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-02-05T07:57:25.192 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 436 kB/s | 51 kB 00:00 2024-02-05T07:57:25.194 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:25.194 INFO:teuthology.orchestra.run.smithi196.stdout:Total 8.3 MB/s | 6.8 MB 00:00 2024-02-05T07:57:25.194 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:57:25.203 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:57:25.203 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:57:25.294 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:57:25.294 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:57:25.387 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:51 ago on Mon 05 Feb 2024 07:55:34 AM UTC. 2024-02-05T07:57:25.552 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-02-05T07:57:25.553 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:25.553 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-02-05T07:57:25.553 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:25.553 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-02-05T07:57:25.553 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 171 k 2024-02-05T07:57:25.553 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:25.553 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-02-05T07:57:25.553 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-02-05T07:57:25.553 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-02-05T07:57:25.553 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:25.553 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 171 k 2024-02-05T07:57:25.553 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 494 k 2024-02-05T07:57:25.553 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-02-05T07:57:25.590 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:57:25.756 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-05T07:57:25.810 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-05T07:57:25.911 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-19.0.0-1241.g164d3be4.el9.x86_64.rpm 479 kB/s | 171 kB 00:00 2024-02-05T07:57:25.912 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:25.912 INFO:teuthology.orchestra.run.smithi134.stdout:Total 476 kB/s | 171 kB 00:00 2024-02-05T07:57:25.912 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-02-05T07:57:25.921 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-02-05T07:57:25.921 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-02-05T07:57:25.979 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-02-05T07:57:25.979 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-02-05T07:57:26.124 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-02-05T07:57:26.235 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-05T07:57:26.395 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-05T07:57:26.466 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-02-05T07:57:26.489 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 6/6 2024-02-05T07:57:26.712 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:27.053 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:27.054 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:27.054 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-02-05T07:57:27.054 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:57:27.054 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:57:27.054 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-02-05T07:57:27.291 DEBUG:teuthology.parallel:result is None 2024-02-05T07:57:27.349 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 6/6 2024-02-05T07:57:27.349 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/6 2024-02-05T07:57:27.349 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-05T07:57:27.349 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-02-05T07:57:27.349 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-05T07:57:27.349 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-05T07:57:27.659 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-05T07:57:27.659 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:27.659 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:57:27.659 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:57:27.659 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-05T07:57:27.659 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-05T07:57:27.660 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-02-05T07:57:27.660 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-05T07:57:27.660 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-05T07:57:27.660 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:27.660 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:27.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-fuse 2024-02-05T07:57:28.463 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:37 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:28.620 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:28.621 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:28.621 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T07:57:28.621 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:28.621 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:57:28.621 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 885 k 2024-02-05T07:57:28.621 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T07:57:28.621 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-05T07:57:28.621 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:28.621 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:57:28.621 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:28.621 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-02-05T07:57:28.622 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:28.622 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 964 k 2024-02-05T07:57:28.622 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.8 M 2024-02-05T07:57:28.622 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:57:28.769 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 538 kB/s | 79 kB 00:00 2024-02-05T07:57:28.927 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-fuse-19.0.0-1241.g164d3be4.el9.x86_ 2.8 MB/s | 885 kB 00:00 2024-02-05T07:57:28.929 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:28.929 INFO:teuthology.orchestra.run.smithi196.stdout:Total 3.1 MB/s | 964 kB 00:00 2024-02-05T07:57:28.929 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:57:28.949 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:57:28.949 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:57:29.027 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:57:29.027 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:57:29.223 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:57:29.353 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-05T07:57:29.428 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2/2 2024-02-05T07:57:30.141 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2/2 2024-02-05T07:57:30.142 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/2 2024-02-05T07:57:30.485 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-05T07:57:30.485 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:30.485 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:57:30.485 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-05T07:57:30.485 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:30.485 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:30.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-volume 2024-02-05T07:57:31.335 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:40 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:31.492 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:31.493 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:31.493 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-02-05T07:57:31.493 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:31.493 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:57:31.493 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 263 k 2024-02-05T07:57:31.493 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-02-05T07:57:31.493 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-05T07:57:31.493 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:31.493 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:57:31.493 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:31.493 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-02-05T07:57:31.494 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:31.494 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 332 k 2024-02-05T07:57:31.494 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 1.5 M 2024-02-05T07:57:31.494 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:57:31.725 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 297 kB/s | 69 kB 00:00 2024-02-05T07:57:31.975 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-volume-19.0.0-1241.g164d3be4.el9.no 547 kB/s | 263 kB 00:00 2024-02-05T07:57:31.976 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:31.977 INFO:teuthology.orchestra.run.smithi196.stdout:Total 686 kB/s | 332 kB 00:00 2024-02-05T07:57:31.977 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:57:31.986 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:57:31.986 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:57:32.029 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:57:32.030 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:57:32.179 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:57:32.348 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-05T07:57:32.387 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2/2 2024-02-05T07:57:32.402 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2/2 2024-02-05T07:57:32.402 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:57:32.402 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-05T07:57:32.403 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:33.154 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 1/2 2024-02-05T07:57:33.518 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-05T07:57:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:33.519 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:57:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T07:57:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-05T07:57:33.519 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:33.519 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:33.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados-devel 2024-02-05T07:57:34.444 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:43 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:34.600 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:34.601 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:34.601 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-02-05T07:57:34.601 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:34.602 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:57:34.602 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 127 k 2024-02-05T07:57:34.602 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:34.602 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:57:34.602 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:34.602 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T07:57:34.602 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:34.602 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 127 k 2024-02-05T07:57:34.602 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 456 k 2024-02-05T07:57:34.602 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:57:34.915 INFO:teuthology.orchestra.run.smithi196.stdout:librados-devel-19.0.0-1241.g164d3be4.el9.x86_64 405 kB/s | 127 kB 00:00 2024-02-05T07:57:34.919 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:34.919 INFO:teuthology.orchestra.run.smithi196.stdout:Total 403 kB/s | 127 kB 00:00 2024-02-05T07:57:34.919 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:57:34.922 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:57:34.922 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:57:34.976 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:57:34.977 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:57:35.122 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:57:35.223 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:35.707 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:35.998 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:35.998 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:35.998 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:57:35.998 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:57:35.998 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:35.998 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:36.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs2 2024-02-05T07:57:36.822 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:45 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:36.928 INFO:teuthology.orchestra.run.smithi196.stdout:Package libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:36.980 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:36.981 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T07:57:36.981 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:37.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs-devel 2024-02-05T07:57:37.620 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:46 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:37.780 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:37.780 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:37.780 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-02-05T07:57:37.780 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:37.781 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:57:37.781 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 31 k 2024-02-05T07:57:37.781 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:37.781 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:57:37.781 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:37.781 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T07:57:37.781 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:37.781 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 31 k 2024-02-05T07:57:37.781 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 140 k 2024-02-05T07:57:37.781 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:57:37.961 INFO:teuthology.orchestra.run.smithi196.stdout:libcephfs-devel-19.0.0-1241.g164d3be4.el9.x86_6 172 kB/s | 31 kB 00:00 2024-02-05T07:57:37.962 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:37.962 INFO:teuthology.orchestra.run.smithi196.stdout:Total 170 kB/s | 31 kB 00:00 2024-02-05T07:57:37.962 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:57:37.965 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:57:37.965 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:57:38.006 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:57:38.006 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:57:38.095 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:57:38.200 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:38.626 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:38.905 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:38.906 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:57:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:57:38.906 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:38.906 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:39.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados2 2024-02-05T07:57:39.669 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:48 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:39.773 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:39.826 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:39.827 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T07:57:39.827 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:39.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librbd1 2024-02-05T07:57:40.461 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:49 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:40.564 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:40.617 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:40.618 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T07:57:40.618 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:40.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rados 2024-02-05T07:57:41.247 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:50 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:41.350 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:41.402 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:41.403 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T07:57:41.404 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:41.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rgw 2024-02-05T07:57:42.037 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:51 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:42.141 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:42.193 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:42.194 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T07:57:42.194 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:42.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-cephfs 2024-02-05T07:57:42.816 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:51 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:42.920 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:42.972 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:42.973 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T07:57:42.973 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:43.039 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rbd 2024-02-05T07:57:43.611 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:52 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:43.715 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T07:57:43.767 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:43.768 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-02-05T07:57:43.768 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:43.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-fuse 2024-02-05T07:57:44.401 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:53 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:44.558 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:44.559 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:44.559 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-02-05T07:57:44.559 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:44.559 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:57:44.559 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 85 k 2024-02-05T07:57:44.559 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:44.559 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:57:44.559 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:44.559 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T07:57:44.560 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:44.560 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 85 k 2024-02-05T07:57:44.560 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 231 k 2024-02-05T07:57:44.560 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:57:44.880 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-fuse-19.0.0-1241.g164d3be4.el9.x86_64.rpm 266 kB/s | 85 kB 00:00 2024-02-05T07:57:44.881 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:44.881 INFO:teuthology.orchestra.run.smithi196.stdout:Total 265 kB/s | 85 kB 00:00 2024-02-05T07:57:44.881 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:57:44.888 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:57:44.888 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:57:44.943 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:57:44.943 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:57:45.125 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:57:45.228 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:45.625 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:45.941 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:57:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:57:45.942 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:45.942 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:46.172 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror 2024-02-05T07:57:46.776 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:55 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:46.933 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:46.933 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:46.933 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-02-05T07:57:46.933 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:46.933 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:57:46.934 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.1 M 2024-02-05T07:57:46.934 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:46.934 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:57:46.934 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:46.934 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T07:57:46.934 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:46.934 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 3.1 M 2024-02-05T07:57:46.935 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 13 M 2024-02-05T07:57:46.935 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:57:47.526 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-mirror-19.0.0-1241.g164d3be4.el9.x86_64.rpm 5.2 MB/s | 3.1 MB 00:00 2024-02-05T07:57:47.527 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:47.527 INFO:teuthology.orchestra.run.smithi196.stdout:Total 5.2 MB/s | 3.1 MB 00:00 2024-02-05T07:57:47.527 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:57:47.536 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:57:47.536 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:57:47.610 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:57:47.610 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:57:48.044 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:57:48.118 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:48.143 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:48.144 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T07:57:48.144 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-05T07:57:48.144 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T07:57:48.144 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T07:57:48.144 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:49.289 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:49.289 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:49.289 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:57:49.290 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:57:49.290 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:49.290 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:49.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-nbd 2024-02-05T07:57:50.123 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:59 ago on Mon 05 Feb 2024 07:55:51 AM UTC. 2024-02-05T07:57:50.278 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-02-05T07:57:50.279 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:50.279 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-02-05T07:57:50.279 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:50.279 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-02-05T07:57:50.279 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 171 k 2024-02-05T07:57:50.279 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:50.279 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-02-05T07:57:50.279 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-02-05T07:57:50.279 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-02-05T07:57:50.279 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:50.280 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 171 k 2024-02-05T07:57:50.280 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 494 k 2024-02-05T07:57:50.280 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-02-05T07:57:50.846 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-nbd-19.0.0-1241.g164d3be4.el9.x86_64.rpm 302 kB/s | 171 kB 00:00 2024-02-05T07:57:50.847 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:57:50.847 INFO:teuthology.orchestra.run.smithi196.stdout:Total 301 kB/s | 171 kB 00:00 2024-02-05T07:57:50.847 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-02-05T07:57:50.855 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-02-05T07:57:50.855 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-02-05T07:57:50.911 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-02-05T07:57:50.912 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-02-05T07:57:51.058 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-02-05T07:57:51.160 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:51.598 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:51.922 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T07:57:51.922 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:51.922 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-02-05T07:57:51.922 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T07:57:51.922 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T07:57:51.922 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-02-05T07:57:52.218 DEBUG:teuthology.parallel:result is None 2024-02-05T07:57:52.219 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:57:52.381 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-05T07:57:52.406 INFO:teuthology.orchestra.run.smithi134.stdout:19.0.0-1241.g164d3be4.el9 2024-02-05T07:57:52.406 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1241.g164d3be4.el9 2024-02-05T07:57:52.406 INFO:teuthology.task.install:The correct ceph version 19.0.0-1241.g164d3be4 is installed. 2024-02-05T07:57:52.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:57:52.547 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-05T07:57:52.568 INFO:teuthology.orchestra.run.smithi196.stdout:19.0.0-1241.g164d3be4.el9 2024-02-05T07:57:52.568 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1241.g164d3be4.el9 2024-02-05T07:57:52.568 INFO:teuthology.task.install:The correct ceph version 19.0.0-1241.g164d3be4 is installed. 2024-02-05T07:57:52.570 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-05T07:57:52.570 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T07:57:52.570 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-05T07:57:52.603 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T07:57:52.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-05T07:57:52.636 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-05T07:57:52.637 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T07:57:52.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-02-05T07:57:52.672 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-05T07:57:52.741 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T07:57:52.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-02-05T07:57:52.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-05T07:57:52.835 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-05T07:57:52.835 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T07:57:52.835 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-05T07:57:52.864 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-05T07:57:52.932 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T07:57:52.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-05T07:57:52.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-05T07:57:53.025 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-05T07:57:53.026 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T07:57:53.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-02-05T07:57:53.053 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-05T07:57:53.121 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T07:57:53.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-02-05T07:57:53.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-05T07:57:53.214 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-05T07:57:53.306 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921', 'cephadm_mode': 'root'} 2024-02-05T07:57:53.306 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:57:53.307 INFO:tasks.cephadm:Cluster fsid is 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T07:57:53.307 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-05T07:57:53.307 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.134', 'mon.c': '[v2:172.21.15.134:3301,v1:172.21.15.134:6790]', 'mon.b': '172.21.15.196'} 2024-02-05T07:57:53.307 INFO:tasks.cephadm:First mon is mon.a on smithi134 2024-02-05T07:57:53.307 INFO:tasks.cephadm:First mgr is y 2024-02-05T07:57:53.307 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-05T07:57:53.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo hostname $(hostname -s) 2024-02-05T07:57:53.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo hostname $(hostname -s) 2024-02-05T07:57:53.363 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-02-05T07:57:53.363 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:57:53.508 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76505/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1241-g164d3be4', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1241.g164d3be4'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-05 00:32:51.050045', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-testing-3-2024-02-04-2254', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-02-05T07:57:53.726 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-guits-testing-3-2024-02-04-2254, sha1 164d3be4a527be775f4d3c343ea591122f1b8921 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:57:53.727 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/x86_64/flavors/default/cephadm 2024-02-05T07:57:53.727 DEBUG:teuthology.orchestra.run.smithi134:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-05T07:57:54.476 INFO:teuthology.orchestra.run.smithi134.stdout:-rw-r--r--. 1 ubuntu ubuntu 766928 Feb 5 07:57 /home/ubuntu/cephtest/cephadm 2024-02-05T07:57:54.476 DEBUG:teuthology.orchestra.run.smithi196:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-05T07:57:54.927 INFO:teuthology.orchestra.run.smithi196.stdout:-rw-r--r--. 1 ubuntu ubuntu 766928 Feb 5 07:57 /home/ubuntu/cephtest/cephadm 2024-02-05T07:57:54.928 DEBUG:teuthology.orchestra.run.smithi134:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-05T07:57:54.946 DEBUG:teuthology.orchestra.run.smithi196:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-05T07:57:54.977 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 on all hosts... 2024-02-05T07:57:54.977 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 pull 2024-02-05T07:57:54.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 pull 2024-02-05T07:57:55.957 INFO:teuthology.orchestra.run.smithi196.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921... 2024-02-05T07:57:55.959 INFO:teuthology.orchestra.run.smithi134.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921... 2024-02-05T07:58:39.293 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T07:58:39.293 INFO:teuthology.orchestra.run.smithi134.stdout: "ceph_version": "ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev)", 2024-02-05T07:58:39.294 INFO:teuthology.orchestra.run.smithi134.stdout: "image_id": "8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1", 2024-02-05T07:58:39.294 INFO:teuthology.orchestra.run.smithi134.stdout: "repo_digests": [ 2024-02-05T07:58:39.294 INFO:teuthology.orchestra.run.smithi134.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0" 2024-02-05T07:58:39.294 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T07:58:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T07:58:53.234 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T07:58:53.234 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_version": "ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev)", 2024-02-05T07:58:53.234 INFO:teuthology.orchestra.run.smithi196.stdout: "image_id": "8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1", 2024-02-05T07:58:53.234 INFO:teuthology.orchestra.run.smithi196.stdout: "repo_digests": [ 2024-02-05T07:58:53.234 INFO:teuthology.orchestra.run.smithi196.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0" 2024-02-05T07:58:53.234 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T07:58:53.234 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T07:58:53.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph 2024-02-05T07:58:53.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph 2024-02-05T07:58:53.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /etc/ceph 2024-02-05T07:58:53.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /etc/ceph 2024-02-05T07:58:53.392 INFO:tasks.cephadm:Writing seed config... 2024-02-05T07:58:53.393 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [global] ms type = async 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-05T07:58:53.394 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-05T07:58:53.395 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-02-05T07:58:53.395 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-02-05T07:58:53.395 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-02-05T07:58:53.395 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-02-05T07:58:53.395 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-02-05T07:58:53.395 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-05T07:58:53.395 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-02-05T07:58:53.396 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T07:58:53.396 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-05T07:58:53.422 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 43e13aa4-c3fc-11ee-95b6-87774f69a715 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-02-05T07:58:53.423 DEBUG:teuthology.orchestra.run.smithi134:mon.a> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.a.service 2024-02-05T07:58:53.465 DEBUG:teuthology.orchestra.run.smithi134:mgr.y> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mgr.y.service 2024-02-05T07:58:53.507 INFO:tasks.cephadm:Bootstrapping... 2024-02-05T07:58:53.507 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 -v bootstrap --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.134 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-05T07:58:53.702 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-02-05T07:58:53.702 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921', '-v', 'bootstrap', '--fsid', '43e13aa4-c3fc-11ee-95b6-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.134', '--skip-admin-label'] 2024-02-05T07:58:53.727 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stdout 4.9.0 2024-02-05T07:58:53.728 INFO:teuthology.orchestra.run.smithi134.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-05T07:58:53.728 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying podman|docker is present... 2024-02-05T07:58:53.751 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stdout 4.9.0 2024-02-05T07:58:53.751 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying lvm2 is present... 2024-02-05T07:58:53.751 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying time synchronization is in place... 2024-02-05T07:58:53.758 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-05T07:58:53.759 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-05T07:58:53.764 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-05T07:58:53.764 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-02-05T07:58:53.770 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout enabled 2024-02-05T07:58:53.777 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout active 2024-02-05T07:58:53.777 INFO:teuthology.orchestra.run.smithi134.stdout:Unit chronyd.service is enabled and running 2024-02-05T07:58:53.777 INFO:teuthology.orchestra.run.smithi134.stdout:Repeating the final host check... 2024-02-05T07:58:53.799 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stdout 4.9.0 2024-02-05T07:58:53.799 INFO:teuthology.orchestra.run.smithi134.stdout:podman (/bin/podman) version 4.9.0 is present 2024-02-05T07:58:53.799 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl is present 2024-02-05T07:58:53.799 INFO:teuthology.orchestra.run.smithi134.stdout:lvcreate is present 2024-02-05T07:58:53.805 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-05T07:58:53.805 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-05T07:58:53.811 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-05T07:58:53.811 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-02-05T07:58:53.817 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout enabled 2024-02-05T07:58:53.824 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout active 2024-02-05T07:58:53.824 INFO:teuthology.orchestra.run.smithi134.stdout:Unit chronyd.service is enabled and running 2024-02-05T07:58:53.824 INFO:teuthology.orchestra.run.smithi134.stdout:Host looks OK 2024-02-05T07:58:53.824 INFO:teuthology.orchestra.run.smithi134.stdout:Cluster fsid: 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T07:58:53.824 INFO:teuthology.orchestra.run.smithi134.stdout:Acquiring lock 139793878860368 on /run/cephadm/43e13aa4-c3fc-11ee-95b6-87774f69a715.lock 2024-02-05T07:58:53.824 INFO:teuthology.orchestra.run.smithi134.stdout:Lock 139793878860368 acquired on /run/cephadm/43e13aa4-c3fc-11ee-95b6-87774f69a715.lock 2024-02-05T07:58:53.825 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying IP 172.21.15.134 port 3300 ... 2024-02-05T07:58:53.825 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying IP 172.21.15.134 port 6789 ... 2024-02-05T07:58:53.825 INFO:teuthology.orchestra.run.smithi134.stdout:Base mon IP(s) is [172.21.15.134:3300, 172.21.15.134:6789], mon addrv is [v2:172.21.15.134:3300,v1:172.21.15.134:6789] 2024-02-05T07:58:53.828 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.134 metric 100 2024-02-05T07:58:53.828 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.134 metric 100 2024-02-05T07:58:53.830 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-05T07:58:53.830 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-05T07:58:53.831 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1783sec hoplimit 64 pref medium 2024-02-05T07:58:53.833 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-05T07:58:53.833 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-05T07:58:53.833 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-05T07:58:53.833 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-05T07:58:53.833 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6de9/64 scope link 2024-02-05T07:58:53.833 INFO:teuthology.orchestra.run.smithi134.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-05T07:58:53.834 INFO:teuthology.orchestra.run.smithi134.stdout:Mon IP `172.21.15.134` is in CIDR network `172.21.0.0/20` 2024-02-05T07:58:53.834 INFO:teuthology.orchestra.run.smithi134.stdout:Mon IP `172.21.15.134` is in CIDR network `172.21.0.0/20` 2024-02-05T07:58:53.834 INFO:teuthology.orchestra.run.smithi134.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-05T07:58:53.834 INFO:teuthology.orchestra.run.smithi134.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-05T07:58:53.835 INFO:teuthology.orchestra.run.smithi134.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921... 2024-02-05T07:58:57.283 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stdout 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 2024-02-05T07:58:57.283 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921... 2024-02-05T07:58:57.283 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Getting image source signatures 2024-02-05T07:58:57.283 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-05T07:58:57.283 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Copying blob sha256:48ac04de0ee67de328b3ea21477b81735e6e7c0783ebcda067ca4695a8266750 2024-02-05T07:58:57.283 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Copying config sha256:8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 2024-02-05T07:58:57.283 INFO:teuthology.orchestra.run.smithi134.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-05T07:59:00.156 INFO:teuthology.orchestra.run.smithi134.stdout:ceph: stdout ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev) 2024-02-05T07:59:00.156 INFO:teuthology.orchestra.run.smithi134.stdout:Ceph version: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev) 2024-02-05T07:59:00.156 INFO:teuthology.orchestra.run.smithi134.stdout:Extracting ceph user uid/gid from container image... 2024-02-05T07:59:01.046 INFO:teuthology.orchestra.run.smithi134.stdout:stat: stdout 167 167 2024-02-05T07:59:01.046 INFO:teuthology.orchestra.run.smithi134.stdout:Creating initial keys... 2024-02-05T07:59:01.968 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph-authtool: stdout AQBFlcBl1Um8GRAAO1Qc+rQ4ERx2nkixKSU8hA== 2024-02-05T07:59:02.941 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph-authtool: stdout AQBGlcBlblsKFhAAoLYBWjwbTRMuLuph7qkR6w== 2024-02-05T07:59:03.896 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph-authtool: stdout AQBHlcBlrBDxERAAS4TYDIHTNxC4AnXt95E94g== 2024-02-05T07:59:03.897 INFO:teuthology.orchestra.run.smithi134.stdout:Creating initial monmap... 2024-02-05T07:59:07.031 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-05T07:59:07.031 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-02-05T07:59:07.031 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T07:59:07.031 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-05T07:59:07.031 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool for a [v2:172.21.15.134:3300,v1:172.21.15.134:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-05T07:59:07.031 INFO:teuthology.orchestra.run.smithi134.stdout:setting min_mon_release = quincy 2024-02-05T07:59:07.031 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: set fsid to 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T07:59:07.031 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-05T07:59:07.031 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T07:59:07.031 INFO:teuthology.orchestra.run.smithi134.stdout:Creating mon... 2024-02-05T07:59:07.920 INFO:teuthology.orchestra.run.smithi134.stdout:create mon.a on 2024-02-05T07:59:08.128 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-05T07:59:08.271 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-05T07:59:08.447 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715.target → /etc/systemd/system/ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715.target. 2024-02-05T07:59:08.447 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715.target → /etc/systemd/system/ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715.target. 2024-02-05T07:59:08.648 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.a 2024-02-05T07:59:08.648 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Failed to reset failed state of unit ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.a.service: Unit ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.a.service not loaded. 2024-02-05T07:59:08.810 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715.target.wants/ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.a.service → /etc/systemd/system/ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@.service. 2024-02-05T07:59:09.368 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-05T07:59:09.368 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout disabled 2024-02-05T07:59:09.375 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-05T07:59:09.376 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-02-05T07:59:09.376 INFO:teuthology.orchestra.run.smithi134.stdout:firewalld.service is not enabled 2024-02-05T07:59:09.376 INFO:teuthology.orchestra.run.smithi134.stdout:Not possible to enable service . firewalld.service is not available 2024-02-05T07:59:09.376 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mon to start... 2024-02-05T07:59:09.376 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mon... 2024-02-05T07:59:10.524 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout cluster: 2024-02-05T07:59:10.524 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout id: 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T07:59:10.524 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-05T07:59:10.524 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-02-05T07:59:10.524 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout services: 2024-02-05T07:59:10.524 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.629166s) 2024-02-05T07:59:10.524 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-05T07:59:10.525 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-05T07:59:10.525 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-02-05T07:59:10.525 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout data: 2024-02-05T07:59:10.525 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-05T07:59:10.525 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-05T07:59:10.525 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-05T07:59:10.525 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout pgs: 2024-02-05T07:59:10.525 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-02-05T07:59:10.525 INFO:teuthology.orchestra.run.smithi134.stdout:mon is available 2024-02-05T07:59:10.525 INFO:teuthology.orchestra.run.smithi134.stdout:Assimilating anything we can from ceph.conf... 2024-02-05T07:59:11.580 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-02-05T07:59:11.580 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [global] 2024-02-05T07:59:11.580 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout fsid = 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T07:59:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.134:3300,v1:172.21.15.134:6789] 2024-02-05T07:59:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-05T07:59:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-05T07:59:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-05T07:59:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-05T07:59:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-02-05T07:59:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-05T07:59:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-05T07:59:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-02-05T07:59:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [osd] 2024-02-05T07:59:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-05T07:59:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-05T07:59:11.581 INFO:teuthology.orchestra.run.smithi134.stdout:Generating new minimal ceph.conf... 2024-02-05T07:59:12.998 INFO:teuthology.orchestra.run.smithi134.stdout:Restarting the monitor... 2024-02-05T07:59:13.013 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:13 smithi134 systemd[1]: Stopping Ceph mon.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T07:59:13.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:13 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a[29574]: 2024-02-05T07:59:13.262+0000 7f9f922d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T07:59:13.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:13 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a[29574]: 2024-02-05T07:59:13.262+0000 7f9f922d8700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-05T07:59:13.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:13 smithi134 podman[29931]: 2024-02-05 07:59:13.325547903 +0000 UTC m=+0.167485299 container died bcae33840e74f42cbabaf91db3b2a1cb4747829a7845f6ecdb585996fabbb580 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-05T07:59:13.900 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:13 smithi134 podman[29931]: 2024-02-05 07:59:13.651890191 +0000 UTC m=+0.493827585 container cleanup bcae33840e74f42cbabaf91db3b2a1cb4747829a7845f6ecdb585996fabbb580 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-05T07:59:13.900 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:13 smithi134 bash[29931]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a 2024-02-05T07:59:13.900 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:13 smithi134 podman[29949]: 2024-02-05 07:59:13.802925185 +0000 UTC m=+0.475475232 container remove bcae33840e74f42cbabaf91db3b2a1cb4747829a7845f6ecdb585996fabbb580 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T07:59:14.212 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-05T07:59:14.212 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 systemd[1]: Stopped Ceph mon.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T07:59:14.212 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 systemd[1]: Starting Ceph mon.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T07:59:14.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 podman[30048]: 2024-02-05 07:59:14.212031344 +0000 UTC m=+0.099287090 container create 6397166ec857eebac547fa2f9d34f0ba0b61385d3127f224987e4a9fefa7ba1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131) 2024-02-05T07:59:14.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 podman[30048]: 2024-02-05 07:59:14.130800842 +0000 UTC m=+0.018056588 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:59:14.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 podman[30048]: 2024-02-05 07:59:14.397724975 +0000 UTC m=+0.284980721 container init 6397166ec857eebac547fa2f9d34f0ba0b61385d3127f224987e4a9fefa7ba1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T07:59:14.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 podman[30048]: 2024-02-05 07:59:14.403268192 +0000 UTC m=+0.290523939 container start 6397166ec857eebac547fa2f9d34f0ba0b61385d3127f224987e4a9fefa7ba1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-05T07:59:14.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: set uid:gid to 167:167 (ceph:ceph) 2024-02-05T07:59:14.491 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev), process ceph-mon, pid 7 2024-02-05T07:59:14.491 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: pidfile_write: ignore empty --pid-file 2024-02-05T07:59:14.491 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: load: jerasure load: lrc 2024-02-05T07:59:14.491 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: RocksDB version: 7.9.2 2024-02-05T07:59:14.491 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Git sha 0 2024-02-05T07:59:14.491 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Compile date 2024-02-04 23:11:06 2024-02-05T07:59:14.491 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: DB SUMMARY 2024-02-05T07:59:14.491 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: DB Session ID: 2T6WQ934JGQHR6NPZW2S 2024-02-05T07:59:14.491 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: CURRENT file: CURRENT 2024-02-05T07:59:14.491 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: IDENTITY file: IDENTITY 2024-02-05T07:59:14.491 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-05T07:59:14.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-05T07:59:14.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 90847 ; 2024-02-05T07:59:14.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.error_if_exists: 0 2024-02-05T07:59:14.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.create_if_missing: 0 2024-02-05T07:59:14.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.paranoid_checks: 1 2024-02-05T07:59:14.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-05T07:59:14.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-05T07:59:14.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-05T07:59:14.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.env: 0x5576c8fa9ae0 2024-02-05T07:59:14.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.fs: PosixFileSystem 2024-02-05T07:59:14.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.info_log: 0x5576c9b338c0 2024-02-05T07:59:14.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_file_opening_threads: 16 2024-02-05T07:59:14.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.statistics: (nil) 2024-02-05T07:59:14.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.use_fsync: 0 2024-02-05T07:59:14.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_log_file_size: 0 2024-02-05T07:59:14.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-05T07:59:14.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-05T07:59:14.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.keep_log_file_num: 1000 2024-02-05T07:59:14.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.recycle_log_file_num: 0 2024-02-05T07:59:14.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.allow_fallocate: 1 2024-02-05T07:59:14.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.allow_mmap_reads: 0 2024-02-05T07:59:14.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.allow_mmap_writes: 0 2024-02-05T07:59:14.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.use_direct_reads: 0 2024-02-05T07:59:14.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-05T07:59:14.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.create_missing_column_families: 0 2024-02-05T07:59:14.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.db_log_dir: 2024-02-05T07:59:14.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.wal_dir: 2024-02-05T07:59:14.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-05T07:59:14.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-05T07:59:14.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-05T07:59:14.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-05T07:59:14.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-05T07:59:14.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-05T07:59:14.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.advise_random_on_open: 1 2024-02-05T07:59:14.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.db_write_buffer_size: 0 2024-02-05T07:59:14.495 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.write_buffer_manager: 0x5576c9b07400 2024-02-05T07:59:14.495 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-05T07:59:14.495 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-05T07:59:14.495 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-05T07:59:14.495 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.rate_limiter: (nil) 2024-02-05T07:59:14.495 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-05T07:59:14.497 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.wal_recovery_mode: 2 2024-02-05T07:59:14.497 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.enable_thread_tracking: 0 2024-02-05T07:59:14.497 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.enable_pipelined_write: 0 2024-02-05T07:59:14.497 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.unordered_write: 0 2024-02-05T07:59:14.497 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-05T07:59:14.497 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-05T07:59:14.497 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-05T07:59:14.497 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-05T07:59:14.498 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.row_cache: None 2024-02-05T07:59:14.498 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.wal_filter: None 2024-02-05T07:59:14.498 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-05T07:59:14.498 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.allow_ingest_behind: 0 2024-02-05T07:59:14.498 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.two_write_queues: 0 2024-02-05T07:59:14.498 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.manual_wal_flush: 0 2024-02-05T07:59:14.498 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.wal_compression: 0 2024-02-05T07:59:14.498 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.atomic_flush: 0 2024-02-05T07:59:14.498 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-05T07:59:14.498 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-05T07:59:14.498 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-05T07:59:14.499 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.log_readahead_size: 0 2024-02-05T07:59:14.499 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-05T07:59:14.499 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.best_efforts_recovery: 0 2024-02-05T07:59:14.499 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-05T07:59:14.499 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-05T07:59:14.499 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.allow_data_in_errors: 0 2024-02-05T07:59:14.499 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.db_host_id: __hostname__ 2024-02-05T07:59:14.499 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-05T07:59:14.499 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_background_jobs: 2 2024-02-05T07:59:14.499 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_background_compactions: -1 2024-02-05T07:59:14.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_subcompactions: 1 2024-02-05T07:59:14.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-05T07:59:14.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-05T07:59:14.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-05T07:59:14.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_total_wal_size: 0 2024-02-05T07:59:14.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-05T07:59:14.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-05T07:59:14.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-05T07:59:14.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-05T07:59:14.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_open_files: -1 2024-02-05T07:59:14.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.bytes_per_sync: 0 2024-02-05T07:59:14.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-05T07:59:14.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-05T07:59:14.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compaction_readahead_size: 0 2024-02-05T07:59:14.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_background_flushes: -1 2024-02-05T07:59:14.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Compression algorithms supported: 2024-02-05T07:59:14.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-05T07:59:14.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: kZSTD supported: 0 2024-02-05T07:59:14.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: kXpressCompression supported: 0 2024-02-05T07:59:14.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: kLZ4HCCompression supported: 1 2024-02-05T07:59:14.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: kZlibCompression supported: 1 2024-02-05T07:59:14.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: kSnappyCompression supported: 1 2024-02-05T07:59:14.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: kLZ4Compression supported: 1 2024-02-05T07:59:14.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: kBZip2Compression supported: 0 2024-02-05T07:59:14.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-05T07:59:14.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-05T07:59:14.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-02-05T07:59:14.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-05T07:59:14.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-05T07:59:14.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.merge_operator: 2024-02-05T07:59:14.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compaction_filter: None 2024-02-05T07:59:14.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compaction_filter_factory: None 2024-02-05T07:59:14.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.sst_partitioner_factory: None 2024-02-05T07:59:14.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-05T07:59:14.503 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-05T07:59:14.503 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5576c9b339c0) 2024-02-05T07:59:14.503 INFO:journalctl@ceph.mon.a.smithi134.stdout: cache_index_and_filter_blocks: 1 2024-02-05T07:59:14.503 INFO:journalctl@ceph.mon.a.smithi134.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-05T07:59:14.503 INFO:journalctl@ceph.mon.a.smithi134.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-05T07:59:14.503 INFO:journalctl@ceph.mon.a.smithi134.stdout: pin_top_level_index_and_filter: 1 2024-02-05T07:59:14.503 INFO:journalctl@ceph.mon.a.smithi134.stdout: index_type: 0 2024-02-05T07:59:14.503 INFO:journalctl@ceph.mon.a.smithi134.stdout: data_block_index_type: 0 2024-02-05T07:59:14.503 INFO:journalctl@ceph.mon.a.smithi134.stdout: index_shortening: 1 2024-02-05T07:59:14.503 INFO:journalctl@ceph.mon.a.smithi134.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-05T07:59:14.503 INFO:journalctl@ceph.mon.a.smithi134.stdout: checksum: 4 2024-02-05T07:59:14.504 INFO:journalctl@ceph.mon.a.smithi134.stdout: no_block_cache: 0 2024-02-05T07:59:14.504 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_cache: 0x5576c9b37090 2024-02-05T07:59:14.504 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_cache_name: BinnedLRUCache 2024-02-05T07:59:14.504 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_cache_options: 2024-02-05T07:59:14.504 INFO:journalctl@ceph.mon.a.smithi134.stdout: capacity : 536870912 2024-02-05T07:59:14.504 INFO:journalctl@ceph.mon.a.smithi134.stdout: num_shard_bits : 4 2024-02-05T07:59:14.504 INFO:journalctl@ceph.mon.a.smithi134.stdout: strict_capacity_limit : 0 2024-02-05T07:59:14.504 INFO:journalctl@ceph.mon.a.smithi134.stdout: high_pri_pool_ratio: 0.000 2024-02-05T07:59:14.504 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_cache_compressed: (nil) 2024-02-05T07:59:14.504 INFO:journalctl@ceph.mon.a.smithi134.stdout: persistent_cache: (nil) 2024-02-05T07:59:14.504 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_size: 4096 2024-02-05T07:59:14.505 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_size_deviation: 10 2024-02-05T07:59:14.505 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_restart_interval: 16 2024-02-05T07:59:14.505 INFO:journalctl@ceph.mon.a.smithi134.stdout: index_block_restart_interval: 1 2024-02-05T07:59:14.505 INFO:journalctl@ceph.mon.a.smithi134.stdout: metadata_block_size: 4096 2024-02-05T07:59:14.505 INFO:journalctl@ceph.mon.a.smithi134.stdout: partition_filters: 0 2024-02-05T07:59:14.505 INFO:journalctl@ceph.mon.a.smithi134.stdout: use_delta_encoding: 1 2024-02-05T07:59:14.505 INFO:journalctl@ceph.mon.a.smithi134.stdout: filter_policy: bloomfilter 2024-02-05T07:59:14.505 INFO:journalctl@ceph.mon.a.smithi134.stdout: whole_key_filtering: 1 2024-02-05T07:59:14.505 INFO:journalctl@ceph.mon.a.smithi134.stdout: verify_compression: 0 2024-02-05T07:59:14.505 INFO:journalctl@ceph.mon.a.smithi134.stdout: read_amp_bytes_per_bit: 0 2024-02-05T07:59:14.505 INFO:journalctl@ceph.mon.a.smithi134.stdout: format_version: 5 2024-02-05T07:59:14.505 INFO:journalctl@ceph.mon.a.smithi134.stdout: enable_index_compression: 1 2024-02-05T07:59:14.506 INFO:journalctl@ceph.mon.a.smithi134.stdout: block_align: 0 2024-02-05T07:59:14.506 INFO:journalctl@ceph.mon.a.smithi134.stdout: max_auto_readahead_size: 262144 2024-02-05T07:59:14.506 INFO:journalctl@ceph.mon.a.smithi134.stdout: prepopulate_block_cache: 0 2024-02-05T07:59:14.506 INFO:journalctl@ceph.mon.a.smithi134.stdout: initial_auto_readahead_size: 8192 2024-02-05T07:59:14.506 INFO:journalctl@ceph.mon.a.smithi134.stdout: num_file_reads_for_auto_readahead: 2 2024-02-05T07:59:14.506 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.write_buffer_size: 33554432 2024-02-05T07:59:14.506 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_write_buffer_number: 2 2024-02-05T07:59:14.506 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compression: NoCompression 2024-02-05T07:59:14.506 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.bottommost_compression: Disabled 2024-02-05T07:59:14.506 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.prefix_extractor: nullptr 2024-02-05T07:59:14.506 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-05T07:59:14.506 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.num_levels: 7 2024-02-05T07:59:14.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-05T07:59:14.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-05T07:59:14.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-05T07:59:14.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-05T07:59:14.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-05T07:59:14.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-05T07:59:14.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-05T07:59:14.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-05T07:59:14.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-05T07:59:14.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-05T07:59:14.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-05T07:59:14.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-05T07:59:14.508 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-05T07:59:14.508 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compression_opts.level: 32767 2024-02-05T07:59:14.508 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compression_opts.strategy: 0 2024-02-05T07:59:14.508 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-05T07:59:14.508 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-05T07:59:14.508 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-05T07:59:14.508 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-05T07:59:14.508 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compression_opts.enabled: false 2024-02-05T07:59:14.508 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-05T07:59:14.508 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-05T07:59:14.508 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-05T07:59:14.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-05T07:59:14.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.target_file_size_base: 67108864 2024-02-05T07:59:14.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-05T07:59:14.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-05T07:59:14.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-05T07:59:14.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-05T07:59:14.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-05T07:59:14.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-05T07:59:14.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-05T07:59:14.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-05T07:59:14.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-05T07:59:14.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-05T07:59:14.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-05T07:59:14.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-05T07:59:14.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-05T07:59:14.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-05T07:59:14.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.arena_block_size: 1048576 2024-02-05T07:59:14.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-05T07:59:14.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-05T07:59:14.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.disable_auto_compactions: 0 2024-02-05T07:59:14.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-05T07:59:14.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-05T07:59:14.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-05T07:59:14.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-05T07:59:14.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-05T07:59:14.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-05T07:59:14.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-05T07:59:14.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-05T07:59:14.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-05T07:59:14.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-05T07:59:14.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.table_properties_collectors: 2024-02-05T07:59:14.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.inplace_update_support: 0 2024-02-05T07:59:14.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-05T07:59:14.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-05T07:59:14.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-05T07:59:14.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-05T07:59:14.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.bloom_locality: 0 2024-02-05T07:59:14.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.max_successive_merges: 0 2024-02-05T07:59:14.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-05T07:59:14.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.paranoid_file_checks: 0 2024-02-05T07:59:14.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.force_consistency_checks: 1 2024-02-05T07:59:14.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.report_bg_io_stats: 0 2024-02-05T07:59:14.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.ttl: 2592000 2024-02-05T07:59:14.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-05T07:59:14.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-05T07:59:14.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-05T07:59:14.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.enable_blob_files: false 2024-02-05T07:59:14.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.min_blob_size: 0 2024-02-05T07:59:14.513 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.blob_file_size: 268435456 2024-02-05T07:59:14.513 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-05T07:59:14.513 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-05T07:59:14.513 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-05T07:59:14.513 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-05T07:59:14.513 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-05T07:59:14.513 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.blob_file_starting_level: 0 2024-02-05T07:59:14.513 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-05T07:59:14.513 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-05T07:59:14.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-05T07:59:14.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 06249f81-884c-41f3-8ba4-d64f00109a59 2024-02-05T07:59:14.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707119954432043, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-05T07:59:14.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-05T07:59:14.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707119954432924, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 86842, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 278, "table_properties": {"data_size": 84924, "index_size": 264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11580, "raw_average_key_size": 48, "raw_value_size": 78545, "raw_average_value_size": 331, "num_data_blocks": 11, "num_entries": 237, "num_filter_entries": 237, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707119954, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "06249f81-884c-41f3-8ba4-d64f00109a59", "db_session_id": "2T6WQ934JGQHR6NPZW2S", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-05T07:59:14.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707119954433004, "job": 1, "event": "recovery_finished"} 2024-02-05T07:59:14.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-05T07:59:14.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-05T07:59:14.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5576c9c28000 2024-02-05T07:59:14.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: DB pointer 0x5576c9c12000 2024-02-05T07:59:14.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-05T07:59:14.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-05T07:59:14.515 INFO:journalctl@ceph.mon.a.smithi134.stdout: ** DB Stats ** 2024-02-05T07:59:14.515 INFO:journalctl@ceph.mon.a.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T07:59:14.515 INFO:journalctl@ceph.mon.a.smithi134.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-05T07:59:14.515 INFO:journalctl@ceph.mon.a.smithi134.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T07:59:14.515 INFO:journalctl@ceph.mon.a.smithi134.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T07:59:14.515 INFO:journalctl@ceph.mon.a.smithi134.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-05T07:59:14.515 INFO:journalctl@ceph.mon.a.smithi134.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T07:59:14.515 INFO:journalctl@ceph.mon.a.smithi134.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T07:59:14.515 INFO:journalctl@ceph.mon.a.smithi134.stdout: 2024-02-05T07:59:14.515 INFO:journalctl@ceph.mon.a.smithi134.stdout: ** Compaction Stats [default] ** 2024-02-05T07:59:14.516 INFO:journalctl@ceph.mon.a.smithi134.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T07:59:14.516 INFO:journalctl@ceph.mon.a.smithi134.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-05T07:59:14.516 INFO:journalctl@ceph.mon.a.smithi134.stdout: L0 2/0 86.64 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T07:59:14.516 INFO:journalctl@ceph.mon.a.smithi134.stdout: Sum 2/0 86.64 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T07:59:14.516 INFO:journalctl@ceph.mon.a.smithi134.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T07:59:14.516 INFO:journalctl@ceph.mon.a.smithi134.stdout: 2024-02-05T07:59:14.516 INFO:journalctl@ceph.mon.a.smithi134.stdout: ** Compaction Stats [default] ** 2024-02-05T07:59:14.516 INFO:journalctl@ceph.mon.a.smithi134.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T07:59:14.516 INFO:journalctl@ceph.mon.a.smithi134.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-05T07:59:14.516 INFO:journalctl@ceph.mon.a.smithi134.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T07:59:14.517 INFO:journalctl@ceph.mon.a.smithi134.stdout: 2024-02-05T07:59:14.517 INFO:journalctl@ceph.mon.a.smithi134.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-05T07:59:14.517 INFO:journalctl@ceph.mon.a.smithi134.stdout: 2024-02-05T07:59:14.517 INFO:journalctl@ceph.mon.a.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T07:59:14.517 INFO:journalctl@ceph.mon.a.smithi134.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-05T07:59:14.517 INFO:journalctl@ceph.mon.a.smithi134.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-05T07:59:14.517 INFO:journalctl@ceph.mon.a.smithi134.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-05T07:59:14.517 INFO:journalctl@ceph.mon.a.smithi134.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-05T07:59:14.517 INFO:journalctl@ceph.mon.a.smithi134.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-05T07:59:14.517 INFO:journalctl@ceph.mon.a.smithi134.stdout: Cumulative compaction: 0.00 GB write, 29.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T07:59:14.517 INFO:journalctl@ceph.mon.a.smithi134.stdout: Interval compaction: 0.00 GB write, 29.45 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T07:59:14.518 INFO:journalctl@ceph.mon.a.smithi134.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-05T07:59:14.518 INFO:journalctl@ceph.mon.a.smithi134.stdout: Block cache BinnedLRUCache@0x5576c9b37090#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-05T07:59:14.518 INFO:journalctl@ceph.mon.a.smithi134.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-02-05T07:59:14.518 INFO:journalctl@ceph.mon.a.smithi134.stdout: 2024-02-05T07:59:14.518 INFO:journalctl@ceph.mon.a.smithi134.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-05T07:59:14.518 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: starting mon.a rank 0 at public addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] at bind addrs [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T07:59:14.518 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: mon.a@-1(???) e1 preinit fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T07:59:14.518 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: mon.a@-1(???).mds e1 new map 2024-02-05T07:59:14.518 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: mon.a@-1(???).mds e1 print_map 2024-02-05T07:59:14.518 INFO:journalctl@ceph.mon.a.smithi134.stdout: e1 2024-02-05T07:59:14.518 INFO:journalctl@ceph.mon.a.smithi134.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-05T07:59:14.518 INFO:journalctl@ceph.mon.a.smithi134.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-05T07:59:14.519 INFO:journalctl@ceph.mon.a.smithi134.stdout: legacy client fscid: -1 2024-02-05T07:59:14.519 INFO:journalctl@ceph.mon.a.smithi134.stdout: 2024-02-05T07:59:14.519 INFO:journalctl@ceph.mon.a.smithi134.stdout: No filesystems configured 2024-02-05T07:59:14.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-05T07:59:14.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T07:59:14.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T07:59:14.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T07:59:14.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-05T07:59:14.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 bash[30048]: 6397166ec857eebac547fa2f9d34f0ba0b61385d3127f224987e4a9fefa7ba1c 2024-02-05T07:59:14.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 systemd[1]: Started Ceph mon.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T07:59:14.519 INFO:teuthology.orchestra.run.smithi134.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-05T07:59:14.743 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-05T07:59:14.743 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: monmap e1: 1 mons at {a=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T07:59:14.743 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: fsmap 2024-02-05T07:59:14.743 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: osdmap e1: 0 total, 0 up, 0 in 2024-02-05T07:59:14.744 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:14 smithi134 ceph-mon[30086]: mgrmap e1: no daemons active 2024-02-05T07:59:15.607 INFO:teuthology.orchestra.run.smithi134.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-05T07:59:15.629 INFO:teuthology.orchestra.run.smithi134.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-05T07:59:15.629 INFO:teuthology.orchestra.run.smithi134.stdout:Creating mgr... 2024-02-05T07:59:15.629 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-05T07:59:15.630 INFO:teuthology.orchestra.run.smithi134.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-05T07:59:15.917 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mgr.y 2024-02-05T07:59:15.917 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Failed to reset failed state of unit ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mgr.y.service: Unit ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mgr.y.service not loaded. 2024-02-05T07:59:16.061 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715.target.wants/ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@.service. 2024-02-05T07:59:16.273 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:16 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3366549763' entity='client.admin' 2024-02-05T07:59:16.274 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:16 smithi134 systemd[1]: Starting Ceph mgr.y for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T07:59:16.528 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:16 smithi134 podman[30330]: 2024-02-05 07:59:16.27271614 +0000 UTC m=+0.094041524 container create da2fb8fd1ea53c8d166129535579b17f315c1234ee36656af87803ed29134d76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS) 2024-02-05T07:59:16.528 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:16 smithi134 podman[30330]: 2024-02-05 07:59:16.196362226 +0000 UTC m=+0.017687608 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T07:59:16.528 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:16 smithi134 podman[30330]: 2024-02-05 07:59:16.466525243 +0000 UTC m=+0.287850633 container init da2fb8fd1ea53c8d166129535579b17f315c1234ee36656af87803ed29134d76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T07:59:16.528 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:16 smithi134 podman[30330]: 2024-02-05 07:59:16.471833086 +0000 UTC m=+0.293158468 container start da2fb8fd1ea53c8d166129535579b17f315c1234ee36656af87803ed29134d76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131) 2024-02-05T07:59:16.582 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-05T07:59:16.582 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout disabled 2024-02-05T07:59:16.590 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-05T07:59:16.590 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-02-05T07:59:16.590 INFO:teuthology.orchestra.run.smithi134.stdout:firewalld.service is not enabled 2024-02-05T07:59:16.590 INFO:teuthology.orchestra.run.smithi134.stdout:Not possible to enable service . firewalld.service is not available 2024-02-05T07:59:16.599 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-05T07:59:16.599 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout disabled 2024-02-05T07:59:16.606 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-05T07:59:16.606 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-02-05T07:59:16.606 INFO:teuthology.orchestra.run.smithi134.stdout:firewalld.service is not enabled 2024-02-05T07:59:16.606 INFO:teuthology.orchestra.run.smithi134.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-02-05T07:59:16.606 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mgr to start... 2024-02-05T07:59:16.606 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mgr... 2024-02-05T07:59:16.836 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:16 smithi134 bash[30330]: da2fb8fd1ea53c8d166129535579b17f315c1234ee36656af87803ed29134d76 2024-02-05T07:59:16.836 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:16 smithi134 systemd[1]: Started Ceph mgr.y for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T07:59:17.096 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:17.024+0000 7f58943f0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T07:59:17.482 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:17 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1604657952' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-05T07:59:17.482 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:17.121+0000 7f58943f0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T07:59:17.791 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-02-05T07:59:17.791 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:59:17.791 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsid": "43e13aa4-c3fc-11ee-95b6-87774f69a715", 2024-02-05T07:59:17.791 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "health": { 2024-02-05T07:59:17.791 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-05T07:59:17.791 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-05T07:59:17.791 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-05T07:59:17.791 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 0 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "a" 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-05T07:59:17.792 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:17.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-05T07:59:17.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-05T07:59:17.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-05T07:59:17.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-05T07:59:17.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-05T07:59:17.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-05T07:59:17.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:17.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-05T07:59:17.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-05T07:59:17.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-05T07:59:17.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-05T07:59:17.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-05T07:59:17.793 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-05T07:59:17.794 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-05T07:59:17.794 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-05T07:59:17.794 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-05T07:59:17.794 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:17.794 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-05T07:59:17.794 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:17.794 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-05T07:59:17.794 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-05T07:59:17.794 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:17.794 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-05T07:59:17.794 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-05T07:59:17.794 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-05T07:59:17.794 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "restful" 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modified": "2024-02-05T07:59:09.321711+0000", 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:59:17.795 INFO:teuthology.orchestra.run.smithi134.stdout:mgr not available, waiting (1/15)... 2024-02-05T07:59:18.110 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:17.778+0000 7f58943f0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T07:59:18.110 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:17.875+0000 7f58943f0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T07:59:18.110 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:18.015+0000 7f58943f0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T07:59:18.450 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:18.108+0000 7f58943f0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T07:59:18.450 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:18.189+0000 7f58943f0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T07:59:18.450 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:18.448+0000 7f58943f0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T07:59:18.733 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:18.555+0000 7f58943f0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T07:59:18.733 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:18.638+0000 7f58943f0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T07:59:18.733 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:18.731+0000 7f58943f0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T07:59:19.028 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:18.907+0000 7f58943f0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T07:59:20.185 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:19 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:19.885+0000 7f58943f0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T07:59:20.186 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:20 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:20.016+0000 7f58943f0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T07:59:20.440 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:20 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:20.359+0000 7f58943f0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T07:59:20.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:20 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1632255217' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-05T07:59:20.739 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:20 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:20.455+0000 7f58943f0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T07:59:20.739 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:20 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:20.651+0000 7f58943f0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:59:21.028 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:20 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:20.833+0000 7f58943f0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T07:59:21.045 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-02-05T07:59:21.045 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:59:21.045 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsid": "43e13aa4-c3fc-11ee-95b6-87774f69a715", 2024-02-05T07:59:21.045 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "health": { 2024-02-05T07:59:21.045 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-05T07:59:21.045 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-05T07:59:21.045 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-05T07:59:21.046 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:21.046 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-05T07:59:21.046 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-05T07:59:21.046 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 0 2024-02-05T07:59:21.046 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:59:21.046 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-05T07:59:21.046 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "a" 2024-02-05T07:59:21.046 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:59:21.046 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-02-05T07:59:21.046 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-05T07:59:21.046 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:21.046 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-05T07:59:21.046 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-05T07:59:21.047 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:21.047 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-05T07:59:21.047 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:21.047 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-05T07:59:21.047 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-05T07:59:21.047 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-05T07:59:21.047 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-05T07:59:21.047 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-05T07:59:21.047 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-05T07:59:21.052 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:21.052 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-05T07:59:21.052 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-05T07:59:21.052 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-05T07:59:21.052 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-05T07:59:21.052 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-05T07:59:21.052 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-05T07:59:21.052 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-05T07:59:21.052 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-05T07:59:21.052 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-05T07:59:21.053 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:21.053 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-05T07:59:21.053 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:21.053 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-05T07:59:21.053 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-05T07:59:21.053 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:21.053 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-05T07:59:21.053 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-05T07:59:21.053 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-05T07:59:21.053 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-05T07:59:21.053 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-05T07:59:21.053 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-05T07:59:21.053 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "restful" 2024-02-05T07:59:21.054 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:59:21.054 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:59:21.054 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:21.054 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-05T07:59:21.054 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:21.054 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modified": "2024-02-05T07:59:09.321711+0000", 2024-02-05T07:59:21.054 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:59:21.054 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:21.054 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-05T07:59:21.054 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:59:21.054 INFO:teuthology.orchestra.run.smithi134.stdout:mgr not available, waiting (2/15)... 2024-02-05T07:59:21.377 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:21.114+0000 7f58943f0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:59:21.377 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:21.375+0000 7f58943f0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T07:59:21.778 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:21.610+0000 7f58943f0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T07:59:22.778 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:22 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:22.413+0000 7f58943f0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T07:59:23.278 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:22 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:22.984+0000 7f58943f0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T07:59:23.278 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:23.084+0000 7f58943f0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T07:59:23.608 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:23.312+0000 7f58943f0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T07:59:23.609 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-mon[30086]: Activating manager daemon y 2024-02-05T07:59:23.609 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-mon[30086]: mgrmap e2: y(active, starting, since 0.00328439s) 2024-02-05T07:59:23.609 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-mon[30086]: from='mgr.14100 172.21.15.134:0/2297834782' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T07:59:23.609 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-mon[30086]: from='mgr.14100 172.21.15.134:0/2297834782' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T07:59:23.609 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-mon[30086]: from='mgr.14100 172.21.15.134:0/2297834782' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T07:59:23.609 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-mon[30086]: from='mgr.14100 172.21.15.134:0/2297834782' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:59:23.609 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-mon[30086]: from='mgr.14100 172.21.15.134:0/2297834782' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T07:59:23.609 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-mon[30086]: Manager daemon y is now available 2024-02-05T07:59:23.609 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-mon[30086]: from='mgr.14100 172.21.15.134:0/2297834782' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T07:59:23.609 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-mon[30086]: from='mgr.14100 172.21.15.134:0/2297834782' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T07:59:23.609 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-mon[30086]: from='mgr.14100 172.21.15.134:0/2297834782' entity='mgr.y' 2024-02-05T07:59:23.610 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-mon[30086]: from='mgr.14100 172.21.15.134:0/2297834782' entity='mgr.y' 2024-02-05T07:59:23.610 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:23 smithi134 ceph-mon[30086]: from='mgr.14100 172.21.15.134:0/2297834782' entity='mgr.y' 2024-02-05T07:59:24.195 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-02-05T07:59:24.195 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:59:24.195 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsid": "43e13aa4-c3fc-11ee-95b6-87774f69a715", 2024-02-05T07:59:24.196 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "health": { 2024-02-05T07:59:24.196 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-05T07:59:24.196 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-05T07:59:24.196 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-05T07:59:24.196 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:24.196 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-05T07:59:24.196 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-05T07:59:24.196 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 0 2024-02-05T07:59:24.196 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:59:24.196 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-05T07:59:24.196 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "a" 2024-02-05T07:59:24.196 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:59:24.196 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:24.197 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-05T07:59:24.198 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-05T07:59:24.198 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-05T07:59:24.198 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-05T07:59:24.198 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-05T07:59:24.198 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-05T07:59:24.198 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-05T07:59:24.198 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-05T07:59:24.198 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-05T07:59:24.198 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:24.198 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-05T07:59:24.198 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:24.198 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-05T07:59:24.198 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "restful" 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modified": "2024-02-05T07:59:09.321711+0000", 2024-02-05T07:59:24.199 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:59:24.200 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:24.200 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-05T07:59:24.200 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:59:24.200 INFO:teuthology.orchestra.run.smithi134.stdout:mgr not available, waiting (3/15)... 2024-02-05T07:59:24.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:24 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2289423041' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-05T07:59:24.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:24 smithi134 ceph-mon[30086]: mgrmap e3: y(active, since 1.00611s) 2024-02-05T07:59:26.557 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:26 smithi134 ceph-mon[30086]: mgrmap e4: y(active, since 2s) 2024-02-05T07:59:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:27 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2056489575' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-05T07:59:27.532 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-02-05T07:59:27.532 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:59:27.532 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsid": "43e13aa4-c3fc-11ee-95b6-87774f69a715", 2024-02-05T07:59:27.532 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "health": { 2024-02-05T07:59:27.532 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-05T07:59:27.532 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-05T07:59:27.532 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-05T07:59:27.532 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:27.532 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-05T07:59:27.532 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 0 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "a" 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-05T07:59:27.533 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-05T07:59:27.534 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-05T07:59:27.537 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-05T07:59:27.537 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-05T07:59:27.537 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-05T07:59:27.537 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:27.537 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-05T07:59:27.537 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-05T07:59:27.537 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-05T07:59:27.538 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "restful" 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ], 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "modified": "2024-02-05T07:59:09.321711+0000", 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout }, 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:59:27.539 INFO:teuthology.orchestra.run.smithi134.stdout:mgr is available 2024-02-05T07:59:28.708 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:28 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1827192536' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-05T07:59:28.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-02-05T07:59:28.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [global] 2024-02-05T07:59:28.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout fsid = 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T07:59:28.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-05T07:59:28.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-05T07:59:28.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-05T07:59:28.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-05T07:59:28.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-02-05T07:59:28.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-05T07:59:28.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-05T07:59:28.712 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 2024-02-05T07:59:28.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout [osd] 2024-02-05T07:59:28.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-05T07:59:28.713 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-05T07:59:28.713 INFO:teuthology.orchestra.run.smithi134.stdout:Enabling cephadm module... 2024-02-05T07:59:29.748 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:29 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1025364739' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-05T07:59:29.748 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:29 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ignoring --setuser ceph since I am not root 2024-02-05T07:59:29.748 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:29 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ignoring --setgroup ceph since I am not root 2024-02-05T07:59:30.250 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:29 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:29.997+0000 7f44b0557200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T07:59:30.251 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:30.099+0000 7f44b0557200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T07:59:30.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:30 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1025364739' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-05T07:59:30.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:30 smithi134 ceph-mon[30086]: mgrmap e5: y(active, since 6s) 2024-02-05T07:59:31.005 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:30.756+0000 7f44b0557200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T07:59:31.005 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:30.859+0000 7f44b0557200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T07:59:31.005 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:31 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:31.003+0000 7f44b0557200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T07:59:31.278 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:31 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:31.100+0000 7f44b0557200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T07:59:31.278 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:31 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:31.183+0000 7f44b0557200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T07:59:31.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:59:31.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-05T07:59:31.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-05T07:59:31.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-05T07:59:31.308 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-05T07:59:31.309 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:59:31.309 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for the mgr to restart... 2024-02-05T07:59:31.309 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mgr epoch 5... 2024-02-05T07:59:31.569 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:31 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3745709175' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-05T07:59:31.569 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:31 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:31.452+0000 7f44b0557200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T07:59:31.569 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:31 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:31.567+0000 7f44b0557200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T07:59:31.935 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:31 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:31.657+0000 7f44b0557200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T07:59:31.935 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:31 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:31.755+0000 7f44b0557200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T07:59:32.278 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:31 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:31.933+0000 7f44b0557200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T07:59:33.278 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:32 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:32.885+0000 7f44b0557200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T07:59:33.278 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:33 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:33.015+0000 7f44b0557200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T07:59:33.634 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:33 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:33.348+0000 7f44b0557200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T07:59:33.635 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:33 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:33.438+0000 7f44b0557200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T07:59:33.635 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:33 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:33.633+0000 7f44b0557200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:59:34.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:33 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:33.815+0000 7f44b0557200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T07:59:34.355 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:34 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:34.095+0000 7f44b0557200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T07:59:34.355 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:34 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:34.353+0000 7f44b0557200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T07:59:34.778 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:34 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:34.588+0000 7f44b0557200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T07:59:35.778 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:35 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:35.378+0000 7f44b0557200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T07:59:36.262 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:35 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:35.950+0000 7f44b0557200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T07:59:36.263 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:36.048+0000 7f44b0557200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T07:59:36.263 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:36.260+0000 7f44b0557200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T07:59:36.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: Active manager daemon y restarted 2024-02-05T07:59:36.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: Activating manager daemon y 2024-02-05T07:59:36.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: osdmap e2: 0 total, 0 up, 0 in 2024-02-05T07:59:36.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: mgrmap e6: y(active, starting, since 0.00336777s) 2024-02-05T07:59:36.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T07:59:36.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T07:59:36.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T07:59:36.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T07:59:36.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T07:59:36.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: Manager daemon y is now available 2024-02-05T07:59:36.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' 2024-02-05T07:59:36.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' 2024-02-05T07:59:36.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:59:36.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:36 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:59:37.585 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:37 smithi134 ceph-mon[30086]: Found migration_current of "None". Setting to last migration. 2024-02-05T07:59:37.586 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:37 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:59:37.586 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:37 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T07:59:37.586 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:37 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T07:59:37.586 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:37 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' 2024-02-05T07:59:37.586 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:37 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' 2024-02-05T07:59:37.586 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:37 smithi134 ceph-mon[30086]: mgrmap e7: y(active, since 1.00576s) 2024-02-05T07:59:37.856 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:59:37.856 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-05T07:59:37.856 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-05T07:59:37.856 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:59:37.856 INFO:teuthology.orchestra.run.smithi134.stdout:mgr epoch 5 is available 2024-02-05T07:59:37.856 INFO:teuthology.orchestra.run.smithi134.stdout:Setting orchestrator backend to cephadm... 2024-02-05T07:59:38.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:38 smithi134 ceph-mon[30086]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-05T07:59:38.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:38 smithi134 ceph-mon[30086]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-05T07:59:38.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:38 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:59:39.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:39 smithi134 ceph-mon[30086]: [05/Feb/2024:07:59:37] ENGINE Bus STARTING 2024-02-05T07:59:39.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:39 smithi134 ceph-mon[30086]: [05/Feb/2024:07:59:37] ENGINE Serving on http://172.21.15.134:8765 2024-02-05T07:59:39.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:39 smithi134 ceph-mon[30086]: [05/Feb/2024:07:59:37] ENGINE Serving on https://172.21.15.134:7150 2024-02-05T07:59:39.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:39 smithi134 ceph-mon[30086]: [05/Feb/2024:07:59:37] ENGINE Bus STARTED 2024-02-05T07:59:39.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:39 smithi134 ceph-mon[30086]: mgrmap e8: y(active, since 2s) 2024-02-05T07:59:39.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:39 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' 2024-02-05T07:59:39.512 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:39 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:59:40.302 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-05T07:59:40.302 INFO:teuthology.orchestra.run.smithi134.stdout:Generating ssh key... 2024-02-05T07:59:40.566 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:40 smithi134 ceph-mon[30086]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:59:41.114 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: Generating public/private rsa key pair. 2024-02-05T07:59:41.114 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: Your identification has been saved in /tmp/tmp1eaotxzt/key. 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: Your public key has been saved in /tmp/tmp1eaotxzt/key.pub. 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: The key fingerprint is: 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: SHA256:GP2vXogfQcaEfvOEfnV+QOwpccXQe8S2rx8omC1DUXk ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: The key's randomart image is: 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: +---[RSA 3072]----+ 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: | ......*.| 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: | ..o...E+ *| 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: | ....+..= +o| 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: | o.+= o =oo| 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: | . S+o+ o +o| 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: | ..=+o . +| 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: | .=o+o. o.| 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: | .o+. . .| 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: | .+ ..| 2024-02-05T07:59:41.115 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: +----[SHA256]-----+ 2024-02-05T07:59:41.707 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-mon[30086]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:59:41.707 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' 2024-02-05T07:59:41.707 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:41 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' 2024-02-05T07:59:42.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:42 smithi134 ceph-mon[30086]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:59:42.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:42 smithi134 ceph-mon[30086]: Generating ssh key... 2024-02-05T07:59:42.781 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVr/AUkPkT/mFcebh0N8b0ZkUJD1FNkrz9PVdgWn0ZdL+gQ1x50BYSHduTFxowAVxyW1Nw28jBeItG+jr1NYBeslRAyohVDzs5B9M0LkICxqPc0jdumz3cnF6DIA6Jo7VScb0XD/ni1t9xNmc62iXkHjzzh8Rdjy6zzlUI2JRF/VtbPySlREO2g+qT2P9SqGD2/oiDmCpcJi/6Mwjucvn97/0g51nBx6TlCZomcbZt929IaYP0CcywG6QUwGeSEr2b4SyPmsExdZ+I0yXw/FPfgAkWbiQuroiilf2XW8EypHeJ798ymQELPZoxsu3TM3DxUO1mGT74qDWY16IlJDMAegGz7f8sAlFxcvArL2NDL+V140BWSjxb3rYgMBJrcasqyx3CdQqIUvZPIasW6zgSg3GIZb8M5BmR2q0QNf615yW1JGJqrmprUn5TfxV/VXhWhRDwYRBnSNrPOyWe5jObtax07taGZgzeSVcrt0JGjced7NnJkxGRK6Tu4YcJ+0U= ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T07:59:42.782 INFO:teuthology.orchestra.run.smithi134.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-05T07:59:42.782 INFO:teuthology.orchestra.run.smithi134.stdout:Adding key to root@localhost authorized_keys... 2024-02-05T07:59:42.782 INFO:teuthology.orchestra.run.smithi134.stdout:Adding host smithi134... 2024-02-05T07:59:43.769 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:43 smithi134 ceph-mon[30086]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:59:44.599 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:44 smithi134 ceph-mon[30086]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi134", "addr": "172.21.15.134", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:59:45.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:45 smithi134 ceph-mon[30086]: Deploying cephadm binary to smithi134 2024-02-05T07:59:48.256 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Added host 'smithi134' with addr '172.21.15.134' 2024-02-05T07:59:48.257 INFO:teuthology.orchestra.run.smithi134.stdout:Deploying unmanaged mon service... 2024-02-05T07:59:48.973 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:48 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' 2024-02-05T07:59:48.973 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:48 smithi134 ceph-mon[30086]: Added host smithi134 2024-02-05T07:59:48.973 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:48 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T07:59:49.774 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-05T07:59:49.774 INFO:teuthology.orchestra.run.smithi134.stdout:Deploying unmanaged mgr service... 2024-02-05T07:59:50.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:49 smithi134 ceph-mon[30086]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:59:50.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:49 smithi134 ceph-mon[30086]: Saving service mon spec with placement count:5 2024-02-05T07:59:50.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:49 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' 2024-02-05T07:59:51.258 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-05T07:59:51.844 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:51 smithi134 ceph-mon[30086]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T07:59:51.844 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:51 smithi134 ceph-mon[30086]: Saving service mgr spec with placement count:2 2024-02-05T07:59:51.845 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:51 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' 2024-02-05T07:59:53.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:53 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' 2024-02-05T07:59:53.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:53 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/714156832' entity='client.admin' 2024-02-05T07:59:53.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:53 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' 2024-02-05T07:59:54.200 INFO:teuthology.orchestra.run.smithi134.stdout:Enabling the dashboard module... 2024-02-05T07:59:54.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:54 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2978865071' entity='client.admin' 2024-02-05T07:59:56.165 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:55 smithi134 ceph-mon[30086]: from='mgr.14120 172.21.15.134:0/935842266' entity='mgr.y' 2024-02-05T07:59:56.166 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:55 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1691525392' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-05T07:59:56.166 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:55 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ignoring --setuser ceph since I am not root 2024-02-05T07:59:56.166 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:55 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ignoring --setgroup ceph since I am not root 2024-02-05T07:59:56.685 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:56.417+0000 7f29773dc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T07:59:56.685 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:56.523+0000 7f29773dc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T07:59:57.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:56 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1691525392' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-05T07:59:57.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:56 smithi134 ceph-mon[30086]: mgrmap e9: y(active, since 19s) 2024-02-05T07:59:57.522 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:57.192+0000 7f29773dc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T07:59:57.522 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:57.294+0000 7f29773dc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T07:59:57.522 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:57.436+0000 7f29773dc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T07:59:57.778 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:57.532+0000 7f29773dc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T07:59:57.778 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:57.616+0000 7f29773dc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T07:59:57.809 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-02-05T07:59:57.809 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-05T07:59:57.809 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-05T07:59:57.809 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-05T07:59:57.809 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-05T07:59:57.809 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-02-05T07:59:57.809 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for the mgr to restart... 2024-02-05T07:59:57.809 INFO:teuthology.orchestra.run.smithi134.stdout:Waiting for mgr epoch 9... 2024-02-05T07:59:58.100 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 07:59:57 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2612665959' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-05T07:59:58.100 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:57.892+0000 7f29773dc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T07:59:58.101 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:58 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:58.006+0000 7f29773dc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T07:59:58.101 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:58 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:58.098+0000 7f29773dc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T07:59:58.376 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:58 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:58.195+0000 7f29773dc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T07:59:58.376 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:58 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:58.374+0000 7f29773dc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T07:59:59.778 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:59 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:59.326+0000 7f29773dc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T07:59:59.778 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:59 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:59.456+0000 7f29773dc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T08:00:00.071 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:59 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:59.787+0000 7f29773dc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T08:00:00.072 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 07:59:59 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T07:59:59.877+0000 7f29773dc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T08:00:00.072 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:00:00 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:00:00.070+0000 7f29773dc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T08:00:00.527 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:00:00 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:00:00.252+0000 7f29773dc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T08:00:00.795 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:00:00 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:00:00.533+0000 7f29773dc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T08:00:00.795 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:00:00 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:00:00.793+0000 7f29773dc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T08:00:01.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:00:01 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:00:01.027+0000 7f29773dc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T08:00:02.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:00:01 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:00:01.815+0000 7f29773dc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T08:00:02.696 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:00:02.385+0000 7f29773dc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T08:00:02.696 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:00:02.482+0000 7f29773dc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T08:00:02.696 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:00:02.694+0000 7f29773dc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T08:00:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-mon[30086]: Active manager daemon y restarted 2024-02-05T08:00:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-mon[30086]: Activating manager daemon y 2024-02-05T08:00:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-mon[30086]: osdmap e3: 0 total, 0 up, 0 in 2024-02-05T08:00:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-mon[30086]: mgrmap e10: y(active, starting, since 0.00338065s) 2024-02-05T08:00:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:00:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T08:00:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T08:00:03.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T08:00:03.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T08:00:03.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-mon[30086]: Manager daemon y is now available 2024-02-05T08:00:03.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:00:03.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:00:04.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T08:00:04.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T08:00:04.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:03 smithi134 ceph-mon[30086]: mgrmap e11: y(active, since 1.00645s) 2024-02-05T08:00:04.702 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout { 2024-02-05T08:00:04.703 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-05T08:00:04.703 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-05T08:00:04.703 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout } 2024-02-05T08:00:04.703 INFO:teuthology.orchestra.run.smithi134.stdout:mgr epoch 9 is available 2024-02-05T08:00:04.703 INFO:teuthology.orchestra.run.smithi134.stdout:Generating a dashboard self-signed certificate... 2024-02-05T08:00:04.971 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:04 smithi134 ceph-mon[30086]: [05/Feb/2024:08:00:03] ENGINE Bus STARTING 2024-02-05T08:00:04.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:04 smithi134 ceph-mon[30086]: [05/Feb/2024:08:00:03] ENGINE Serving on https://172.21.15.134:7150 2024-02-05T08:00:04.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:04 smithi134 ceph-mon[30086]: [05/Feb/2024:08:00:03] ENGINE Serving on http://172.21.15.134:8765 2024-02-05T08:00:04.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:04 smithi134 ceph-mon[30086]: [05/Feb/2024:08:00:03] ENGINE Bus STARTED 2024-02-05T08:00:04.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:04 smithi134 ceph-mon[30086]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-05T08:00:04.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:04 smithi134 ceph-mon[30086]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-05T08:00:04.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:04 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:06.007 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-05T08:00:06.007 INFO:teuthology.orchestra.run.smithi134.stdout:Creating initial admin user... 2024-02-05T08:00:06.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:06.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:06 smithi134 ceph-mon[30086]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:00:06.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:06 smithi134 ceph-mon[30086]: mgrmap e12: y(active, since 2s) 2024-02-05T08:00:06.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:06.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:07.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:07 smithi134 ceph-mon[30086]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:00:07.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:07 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:07.702 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$aTaUZSMb4mwRcnjjfyQfw.4t8Q/HqPse6ErKw.sJZcn6D5dXLKg5m", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707120006, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-05T08:00:07.702 INFO:teuthology.orchestra.run.smithi134.stdout:Fetching dashboard port number... 2024-02-05T08:00:09.454 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stdout 8443 2024-02-05T08:00:09.463 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-05T08:00:09.463 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout disabled 2024-02-05T08:00:09.470 INFO:teuthology.orchestra.run.smithi134.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-05T08:00:09.470 INFO:teuthology.orchestra.run.smithi134.stdout:systemctl: stdout inactive 2024-02-05T08:00:09.470 INFO:teuthology.orchestra.run.smithi134.stdout:firewalld.service is not enabled 2024-02-05T08:00:09.470 INFO:teuthology.orchestra.run.smithi134.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-05T08:00:09.472 INFO:teuthology.orchestra.run.smithi134.stdout:Ceph Dashboard is now available at: 2024-02-05T08:00:09.472 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:00:09.472 INFO:teuthology.orchestra.run.smithi134.stdout: URL: https://smithi134.front.sepia.ceph.com:8443/ 2024-02-05T08:00:09.472 INFO:teuthology.orchestra.run.smithi134.stdout: User: admin 2024-02-05T08:00:09.472 INFO:teuthology.orchestra.run.smithi134.stdout: Password: 9rq573pqmi 2024-02-05T08:00:09.472 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:00:09.472 INFO:teuthology.orchestra.run.smithi134.stdout:Saving cluster configuration to /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config directory 2024-02-05T08:00:09.473 INFO:teuthology.orchestra.run.smithi134.stdout:Enabling autotune for osd_memory_target 2024-02-05T08:00:10.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:09 smithi134 ceph-mon[30086]: mgrmap e13: y(active, since 6s) 2024-02-05T08:00:10.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:09 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2971852873' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-05T08:00:12.017 INFO:teuthology.orchestra.run.smithi134.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-05T08:00:12.017 INFO:teuthology.orchestra.run.smithi134.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-05T08:00:12.017 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:00:12.017 INFO:teuthology.orchestra.run.smithi134.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-05T08:00:12.017 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:00:12.017 INFO:teuthology.orchestra.run.smithi134.stdout:Or, if you are only running a single cluster on this host: 2024-02-05T08:00:12.017 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:00:12.017 INFO:teuthology.orchestra.run.smithi134.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-05T08:00:12.017 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:00:12.017 INFO:teuthology.orchestra.run.smithi134.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-05T08:00:12.018 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:00:12.018 INFO:teuthology.orchestra.run.smithi134.stdout: ceph telemetry on 2024-02-05T08:00:12.018 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:00:12.018 INFO:teuthology.orchestra.run.smithi134.stdout:For more information see: 2024-02-05T08:00:12.018 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:00:12.018 INFO:teuthology.orchestra.run.smithi134.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-05T08:00:12.018 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:00:12.018 INFO:teuthology.orchestra.run.smithi134.stdout:Bootstrap complete. 2024-02-05T08:00:12.058 INFO:tasks.cephadm:Fetching config... 2024-02-05T08:00:12.058 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T08:00:12.059 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-05T08:00:12.074 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-05T08:00:12.074 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T08:00:12.074 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-05T08:00:12.131 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-05T08:00:12.131 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T08:00:12.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-05T08:00:12.198 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-05T08:00:12.198 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T08:00:12.198 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-05T08:00:12.253 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-05T08:00:12.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVr/AUkPkT/mFcebh0N8b0ZkUJD1FNkrz9PVdgWn0ZdL+gQ1x50BYSHduTFxowAVxyW1Nw28jBeItG+jr1NYBeslRAyohVDzs5B9M0LkICxqPc0jdumz3cnF6DIA6Jo7VScb0XD/ni1t9xNmc62iXkHjzzh8Rdjy6zzlUI2JRF/VtbPySlREO2g+qT2P9SqGD2/oiDmCpcJi/6Mwjucvn97/0g51nBx6TlCZomcbZt929IaYP0CcywG6QUwGeSEr2b4SyPmsExdZ+I0yXw/FPfgAkWbiQuroiilf2XW8EypHeJ798ymQELPZoxsu3TM3DxUO1mGT74qDWY16IlJDMAegGz7f8sAlFxcvArL2NDL+V140BWSjxb3rYgMBJrcasqyx3CdQqIUvZPIasW6zgSg3GIZb8M5BmR2q0QNf615yW1JGJqrmprUn5TfxV/VXhWhRDwYRBnSNrPOyWe5jObtax07taGZgzeSVcrt0JGjced7NnJkxGRK6Tu4YcJ+0U= ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-05T08:00:12.333 INFO:teuthology.orchestra.run.smithi134.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVr/AUkPkT/mFcebh0N8b0ZkUJD1FNkrz9PVdgWn0ZdL+gQ1x50BYSHduTFxowAVxyW1Nw28jBeItG+jr1NYBeslRAyohVDzs5B9M0LkICxqPc0jdumz3cnF6DIA6Jo7VScb0XD/ni1t9xNmc62iXkHjzzh8Rdjy6zzlUI2JRF/VtbPySlREO2g+qT2P9SqGD2/oiDmCpcJi/6Mwjucvn97/0g51nBx6TlCZomcbZt929IaYP0CcywG6QUwGeSEr2b4SyPmsExdZ+I0yXw/FPfgAkWbiQuroiilf2XW8EypHeJ798ymQELPZoxsu3TM3DxUO1mGT74qDWY16IlJDMAegGz7f8sAlFxcvArL2NDL+V140BWSjxb3rYgMBJrcasqyx3CdQqIUvZPIasW6zgSg3GIZb8M5BmR2q0QNf615yW1JGJqrmprUn5TfxV/VXhWhRDwYRBnSNrPOyWe5jObtax07taGZgzeSVcrt0JGjced7NnJkxGRK6Tu4YcJ+0U= ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T08:00:12.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVr/AUkPkT/mFcebh0N8b0ZkUJD1FNkrz9PVdgWn0ZdL+gQ1x50BYSHduTFxowAVxyW1Nw28jBeItG+jr1NYBeslRAyohVDzs5B9M0LkICxqPc0jdumz3cnF6DIA6Jo7VScb0XD/ni1t9xNmc62iXkHjzzh8Rdjy6zzlUI2JRF/VtbPySlREO2g+qT2P9SqGD2/oiDmCpcJi/6Mwjucvn97/0g51nBx6TlCZomcbZt929IaYP0CcywG6QUwGeSEr2b4SyPmsExdZ+I0yXw/FPfgAkWbiQuroiilf2XW8EypHeJ798ymQELPZoxsu3TM3DxUO1mGT74qDWY16IlJDMAegGz7f8sAlFxcvArL2NDL+V140BWSjxb3rYgMBJrcasqyx3CdQqIUvZPIasW6zgSg3GIZb8M5BmR2q0QNf615yW1JGJqrmprUn5TfxV/VXhWhRDwYRBnSNrPOyWe5jObtax07taGZgzeSVcrt0JGjced7NnJkxGRK6Tu4YcJ+0U= ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-05T08:00:12.396 INFO:teuthology.orchestra.run.smithi196.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVr/AUkPkT/mFcebh0N8b0ZkUJD1FNkrz9PVdgWn0ZdL+gQ1x50BYSHduTFxowAVxyW1Nw28jBeItG+jr1NYBeslRAyohVDzs5B9M0LkICxqPc0jdumz3cnF6DIA6Jo7VScb0XD/ni1t9xNmc62iXkHjzzh8Rdjy6zzlUI2JRF/VtbPySlREO2g+qT2P9SqGD2/oiDmCpcJi/6Mwjucvn97/0g51nBx6TlCZomcbZt929IaYP0CcywG6QUwGeSEr2b4SyPmsExdZ+I0yXw/FPfgAkWbiQuroiilf2XW8EypHeJ798ymQELPZoxsu3TM3DxUO1mGT74qDWY16IlJDMAegGz7f8sAlFxcvArL2NDL+V140BWSjxb3rYgMBJrcasqyx3CdQqIUvZPIasW6zgSg3GIZb8M5BmR2q0QNf615yW1JGJqrmprUn5TfxV/VXhWhRDwYRBnSNrPOyWe5jObtax07taGZgzeSVcrt0JGjced7NnJkxGRK6Tu4YcJ+0U= ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T08:00:12.411 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-05T08:00:12.646 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:00:12.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:12 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/410960377' entity='client.admin' 2024-02-05T08:00:15.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:14 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:15.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:14 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:15.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:14 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi134", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:00:15.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:14 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:15.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:14 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:00:15.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:14 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:15.585 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-05T08:00:15.586 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-05T08:00:15.817 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:00:16.269 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:15 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/252993826' entity='client.admin' 2024-02-05T08:00:16.269 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:15 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:00:16.269 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:15 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:18.377 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi196 2024-02-05T08:00:18.377 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T08:00:18.377 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.conf 2024-02-05T08:00:18.397 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T08:00:18.397 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-05T08:00:18.453 INFO:tasks.cephadm:Adding host smithi196 to orchestrator... 2024-02-05T08:00:18.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch host add smithi196 2024-02-05T08:00:18.690 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:00:18.929 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:18 smithi134 ceph-mon[30086]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:00:18.929 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:18 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:18.929 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:18 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:00:18.929 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:18 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:18.929 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:18 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:00:18.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:18 smithi134 ceph-mon[30086]: Updating smithi134:/etc/ceph/ceph.conf 2024-02-05T08:00:18.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:18 smithi134 ceph-mon[30086]: Updating smithi134:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:00:18.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:18 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:18.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:18 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:18.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:18 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:19.946 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:19 smithi134 ceph-mon[30086]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-02-05T08:00:19.946 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:19 smithi134 ceph-mon[30086]: Updating smithi134:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T08:00:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:20 smithi134 ceph-mon[30086]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:00:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:21 smithi134 ceph-mon[30086]: Deploying cephadm binary to smithi196 2024-02-05T08:00:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:23 smithi134 ceph-mon[30086]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:24.978 INFO:teuthology.orchestra.run.smithi134.stdout:Added host 'smithi196' with addr '172.21.15.196' 2024-02-05T08:00:25.634 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:25 smithi134 ceph-mon[30086]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:25.634 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:25 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:25.634 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:25 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:00:25.635 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch host ls --format=json 2024-02-05T08:00:25.873 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:00:26.756 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:26 smithi134 ceph-mon[30086]: Added host smithi196 2024-02-05T08:00:26.757 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:26 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:26.757 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:26 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:27.445 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:00:27.445 INFO:teuthology.orchestra.run.smithi134.stdout:[{"addr": "172.21.15.134", "hostname": "smithi134", "labels": [], "status": ""}, {"addr": "172.21.15.196", "hostname": "smithi196", "labels": [], "status": ""}] 2024-02-05T08:00:27.754 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:27 smithi134 ceph-mon[30086]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:28.032 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-05T08:00:28.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd crush tunables default 2024-02-05T08:00:28.264 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:00:28.640 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:28 smithi134 ceph-mon[30086]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:00:28.640 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:28 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:29.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:29 smithi134 ceph-mon[30086]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:30.653 INFO:teuthology.orchestra.run.smithi134.stderr:adjusted tunables profile to default 2024-02-05T08:00:30.949 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:30 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/418301971' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-05T08:00:31.260 INFO:tasks.cephadm:Adding mon.a on smithi134 2024-02-05T08:00:31.260 INFO:tasks.cephadm:Adding mon.c on smithi134 2024-02-05T08:00:31.260 INFO:tasks.cephadm:Adding mon.b on smithi196 2024-02-05T08:00:31.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch apply mon '3;smithi134:172.21.15.134=a;smithi134:[v2:172.21.15.134:3301,v1:172.21.15.134:6790]=c;smithi196:172.21.15.196=b' 2024-02-05T08:00:31.493 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:00:32.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/418301971' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-05T08:00:32.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T08:00:32.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:32.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:32.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:32.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:32.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:32.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:00:32.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:32.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:00:32.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: Updating smithi196:/etc/ceph/ceph.conf 2024-02-05T08:00:32.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:32.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:32.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:31 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:33.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:32 smithi134 ceph-mon[30086]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:00:33.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:32 smithi134 ceph-mon[30086]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-02-05T08:00:33.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:32 smithi134 ceph-mon[30086]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T08:00:33.283 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mon update... 2024-02-05T08:00:33.849 DEBUG:teuthology.orchestra.run.smithi134:mon.c> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.c.service 2024-02-05T08:00:33.852 DEBUG:teuthology.orchestra.run.smithi196:mon.b> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.b.service 2024-02-05T08:00:33.854 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-05T08:00:33.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph mon dump -f json 2024-02-05T08:00:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:33 smithi134 ceph-mon[30086]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:34.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:33 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:34.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:33 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:00:34.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:33 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:34.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:33 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:00:34.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:33 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:34.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:33 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:34.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:33 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:00:35.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:34 smithi134 ceph-mon[30086]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi134:172.21.15.134=a;smithi134:[v2:172.21.15.134:3301,v1:172.21.15.134:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:00:35.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:34 smithi134 ceph-mon[30086]: Saving service mon spec with placement smithi134:172.21.15.134=a;smithi134:[v2:172.21.15.134:3301,v1:172.21.15.134:6790]=c;smithi196:172.21.15.196=b;count:3 2024-02-05T08:00:35.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:34 smithi134 ceph-mon[30086]: Deploying daemon mon.b on smithi196 2024-02-05T08:00:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:35 smithi134 ceph-mon[30086]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:37.381 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T08:00:37.382 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"43e13aa4-c3fc-11ee-95b6-87774f69a715","modified":"2024-02-05T07:59:04.254061Z","created":"2024-02-05T07:59:04.254061Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-05T08:00:37.384 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-02-05T08:00:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:37 smithi134 ceph-mon[30086]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:38.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:37 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/2242712204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:00:39.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:38 smithi196 podman[32981]: 2024-02-05 08:00:38.821903405 +0000 UTC m=+0.016830222 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:00:39.290 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-05T08:00:39.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph mon dump -f json 2024-02-05T08:00:39.524 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:00:39.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 podman[32981]: 2024-02-05 08:00:39.080971141 +0000 UTC m=+0.275897953 container init dd595281e22390b0715271bd408ec79e8a529e0f4f63da3b3079ad71b2042612 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-b, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-05T08:00:39.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 podman[32981]: 2024-02-05 08:00:39.086476355 +0000 UTC m=+0.281403170 container start dd595281e22390b0715271bd408ec79e8a529e0f4f63da3b3079ad71b2042612 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-b, org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T08:00:39.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: set uid:gid to 167:167 (ceph:ceph) 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev), process ceph-mon, pid 7 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: pidfile_write: ignore empty --pid-file 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: load: jerasure load: lrc 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: RocksDB version: 7.9.2 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Git sha 0 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Compile date 2024-02-04 23:11:06 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: DB SUMMARY 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: DB Session ID: LFFRBAXM95JG70Z2A6CW 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: CURRENT file: CURRENT 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: IDENTITY file: IDENTITY 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.error_if_exists: 0 2024-02-05T08:00:39.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.create_if_missing: 0 2024-02-05T08:00:39.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.paranoid_checks: 1 2024-02-05T08:00:39.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-05T08:00:39.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-05T08:00:39.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-05T08:00:39.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.env: 0x563c67b00ae0 2024-02-05T08:00:39.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.fs: PosixFileSystem 2024-02-05T08:00:39.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.info_log: 0x563c687ab8c0 2024-02-05T08:00:39.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_file_opening_threads: 16 2024-02-05T08:00:39.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.statistics: (nil) 2024-02-05T08:00:39.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.use_fsync: 0 2024-02-05T08:00:39.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_log_file_size: 0 2024-02-05T08:00:39.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-05T08:00:39.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-05T08:00:39.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.keep_log_file_num: 1000 2024-02-05T08:00:39.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.recycle_log_file_num: 0 2024-02-05T08:00:39.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.allow_fallocate: 1 2024-02-05T08:00:39.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.allow_mmap_reads: 0 2024-02-05T08:00:39.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.allow_mmap_writes: 0 2024-02-05T08:00:39.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.use_direct_reads: 0 2024-02-05T08:00:39.569 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-05T08:00:39.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.create_missing_column_families: 0 2024-02-05T08:00:39.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.db_log_dir: 2024-02-05T08:00:39.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.wal_dir: 2024-02-05T08:00:39.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-05T08:00:39.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-05T08:00:39.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-05T08:00:39.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-05T08:00:39.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-05T08:00:39.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-05T08:00:39.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.advise_random_on_open: 1 2024-02-05T08:00:39.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.db_write_buffer_size: 0 2024-02-05T08:00:39.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.write_buffer_manager: 0x563c6877f400 2024-02-05T08:00:39.570 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.rate_limiter: (nil) 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.wal_recovery_mode: 2 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.enable_thread_tracking: 0 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.enable_pipelined_write: 0 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.unordered_write: 0 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.row_cache: None 2024-02-05T08:00:39.571 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.wal_filter: None 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.allow_ingest_behind: 0 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.two_write_queues: 0 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.manual_wal_flush: 0 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.wal_compression: 0 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.atomic_flush: 0 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.log_readahead_size: 0 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.best_efforts_recovery: 0 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-05T08:00:39.572 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.allow_data_in_errors: 0 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.db_host_id: __hostname__ 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_background_jobs: 2 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_background_compactions: -1 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_subcompactions: 1 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_total_wal_size: 0 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-05T08:00:39.573 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_open_files: -1 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.bytes_per_sync: 0 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compaction_readahead_size: 0 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_background_flushes: -1 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Compression algorithms supported: 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: kZSTD supported: 0 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: kXpressCompression supported: 0 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: kLZ4HCCompression supported: 1 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: kZlibCompression supported: 1 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: kSnappyCompression supported: 1 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: kLZ4Compression supported: 1 2024-02-05T08:00:39.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: kBZip2Compression supported: 0 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.merge_operator: 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compaction_filter: None 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compaction_filter_factory: None 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.sst_partitioner_factory: None 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563c687ab9c0) 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks: 1 2024-02-05T08:00:39.575 INFO:journalctl@ceph.mon.b.smithi196.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-05T08:00:39.576 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-05T08:00:39.576 INFO:journalctl@ceph.mon.b.smithi196.stdout: pin_top_level_index_and_filter: 1 2024-02-05T08:00:39.576 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_type: 0 2024-02-05T08:00:39.576 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_index_type: 0 2024-02-05T08:00:39.576 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_shortening: 1 2024-02-05T08:00:39.576 INFO:journalctl@ceph.mon.b.smithi196.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-05T08:00:39.576 INFO:journalctl@ceph.mon.b.smithi196.stdout: checksum: 4 2024-02-05T08:00:39.576 INFO:journalctl@ceph.mon.b.smithi196.stdout: no_block_cache: 0 2024-02-05T08:00:39.576 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache: 0x563c687af090 2024-02-05T08:00:39.576 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_name: BinnedLRUCache 2024-02-05T08:00:39.576 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_options: 2024-02-05T08:00:39.576 INFO:journalctl@ceph.mon.b.smithi196.stdout: capacity : 536870912 2024-02-05T08:00:39.576 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_shard_bits : 4 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: strict_capacity_limit : 0 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: high_pri_pool_ratio: 0.000 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_cache_compressed: (nil) 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: persistent_cache: (nil) 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size: 4096 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_size_deviation: 10 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_restart_interval: 16 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: index_block_restart_interval: 1 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: metadata_block_size: 4096 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: partition_filters: 0 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: use_delta_encoding: 1 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: filter_policy: bloomfilter 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: whole_key_filtering: 1 2024-02-05T08:00:39.577 INFO:journalctl@ceph.mon.b.smithi196.stdout: verify_compression: 0 2024-02-05T08:00:39.578 INFO:journalctl@ceph.mon.b.smithi196.stdout: read_amp_bytes_per_bit: 0 2024-02-05T08:00:39.578 INFO:journalctl@ceph.mon.b.smithi196.stdout: format_version: 5 2024-02-05T08:00:39.578 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_index_compression: 1 2024-02-05T08:00:39.578 INFO:journalctl@ceph.mon.b.smithi196.stdout: block_align: 0 2024-02-05T08:00:39.578 INFO:journalctl@ceph.mon.b.smithi196.stdout: max_auto_readahead_size: 262144 2024-02-05T08:00:39.578 INFO:journalctl@ceph.mon.b.smithi196.stdout: prepopulate_block_cache: 0 2024-02-05T08:00:39.578 INFO:journalctl@ceph.mon.b.smithi196.stdout: initial_auto_readahead_size: 8192 2024-02-05T08:00:39.578 INFO:journalctl@ceph.mon.b.smithi196.stdout: num_file_reads_for_auto_readahead: 2 2024-02-05T08:00:39.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.write_buffer_size: 33554432 2024-02-05T08:00:39.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_write_buffer_number: 2 2024-02-05T08:00:39.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compression: NoCompression 2024-02-05T08:00:39.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.bottommost_compression: Disabled 2024-02-05T08:00:39.578 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.prefix_extractor: nullptr 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.num_levels: 7 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-05T08:00:39.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compression_opts.level: 32767 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compression_opts.strategy: 0 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compression_opts.enabled: false 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.target_file_size_base: 67108864 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-05T08:00:39.580 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.arena_block_size: 1048576 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-05T08:00:39.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.disable_auto_compactions: 0 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.table_properties_collectors: 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.inplace_update_support: 0 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-05T08:00:39.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.bloom_locality: 0 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.max_successive_merges: 0 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.paranoid_file_checks: 0 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.force_consistency_checks: 1 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.report_bg_io_stats: 0 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.ttl: 2592000 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.enable_blob_files: false 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.min_blob_size: 0 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.blob_file_size: 268435456 2024-02-05T08:00:39.583 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-05T08:00:39.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-05T08:00:39.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-05T08:00:39.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-05T08:00:39.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-05T08:00:39.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.blob_file_starting_level: 0 2024-02-05T08:00:39.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-05T08:00:39.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-05T08:00:39.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-05T08:00:39.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cd9403ae-2d16-4995-8669-58a6e801019b 2024-02-05T08:00:39.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707120039115674, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-05T08:00:39.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-05T08:00:39.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707120039116089, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707120039, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cd9403ae-2d16-4995-8669-58a6e801019b", "db_session_id": "LFFRBAXM95JG70Z2A6CW", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-05T08:00:39.584 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707120039116167, "job": 1, "event": "recovery_finished"} 2024-02-05T08:00:39.585 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-05T08:00:39.585 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-05T08:00:39.585 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563c688a0000 2024-02-05T08:00:39.585 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: DB pointer 0x563c6888a000 2024-02-05T08:00:39.585 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-05T08:00:39.585 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-05T08:00:39.585 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** DB Stats ** 2024-02-05T08:00:39.585 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T08:00:39.585 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-05T08:00:39.585 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T08:00:39.585 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T08:00:39.585 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-05T08:00:39.585 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** Compaction Stats [default] ** 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-02-05T08:00:39.586 INFO:journalctl@ceph.mon.b.smithi196.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache BinnedLRUCache@0x563c687af090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-02-05T08:00:39.587 INFO:journalctl@ceph.mon.b.smithi196.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-05T08:00:39.588 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-05T08:00:39.588 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: using public_addr v2:172.21.15.196:0/0 -> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] 2024-02-05T08:00:39.588 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: starting mon.b rank -1 at public addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] at bind addrs [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T08:00:39.588 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(???) e0 preinit fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T08:00:39.588 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(synchronizing).mds e1 new map 2024-02-05T08:00:39.588 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(synchronizing).mds e1 print_map 2024-02-05T08:00:39.588 INFO:journalctl@ceph.mon.b.smithi196.stdout: e1 2024-02-05T08:00:39.588 INFO:journalctl@ceph.mon.b.smithi196.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-05T08:00:39.588 INFO:journalctl@ceph.mon.b.smithi196.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-05T08:00:39.588 INFO:journalctl@ceph.mon.b.smithi196.stdout: legacy client fscid: -1 2024-02-05T08:00:39.588 INFO:journalctl@ceph.mon.b.smithi196.stdout: 2024-02-05T08:00:39.588 INFO:journalctl@ceph.mon.b.smithi196.stdout: No filesystems configured 2024-02-05T08:00:39.588 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/418301971' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-05T08:00:39.589 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/418301971' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: Updating smithi196:/etc/ceph/ceph.conf 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:39.590 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi134:172.21.15.134=a;smithi134:[v2:172.21.15.134:3301,v1:172.21.15.134:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: Saving service mon spec with placement smithi134:172.21.15.134=a;smithi134:[v2:172.21.15.134:3301,v1:172.21.15.134:6790]=c;smithi196:172.21.15.196=b;count:3 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: Deploying daemon mon.b on smithi196 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:39.591 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:39.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/2242712204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:00:39.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 ceph-mon[33018]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-05T08:00:39.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 bash[32981]: dd595281e22390b0715271bd408ec79e8a529e0f4f63da3b3079ad71b2042612 2024-02-05T08:00:39.592 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:39 smithi196 systemd[1]: Started Ceph mon.b for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:00:44.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:00:44.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:44.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: mon.a calling monitor election 2024-02-05T08:00:44.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:44.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:44.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:44.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: mon.b calling monitor election 2024-02-05T08:00:44.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:44.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:44.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:44.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:44.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-05T08:00:44.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: monmap e2: 2 mons at {a=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T08:00:44.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: fsmap 2024-02-05T08:00:44.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T08:00:44.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: mgrmap e13: y(active, since 41s) 2024-02-05T08:00:44.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: overall HEALTH_OK 2024-02-05T08:00:44.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:44.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:44.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:44.530 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:44.530 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:00:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: mon.a calling monitor election 2024-02-05T08:00:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: mon.b calling monitor election 2024-02-05T08:00:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:44.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-05T08:00:44.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: monmap e2: 2 mons at {a=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T08:00:44.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: fsmap 2024-02-05T08:00:44.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T08:00:44.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: mgrmap e13: y(active, since 41s) 2024-02-05T08:00:44.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: overall HEALTH_OK 2024-02-05T08:00:44.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:44.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:44.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:44.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:44.566 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:44.643 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T08:00:44.643 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":2,"fsid":"43e13aa4-c3fc-11ee-95b6-87774f69a715","modified":"2024-02-05T08:00:39.136940Z","created":"2024-02-05T07:59:04.254061Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-05T08:00:44.646 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 2 2024-02-05T08:00:45.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:45 smithi134 ceph-mon[30086]: Deploying daemon mon.c on smithi134 2024-02-05T08:00:45.730 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:45 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/3975265034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:00:45.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:45 smithi134 ceph-mon[30086]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:45.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:45 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:45.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:45 smithi196 ceph-mon[33018]: Deploying daemon mon.c on smithi134 2024-02-05T08:00:45.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:45 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/3975265034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:00:45.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:45 smithi196 ceph-mon[33018]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:45.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:45 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:46.363 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-05T08:00:46.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph mon dump -f json 2024-02-05T08:00:46.611 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:00:47.355 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 podman[39240]: 2024-02-05 08:00:47.354432161 +0000 UTC m=+0.193468189 container create 59ded051c19d862b7d30921ee79822829ec82c52091b6c8006f09a7cee7aff32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-05T08:00:47.616 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 podman[39240]: 2024-02-05 08:00:47.531739454 +0000 UTC m=+0.370775440 container init 59ded051c19d862b7d30921ee79822829ec82c52091b6c8006f09a7cee7aff32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131) 2024-02-05T08:00:47.616 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 podman[39240]: 2024-02-05 08:00:47.537359616 +0000 UTC m=+0.376395605 container start 59ded051c19d862b7d30921ee79822829ec82c52091b6c8006f09a7cee7aff32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0) 2024-02-05T08:00:47.616 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: set uid:gid to 167:167 (ceph:ceph) 2024-02-05T08:00:47.616 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev), process ceph-mon, pid 7 2024-02-05T08:00:47.616 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: pidfile_write: ignore empty --pid-file 2024-02-05T08:00:47.616 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: load: jerasure load: lrc 2024-02-05T08:00:47.616 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: RocksDB version: 7.9.2 2024-02-05T08:00:47.616 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Git sha 0 2024-02-05T08:00:47.617 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Compile date 2024-02-04 23:11:06 2024-02-05T08:00:47.617 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: DB SUMMARY 2024-02-05T08:00:47.617 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: DB Session ID: GYSVEM7ID1BMWQQIGA56 2024-02-05T08:00:47.617 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: CURRENT file: CURRENT 2024-02-05T08:00:47.617 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: IDENTITY file: IDENTITY 2024-02-05T08:00:47.617 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-05T08:00:47.617 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-05T08:00:47.617 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-02-05T08:00:47.617 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.error_if_exists: 0 2024-02-05T08:00:47.617 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.create_if_missing: 0 2024-02-05T08:00:47.617 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.paranoid_checks: 1 2024-02-05T08:00:47.618 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-05T08:00:47.618 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-05T08:00:47.618 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-05T08:00:47.618 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.env: 0x5617af65dae0 2024-02-05T08:00:47.618 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.fs: PosixFileSystem 2024-02-05T08:00:47.618 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.info_log: 0x5617b11038c0 2024-02-05T08:00:47.618 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_file_opening_threads: 16 2024-02-05T08:00:47.620 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.statistics: (nil) 2024-02-05T08:00:47.620 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.use_fsync: 0 2024-02-05T08:00:47.620 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_log_file_size: 0 2024-02-05T08:00:47.620 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-05T08:00:47.621 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-05T08:00:47.621 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.keep_log_file_num: 1000 2024-02-05T08:00:47.621 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.recycle_log_file_num: 0 2024-02-05T08:00:47.621 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.allow_fallocate: 1 2024-02-05T08:00:47.621 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.allow_mmap_reads: 0 2024-02-05T08:00:47.621 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.allow_mmap_writes: 0 2024-02-05T08:00:47.621 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.use_direct_reads: 0 2024-02-05T08:00:47.621 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-05T08:00:47.621 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.create_missing_column_families: 0 2024-02-05T08:00:47.621 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.db_log_dir: 2024-02-05T08:00:47.621 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.wal_dir: 2024-02-05T08:00:47.621 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-05T08:00:47.621 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.advise_random_on_open: 1 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.db_write_buffer_size: 0 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.write_buffer_manager: 0x5617b10d7400 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.rate_limiter: (nil) 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.wal_recovery_mode: 2 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.enable_thread_tracking: 0 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.enable_pipelined_write: 0 2024-02-05T08:00:47.622 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.unordered_write: 0 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.row_cache: None 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.wal_filter: None 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.allow_ingest_behind: 0 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.two_write_queues: 0 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.manual_wal_flush: 0 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.wal_compression: 0 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.atomic_flush: 0 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-05T08:00:47.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.log_readahead_size: 0 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.best_efforts_recovery: 0 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.allow_data_in_errors: 0 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.db_host_id: __hostname__ 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_background_jobs: 2 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_background_compactions: -1 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_subcompactions: 1 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-05T08:00:47.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_total_wal_size: 0 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_open_files: -1 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.bytes_per_sync: 0 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compaction_readahead_size: 0 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_background_flushes: -1 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Compression algorithms supported: 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: kZSTD supported: 0 2024-02-05T08:00:47.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: kXpressCompression supported: 0 2024-02-05T08:00:47.626 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: kLZ4HCCompression supported: 1 2024-02-05T08:00:47.626 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: kZlibCompression supported: 1 2024-02-05T08:00:47.626 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: kSnappyCompression supported: 1 2024-02-05T08:00:47.626 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: kLZ4Compression supported: 1 2024-02-05T08:00:47.626 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: kBZip2Compression supported: 0 2024-02-05T08:00:47.626 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-05T08:00:47.626 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-05T08:00:47.626 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-02-05T08:00:47.626 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-05T08:00:47.626 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-05T08:00:47.626 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.merge_operator: 2024-02-05T08:00:47.626 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compaction_filter: None 2024-02-05T08:00:47.626 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compaction_filter_factory: None 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.sst_partitioner_factory: None 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5617b11039c0) 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout: cache_index_and_filter_blocks: 1 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout: pin_top_level_index_and_filter: 1 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout: index_type: 0 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout: data_block_index_type: 0 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout: index_shortening: 1 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout: checksum: 4 2024-02-05T08:00:47.627 INFO:journalctl@ceph.mon.c.smithi134.stdout: no_block_cache: 0 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: block_cache: 0x5617b1107090 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: block_cache_name: BinnedLRUCache 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: block_cache_options: 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: capacity : 536870912 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: num_shard_bits : 4 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: strict_capacity_limit : 0 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: high_pri_pool_ratio: 0.000 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: block_cache_compressed: (nil) 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: persistent_cache: (nil) 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: block_size: 4096 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: block_size_deviation: 10 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: block_restart_interval: 16 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: index_block_restart_interval: 1 2024-02-05T08:00:47.628 INFO:journalctl@ceph.mon.c.smithi134.stdout: metadata_block_size: 4096 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout: partition_filters: 0 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout: use_delta_encoding: 1 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout: filter_policy: bloomfilter 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout: whole_key_filtering: 1 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout: verify_compression: 0 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout: read_amp_bytes_per_bit: 0 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout: format_version: 5 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout: enable_index_compression: 1 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout: block_align: 0 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout: max_auto_readahead_size: 262144 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout: prepopulate_block_cache: 0 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout: initial_auto_readahead_size: 8192 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout: num_file_reads_for_auto_readahead: 2 2024-02-05T08:00:47.629 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.write_buffer_size: 33554432 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_write_buffer_number: 2 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compression: NoCompression 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.bottommost_compression: Disabled 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.prefix_extractor: nullptr 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.num_levels: 7 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-05T08:00:47.630 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compression_opts.level: 32767 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compression_opts.strategy: 0 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compression_opts.enabled: false 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-05T08:00:47.631 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.target_file_size_base: 67108864 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-05T08:00:47.632 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.arena_block_size: 1048576 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.disable_auto_compactions: 0 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-05T08:00:47.633 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.table_properties_collectors: 2024-02-05T08:00:47.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.inplace_update_support: 0 2024-02-05T08:00:47.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-05T08:00:47.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-05T08:00:47.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-05T08:00:47.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-05T08:00:47.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.bloom_locality: 0 2024-02-05T08:00:47.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.max_successive_merges: 0 2024-02-05T08:00:47.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-05T08:00:47.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.paranoid_file_checks: 0 2024-02-05T08:00:47.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.force_consistency_checks: 1 2024-02-05T08:00:47.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.report_bg_io_stats: 0 2024-02-05T08:00:47.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.ttl: 2592000 2024-02-05T08:00:47.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.enable_blob_files: false 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.min_blob_size: 0 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.blob_file_size: 268435456 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.blob_file_starting_level: 0 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-05T08:00:47.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 73b24843-e99a-4108-8c89-1dbdc27ab20d 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707120047566803, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707120047567200, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707120047, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "73b24843-e99a-4108-8c89-1dbdc27ab20d", "db_session_id": "GYSVEM7ID1BMWQQIGA56", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707120047567279, "job": 1, "event": "recovery_finished"} 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5617b11f8000 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: DB pointer 0x5617b11e2000 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout: ** DB Stats ** 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-05T08:00:47.636 INFO:journalctl@ceph.mon.c.smithi134.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: ** Compaction Stats [default] ** 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: ** Compaction Stats [default] ** 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T08:00:47.637 INFO:journalctl@ceph.mon.c.smithi134.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-05T08:00:47.638 INFO:journalctl@ceph.mon.c.smithi134.stdout: Block cache BinnedLRUCache@0x5617b1107090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout: 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: using public_addrv [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: starting mon.c rank -1 at public addrs [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] at bind addrs [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(???) e0 preinit fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(synchronizing).mds e1 new map 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout: e1 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout: legacy client fscid: -1 2024-02-05T08:00:47.639 INFO:journalctl@ceph.mon.c.smithi134.stdout: 2024-02-05T08:00:47.640 INFO:journalctl@ceph.mon.c.smithi134.stdout: No filesystems configured 2024-02-05T08:00:47.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-05T08:00:47.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-05T08:00:47.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-05T08:00:47.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-05T08:00:47.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-05T08:00:47.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-05T08:00:47.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-05T08:00:47.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T08:00:47.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T08:00:47.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T08:00:47.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:47.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:00:47.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:47.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:47.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/418301971' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-05T08:00:47.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/418301971' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-05T08:00:47.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T08:00:47.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:47.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:47.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:47.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:47.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:47.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: Updating smithi196:/etc/ceph/ceph.conf 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:47.642 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi134:172.21.15.134=a;smithi134:[v2:172.21.15.134:3301,v1:172.21.15.134:6790]=c;smithi196:172.21.15.196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: Saving service mon spec with placement smithi134:172.21.15.134=a;smithi134:[v2:172.21.15.134:3301,v1:172.21.15.134:6790]=c;smithi196:172.21.15.196=b;count:3 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: Deploying daemon mon.b on smithi196 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/2242712204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.a calling monitor election 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:47.643 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:47.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:47.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.b calling monitor election 2024-02-05T08:00:47.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:47.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:47.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:47.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:47.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-05T08:00:47.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: monmap e2: 2 mons at {a=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T08:00:47.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: fsmap 2024-02-05T08:00:47.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T08:00:47.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mgrmap e13: y(active, since 41s) 2024-02-05T08:00:47.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: overall HEALTH_OK 2024-02-05T08:00:47.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:47.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:47.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:47.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:47.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:47.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: Deploying daemon mon.c on smithi134 2024-02-05T08:00:47.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/3975265034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:00:47.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:47.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:47.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 ceph-mon[39278]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-05T08:00:47.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 bash[39240]: 59ded051c19d862b7d30921ee79822829ec82c52091b6c8006f09a7cee7aff32 2024-02-05T08:00:47.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:47 smithi134 systemd[1]: Started Ceph mon.c for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:00:53.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:00:53.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: mon.a calling monitor election 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: mon.b calling monitor election 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: mon.c calling monitor election 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: monmap e3: 3 mons at {a=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: fsmap 2024-02-05T08:00:53.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T08:00:53.030 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: mgrmap e13: y(active, since 49s) 2024-02-05T08:00:53.030 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: overall HEALTH_OK 2024-02-05T08:00:53.030 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:53.030 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:53.030 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:53.030 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:53.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:00:53.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: mon.a calling monitor election 2024-02-05T08:00:53.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: mon.b calling monitor election 2024-02-05T08:00:53.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:53.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: mon.c calling monitor election 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: monmap e3: 3 mons at {a=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: fsmap 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: mgrmap e13: y(active, since 49s) 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: overall HEALTH_OK 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:53.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:53.032 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:53.032 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:52 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:00:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: mon.a calling monitor election 2024-02-05T08:00:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: mon.b calling monitor election 2024-02-05T08:00:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:00:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: mon.c calling monitor election 2024-02-05T08:00:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:53.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:53.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-05T08:00:53.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: monmap e3: 3 mons at {a=[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0],b=[v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0],c=[v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-05T08:00:53.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: fsmap 2024-02-05T08:00:53.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T08:00:53.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: mgrmap e13: y(active, since 49s) 2024-02-05T08:00:53.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: overall HEALTH_OK 2024-02-05T08:00:53.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:53.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:53.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:53.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:52 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:53.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:53 smithi196 ceph-mon[33018]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:53.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:53 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:53.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:53 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:00:53.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:53 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:54.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:53 smithi134 ceph-mon[30086]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:54.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:53 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:54.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:53 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:00:54.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:53 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:53 smithi134 ceph-mon[39278]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:53 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:53 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:00:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:53 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:00:54.422 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-02-05T08:00:54.422 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":3,"fsid":"43e13aa4-c3fc-11ee-95b6-87774f69a715","modified":"2024-02-05T08:00:47.591087Z","created":"2024-02-05T07:59:04.254061Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3300","nonce":0},{"type":"v1","addr":"172.21.15.134:6789","nonce":0}]},"addr":"172.21.15.134:6789/0","public_addr":"172.21.15.134:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:3301","nonce":0},{"type":"v1","addr":"172.21.15.134:6790","nonce":0}]},"addr":"172.21.15.134:6790/0","public_addr":"172.21.15.134:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-02-05T08:00:54.424 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 3 2024-02-05T08:00:55.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:00:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: Updating smithi134:/etc/ceph/ceph.conf 2024-02-05T08:00:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: Updating smithi196:/etc/ceph/ceph.conf 2024-02-05T08:00:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/2350077061' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:00:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: Updating smithi134:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:00:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:00:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: Reconfiguring mon.a (unknown last config time)... 2024-02-05T08:00:55.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:55.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T08:00:55.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:55.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:55 smithi196 ceph-mon[33018]: Reconfiguring daemon mon.a on smithi134 2024-02-05T08:00:55.068 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-05T08:00:55.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph config generate-minimal-conf 2024-02-05T08:00:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:00:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: Updating smithi134:/etc/ceph/ceph.conf 2024-02-05T08:00:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: Updating smithi196:/etc/ceph/ceph.conf 2024-02-05T08:00:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/2350077061' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:00:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: Updating smithi134:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:00:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:00:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:55.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: Reconfiguring mon.a (unknown last config time)... 2024-02-05T08:00:55.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:55.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T08:00:55.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:55.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[39278]: Reconfiguring daemon mon.a on smithi134 2024-02-05T08:00:55.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:55.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:00:55.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: Updating smithi134:/etc/ceph/ceph.conf 2024-02-05T08:00:55.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: Updating smithi196:/etc/ceph/ceph.conf 2024-02-05T08:00:55.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/2350077061' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:00:55.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: Updating smithi134:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:00:55.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:00:55.281 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.281 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.281 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.281 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.281 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:55.281 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.281 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.281 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.281 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.281 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:55.282 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: Reconfiguring mon.a (unknown last config time)... 2024-02-05T08:00:55.282 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:55.282 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T08:00:55.282 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:55.282 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:55 smithi134 ceph-mon[30086]: Reconfiguring daemon mon.a on smithi134 2024-02-05T08:00:55.971 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:00:57.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:57.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:57.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[39278]: Reconfiguring mon.c (monmap changed)... 2024-02-05T08:00:57.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:57.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T08:00:57.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:57.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[39278]: Reconfiguring daemon mon.c on smithi134 2024-02-05T08:00:57.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[39278]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:57.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:57.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:57.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[30086]: Reconfiguring mon.c (monmap changed)... 2024-02-05T08:00:57.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:57.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T08:00:57.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:57.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[30086]: Reconfiguring daemon mon.c on smithi134 2024-02-05T08:00:57.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:57 smithi134 ceph-mon[30086]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:57 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:57 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:57 smithi196 ceph-mon[33018]: Reconfiguring mon.c (monmap changed)... 2024-02-05T08:00:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:57 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:57 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T08:00:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:57 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:57 smithi196 ceph-mon[33018]: Reconfiguring daemon mon.c on smithi134 2024-02-05T08:00:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:57 smithi196 ceph-mon[33018]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:58.992 INFO:teuthology.orchestra.run.smithi134.stdout:# minimal ceph.conf for 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T08:00:58.992 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-02-05T08:00:58.992 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T08:00:58.992 INFO:teuthology.orchestra.run.smithi134.stdout: mon_host = [v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0] [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] [v2:172.21.15.134:3301/0,v1:172.21.15.134:6790/0] 2024-02-05T08:00:59.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:59 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:59.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:59 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:59.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:59 smithi196 ceph-mon[33018]: Reconfiguring mon.b (monmap changed)... 2024-02-05T08:00:59.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:59 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:59.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:59 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T08:00:59.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:59 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:59.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:59 smithi196 ceph-mon[33018]: Reconfiguring daemon mon.b on smithi196 2024-02-05T08:00:59.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:59 smithi196 ceph-mon[33018]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:59.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:00:59 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2870126330' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:59.577 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:59.577 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:59.577 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[30086]: Reconfiguring mon.b (monmap changed)... 2024-02-05T08:00:59.577 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:59.577 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T08:00:59.577 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:59.577 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[30086]: Reconfiguring daemon mon.b on smithi196 2024-02-05T08:00:59.577 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[30086]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:59.577 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2870126330' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:59.577 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:59.577 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:00:59.578 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[39278]: Reconfiguring mon.b (monmap changed)... 2024-02-05T08:00:59.578 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T08:00:59.578 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T08:00:59.578 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:59.578 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[39278]: Reconfiguring daemon mon.b on smithi196 2024-02-05T08:00:59.578 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[39278]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:00:59.578 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:00:59 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2870126330' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:00:59.606 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-05T08:00:59.606 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T08:00:59.606 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.conf 2024-02-05T08:00:59.636 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T08:00:59.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-05T08:00:59.705 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T08:00:59.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.conf 2024-02-05T08:00:59.735 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T08:00:59.736 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-05T08:00:59.803 INFO:tasks.cephadm:Adding mgr.y on smithi134 2024-02-05T08:00:59.803 INFO:tasks.cephadm:Adding mgr.x on smithi196 2024-02-05T08:00:59.803 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch apply mgr '2;smithi134=y;smithi196=x' 2024-02-05T08:01:00.076 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:01:01.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:00 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:01.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:00 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:01.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:00 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:01.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:00 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:01.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:00 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:01.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:00 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:01.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:00 smithi196 ceph-mon[33018]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:01.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:01.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:01.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:01.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:01.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:01.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:01.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[30086]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:01.280 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:01.280 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:01.280 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:01.280 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:01.280 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:01.280 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:01.281 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:00 smithi134 ceph-mon[39278]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:01.695 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mgr update... 2024-02-05T08:01:02.210 DEBUG:teuthology.orchestra.run.smithi196:mgr.x> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mgr.x.service 2024-02-05T08:01:02.212 INFO:tasks.cephadm:Deploying OSDs... 2024-02-05T08:01:02.212 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T08:01:02.212 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-02-05T08:01:02.232 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-05T08:01:02.232 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-02-05T08:01:02.289 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-05T08:01:02.289 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T08:01:02.289 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-02-05T08:01:02.289 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T08:01:02.289 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T08:01:02.290 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-02-05 08:00:46.999524001 +0000 2024-02-05T08:01:02.290 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-02-05 07:54:44.715756993 +0000 2024-02-05T08:01:02.290 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-02-05 07:54:44.715756993 +0000 2024-02-05T08:01:02.290 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-02-05 07:54:44.715756993 +0000 2024-02-05T08:01:02.290 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-05T08:01:02.359 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-02-05T08:01:02.359 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-02-05T08:01:02.359 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000227664 s, 2.2 MB/s 2024-02-05T08:01:02.360 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-05T08:01:02.417 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-02-05T08:01:02.475 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-05T08:01:02.475 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T08:01:02.475 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-05T08:01:02.475 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T08:01:02.475 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T08:01:02.475 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-02-05 08:00:46.999524001 +0000 2024-02-05T08:01:02.475 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-02-05 07:54:45.043749071 +0000 2024-02-05T08:01:02.475 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-02-05 07:54:45.043749071 +0000 2024-02-05T08:01:02.475 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-02-05 07:54:45.043749071 +0000 2024-02-05T08:01:02.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-05T08:01:02.543 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-02-05T08:01:02.543 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-02-05T08:01:02.543 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000214238 s, 2.4 MB/s 2024-02-05T08:01:02.545 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-05T08:01:02.603 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-02-05T08:01:02.659 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-05T08:01:02.659 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T08:01:02.659 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-05T08:01:02.659 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T08:01:02.659 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T08:01:02.660 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-02-05 08:00:47.000523977 +0000 2024-02-05T08:01:02.660 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-02-05 07:54:45.374741077 +0000 2024-02-05T08:01:02.660 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-02-05 07:54:45.374741077 +0000 2024-02-05T08:01:02.660 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-02-05 07:54:45.374741077 +0000 2024-02-05T08:01:02.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-05T08:01:02.726 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-02-05T08:01:02.727 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-02-05T08:01:02.727 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000227727 s, 2.2 MB/s 2024-02-05T08:01:02.728 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-05T08:01:02.784 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-02-05T08:01:02.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:02 smithi196 ceph-mon[33018]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi134=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:01:02.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:02 smithi196 ceph-mon[33018]: Saving service mgr spec with placement smithi134=y;smithi196=x;count:2 2024-02-05T08:01:02.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:02.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:02.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:02.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:02.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:02.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T08:01:02.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-05T08:01:02.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T08:01:02.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:02.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:02 smithi196 ceph-mon[33018]: Deploying daemon mgr.x on smithi196 2024-02-05T08:01:02.839 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-05T08:01:02.839 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T08:01:02.839 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-02-05T08:01:02.839 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T08:01:02.840 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T08:01:02.840 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-02-05 08:00:47.000523977 +0000 2024-02-05T08:01:02.840 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-02-05 07:54:45.736732334 +0000 2024-02-05T08:01:02.840 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-02-05 07:54:45.736732334 +0000 2024-02-05T08:01:02.840 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-02-05 07:54:45.736732334 +0000 2024-02-05T08:01:02.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-05T08:01:02.905 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-02-05T08:01:02.906 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-02-05T08:01:02.906 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000184365 s, 2.8 MB/s 2024-02-05T08:01:02.907 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-05T08:01:02.963 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T08:01:02.963 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-02-05T08:01:02.979 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-05T08:01:02.979 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-02-05T08:01:03.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[30086]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi134=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:01:03.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[30086]: Saving service mgr spec with placement smithi134=y;smithi196=x;count:2 2024-02-05T08:01:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T08:01:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-05T08:01:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T08:01:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:03.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[30086]: Deploying daemon mgr.x on smithi196 2024-02-05T08:01:03.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[39278]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi134=y;smithi196=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:01:03.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[39278]: Saving service mgr spec with placement smithi134=y;smithi196=x;count:2 2024-02-05T08:01:03.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:03.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:03.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:03.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:03.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:03.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T08:01:03.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-05T08:01:03.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T08:01:03.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:03.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:02 smithi134 ceph-mon[39278]: Deploying daemon mgr.x on smithi196 2024-02-05T08:01:03.034 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-05T08:01:03.035 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T08:01:03.035 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-05T08:01:03.035 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T08:01:03.035 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T08:01:03.035 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-02-05 08:00:38.621238986 +0000 2024-02-05T08:01:03.035 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-02-05 07:54:44.718756169 +0000 2024-02-05T08:01:03.035 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-02-05 07:54:44.718756169 +0000 2024-02-05T08:01:03.035 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-02-05 07:54:44.718756169 +0000 2024-02-05T08:01:03.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-05T08:01:03.100 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-02-05T08:01:03.100 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-02-05T08:01:03.101 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.00033419 s, 1.5 MB/s 2024-02-05T08:01:03.102 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-05T08:01:03.158 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-02-05T08:01:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-05T08:01:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T08:01:03.213 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-02-05T08:01:03.213 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T08:01:03.213 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T08:01:03.213 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-02-05 08:00:38.621238986 +0000 2024-02-05T08:01:03.213 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-02-05 07:54:45.050748722 +0000 2024-02-05T08:01:03.213 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-02-05 07:54:45.050748722 +0000 2024-02-05T08:01:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-02-05 07:54:45.050748722 +0000 2024-02-05T08:01:03.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-05T08:01:03.278 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-02-05T08:01:03.279 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-02-05T08:01:03.279 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000242422 s, 2.1 MB/s 2024-02-05T08:01:03.280 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-05T08:01:03.336 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-02-05T08:01:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-05T08:01:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T08:01:03.391 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-05T08:01:03.391 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T08:01:03.391 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T08:01:03.391 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-02-05 08:00:38.621238986 +0000 2024-02-05T08:01:03.391 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-02-05 07:54:45.391741072 +0000 2024-02-05T08:01:03.392 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-02-05 07:54:45.391741072 +0000 2024-02-05T08:01:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-02-05 07:54:45.391741072 +0000 2024-02-05T08:01:03.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-05T08:01:03.622 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-02-05T08:01:03.622 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-02-05T08:01:03.622 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000198497 s, 2.6 MB/s 2024-02-05T08:01:03.623 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-05T08:01:03.638 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-02-05T08:01:03.694 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-05T08:01:03.694 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T08:01:03.694 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-02-05T08:01:03.694 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T08:01:03.694 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T08:01:03.694 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-02-05 08:01:03.576681828 +0000 2024-02-05T08:01:03.695 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-02-05 07:54:45.739733265 +0000 2024-02-05T08:01:03.695 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-02-05 07:54:45.739733265 +0000 2024-02-05T08:01:03.695 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-02-05 07:54:45.739733265 +0000 2024-02-05T08:01:03.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-05T08:01:03.796 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-02-05T08:01:03.796 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-02-05T08:01:03.796 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000194846 s, 2.6 MB/s 2024-02-05T08:01:03.798 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-05T08:01:03.813 INFO:tasks.cephadm:Deploying osd.0 on smithi134 with /dev/vg_nvme/lv_4... 2024-02-05T08:01:03.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-05T08:01:04.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:03 smithi196 ceph-mon[33018]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:04.026 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:03 smithi196 systemd[1]: Starting Ceph mgr.x for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:01:04.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:03 smithi134 ceph-mon[30086]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:04.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:03 smithi134 ceph-mon[39278]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:04.060 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:01:04.311 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:04 smithi196 podman[35121]: 2024-02-05 08:01:03.929125629 +0000 UTC m=+0.018614771 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:01:04.311 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:04 smithi196 podman[35121]: 2024-02-05 08:01:04.033107836 +0000 UTC m=+0.122596973 container create 40e009f8fe59644dae71723ffcf98b5c22513d7b91d6622ec819eb88c46af66a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-05T08:01:04.311 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:04 smithi196 podman[35121]: 2024-02-05 08:01:04.243718213 +0000 UTC m=+0.333207350 container init 40e009f8fe59644dae71723ffcf98b5c22513d7b91d6622ec819eb88c46af66a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131) 2024-02-05T08:01:04.311 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:04 smithi196 podman[35121]: 2024-02-05 08:01:04.248734514 +0000 UTC m=+0.338223651 container start 40e009f8fe59644dae71723ffcf98b5c22513d7b91d6622ec819eb88c46af66a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True) 2024-02-05T08:01:04.311 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:04 smithi196 ceph-mgr[35159]: -- 172.21.15.196:0/2366104111 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c5c92a1520 con 0x55c5c9277000 2024-02-05T08:01:04.311 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:04 smithi196 bash[35121]: 40e009f8fe59644dae71723ffcf98b5c22513d7b91d6622ec819eb88c46af66a 2024-02-05T08:01:04.564 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:04 smithi196 systemd[1]: Started Ceph mgr.x for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:01:05.315 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:04.815+0000 7f15ea511200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T08:01:05.315 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:04.912+0000 7f15ea511200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T08:01:05.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:05 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:05.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:05 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:05.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:05 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:05.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:05 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:05.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:05 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:05.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:05 smithi134 ceph-mon[30086]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:05.446 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:05 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:05.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:05 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:05.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:05 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:05.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:05 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:05.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:05 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:05.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:05 smithi134 ceph-mon[39278]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:05.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:05 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:05.608 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:05 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:05.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:05 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:05.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:05 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:05.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:05 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:05.609 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:05 smithi196 ceph-mon[33018]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:05.946 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:05 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:05.606+0000 7f15ea511200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T08:01:05.946 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:05 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:05.699+0000 7f15ea511200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T08:01:05.946 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:05 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:05.849+0000 7f15ea511200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T08:01:05.946 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:05 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:05.944+0000 7f15ea511200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T08:01:06.018 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-05T08:01:06.019 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-05T08:01:06.019 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-02-05T08:01:06.019 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-02-05T08:01:06.019 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131651 s, 796 MB/s 2024-02-05T08:01:06.019 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-02-05T08:01:06.293 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:06.030+0000 7f15ea511200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T08:01:06.293 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:06.291+0000 7f15ea511200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T08:01:06.564 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:06.401+0000 7f15ea511200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T08:01:06.565 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:06.486+0000 7f15ea511200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T08:01:06.606 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi134:vg_nvme/lv_4 2024-02-05T08:01:06.858 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:06.580+0000 7f15ea511200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T08:01:06.858 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:06.756+0000 7f15ea511200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T08:01:06.895 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:06.895 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:06.895 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:06.895 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:06.895 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:06.895 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T08:01:06.895 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T08:01:06.895 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:06.896 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:06.896 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:06.896 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:06.896 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:06.896 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:06.896 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T08:01:06.896 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T08:01:06.896 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:07.298 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:01:07.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:07.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:07.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:07.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:07.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:07.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T08:01:07.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T08:01:07.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:07.981 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:07 smithi134 ceph-mon[30086]: Reconfiguring mgr.y (unknown last config time)... 2024-02-05T08:01:07.981 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:07 smithi134 ceph-mon[30086]: Reconfiguring daemon mgr.y on smithi134 2024-02-05T08:01:07.981 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:07 smithi134 ceph-mon[30086]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:07.981 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:07 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:07.982 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:07 smithi134 ceph-mon[39278]: Reconfiguring mgr.y (unknown last config time)... 2024-02-05T08:01:07.982 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:07 smithi134 ceph-mon[39278]: Reconfiguring daemon mgr.y on smithi134 2024-02-05T08:01:07.982 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:07 smithi134 ceph-mon[39278]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:07.982 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:07 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:08.065 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:07 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:07.719+0000 7f15ea511200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T08:01:08.065 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:07 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:07.847+0000 7f15ea511200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T08:01:08.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:07 smithi196 ceph-mon[33018]: Reconfiguring mgr.y (unknown last config time)... 2024-02-05T08:01:08.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:07 smithi196 ceph-mon[33018]: Reconfiguring daemon mgr.y on smithi134 2024-02-05T08:01:08.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:07 smithi196 ceph-mon[33018]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:08.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:07 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:08.464 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:08 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:08.180+0000 7f15ea511200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T08:01:08.464 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:08 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:08.269+0000 7f15ea511200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T08:01:08.464 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:08 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:08.462+0000 7f15ea511200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T08:01:08.815 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:08 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:08.644+0000 7f15ea511200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T08:01:09.187 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:08 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:08.924+0000 7f15ea511200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T08:01:09.187 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:09 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:09.185+0000 7f15ea511200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T08:01:09.565 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:09 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:09.419+0000 7f15ea511200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T08:01:10.216 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:09 smithi196 ceph-mon[33018]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:10.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:09 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:01:10.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:09 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:01:10.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:09 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:10.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:09 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:10.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:09 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:10.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:09 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:10.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[30086]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:10.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:01:10.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:01:10.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:10.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:10.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:10.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[39278]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:01:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:01:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:09 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:10.565 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:10 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:10.214+0000 7f15ea511200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T08:01:10.953 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:10 smithi134 ceph-mon[30086]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:01:10.953 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:10 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:10.953 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:10 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:10.953 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:10 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:10.954 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:10 smithi134 ceph-mon[39278]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:01:10.954 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:10 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:10.954 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:10 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:10.954 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:10 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:11.065 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:10 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:10.781+0000 7f15ea511200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T08:01:11.065 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:10 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:10.879+0000 7f15ea511200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T08:01:11.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:10 smithi196 ceph-mon[33018]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:01:11.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:10 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:11.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:10 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:11.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:10 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:11.565 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:01:11 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:01:11.094+0000 7f15ea511200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T08:01:12.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[30086]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:12.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[30086]: Standby manager daemon x started 2024-02-05T08:01:12.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[30086]: from='mgr.? 172.21.15.196:0/1777019356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T08:01:12.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[30086]: from='mgr.? 172.21.15.196:0/1777019356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T08:01:12.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[30086]: from='mgr.? 172.21.15.196:0/1777019356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T08:01:12.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[30086]: from='mgr.? 172.21.15.196:0/1777019356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T08:01:12.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3126898017' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d90a5ea0-9a19-452e-9564-ca7b7a7c45e4"}]: dispatch 2024-02-05T08:01:12.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3126898017' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d90a5ea0-9a19-452e-9564-ca7b7a7c45e4"}]': finished 2024-02-05T08:01:12.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[30086]: osdmap e5: 1 total, 0 up, 1 in 2024-02-05T08:01:12.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:12.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[39278]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:12.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[39278]: Standby manager daemon x started 2024-02-05T08:01:12.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[39278]: from='mgr.? 172.21.15.196:0/1777019356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T08:01:12.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[39278]: from='mgr.? 172.21.15.196:0/1777019356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T08:01:12.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[39278]: from='mgr.? 172.21.15.196:0/1777019356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T08:01:12.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[39278]: from='mgr.? 172.21.15.196:0/1777019356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T08:01:12.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3126898017' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d90a5ea0-9a19-452e-9564-ca7b7a7c45e4"}]: dispatch 2024-02-05T08:01:12.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3126898017' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d90a5ea0-9a19-452e-9564-ca7b7a7c45e4"}]': finished 2024-02-05T08:01:12.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[39278]: osdmap e5: 1 total, 0 up, 1 in 2024-02-05T08:01:12.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:11 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:12.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:11 smithi196 ceph-mon[33018]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:12.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:11 smithi196 ceph-mon[33018]: Standby manager daemon x started 2024-02-05T08:01:12.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:11 smithi196 ceph-mon[33018]: from='mgr.? 172.21.15.196:0/1777019356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T08:01:12.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:11 smithi196 ceph-mon[33018]: from='mgr.? 172.21.15.196:0/1777019356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T08:01:12.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:11 smithi196 ceph-mon[33018]: from='mgr.? 172.21.15.196:0/1777019356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T08:01:12.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:11 smithi196 ceph-mon[33018]: from='mgr.? 172.21.15.196:0/1777019356' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T08:01:12.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:11 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3126898017' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d90a5ea0-9a19-452e-9564-ca7b7a7c45e4"}]: dispatch 2024-02-05T08:01:12.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:11 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3126898017' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d90a5ea0-9a19-452e-9564-ca7b7a7c45e4"}]': finished 2024-02-05T08:01:12.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:11 smithi196 ceph-mon[33018]: osdmap e5: 1 total, 0 up, 1 in 2024-02-05T08:01:12.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:11 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:13.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:12 smithi134 ceph-mon[30086]: mgrmap e14: y(active, since 69s), standbys: x 2024-02-05T08:01:13.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:12 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T08:01:13.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:12 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/953942623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:01:13.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:12 smithi134 ceph-mon[30086]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:13.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:12 smithi134 ceph-mon[39278]: mgrmap e14: y(active, since 69s), standbys: x 2024-02-05T08:01:13.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:12 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T08:01:13.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:12 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/953942623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:01:13.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:12 smithi134 ceph-mon[39278]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:13.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:12 smithi196 ceph-mon[33018]: mgrmap e14: y(active, since 69s), standbys: x 2024-02-05T08:01:13.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:12 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T08:01:13.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:12 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/953942623' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:01:13.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:12 smithi196 ceph-mon[33018]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:15.761 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:15 smithi134 ceph-mon[30086]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:15.761 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:15 smithi134 ceph-mon[39278]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:15 smithi196 ceph-mon[33018]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:17.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:17 smithi134 ceph-mon[30086]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:17.931 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:17 smithi134 ceph-mon[39278]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:18.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:17 smithi196 ceph-mon[33018]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:18.815 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:18 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-05T08:01:18.815 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:18 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:18.815 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:18 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-05T08:01:18.815 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:18 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:19.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:18 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-05T08:01:19.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:18 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:19.897 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:19 smithi134 ceph-mon[30086]: Deploying daemon osd.0 on smithi134 2024-02-05T08:01:19.898 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:19 smithi134 ceph-mon[30086]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:19.898 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:19 smithi134 ceph-mon[39278]: Deploying daemon osd.0 on smithi134 2024-02-05T08:01:19.898 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:19 smithi134 ceph-mon[39278]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:20.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:19 smithi196 ceph-mon[33018]: Deploying daemon osd.0 on smithi134 2024-02-05T08:01:20.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:19 smithi196 ceph-mon[33018]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:22.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:21 smithi134 ceph-mon[30086]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:22.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:21 smithi134 ceph-mon[39278]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:21 smithi196 ceph-mon[33018]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:23.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:23 smithi196 ceph-mon[33018]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:23.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:23 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:23.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:23 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:23.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:23 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:24.015 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:23 smithi134 ceph-mon[30086]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:24.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:23 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:24.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:23 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:24.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:23 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:24.016 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:23 smithi134 ceph-mon[39278]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:24.016 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:23 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:24.016 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:23 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:24.016 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:23 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:25.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:25 smithi134 ceph-mon[30086]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:25.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:25 smithi134 ceph-mon[39278]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:25 smithi196 ceph-mon[33018]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:26.517 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 0 on host 'smithi134' 2024-02-05T08:01:26.684 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:26 smithi134 ceph-mon[30086]: from='osd.0 [v2:172.21.15.134:6802/2540271081,v1:172.21.15.134:6803/2540271081]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-05T08:01:26.684 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:26 smithi134 ceph-mon[30086]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-05T08:01:26.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:26 smithi134 ceph-mon[39278]: from='osd.0 [v2:172.21.15.134:6802/2540271081,v1:172.21.15.134:6803/2540271081]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-05T08:01:26.684 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:26 smithi134 ceph-mon[39278]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-05T08:01:26.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:26 smithi196 ceph-mon[33018]: from='osd.0 [v2:172.21.15.134:6802/2540271081,v1:172.21.15.134:6803/2540271081]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-05T08:01:26.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:26 smithi196 ceph-mon[33018]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-05T08:01:27.070 DEBUG:teuthology.orchestra.run.smithi134:osd.0> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.0.service 2024-02-05T08:01:27.072 INFO:tasks.cephadm:Deploying osd.1 on smithi134 with /dev/vg_nvme/lv_3... 2024-02-05T08:01:27.072 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-05T08:01:27.359 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:01:27.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:27.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[39278]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-05T08:01:27.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[39278]: osdmap e6: 1 total, 0 up, 1 in 2024-02-05T08:01:27.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[39278]: from='osd.0 [v2:172.21.15.134:6802/2540271081,v1:172.21.15.134:6803/2540271081]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:01:27.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:27.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[39278]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:01:27.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:27.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[39278]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:27.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:27.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:27.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:27.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[30086]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-05T08:01:27.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[30086]: osdmap e6: 1 total, 0 up, 1 in 2024-02-05T08:01:27.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[30086]: from='osd.0 [v2:172.21.15.134:6802/2540271081,v1:172.21.15.134:6803/2540271081]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:01:27.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:27.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[30086]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:01:27.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:27.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[30086]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:27.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:27.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:27.769 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0[43745]: 2024-02-05T08:01:27.473+0000 7f74901e0700 -1 osd.0 0 waiting for initial osdmap 2024-02-05T08:01:27.769 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:01:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0[43745]: 2024-02-05T08:01:27.476+0000 7f748bfda700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T08:01:27.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:27 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:27.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:27 smithi196 ceph-mon[33018]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-05T08:01:27.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:27 smithi196 ceph-mon[33018]: osdmap e6: 1 total, 0 up, 1 in 2024-02-05T08:01:27.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:27 smithi196 ceph-mon[33018]: from='osd.0 [v2:172.21.15.134:6802/2540271081,v1:172.21.15.134:6803/2540271081]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:01:27.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:27 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:27.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:27 smithi196 ceph-mon[33018]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:01:27.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:27 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:27.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:27 smithi196 ceph-mon[33018]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:27.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:27 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:27.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:27 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:28.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:28 smithi134 ceph-mon[30086]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-02-05T08:01:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:28 smithi134 ceph-mon[30086]: osdmap e7: 1 total, 0 up, 1 in 2024-02-05T08:01:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:28 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:28 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:28 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:28.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:28 smithi134 ceph-mon[39278]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-02-05T08:01:28.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:28 smithi134 ceph-mon[39278]: osdmap e7: 1 total, 0 up, 1 in 2024-02-05T08:01:28.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:28 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:28.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:28 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:28.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:28 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:28.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:28 smithi196 ceph-mon[33018]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-02-05T08:01:28.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:28 smithi196 ceph-mon[33018]: osdmap e7: 1 total, 0 up, 1 in 2024-02-05T08:01:28.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:28 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:28.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:28 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:28.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:28 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:29.669 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:29 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:01:29.669 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:29 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:01:29.669 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:29 smithi134 ceph-mon[30086]: osd.0 [v2:172.21.15.134:6802/2540271081,v1:172.21.15.134:6803/2540271081] boot 2024-02-05T08:01:29.669 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:29 smithi134 ceph-mon[30086]: osdmap e8: 1 total, 1 up, 1 in 2024-02-05T08:01:29.669 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:29.670 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:29 smithi134 ceph-mon[30086]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:29.670 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:29 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:01:29.670 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:29 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:01:29.670 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:29 smithi134 ceph-mon[39278]: osd.0 [v2:172.21.15.134:6802/2540271081,v1:172.21.15.134:6803/2540271081] boot 2024-02-05T08:01:29.670 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:29 smithi134 ceph-mon[39278]: osdmap e8: 1 total, 1 up, 1 in 2024-02-05T08:01:29.670 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:29.670 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:29 smithi134 ceph-mon[39278]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:29.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:29 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:01:29.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:29 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:01:29.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:29 smithi196 ceph-mon[33018]: osd.0 [v2:172.21.15.134:6802/2540271081,v1:172.21.15.134:6803/2540271081] boot 2024-02-05T08:01:29.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:29 smithi196 ceph-mon[33018]: osdmap e8: 1 total, 1 up, 1 in 2024-02-05T08:01:29.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:01:29.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:29 smithi196 ceph-mon[33018]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T08:01:30.493 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-05T08:01:30.493 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-05T08:01:30.493 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-02-05T08:01:30.493 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-02-05T08:01:30.493 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124291 s, 844 MB/s 2024-02-05T08:01:30.493 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-02-05T08:01:30.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:30 smithi134 ceph-mon[30086]: osdmap e9: 1 total, 1 up, 1 in 2024-02-05T08:01:30.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:30 smithi134 ceph-mon[39278]: osdmap e9: 1 total, 1 up, 1 in 2024-02-05T08:01:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:30 smithi196 ceph-mon[33018]: osdmap e9: 1 total, 1 up, 1 in 2024-02-05T08:01:31.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi134:vg_nvme/lv_3 2024-02-05T08:01:31.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:31 smithi134 ceph-mon[30086]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:31.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:31 smithi134 ceph-mon[39278]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:31.788 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:01:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:31 smithi196 ceph-mon[33018]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:32.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[30086]: Detected new or changed devices on smithi134 2024-02-05T08:01:32.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:32.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:32.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:01:32.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[30086]: Adjusting osd_memory_target on smithi134 to 16064M 2024-02-05T08:01:32.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:32.829 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:32.829 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:32.829 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:32.829 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:32.829 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:32.829 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:32.829 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:32.829 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[39278]: Detected new or changed devices on smithi134 2024-02-05T08:01:32.829 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:32.829 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:32.829 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:01:32.829 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[39278]: Adjusting osd_memory_target on smithi134 to 16064M 2024-02-05T08:01:32.830 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:32.830 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:32.830 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:32.830 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:32.830 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:32.830 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:32.830 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:32.830 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:32 smithi196 ceph-mon[33018]: Detected new or changed devices on smithi134 2024-02-05T08:01:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:01:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:32 smithi196 ceph-mon[33018]: Adjusting osd_memory_target on smithi134 to 16064M 2024-02-05T08:01:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:33.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:33.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:33.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:33.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:33.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:33.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:33 smithi134 ceph-mon[30086]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:33 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:01:34.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:33 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:01:34.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:33 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:34.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:33 smithi134 ceph-mon[39278]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:34.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:33 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:01:34.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:33 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:01:34.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:33 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:33 smithi196 ceph-mon[33018]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:34.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:33 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:01:34.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:33 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:01:34.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:33 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:34.871 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:34 smithi134 ceph-mon[30086]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:01:34.871 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:34 smithi134 ceph-mon[39278]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:01:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:34 smithi196 ceph-mon[33018]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:01:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:35 smithi134 ceph-mon[30086]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:35 smithi134 ceph-mon[39278]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:35 smithi196 ceph-mon[33018]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1093461418' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4"}]: dispatch 2024-02-05T08:01:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1093461418' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4"}]': finished 2024-02-05T08:01:37.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:36 smithi134 ceph-mon[30086]: osdmap e10: 2 total, 1 up, 2 in 2024-02-05T08:01:37.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:36 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1093461418' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4"}]: dispatch 2024-02-05T08:01:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1093461418' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4"}]': finished 2024-02-05T08:01:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:36 smithi134 ceph-mon[39278]: osdmap e10: 2 total, 1 up, 2 in 2024-02-05T08:01:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:36 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1093461418' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4"}]: dispatch 2024-02-05T08:01:37.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1093461418' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4"}]': finished 2024-02-05T08:01:37.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:36 smithi196 ceph-mon[33018]: osdmap e10: 2 total, 1 up, 2 in 2024-02-05T08:01:37.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:36 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:37 smithi196 ceph-mon[33018]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:37 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2594622469' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:01:38.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:37 smithi134 ceph-mon[30086]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:38.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:37 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2594622469' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:01:38.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:37 smithi134 ceph-mon[39278]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:38.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:37 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2594622469' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:01:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:39 smithi134 ceph-mon[30086]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:40.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:39 smithi134 ceph-mon[39278]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:40.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:39 smithi196 ceph-mon[33018]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:41.978 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:41 smithi134 ceph-mon[30086]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:41.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:41 smithi134 ceph-mon[39278]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:42.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:41 smithi196 ceph-mon[33018]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:43.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:42 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-05T08:01:43.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:42 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:43.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:42 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-05T08:01:43.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:42 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:43.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:42 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-05T08:01:43.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:42 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:44.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:43 smithi134 ceph-mon[30086]: Deploying daemon osd.1 on smithi134 2024-02-05T08:01:44.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:43 smithi134 ceph-mon[30086]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:44.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:43 smithi134 ceph-mon[39278]: Deploying daemon osd.1 on smithi134 2024-02-05T08:01:44.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:43 smithi134 ceph-mon[39278]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:43 smithi196 ceph-mon[33018]: Deploying daemon osd.1 on smithi134 2024-02-05T08:01:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:43 smithi196 ceph-mon[33018]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:45.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:45 smithi134 ceph-mon[30086]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:45.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:45 smithi134 ceph-mon[39278]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:45.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:45 smithi196 ceph-mon[33018]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:47.872 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:47 smithi134 ceph-mon[30086]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:47.872 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:47 smithi134 ceph-mon[39278]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:47 smithi196 ceph-mon[33018]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:48 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:49.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:48 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:49.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:48 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:48 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:48 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:48 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:49.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:48 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:49.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:48 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:49.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:48 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:50.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:49 smithi134 ceph-mon[30086]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:50.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:49 smithi134 ceph-mon[39278]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:49 smithi196 ceph-mon[33018]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:51.434 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 1 on host 'smithi134' 2024-02-05T08:01:51.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:51.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:51.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:51.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:51.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:51.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[30086]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:51.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[30086]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T08:01:51.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:51.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:51.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:51.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:51.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:51.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:51.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:51.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:51.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[39278]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:51.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[39278]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T08:01:51.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:51.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:51.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:51 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:51 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:51 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:51 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:51 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:51 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:51 smithi196 ceph-mon[33018]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:51 smithi196 ceph-mon[33018]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T08:01:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:51 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:51 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:51 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:01:52.116 DEBUG:teuthology.orchestra.run.smithi134:osd.1> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.1.service 2024-02-05T08:01:52.119 INFO:tasks.cephadm:Deploying osd.2 on smithi134 with /dev/vg_nvme/lv_2... 2024-02-05T08:01:52.119 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-05T08:01:52.893 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:01:52.995 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:52 smithi134 ceph-mon[30086]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-05T08:01:52.995 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:52 smithi134 ceph-mon[30086]: osdmap e11: 2 total, 1 up, 2 in 2024-02-05T08:01:52.995 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:52 smithi134 ceph-mon[30086]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:01:52.995 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:52 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:52.996 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:52 smithi134 ceph-mon[39278]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-05T08:01:52.996 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:52 smithi134 ceph-mon[39278]: osdmap e11: 2 total, 1 up, 2 in 2024-02-05T08:01:52.996 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:52 smithi134 ceph-mon[39278]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:01:52.996 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:52 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:52 smithi196 ceph-mon[33018]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-05T08:01:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:52 smithi196 ceph-mon[33018]: osdmap e11: 2 total, 1 up, 2 in 2024-02-05T08:01:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:52 smithi196 ceph-mon[33018]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:01:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:52 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:53.266 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1[49860]: 2024-02-05T08:01:53.033+0000 7f611bb83700 -1 osd.1 0 waiting for initial osdmap 2024-02-05T08:01:53.266 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1[49860]: 2024-02-05T08:01:53.036+0000 7f6113975700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T08:01:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[30086]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-02-05T08:01:53.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[30086]: osdmap e12: 2 total, 1 up, 2 in 2024-02-05T08:01:53.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:53.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:53.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[30086]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:53.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[30086]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' 2024-02-05T08:01:53.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[39278]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-02-05T08:01:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[39278]: osdmap e12: 2 total, 1 up, 2 in 2024-02-05T08:01:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[39278]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[39278]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' 2024-02-05T08:01:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:53 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:54.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:53 smithi196 ceph-mon[33018]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-02-05T08:01:54.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:53 smithi196 ceph-mon[33018]: osdmap e12: 2 total, 1 up, 2 in 2024-02-05T08:01:54.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:53 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:54.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:53 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:54.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:53 smithi196 ceph-mon[33018]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-05T08:01:54.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:53 smithi196 ceph-mon[33018]: from='osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960]' entity='osd.1' 2024-02-05T08:01:54.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:53 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:55.017 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:54 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:01:55.017 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:54 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:01:55.018 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:54 smithi134 ceph-mon[30086]: osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960] boot 2024-02-05T08:01:55.018 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:54 smithi134 ceph-mon[30086]: osdmap e13: 2 total, 2 up, 2 in 2024-02-05T08:01:55.018 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:54 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:55.018 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:54 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:01:55.018 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:54 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:01:55.018 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:54 smithi134 ceph-mon[39278]: osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960] boot 2024-02-05T08:01:55.018 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:54 smithi134 ceph-mon[39278]: osdmap e13: 2 total, 2 up, 2 in 2024-02-05T08:01:55.018 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:54 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:54 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:01:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:54 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:01:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:54 smithi196 ceph-mon[33018]: osd.1 [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960] boot 2024-02-05T08:01:55.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:54 smithi196 ceph-mon[33018]: osdmap e13: 2 total, 2 up, 2 in 2024-02-05T08:01:55.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:54 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:01:55.388 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-05T08:01:55.388 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-05T08:01:55.388 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-02-05T08:01:55.388 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-02-05T08:01:55.389 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125246 s, 837 MB/s 2024-02-05T08:01:55.389 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-02-05T08:01:55.978 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:55 smithi134 ceph-mon[30086]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:01:55.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:55 smithi134 ceph-mon[39278]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:01:56.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi134:vg_nvme/lv_2 2024-02-05T08:01:56.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:55 smithi196 ceph-mon[33018]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:01:56.593 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:01:57.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[39278]: osdmap e14: 2 total, 2 up, 2 in 2024-02-05T08:01:57.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:57.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:57.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:01:57.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:01:57.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:57.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:57.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:57.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:57.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[30086]: osdmap e14: 2 total, 2 up, 2 in 2024-02-05T08:01:57.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:57.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:57.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:01:57.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:01:57.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:57.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:57.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:57.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:56 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:57.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:56 smithi196 ceph-mon[33018]: osdmap e14: 2 total, 2 up, 2 in 2024-02-05T08:01:57.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:56 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:57.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:56 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:57.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:56 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:01:57.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:56 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:01:57.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:56 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:57.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:56 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:57.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:56 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:01:57.131 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:56 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:01:57.964 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:57 smithi134 ceph-mon[30086]: Detected new or changed devices on smithi134 2024-02-05T08:01:57.965 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:57 smithi134 ceph-mon[30086]: Adjusting osd_memory_target on smithi134 to 8032M 2024-02-05T08:01:57.965 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:57 smithi134 ceph-mon[30086]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:01:57.965 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:57 smithi134 ceph-mon[39278]: Detected new or changed devices on smithi134 2024-02-05T08:01:57.965 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:57 smithi134 ceph-mon[39278]: Adjusting osd_memory_target on smithi134 to 8032M 2024-02-05T08:01:57.965 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:57 smithi134 ceph-mon[39278]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:01:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:57 smithi196 ceph-mon[33018]: Detected new or changed devices on smithi134 2024-02-05T08:01:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:57 smithi196 ceph-mon[33018]: Adjusting osd_memory_target on smithi134 to 8032M 2024-02-05T08:01:58.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:57 smithi196 ceph-mon[33018]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:01:58.985 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:58 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:01:58.986 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:58 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:01:58.986 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:58 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:58.986 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:58 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:01:58.986 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:58 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:01:58.986 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:58 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:01:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:58 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:01:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:58 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:01:59.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:58 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:00.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:59 smithi134 ceph-mon[30086]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:02:00.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:01:59 smithi134 ceph-mon[30086]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:00.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:59 smithi134 ceph-mon[39278]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:02:00.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:01:59 smithi134 ceph-mon[39278]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:00.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:59 smithi196 ceph-mon[33018]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:02:00.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:01:59 smithi196 ceph-mon[33018]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:02.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:01 smithi196 ceph-mon[33018]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:02.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:01 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3892164281' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708"}]: dispatch 2024-02-05T08:02:02.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:01 smithi196 ceph-mon[33018]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708"}]: dispatch 2024-02-05T08:02:02.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:01 smithi196 ceph-mon[33018]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708"}]': finished 2024-02-05T08:02:02.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:01 smithi196 ceph-mon[33018]: osdmap e15: 3 total, 2 up, 3 in 2024-02-05T08:02:02.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:01 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:02.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:01 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1926997494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:02:02.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[39278]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3892164281' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708"}]: dispatch 2024-02-05T08:02:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[39278]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708"}]: dispatch 2024-02-05T08:02:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[39278]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708"}]': finished 2024-02-05T08:02:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[39278]: osdmap e15: 3 total, 2 up, 3 in 2024-02-05T08:02:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1926997494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:02:02.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[30086]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:02.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3892164281' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708"}]: dispatch 2024-02-05T08:02:02.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[30086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708"}]: dispatch 2024-02-05T08:02:02.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[30086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708"}]': finished 2024-02-05T08:02:02.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[30086]: osdmap e15: 3 total, 2 up, 3 in 2024-02-05T08:02:02.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:02.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:01 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1926997494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:02:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:03 smithi196 ceph-mon[33018]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:03 smithi134 ceph-mon[30086]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:03 smithi134 ceph-mon[39278]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:05.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:05 smithi134 ceph-mon[30086]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:05.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:05 smithi134 ceph-mon[39278]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:05 smithi196 ceph-mon[33018]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:08.035 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:07 smithi134 ceph-mon[30086]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:08.035 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:07 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-05T08:02:08.036 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:07 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:08.036 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:07 smithi134 ceph-mon[39278]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:08.036 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:07 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-05T08:02:08.036 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:07 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:07 smithi196 ceph-mon[33018]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:08.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:07 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-05T08:02:08.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:07 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:08.899 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:08 smithi134 ceph-mon[30086]: Deploying daemon osd.2 on smithi134 2024-02-05T08:02:08.899 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:08 smithi134 ceph-mon[39278]: Deploying daemon osd.2 on smithi134 2024-02-05T08:02:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:08 smithi196 ceph-mon[33018]: Deploying daemon osd.2 on smithi134 2024-02-05T08:02:10.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:09 smithi196 ceph-mon[33018]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:10.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:09 smithi134 ceph-mon[30086]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:10.171 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:09 smithi134 ceph-mon[39278]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:11.921 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:11 smithi134 ceph-mon[30086]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:11.921 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:11 smithi134 ceph-mon[39278]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:12.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:11 smithi196 ceph-mon[33018]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:13 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:14.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:13 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:14.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:13 smithi134 ceph-mon[30086]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:14.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:13 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:02:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:13 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:13 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:13 smithi134 ceph-mon[39278]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:13 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:02:14.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:13 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:14.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:13 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:14.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:13 smithi196 ceph-mon[33018]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:14.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:13 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:02:15.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:15 smithi134 ceph-mon[30086]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:15.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:15 smithi134 ceph-mon[39278]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:15 smithi196 ceph-mon[33018]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:16.438 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 2 on host 'smithi134' 2024-02-05T08:02:17.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[30086]: from='osd.2 [v2:172.21.15.134:6818/153827987,v1:172.21.15.134:6819/153827987]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T08:02:17.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[30086]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T08:02:17.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:17.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:02:17.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:02:17.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[39278]: from='osd.2 [v2:172.21.15.134:6818/153827987,v1:172.21.15.134:6819/153827987]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T08:02:17.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[39278]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T08:02:17.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:17.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:02:17.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:02:17.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:16 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.053 DEBUG:teuthology.orchestra.run.smithi134:osd.2> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.2.service 2024-02-05T08:02:17.055 INFO:tasks.cephadm:Deploying osd.3 on smithi134 with /dev/vg_nvme/lv_1... 2024-02-05T08:02:17.055 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-05T08:02:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:16 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:16 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:16 smithi196 ceph-mon[33018]: from='osd.2 [v2:172.21.15.134:6818/153827987,v1:172.21.15.134:6819/153827987]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T08:02:17.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:16 smithi196 ceph-mon[33018]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T08:02:17.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:16 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:17.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:16 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:02:17.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:16 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:16 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:16 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:02:17.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:16 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:17.676 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:17 smithi134 ceph-mon[39278]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-05T08:02:17.676 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:17 smithi134 ceph-mon[39278]: osdmap e16: 3 total, 2 up, 3 in 2024-02-05T08:02:17.676 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:17 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:17.676 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:17 smithi134 ceph-mon[39278]: from='osd.2 [v2:172.21.15.134:6818/153827987,v1:172.21.15.134:6819/153827987]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:02:17.755 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:02:17.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:17 smithi134 ceph-mon[30086]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-05T08:02:17.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:17 smithi134 ceph-mon[30086]: osdmap e16: 3 total, 2 up, 3 in 2024-02-05T08:02:17.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:17 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:17.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:17 smithi134 ceph-mon[30086]: from='osd.2 [v2:172.21.15.134:6818/153827987,v1:172.21.15.134:6819/153827987]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:02:17.971 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:17 smithi134 ceph-mon[30086]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:02:17.971 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:17 smithi134 ceph-mon[30086]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:17.971 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:17 smithi134 ceph-mon[39278]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:02:17.971 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:17 smithi134 ceph-mon[39278]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:17 smithi196 ceph-mon[33018]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-05T08:02:18.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:17 smithi196 ceph-mon[33018]: osdmap e16: 3 total, 2 up, 3 in 2024-02-05T08:02:18.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:17 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:18.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:17 smithi196 ceph-mon[33018]: from='osd.2 [v2:172.21.15.134:6818/153827987,v1:172.21.15.134:6819/153827987]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:02:18.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:17 smithi196 ceph-mon[33018]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:02:18.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:17 smithi196 ceph-mon[33018]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:18.278 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:02:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2[56265]: 2024-02-05T08:02:18.103+0000 7f4ee8bac700 -1 osd.2 0 waiting for initial osdmap 2024-02-05T08:02:18.278 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:02:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2[56265]: 2024-02-05T08:02:18.107+0000 7f4ee019d700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T08:02:18.979 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:18 smithi134 ceph-mon[30086]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-02-05T08:02:18.979 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:18 smithi134 ceph-mon[30086]: osdmap e17: 3 total, 2 up, 3 in 2024-02-05T08:02:18.979 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:18 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:18.979 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:18 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:18.979 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:18 smithi134 ceph-mon[30086]: from='osd.2 ' entity='osd.2' 2024-02-05T08:02:18.980 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:18 smithi134 ceph-mon[39278]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-02-05T08:02:18.980 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:18 smithi134 ceph-mon[39278]: osdmap e17: 3 total, 2 up, 3 in 2024-02-05T08:02:18.980 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:18 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:18.980 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:18 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:18.980 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:18 smithi134 ceph-mon[39278]: from='osd.2 ' entity='osd.2' 2024-02-05T08:02:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:18 smithi196 ceph-mon[33018]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-02-05T08:02:19.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:18 smithi196 ceph-mon[33018]: osdmap e17: 3 total, 2 up, 3 in 2024-02-05T08:02:19.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:18 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:19.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:18 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:19.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:18 smithi196 ceph-mon[33018]: from='osd.2 ' entity='osd.2' 2024-02-05T08:02:19.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:02:19.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:02:19.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:19.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[30086]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:19.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[30086]: osd.2 [v2:172.21.15.134:6818/153827987,v1:172.21.15.134:6819/153827987] boot 2024-02-05T08:02:19.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[30086]: osdmap e18: 3 total, 3 up, 3 in 2024-02-05T08:02:19.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:19.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:02:19.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:02:19.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:19.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[39278]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:19.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[39278]: osd.2 [v2:172.21.15.134:6818/153827987,v1:172.21.15.134:6819/153827987] boot 2024-02-05T08:02:19.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[39278]: osdmap e18: 3 total, 3 up, 3 in 2024-02-05T08:02:19.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:19 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:19 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:02:20.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:19 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:02:20.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:19 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:20.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:19 smithi196 ceph-mon[33018]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-05T08:02:20.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:19 smithi196 ceph-mon[33018]: osd.2 [v2:172.21.15.134:6818/153827987,v1:172.21.15.134:6819/153827987] boot 2024-02-05T08:02:20.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:19 smithi196 ceph-mon[33018]: osdmap e18: 3 total, 3 up, 3 in 2024-02-05T08:02:20.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:19 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:02:20.502 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-05T08:02:20.503 INFO:teuthology.orchestra.run.smithi134.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-05T08:02:20.503 INFO:teuthology.orchestra.run.smithi134.stderr: stderr: 10+0 records in 2024-02-05T08:02:20.503 INFO:teuthology.orchestra.run.smithi134.stderr:10+0 records out 2024-02-05T08:02:20.503 INFO:teuthology.orchestra.run.smithi134.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123558 s, 849 MB/s 2024-02-05T08:02:20.503 INFO:teuthology.orchestra.run.smithi134.stderr:--> Zapping successful for: 2024-02-05T08:02:21.176 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi134:vg_nvme/lv_1 2024-02-05T08:02:21.683 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:02:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:21 smithi134 ceph-mon[30086]: osdmap e19: 3 total, 3 up, 3 in 2024-02-05T08:02:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:21 smithi134 ceph-mon[30086]: pgmap v78: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:21 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:02:22.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:21 smithi134 ceph-mon[39278]: osdmap e19: 3 total, 3 up, 3 in 2024-02-05T08:02:22.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:21 smithi134 ceph-mon[39278]: pgmap v78: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:22.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:21 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:02:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:21 smithi196 ceph-mon[33018]: osdmap e19: 3 total, 3 up, 3 in 2024-02-05T08:02:22.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:21 smithi196 ceph-mon[33018]: pgmap v78: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:22.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:21 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:02:23.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: Detected new or changed devices on smithi134 2024-02-05T08:02:23.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:23.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-05T08:02:23.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: osdmap e20: 3 total, 3 up, 3 in 2024-02-05T08:02:23.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:02:23.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:23.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:23.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:23.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:23.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: Adjusting osd_memory_target on smithi134 to 5354M 2024-02-05T08:02:23.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:23.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:23.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:02:23.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:23.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: Detected new or changed devices on smithi134 2024-02-05T08:02:23.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:23.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-05T08:02:23.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: osdmap e20: 3 total, 3 up, 3 in 2024-02-05T08:02:23.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:02:23.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:23.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:23.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:23.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:23.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: Adjusting osd_memory_target on smithi134 to 5354M 2024-02-05T08:02:23.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:23.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:23.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:02:23.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:22 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:23.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: Detected new or changed devices on smithi134 2024-02-05T08:02:23.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:23.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-05T08:02:23.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: osdmap e20: 3 total, 3 up, 3 in 2024-02-05T08:02:23.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:02:23.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:23.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:23.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:23.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:23.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: Adjusting osd_memory_target on smithi134 to 5354M 2024-02-05T08:02:23.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:23.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:23.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:02:23.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:22 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:23.910 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:02:23 smithi134 sudo[60800]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T08:02:23.910 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:02:23 smithi134 sudo[60800]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T08:02:23.910 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:02:23 smithi134 sudo[60800]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T08:02:23.911 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:02:23 smithi134 sudo[60800]: pam_unix(sudo:session): session closed for user root 2024-02-05T08:02:23.911 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:23 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:02:23.911 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:23 smithi134 ceph-mon[30086]: osdmap e21: 3 total, 3 up, 3 in 2024-02-05T08:02:23.911 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:23 smithi134 ceph-mon[30086]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:23.911 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:23 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:02:23.911 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:23 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:02:23.911 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:23 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:23.911 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:23 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:02:23.911 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:23 smithi134 ceph-mon[39278]: osdmap e21: 3 total, 3 up, 3 in 2024-02-05T08:02:23.912 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:23 smithi134 ceph-mon[39278]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:23.912 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:23 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:02:23.912 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:23 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:02:23.912 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:23 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:24.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:23 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:02:24.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:23 smithi196 ceph-mon[33018]: osdmap e21: 3 total, 3 up, 3 in 2024-02-05T08:02:24.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:23 smithi196 ceph-mon[33018]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:24.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:23 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:02:24.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:23 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:02:24.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:23 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:24.190 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:02:23 smithi134 sudo[60855]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T08:02:24.190 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:02:23 smithi134 sudo[60855]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T08:02:24.190 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:02:23 smithi134 sudo[60855]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T08:02:24.190 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:02:23 smithi134 sudo[60855]: pam_unix(sudo:session): session closed for user root 2024-02-05T08:02:24.455 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60914]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T08:02:24.455 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60914]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T08:02:24.455 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60914]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T08:02:24.455 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60914]: pam_unix(sudo:session): session closed for user root 2024-02-05T08:02:24.705 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60936]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T08:02:24.706 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60936]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T08:02:24.706 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60936]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T08:02:24.706 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60936]: pam_unix(sudo:session): session closed for user root 2024-02-05T08:02:24.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:24 smithi134 ceph-mon[30086]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:02:24.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:24 smithi134 ceph-mon[30086]: osdmap e22: 3 total, 3 up, 3 in 2024-02-05T08:02:24.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:24 smithi134 ceph-mon[30086]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-05T08:02:24.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:24 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:24.970 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60957]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T08:02:24.971 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60957]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T08:02:24.971 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60957]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T08:02:24.971 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60957]: pam_unix(sudo:session): session closed for user root 2024-02-05T08:02:24.971 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:24 smithi134 ceph-mon[39278]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:02:24.971 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:24 smithi134 ceph-mon[39278]: osdmap e22: 3 total, 3 up, 3 in 2024-02-05T08:02:24.971 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:24 smithi134 ceph-mon[39278]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-05T08:02:24.971 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:24 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:25.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:24 smithi196 ceph-mon[33018]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi134:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:02:25.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:24 smithi196 ceph-mon[33018]: osdmap e22: 3 total, 3 up, 3 in 2024-02-05T08:02:25.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:24 smithi196 ceph-mon[33018]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-05T08:02:25.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:24 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:25.231 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60969]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T08:02:25.232 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60969]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T08:02:25.232 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60969]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T08:02:25.232 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:02:24 smithi134 sudo[60969]: pam_unix(sudo:session): session closed for user root 2024-02-05T08:02:25.495 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:25 smithi134 sudo[60993]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T08:02:25.495 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:25 smithi134 sudo[60993]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T08:02:25.495 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:25 smithi134 sudo[60993]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T08:02:25.496 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:25 smithi134 sudo[60993]: pam_unix(sudo:session): session closed for user root 2024-02-05T08:02:25.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:25 smithi134 sudo[61016]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T08:02:25.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:25 smithi134 sudo[61016]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T08:02:25.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:25 smithi134 sudo[61016]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T08:02:25.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:25 smithi134 sudo[61016]: pam_unix(sudo:session): session closed for user root 2024-02-05T08:02:25.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:25 smithi196 sudo[35871]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T08:02:25.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:25 smithi196 sudo[35871]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T08:02:25.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:25 smithi196 sudo[35871]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T08:02:25.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:25 smithi196 sudo[35871]: pam_unix(sudo:session): session closed for user root 2024-02-05T08:02:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[30086]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[30086]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T08:02:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[30086]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T08:02:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:02:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:02:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:02:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:02:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:02:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:02:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[39278]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[39278]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T08:02:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[39278]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T08:02:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:02:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:02:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:02:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:02:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:02:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:02:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:25 smithi196 sudo[35874]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T08:02:26.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:25 smithi196 sudo[35874]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T08:02:26.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:25 smithi196 sudo[35874]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T08:02:26.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:25 smithi196 sudo[35874]: pam_unix(sudo:session): session closed for user root 2024-02-05T08:02:26.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:26 smithi196 ceph-mon[33018]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:26.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:26 smithi196 ceph-mon[33018]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T08:02:26.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:26 smithi196 ceph-mon[33018]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T08:02:26.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:26 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:02:26.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:26 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:02:26.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:26 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:02:26.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:26 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:02:26.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:26 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:02:26.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:26 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:02:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 sudo[61069]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T08:02:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 sudo[61069]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T08:02:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 sudo[61069]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T08:02:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 sudo[61069]: pam_unix(sudo:session): session closed for user root 2024-02-05T08:02:27.002 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 sudo[61073]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T08:02:27.002 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 sudo[61073]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T08:02:27.002 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 sudo[61073]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T08:02:27.002 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:26 smithi134 sudo[61073]: pam_unix(sudo:session): session closed for user root 2024-02-05T08:02:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[30086]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T08:02:27.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[30086]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T08:02:27.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:27.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[30086]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T08:02:27.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:02:27.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:02:27.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:02:27.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1358470688' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6774e330-3202-4671-894e-172cc2f6189d"}]: dispatch 2024-02-05T08:02:27.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[30086]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T08:02:27.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1358470688' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6774e330-3202-4671-894e-172cc2f6189d"}]': finished 2024-02-05T08:02:27.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[30086]: osdmap e23: 4 total, 3 up, 4 in 2024-02-05T08:02:27.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[30086]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[39278]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[39278]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[39278]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1358470688' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6774e330-3202-4671-894e-172cc2f6189d"}]: dispatch 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[39278]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1358470688' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6774e330-3202-4671-894e-172cc2f6189d"}]': finished 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[39278]: osdmap e23: 4 total, 3 up, 4 in 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:27.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:27 smithi134 ceph-mon[39278]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:27 smithi196 ceph-mon[33018]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T08:02:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:27 smithi196 ceph-mon[33018]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T08:02:27.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:27 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:27.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:27 smithi196 ceph-mon[33018]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T08:02:27.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:27 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:02:27.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:27 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:02:27.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:27 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:02:27.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:27 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1358470688' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6774e330-3202-4671-894e-172cc2f6189d"}]: dispatch 2024-02-05T08:02:27.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:27 smithi196 ceph-mon[33018]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T08:02:27.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:27 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1358470688' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6774e330-3202-4671-894e-172cc2f6189d"}]': finished 2024-02-05T08:02:27.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:27 smithi196 ceph-mon[33018]: osdmap e23: 4 total, 3 up, 4 in 2024-02-05T08:02:27.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:27 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:27.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:27 smithi196 ceph-mon[33018]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:28 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1984687609' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:02:28.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:28 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1984687609' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:02:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:28 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1984687609' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:02:29.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:29 smithi134 ceph-mon[30086]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:29.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:29 smithi134 ceph-mon[39278]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:29 smithi196 ceph-mon[33018]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:31.977 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:31 smithi134 ceph-mon[30086]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-05T08:02:31.977 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:31 smithi134 ceph-mon[39278]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-05T08:02:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:31 smithi196 ceph-mon[33018]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-05T08:02:33.869 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:33 smithi134 ceph-mon[30086]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-05T08:02:33.869 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:33 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-05T08:02:33.869 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:33 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:33.869 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:33 smithi134 ceph-mon[39278]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-05T08:02:33.869 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:33 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-05T08:02:33.870 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:33 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:33 smithi196 ceph-mon[33018]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-05T08:02:34.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:33 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-05T08:02:34.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:33 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:34 smithi196 ceph-mon[33018]: Deploying daemon osd.3 on smithi134 2024-02-05T08:02:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:34 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:02:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:34 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:02:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:34 smithi196 ceph-mon[33018]: osdmap e24: 4 total, 3 up, 4 in 2024-02-05T08:02:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:34 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:35.092 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:34 smithi134 ceph-mon[30086]: Deploying daemon osd.3 on smithi134 2024-02-05T08:02:35.092 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:34 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:02:35.092 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:34 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:02:35.092 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:34 smithi134 ceph-mon[30086]: osdmap e24: 4 total, 3 up, 4 in 2024-02-05T08:02:35.092 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:34 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:35.092 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:34 smithi134 ceph-mon[39278]: Deploying daemon osd.3 on smithi134 2024-02-05T08:02:35.092 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:34 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:02:35.093 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:34 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:02:35.093 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:34 smithi134 ceph-mon[39278]: osdmap e24: 4 total, 3 up, 4 in 2024-02-05T08:02:35.093 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:34 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:35 smithi134 ceph-mon[30086]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 96 KiB/s wr, 6 op/s 2024-02-05T08:02:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:35 smithi134 ceph-mon[39278]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 96 KiB/s wr, 6 op/s 2024-02-05T08:02:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:35 smithi196 ceph-mon[33018]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 96 KiB/s wr, 6 op/s 2024-02-05T08:02:37.871 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:37 smithi134 ceph-mon[30086]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-05T08:02:37.872 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:37 smithi134 ceph-mon[39278]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-05T08:02:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:37 smithi196 ceph-mon[33018]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-05T08:02:39.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:39 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:39.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:39 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:39.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:39 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:02:39.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:39 smithi134 ceph-mon[39278]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-05T08:02:39.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:39 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:39.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:39 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:39.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:39 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:02:39.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:39 smithi134 ceph-mon[30086]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-05T08:02:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:39 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:02:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:39 smithi196 ceph-mon[33018]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-05T08:02:41.883 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[30086]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:41.884 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:41.884 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:41.884 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:41.884 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:02:41.884 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:41.884 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[30086]: from='osd.3 [v2:172.21.15.134:6826/1527507056,v1:172.21.15.134:6827/1527507056]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T08:02:41.884 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[30086]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T08:02:41.885 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[39278]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:41.885 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:41.885 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:41.885 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:41.885 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:02:41.885 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:41.886 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[39278]: from='osd.3 [v2:172.21.15.134:6826/1527507056,v1:172.21.15.134:6827/1527507056]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T08:02:41.886 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:41 smithi134 ceph-mon[39278]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T08:02:42.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:41 smithi196 ceph-mon[33018]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:42.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:41 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:42.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:41 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:42.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:41 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:42.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:41 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:02:42.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:41 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:42.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:41 smithi196 ceph-mon[33018]: from='osd.3 [v2:172.21.15.134:6826/1527507056,v1:172.21.15.134:6827/1527507056]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T08:02:42.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:41 smithi196 ceph-mon[33018]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T08:02:42.222 INFO:teuthology.orchestra.run.smithi134.stdout:Created osd(s) 3 on host 'smithi134' 2024-02-05T08:02:42.848 DEBUG:teuthology.orchestra.run.smithi134:osd.3> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.3.service 2024-02-05T08:02:42.851 INFO:tasks.cephadm:Deploying osd.4 on smithi196 with /dev/vg_nvme/lv_4... 2024-02-05T08:02:42.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-05T08:02:43.094 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:02:43.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:43 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:43.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:43 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:43.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:43 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:02:43.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:43 smithi196 ceph-mon[33018]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-05T08:02:43.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:43 smithi196 ceph-mon[33018]: osdmap e25: 4 total, 3 up, 4 in 2024-02-05T08:02:43.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:43 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:43.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:43 smithi196 ceph-mon[33018]: from='osd.3 [v2:172.21.15.134:6826/1527507056,v1:172.21.15.134:6827/1527507056]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:02:43.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:43 smithi196 ceph-mon[33018]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:02:43.455 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:43 smithi196 ceph-mon[33018]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:02:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[39278]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-05T08:02:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[39278]: osdmap e25: 4 total, 3 up, 4 in 2024-02-05T08:02:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[39278]: from='osd.3 [v2:172.21.15.134:6826/1527507056,v1:172.21.15.134:6827/1527507056]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:02:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[39278]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:02:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[39278]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:43.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:43.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:43.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:02:43.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[30086]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-05T08:02:43.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[30086]: osdmap e25: 4 total, 3 up, 4 in 2024-02-05T08:02:43.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:43.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[30086]: from='osd.3 [v2:172.21.15.134:6826/1527507056,v1:172.21.15.134:6827/1527507056]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:02:43.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[30086]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:02:43.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-mon[30086]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:43.994 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3[62874]: 2024-02-05T08:02:43.875+0000 7fe2a5583700 -1 osd.3 0 waiting for initial osdmap 2024-02-05T08:02:43.995 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:02:43 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3[62874]: 2024-02-05T08:02:43.880+0000 7fe29cb74700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T08:02:44.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:44 smithi196 ceph-mon[33018]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-02-05T08:02:44.526 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:44 smithi196 ceph-mon[33018]: osdmap e26: 4 total, 3 up, 4 in 2024-02-05T08:02:44.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:44.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:44 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:44.527 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:44 smithi196 ceph-mon[33018]: from='osd.3 ' entity='osd.3' 2024-02-05T08:02:44.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:44 smithi134 ceph-mon[30086]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-02-05T08:02:44.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:44 smithi134 ceph-mon[30086]: osdmap e26: 4 total, 3 up, 4 in 2024-02-05T08:02:44.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:44.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:44 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:44.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:44 smithi134 ceph-mon[30086]: from='osd.3 ' entity='osd.3' 2024-02-05T08:02:44.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:44 smithi134 ceph-mon[39278]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]': finished 2024-02-05T08:02:44.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:44 smithi134 ceph-mon[39278]: osdmap e26: 4 total, 3 up, 4 in 2024-02-05T08:02:44.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:44 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:44.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:44 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:44.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:44 smithi134 ceph-mon[39278]: from='osd.3 ' entity='osd.3' 2024-02-05T08:02:45.095 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-05T08:02:45.095 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-05T08:02:45.095 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-02-05T08:02:45.095 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-02-05T08:02:45.095 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113997 s, 920 MB/s 2024-02-05T08:02:45.095 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-02-05T08:02:45.672 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:45 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:02:45.672 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:45 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:02:45.672 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:45 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:45.673 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:45 smithi196 ceph-mon[33018]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:45.673 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:45 smithi196 ceph-mon[33018]: osd.3 [v2:172.21.15.134:6826/1527507056,v1:172.21.15.134:6827/1527507056] boot 2024-02-05T08:02:45.673 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:45 smithi196 ceph-mon[33018]: osdmap e27: 4 total, 4 up, 4 in 2024-02-05T08:02:45.673 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:45 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:45.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi196:vg_nvme/lv_4 2024-02-05T08:02:45.760 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:02:45.760 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:02:45.760 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:45.760 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[30086]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:45.760 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[30086]: osd.3 [v2:172.21.15.134:6826/1527507056,v1:172.21.15.134:6827/1527507056] boot 2024-02-05T08:02:45.760 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[30086]: osdmap e27: 4 total, 4 up, 4 in 2024-02-05T08:02:45.760 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:45.761 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:02:45.761 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:02:45.761 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:45.761 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[39278]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-05T08:02:45.761 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[39278]: osd.3 [v2:172.21.15.134:6826/1527507056,v1:172.21.15.134:6827/1527507056] boot 2024-02-05T08:02:45.761 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[39278]: osdmap e27: 4 total, 4 up, 4 in 2024-02-05T08:02:45.761 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:45 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:02:45.945 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:02:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:46 smithi196 ceph-mon[33018]: Detected new or changed devices on smithi134 2024-02-05T08:02:47.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:46 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:47.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:46 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:47.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:46 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:47.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:46 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:47.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:46 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:47.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:46 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:47.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:46 smithi196 ceph-mon[33018]: Adjusting osd_memory_target on smithi134 to 4016M 2024-02-05T08:02:47.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:46 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:47.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:46 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:47.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:46 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:02:47.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:46 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:47.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:46 smithi196 ceph-mon[33018]: osdmap e28: 4 total, 4 up, 4 in 2024-02-05T08:02:47.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[30086]: Detected new or changed devices on smithi134 2024-02-05T08:02:47.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:47.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:47.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:47.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:47.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:47.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:47.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[30086]: Adjusting osd_memory_target on smithi134 to 4016M 2024-02-05T08:02:47.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:47.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:47.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:02:47.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:47.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[30086]: osdmap e28: 4 total, 4 up, 4 in 2024-02-05T08:02:47.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[39278]: Detected new or changed devices on smithi134 2024-02-05T08:02:47.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:47.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:47.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:47.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:47.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:47.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:02:47.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[39278]: Adjusting osd_memory_target on smithi134 to 4016M 2024-02-05T08:02:47.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:47.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:47.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:02:47.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:02:47.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:46 smithi134 ceph-mon[39278]: osdmap e28: 4 total, 4 up, 4 in 2024-02-05T08:02:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:47 smithi196 ceph-mon[33018]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:48.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:47 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:02:48.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:47 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:02:48.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:47 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:48.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:47 smithi134 ceph-mon[30086]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:48.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:47 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:02:48.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:47 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:02:48.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:47 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:48.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:47 smithi134 ceph-mon[39278]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:48.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:02:48.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:02:48.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:49.044 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:48 smithi196 ceph-mon[33018]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:02:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:48 smithi134 ceph-mon[30086]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:02:49.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:48 smithi134 ceph-mon[39278]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:02:49.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:49 smithi196 ceph-mon[33018]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:50.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:49 smithi134 ceph-mon[30086]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:50.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:49 smithi134 ceph-mon[39278]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:51.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:50 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/2010605123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693"}]: dispatch 2024-02-05T08:02:51.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:50 smithi134 ceph-mon[30086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693"}]: dispatch 2024-02-05T08:02:51.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:50 smithi134 ceph-mon[30086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693"}]': finished 2024-02-05T08:02:51.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:50 smithi134 ceph-mon[30086]: osdmap e29: 5 total, 4 up, 5 in 2024-02-05T08:02:51.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:50 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:02:51.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:50 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/2010605123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693"}]: dispatch 2024-02-05T08:02:51.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:50 smithi134 ceph-mon[39278]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693"}]: dispatch 2024-02-05T08:02:51.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:50 smithi134 ceph-mon[39278]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693"}]': finished 2024-02-05T08:02:51.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:50 smithi134 ceph-mon[39278]: osdmap e29: 5 total, 4 up, 5 in 2024-02-05T08:02:51.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:50 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:02:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:50 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/2010605123' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693"}]: dispatch 2024-02-05T08:02:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:50 smithi196 ceph-mon[33018]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693"}]: dispatch 2024-02-05T08:02:51.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:50 smithi196 ceph-mon[33018]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693"}]': finished 2024-02-05T08:02:51.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:50 smithi196 ceph-mon[33018]: osdmap e29: 5 total, 4 up, 5 in 2024-02-05T08:02:51.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:50 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:02:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:51 smithi134 ceph-mon[30086]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:51 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/3723556613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:02:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:51 smithi134 ceph-mon[39278]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:51 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/3723556613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:02:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:51 smithi196 ceph-mon[33018]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:51 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/3723556613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:02:53.944 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:53 smithi196 ceph-mon[33018]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:53 smithi134 ceph-mon[30086]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:54.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:53 smithi134 ceph-mon[39278]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:55.517 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:55 smithi196 ceph-mon[33018]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:55.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:55 smithi134 ceph-mon[30086]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:55.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:55 smithi134 ceph-mon[39278]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:58.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:57 smithi196 ceph-mon[33018]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:58.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:57 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-05T08:02:58.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:57 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:58.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:57 smithi196 ceph-mon[33018]: Deploying daemon osd.4 on smithi196 2024-02-05T08:02:58.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:57 smithi134 ceph-mon[30086]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:58.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:57 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-05T08:02:58.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:57 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:58.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:57 smithi134 ceph-mon[30086]: Deploying daemon osd.4 on smithi196 2024-02-05T08:02:58.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:57 smithi134 ceph-mon[39278]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:02:58.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:57 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-05T08:02:58.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:57 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:02:58.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:57 smithi134 ceph-mon[39278]: Deploying daemon osd.4 on smithi196 2024-02-05T08:02:59.902 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:02:59 smithi196 ceph-mon[33018]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:02:59 smithi134 ceph-mon[30086]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:00.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:02:59 smithi134 ceph-mon[39278]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:01.851 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:01 smithi196 ceph-mon[33018]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:01 smithi134 ceph-mon[30086]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:02.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:01 smithi134 ceph-mon[39278]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:03.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:03.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:03.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:03 smithi196 ceph-mon[33018]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:03 smithi134 ceph-mon[30086]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:03 smithi134 ceph-mon[39278]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:05.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:05 smithi134 ceph-mon[30086]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:05.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:05 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:05.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:05 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:05.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:05 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:05.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:05 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:03:05.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:05 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:05.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:05 smithi134 ceph-mon[39278]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:05.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:05 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:05.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:05 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:05.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:05 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:05.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:05 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:03:05.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:05 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:05 smithi196 ceph-mon[33018]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:05 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:05 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:05 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:05.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:05 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:03:05.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:05 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:06.238 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 4 on host 'smithi196' 2024-02-05T08:03:06.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:06 smithi134 ceph-mon[30086]: from='osd.4 [v2:172.21.15.196:6800/3005156808,v1:172.21.15.196:6801/3005156808]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T08:03:06.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:06 smithi134 ceph-mon[30086]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T08:03:06.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:06.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:06.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:06.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:06 smithi134 ceph-mon[39278]: from='osd.4 [v2:172.21.15.196:6800/3005156808,v1:172.21.15.196:6801/3005156808]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T08:03:06.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:06 smithi134 ceph-mon[39278]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T08:03:06.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:06.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:06.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:06.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:06 smithi196 ceph-mon[33018]: from='osd.4 [v2:172.21.15.196:6800/3005156808,v1:172.21.15.196:6801/3005156808]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T08:03:06.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:06 smithi196 ceph-mon[33018]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T08:03:06.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:06.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:06.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:06.839 DEBUG:teuthology.orchestra.run.smithi196:osd.4> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.4.service 2024-02-05T08:03:06.842 INFO:tasks.cephadm:Deploying osd.5 on smithi196 with /dev/vg_nvme/lv_3... 2024-02-05T08:03:06.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-05T08:03:07.617 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:03:07.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:07 smithi134 ceph-mon[30086]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-05T08:03:07.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:07 smithi134 ceph-mon[30086]: from='osd.4 [v2:172.21.15.196:6800/3005156808,v1:172.21.15.196:6801/3005156808]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:03:07.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:07 smithi134 ceph-mon[30086]: osdmap e30: 5 total, 4 up, 5 in 2024-02-05T08:03:07.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:07 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:03:07.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:07 smithi134 ceph-mon[30086]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:03:07.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:07 smithi134 ceph-mon[30086]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:07.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:07 smithi134 ceph-mon[39278]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-05T08:03:07.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:07 smithi134 ceph-mon[39278]: from='osd.4 [v2:172.21.15.196:6800/3005156808,v1:172.21.15.196:6801/3005156808]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:03:07.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:07 smithi134 ceph-mon[39278]: osdmap e30: 5 total, 4 up, 5 in 2024-02-05T08:03:07.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:07 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:03:07.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:07 smithi134 ceph-mon[39278]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:03:07.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:07 smithi134 ceph-mon[39278]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:07.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:07 smithi196 ceph-mon[33018]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-05T08:03:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:07 smithi196 ceph-mon[33018]: from='osd.4 [v2:172.21.15.196:6800/3005156808,v1:172.21.15.196:6801/3005156808]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:03:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:07 smithi196 ceph-mon[33018]: osdmap e30: 5 total, 4 up, 5 in 2024-02-05T08:03:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:07 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:03:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:07 smithi196 ceph-mon[33018]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:03:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:07 smithi196 ceph-mon[33018]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-02-05T08:03:07.815 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:03:07 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:03:07.511+0000 7efd210ce700 -1 osd.4 0 waiting for initial osdmap 2024-02-05T08:03:07.815 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:03:07 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:03:07.517+0000 7efd1cec8700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T08:03:08.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[30086]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-02-05T08:03:08.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[30086]: osdmap e31: 5 total, 4 up, 5 in 2024-02-05T08:03:08.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:03:08.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:03:08.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[30086]: osd.4 [v2:172.21.15.196:6800/3005156808,v1:172.21.15.196:6801/3005156808] boot 2024-02-05T08:03:08.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[30086]: osdmap e32: 5 total, 5 up, 5 in 2024-02-05T08:03:08.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:03:08.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[39278]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-02-05T08:03:08.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[39278]: osdmap e31: 5 total, 4 up, 5 in 2024-02-05T08:03:08.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:03:08.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:03:08.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[39278]: osd.4 [v2:172.21.15.196:6800/3005156808,v1:172.21.15.196:6801/3005156808] boot 2024-02-05T08:03:08.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[39278]: osdmap e32: 5 total, 5 up, 5 in 2024-02-05T08:03:08.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:08 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:03:08.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:08 smithi196 ceph-mon[33018]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-02-05T08:03:08.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:08 smithi196 ceph-mon[33018]: osdmap e31: 5 total, 4 up, 5 in 2024-02-05T08:03:08.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:08 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:03:08.809 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:08 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:03:08.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:08 smithi196 ceph-mon[33018]: osd.4 [v2:172.21.15.196:6800/3005156808,v1:172.21.15.196:6801/3005156808] boot 2024-02-05T08:03:08.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:08 smithi196 ceph-mon[33018]: osdmap e32: 5 total, 5 up, 5 in 2024-02-05T08:03:08.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:08 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:03:09.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:09 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:03:09.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:09 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:03:09.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:09 smithi196 ceph-mon[33018]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:09.688 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:09 smithi196 ceph-mon[33018]: osdmap e33: 5 total, 5 up, 5 in 2024-02-05T08:03:09.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:09 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:03:09.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:09 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:03:09.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:09 smithi134 ceph-mon[30086]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:09.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:09 smithi134 ceph-mon[30086]: osdmap e33: 5 total, 5 up, 5 in 2024-02-05T08:03:09.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:09 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:03:09.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:09 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:03:09.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:09 smithi134 ceph-mon[39278]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:09.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:09 smithi134 ceph-mon[39278]: osdmap e33: 5 total, 5 up, 5 in 2024-02-05T08:03:10.714 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-05T08:03:10.714 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-05T08:03:10.714 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-02-05T08:03:10.714 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-02-05T08:03:10.714 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112759 s, 930 MB/s 2024-02-05T08:03:10.714 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-02-05T08:03:11.493 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi196:vg_nvme/lv_3 2024-02-05T08:03:11.729 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:03:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[30086]: osdmap e34: 5 total, 5 up, 5 in 2024-02-05T08:03:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[30086]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[30086]: Detected new or changed devices on smithi196 2024-02-05T08:03:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:11.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:11.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:03:11.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[30086]: Adjusting osd_memory_target on smithi196 to 17088M 2024-02-05T08:03:11.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:11.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:11.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:03:11.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[39278]: osdmap e34: 5 total, 5 up, 5 in 2024-02-05T08:03:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[39278]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[39278]: Detected new or changed devices on smithi196 2024-02-05T08:03:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:11.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:03:11.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[39278]: Adjusting osd_memory_target on smithi196 to 17088M 2024-02-05T08:03:11.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:11.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:11.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:03:11.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:11 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:11.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:11 smithi196 ceph-mon[33018]: osdmap e34: 5 total, 5 up, 5 in 2024-02-05T08:03:11.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:11 smithi196 ceph-mon[33018]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:11.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:11 smithi196 ceph-mon[33018]: Detected new or changed devices on smithi196 2024-02-05T08:03:11.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:11 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:11.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:11 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:11.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:11 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:03:11.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:11 smithi196 ceph-mon[33018]: Adjusting osd_memory_target on smithi196 to 17088M 2024-02-05T08:03:11.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:11 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:11.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:11 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:11.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:11 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:03:11.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:11 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:12.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:12 smithi134 ceph-mon[30086]: osdmap e35: 5 total, 5 up, 5 in 2024-02-05T08:03:12.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:12 smithi134 ceph-mon[39278]: osdmap e35: 5 total, 5 up, 5 in 2024-02-05T08:03:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:12 smithi196 ceph-mon[33018]: osdmap e35: 5 total, 5 up, 5 in 2024-02-05T08:03:13.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:13 smithi134 ceph-mon[30086]: osdmap e36: 5 total, 5 up, 5 in 2024-02-05T08:03:13.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:13 smithi134 ceph-mon[30086]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:13.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:13 smithi134 ceph-mon[39278]: osdmap e36: 5 total, 5 up, 5 in 2024-02-05T08:03:13.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:13 smithi134 ceph-mon[39278]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:13.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:13 smithi196 ceph-mon[33018]: osdmap e36: 5 total, 5 up, 5 in 2024-02-05T08:03:13.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:13 smithi196 ceph-mon[33018]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:14.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:14 smithi196 ceph-mon[33018]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:03:14.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:14 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:03:14.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:14 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:03:14.706 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:14 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:14.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:14 smithi134 ceph-mon[30086]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:03:14.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:14 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:03:14.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:14 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:03:14.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:14 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:14.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:14 smithi134 ceph-mon[39278]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:03:14.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:14 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:03:14.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:14 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:03:14.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:14 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:15.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:15 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:03:15.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:15 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:03:15.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:15 smithi134 ceph-mon[30086]: osdmap e37: 5 total, 5 up, 5 in 2024-02-05T08:03:15.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:15 smithi134 ceph-mon[30086]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-02-05T08:03:15.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:15 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:03:15.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:15 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:03:15.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:15 smithi134 ceph-mon[39278]: osdmap e37: 5 total, 5 up, 5 in 2024-02-05T08:03:15.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:15 smithi134 ceph-mon[39278]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-02-05T08:03:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:15 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:03:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:15 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:03:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:15 smithi196 ceph-mon[33018]: osdmap e37: 5 total, 5 up, 5 in 2024-02-05T08:03:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:15 smithi196 ceph-mon[33018]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 135 KiB/s, 0 objects/s recovering 2024-02-05T08:03:16.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:16 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/4138072690' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955"}]: dispatch 2024-02-05T08:03:16.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:16 smithi134 ceph-mon[30086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955"}]: dispatch 2024-02-05T08:03:16.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:16 smithi134 ceph-mon[30086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955"}]': finished 2024-02-05T08:03:16.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:16 smithi134 ceph-mon[30086]: osdmap e38: 6 total, 5 up, 6 in 2024-02-05T08:03:16.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:16 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:16.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:16 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/4138072690' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955"}]: dispatch 2024-02-05T08:03:16.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:16 smithi134 ceph-mon[39278]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955"}]: dispatch 2024-02-05T08:03:16.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:16 smithi134 ceph-mon[39278]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955"}]': finished 2024-02-05T08:03:16.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:16 smithi134 ceph-mon[39278]: osdmap e38: 6 total, 5 up, 6 in 2024-02-05T08:03:16.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:16 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:16 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/4138072690' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955"}]: dispatch 2024-02-05T08:03:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:16 smithi196 ceph-mon[33018]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955"}]: dispatch 2024-02-05T08:03:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:16 smithi196 ceph-mon[33018]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955"}]': finished 2024-02-05T08:03:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:16 smithi196 ceph-mon[33018]: osdmap e38: 6 total, 5 up, 6 in 2024-02-05T08:03:16.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:16 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:17.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:17 smithi134 ceph-mon[30086]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-02-05T08:03:17.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:17 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/158572409' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:03:17.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:17 smithi134 ceph-mon[39278]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-02-05T08:03:17.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:17 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/158572409' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:03:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:17 smithi196 ceph-mon[33018]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-02-05T08:03:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:17 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/158572409' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:03:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:19 smithi196 ceph-mon[33018]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-02-05T08:03:20.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:19 smithi134 ceph-mon[30086]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-02-05T08:03:20.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:19 smithi134 ceph-mon[39278]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-02-05T08:03:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:20 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:03:21.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:20 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:03:21.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:20 smithi196 ceph-mon[33018]: osdmap e39: 6 total, 5 up, 6 in 2024-02-05T08:03:21.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:20 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:20 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:03:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:20 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:03:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:20 smithi134 ceph-mon[30086]: osdmap e39: 6 total, 5 up, 6 in 2024-02-05T08:03:21.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:20 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:20 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:03:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:20 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:03:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:20 smithi134 ceph-mon[39278]: osdmap e39: 6 total, 5 up, 6 in 2024-02-05T08:03:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:20 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:22.123 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:21 smithi196 ceph-mon[33018]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-02-05T08:03:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:21 smithi134 ceph-mon[39278]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-02-05T08:03:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:21 smithi134 ceph-mon[30086]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-02-05T08:03:24.129 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:23 smithi196 ceph-mon[33018]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:24.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:23 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-05T08:03:24.146 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:23 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:23 smithi134 ceph-mon[30086]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:23 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-05T08:03:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:23 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:23 smithi134 ceph-mon[39278]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:23 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-05T08:03:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:23 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:25.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:24 smithi196 ceph-mon[33018]: Deploying daemon osd.5 on smithi196 2024-02-05T08:03:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:24 smithi134 ceph-mon[30086]: Deploying daemon osd.5 on smithi196 2024-02-05T08:03:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:24 smithi134 ceph-mon[39278]: Deploying daemon osd.5 on smithi196 2024-02-05T08:03:25.882 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:25 smithi196 ceph-mon[33018]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:25 smithi134 ceph-mon[30086]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:25 smithi134 ceph-mon[39278]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:27.912 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:27 smithi196 ceph-mon[33018]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:27 smithi134 ceph-mon[30086]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:28.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:27 smithi134 ceph-mon[39278]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:29 smithi196 ceph-mon[33018]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:30.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:30.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:30.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:29 smithi134 ceph-mon[30086]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:30.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:29 smithi134 ceph-mon[39278]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:31.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:31 smithi196 ceph-mon[33018]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:31 smithi134 ceph-mon[30086]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:32.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:31 smithi134 ceph-mon[39278]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:32.470 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 5 on host 'smithi196' 2024-02-05T08:03:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:32 smithi196 ceph-mon[33018]: from='osd.5 [v2:172.21.15.196:6808/1603224792,v1:172.21.15.196:6809/1603224792]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T08:03:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:32 smithi196 ceph-mon[33018]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T08:03:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:32 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:33.133 DEBUG:teuthology.orchestra.run.smithi196:osd.5> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.5.service 2024-02-05T08:03:33.136 INFO:tasks.cephadm:Deploying osd.6 on smithi196 with /dev/vg_nvme/lv_2... 2024-02-05T08:03:33.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-05T08:03:33.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:32 smithi134 ceph-mon[30086]: from='osd.5 [v2:172.21.15.196:6808/1603224792,v1:172.21.15.196:6809/1603224792]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T08:03:33.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:32 smithi134 ceph-mon[30086]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T08:03:33.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:33.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:33.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:33.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:32 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:33.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:32 smithi134 ceph-mon[39278]: from='osd.5 [v2:172.21.15.196:6808/1603224792,v1:172.21.15.196:6809/1603224792]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T08:03:33.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:32 smithi134 ceph-mon[39278]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T08:03:33.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:33.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:33.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:33.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:32 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:33.883 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:03:34.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:33 smithi196 ceph-mon[33018]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:34.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:33 smithi196 ceph-mon[33018]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-05T08:03:34.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:33 smithi196 ceph-mon[33018]: from='osd.5 [v2:172.21.15.196:6808/1603224792,v1:172.21.15.196:6809/1603224792]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:03:34.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:33 smithi196 ceph-mon[33018]: osdmap e40: 6 total, 5 up, 6 in 2024-02-05T08:03:34.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:33 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:34.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:33 smithi196 ceph-mon[33018]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:03:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:33 smithi134 ceph-mon[30086]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:33 smithi134 ceph-mon[30086]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-05T08:03:34.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:33 smithi134 ceph-mon[30086]: from='osd.5 [v2:172.21.15.196:6808/1603224792,v1:172.21.15.196:6809/1603224792]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:03:34.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:33 smithi134 ceph-mon[30086]: osdmap e40: 6 total, 5 up, 6 in 2024-02-05T08:03:34.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:33 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:34.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:33 smithi134 ceph-mon[30086]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:03:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:33 smithi134 ceph-mon[39278]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:33 smithi134 ceph-mon[39278]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-05T08:03:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:33 smithi134 ceph-mon[39278]: from='osd.5 [v2:172.21.15.196:6808/1603224792,v1:172.21.15.196:6809/1603224792]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:03:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:33 smithi134 ceph-mon[39278]: osdmap e40: 6 total, 5 up, 6 in 2024-02-05T08:03:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:33 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:33 smithi134 ceph-mon[39278]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:03:34.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:03:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:03:34.296+0000 7f84d308e700 -1 osd.5 0 waiting for initial osdmap 2024-02-05T08:03:34.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:03:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:03:34.303+0000 7f84cae80700 -1 osd.5 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T08:03:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:34 smithi196 ceph-mon[33018]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-02-05T08:03:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:34 smithi196 ceph-mon[33018]: osdmap e41: 6 total, 5 up, 6 in 2024-02-05T08:03:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:34 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:34 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:34 smithi196 ceph-mon[33018]: from='osd.5 ' entity='osd.5' 2024-02-05T08:03:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:34 smithi196 ceph-mon[33018]: osd.5 [v2:172.21.15.196:6808/1603224792,v1:172.21.15.196:6809/1603224792] boot 2024-02-05T08:03:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:34 smithi196 ceph-mon[33018]: osdmap e42: 6 total, 6 up, 6 in 2024-02-05T08:03:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:34 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[30086]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-02-05T08:03:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[30086]: osdmap e41: 6 total, 5 up, 6 in 2024-02-05T08:03:35.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:35.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:35.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[30086]: from='osd.5 ' entity='osd.5' 2024-02-05T08:03:35.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[30086]: osd.5 [v2:172.21.15.196:6808/1603224792,v1:172.21.15.196:6809/1603224792] boot 2024-02-05T08:03:35.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[30086]: osdmap e42: 6 total, 6 up, 6 in 2024-02-05T08:03:35.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[39278]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-02-05T08:03:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[39278]: osdmap e41: 6 total, 5 up, 6 in 2024-02-05T08:03:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:35.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[39278]: from='osd.5 ' entity='osd.5' 2024-02-05T08:03:35.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[39278]: osd.5 [v2:172.21.15.196:6808/1603224792,v1:172.21.15.196:6809/1603224792] boot 2024-02-05T08:03:35.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[39278]: osdmap e42: 6 total, 6 up, 6 in 2024-02-05T08:03:35.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:34 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:03:36.060 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:35 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:03:36.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:35 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:03:36.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:35 smithi196 ceph-mon[33018]: pgmap v138: 1 pgs: 1 unknown; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:36.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:35 smithi196 ceph-mon[33018]: osdmap e43: 6 total, 6 up, 6 in 2024-02-05T08:03:36.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:35 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:03:36.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:35 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:03:36.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:35 smithi134 ceph-mon[30086]: pgmap v138: 1 pgs: 1 unknown; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:36.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:35 smithi134 ceph-mon[30086]: osdmap e43: 6 total, 6 up, 6 in 2024-02-05T08:03:36.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:35 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:03:36.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:35 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:03:36.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:35 smithi134 ceph-mon[39278]: pgmap v138: 1 pgs: 1 unknown; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-02-05T08:03:36.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:35 smithi134 ceph-mon[39278]: osdmap e43: 6 total, 6 up, 6 in 2024-02-05T08:03:36.662 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-05T08:03:36.662 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-05T08:03:36.662 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-02-05T08:03:36.663 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-02-05T08:03:36.663 INFO:teuthology.orchestra.run.smithi196.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138709 s, 756 MB/s 2024-02-05T08:03:36.663 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-02-05T08:03:37.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi196:vg_nvme/lv_2 2024-02-05T08:03:37.702 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:37 smithi196 ceph-mon[33018]: osdmap e44: 6 total, 6 up, 6 in 2024-02-05T08:03:37.702 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:37 smithi196 ceph-mon[33018]: pgmap v141: 1 pgs: 1 unknown; 577 KiB data, 342 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:37.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:37 smithi134 ceph-mon[30086]: osdmap e44: 6 total, 6 up, 6 in 2024-02-05T08:03:37.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:37 smithi134 ceph-mon[30086]: pgmap v141: 1 pgs: 1 unknown; 577 KiB data, 342 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:37.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:37 smithi134 ceph-mon[39278]: osdmap e44: 6 total, 6 up, 6 in 2024-02-05T08:03:37.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:37 smithi134 ceph-mon[39278]: pgmap v141: 1 pgs: 1 unknown; 577 KiB data, 342 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:38.026 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:03:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:38 smithi196 ceph-mon[33018]: osdmap e45: 6 total, 6 up, 6 in 2024-02-05T08:03:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:38 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:38 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:38 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:03:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:38 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:03:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:38 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:38 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:38 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:03:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:38 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:38 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:38 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:38 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:03:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:38 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[30086]: osdmap e45: 6 total, 6 up, 6 in 2024-02-05T08:03:38.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:03:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:03:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:03:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:03:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[39278]: osdmap e45: 6 total, 6 up, 6 in 2024-02-05T08:03:38.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:03:38.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:03:38.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:38.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:03:38.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:38.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:38.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:38.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:03:38.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:38 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:39.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:39 smithi134 ceph-mon[39278]: Detected new or changed devices on smithi196 2024-02-05T08:03:39.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:39 smithi134 ceph-mon[39278]: Adjusting osd_memory_target on smithi196 to 8544M 2024-02-05T08:03:39.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:39 smithi134 ceph-mon[39278]: osdmap e46: 6 total, 6 up, 6 in 2024-02-05T08:03:39.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:39 smithi134 ceph-mon[39278]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 342 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:39 smithi134 ceph-mon[30086]: Detected new or changed devices on smithi196 2024-02-05T08:03:39.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:39 smithi134 ceph-mon[30086]: Adjusting osd_memory_target on smithi196 to 8544M 2024-02-05T08:03:39.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:39 smithi134 ceph-mon[30086]: osdmap e46: 6 total, 6 up, 6 in 2024-02-05T08:03:39.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:39 smithi134 ceph-mon[30086]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 342 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:39 smithi196 ceph-mon[33018]: Detected new or changed devices on smithi196 2024-02-05T08:03:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:39 smithi196 ceph-mon[33018]: Adjusting osd_memory_target on smithi196 to 8544M 2024-02-05T08:03:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:39 smithi196 ceph-mon[33018]: osdmap e46: 6 total, 6 up, 6 in 2024-02-05T08:03:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:39 smithi196 ceph-mon[33018]: pgmap v144: 1 pgs: 1 unknown; 577 KiB data, 342 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:40.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:40 smithi196 ceph-mon[33018]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:03:40.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:40 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:03:40.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:40 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:03:40.627 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:40 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:40.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:40 smithi134 ceph-mon[30086]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:03:40.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:40 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:03:40.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:40 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:03:40.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:40 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:40.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:40 smithi134 ceph-mon[39278]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:03:40.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:40 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:03:40.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:40 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:03:40.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:40 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:41.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:41 smithi196 ceph-mon[33018]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-02-05T08:03:41.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:41 smithi134 ceph-mon[30086]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-02-05T08:03:41.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:41 smithi134 ceph-mon[39278]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-02-05T08:03:43.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:42 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/3416872430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a"}]: dispatch 2024-02-05T08:03:43.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:42 smithi134 ceph-mon[30086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a"}]: dispatch 2024-02-05T08:03:43.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:42 smithi134 ceph-mon[30086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a"}]': finished 2024-02-05T08:03:43.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:42 smithi134 ceph-mon[30086]: osdmap e47: 7 total, 6 up, 7 in 2024-02-05T08:03:43.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:42 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:03:43.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:42 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/3416872430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a"}]: dispatch 2024-02-05T08:03:43.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:42 smithi134 ceph-mon[39278]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a"}]: dispatch 2024-02-05T08:03:43.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:42 smithi134 ceph-mon[39278]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a"}]': finished 2024-02-05T08:03:43.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:42 smithi134 ceph-mon[39278]: osdmap e47: 7 total, 6 up, 7 in 2024-02-05T08:03:43.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:42 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:03:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:42 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/3416872430' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a"}]: dispatch 2024-02-05T08:03:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:42 smithi196 ceph-mon[33018]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a"}]: dispatch 2024-02-05T08:03:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:42 smithi196 ceph-mon[33018]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a"}]': finished 2024-02-05T08:03:43.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:42 smithi196 ceph-mon[33018]: osdmap e47: 7 total, 6 up, 7 in 2024-02-05T08:03:43.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:42 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:03:43.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:43 smithi196 ceph-mon[33018]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T08:03:43.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:43 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/2664417596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:03:44.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:43 smithi134 ceph-mon[30086]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T08:03:44.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:43 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/2664417596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:03:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:43 smithi134 ceph-mon[39278]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T08:03:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:43 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/2664417596' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:03:45.773 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:45 smithi196 ceph-mon[33018]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-02-05T08:03:45.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:45 smithi134 ceph-mon[30086]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-02-05T08:03:45.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:45 smithi134 ceph-mon[39278]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-02-05T08:03:47.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:47 smithi196 ceph-mon[33018]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-02-05T08:03:48.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:47 smithi134 ceph-mon[30086]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-02-05T08:03:48.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:47 smithi134 ceph-mon[39278]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-02-05T08:03:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:49 smithi196 ceph-mon[33018]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T08:03:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:49 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-05T08:03:50.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:49 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:50.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:49 smithi134 ceph-mon[30086]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T08:03:50.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:49 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-05T08:03:50.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:49 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:50.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:49 smithi134 ceph-mon[39278]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T08:03:50.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:49 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-05T08:03:50.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:49 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:03:50.925 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:50 smithi196 ceph-mon[33018]: 1.0 scrub starts 2024-02-05T08:03:50.925 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:50 smithi196 ceph-mon[33018]: 1.0 scrub ok 2024-02-05T08:03:50.925 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:50 smithi196 ceph-mon[33018]: Deploying daemon osd.6 on smithi196 2024-02-05T08:03:50.925 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:50 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:03:50.925 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:50 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:03:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:50 smithi134 ceph-mon[30086]: 1.0 scrub starts 2024-02-05T08:03:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:50 smithi134 ceph-mon[30086]: 1.0 scrub ok 2024-02-05T08:03:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:50 smithi134 ceph-mon[30086]: Deploying daemon osd.6 on smithi196 2024-02-05T08:03:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:50 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:03:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:50 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:03:51.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:50 smithi134 ceph-mon[39278]: 1.0 scrub starts 2024-02-05T08:03:51.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:50 smithi134 ceph-mon[39278]: 1.0 scrub ok 2024-02-05T08:03:51.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:50 smithi134 ceph-mon[39278]: Deploying daemon osd.6 on smithi196 2024-02-05T08:03:51.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:50 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:03:51.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:50 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:03:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:51 smithi134 ceph-mon[30086]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:51 smithi134 ceph-mon[30086]: osdmap e48: 7 total, 6 up, 7 in 2024-02-05T08:03:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:51 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:03:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:51 smithi134 ceph-mon[39278]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:51 smithi134 ceph-mon[39278]: osdmap e48: 7 total, 6 up, 7 in 2024-02-05T08:03:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:51 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:03:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:51 smithi196 ceph-mon[33018]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:52.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:51 smithi196 ceph-mon[33018]: osdmap e48: 7 total, 6 up, 7 in 2024-02-05T08:03:52.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:51 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:03:54.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:53 smithi196 ceph-mon[33018]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:53 smithi134 ceph-mon[30086]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:54.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:53 smithi134 ceph-mon[39278]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:55.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:55 smithi196 ceph-mon[33018]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:55.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:55.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:55.574 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:55.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:55 smithi134 ceph-mon[30086]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:55.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:55.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:55.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:55.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:55 smithi134 ceph-mon[39278]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:55.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:03:55.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:55.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:58.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:57 smithi196 ceph-mon[33018]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:58.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:57 smithi134 ceph-mon[30086]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:58.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:57 smithi134 ceph-mon[39278]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:03:59.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:58 smithi196 ceph-mon[33018]: from='osd.6 [v2:172.21.15.196:6816/3727413143,v1:172.21.15.196:6817/3727413143]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T08:03:59.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:58 smithi196 ceph-mon[33018]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T08:03:59.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:58 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:59.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:58 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:59.091 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 6 on host 'smithi196' 2024-02-05T08:03:59.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:58 smithi134 ceph-mon[30086]: from='osd.6 [v2:172.21.15.196:6816/3727413143,v1:172.21.15.196:6817/3727413143]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T08:03:59.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:58 smithi134 ceph-mon[30086]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T08:03:59.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:58 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:59.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:58 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:59.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:58 smithi134 ceph-mon[39278]: from='osd.6 [v2:172.21.15.196:6816/3727413143,v1:172.21.15.196:6817/3727413143]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T08:03:59.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:58 smithi134 ceph-mon[39278]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T08:03:59.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:58 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:59.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:58 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:03:59.727 DEBUG:teuthology.orchestra.run.smithi196:osd.6> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.6.service 2024-02-05T08:03:59.729 INFO:tasks.cephadm:Deploying osd.7 on smithi196 with /dev/vg_nvme/lv_1... 2024-02-05T08:03:59.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-05T08:04:00.002 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:03:59.833+0000 7fd817300700 -1 osd.6 0 waiting for initial osdmap 2024-02-05T08:04:00.002 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:03:59.839+0000 7fd8130fa700 -1 osd.6 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T08:04:00.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-mon[33018]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:04:00.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-mon[33018]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-05T08:04:00.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-mon[33018]: osdmap e49: 7 total, 6 up, 7 in 2024-02-05T08:04:00.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-mon[33018]: from='osd.6 [v2:172.21.15.196:6816/3727413143,v1:172.21.15.196:6817/3727413143]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:04:00.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:00.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-mon[33018]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:04:00.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:00.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:00.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:00.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:00.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:00.003 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:03:59 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:00.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[39278]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:04:00.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[39278]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-05T08:04:00.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[39278]: osdmap e49: 7 total, 6 up, 7 in 2024-02-05T08:04:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[39278]: from='osd.6 [v2:172.21.15.196:6816/3727413143,v1:172.21.15.196:6817/3727413143]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:04:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[39278]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:04:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:00.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[30086]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:04:00.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[30086]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-05T08:04:00.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[30086]: osdmap e49: 7 total, 6 up, 7 in 2024-02-05T08:04:00.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[30086]: from='osd.6 [v2:172.21.15.196:6816/3727413143,v1:172.21.15.196:6817/3727413143]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:04:00.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:00.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[30086]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:04:00.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:00.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:00.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:00.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:00.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:00.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:03:59 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:00.432 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:04:01.111 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:00 smithi196 ceph-mon[33018]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-02-05T08:04:01.111 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:00 smithi196 ceph-mon[33018]: osdmap e50: 7 total, 6 up, 7 in 2024-02-05T08:04:01.111 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:00 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:01.111 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:00 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:01.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:00 smithi134 ceph-mon[30086]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-02-05T08:04:01.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:00 smithi134 ceph-mon[30086]: osdmap e50: 7 total, 6 up, 7 in 2024-02-05T08:04:01.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:00 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:01.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:00 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:01.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:00 smithi134 ceph-mon[39278]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-02-05T08:04:01.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:00 smithi134 ceph-mon[39278]: osdmap e50: 7 total, 6 up, 7 in 2024-02-05T08:04:01.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:00 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:01.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:00 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:04:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:04:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[30086]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:04:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[30086]: osd.6 [v2:172.21.15.196:6816/3727413143,v1:172.21.15.196:6817/3727413143] boot 2024-02-05T08:04:02.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[30086]: osdmap e51: 7 total, 7 up, 7 in 2024-02-05T08:04:02.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:04:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:04:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[39278]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:04:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[39278]: osd.6 [v2:172.21.15.196:6816/3727413143,v1:172.21.15.196:6817/3727413143] boot 2024-02-05T08:04:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[39278]: osdmap e51: 7 total, 7 up, 7 in 2024-02-05T08:04:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:01 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:01 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:04:02.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:01 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:04:02.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:01 smithi196 ceph-mon[33018]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-02-05T08:04:02.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:01 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:02.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:01 smithi196 ceph-mon[33018]: osd.6 [v2:172.21.15.196:6816/3727413143,v1:172.21.15.196:6817/3727413143] boot 2024-02-05T08:04:02.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:01 smithi196 ceph-mon[33018]: osdmap e51: 7 total, 7 up, 7 in 2024-02-05T08:04:02.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:01 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:04:03.014 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:02 smithi196 ceph-mon[33018]: osdmap e52: 7 total, 7 up, 7 in 2024-02-05T08:04:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:02 smithi134 ceph-mon[30086]: osdmap e52: 7 total, 7 up, 7 in 2024-02-05T08:04:03.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:02 smithi134 ceph-mon[39278]: osdmap e52: 7 total, 7 up, 7 in 2024-02-05T08:04:03.457 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-05T08:04:03.457 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-05T08:04:03.458 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-02-05T08:04:03.458 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-02-05T08:04:03.458 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123707 s, 848 MB/s 2024-02-05T08:04:03.458 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-02-05T08:04:04.220 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:03 smithi196 ceph-mon[33018]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:04.220 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:03 smithi196 ceph-mon[33018]: osdmap e53: 7 total, 7 up, 7 in 2024-02-05T08:04:04.221 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:03 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:04.221 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:03 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:04.221 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:03 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:04.221 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:03 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:04.221 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:03 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:04.221 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:03 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:04.221 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:03 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:04.221 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:03 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:04.221 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:03 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:04.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi196:vg_nvme/lv_1 2024-02-05T08:04:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[30086]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[30086]: osdmap e53: 7 total, 7 up, 7 in 2024-02-05T08:04:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:04.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:04.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:04.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:04.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:04.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[39278]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[39278]: osdmap e53: 7 total, 7 up, 7 in 2024-02-05T08:04:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:04.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:04.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:04.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:04.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:03 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:04.481 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:04:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:04 smithi134 ceph-mon[30086]: Detected new or changed devices on smithi196 2024-02-05T08:04:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:04 smithi134 ceph-mon[30086]: Adjusting osd_memory_target on smithi196 to 5696M 2024-02-05T08:04:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:04 smithi134 ceph-mon[30086]: osdmap e54: 7 total, 7 up, 7 in 2024-02-05T08:04:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:04 smithi134 ceph-mon[39278]: Detected new or changed devices on smithi196 2024-02-05T08:04:05.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:04 smithi134 ceph-mon[39278]: Adjusting osd_memory_target on smithi196 to 5696M 2024-02-05T08:04:05.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:04 smithi134 ceph-mon[39278]: osdmap e54: 7 total, 7 up, 7 in 2024-02-05T08:04:05.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:04 smithi196 ceph-mon[33018]: Detected new or changed devices on smithi196 2024-02-05T08:04:05.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:04 smithi196 ceph-mon[33018]: Adjusting osd_memory_target on smithi196 to 5696M 2024-02-05T08:04:05.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:04 smithi196 ceph-mon[33018]: osdmap e54: 7 total, 7 up, 7 in 2024-02-05T08:04:06.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:05 smithi134 ceph-mon[30086]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:06.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:05 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:04:06.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:05 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:04:06.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:05 smithi134 ceph-mon[30086]: osdmap e55: 7 total, 7 up, 7 in 2024-02-05T08:04:06.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:05 smithi134 ceph-mon[39278]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:06.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:05 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:04:06.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:05 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:04:06.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:05 smithi134 ceph-mon[39278]: osdmap e55: 7 total, 7 up, 7 in 2024-02-05T08:04:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:05 smithi196 ceph-mon[33018]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:05 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:04:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:05 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:04:06.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:05 smithi196 ceph-mon[33018]: osdmap e55: 7 total, 7 up, 7 in 2024-02-05T08:04:07.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:04:07.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:04:07.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:06 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:04:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:04:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:06 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:07.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T08:04:07.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T08:04:07.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:06 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:08.028 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:07 smithi196 ceph-mon[33018]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:08.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:07 smithi196 ceph-mon[33018]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:07 smithi134 ceph-mon[30086]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:07 smithi134 ceph-mon[30086]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:08.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:07 smithi134 ceph-mon[39278]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:07 smithi134 ceph-mon[39278]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:10.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[30086]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T08:04:10.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/1864036729' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057"}]: dispatch 2024-02-05T08:04:10.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[30086]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057"}]: dispatch 2024-02-05T08:04:10.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[30086]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057"}]': finished 2024-02-05T08:04:10.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[30086]: osdmap e56: 8 total, 7 up, 8 in 2024-02-05T08:04:10.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:10.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/2083845091' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:04:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[39278]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T08:04:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/1864036729' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057"}]: dispatch 2024-02-05T08:04:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[39278]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057"}]: dispatch 2024-02-05T08:04:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[39278]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057"}]': finished 2024-02-05T08:04:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[39278]: osdmap e56: 8 total, 7 up, 8 in 2024-02-05T08:04:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:10.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:09 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/2083845091' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:04:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:09 smithi196 ceph-mon[33018]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T08:04:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:09 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/1864036729' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057"}]: dispatch 2024-02-05T08:04:10.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:09 smithi196 ceph-mon[33018]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057"}]: dispatch 2024-02-05T08:04:10.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:09 smithi196 ceph-mon[33018]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057"}]': finished 2024-02-05T08:04:10.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:09 smithi196 ceph-mon[33018]: osdmap e56: 8 total, 7 up, 8 in 2024-02-05T08:04:10.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:09 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:10.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:09 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/2083845091' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T08:04:12.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:11 smithi134 ceph-mon[30086]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-02-05T08:04:12.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:11 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:04:12.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:11 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:04:12.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:11 smithi134 ceph-mon[39278]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-02-05T08:04:12.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:11 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:04:12.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:11 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:04:12.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:11 smithi196 ceph-mon[33018]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-02-05T08:04:12.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:11 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:04:12.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:11 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:04:13.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:12 smithi134 ceph-mon[30086]: osdmap e57: 8 total, 7 up, 8 in 2024-02-05T08:04:13.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:12 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:13.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:12 smithi134 ceph-mon[39278]: osdmap e57: 8 total, 7 up, 8 in 2024-02-05T08:04:13.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:12 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:12 smithi196 ceph-mon[33018]: osdmap e57: 8 total, 7 up, 8 in 2024-02-05T08:04:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:12 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:14.265 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:13 smithi196 ceph-mon[33018]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-02-05T08:04:14.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:13 smithi134 ceph-mon[30086]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-02-05T08:04:14.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:13 smithi134 ceph-mon[39278]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-02-05T08:04:15.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:15 smithi134 ceph-mon[30086]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T08:04:15.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:15 smithi134 ceph-mon[39278]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T08:04:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:15 smithi196 ceph-mon[33018]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T08:04:16.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:16 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-05T08:04:16.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:16 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:16.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:16 smithi134 ceph-mon[30086]: Deploying daemon osd.7 on smithi196 2024-02-05T08:04:16.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:16 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-05T08:04:16.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:16 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:16.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:16 smithi134 ceph-mon[39278]: Deploying daemon osd.7 on smithi196 2024-02-05T08:04:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:16 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-05T08:04:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:16 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:16 smithi196 ceph-mon[33018]: Deploying daemon osd.7 on smithi196 2024-02-05T08:04:17.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:17 smithi134 ceph-mon[30086]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:17.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:17 smithi134 ceph-mon[39278]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:17.787 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:17 smithi196 ceph-mon[33018]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:19 smithi196 ceph-mon[33018]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:20.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:19 smithi134 ceph-mon[30086]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:20.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:19 smithi134 ceph-mon[39278]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:22.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:21 smithi196 ceph-mon[33018]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:22.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:21 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:22.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:21 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:22.022 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:21 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:21 smithi134 ceph-mon[30086]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:21 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:21 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:21 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:22.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:21 smithi134 ceph-mon[39278]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:22.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:21 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:22.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:21 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:22.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:21 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:24.132 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:23 smithi196 ceph-mon[33018]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:23 smithi134 ceph-mon[30086]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:24.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:23 smithi134 ceph-mon[39278]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:25.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:24 smithi196 ceph-mon[33018]: from='osd.7 [v2:172.21.15.196:6824/4206571010,v1:172.21.15.196:6825/4206571010]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T08:04:25.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:24 smithi196 ceph-mon[33018]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T08:04:25.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:24 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:25.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:24 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:25.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:24 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:25.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:24 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:25.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:24 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[30086]: from='osd.7 [v2:172.21.15.196:6824/4206571010,v1:172.21.15.196:6825/4206571010]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T08:04:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[30086]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T08:04:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:25.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:25.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:25.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:25.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:25.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[39278]: from='osd.7 [v2:172.21.15.196:6824/4206571010,v1:172.21.15.196:6825/4206571010]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T08:04:25.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[39278]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T08:04:25.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:25.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:25.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:25.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:25.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:24 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:25.451 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 7 on host 'smithi196' 2024-02-05T08:04:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:25 smithi196 ceph-mon[33018]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:25 smithi196 ceph-mon[33018]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-05T08:04:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:25 smithi196 ceph-mon[33018]: osdmap e58: 8 total, 7 up, 8 in 2024-02-05T08:04:26.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:25 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:26.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:25 smithi196 ceph-mon[33018]: from='osd.7 [v2:172.21.15.196:6824/4206571010,v1:172.21.15.196:6825/4206571010]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:04:26.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:25 smithi196 ceph-mon[33018]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:04:26.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:25 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:26.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:25 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:26.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:25 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:26.121 DEBUG:teuthology.orchestra.run.smithi196:osd.7> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.7.service 2024-02-05T08:04:26.124 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-05T08:04:26.124 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd stat -f json 2024-02-05T08:04:26.154 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[30086]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:26.154 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[30086]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-05T08:04:26.154 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[30086]: osdmap e58: 8 total, 7 up, 8 in 2024-02-05T08:04:26.155 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:26.155 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[30086]: from='osd.7 [v2:172.21.15.196:6824/4206571010,v1:172.21.15.196:6825/4206571010]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:04:26.155 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[30086]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:04:26.155 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:26.155 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:26.155 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:26.155 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[39278]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:26.155 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[39278]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-05T08:04:26.155 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[39278]: osdmap e58: 8 total, 7 up, 8 in 2024-02-05T08:04:26.155 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:26.155 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[39278]: from='osd.7 [v2:172.21.15.196:6824/4206571010,v1:172.21.15.196:6825/4206571010]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:04:26.156 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[39278]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-02-05T08:04:26.156 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:26.156 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:26.156 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:25 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:26.375 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:04:26 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:04:26.256+0000 7f5cedbc1700 -1 osd.7 0 waiting for initial osdmap 2024-02-05T08:04:26.375 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:04:26 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:04:26.263+0000 7f5ce59b3700 -1 osd.7 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T08:04:26.375 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:04:27.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:26 smithi134 ceph-mon[30086]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-02-05T08:04:27.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:26 smithi134 ceph-mon[30086]: osdmap e59: 8 total, 7 up, 8 in 2024-02-05T08:04:27.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:26 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:27.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:26 smithi134 ceph-mon[30086]: from='osd.7 ' entity='osd.7' 2024-02-05T08:04:27.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:26 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:27.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:26 smithi134 ceph-mon[39278]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-02-05T08:04:27.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:26 smithi134 ceph-mon[39278]: osdmap e59: 8 total, 7 up, 8 in 2024-02-05T08:04:27.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:26 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:27.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:26 smithi134 ceph-mon[39278]: from='osd.7 ' entity='osd.7' 2024-02-05T08:04:27.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:26 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:26 smithi196 ceph-mon[33018]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-02-05T08:04:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:26 smithi196 ceph-mon[33018]: osdmap e59: 8 total, 7 up, 8 in 2024-02-05T08:04:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:26 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:26 smithi196 ceph-mon[33018]: from='osd.7 ' entity='osd.7' 2024-02-05T08:04:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:26 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:28.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:27 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:04:28.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:27 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:04:28.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:27 smithi196 ceph-mon[33018]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:28.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:27 smithi196 ceph-mon[33018]: osd.7 [v2:172.21.15.196:6824/4206571010,v1:172.21.15.196:6825/4206571010] boot 2024-02-05T08:04:28.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:27 smithi196 ceph-mon[33018]: osdmap e60: 8 total, 8 up, 8 in 2024-02-05T08:04:28.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:27 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:28.220 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:04:28.220 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:27 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:04:28.221 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:27 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:04:28.221 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:27 smithi134 ceph-mon[30086]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:28.221 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:27 smithi134 ceph-mon[30086]: osd.7 [v2:172.21.15.196:6824/4206571010,v1:172.21.15.196:6825/4206571010] boot 2024-02-05T08:04:28.221 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:27 smithi134 ceph-mon[30086]: osdmap e60: 8 total, 8 up, 8 in 2024-02-05T08:04:28.221 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:27 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:28.221 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:27 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:04:28.221 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:27 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:04:28.221 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:27 smithi134 ceph-mon[39278]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:04:28.221 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:27 smithi134 ceph-mon[39278]: osd.7 [v2:172.21.15.196:6824/4206571010,v1:172.21.15.196:6825/4206571010] boot 2024-02-05T08:04:28.221 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:27 smithi134 ceph-mon[39278]: osdmap e60: 8 total, 8 up, 8 in 2024-02-05T08:04:28.221 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:27 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:04:28.895 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":61,"num_osds":8,"num_up_osds":8,"osd_up_since":1707120266,"num_in_osds":8,"osd_in_since":1707120249,"num_remapped_pgs":1} 2024-02-05T08:04:28.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T08:04:29.172 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:04:29.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:28 smithi134 ceph-mon[30086]: osdmap e61: 8 total, 8 up, 8 in 2024-02-05T08:04:29.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:28 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2728554454' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-05T08:04:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:28 smithi134 ceph-mon[39278]: osdmap e61: 8 total, 8 up, 8 in 2024-02-05T08:04:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:28 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2728554454' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-05T08:04:29.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:28 smithi196 ceph-mon[33018]: osdmap e61: 8 total, 8 up, 8 in 2024-02-05T08:04:29.312 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:28 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2728554454' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-05T08:04:30.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:04:30.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:04:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:04:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: osdmap e62: 8 total, 8 up, 8 in 2024-02-05T08:04:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[39278]: osdmap e63: 8 total, 8 up, 8 in 2024-02-05T08:04:30.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:04:30.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:04:30.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:04:30.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: osdmap e62: 8 total, 8 up, 8 in 2024-02-05T08:04:30.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:30.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:30.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:30.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:30.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:30.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:30.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:30.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:30.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:30.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:30.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:29 smithi134 ceph-mon[30086]: osdmap e63: 8 total, 8 up, 8 in 2024-02-05T08:04:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:04:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:04:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:04:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: osdmap e62: 8 total, 8 up, 8 in 2024-02-05T08:04:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:04:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:29 smithi196 ceph-mon[33018]: osdmap e63: 8 total, 8 up, 8 in 2024-02-05T08:04:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:30 smithi134 ceph-mon[30086]: Detected new or changed devices on smithi196 2024-02-05T08:04:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:30 smithi134 ceph-mon[30086]: Adjusting osd_memory_target on smithi196 to 4272M 2024-02-05T08:04:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:30 smithi134 ceph-mon[30086]: osdmap e64: 8 total, 8 up, 8 in 2024-02-05T08:04:31.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:30 smithi134 ceph-mon[39278]: Detected new or changed devices on smithi196 2024-02-05T08:04:31.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:30 smithi134 ceph-mon[39278]: Adjusting osd_memory_target on smithi196 to 4272M 2024-02-05T08:04:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:30 smithi134 ceph-mon[39278]: osdmap e64: 8 total, 8 up, 8 in 2024-02-05T08:04:31.104 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:04:31.104 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":64,"fsid":"43e13aa4-c3fc-11ee-95b6-87774f69a715","created":"2024-02-05T07:59:09.321262+0000","modified":"2024-02-05T08:04:30.510337+0000","last_up_change":"2024-02-05T08:04:26.849889+0000","last_in_change":"2024-02-05T08:04:09.165031+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T08:02:21.034012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d90a5ea0-9a19-452e-9564-ca7b7a7c45e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6803","nonce":2540271081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6805","nonce":2540271081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6809","nonce":2540271081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6807","nonce":2540271081}]},"public_addr":"172.21.15.134:6803/2540271081","cluster_addr":"172.21.15.134:6805/2540271081","heartbeat_back_addr":"172.21.15.134:6809/2540271081","heartbeat_front_addr":"172.21.15.134:6807/2540271081","state":["exists","up"]},{"osd":1,"uuid":"4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":2032802960},{"type":"v1","addr":"172.21.15.134:6811","nonce":2032802960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":2032802960},{"type":"v1","addr":"172.21.15.134:6813","nonce":2032802960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2032802960},{"type":"v1","addr":"172.21.15.134:6817","nonce":2032802960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":2032802960},{"type":"v1","addr":"172.21.15.134:6815","nonce":2032802960}]},"public_addr":"172.21.15.134:6811/2032802960","cluster_addr":"172.21.15.134:6813/2032802960","heartbeat_back_addr":"172.21.15.134:6817/2032802960","heartbeat_front_addr":"172.21.15.134:6815/2032802960","state":["exists","up"]},{"osd":2,"uuid":"078e578b-855b-49ff-83bd-0548a9c5a708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6819","nonce":153827987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6821","nonce":153827987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6825","nonce":153827987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6823","nonce":153827987}]},"public_addr":"172.21.15.134:6819/153827987","cluster_addr":"172.21.15.134:6821/153827987","heartbeat_back_addr":"172.21.15.134:6825/153827987","heartbeat_front_addr":"172.21.15.134:6823/153827987","state":["exists","up"]},{"osd":3,"uuid":"6774e330-3202-4671-894e-172cc2f6189d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6827","nonce":1527507056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6829","nonce":1527507056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6833","nonce":1527507056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6831","nonce":1527507056}]},"public_addr":"172.21.15.134:6827/1527507056","cluster_addr":"172.21.15.134:6829/1527507056","heartbeat_back_addr":"172.21.15.134:6833/1527507056","heartbeat_front_addr":"172.21.15.134:6831/1527507056","state":["exists","up"]},{"osd":4,"uuid":"52f70cfa-ab0e-4751-a5f8-1045c61c1693","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6801","nonce":3005156808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6803","nonce":3005156808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6807","nonce":3005156808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6805","nonce":3005156808}]},"public_addr":"172.21.15.196:6801/3005156808","cluster_addr":"172.21.15.196:6803/3005156808","heartbeat_back_addr":"172.21.15.196:6807/3005156808","heartbeat_front_addr":"172.21.15.196:6805/3005156808","state":["exists","up"]},{"osd":5,"uuid":"b4b21769-d40f-43fd-b7e9-2477b74d3955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6809","nonce":1603224792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6811","nonce":1603224792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6815","nonce":1603224792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6813","nonce":1603224792}]},"public_addr":"172.21.15.196:6809/1603224792","cluster_addr":"172.21.15.196:6811/1603224792","heartbeat_back_addr":"172.21.15.196:6815/1603224792","heartbeat_front_addr":"172.21.15.196:6813/1603224792","state":["exists","up"]},{"osd":6,"uuid":"af1cced4-0015-47f5-b105-27641bfe4f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6817","nonce":3727413143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6819","nonce":3727413143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6823","nonce":3727413143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6821","nonce":3727413143}]},"public_addr":"172.21.15.196:6817/3727413143","cluster_addr":"172.21.15.196:6819/3727413143","heartbeat_back_addr":"172.21.15.196:6823/3727413143","heartbeat_front_addr":"172.21.15.196:6821/3727413143","state":["exists","up"]},{"osd":7,"uuid":"c54beeae-9f7c-4723-a369-13bb531b4057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6825","nonce":4206571010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6827","nonce":4206571010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6831","nonce":4206571010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6829","nonce":4206571010}]},"public_addr":"172.21.15.196:6825/4206571010","cluster_addr":"172.21.15.196:6827/4206571010","heartbeat_back_addr":"172.21.15.196:6831/4206571010","heartbeat_front_addr":"172.21.15.196:6829/4206571010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:03:50.324792+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:28.076106+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:03:19.266508+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:04.796357+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:11.448792+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:03:32.966320+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:03:58.970755+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:25.455681+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3161126612":"2024-02-06T08:00:02.709280+0000","172.21.15.134:0/2952444615":"2024-02-06T08:00:02.709280+0000","172.21.15.134:6801/3607168099":"2024-02-06T08:00:02.709280+0000","172.21.15.134:0/4225724311":"2024-02-06T08:00:02.709280+0000","172.21.15.134:6801/673876301":"2024-02-06T07:59:36.275039+0000","172.21.15.134:6800/673876301":"2024-02-06T07:59:36.275039+0000","172.21.15.134:6800/3607168099":"2024-02-06T08:00:02.709280+0000","172.21.15.134:0/301941854":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/464208089":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/1046500326":"2024-02-06T07:59:36.275039+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T08:04:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:30 smithi196 ceph-mon[33018]: Detected new or changed devices on smithi196 2024-02-05T08:04:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:30 smithi196 ceph-mon[33018]: Adjusting osd_memory_target on smithi196 to 4272M 2024-02-05T08:04:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:30 smithi196 ceph-mon[33018]: osdmap e64: 8 total, 8 up, 8 in 2024-02-05T08:04:31.721 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T08:02:21.034012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-05T08:04:31.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-05T08:04:31.960 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:04:32.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:31 smithi134 ceph-mon[39278]: pgmap v188: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-02-05T08:04:32.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:31 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2511533473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:04:32.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:31 smithi134 ceph-mon[30086]: pgmap v188: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-02-05T08:04:32.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:31 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2511533473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:04:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:31 smithi196 ceph-mon[33018]: pgmap v188: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-02-05T08:04:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:31 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2511533473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:04:33.814 INFO:teuthology.orchestra.run.smithi134.stdout:pg_num: 1 2024-02-05T08:04:34.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:33 smithi134 ceph-mon[30086]: pgmap v189: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-02-05T08:04:34.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:33 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3753891611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-05T08:04:34.130 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:33 smithi134 ceph-mon[39278]: pgmap v189: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-02-05T08:04:34.131 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:33 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3753891611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-05T08:04:34.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:33 smithi196 ceph-mon[33018]: pgmap v189: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-02-05T08:04:34.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:33 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3753891611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-05T08:04:34.424 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi134 2024-02-05T08:04:34.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi134=foo.a' 2024-02-05T08:04:34.679 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:04:35.775 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:35 smithi196 ceph-mon[33018]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T08:04:35.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:35 smithi134 ceph-mon[30086]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T08:04:35.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:35 smithi134 ceph-mon[39278]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T08:04:36.495 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled rgw.foo.a update... 2024-02-05T08:04:37.100 DEBUG:teuthology.orchestra.run.smithi134:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@rgw.foo.a.service 2024-02-05T08:04:37.103 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi196 2024-02-05T08:04:37.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-02-05T08:04:37.340 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:04:37.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:37 smithi196 ceph-mon[33018]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi134=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:37.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:37 smithi196 ceph-mon[33018]: Saving service rgw.foo.a spec with placement smithi134=foo.a;count:1 2024-02-05T08:04:37.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:37 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:37.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:37 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:37.734 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:37 smithi196 ceph-mon[33018]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-02-05T08:04:37.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:37 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:37.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:37 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:37.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:37 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:37.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:37 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-05T08:04:37.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:37 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-05T08:04:37.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:37 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:37.735 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:37 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:37.774 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[30086]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi134=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:37.774 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[30086]: Saving service rgw.foo.a spec with placement smithi134=foo.a;count:1 2024-02-05T08:04:37.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:37.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:37.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[30086]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-02-05T08:04:37.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:37.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:37.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:37.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-05T08:04:37.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-05T08:04:37.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:37.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:37.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[39278]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi134=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:37.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[39278]: Saving service rgw.foo.a spec with placement smithi134=foo.a;count:1 2024-02-05T08:04:37.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:37.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:37.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[39278]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-02-05T08:04:37.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:37.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:37.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:37.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-05T08:04:37.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-05T08:04:37.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:37.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:37 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:38.753 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:38 smithi196 ceph-mon[33018]: Deploying daemon rgw.foo.a on smithi134 2024-02-05T08:04:38.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:38 smithi134 ceph-mon[30086]: Deploying daemon rgw.foo.a on smithi134 2024-02-05T08:04:38.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:38 smithi134 ceph-mon[39278]: Deploying daemon rgw.foo.a on smithi134 2024-02-05T08:04:39.491 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:04:39 smithi134 systemd[1]: Starting Ceph rgw.foo.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:04:39.510 INFO:teuthology.orchestra.run.smithi196.stderr:pool 'datapool' created 2024-02-05T08:04:39.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:39 smithi134 ceph-mon[39278]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:04:39.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:39 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/625664244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T08:04:39.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:39 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T08:04:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:39 smithi134 ceph-mon[30086]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:04:39.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:39 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/625664244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T08:04:39.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:39 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T08:04:39.778 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:04:39 smithi134 podman[68279]: 2024-02-05 08:04:39.490677685 +0000 UTC m=+0.108779762 container create 784ffda1aee4a1abe76ad3e7e2b0e689e88b540e70ce18ffeb18b273a2c10ef9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-rgw-foo-a, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-05T08:04:39.778 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:04:39 smithi134 podman[68279]: 2024-02-05 08:04:39.399728011 +0000 UTC m=+0.017830088 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:04:39.778 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:04:39 smithi134 podman[68279]: 2024-02-05 08:04:39.726787561 +0000 UTC m=+0.344889633 container init 784ffda1aee4a1abe76ad3e7e2b0e689e88b540e70ce18ffeb18b273a2c10ef9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD) 2024-02-05T08:04:39.778 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:04:39 smithi134 podman[68279]: 2024-02-05 08:04:39.731824543 +0000 UTC m=+0.349926616 container start 784ffda1aee4a1abe76ad3e7e2b0e689e88b540e70ce18ffeb18b273a2c10ef9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-rgw-foo-a, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131) 2024-02-05T08:04:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:39 smithi196 ceph-mon[33018]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:04:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:39 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/625664244' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T08:04:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:39 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T08:04:40.277 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:04:39 smithi134 bash[68279]: 784ffda1aee4a1abe76ad3e7e2b0e689e88b540e70ce18ffeb18b273a2c10ef9 2024-02-05T08:04:40.277 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:04:39 smithi134 systemd[1]: Started Ceph rgw.foo.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:04:40.288 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- rbd pool init datapool 2024-02-05T08:04:40.528 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:04:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:40 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-05T08:04:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:40 smithi196 ceph-mon[33018]: osdmap e65: 8 total, 8 up, 8 in 2024-02-05T08:04:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:40 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:40 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:40 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:40 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:40 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:40 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:40.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-05T08:04:40.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[30086]: osdmap e65: 8 total, 8 up, 8 in 2024-02-05T08:04:40.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:40.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-05T08:04:40.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[39278]: osdmap e65: 8 total, 8 up, 8 in 2024-02-05T08:04:40.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:40.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:40 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:41.616 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:41 smithi196 ceph-mon[33018]: Saving service rgw.foo.a spec with placement smithi134=foo.a;count:1 2024-02-05T08:04:41.616 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:41 smithi196 ceph-mon[33018]: osdmap e66: 8 total, 8 up, 8 in 2024-02-05T08:04:41.616 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:41 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4286562240' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T08:04:41.616 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:41 smithi196 ceph-mon[33018]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T08:04:41.616 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:41 smithi196 ceph-mon[33018]: pgmap v195: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:04:41.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:41 smithi134 ceph-mon[30086]: Saving service rgw.foo.a spec with placement smithi134=foo.a;count:1 2024-02-05T08:04:41.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:41 smithi134 ceph-mon[30086]: osdmap e66: 8 total, 8 up, 8 in 2024-02-05T08:04:41.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:41 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4286562240' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T08:04:41.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:41 smithi134 ceph-mon[30086]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T08:04:41.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:41 smithi134 ceph-mon[30086]: pgmap v195: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:04:41.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:41 smithi134 ceph-mon[39278]: Saving service rgw.foo.a spec with placement smithi134=foo.a;count:1 2024-02-05T08:04:41.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:41 smithi134 ceph-mon[39278]: osdmap e66: 8 total, 8 up, 8 in 2024-02-05T08:04:41.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:41 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4286562240' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T08:04:41.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:41 smithi134 ceph-mon[39278]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T08:04:41.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:41 smithi134 ceph-mon[39278]: pgmap v195: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:04:42.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[30086]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:04:42.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[30086]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-05T08:04:42.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[30086]: osdmap e67: 8 total, 8 up, 8 in 2024-02-05T08:04:42.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:42.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:42.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:42.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:42.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[30086]: Checking dashboard <-> RGW credentials 2024-02-05T08:04:42.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/1144071907' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T08:04:42.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T08:04:42.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[39278]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:04:42.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[39278]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-05T08:04:42.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[39278]: osdmap e67: 8 total, 8 up, 8 in 2024-02-05T08:04:42.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:42.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:42.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:42.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:42.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[39278]: Checking dashboard <-> RGW credentials 2024-02-05T08:04:42.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/1144071907' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T08:04:42.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:42 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T08:04:42.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:42 smithi196 ceph-mon[33018]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:04:42.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:42 smithi196 ceph-mon[33018]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-05T08:04:42.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:42 smithi196 ceph-mon[33018]: osdmap e67: 8 total, 8 up, 8 in 2024-02-05T08:04:42.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:42 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:42.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:42 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:42.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:42 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:42.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:42 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:42.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:42 smithi196 ceph-mon[33018]: Checking dashboard <-> RGW credentials 2024-02-05T08:04:42.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:42 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/1144071907' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T08:04:42.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:42 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T08:04:43.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:43 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-05T08:04:43.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:43 smithi196 ceph-mon[33018]: osdmap e68: 8 total, 8 up, 8 in 2024-02-05T08:04:43.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:43 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1272526039' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T08:04:43.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:43 smithi196 ceph-mon[33018]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T08:04:43.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:43 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T08:04:43.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:43 smithi196 ceph-mon[33018]: pgmap v198: 68 pgs: 65 unknown, 3 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:04:43.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:43 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-05T08:04:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[39278]: osdmap e68: 8 total, 8 up, 8 in 2024-02-05T08:04:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1272526039' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T08:04:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[39278]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T08:04:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T08:04:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[39278]: pgmap v198: 68 pgs: 65 unknown, 3 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:04:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:44.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-05T08:04:44.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[30086]: osdmap e68: 8 total, 8 up, 8 in 2024-02-05T08:04:44.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1272526039' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T08:04:44.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[30086]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T08:04:44.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T08:04:44.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[30086]: pgmap v198: 68 pgs: 65 unknown, 3 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:04:44.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:43 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:44.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:44 smithi196 ceph-mon[33018]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T08:04:44.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:44 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T08:04:44.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:44 smithi196 ceph-mon[33018]: osdmap e69: 8 total, 8 up, 8 in 2024-02-05T08:04:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:44 smithi134 ceph-mon[30086]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T08:04:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:44 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T08:04:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:44 smithi134 ceph-mon[30086]: osdmap e69: 8 total, 8 up, 8 in 2024-02-05T08:04:45.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:44 smithi134 ceph-mon[39278]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T08:04:45.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:44 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T08:04:45.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:44 smithi134 ceph-mon[39278]: osdmap e69: 8 total, 8 up, 8 in 2024-02-05T08:04:45.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.196 --placement '1;smithi196=iscsi.a' 2024-02-05T08:04:45.399 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:04:45.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:45 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:04:45.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:45 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:04:45.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:45 smithi196 ceph-mon[33018]: osdmap e70: 8 total, 8 up, 8 in 2024-02-05T08:04:45.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:45 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1272526039' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T08:04:45.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:45 smithi196 ceph-mon[33018]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T08:04:45.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:45 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T08:04:45.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:45 smithi196 ceph-mon[33018]: pgmap v201: 100 pgs: 18 creating+peering, 35 active+clean, 47 unknown; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-05T08:04:45.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:04:45.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:04:45.929 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[30086]: osdmap e70: 8 total, 8 up, 8 in 2024-02-05T08:04:45.929 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1272526039' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T08:04:45.929 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[30086]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T08:04:45.929 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T08:04:45.929 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[30086]: pgmap v201: 100 pgs: 18 creating+peering, 35 active+clean, 47 unknown; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-05T08:04:45.929 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:04:45.929 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:04:45.929 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[39278]: osdmap e70: 8 total, 8 up, 8 in 2024-02-05T08:04:45.929 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1272526039' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T08:04:45.929 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[39278]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T08:04:45.930 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T08:04:45.930 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:45 smithi134 ceph-mon[39278]: pgmap v201: 100 pgs: 18 creating+peering, 35 active+clean, 47 unknown; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-02-05T08:04:46.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:46 smithi196 ceph-mon[33018]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T08:04:46.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:46 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T08:04:46.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:46 smithi196 ceph-mon[33018]: osdmap e71: 8 total, 8 up, 8 in 2024-02-05T08:04:46.941 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:46 smithi134 ceph-mon[30086]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T08:04:46.941 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:46 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T08:04:46.941 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:46 smithi134 ceph-mon[30086]: osdmap e71: 8 total, 8 up, 8 in 2024-02-05T08:04:46.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:46 smithi134 ceph-mon[39278]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T08:04:46.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:46 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T08:04:46.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:46 smithi134 ceph-mon[39278]: osdmap e71: 8 total, 8 up, 8 in 2024-02-05T08:04:47.200 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled iscsi.datapool update... 2024-02-05T08:04:47.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:47 smithi196 ceph-mon[33018]: osdmap e72: 8 total, 8 up, 8 in 2024-02-05T08:04:47.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:47 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T08:04:47.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:47 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1272526039' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T08:04:47.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:47 smithi196 ceph-mon[33018]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T08:04:47.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:47 smithi196 ceph-mon[33018]: pgmap v204: 132 pgs: 27 creating+peering, 56 active+clean, 49 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-05T08:04:47.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:47 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:47.835 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-02-05T08:04:47.836 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T08:04:47.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-02-05T08:04:47.861 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:47 smithi134 ceph-mon[30086]: osdmap e72: 8 total, 8 up, 8 in 2024-02-05T08:04:47.861 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:47 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T08:04:47.861 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:47 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1272526039' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T08:04:47.861 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:47 smithi134 ceph-mon[30086]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T08:04:47.861 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:47 smithi134 ceph-mon[30086]: pgmap v204: 132 pgs: 27 creating+peering, 56 active+clean, 49 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-05T08:04:47.861 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:47 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:47.861 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:47 smithi134 ceph-mon[39278]: osdmap e72: 8 total, 8 up, 8 in 2024-02-05T08:04:47.862 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:47 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T08:04:47.862 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:47 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1272526039' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T08:04:47.862 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:47 smithi134 ceph-mon[39278]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T08:04:47.862 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:47 smithi134 ceph-mon[39278]: pgmap v204: 132 pgs: 27 creating+peering, 56 active+clean, 49 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-02-05T08:04:47.862 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:47 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:47.865 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T08:04:47.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-02-05T08:04:47.894 DEBUG:teuthology.orchestra.run.smithi196:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@iscsi.iscsi.a.service 2024-02-05T08:04:47.936 INFO:tasks.cephadm:Adding prometheus.a on smithi196 2024-02-05T08:04:47.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch apply prometheus '1;smithi196=a' 2024-02-05T08:04:48.214 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:04:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:48 smithi196 ceph-mon[33018]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.196", "placement": "1;smithi196=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:48 smithi196 ceph-mon[33018]: Saving service iscsi.datapool spec with placement smithi196=iscsi.a;count:1 2024-02-05T08:04:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:48 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T08:04:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:48 smithi196 ceph-mon[33018]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T08:04:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:48 smithi196 ceph-mon[33018]: osdmap e73: 8 total, 8 up, 8 in 2024-02-05T08:04:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:48 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1272526039' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T08:04:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:48 smithi196 ceph-mon[33018]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T08:04:48.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:48 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T08:04:49.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[30086]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.196", "placement": "1;smithi196=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:49.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[30086]: Saving service iscsi.datapool spec with placement smithi196=iscsi.a;count:1 2024-02-05T08:04:49.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T08:04:49.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[30086]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T08:04:49.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[30086]: osdmap e73: 8 total, 8 up, 8 in 2024-02-05T08:04:49.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1272526039' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T08:04:49.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[30086]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T08:04:49.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T08:04:49.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[39278]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.196", "placement": "1;smithi196=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:49.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[39278]: Saving service iscsi.datapool spec with placement smithi196=iscsi.a;count:1 2024-02-05T08:04:49.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T08:04:49.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[39278]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T08:04:49.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[39278]: osdmap e73: 8 total, 8 up, 8 in 2024-02-05T08:04:49.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1272526039' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T08:04:49.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[39278]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T08:04:49.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:48 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T08:04:49.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:49 smithi196 ceph-mon[33018]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T08:04:49.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:49 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T08:04:49.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:49 smithi196 ceph-mon[33018]: osdmap e74: 8 total, 8 up, 8 in 2024-02-05T08:04:49.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:49 smithi196 ceph-mon[33018]: pgmap v207: 132 pgs: 3 creating+activating, 18 creating+peering, 87 active+clean, 24 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-05T08:04:49.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:49 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:49.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:49 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:49.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:49 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:49.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:49 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-05T08:04:49.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:49 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-05T08:04:49.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:49 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:49.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:49 smithi196 ceph-mon[33018]: Deploying daemon iscsi.iscsi.a on smithi196 2024-02-05T08:04:49.740 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:49 smithi196 ceph-mon[33018]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:04:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[30086]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T08:04:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T08:04:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[30086]: osdmap e74: 8 total, 8 up, 8 in 2024-02-05T08:04:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[30086]: pgmap v207: 132 pgs: 3 creating+activating, 18 creating+peering, 87 active+clean, 24 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-05T08:04:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-05T08:04:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-05T08:04:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[30086]: Deploying daemon iscsi.iscsi.a on smithi196 2024-02-05T08:04:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[30086]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:04:50.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[39278]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T08:04:50.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2578723559' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T08:04:50.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[39278]: osdmap e74: 8 total, 8 up, 8 in 2024-02-05T08:04:50.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[39278]: pgmap v207: 132 pgs: 3 creating+activating, 18 creating+peering, 87 active+clean, 24 unknown; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-05T08:04:50.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:50.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:50.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:50.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-05T08:04:50.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-05T08:04:50.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:50.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[39278]: Deploying daemon iscsi.iscsi.a on smithi196 2024-02-05T08:04:50.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:49 smithi134 ceph-mon[39278]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:04:50.182 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled prometheus update... 2024-02-05T08:04:50.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:50 smithi196 ceph-mon[33018]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:04:50.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:50 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:04:50.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:50 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:51.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:50 smithi134 ceph-mon[30086]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:04:51.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:50 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:04:51.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:50 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:51.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:50 smithi134 ceph-mon[39278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:04:51.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:50 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:04:51.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:50 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:51.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:51 smithi196 ceph-mon[33018]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:51.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:51 smithi196 ceph-mon[33018]: Saving service prometheus spec with placement smithi196=a;count:1 2024-02-05T08:04:51.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:51 smithi196 ceph-mon[33018]: pgmap v208: 132 pgs: 3 creating+activating, 129 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.8 KiB/s wr, 139 op/s 2024-02-05T08:04:51.933 DEBUG:teuthology.orchestra.run.smithi196:prometheus.a> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@prometheus.a.service 2024-02-05T08:04:51.935 INFO:tasks.cephadm:Adding node-exporter.a on smithi134 2024-02-05T08:04:51.935 INFO:tasks.cephadm:Adding node-exporter.b on smithi196 2024-02-05T08:04:51.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch apply node-exporter '2;smithi134=a;smithi196=b' 2024-02-05T08:04:52.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:51 smithi134 ceph-mon[30086]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:52.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:51 smithi134 ceph-mon[30086]: Saving service prometheus spec with placement smithi196=a;count:1 2024-02-05T08:04:52.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:51 smithi134 ceph-mon[30086]: pgmap v208: 132 pgs: 3 creating+activating, 129 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.8 KiB/s wr, 139 op/s 2024-02-05T08:04:52.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:51 smithi134 ceph-mon[39278]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:52.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:51 smithi134 ceph-mon[39278]: Saving service prometheus spec with placement smithi196=a;count:1 2024-02-05T08:04:52.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:51 smithi134 ceph-mon[39278]: pgmap v208: 132 pgs: 3 creating+activating, 129 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.8 KiB/s wr, 139 op/s 2024-02-05T08:04:52.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:04:52 smithi196 podman[62711]: 2024-02-05 08:04:52.226390051 +0000 UTC m=+0.136531242 container create 3bf96afe72ea5e07c571425e06d627fec64ed7a70b5b13187139170e6e17306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-05T08:04:52.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:04:52 smithi196 podman[62711]: 2024-02-05 08:04:52.501202983 +0000 UTC m=+0.411344172 container init 3bf96afe72ea5e07c571425e06d627fec64ed7a70b5b13187139170e6e17306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T08:04:52.678 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:04:52.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:04:52 smithi196 podman[62711]: 2024-02-05 08:04:52.507737623 +0000 UTC m=+0.417878815 container start 3bf96afe72ea5e07c571425e06d627fec64ed7a70b5b13187139170e6e17306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-05T08:04:52.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:04:52 smithi196 bash[62711]: 3bf96afe72ea5e07c571425e06d627fec64ed7a70b5b13187139170e6e17306d 2024-02-05T08:04:52.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:04:52 smithi196 systemd[1]: Started Ceph iscsi.iscsi.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:04:53.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:04:52 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug Started the configuration object watcher 2024-02-05T08:04:53.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:04:52 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug Checking for config object changes every 1s 2024-02-05T08:04:53.065 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:04:52 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug Processing osd blocklist entries for this node 2024-02-05T08:04:53.407 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:04:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug Reading the configuration object to update local LIO configuration 2024-02-05T08:04:53.407 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:04:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug Configuration does not have an entry for this host(smithi196.front.sepia.ceph.com) - nothing to define to LIO 2024-02-05T08:04:53.407 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:04:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-02-05T08:04:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[30086]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-05T08:04:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[30086]: pgmap v209: 132 pgs: 3 creating+activating, 129 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.3 KiB/s wr, 135 op/s 2024-02-05T08:04:54.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:54.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/4164837324' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-05T08:04:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[39278]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-05T08:04:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[39278]: pgmap v209: 132 pgs: 3 creating+activating, 129 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.3 KiB/s wr, 135 op/s 2024-02-05T08:04:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:54.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:53 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/4164837324' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-05T08:04:54.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:53 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:53 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:53 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:53 smithi196 ceph-mon[33018]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-05T08:04:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:53 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:53 smithi196 ceph-mon[33018]: pgmap v209: 132 pgs: 3 creating+activating, 129 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.3 KiB/s wr, 135 op/s 2024-02-05T08:04:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:53 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:53 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:04:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:53 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/4164837324' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-05T08:04:55.281 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled node-exporter update... 2024-02-05T08:04:55.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:55 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:04:55.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:55 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:04:55.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:55 smithi134 ceph-mon[30086]: osdmap e75: 8 total, 8 up, 8 in 2024-02-05T08:04:55.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:55 smithi134 ceph-mon[30086]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-05T08:04:55.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:55 smithi134 ceph-mon[30086]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 5.0 KiB/s wr, 157 op/s 2024-02-05T08:04:55.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:55 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:55.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:55 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:04:55.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:55 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:04:55.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:55 smithi134 ceph-mon[39278]: osdmap e75: 8 total, 8 up, 8 in 2024-02-05T08:04:55.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:55 smithi134 ceph-mon[39278]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-05T08:04:55.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:55 smithi134 ceph-mon[39278]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 5.0 KiB/s wr, 157 op/s 2024-02-05T08:04:55.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:55 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:55 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:04:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:55 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:04:55.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:55 smithi196 ceph-mon[33018]: osdmap e75: 8 total, 8 up, 8 in 2024-02-05T08:04:55.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:55 smithi196 ceph-mon[33018]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-05T08:04:55.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:55 smithi196 ceph-mon[33018]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 5.0 KiB/s wr, 157 op/s 2024-02-05T08:04:55.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:55 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:56.129 DEBUG:teuthology.orchestra.run.smithi134:node-exporter.a> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@node-exporter.a.service 2024-02-05T08:04:56.132 DEBUG:teuthology.orchestra.run.smithi196:node-exporter.b> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@node-exporter.b.service 2024-02-05T08:04:56.134 INFO:tasks.cephadm:Adding alertmanager.a on smithi134 2024-02-05T08:04:56.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch apply alertmanager '1;smithi134=a' 2024-02-05T08:04:56.431 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:04:56.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:56 smithi196 ceph-mon[33018]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi134=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:56.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:56 smithi196 ceph-mon[33018]: Saving service node-exporter spec with placement smithi134=a;smithi196=b;count:2 2024-02-05T08:04:57.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:56 smithi134 ceph-mon[30086]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi134=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:57.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:56 smithi134 ceph-mon[30086]: Saving service node-exporter spec with placement smithi134=a;smithi196=b;count:2 2024-02-05T08:04:57.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:56 smithi134 ceph-mon[39278]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi134=a;smithi196=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:04:57.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:56 smithi134 ceph-mon[39278]: Saving service node-exporter spec with placement smithi134=a;smithi196=b;count:2 2024-02-05T08:04:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:57 smithi134 ceph-mon[30086]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.4 KiB/s wr, 137 op/s 2024-02-05T08:04:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:57 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:57 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:57 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:58.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:57 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:58.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:57 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:57 smithi134 ceph-mon[39278]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.4 KiB/s wr, 137 op/s 2024-02-05T08:04:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:57 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:57 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:57 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:57 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:57 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:57 smithi196 ceph-mon[33018]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.4 KiB/s wr, 137 op/s 2024-02-05T08:04:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:57 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:57 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:57 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:04:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:57 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:04:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:57 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:04:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:58 smithi134 ceph-mon[30086]: Deploying daemon prometheus.a on smithi196 2024-02-05T08:04:59.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:58 smithi134 ceph-mon[39278]: Deploying daemon prometheus.a on smithi196 2024-02-05T08:04:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:58 smithi196 ceph-mon[33018]: Deploying daemon prometheus.a on smithi196 2024-02-05T08:04:59.091 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled alertmanager update... 2024-02-05T08:05:00.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:59 smithi134 ceph-mon[30086]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.6 KiB/s wr, 113 op/s 2024-02-05T08:05:00.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:04:59 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:00.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:59 smithi134 ceph-mon[39278]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.6 KiB/s wr, 113 op/s 2024-02-05T08:05:00.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:04:59 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:00.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:59 smithi196 ceph-mon[33018]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.6 KiB/s wr, 113 op/s 2024-02-05T08:05:00.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:04:59 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:00.117 DEBUG:teuthology.orchestra.run.smithi134:alertmanager.a> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@alertmanager.a.service 2024-02-05T08:05:00.120 INFO:tasks.cephadm:Adding grafana.a on smithi196 2024-02-05T08:05:00.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph orch apply grafana '1;smithi196=a' 2024-02-05T08:05:00.927 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:05:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:00 smithi134 ceph-mon[30086]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi134=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:05:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:00 smithi134 ceph-mon[30086]: Saving service alertmanager spec with placement smithi134=a;count:1 2024-02-05T08:05:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:00 smithi134 ceph-mon[39278]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi134=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:05:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:00 smithi134 ceph-mon[39278]: Saving service alertmanager spec with placement smithi134=a;count:1 2024-02-05T08:05:01.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:00 smithi196 ceph-mon[33018]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi134=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:05:01.049 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:00 smithi196 ceph-mon[33018]: Saving service alertmanager spec with placement smithi134=a;count:1 2024-02-05T08:05:02.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:01 smithi134 ceph-mon[30086]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 1.2 KiB/s wr, 41 op/s 2024-02-05T08:05:02.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:01 smithi134 ceph-mon[39278]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 1.2 KiB/s wr, 41 op/s 2024-02-05T08:05:02.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:01 smithi196 ceph-mon[33018]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 1.2 KiB/s wr, 41 op/s 2024-02-05T08:05:02.958 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:05:02 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:05:02.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-05T08:05:02.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-05T08:05:02.959 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:02 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-05T08:05:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-05T08:05:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-05T08:05:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:02 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-05T08:05:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]: dispatch 2024-02-05T08:05:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-02-05T08:05:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:02 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-02-05T08:05:03.617 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled grafana update... 2024-02-05T08:05:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:03 smithi196 ceph-mon[33018]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 1023 B/s wr, 30 op/s 2024-02-05T08:05:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:03 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-05T08:05:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:03 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-05T08:05:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:03 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-02-05T08:05:04.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:03 smithi196 ceph-mon[33018]: osdmap e76: 8 total, 8 up, 8 in 2024-02-05T08:05:04.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:03 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:04.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:03 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[30086]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 1023 B/s wr, 30 op/s 2024-02-05T08:05:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-05T08:05:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-05T08:05:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-02-05T08:05:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[30086]: osdmap e76: 8 total, 8 up, 8 in 2024-02-05T08:05:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[39278]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 1023 B/s wr, 30 op/s 2024-02-05T08:05:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [1, 2]}]': finished 2024-02-05T08:05:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-02-05T08:05:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-02-05T08:05:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[39278]: osdmap e76: 8 total, 8 up, 8 in 2024-02-05T08:05:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:03 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:05.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:04 smithi196 ceph-mon[33018]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:05:05.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:04 smithi196 ceph-mon[33018]: Saving service grafana spec with placement smithi196=a;count:1 2024-02-05T08:05:05.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:04 smithi196 ceph-mon[33018]: osdmap e77: 8 total, 8 up, 8 in 2024-02-05T08:05:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:04 smithi134 ceph-mon[30086]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:05:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:04 smithi134 ceph-mon[30086]: Saving service grafana spec with placement smithi196=a;count:1 2024-02-05T08:05:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:04 smithi134 ceph-mon[30086]: osdmap e77: 8 total, 8 up, 8 in 2024-02-05T08:05:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:04 smithi134 ceph-mon[39278]: from='client.24412 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi196=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T08:05:05.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:04 smithi134 ceph-mon[39278]: Saving service grafana spec with placement smithi196=a;count:1 2024-02-05T08:05:05.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:04 smithi134 ceph-mon[39278]: osdmap e77: 8 total, 8 up, 8 in 2024-02-05T08:05:05.570 DEBUG:teuthology.orchestra.run.smithi196:grafana.a> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@grafana.a.service 2024-02-05T08:05:05.573 INFO:tasks.cephadm:Setting up client nodes... 2024-02-05T08:05:05.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-05T08:05:05.820 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:05.939 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:05 smithi134 ceph-mon[30086]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 41 B/s, 0 objects/s recovering 2024-02-05T08:05:05.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:05 smithi134 ceph-mon[39278]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 41 B/s, 0 objects/s recovering 2024-02-05T08:05:06.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:05 smithi196 ceph-mon[33018]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 41 B/s, 0 objects/s recovering 2024-02-05T08:05:07.892 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-02-05T08:05:07.892 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQCzlsBlW4v1NBAArlGAYfQXJ/shAF6nMBB34A== 2024-02-05T08:05:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:07 smithi196 ceph-mon[33018]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 41 B/s, 0 objects/s recovering 2024-02-05T08:05:08.218 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:07 smithi134 ceph-mon[30086]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 41 B/s, 0 objects/s recovering 2024-02-05T08:05:08.218 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:07 smithi134 ceph-mon[39278]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 41 B/s, 0 objects/s recovering 2024-02-05T08:05:08.783 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-02-05T08:05:08.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-05T08:05:08.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-05T08:05:08.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-05T08:05:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:08 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1019393557' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T08:05:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:08 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T08:05:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:08 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T08:05:09.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:08 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1019393557' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T08:05:09.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:08 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T08:05:09.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:08 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T08:05:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:08 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1019393557' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T08:05:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:08 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T08:05:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:08 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T08:05:09.417 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:05:10.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:09 smithi134 ceph-mon[30086]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 68 B/s, 2 objects/s recovering 2024-02-05T08:05:10.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:09 smithi134 ceph-mon[39278]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 68 B/s, 2 objects/s recovering 2024-02-05T08:05:10.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:09 smithi196 ceph-mon[33018]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 68 B/s, 2 objects/s recovering 2024-02-05T08:05:12.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:11 smithi134 ceph-mon[30086]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 68 B/s, 2 objects/s recovering 2024-02-05T08:05:12.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:11 smithi134 ceph-mon[39278]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 68 B/s, 2 objects/s recovering 2024-02-05T08:05:12.301 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:11 smithi196 ceph-mon[33018]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 68 B/s, 2 objects/s recovering 2024-02-05T08:05:12.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:05:12 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:05:12.883 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:12 smithi196 podman[64597]: 2024-02-05 08:05:12.77949896 +0000 UTC m=+0.021467819 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-02-05T08:05:12.883 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:12 smithi196 podman[64597]: 2024-02-05 08:05:12.882772604 +0000 UTC m=+0.124741461 container create b621d9607e33a0ac0960a40881f88137acf9b2abd8619dbe349fc1b68b5d1221 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T08:05:13.049 INFO:teuthology.orchestra.run.smithi196.stdout:[client.1] 2024-02-05T08:05:13.049 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQC5lsBlYiWqAhAAVK014P378I67/3Gq96H0Uw== 2024-02-05T08:05:13.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:12 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:05:13.144 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:12 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:05:13.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:12 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:05:13.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:12 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:05:13.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:12 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:05:13.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:12 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:05:13.565 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 podman[64597]: 2024-02-05 08:05:13.144577844 +0000 UTC m=+0.386546703 container init b621d9607e33a0ac0960a40881f88137acf9b2abd8619dbe349fc1b68b5d1221 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T08:05:13.565 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 podman[64597]: 2024-02-05 08:05:13.15053929 +0000 UTC m=+0.392508149 container start b621d9607e33a0ac0960a40881f88137acf9b2abd8619dbe349fc1b68b5d1221 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T08:05:13.565 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.173Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-02-05T08:05:13.565 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.173Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-02-05T08:05:13.565 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.173Z caller=main.go:570 level=info host_details="(Linux 5.14.0-416.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jan 31 22:52:48 UTC 2024 x86_64 smithi196 (none))" 2024-02-05T08:05:13.565 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.173Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-05T08:05:13.566 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.173Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-05T08:05:13.566 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.174Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-02-05T08:05:13.566 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.175Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-02-05T08:05:13.566 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.176Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-02-05T08:05:13.566 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.176Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-02-05T08:05:13.566 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.178Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-05T08:05:13.566 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.178Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.098µs 2024-02-05T08:05:13.566 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.178Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-05T08:05:13.566 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.178Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-05T08:05:13.566 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.178Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.3µs wal_replay_duration=282.638µs wbl_replay_duration=87ns total_replay_duration=317.722µs 2024-02-05T08:05:13.566 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.179Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-02-05T08:05:13.566 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.179Z caller=main.go:1029 level=info msg="TSDB started" 2024-02-05T08:05:13.566 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.179Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-05T08:05:13.567 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.196Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.939586ms db_storage=739ns remote_storage=1.107µs web_handler=370ns query_engine=485ns scrape=374.368µs scrape_sd=63.861µs notify=975ns notify_sd=1.89µs rules=16.329863ms tracing=5.213µs 2024-02-05T08:05:13.567 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.196Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-02-05T08:05:13.567 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:05:13.196Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-02-05T08:05:13.567 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 bash[64597]: b621d9607e33a0ac0960a40881f88137acf9b2abd8619dbe349fc1b68b5d1221 2024-02-05T08:05:13.567 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:05:13 smithi196 systemd[1]: Started Ceph prometheus.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:05:13.956 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-02-05T08:05:13.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-05T08:05:13.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-05T08:05:13.981 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-mon[33018]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s; 55 B/s, 1 objects/s recovering 2024-02-05T08:05:13.981 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:13.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-mon[33018]: osdmap e78: 8 total, 8 up, 8 in 2024-02-05T08:05:13.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-mon[33018]: from='client.? 172.21.15.196:0/3714425827' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T08:05:13.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T08:05:13.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T08:05:13.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:13.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:13.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:13.982 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:13 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-05T08:05:13.998 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-05T08:05:13.998 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-05T08:05:13.999 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph mgr dump --format=json 2024-02-05T08:05:14.248 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:14.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[30086]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s; 55 B/s, 1 objects/s recovering 2024-02-05T08:05:14.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:14.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[30086]: osdmap e78: 8 total, 8 up, 8 in 2024-02-05T08:05:14.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[30086]: from='client.? 172.21.15.196:0/3714425827' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T08:05:14.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T08:05:14.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T08:05:14.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:14.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:14.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:14.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-05T08:05:14.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[39278]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s; 55 B/s, 1 objects/s recovering 2024-02-05T08:05:14.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:14.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[39278]: osdmap e78: 8 total, 8 up, 8 in 2024-02-05T08:05:14.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[39278]: from='client.? 172.21.15.196:0/3714425827' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T08:05:14.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T08:05:14.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T08:05:14.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:14.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:14.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' 2024-02-05T08:05:14.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:13 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-05T08:05:14.631 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:14 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ignoring --setuser ceph since I am not root 2024-02-05T08:05:14.632 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:14 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ignoring --setgroup ceph since I am not root 2024-02-05T08:05:14.814 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: ignoring --setuser ceph since I am not root 2024-02-05T08:05:14.814 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: ignoring --setgroup ceph since I am not root 2024-02-05T08:05:14.814 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:14 smithi196 ceph-mgr[35159]: -- 172.21.15.196:0/1337682546 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cecc137520 con 0x55cecc10c800 2024-02-05T08:05:15.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:14 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:14.922+0000 7f216cce6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T08:05:15.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:15 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:15.019+0000 7f216cce6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T08:05:15.314 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:14.903+0000 7fe067343200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T08:05:15.314 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:14.997+0000 7fe067343200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T08:05:15.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:15 smithi134 ceph-mon[30086]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-05T08:05:15.574 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:15 smithi134 ceph-mon[30086]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-05T08:05:15.574 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:15 smithi134 ceph-mon[39278]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-05T08:05:15.575 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:15 smithi134 ceph-mon[39278]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-05T08:05:15.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:15 smithi196 ceph-mon[33018]: from='mgr.14152 172.21.15.134:0/1230449147' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-05T08:05:15.646 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:15 smithi196 ceph-mon[33018]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-05T08:05:15.896 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:15 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:15.645+0000 7fe067343200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T08:05:15.896 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:15 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:15.744+0000 7fe067343200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T08:05:15.920 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:15 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:15.677+0000 7f216cce6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T08:05:15.921 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:15 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:15.772+0000 7f216cce6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T08:05:16.087 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:05:16.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:15 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:15.920+0000 7f216cce6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T08:05:16.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:16.018+0000 7f216cce6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T08:05:16.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:16.105+0000 7f216cce6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T08:05:16.314 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:15 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:15.896+0000 7fe067343200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T08:05:16.314 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:15 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:15.996+0000 7fe067343200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T08:05:16.314 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:16.084+0000 7fe067343200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T08:05:16.555 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:16 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3684360145' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-05T08:05:16.555 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:16.392+0000 7f216cce6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T08:05:16.555 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:16.511+0000 7f216cce6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T08:05:16.555 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:16 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3684360145' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-05T08:05:16.628 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:16 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3684360145' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-05T08:05:16.629 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:16.356+0000 7fe067343200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T08:05:16.629 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:16.468+0000 7fe067343200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T08:05:16.629 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:16.556+0000 7fe067343200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T08:05:16.820 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:16.603+0000 7f216cce6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T08:05:16.820 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:16.704+0000 7f216cce6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T08:05:16.861 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":1525062017},{"type":"v1","addr":"172.21.15.134:6801","nonce":1525062017}]},"active_addr":"172.21.15.134:6801/1525062017","active_change":"2024-02-05T08:00:02.709451+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[{"gid":24118,"name":"x","mgr_features":4540701272860131327,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.134:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":1645475987}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":69097091}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":3200884805}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":788812699}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":3325835814}]}]} 2024-02-05T08:05:16.865 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-05T08:05:16.865 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-05T08:05:16.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T08:05:17.064 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:16.657+0000 7fe067343200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T08:05:17.064 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:16.832+0000 7fe067343200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T08:05:17.131 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:17.253 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:16.897+0000 7f216cce6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T08:05:18.064 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:17 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:17.793+0000 7fe067343200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T08:05:18.065 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:17 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:17.923+0000 7fe067343200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T08:05:18.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:17.915+0000 7f216cce6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T08:05:18.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:18.046+0000 7f216cce6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T08:05:18.543 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:18 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:18.257+0000 7fe067343200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T08:05:18.543 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:18 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:18.348+0000 7fe067343200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T08:05:18.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:18 smithi196 ceph-mon[33018]: 1.0 deep-scrub starts 2024-02-05T08:05:18.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:18 smithi196 ceph-mon[33018]: 1.0 deep-scrub ok 2024-02-05T08:05:18.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:18 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:05:18.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:18 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:05:18.544 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:18 smithi196 ceph-mon[33018]: osdmap e79: 8 total, 8 up, 8 in 2024-02-05T08:05:18.638 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-mon[30086]: 1.0 deep-scrub starts 2024-02-05T08:05:18.638 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-mon[30086]: 1.0 deep-scrub ok 2024-02-05T08:05:18.639 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:05:18.639 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:05:18.639 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-mon[30086]: osdmap e79: 8 total, 8 up, 8 in 2024-02-05T08:05:18.639 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:18.387+0000 7f216cce6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T08:05:18.640 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:18.484+0000 7f216cce6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T08:05:18.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-mon[39278]: 1.0 deep-scrub starts 2024-02-05T08:05:18.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-mon[39278]: 1.0 deep-scrub ok 2024-02-05T08:05:18.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:05:18.640 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:05:18.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-mon[39278]: osdmap e79: 8 total, 8 up, 8 in 2024-02-05T08:05:18.814 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:18 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:18.542+0000 7fe067343200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T08:05:18.814 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:18 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:18.726+0000 7fe067343200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T08:05:19.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:18.687+0000 7f216cce6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T08:05:19.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:18 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:18.881+0000 7f216cce6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T08:05:19.096 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:05:19.096 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":79,"fsid":"43e13aa4-c3fc-11ee-95b6-87774f69a715","created":"2024-02-05T07:59:09.321262+0000","modified":"2024-02-05T08:05:17.345931+0000","last_up_change":"2024-02-05T08:04:26.849889+0000","last_in_change":"2024-02-05T08:04:09.165031+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T08:02:21.034012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-05T08:04:39.400746+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-05T08:04:40.005769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-05T08:04:41.851891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-05T08:04:43.793804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-05T08:04:45.853999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d90a5ea0-9a19-452e-9564-ca7b7a7c45e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6803","nonce":2540271081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6805","nonce":2540271081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6809","nonce":2540271081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6807","nonce":2540271081}]},"public_addr":"172.21.15.134:6803/2540271081","cluster_addr":"172.21.15.134:6805/2540271081","heartbeat_back_addr":"172.21.15.134:6809/2540271081","heartbeat_front_addr":"172.21.15.134:6807/2540271081","state":["exists","up"]},{"osd":1,"uuid":"4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":2032802960},{"type":"v1","addr":"172.21.15.134:6811","nonce":2032802960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":2032802960},{"type":"v1","addr":"172.21.15.134:6813","nonce":2032802960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2032802960},{"type":"v1","addr":"172.21.15.134:6817","nonce":2032802960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":2032802960},{"type":"v1","addr":"172.21.15.134:6815","nonce":2032802960}]},"public_addr":"172.21.15.134:6811/2032802960","cluster_addr":"172.21.15.134:6813/2032802960","heartbeat_back_addr":"172.21.15.134:6817/2032802960","heartbeat_front_addr":"172.21.15.134:6815/2032802960","state":["exists","up"]},{"osd":2,"uuid":"078e578b-855b-49ff-83bd-0548a9c5a708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6819","nonce":153827987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6821","nonce":153827987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6825","nonce":153827987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6823","nonce":153827987}]},"public_addr":"172.21.15.134:6819/153827987","cluster_addr":"172.21.15.134:6821/153827987","heartbeat_back_addr":"172.21.15.134:6825/153827987","heartbeat_front_addr":"172.21.15.134:6823/153827987","state":["exists","up"]},{"osd":3,"uuid":"6774e330-3202-4671-894e-172cc2f6189d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6827","nonce":1527507056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6829","nonce":1527507056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6833","nonce":1527507056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6831","nonce":1527507056}]},"public_addr":"172.21.15.134:6827/1527507056","cluster_addr":"172.21.15.134:6829/1527507056","heartbeat_back_addr":"172.21.15.134:6833/1527507056","heartbeat_front_addr":"172.21.15.134:6831/1527507056","state":["exists","up"]},{"osd":4,"uuid":"52f70cfa-ab0e-4751-a5f8-1045c61c1693","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6801","nonce":3005156808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6803","nonce":3005156808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6807","nonce":3005156808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6805","nonce":3005156808}]},"public_addr":"172.21.15.196:6801/3005156808","cluster_addr":"172.21.15.196:6803/3005156808","heartbeat_back_addr":"172.21.15.196:6807/3005156808","heartbeat_front_addr":"172.21.15.196:6805/3005156808","state":["exists","up"]},{"osd":5,"uuid":"b4b21769-d40f-43fd-b7e9-2477b74d3955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6809","nonce":1603224792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6811","nonce":1603224792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6815","nonce":1603224792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6813","nonce":1603224792}]},"public_addr":"172.21.15.196:6809/1603224792","cluster_addr":"172.21.15.196:6811/1603224792","heartbeat_back_addr":"172.21.15.196:6815/1603224792","heartbeat_front_addr":"172.21.15.196:6813/1603224792","state":["exists","up"]},{"osd":6,"uuid":"af1cced4-0015-47f5-b105-27641bfe4f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6817","nonce":3727413143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6819","nonce":3727413143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6823","nonce":3727413143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6821","nonce":3727413143}]},"public_addr":"172.21.15.196:6817/3727413143","cluster_addr":"172.21.15.196:6819/3727413143","heartbeat_back_addr":"172.21.15.196:6823/3727413143","heartbeat_front_addr":"172.21.15.196:6821/3727413143","state":["exists","up"]},{"osd":7,"uuid":"c54beeae-9f7c-4723-a369-13bb531b4057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6825","nonce":4206571010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6827","nonce":4206571010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6831","nonce":4206571010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6829","nonce":4206571010}]},"public_addr":"172.21.15.196:6825/4206571010","cluster_addr":"172.21.15.196:6827/4206571010","heartbeat_back_addr":"172.21.15.196:6831/4206571010","heartbeat_front_addr":"172.21.15.196:6829/4206571010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:05:12.234291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:28.076106+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:44.267635+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:04.796357+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:11.448792+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:54.326027+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:05:17.260063+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:25.455681+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/3161126612":"2024-02-06T08:00:02.709280+0000","172.21.15.134:0/2952444615":"2024-02-06T08:00:02.709280+0000","172.21.15.134:6801/3607168099":"2024-02-06T08:00:02.709280+0000","172.21.15.134:0/4225724311":"2024-02-06T08:00:02.709280+0000","172.21.15.134:6801/673876301":"2024-02-06T07:59:36.275039+0000","172.21.15.134:6800/673876301":"2024-02-06T07:59:36.275039+0000","172.21.15.134:6800/3607168099":"2024-02-06T08:00:02.709280+0000","172.21.15.134:0/301941854":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/464208089":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/1046500326":"2024-02-06T07:59:36.275039+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T08:05:19.276 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:19 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:19.011+0000 7fe067343200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T08:05:19.349 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:19 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:19.184+0000 7f216cce6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T08:05:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:19 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3187985667' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:05:19.564 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:19 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:19.275+0000 7fe067343200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T08:05:19.565 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:19 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:19.513+0000 7fe067343200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T08:05:19.691 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:19 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3187985667' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:05:19.691 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:19 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:19.451+0000 7f216cce6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T08:05:19.691 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:19 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3187985667' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:05:19.833 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-05T08:05:19.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T08:05:20.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:19 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:19.691+0000 7f216cce6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T08:05:20.085 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:20.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:20 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:05:20.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:20 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:05:20.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:20 smithi134 ceph-mon[30086]: osdmap e80: 8 total, 8 up, 8 in 2024-02-05T08:05:20.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:20 smithi134 ceph-mon[30086]: osdmap e81: 8 total, 8 up, 8 in 2024-02-05T08:05:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:20 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:05:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:20 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:05:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:20 smithi134 ceph-mon[39278]: osdmap e80: 8 total, 8 up, 8 in 2024-02-05T08:05:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:20 smithi134 ceph-mon[39278]: osdmap e81: 8 total, 8 up, 8 in 2024-02-05T08:05:20.777 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:20 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:20.547+0000 7f216cce6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T08:05:20.814 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:20 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:20.317+0000 7fe067343200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T08:05:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:20 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:05:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:20 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:05:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:20 smithi196 ceph-mon[33018]: osdmap e80: 8 total, 8 up, 8 in 2024-02-05T08:05:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:20 smithi196 ceph-mon[33018]: osdmap e81: 8 total, 8 up, 8 in 2024-02-05T08:05:21.209 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:20 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:20.893+0000 7fe067343200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T08:05:21.209 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:20 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:20.991+0000 7fe067343200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T08:05:21.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:21.134+0000 7f216cce6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T08:05:21.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:21.236+0000 7f216cce6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T08:05:21.562 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-mon[30086]: Standby manager daemon x restarted 2024-02-05T08:05:21.562 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-mon[30086]: Standby manager daemon x started 2024-02-05T08:05:21.562 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-mon[30086]: from='mgr.? 172.21.15.196:0/3980025535' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T08:05:21.563 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-mon[30086]: from='mgr.? 172.21.15.196:0/3980025535' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T08:05:21.563 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-mon[30086]: from='mgr.? 172.21.15.196:0/3980025535' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T08:05:21.563 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-mon[30086]: from='mgr.? 172.21.15.196:0/3980025535' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T08:05:21.563 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:21.463+0000 7f216cce6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T08:05:21.563 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-mon[39278]: Standby manager daemon x restarted 2024-02-05T08:05:21.563 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-mon[39278]: Standby manager daemon x started 2024-02-05T08:05:21.563 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-mon[39278]: from='mgr.? 172.21.15.196:0/3980025535' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T08:05:21.563 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-mon[39278]: from='mgr.? 172.21.15.196:0/3980025535' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T08:05:21.563 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-mon[39278]: from='mgr.? 172.21.15.196:0/3980025535' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T08:05:21.564 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-mon[39278]: from='mgr.? 172.21.15.196:0/3980025535' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T08:05:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:21 smithi196 ceph-mon[33018]: Standby manager daemon x restarted 2024-02-05T08:05:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:21 smithi196 ceph-mon[33018]: Standby manager daemon x started 2024-02-05T08:05:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:21 smithi196 ceph-mon[33018]: from='mgr.? 172.21.15.196:0/3980025535' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T08:05:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:21 smithi196 ceph-mon[33018]: from='mgr.? 172.21.15.196:0/3980025535' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T08:05:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:21 smithi196 ceph-mon[33018]: from='mgr.? 172.21.15.196:0/3980025535' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T08:05:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:21 smithi196 ceph-mon[33018]: from='mgr.? 172.21.15.196:0/3980025535' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T08:05:21.565 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:21 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:21.207+0000 7fe067343200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T08:05:21.565 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:21 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: [05/Feb/2024:08:05:21] ENGINE Bus STARTING 2024-02-05T08:05:21.565 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:21 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: CherryPy Checker: 2024-02-05T08:05:21.565 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:21 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: The Application mounted at '' has an empty config. 2024-02-05T08:05:21.565 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:21 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: 2024-02-05T08:05:21.565 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:21 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: [05/Feb/2024:08:05:21] ENGINE Serving on http://:::9283 2024-02-05T08:05:21.565 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:05:21 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x[35136]: [05/Feb/2024:08:05:21] ENGINE Bus STARTED 2024-02-05T08:05:21.869 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:05:21] ENGINE Bus STARTING 2024-02-05T08:05:21.869 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: CherryPy Checker: 2024-02-05T08:05:21.869 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: The Application mounted at '' has an empty config. 2024-02-05T08:05:21.869 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:05:21.869 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:05:21] ENGINE Serving on http://:::9283 2024-02-05T08:05:21.869 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:05:21] ENGINE Bus STARTED 2024-02-05T08:05:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:05:22.119 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":82,"fsid":"43e13aa4-c3fc-11ee-95b6-87774f69a715","created":"2024-02-05T07:59:09.321262+0000","modified":"2024-02-05T08:05:21.465985+0000","last_up_change":"2024-02-05T08:04:26.849889+0000","last_in_change":"2024-02-05T08:04:09.165031+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T08:02:21.034012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-05T08:04:39.400746+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-05T08:04:40.005769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-05T08:04:41.851891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-05T08:04:43.793804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-05T08:04:45.853999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d90a5ea0-9a19-452e-9564-ca7b7a7c45e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6803","nonce":2540271081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6805","nonce":2540271081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6809","nonce":2540271081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6807","nonce":2540271081}]},"public_addr":"172.21.15.134:6803/2540271081","cluster_addr":"172.21.15.134:6805/2540271081","heartbeat_back_addr":"172.21.15.134:6809/2540271081","heartbeat_front_addr":"172.21.15.134:6807/2540271081","state":["exists","up"]},{"osd":1,"uuid":"4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":2032802960},{"type":"v1","addr":"172.21.15.134:6811","nonce":2032802960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":2032802960},{"type":"v1","addr":"172.21.15.134:6813","nonce":2032802960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":2032802960},{"type":"v1","addr":"172.21.15.134:6817","nonce":2032802960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":2032802960},{"type":"v1","addr":"172.21.15.134:6815","nonce":2032802960}]},"public_addr":"172.21.15.134:6811/2032802960","cluster_addr":"172.21.15.134:6813/2032802960","heartbeat_back_addr":"172.21.15.134:6817/2032802960","heartbeat_front_addr":"172.21.15.134:6815/2032802960","state":["exists","up"]},{"osd":2,"uuid":"078e578b-855b-49ff-83bd-0548a9c5a708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6819","nonce":153827987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6821","nonce":153827987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6825","nonce":153827987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6823","nonce":153827987}]},"public_addr":"172.21.15.134:6819/153827987","cluster_addr":"172.21.15.134:6821/153827987","heartbeat_back_addr":"172.21.15.134:6825/153827987","heartbeat_front_addr":"172.21.15.134:6823/153827987","state":["exists","up"]},{"osd":3,"uuid":"6774e330-3202-4671-894e-172cc2f6189d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6827","nonce":1527507056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6829","nonce":1527507056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6833","nonce":1527507056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6831","nonce":1527507056}]},"public_addr":"172.21.15.134:6827/1527507056","cluster_addr":"172.21.15.134:6829/1527507056","heartbeat_back_addr":"172.21.15.134:6833/1527507056","heartbeat_front_addr":"172.21.15.134:6831/1527507056","state":["exists","up"]},{"osd":4,"uuid":"52f70cfa-ab0e-4751-a5f8-1045c61c1693","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6801","nonce":3005156808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6803","nonce":3005156808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6807","nonce":3005156808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6805","nonce":3005156808}]},"public_addr":"172.21.15.196:6801/3005156808","cluster_addr":"172.21.15.196:6803/3005156808","heartbeat_back_addr":"172.21.15.196:6807/3005156808","heartbeat_front_addr":"172.21.15.196:6805/3005156808","state":["exists","up"]},{"osd":5,"uuid":"b4b21769-d40f-43fd-b7e9-2477b74d3955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6809","nonce":1603224792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6811","nonce":1603224792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6815","nonce":1603224792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6813","nonce":1603224792}]},"public_addr":"172.21.15.196:6809/1603224792","cluster_addr":"172.21.15.196:6811/1603224792","heartbeat_back_addr":"172.21.15.196:6815/1603224792","heartbeat_front_addr":"172.21.15.196:6813/1603224792","state":["exists","up"]},{"osd":6,"uuid":"af1cced4-0015-47f5-b105-27641bfe4f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6817","nonce":3727413143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6819","nonce":3727413143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6823","nonce":3727413143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6821","nonce":3727413143}]},"public_addr":"172.21.15.196:6817/3727413143","cluster_addr":"172.21.15.196:6819/3727413143","heartbeat_back_addr":"172.21.15.196:6823/3727413143","heartbeat_front_addr":"172.21.15.196:6821/3727413143","state":["exists","up"]},{"osd":7,"uuid":"c54beeae-9f7c-4723-a369-13bb531b4057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6825","nonce":4206571010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6827","nonce":4206571010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6831","nonce":4206571010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6829","nonce":4206571010}]},"public_addr":"172.21.15.196:6825/4206571010","cluster_addr":"172.21.15.196:6827/4206571010","heartbeat_back_addr":"172.21.15.196:6831/4206571010","heartbeat_front_addr":"172.21.15.196:6829/4206571010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:05:12.234291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:28.076106+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:44.267635+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:05:19.021118+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:11.448792+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:54.326027+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:05:17.260063+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:04:25.455681+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/788812699":"2024-02-06T08:05:21.465844+0000","172.21.15.134:0/1046500326":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/464208089":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/301941854":"2024-02-06T07:59:36.275039+0000","172.21.15.134:6800/3607168099":"2024-02-06T08:00:02.709280+0000","172.21.15.134:6800/673876301":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/4225724311":"2024-02-06T08:00:02.709280+0000","172.21.15.134:6800/1525062017":"2024-02-06T08:05:21.465844+0000","172.21.15.134:6801/3607168099":"2024-02-06T08:00:02.709280+0000","172.21.15.134:6801/673876301":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/3200884805":"2024-02-06T08:05:21.465844+0000","172.21.15.134:0/2952444615":"2024-02-06T08:00:02.709280+0000","172.21.15.134:0/3325835814":"2024-02-06T08:05:21.465844+0000","172.21.15.134:0/3161126612":"2024-02-06T08:00:02.709280+0000","172.21.15.134:0/69097091":"2024-02-06T08:05:21.465844+0000","172.21.15.134:6801/1525062017":"2024-02-06T08:05:21.465844+0000","172.21.15.134:0/1645475987":"2024-02-06T08:05:21.465844+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T08:05:22.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-05T08:05:22.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: Active manager daemon y restarted 2024-02-05T08:05:22.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: Activating manager daemon y 2024-02-05T08:05:22.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: osdmap e82: 8 total, 8 up, 8 in 2024-02-05T08:05:22.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: mgrmap e19: y(active, starting, since 0.00752988s), standbys: x 2024-02-05T08:05:22.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:05:22.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:05:22.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: Manager daemon y is now available 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:05:22.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:05:22.459 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:22.459 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:05:22.459 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T08:05:22.459 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T08:05:22.459 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2428282938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:05:22.459 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-05T08:05:22.459 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: Active manager daemon y restarted 2024-02-05T08:05:22.459 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: Activating manager daemon y 2024-02-05T08:05:22.459 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: osdmap e82: 8 total, 8 up, 8 in 2024-02-05T08:05:22.459 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: mgrmap e19: y(active, starting, since 0.00752988s), standbys: x 2024-02-05T08:05:22.459 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:05:22.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:05:22.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:05:22.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T08:05:22.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T08:05:22.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:05:22.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:05:22.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:05:22.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:05:22.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:05:22.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:05:22.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:05:22.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:05:22.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T08:05:22.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T08:05:22.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T08:05:22.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: Manager daemon y is now available 2024-02-05T08:05:22.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:05:22.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:05:22.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:22.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:05:22.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T08:05:22.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T08:05:22.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:22 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2428282938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:05:22.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-05T08:05:22.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: Active manager daemon y restarted 2024-02-05T08:05:22.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: Activating manager daemon y 2024-02-05T08:05:22.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: osdmap e82: 8 total, 8 up, 8 in 2024-02-05T08:05:22.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: mgrmap e19: y(active, starting, since 0.00752988s), standbys: x 2024-02-05T08:05:22.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T08:05:22.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T08:05:22.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T08:05:22.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T08:05:22.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T08:05:22.659 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T08:05:22.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:05:22.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T08:05:22.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T08:05:22.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T08:05:22.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T08:05:22.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T08:05:22.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T08:05:22.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T08:05:22.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T08:05:22.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T08:05:22.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: Manager daemon y is now available 2024-02-05T08:05:22.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:05:22.660 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:05:22.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:22.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:05:22.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T08:05:22.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T08:05:22.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2428282938' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:05:22.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-05T08:05:22.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-05T08:05:22.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-05T08:05:22.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-05T08:05:22.812 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-05T08:05:22.812 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-05T08:05:22.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-05T08:05:22.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-05T08:05:23.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:05:22 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:05:23.715 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:23 smithi196 ceph-mon[33018]: [05/Feb/2024:08:05:21] ENGINE Bus STARTING 2024-02-05T08:05:23.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:23 smithi196 ceph-mon[33018]: [05/Feb/2024:08:05:22] ENGINE Serving on https://172.21.15.134:7150 2024-02-05T08:05:23.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:23 smithi196 ceph-mon[33018]: [05/Feb/2024:08:05:22] ENGINE Serving on http://172.21.15.134:8765 2024-02-05T08:05:23.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:23 smithi196 ceph-mon[33018]: [05/Feb/2024:08:05:22] ENGINE Bus STARTED 2024-02-05T08:05:23.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:23 smithi196 ceph-mon[33018]: mgrmap e20: y(active, since 1.01138s), standbys: x 2024-02-05T08:05:23.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:23 smithi134 ceph-mon[30086]: [05/Feb/2024:08:05:21] ENGINE Bus STARTING 2024-02-05T08:05:23.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:23 smithi134 ceph-mon[30086]: [05/Feb/2024:08:05:22] ENGINE Serving on https://172.21.15.134:7150 2024-02-05T08:05:23.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:23 smithi134 ceph-mon[30086]: [05/Feb/2024:08:05:22] ENGINE Serving on http://172.21.15.134:8765 2024-02-05T08:05:23.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:23 smithi134 ceph-mon[30086]: [05/Feb/2024:08:05:22] ENGINE Bus STARTED 2024-02-05T08:05:23.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:23 smithi134 ceph-mon[30086]: mgrmap e20: y(active, since 1.01138s), standbys: x 2024-02-05T08:05:23.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:23 smithi134 ceph-mon[39278]: [05/Feb/2024:08:05:21] ENGINE Bus STARTING 2024-02-05T08:05:23.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:23 smithi134 ceph-mon[39278]: [05/Feb/2024:08:05:22] ENGINE Serving on https://172.21.15.134:7150 2024-02-05T08:05:23.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:23 smithi134 ceph-mon[39278]: [05/Feb/2024:08:05:22] ENGINE Serving on http://172.21.15.134:8765 2024-02-05T08:05:23.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:23 smithi134 ceph-mon[39278]: [05/Feb/2024:08:05:22] ENGINE Bus STARTED 2024-02-05T08:05:23.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:23 smithi134 ceph-mon[39278]: mgrmap e20: y(active, since 1.01138s), standbys: x 2024-02-05T08:05:23.812 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:23.812 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:23.827 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:23.857 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:24.523 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:24 smithi134 ceph-mon[30086]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:05:24.523 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:24 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:24.523 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:24 smithi134 ceph-mon[39278]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:05:24.523 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:24 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:24 smithi196 ceph-mon[33018]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:05:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:24 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:25.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:25 smithi134 ceph-mon[30086]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:05:25.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:25 smithi134 ceph-mon[30086]: mgrmap e21: y(active, since 3s), standbys: x 2024-02-05T08:05:25.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:25 smithi134 ceph-mon[30086]: osdmap e83: 8 total, 8 up, 8 in 2024-02-05T08:05:25.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:25 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:25.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:25 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:25.652 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:25 smithi134 ceph-mon[39278]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:05:25.652 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:25 smithi134 ceph-mon[39278]: mgrmap e21: y(active, since 3s), standbys: x 2024-02-05T08:05:25.652 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:25 smithi134 ceph-mon[39278]: osdmap e83: 8 total, 8 up, 8 in 2024-02-05T08:05:25.652 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:25 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:25.652 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:25 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:25 smithi196 ceph-mon[33018]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:05:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:25 smithi196 ceph-mon[33018]: mgrmap e21: y(active, since 3s), standbys: x 2024-02-05T08:05:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:25 smithi196 ceph-mon[33018]: osdmap e83: 8 total, 8 up, 8 in 2024-02-05T08:05:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:25 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:25 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:26 smithi196 ceph-mon[33018]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:05:27.084 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:26 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:27.084 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:26 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:27.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:26 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:27.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:26 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:27.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:26 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:27.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:26 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:27.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:26 smithi196 ceph-mon[33018]: Adjusting osd_memory_target on smithi196 to 3760M 2024-02-05T08:05:27.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:26 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:27.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:26 smithi196 ceph-mon[33018]: mgrmap e22: y(active, since 4s), standbys: x 2024-02-05T08:05:27.203 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[30086]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:05:27.203 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:27.203 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:27.203 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:27.203 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:27.203 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:27.203 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:27.203 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[30086]: Adjusting osd_memory_target on smithi196 to 3760M 2024-02-05T08:05:27.203 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:27.204 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[30086]: mgrmap e22: y(active, since 4s), standbys: x 2024-02-05T08:05:27.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[39278]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:05:27.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:27.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:27.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:27.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:27.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:27.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:27.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[39278]: Adjusting osd_memory_target on smithi196 to 3760M 2024-02-05T08:05:27.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:27.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:26 smithi134 ceph-mon[39278]: mgrmap e22: y(active, since 4s), standbys: x 2024-02-05T08:05:28.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:05:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:05:29.195 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:28 smithi134 ceph-mon[30086]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:05:29.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:28 smithi134 ceph-mon[39278]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:05:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:28 smithi196 ceph-mon[33018]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail 2024-02-05T08:05:29.761 INFO:teuthology.orchestra.run.smithi134.stdout:77309411368 2024-02-05T08:05:29.762 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-05T08:05:30.185 INFO:teuthology.orchestra.run.smithi134.stdout:55834574893 2024-02-05T08:05:30.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-05T08:05:30.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:30 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:05:30.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:30 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:05:30.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:30 smithi134 ceph-mon[30086]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-05T08:05:30.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:30 smithi134 ceph-mon[30086]: osdmap e84: 8 total, 8 up, 8 in 2024-02-05T08:05:30.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:30 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:05:30.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:30 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:05:30.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:30 smithi134 ceph-mon[39278]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-05T08:05:30.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:30 smithi134 ceph-mon[39278]: osdmap e84: 8 total, 8 up, 8 in 2024-02-05T08:05:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:30 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:05:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:30 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:05:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:30 smithi196 ceph-mon[33018]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-05T08:05:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:30 smithi196 ceph-mon[33018]: osdmap e84: 8 total, 8 up, 8 in 2024-02-05T08:05:31.038 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:31.039 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:31.039 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:31.040 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:31.661 INFO:teuthology.orchestra.run.smithi134.stdout:257698037774 2024-02-05T08:05:31.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-05T08:05:31.665 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:32.925 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:33.261 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:05:32 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:05:33.282 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:33.382 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:33 smithi134 ceph-mon[30086]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-05T08:05:33.382 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:33 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:33.382 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:33 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:33.382 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:33 smithi134 ceph-mon[39278]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-05T08:05:33.383 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:33 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:33.383 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:33 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:33 smithi196 ceph-mon[33018]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-05T08:05:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:33 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:33 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:34.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:34.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:34.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:34.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:34.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:34.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:34.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:34.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:34.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:05:34.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:05:34.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:34.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:34.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:34.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:34.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:34.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:34.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:34.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:34.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:05:34.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:05:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:34 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:34.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:34.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:05:34.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:34.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:05:34.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:05:35.289 INFO:teuthology.orchestra.run.smithi134.stdout:55834574894 2024-02-05T08:05:35.514 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[30086]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-05T08:05:35.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[30086]: Adjusting osd_memory_target on smithi134 to 3760M 2024-02-05T08:05:35.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[30086]: Updating smithi134:/etc/ceph/ceph.conf 2024-02-05T08:05:35.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[30086]: Updating smithi196:/etc/ceph/ceph.conf 2024-02-05T08:05:35.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[30086]: Updating smithi134:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:05:35.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[30086]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:05:35.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[30086]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-02-05T08:05:35.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[30086]: osdmap e85: 8 total, 8 up, 8 in 2024-02-05T08:05:35.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.516 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[39278]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-05T08:05:35.516 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[39278]: Adjusting osd_memory_target on smithi134 to 3760M 2024-02-05T08:05:35.516 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[39278]: Updating smithi134:/etc/ceph/ceph.conf 2024-02-05T08:05:35.516 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[39278]: Updating smithi196:/etc/ceph/ceph.conf 2024-02-05T08:05:35.516 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[39278]: Updating smithi134:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:05:35.516 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[39278]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:05:35.516 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[39278]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-02-05T08:05:35.516 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[39278]: osdmap e85: 8 total, 8 up, 8 in 2024-02-05T08:05:35.516 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.516 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.517 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.517 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.517 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:35 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:35 smithi196 ceph-mon[33018]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 14 op/s 2024-02-05T08:05:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:35 smithi196 ceph-mon[33018]: Adjusting osd_memory_target on smithi134 to 3760M 2024-02-05T08:05:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:35 smithi196 ceph-mon[33018]: Updating smithi134:/etc/ceph/ceph.conf 2024-02-05T08:05:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:35 smithi196 ceph-mon[33018]: Updating smithi196:/etc/ceph/ceph.conf 2024-02-05T08:05:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:35 smithi196 ceph-mon[33018]: Updating smithi134:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:05:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:35 smithi196 ceph-mon[33018]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T08:05:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:35 smithi196 ceph-mon[33018]: Updating smithi134:/etc/ceph/ceph.client.admin.keyring 2024-02-05T08:05:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:35 smithi196 ceph-mon[33018]: osdmap e85: 8 total, 8 up, 8 in 2024-02-05T08:05:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:35 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:35 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:35 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:35 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:35 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:05:36.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:36 smithi134 ceph-mon[30086]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-02-05T08:05:36.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:36 smithi134 ceph-mon[30086]: Updating smithi134:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T08:05:36.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:36 smithi134 ceph-mon[30086]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T08:05:36.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:36 smithi134 ceph-mon[30086]: Deploying daemon alertmanager.a on smithi134 2024-02-05T08:05:36.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1707693591' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T08:05:36.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:36 smithi134 ceph-mon[39278]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-02-05T08:05:36.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:36 smithi134 ceph-mon[39278]: Updating smithi134:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T08:05:36.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:36 smithi134 ceph-mon[39278]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T08:05:36.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:36 smithi134 ceph-mon[39278]: Deploying daemon alertmanager.a on smithi134 2024-02-05T08:05:36.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1707693591' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T08:05:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:36 smithi196 ceph-mon[33018]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-02-05T08:05:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:36 smithi196 ceph-mon[33018]: Updating smithi134:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T08:05:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:36 smithi196 ceph-mon[33018]: Updating smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T08:05:36.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:36 smithi196 ceph-mon[33018]: Deploying daemon alertmanager.a on smithi134 2024-02-05T08:05:36.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1707693591' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T08:05:37.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:37 smithi134 ceph-mon[30086]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-05T08:05:37.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:37 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:05:37.400 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:37 smithi134 ceph-mon[39278]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-05T08:05:37.400 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:37 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:05:37.428 INFO:teuthology.orchestra.run.smithi134.stdout:137438953502 2024-02-05T08:05:37.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-05T08:05:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:37 smithi196 ceph-mon[33018]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-02-05T08:05:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:37 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:05:38.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:05:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:05:39.378 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:39 smithi134 ceph-mon[30086]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:05:39.378 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:39 smithi134 ceph-mon[39278]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:05:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:39 smithi196 ceph-mon[33018]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:05:39.941 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574894 for osd.1 2024-02-05T08:05:39.942 DEBUG:teuthology.parallel:result is None 2024-02-05T08:05:41.307 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:41 smithi134 ceph-mon[30086]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T08:05:41.307 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:41 smithi134 ceph-mon[39278]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T08:05:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:41 smithi196 ceph-mon[33018]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T08:05:42.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:42 smithi196 ceph-mon[33018]: 2.1 scrub starts 2024-02-05T08:05:42.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:42 smithi196 ceph-mon[33018]: 2.1 scrub ok 2024-02-05T08:05:42.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:42 smithi134 ceph-mon[30086]: 2.1 scrub starts 2024-02-05T08:05:42.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:42 smithi134 ceph-mon[30086]: 2.1 scrub ok 2024-02-05T08:05:42.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:42 smithi134 ceph-mon[39278]: 2.1 scrub starts 2024-02-05T08:05:42.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:42 smithi134 ceph-mon[39278]: 2.1 scrub ok 2024-02-05T08:05:42.802 INFO:teuthology.orchestra.run.smithi134.stdout:115964117029 2024-02-05T08:05:42.802 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-05T08:05:43.284 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:05:42 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:05:43.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:43 smithi134 ceph-mon[30086]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-02-05T08:05:43.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:43 smithi134 ceph-mon[30086]: 3.1e scrub starts 2024-02-05T08:05:43.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:43 smithi134 ceph-mon[30086]: 3.1e scrub ok 2024-02-05T08:05:43.440 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:43 smithi134 ceph-mon[39278]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-02-05T08:05:43.440 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:43 smithi134 ceph-mon[39278]: 3.1e scrub starts 2024-02-05T08:05:43.440 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:43 smithi134 ceph-mon[39278]: 3.1e scrub ok 2024-02-05T08:05:43.558 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:43 smithi196 ceph-mon[33018]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-02-05T08:05:43.558 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:43 smithi196 ceph-mon[33018]: 3.1e scrub starts 2024-02-05T08:05:43.558 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:43 smithi196 ceph-mon[33018]: 3.1e scrub ok 2024-02-05T08:05:43.664 INFO:teuthology.orchestra.run.smithi134.stdout:219043332118 2024-02-05T08:05:43.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-05T08:05:44.314 INFO:teuthology.orchestra.run.smithi134.stdout:180388626458 2024-02-05T08:05:44.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-05T08:05:44.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:44 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:44.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:44 smithi196 ceph-mon[33018]: 4.4 deep-scrub starts 2024-02-05T08:05:44.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:44 smithi196 ceph-mon[33018]: 4.4 deep-scrub ok 2024-02-05T08:05:44.646 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:44 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:44.647 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:44 smithi134 ceph-mon[30086]: 4.4 deep-scrub starts 2024-02-05T08:05:44.647 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:44 smithi134 ceph-mon[30086]: 4.4 deep-scrub ok 2024-02-05T08:05:44.647 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:44 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:44.647 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:44 smithi134 ceph-mon[39278]: 4.4 deep-scrub starts 2024-02-05T08:05:44.647 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:44 smithi134 ceph-mon[39278]: 4.4 deep-scrub ok 2024-02-05T08:05:44.780 INFO:teuthology.orchestra.run.smithi134.stdout:77309411371 2024-02-05T08:05:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:45 smithi196 ceph-mon[33018]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:05:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:45 smithi196 ceph-mon[33018]: 2.0 scrub starts 2024-02-05T08:05:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:45 smithi196 ceph-mon[33018]: 2.0 scrub ok 2024-02-05T08:05:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:45 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:05:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:45 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:05:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:45 smithi196 ceph-mon[33018]: osdmap e86: 8 total, 8 up, 8 in 2024-02-05T08:05:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:45 smithi196 ceph-mon[33018]: osdmap e87: 8 total, 8 up, 8 in 2024-02-05T08:05:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:45 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/911887484' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T08:05:45.704 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[30086]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:05:45.704 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[30086]: 2.0 scrub starts 2024-02-05T08:05:45.704 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[30086]: 2.0 scrub ok 2024-02-05T08:05:45.704 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:05:45.704 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:05:45.704 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[30086]: osdmap e86: 8 total, 8 up, 8 in 2024-02-05T08:05:45.704 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[30086]: osdmap e87: 8 total, 8 up, 8 in 2024-02-05T08:05:45.704 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/911887484' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T08:05:45.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[39278]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:05:45.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[39278]: 2.0 scrub starts 2024-02-05T08:05:45.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[39278]: 2.0 scrub ok 2024-02-05T08:05:45.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:05:45.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:05:45.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[39278]: osdmap e86: 8 total, 8 up, 8 in 2024-02-05T08:05:45.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[39278]: osdmap e87: 8 total, 8 up, 8 in 2024-02-05T08:05:45.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:45 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/911887484' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T08:05:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:46 smithi196 ceph-mon[33018]: 4.7 scrub starts 2024-02-05T08:05:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:46 smithi196 ceph-mon[33018]: 4.7 scrub ok 2024-02-05T08:05:46.638 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:46 smithi134 ceph-mon[30086]: 4.7 scrub starts 2024-02-05T08:05:46.638 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:46 smithi134 ceph-mon[30086]: 4.7 scrub ok 2024-02-05T08:05:46.638 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:46 smithi134 ceph-mon[39278]: 4.7 scrub starts 2024-02-05T08:05:46.638 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:46 smithi134 ceph-mon[39278]: 4.7 scrub ok 2024-02-05T08:05:46.669 INFO:teuthology.orchestra.run.smithi134.stdout:257698037777 2024-02-05T08:05:47.262 INFO:teuthology.orchestra.run.smithi134.stdout:34359738421 2024-02-05T08:05:47.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-05T08:05:47.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[30086]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:05:47.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[30086]: 3.13 scrub starts 2024-02-05T08:05:47.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[30086]: 3.13 scrub ok 2024-02-05T08:05:47.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[30086]: 4.1f scrub starts 2024-02-05T08:05:47.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[30086]: 4.1f scrub ok 2024-02-05T08:05:47.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[30086]: 3.1c deep-scrub starts 2024-02-05T08:05:47.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[30086]: 3.1c deep-scrub ok 2024-02-05T08:05:47.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3624307734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T08:05:47.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[30086]: 4.1c scrub starts 2024-02-05T08:05:47.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[30086]: 4.1c scrub ok 2024-02-05T08:05:47.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[30086]: 5.12 scrub starts 2024-02-05T08:05:47.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[30086]: 5.12 scrub ok 2024-02-05T08:05:47.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[39278]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:05:47.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[39278]: 3.13 scrub starts 2024-02-05T08:05:47.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[39278]: 3.13 scrub ok 2024-02-05T08:05:47.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[39278]: 4.1f scrub starts 2024-02-05T08:05:47.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[39278]: 4.1f scrub ok 2024-02-05T08:05:47.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[39278]: 3.1c deep-scrub starts 2024-02-05T08:05:47.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[39278]: 3.1c deep-scrub ok 2024-02-05T08:05:47.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3624307734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T08:05:47.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[39278]: 4.1c scrub starts 2024-02-05T08:05:47.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[39278]: 4.1c scrub ok 2024-02-05T08:05:47.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[39278]: 5.12 scrub starts 2024-02-05T08:05:47.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-mon[39278]: 5.12 scrub ok 2024-02-05T08:05:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:47 smithi196 ceph-mon[33018]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:05:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:47 smithi196 ceph-mon[33018]: 3.13 scrub starts 2024-02-05T08:05:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:47 smithi196 ceph-mon[33018]: 3.13 scrub ok 2024-02-05T08:05:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:47 smithi196 ceph-mon[33018]: 4.1f scrub starts 2024-02-05T08:05:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:47 smithi196 ceph-mon[33018]: 4.1f scrub ok 2024-02-05T08:05:47.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:47 smithi196 ceph-mon[33018]: 3.1c deep-scrub starts 2024-02-05T08:05:47.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:47 smithi196 ceph-mon[33018]: 3.1c deep-scrub ok 2024-02-05T08:05:47.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:47 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3624307734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T08:05:47.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:47 smithi196 ceph-mon[33018]: 4.1c scrub starts 2024-02-05T08:05:47.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:47 smithi196 ceph-mon[33018]: 4.1c scrub ok 2024-02-05T08:05:47.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:47 smithi196 ceph-mon[33018]: 5.12 scrub starts 2024-02-05T08:05:47.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:47 smithi196 ceph-mon[33018]: 5.12 scrub ok 2024-02-05T08:05:47.838 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:05:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:05:48.049 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411371 for osd.2 2024-02-05T08:05:48.050 DEBUG:teuthology.parallel:result is None 2024-02-05T08:05:48.276 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:48.276 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:48.276 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:49.262 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:49.294 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037774 got 257698037777 for osd.7 2024-02-05T08:05:49.294 DEBUG:teuthology.parallel:result is None 2024-02-05T08:05:49.305 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:05:49.466 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[30086]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:05:49.466 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[30086]: 3.9 scrub starts 2024-02-05T08:05:49.466 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[30086]: 3.9 scrub ok 2024-02-05T08:05:49.466 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[30086]: 3.7 scrub starts 2024-02-05T08:05:49.467 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[30086]: 3.7 scrub ok 2024-02-05T08:05:49.467 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[30086]: 6.d scrub starts 2024-02-05T08:05:49.467 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[30086]: 6.d scrub ok 2024-02-05T08:05:49.467 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:05:49.467 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:05:49.467 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[30086]: 4.12 scrub starts 2024-02-05T08:05:49.467 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[30086]: 4.12 scrub ok 2024-02-05T08:05:49.467 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[39278]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:05:49.467 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[39278]: 3.9 scrub starts 2024-02-05T08:05:49.467 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[39278]: 3.9 scrub ok 2024-02-05T08:05:49.467 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[39278]: 3.7 scrub starts 2024-02-05T08:05:49.468 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[39278]: 3.7 scrub ok 2024-02-05T08:05:49.468 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[39278]: 6.d scrub starts 2024-02-05T08:05:49.468 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[39278]: 6.d scrub ok 2024-02-05T08:05:49.468 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:05:49.468 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:05:49.468 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[39278]: 4.12 scrub starts 2024-02-05T08:05:49.468 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:49 smithi134 ceph-mon[39278]: 4.12 scrub ok 2024-02-05T08:05:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:49 smithi196 ceph-mon[33018]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:05:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:49 smithi196 ceph-mon[33018]: 3.9 scrub starts 2024-02-05T08:05:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:49 smithi196 ceph-mon[33018]: 3.9 scrub ok 2024-02-05T08:05:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:49 smithi196 ceph-mon[33018]: 3.7 scrub starts 2024-02-05T08:05:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:49 smithi196 ceph-mon[33018]: 3.7 scrub ok 2024-02-05T08:05:49.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:49 smithi196 ceph-mon[33018]: 6.d scrub starts 2024-02-05T08:05:49.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:49 smithi196 ceph-mon[33018]: 6.d scrub ok 2024-02-05T08:05:49.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:49 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:05:49.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:49 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:05:49.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:49 smithi196 ceph-mon[33018]: 4.12 scrub starts 2024-02-05T08:05:49.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:49 smithi196 ceph-mon[33018]: 4.12 scrub ok 2024-02-05T08:05:50.542 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:50 smithi134 ceph-mon[30086]: 5.10 deep-scrub starts 2024-02-05T08:05:50.543 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:50 smithi134 ceph-mon[30086]: 5.10 deep-scrub ok 2024-02-05T08:05:50.543 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:50 smithi134 ceph-mon[30086]: 3.10 scrub starts 2024-02-05T08:05:50.543 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:50 smithi134 ceph-mon[30086]: 3.10 scrub ok 2024-02-05T08:05:50.543 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:50 smithi134 ceph-mon[39278]: 5.10 deep-scrub starts 2024-02-05T08:05:50.543 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:50 smithi134 ceph-mon[39278]: 5.10 deep-scrub ok 2024-02-05T08:05:50.543 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:50 smithi134 ceph-mon[39278]: 3.10 scrub starts 2024-02-05T08:05:50.543 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:50 smithi134 ceph-mon[39278]: 3.10 scrub ok 2024-02-05T08:05:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:50 smithi196 ceph-mon[33018]: 5.10 deep-scrub starts 2024-02-05T08:05:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:50 smithi196 ceph-mon[33018]: 5.10 deep-scrub ok 2024-02-05T08:05:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:50 smithi196 ceph-mon[33018]: 3.10 scrub starts 2024-02-05T08:05:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:50 smithi196 ceph-mon[33018]: 3.10 scrub ok 2024-02-05T08:05:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:51 smithi196 ceph-mon[33018]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:05:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:51 smithi196 ceph-mon[33018]: 3.a deep-scrub starts 2024-02-05T08:05:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:51 smithi196 ceph-mon[33018]: 3.a deep-scrub ok 2024-02-05T08:05:51.596 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:51 smithi134 ceph-mon[30086]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:05:51.596 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:51 smithi134 ceph-mon[30086]: 3.a deep-scrub starts 2024-02-05T08:05:51.596 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:51 smithi134 ceph-mon[30086]: 3.a deep-scrub ok 2024-02-05T08:05:51.596 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:51 smithi134 ceph-mon[39278]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:05:51.596 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:51 smithi134 ceph-mon[39278]: 3.a deep-scrub starts 2024-02-05T08:05:51.596 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:51 smithi134 ceph-mon[39278]: 3.a deep-scrub ok 2024-02-05T08:05:52.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:52 smithi134 ceph-mon[30086]: 6.1e scrub starts 2024-02-05T08:05:52.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:52 smithi134 ceph-mon[30086]: 6.1e scrub ok 2024-02-05T08:05:52.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:52 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:05:52.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:52 smithi134 ceph-mon[30086]: 3.4 scrub starts 2024-02-05T08:05:52.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:52 smithi134 ceph-mon[30086]: 3.4 scrub ok 2024-02-05T08:05:52.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:52 smithi134 ceph-mon[39278]: 6.1e scrub starts 2024-02-05T08:05:52.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:52 smithi134 ceph-mon[39278]: 6.1e scrub ok 2024-02-05T08:05:52.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:52 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:05:52.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:52 smithi134 ceph-mon[39278]: 3.4 scrub starts 2024-02-05T08:05:52.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:52 smithi134 ceph-mon[39278]: 3.4 scrub ok 2024-02-05T08:05:52.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:52 smithi196 ceph-mon[33018]: 6.1e scrub starts 2024-02-05T08:05:52.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:52 smithi196 ceph-mon[33018]: 6.1e scrub ok 2024-02-05T08:05:52.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:52 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:05:52.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:52 smithi196 ceph-mon[33018]: 3.4 scrub starts 2024-02-05T08:05:52.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:52 smithi196 ceph-mon[33018]: 3.4 scrub ok 2024-02-05T08:05:53.312 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:05:52 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:05:53.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:53 smithi196 ceph-mon[33018]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:05:53.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:53 smithi196 ceph-mon[33018]: 4.0 scrub starts 2024-02-05T08:05:53.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:53 smithi196 ceph-mon[33018]: 4.0 scrub ok 2024-02-05T08:05:53.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:53 smithi196 ceph-mon[33018]: 3.6 scrub starts 2024-02-05T08:05:53.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:53 smithi196 ceph-mon[33018]: 3.6 scrub ok 2024-02-05T08:05:53.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:53 smithi196 ceph-mon[33018]: 2.2 deep-scrub starts 2024-02-05T08:05:53.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:53 smithi196 ceph-mon[33018]: 2.2 deep-scrub ok 2024-02-05T08:05:53.634 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[30086]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:05:53.634 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[30086]: 4.0 scrub starts 2024-02-05T08:05:53.634 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[30086]: 4.0 scrub ok 2024-02-05T08:05:53.634 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[30086]: 3.6 scrub starts 2024-02-05T08:05:53.634 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[30086]: 3.6 scrub ok 2024-02-05T08:05:53.634 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[30086]: 2.2 deep-scrub starts 2024-02-05T08:05:53.634 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[30086]: 2.2 deep-scrub ok 2024-02-05T08:05:53.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[39278]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:05:53.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[39278]: 4.0 scrub starts 2024-02-05T08:05:53.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[39278]: 4.0 scrub ok 2024-02-05T08:05:53.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[39278]: 3.6 scrub starts 2024-02-05T08:05:53.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[39278]: 3.6 scrub ok 2024-02-05T08:05:53.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[39278]: 2.2 deep-scrub starts 2024-02-05T08:05:53.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:53 smithi134 ceph-mon[39278]: 2.2 deep-scrub ok 2024-02-05T08:05:54.581 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:54 smithi196 ceph-mon[33018]: 5.1a scrub starts 2024-02-05T08:05:54.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:54 smithi196 ceph-mon[33018]: 5.1a scrub ok 2024-02-05T08:05:54.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:54 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:54.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:54 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:05:54.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:54 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:05:54.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:54 smithi196 ceph-mon[33018]: 3.14 scrub starts 2024-02-05T08:05:54.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:54 smithi196 ceph-mon[33018]: 3.14 scrub ok 2024-02-05T08:05:54.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:54 smithi196 ceph-mon[33018]: 3.1f scrub starts 2024-02-05T08:05:54.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:54 smithi196 ceph-mon[33018]: 3.1f scrub ok 2024-02-05T08:05:54.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:54 smithi196 ceph-mon[33018]: osdmap e88: 8 total, 8 up, 8 in 2024-02-05T08:05:54.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:54 smithi196 ceph-mon[33018]: 4.f scrub starts 2024-02-05T08:05:54.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:54 smithi196 ceph-mon[33018]: 4.f scrub ok 2024-02-05T08:05:54.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[30086]: 5.1a scrub starts 2024-02-05T08:05:54.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[30086]: 5.1a scrub ok 2024-02-05T08:05:54.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:54.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:05:54.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:05:54.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[30086]: 3.14 scrub starts 2024-02-05T08:05:54.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[30086]: 3.14 scrub ok 2024-02-05T08:05:54.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[30086]: 3.1f scrub starts 2024-02-05T08:05:54.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[30086]: 3.1f scrub ok 2024-02-05T08:05:54.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[30086]: osdmap e88: 8 total, 8 up, 8 in 2024-02-05T08:05:54.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[30086]: 4.f scrub starts 2024-02-05T08:05:54.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[30086]: 4.f scrub ok 2024-02-05T08:05:54.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[39278]: 5.1a scrub starts 2024-02-05T08:05:54.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[39278]: 5.1a scrub ok 2024-02-05T08:05:54.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:05:54.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:05:54.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:05:54.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[39278]: 3.14 scrub starts 2024-02-05T08:05:54.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[39278]: 3.14 scrub ok 2024-02-05T08:05:54.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[39278]: 3.1f scrub starts 2024-02-05T08:05:54.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[39278]: 3.1f scrub ok 2024-02-05T08:05:54.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[39278]: osdmap e88: 8 total, 8 up, 8 in 2024-02-05T08:05:54.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[39278]: 4.f scrub starts 2024-02-05T08:05:54.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:54 smithi134 ceph-mon[39278]: 4.f scrub ok 2024-02-05T08:05:55.455 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:55 smithi134 ceph-mon[30086]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:05:55.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:55 smithi134 ceph-mon[30086]: 3.d scrub starts 2024-02-05T08:05:55.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:55 smithi134 ceph-mon[30086]: 3.d scrub ok 2024-02-05T08:05:55.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:55 smithi134 ceph-mon[30086]: 5.11 deep-scrub starts 2024-02-05T08:05:55.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:55 smithi134 ceph-mon[30086]: 5.11 deep-scrub ok 2024-02-05T08:05:55.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:55 smithi134 ceph-mon[30086]: osdmap e89: 8 total, 8 up, 8 in 2024-02-05T08:05:55.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:55 smithi134 ceph-mon[39278]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:05:55.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:55 smithi134 ceph-mon[39278]: 3.d scrub starts 2024-02-05T08:05:55.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:55 smithi134 ceph-mon[39278]: 3.d scrub ok 2024-02-05T08:05:55.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:55 smithi134 ceph-mon[39278]: 5.11 deep-scrub starts 2024-02-05T08:05:55.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:55 smithi134 ceph-mon[39278]: 5.11 deep-scrub ok 2024-02-05T08:05:55.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:55 smithi134 ceph-mon[39278]: osdmap e89: 8 total, 8 up, 8 in 2024-02-05T08:05:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:55 smithi196 ceph-mon[33018]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:05:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:55 smithi196 ceph-mon[33018]: 3.d scrub starts 2024-02-05T08:05:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:55 smithi196 ceph-mon[33018]: 3.d scrub ok 2024-02-05T08:05:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:55 smithi196 ceph-mon[33018]: 5.11 deep-scrub starts 2024-02-05T08:05:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:55 smithi196 ceph-mon[33018]: 5.11 deep-scrub ok 2024-02-05T08:05:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:55 smithi196 ceph-mon[33018]: osdmap e89: 8 total, 8 up, 8 in 2024-02-05T08:05:56.385 INFO:teuthology.orchestra.run.smithi134.stdout:115964117032 2024-02-05T08:05:56.769 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[30086]: 6.1a scrub starts 2024-02-05T08:05:56.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[30086]: 6.1a scrub ok 2024-02-05T08:05:56.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[30086]: 4.3 scrub starts 2024-02-05T08:05:56.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[30086]: 4.3 scrub ok 2024-02-05T08:05:56.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[30086]: 3.1d scrub starts 2024-02-05T08:05:56.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[30086]: 3.1d scrub ok 2024-02-05T08:05:56.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[30086]: 5.b scrub starts 2024-02-05T08:05:56.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[30086]: 5.b scrub ok 2024-02-05T08:05:56.770 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[39278]: 6.1a scrub starts 2024-02-05T08:05:56.770 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[39278]: 6.1a scrub ok 2024-02-05T08:05:56.770 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[39278]: 4.3 scrub starts 2024-02-05T08:05:56.770 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[39278]: 4.3 scrub ok 2024-02-05T08:05:56.770 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[39278]: 3.1d scrub starts 2024-02-05T08:05:56.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[39278]: 3.1d scrub ok 2024-02-05T08:05:56.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[39278]: 5.b scrub starts 2024-02-05T08:05:56.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:56 smithi134 ceph-mon[39278]: 5.b scrub ok 2024-02-05T08:05:56.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:56 smithi196 ceph-mon[33018]: 6.1a scrub starts 2024-02-05T08:05:56.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:56 smithi196 ceph-mon[33018]: 6.1a scrub ok 2024-02-05T08:05:56.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:56 smithi196 ceph-mon[33018]: 4.3 scrub starts 2024-02-05T08:05:56.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:56 smithi196 ceph-mon[33018]: 4.3 scrub ok 2024-02-05T08:05:56.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:56 smithi196 ceph-mon[33018]: 3.1d scrub starts 2024-02-05T08:05:56.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:56 smithi196 ceph-mon[33018]: 3.1d scrub ok 2024-02-05T08:05:56.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:56 smithi196 ceph-mon[33018]: 5.b scrub starts 2024-02-05T08:05:56.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:56 smithi196 ceph-mon[33018]: 5.b scrub ok 2024-02-05T08:05:57.455 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[30086]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:05:57.455 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[30086]: 6.1c scrub starts 2024-02-05T08:05:57.455 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[30086]: 6.1c scrub ok 2024-02-05T08:05:57.455 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:05:57.455 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:05:57.455 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[30086]: 6.18 scrub starts 2024-02-05T08:05:57.455 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[30086]: 6.18 scrub ok 2024-02-05T08:05:57.455 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[30086]: osdmap e90: 8 total, 8 up, 8 in 2024-02-05T08:05:57.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2142416363' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T08:05:57.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[30086]: 4.15 scrub starts 2024-02-05T08:05:57.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[30086]: 4.15 scrub ok 2024-02-05T08:05:57.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[39278]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:05:57.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[39278]: 6.1c scrub starts 2024-02-05T08:05:57.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[39278]: 6.1c scrub ok 2024-02-05T08:05:57.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:05:57.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:05:57.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[39278]: 6.18 scrub starts 2024-02-05T08:05:57.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[39278]: 6.18 scrub ok 2024-02-05T08:05:57.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[39278]: osdmap e90: 8 total, 8 up, 8 in 2024-02-05T08:05:57.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2142416363' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T08:05:57.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[39278]: 4.15 scrub starts 2024-02-05T08:05:57.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-mon[39278]: 4.15 scrub ok 2024-02-05T08:05:57.666 INFO:teuthology.orchestra.run.smithi134.stdout:180388626461 2024-02-05T08:05:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:57 smithi196 ceph-mon[33018]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:05:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:57 smithi196 ceph-mon[33018]: 6.1c scrub starts 2024-02-05T08:05:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:57 smithi196 ceph-mon[33018]: 6.1c scrub ok 2024-02-05T08:05:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:57 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:05:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:57 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:05:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:57 smithi196 ceph-mon[33018]: 6.18 scrub starts 2024-02-05T08:05:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:57 smithi196 ceph-mon[33018]: 6.18 scrub ok 2024-02-05T08:05:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:57 smithi196 ceph-mon[33018]: osdmap e90: 8 total, 8 up, 8 in 2024-02-05T08:05:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:57 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2142416363' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T08:05:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:57 smithi196 ceph-mon[33018]: 4.15 scrub starts 2024-02-05T08:05:57.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:57 smithi196 ceph-mon[33018]: 4.15 scrub ok 2024-02-05T08:05:57.988 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:05:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:05:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:05:58.623 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[30086]: 6.15 scrub starts 2024-02-05T08:05:58.623 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[30086]: 6.15 scrub ok 2024-02-05T08:05:58.623 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[30086]: 5.1c deep-scrub starts 2024-02-05T08:05:58.623 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[30086]: 5.1c deep-scrub ok 2024-02-05T08:05:58.623 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[30086]: 5.3 scrub starts 2024-02-05T08:05:58.623 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[30086]: 5.3 scrub ok 2024-02-05T08:05:58.623 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[30086]: 5.13 scrub starts 2024-02-05T08:05:58.623 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[30086]: 5.13 scrub ok 2024-02-05T08:05:58.623 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1783794040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T08:05:58.623 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[39278]: 6.15 scrub starts 2024-02-05T08:05:58.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[39278]: 6.15 scrub ok 2024-02-05T08:05:58.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[39278]: 5.1c deep-scrub starts 2024-02-05T08:05:58.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[39278]: 5.1c deep-scrub ok 2024-02-05T08:05:58.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[39278]: 5.3 scrub starts 2024-02-05T08:05:58.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[39278]: 5.3 scrub ok 2024-02-05T08:05:58.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[39278]: 5.13 scrub starts 2024-02-05T08:05:58.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[39278]: 5.13 scrub ok 2024-02-05T08:05:58.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:58 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1783794040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T08:05:58.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:58 smithi196 ceph-mon[33018]: 6.15 scrub starts 2024-02-05T08:05:58.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:58 smithi196 ceph-mon[33018]: 6.15 scrub ok 2024-02-05T08:05:58.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:58 smithi196 ceph-mon[33018]: 5.1c deep-scrub starts 2024-02-05T08:05:58.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:58 smithi196 ceph-mon[33018]: 5.1c deep-scrub ok 2024-02-05T08:05:58.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:58 smithi196 ceph-mon[33018]: 5.3 scrub starts 2024-02-05T08:05:58.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:58 smithi196 ceph-mon[33018]: 5.3 scrub ok 2024-02-05T08:05:58.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:58 smithi196 ceph-mon[33018]: 5.13 scrub starts 2024-02-05T08:05:58.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:58 smithi196 ceph-mon[33018]: 5.13 scrub ok 2024-02-05T08:05:58.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:58 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1783794040' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T08:05:59.384 INFO:teuthology.orchestra.run.smithi134.stdout:219043332122 2024-02-05T08:05:59.689 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[30086]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T08:05:59.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[30086]: 3.f scrub starts 2024-02-05T08:05:59.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[30086]: 3.f scrub ok 2024-02-05T08:05:59.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[30086]: 4.18 deep-scrub starts 2024-02-05T08:05:59.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[30086]: 4.18 deep-scrub ok 2024-02-05T08:05:59.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[30086]: 6.0 scrub starts 2024-02-05T08:05:59.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[30086]: 6.0 scrub ok 2024-02-05T08:05:59.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[30086]: 4.1d deep-scrub starts 2024-02-05T08:05:59.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[30086]: 4.1d deep-scrub ok 2024-02-05T08:05:59.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:05:59.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:05:59.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[39278]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T08:05:59.691 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[39278]: 3.f scrub starts 2024-02-05T08:05:59.691 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[39278]: 3.f scrub ok 2024-02-05T08:05:59.691 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[39278]: 4.18 deep-scrub starts 2024-02-05T08:05:59.691 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[39278]: 4.18 deep-scrub ok 2024-02-05T08:05:59.691 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[39278]: 6.0 scrub starts 2024-02-05T08:05:59.691 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[39278]: 6.0 scrub ok 2024-02-05T08:05:59.691 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[39278]: 4.1d deep-scrub starts 2024-02-05T08:05:59.691 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[39278]: 4.1d deep-scrub ok 2024-02-05T08:05:59.691 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:05:59.691 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:05:59 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:05:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:59 smithi196 ceph-mon[33018]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T08:05:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:59 smithi196 ceph-mon[33018]: 3.f scrub starts 2024-02-05T08:05:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:59 smithi196 ceph-mon[33018]: 3.f scrub ok 2024-02-05T08:05:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:59 smithi196 ceph-mon[33018]: 4.18 deep-scrub starts 2024-02-05T08:05:59.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:59 smithi196 ceph-mon[33018]: 4.18 deep-scrub ok 2024-02-05T08:05:59.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:59 smithi196 ceph-mon[33018]: 6.0 scrub starts 2024-02-05T08:05:59.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:59 smithi196 ceph-mon[33018]: 6.0 scrub ok 2024-02-05T08:05:59.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:59 smithi196 ceph-mon[33018]: 4.1d deep-scrub starts 2024-02-05T08:05:59.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:59 smithi196 ceph-mon[33018]: 4.1d deep-scrub ok 2024-02-05T08:05:59.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:59 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:05:59.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:05:59 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:06:00.102 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332118 got 219043332122 for osd.6 2024-02-05T08:06:00.103 DEBUG:teuthology.parallel:result is None 2024-02-05T08:06:00.494 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117029 got 115964117032 for osd.3 2024-02-05T08:06:00.494 DEBUG:teuthology.parallel:result is None 2024-02-05T08:06:00.660 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:00 smithi134 ceph-mon[39278]: 3.8 scrub starts 2024-02-05T08:06:00.660 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:00 smithi134 ceph-mon[39278]: 3.8 scrub ok 2024-02-05T08:06:00.660 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:00 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1334131477' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T08:06:00.660 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:00 smithi134 ceph-mon[39278]: 6.a scrub starts 2024-02-05T08:06:00.660 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:00 smithi134 ceph-mon[39278]: 6.a scrub ok 2024-02-05T08:06:00.660 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:00 smithi134 ceph-mon[39278]: osdmap e91: 8 total, 8 up, 8 in 2024-02-05T08:06:00.661 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:00 smithi134 ceph-mon[30086]: 3.8 scrub starts 2024-02-05T08:06:00.661 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:00 smithi134 ceph-mon[30086]: 3.8 scrub ok 2024-02-05T08:06:00.661 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:00 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1334131477' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T08:06:00.661 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:00 smithi134 ceph-mon[30086]: 6.a scrub starts 2024-02-05T08:06:00.661 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:00 smithi134 ceph-mon[30086]: 6.a scrub ok 2024-02-05T08:06:00.661 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:00 smithi134 ceph-mon[30086]: osdmap e91: 8 total, 8 up, 8 in 2024-02-05T08:06:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:00 smithi196 ceph-mon[33018]: 3.8 scrub starts 2024-02-05T08:06:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:00 smithi196 ceph-mon[33018]: 3.8 scrub ok 2024-02-05T08:06:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:00 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1334131477' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T08:06:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:00 smithi196 ceph-mon[33018]: 6.a scrub starts 2024-02-05T08:06:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:00 smithi196 ceph-mon[33018]: 6.a scrub ok 2024-02-05T08:06:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:00 smithi196 ceph-mon[33018]: osdmap e91: 8 total, 8 up, 8 in 2024-02-05T08:06:00.870 INFO:teuthology.orchestra.run.smithi134.stdout:34359738424 2024-02-05T08:06:01.449 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[30086]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-02-05T08:06:01.449 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[30086]: 5.9 scrub starts 2024-02-05T08:06:01.449 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[30086]: 5.9 scrub ok 2024-02-05T08:06:01.449 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[30086]: 6.1b scrub starts 2024-02-05T08:06:01.449 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[30086]: 6.1b scrub ok 2024-02-05T08:06:01.449 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[30086]: 4.5 scrub starts 2024-02-05T08:06:01.450 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[30086]: 4.5 scrub ok 2024-02-05T08:06:01.450 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/417690463' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T08:06:01.450 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[39278]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-02-05T08:06:01.450 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[39278]: 5.9 scrub starts 2024-02-05T08:06:01.450 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[39278]: 5.9 scrub ok 2024-02-05T08:06:01.450 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[39278]: 6.1b scrub starts 2024-02-05T08:06:01.450 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[39278]: 6.1b scrub ok 2024-02-05T08:06:01.450 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[39278]: 4.5 scrub starts 2024-02-05T08:06:01.450 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[39278]: 4.5 scrub ok 2024-02-05T08:06:01.450 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:01 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/417690463' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T08:06:01.476 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738421 got 34359738424 for osd.0 2024-02-05T08:06:01.477 DEBUG:teuthology.parallel:result is None 2024-02-05T08:06:01.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:01 smithi196 ceph-mon[33018]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-02-05T08:06:01.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:01 smithi196 ceph-mon[33018]: 5.9 scrub starts 2024-02-05T08:06:01.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:01 smithi196 ceph-mon[33018]: 5.9 scrub ok 2024-02-05T08:06:01.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:01 smithi196 ceph-mon[33018]: 6.1b scrub starts 2024-02-05T08:06:01.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:01 smithi196 ceph-mon[33018]: 6.1b scrub ok 2024-02-05T08:06:01.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:01 smithi196 ceph-mon[33018]: 4.5 scrub starts 2024-02-05T08:06:01.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:01 smithi196 ceph-mon[33018]: 4.5 scrub ok 2024-02-05T08:06:01.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:01 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/417690463' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T08:06:01.865 INFO:teuthology.orchestra.run.smithi134.stdout:137438953508 2024-02-05T08:06:02.096 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626458 got 180388626461 for osd.5 2024-02-05T08:06:02.096 DEBUG:teuthology.parallel:result is None 2024-02-05T08:06:02.698 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:02 smithi134 ceph-mon[30086]: 6.2 scrub starts 2024-02-05T08:06:02.698 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:02 smithi134 ceph-mon[30086]: 6.2 scrub ok 2024-02-05T08:06:02.698 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:02 smithi134 ceph-mon[30086]: 5.f scrub starts 2024-02-05T08:06:02.698 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:02 smithi134 ceph-mon[30086]: 5.f scrub ok 2024-02-05T08:06:02.698 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:02 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/533602631' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T08:06:02.698 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:02 smithi134 ceph-mon[39278]: 6.2 scrub starts 2024-02-05T08:06:02.699 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:02 smithi134 ceph-mon[39278]: 6.2 scrub ok 2024-02-05T08:06:02.699 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:02 smithi134 ceph-mon[39278]: 5.f scrub starts 2024-02-05T08:06:02.699 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:02 smithi134 ceph-mon[39278]: 5.f scrub ok 2024-02-05T08:06:02.699 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:02 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/533602631' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T08:06:02.725 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953502 got 137438953508 for osd.4 2024-02-05T08:06:02.725 DEBUG:teuthology.parallel:result is None 2024-02-05T08:06:02.726 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-05T08:06:02.726 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:06:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:02 smithi196 ceph-mon[33018]: 6.2 scrub starts 2024-02-05T08:06:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:02 smithi196 ceph-mon[33018]: 6.2 scrub ok 2024-02-05T08:06:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:02 smithi196 ceph-mon[33018]: 5.f scrub starts 2024-02-05T08:06:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:02 smithi196 ceph-mon[33018]: 5.f scrub ok 2024-02-05T08:06:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:02 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/533602631' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T08:06:03.205 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:03.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:06:02 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:06:03.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[30086]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-02-05T08:06:03.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[30086]: 3.e scrub starts 2024-02-05T08:06:03.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[30086]: 3.e scrub ok 2024-02-05T08:06:03.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[30086]: 6.13 scrub starts 2024-02-05T08:06:03.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[30086]: 6.13 scrub ok 2024-02-05T08:06:03.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[30086]: 3.16 scrub starts 2024-02-05T08:06:03.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[30086]: 3.16 scrub ok 2024-02-05T08:06:03.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[39278]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-02-05T08:06:03.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[39278]: 3.e scrub starts 2024-02-05T08:06:03.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[39278]: 3.e scrub ok 2024-02-05T08:06:03.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[39278]: 6.13 scrub starts 2024-02-05T08:06:03.646 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[39278]: 6.13 scrub ok 2024-02-05T08:06:03.646 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[39278]: 3.16 scrub starts 2024-02-05T08:06:03.646 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:03 smithi134 ceph-mon[39278]: 3.16 scrub ok 2024-02-05T08:06:03.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:03 smithi196 ceph-mon[33018]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-02-05T08:06:03.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:03 smithi196 ceph-mon[33018]: 3.e scrub starts 2024-02-05T08:06:03.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:03 smithi196 ceph-mon[33018]: 3.e scrub ok 2024-02-05T08:06:03.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:03 smithi196 ceph-mon[33018]: 6.13 scrub starts 2024-02-05T08:06:03.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:03 smithi196 ceph-mon[33018]: 6.13 scrub ok 2024-02-05T08:06:03.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:03 smithi196 ceph-mon[33018]: 3.16 scrub starts 2024-02-05T08:06:03.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:03 smithi196 ceph-mon[33018]: 3.16 scrub ok 2024-02-05T08:06:04.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:04 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:04.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:04 smithi134 ceph-mon[30086]: 6.10 scrub starts 2024-02-05T08:06:04.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:04 smithi134 ceph-mon[30086]: 6.10 scrub ok 2024-02-05T08:06:04.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:04 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:04.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:04 smithi134 ceph-mon[39278]: 6.10 scrub starts 2024-02-05T08:06:04.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:04 smithi134 ceph-mon[39278]: 6.10 scrub ok 2024-02-05T08:06:04.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:04 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:04.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:04 smithi196 ceph-mon[33018]: 6.10 scrub starts 2024-02-05T08:06:04.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:04 smithi196 ceph-mon[33018]: 6.10 scrub ok 2024-02-05T08:06:05.427 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[30086]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[30086]: 6.8 scrub starts 2024-02-05T08:06:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[30086]: 6.8 scrub ok 2024-02-05T08:06:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[30086]: 5.17 scrub starts 2024-02-05T08:06:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[30086]: 5.17 scrub ok 2024-02-05T08:06:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[30086]: 4.b scrub starts 2024-02-05T08:06:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[30086]: 4.b scrub ok 2024-02-05T08:06:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[30086]: 5.d scrub starts 2024-02-05T08:06:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[30086]: 5.d scrub ok 2024-02-05T08:06:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[30086]: 5.16 scrub starts 2024-02-05T08:06:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[30086]: 5.16 scrub ok 2024-02-05T08:06:05.429 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[39278]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:05.429 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[39278]: 6.8 scrub starts 2024-02-05T08:06:05.429 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[39278]: 6.8 scrub ok 2024-02-05T08:06:05.429 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[39278]: 5.17 scrub starts 2024-02-05T08:06:05.429 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[39278]: 5.17 scrub ok 2024-02-05T08:06:05.429 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[39278]: 4.b scrub starts 2024-02-05T08:06:05.429 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[39278]: 4.b scrub ok 2024-02-05T08:06:05.429 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[39278]: 5.d scrub starts 2024-02-05T08:06:05.429 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[39278]: 5.d scrub ok 2024-02-05T08:06:05.429 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[39278]: 5.16 scrub starts 2024-02-05T08:06:05.429 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:05 smithi134 ceph-mon[39278]: 5.16 scrub ok 2024-02-05T08:06:05.777 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:05 smithi134 podman[77469]: 2024-02-05 08:06:05.65727807 +0000 UTC m=+0.449431879 container create 6843e7bbe75accc92cdc239d5e3a71215e76d7a9c70662bb801beb035db43e83 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T08:06:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:05 smithi196 ceph-mon[33018]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:05 smithi196 ceph-mon[33018]: 6.8 scrub starts 2024-02-05T08:06:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:05 smithi196 ceph-mon[33018]: 6.8 scrub ok 2024-02-05T08:06:05.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:05 smithi196 ceph-mon[33018]: 5.17 scrub starts 2024-02-05T08:06:05.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:05 smithi196 ceph-mon[33018]: 5.17 scrub ok 2024-02-05T08:06:05.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:05 smithi196 ceph-mon[33018]: 4.b scrub starts 2024-02-05T08:06:05.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:05 smithi196 ceph-mon[33018]: 4.b scrub ok 2024-02-05T08:06:05.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:05 smithi196 ceph-mon[33018]: 5.d scrub starts 2024-02-05T08:06:05.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:05 smithi196 ceph-mon[33018]: 5.d scrub ok 2024-02-05T08:06:05.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:05 smithi196 ceph-mon[33018]: 5.16 scrub starts 2024-02-05T08:06:05.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:05 smithi196 ceph-mon[33018]: 5.16 scrub ok 2024-02-05T08:06:06.264 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:06 smithi134 podman[77469]: 2024-02-05 08:06:06.219106473 +0000 UTC m=+1.011260241 container init 6843e7bbe75accc92cdc239d5e3a71215e76d7a9c70662bb801beb035db43e83 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T08:06:06.264 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:06 smithi134 podman[77469]: 2024-02-05 08:06:06.226723539 +0000 UTC m=+1.018877312 container start 6843e7bbe75accc92cdc239d5e3a71215e76d7a9c70662bb801beb035db43e83 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T08:06:06.264 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[77504]: ts=2024-02-05T08:06:06.238Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-05T08:06:06.264 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[77504]: ts=2024-02-05T08:06:06.238Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-05T08:06:06.264 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[77504]: ts=2024-02-05T08:06:06.239Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.134 port=9094 2024-02-05T08:06:06.264 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[77504]: ts=2024-02-05T08:06:06.240Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-05T08:06:06.264 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[77504]: ts=2024-02-05T08:06:06.264Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-05T08:06:06.264 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[77504]: ts=2024-02-05T08:06:06.264Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-05T08:06:06.322 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:06:06.326 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:06:06.521 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:06:06] ENGINE Bus STOPPING 2024-02-05T08:06:06.521 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[77504]: ts=2024-02-05T08:06:06.266Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-05T08:06:06.521 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[77504]: ts=2024-02-05T08:06:06.266Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-05T08:06:06.521 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:06 smithi134 bash[77469]: 6843e7bbe75accc92cdc239d5e3a71215e76d7a9c70662bb801beb035db43e83 2024-02-05T08:06:06.521 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:06 smithi134 systemd[1]: Started Ceph alertmanager.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:06:06.777 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:06:06] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T08:06:06.778 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:06:06] ENGINE Bus STOPPED 2024-02-05T08:06:06.778 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:06:06] ENGINE Bus STARTING 2024-02-05T08:06:06.778 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:06:06] ENGINE Serving on http://:::9283 2024-02-05T08:06:06.778 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:06 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:06:06] ENGINE Bus STARTED 2024-02-05T08:06:07.018 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-02-05T08:06:05.490243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1008,"num_read_kb":854,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":622,"ondisk_log_size":622,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":245908,"kb_used_data":8956,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749715308,"statfs":{"total":767960285184,"available":767708475392,"internally_reserved":0,"allocated":9170944,"data_stored":5261412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":242535987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.156673"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529806+0000","last_change":"2024-02-05T08:05:55.114592+0000","last_active":"2024-02-05T08:05:59.529806+0000","last_peered":"2024-02-05T08:05:59.529806+0000","last_clean":"2024-02-05T08:05:59.529806+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:05:59.529806+0000","last_undegraded":"2024-02-05T08:05:59.529806+0000","last_fullsized":"2024-02-05T08:05:59.529806+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:55.114498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.114498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.383006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527645+0000","last_change":"2024-02-05T08:04:45.560680+0000","last_active":"2024-02-05T08:05:59.527645+0000","last_peered":"2024-02-05T08:05:59.527645+0000","last_clean":"2024-02-05T08:05:59.527645+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:05:59.527645+0000","last_undegraded":"2024-02-05T08:05:59.527645+0000","last_fullsized":"2024-02-05T08:05:59.527645+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527961+0000","last_change":"2024-02-05T08:05:53.302720+0000","last_active":"2024-02-05T08:05:59.527961+0000","last_peered":"2024-02-05T08:05:59.527961+0000","last_clean":"2024-02-05T08:05:59.527961+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:05:59.527961+0000","last_undegraded":"2024-02-05T08:05:59.527961+0000","last_fullsized":"2024-02-05T08:05:59.527961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:53.302642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.302642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:57.391075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"90'13","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529707+0000","last_change":"2024-02-05T08:05:58.103861+0000","last_active":"2024-02-05T08:05:59.529707+0000","last_peered":"2024-02-05T08:05:59.529707+0000","last_clean":"2024-02-05T08:05:59.529707+0000","last_became_active":"2024-02-05T08:04:43.551169+0000","last_became_peered":"2024-02-05T08:04:43.551169+0000","last_unstale":"2024-02-05T08:05:59.529707+0000","last_undegraded":"2024-02-05T08:05:59.529707+0000","last_fullsized":"2024-02-05T08:05:59.529707+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.103749+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:14.381872+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.227351+0000","last_change":"2024-02-05T08:06:00.227371+0000","last_active":"2024-02-05T08:06:00.227351+0000","last_peered":"2024-02-05T08:06:00.227351+0000","last_clean":"2024-02-05T08:06:00.227351+0000","last_became_active":"2024-02-05T08:04:48.172811+0000","last_became_peered":"2024-02-05T08:04:48.172811+0000","last_unstale":"2024-02-05T08:06:00.227351+0000","last_undegraded":"2024-02-05T08:06:00.227351+0000","last_fullsized":"2024-02-05T08:06:00.227351+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:00.227295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:00.227295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:06.714686+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529726+0000","last_change":"2024-02-05T08:04:45.564397+0000","last_active":"2024-02-05T08:05:59.529726+0000","last_peered":"2024-02-05T08:05:59.529726+0000","last_clean":"2024-02-05T08:05:59.529726+0000","last_became_active":"2024-02-05T08:04:45.563795+0000","last_became_peered":"2024-02-05T08:04:45.563795+0000","last_unstale":"2024-02-05T08:05:59.529726+0000","last_undegraded":"2024-02-05T08:05:59.529726+0000","last_fullsized":"2024-02-05T08:05:59.529726+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.204318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226552+0000","last_change":"2024-02-05T08:05:42.259337+0000","last_active":"2024-02-05T08:06:00.226552+0000","last_peered":"2024-02-05T08:06:00.226552+0000","last_clean":"2024-02-05T08:06:00.226552+0000","last_became_active":"2024-02-05T08:04:41.535435+0000","last_became_peered":"2024-02-05T08:04:41.535435+0000","last_unstale":"2024-02-05T08:06:00.226552+0000","last_undegraded":"2024-02-05T08:06:00.226552+0000","last_fullsized":"2024-02-05T08:06:00.226552+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:42.259257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:42.259257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:48.329795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226085+0000","last_change":"2024-02-05T08:04:43.560996+0000","last_active":"2024-02-05T08:06:00.226085+0000","last_peered":"2024-02-05T08:06:00.226085+0000","last_clean":"2024-02-05T08:06:00.226085+0000","last_became_active":"2024-02-05T08:04:43.560660+0000","last_became_peered":"2024-02-05T08:04:43.560660+0000","last_unstale":"2024-02-05T08:06:00.226085+0000","last_undegraded":"2024-02-05T08:06:00.226085+0000","last_fullsized":"2024-02-05T08:06:00.226085+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528432+0000","last_change":"2024-02-05T08:05:56.285954+0000","last_active":"2024-02-05T08:05:59.528432+0000","last_peered":"2024-02-05T08:05:59.528432+0000","last_clean":"2024-02-05T08:05:59.528432+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:05:59.528432+0000","last_undegraded":"2024-02-05T08:05:59.528432+0000","last_fullsized":"2024-02-05T08:05:59.528432+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:56.285876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:56.285876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:18.137707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268950+0000","last_change":"2024-02-05T08:04:45.560281+0000","last_active":"2024-02-05T08:06:00.268950+0000","last_peered":"2024-02-05T08:06:00.268950+0000","last_clean":"2024-02-05T08:06:00.268950+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:06:00.268950+0000","last_undegraded":"2024-02-05T08:06:00.268950+0000","last_fullsized":"2024-02-05T08:06:00.268950+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:14.202262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269001+0000","last_change":"2024-02-05T08:05:55.576535+0000","last_active":"2024-02-05T08:06:00.269001+0000","last_peered":"2024-02-05T08:06:00.269001+0000","last_clean":"2024-02-05T08:06:00.269001+0000","last_became_active":"2024-02-05T08:04:41.540764+0000","last_became_peered":"2024-02-05T08:04:41.540764+0000","last_unstale":"2024-02-05T08:06:00.269001+0000","last_undegraded":"2024-02-05T08:06:00.269001+0000","last_fullsized":"2024-02-05T08:06:00.269001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:55.576441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.576441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.917805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529949+0000","last_change":"2024-02-05T08:04:43.557923+0000","last_active":"2024-02-05T08:05:59.529949+0000","last_peered":"2024-02-05T08:05:59.529949+0000","last_clean":"2024-02-05T08:05:59.529949+0000","last_became_active":"2024-02-05T08:04:43.557652+0000","last_became_peered":"2024-02-05T08:04:43.557652+0000","last_unstale":"2024-02-05T08:05:59.529949+0000","last_undegraded":"2024-02-05T08:05:59.529949+0000","last_fullsized":"2024-02-05T08:05:59.529949+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.987638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.267980+0000","last_change":"2024-02-05T08:04:47.587766+0000","last_active":"2024-02-05T08:06:00.267980+0000","last_peered":"2024-02-05T08:06:00.267980+0000","last_clean":"2024-02-05T08:06:00.267980+0000","last_became_active":"2024-02-05T08:04:47.587515+0000","last_became_peered":"2024-02-05T08:04:47.587515+0000","last_unstale":"2024-02-05T08:06:00.267980+0000","last_undegraded":"2024-02-05T08:06:00.267980+0000","last_fullsized":"2024-02-05T08:06:00.267980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.970614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718175+0000","last_change":"2024-02-05T08:05:52.725804+0000","last_active":"2024-02-05T08:05:59.718175+0000","last_peered":"2024-02-05T08:05:59.718175+0000","last_clean":"2024-02-05T08:05:59.718175+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:05:59.718175+0000","last_undegraded":"2024-02-05T08:05:59.718175+0000","last_fullsized":"2024-02-05T08:05:59.718175+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:52.725679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.725679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.406845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268975+0000","last_change":"2024-02-05T08:05:46.458359+0000","last_active":"2024-02-05T08:06:00.268975+0000","last_peered":"2024-02-05T08:06:00.268975+0000","last_clean":"2024-02-05T08:06:00.268975+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:06:00.268975+0000","last_undegraded":"2024-02-05T08:06:00.268975+0000","last_fullsized":"2024-02-05T08:06:00.268975+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.458244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.877756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530213+0000","last_change":"2024-02-05T08:04:43.557900+0000","last_active":"2024-02-05T08:05:59.530213+0000","last_peered":"2024-02-05T08:05:59.530213+0000","last_clean":"2024-02-05T08:05:59.530213+0000","last_became_active":"2024-02-05T08:04:43.557559+0000","last_became_peered":"2024-02-05T08:04:43.557559+0000","last_unstale":"2024-02-05T08:05:59.530213+0000","last_undegraded":"2024-02-05T08:05:59.530213+0000","last_fullsized":"2024-02-05T08:05:59.530213+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:09.465379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530663+0000","last_change":"2024-02-05T08:05:51.076610+0000","last_active":"2024-02-05T08:05:59.530663+0000","last_peered":"2024-02-05T08:05:59.530663+0000","last_clean":"2024-02-05T08:05:59.530663+0000","last_became_active":"2024-02-05T08:04:48.170756+0000","last_became_peered":"2024-02-05T08:04:48.170756+0000","last_unstale":"2024-02-05T08:05:59.530663+0000","last_undegraded":"2024-02-05T08:05:59.530663+0000","last_fullsized":"2024-02-05T08:05:59.530663+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:51.076514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:51.076514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:11.206152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527244+0000","last_change":"2024-02-05T08:04:45.559908+0000","last_active":"2024-02-05T08:05:59.527244+0000","last_peered":"2024-02-05T08:05:59.527244+0000","last_clean":"2024-02-05T08:05:59.527244+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:05:59.527244+0000","last_undegraded":"2024-02-05T08:05:59.527244+0000","last_fullsized":"2024-02-05T08:05:59.527244+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:11.185330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528706+0000","last_change":"2024-02-05T08:04:41.541812+0000","last_active":"2024-02-05T08:05:59.528706+0000","last_peered":"2024-02-05T08:05:59.528706+0000","last_clean":"2024-02-05T08:05:59.528706+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:05:59.528706+0000","last_undegraded":"2024-02-05T08:05:59.528706+0000","last_fullsized":"2024-02-05T08:05:59.528706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546521+0000","last_change":"2024-02-05T08:05:46.814239+0000","last_active":"2024-02-05T08:06:00.546521+0000","last_peered":"2024-02-05T08:06:00.546521+0000","last_clean":"2024-02-05T08:06:00.546521+0000","last_became_active":"2024-02-05T08:04:43.557496+0000","last_became_peered":"2024-02-05T08:04:43.557496+0000","last_unstale":"2024-02-05T08:06:00.546521+0000","last_undegraded":"2024-02-05T08:06:00.546521+0000","last_fullsized":"2024-02-05T08:06:00.546521+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:05:46.814139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.814139+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:58.081182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225446+0000","last_change":"2024-02-05T08:04:48.170578+0000","last_active":"2024-02-05T08:06:00.225446+0000","last_peered":"2024-02-05T08:06:00.225446+0000","last_clean":"2024-02-05T08:06:00.225446+0000","last_became_active":"2024-02-05T08:04:48.169770+0000","last_became_peered":"2024-02-05T08:04:48.169770+0000","last_unstale":"2024-02-05T08:06:00.225446+0000","last_undegraded":"2024-02-05T08:06:00.225446+0000","last_fullsized":"2024-02-05T08:06:00.225446+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.961379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530140+0000","last_change":"2024-02-05T08:05:57.052122+0000","last_active":"2024-02-05T08:05:59.530140+0000","last_peered":"2024-02-05T08:05:59.530140+0000","last_clean":"2024-02-05T08:05:59.530140+0000","last_became_active":"2024-02-05T08:04:45.563948+0000","last_became_peered":"2024-02-05T08:04:45.563948+0000","last_unstale":"2024-02-05T08:05:59.530140+0000","last_undegraded":"2024-02-05T08:05:59.530140+0000","last_fullsized":"2024-02-05T08:05:59.530140+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.052030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:14.589576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530056+0000","last_change":"2024-02-05T08:04:41.534341+0000","last_active":"2024-02-05T08:05:59.530056+0000","last_peered":"2024-02-05T08:05:59.530056+0000","last_clean":"2024-02-05T08:05:59.530056+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:05:59.530056+0000","last_undegraded":"2024-02-05T08:05:59.530056+0000","last_fullsized":"2024-02-05T08:05:59.530056+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.393385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"90'17","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226406+0000","last_change":"2024-02-05T08:05:58.285391+0000","last_active":"2024-02-05T08:06:00.226406+0000","last_peered":"2024-02-05T08:06:00.226406+0000","last_clean":"2024-02-05T08:06:00.226406+0000","last_became_active":"2024-02-05T08:04:43.557596+0000","last_became_peered":"2024-02-05T08:04:43.557596+0000","last_unstale":"2024-02-05T08:06:00.226406+0000","last_undegraded":"2024-02-05T08:06:00.226406+0000","last_fullsized":"2024-02-05T08:06:00.226406+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.285304+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.927379+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718547+0000","last_change":"2024-02-05T08:05:55.766576+0000","last_active":"2024-02-05T08:05:59.718547+0000","last_peered":"2024-02-05T08:05:59.718547+0000","last_clean":"2024-02-05T08:05:59.718547+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:05:59.718547+0000","last_undegraded":"2024-02-05T08:05:59.718547+0000","last_fullsized":"2024-02-05T08:05:59.718547+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:05:55.766478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.766478+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:59.016044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.721216+0000","last_change":"2024-02-05T08:04:45.565057+0000","last_active":"2024-02-05T08:05:59.721216+0000","last_peered":"2024-02-05T08:05:59.721216+0000","last_clean":"2024-02-05T08:05:59.721216+0000","last_became_active":"2024-02-05T08:04:45.564774+0000","last_became_peered":"2024-02-05T08:04:45.564774+0000","last_unstale":"2024-02-05T08:05:59.721216+0000","last_undegraded":"2024-02-05T08:05:59.721216+0000","last_fullsized":"2024-02-05T08:05:59.721216+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527726+0000","last_change":"2024-02-05T08:04:41.532653+0000","last_active":"2024-02-05T08:05:59.527726+0000","last_peered":"2024-02-05T08:05:59.527726+0000","last_clean":"2024-02-05T08:05:59.527726+0000","last_became_active":"2024-02-05T08:04:41.532339+0000","last_became_peered":"2024-02-05T08:04:41.532339+0000","last_unstale":"2024-02-05T08:05:59.527726+0000","last_undegraded":"2024-02-05T08:05:59.527726+0000","last_fullsized":"2024-02-05T08:05:59.527726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528405+0000","last_change":"2024-02-05T08:04:43.606275+0000","last_active":"2024-02-05T08:05:59.528405+0000","last_peered":"2024-02-05T08:05:59.528405+0000","last_clean":"2024-02-05T08:05:59.528405+0000","last_became_active":"2024-02-05T08:04:43.605952+0000","last_became_peered":"2024-02-05T08:04:43.605952+0000","last_unstale":"2024-02-05T08:05:59.528405+0000","last_undegraded":"2024-02-05T08:05:59.528405+0000","last_fullsized":"2024-02-05T08:05:59.528405+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:09.695654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527368+0000","last_change":"2024-02-05T08:04:47.578164+0000","last_active":"2024-02-05T08:05:59.527368+0000","last_peered":"2024-02-05T08:05:59.527368+0000","last_clean":"2024-02-05T08:05:59.527368+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:05:59.527368+0000","last_undegraded":"2024-02-05T08:05:59.527368+0000","last_fullsized":"2024-02-05T08:05:59.527368+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528636+0000","last_change":"2024-02-05T08:04:45.564760+0000","last_active":"2024-02-05T08:05:59.528636+0000","last_peered":"2024-02-05T08:05:59.528636+0000","last_clean":"2024-02-05T08:05:59.528636+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:05:59.528636+0000","last_undegraded":"2024-02-05T08:05:59.528636+0000","last_fullsized":"2024-02-05T08:05:59.528636+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.404706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225749+0000","last_change":"2024-02-05T08:04:41.534721+0000","last_active":"2024-02-05T08:06:00.225749+0000","last_peered":"2024-02-05T08:06:00.225749+0000","last_clean":"2024-02-05T08:06:00.225749+0000","last_became_active":"2024-02-05T08:04:41.534448+0000","last_became_peered":"2024-02-05T08:04:41.534448+0000","last_unstale":"2024-02-05T08:06:00.225749+0000","last_undegraded":"2024-02-05T08:06:00.225749+0000","last_fullsized":"2024-02-05T08:06:00.225749+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.721085+0000","last_change":"2024-02-05T08:05:46.208637+0000","last_active":"2024-02-05T08:05:59.721085+0000","last_peered":"2024-02-05T08:05:59.721085+0000","last_clean":"2024-02-05T08:05:59.721085+0000","last_became_active":"2024-02-05T08:05:03.789351+0000","last_became_peered":"2024-02-05T08:05:03.789351+0000","last_unstale":"2024-02-05T08:05:59.721085+0000","last_undegraded":"2024-02-05T08:05:59.721085+0000","last_fullsized":"2024-02-05T08:05:59.721085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:05:46.208534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.208534+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:00.233101+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.717804+0000","last_change":"2024-02-05T08:04:47.579487+0000","last_active":"2024-02-05T08:05:59.717804+0000","last_peered":"2024-02-05T08:05:59.717804+0000","last_clean":"2024-02-05T08:05:59.717804+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:05:59.717804+0000","last_undegraded":"2024-02-05T08:05:59.717804+0000","last_fullsized":"2024-02-05T08:05:59.717804+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:10.763840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720346+0000","last_change":"2024-02-05T08:05:54.367693+0000","last_active":"2024-02-05T08:05:59.720346+0000","last_peered":"2024-02-05T08:05:59.720346+0000","last_clean":"2024-02-05T08:05:59.720346+0000","last_became_active":"2024-02-05T08:04:45.560874+0000","last_became_peered":"2024-02-05T08:04:45.560874+0000","last_unstale":"2024-02-05T08:05:59.720346+0000","last_undegraded":"2024-02-05T08:05:59.720346+0000","last_fullsized":"2024-02-05T08:05:59.720346+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:05:54.367588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:17.218061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528041+0000","last_change":"2024-02-05T08:04:41.536675+0000","last_active":"2024-02-05T08:05:59.528041+0000","last_peered":"2024-02-05T08:05:59.528041+0000","last_clean":"2024-02-05T08:05:59.528041+0000","last_became_active":"2024-02-05T08:04:41.536019+0000","last_became_peered":"2024-02-05T08:04:41.536019+0000","last_unstale":"2024-02-05T08:05:59.528041+0000","last_undegraded":"2024-02-05T08:05:59.528041+0000","last_fullsized":"2024-02-05T08:05:59.528041+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225759+0000","last_change":"2024-02-05T08:04:43.555567+0000","last_active":"2024-02-05T08:06:00.225759+0000","last_peered":"2024-02-05T08:06:00.225759+0000","last_clean":"2024-02-05T08:06:00.225759+0000","last_became_active":"2024-02-05T08:04:43.552866+0000","last_became_peered":"2024-02-05T08:04:43.552866+0000","last_unstale":"2024-02-05T08:06:00.225759+0000","last_undegraded":"2024-02-05T08:06:00.225759+0000","last_fullsized":"2024-02-05T08:06:00.225759+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:03.755886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:02.190986+0000","last_change":"2024-02-05T08:06:02.191006+0000","last_active":"2024-02-05T08:06:02.190986+0000","last_peered":"2024-02-05T08:06:02.190986+0000","last_clean":"2024-02-05T08:06:02.190986+0000","last_became_active":"2024-02-05T08:04:48.172793+0000","last_became_peered":"2024-02-05T08:04:48.172793+0000","last_unstale":"2024-02-05T08:06:02.190986+0000","last_undegraded":"2024-02-05T08:06:02.190986+0000","last_fullsized":"2024-02-05T08:06:02.190986+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:02.190931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:02.190931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:27.955974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718344+0000","last_change":"2024-02-05T08:05:48.748227+0000","last_active":"2024-02-05T08:05:59.718344+0000","last_peered":"2024-02-05T08:05:59.718344+0000","last_clean":"2024-02-05T08:05:59.718344+0000","last_became_active":"2024-02-05T08:04:45.563610+0000","last_became_peered":"2024-02-05T08:04:45.563610+0000","last_unstale":"2024-02-05T08:05:59.718344+0000","last_undegraded":"2024-02-05T08:05:59.718344+0000","last_fullsized":"2024-02-05T08:05:59.718344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:48.748132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:48.748132+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.748132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:17.563379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269194+0000","last_change":"2024-02-05T08:04:41.542169+0000","last_active":"2024-02-05T08:06:00.269194+0000","last_peered":"2024-02-05T08:06:00.269194+0000","last_clean":"2024-02-05T08:06:00.269194+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:06:00.269194+0000","last_undegraded":"2024-02-05T08:06:00.269194+0000","last_fullsized":"2024-02-05T08:06:00.269194+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226059+0000","last_change":"2024-02-05T08:04:43.606219+0000","last_active":"2024-02-05T08:06:00.226059+0000","last_peered":"2024-02-05T08:06:00.226059+0000","last_clean":"2024-02-05T08:06:00.226059+0000","last_became_active":"2024-02-05T08:04:43.605932+0000","last_became_peered":"2024-02-05T08:04:43.605932+0000","last_unstale":"2024-02-05T08:06:00.226059+0000","last_undegraded":"2024-02-05T08:06:00.226059+0000","last_fullsized":"2024-02-05T08:06:00.226059+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:00.517159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528230+0000","last_change":"2024-02-05T08:04:47.582343+0000","last_active":"2024-02-05T08:05:59.528230+0000","last_peered":"2024-02-05T08:05:59.528230+0000","last_clean":"2024-02-05T08:05:59.528230+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:05:59.528230+0000","last_undegraded":"2024-02-05T08:05:59.528230+0000","last_fullsized":"2024-02-05T08:05:59.528230+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226266+0000","last_change":"2024-02-05T08:05:57.270638+0000","last_active":"2024-02-05T08:06:00.226266+0000","last_peered":"2024-02-05T08:06:00.226266+0000","last_clean":"2024-02-05T08:06:00.226266+0000","last_became_active":"2024-02-05T08:04:45.568484+0000","last_became_peered":"2024-02-05T08:04:45.568484+0000","last_unstale":"2024-02-05T08:06:00.226266+0000","last_undegraded":"2024-02-05T08:06:00.226266+0000","last_fullsized":"2024-02-05T08:06:00.226266+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:57.270554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.270554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:03.898076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718661+0000","last_change":"2024-02-05T08:04:41.540005+0000","last_active":"2024-02-05T08:05:59.718661+0000","last_peered":"2024-02-05T08:05:59.718661+0000","last_clean":"2024-02-05T08:05:59.718661+0000","last_became_active":"2024-02-05T08:04:41.539698+0000","last_became_peered":"2024-02-05T08:04:41.539698+0000","last_unstale":"2024-02-05T08:05:59.718661+0000","last_undegraded":"2024-02-05T08:05:59.718661+0000","last_fullsized":"2024-02-05T08:05:59.718661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.105779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527388+0000","last_change":"2024-02-05T08:05:48.910387+0000","last_active":"2024-02-05T08:05:59.527388+0000","last_peered":"2024-02-05T08:05:59.527388+0000","last_clean":"2024-02-05T08:05:59.527388+0000","last_became_active":"2024-02-05T08:04:43.555031+0000","last_became_peered":"2024-02-05T08:04:43.555031+0000","last_unstale":"2024-02-05T08:05:59.527388+0000","last_undegraded":"2024-02-05T08:05:59.527388+0000","last_fullsized":"2024-02-05T08:05:59.527388+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:05:48.910304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.910304+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:49.156858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225328+0000","last_change":"2024-02-05T08:04:47.586079+0000","last_active":"2024-02-05T08:06:00.225328+0000","last_peered":"2024-02-05T08:06:00.225328+0000","last_clean":"2024-02-05T08:06:00.225328+0000","last_became_active":"2024-02-05T08:04:47.585496+0000","last_became_peered":"2024-02-05T08:04:47.585496+0000","last_unstale":"2024-02-05T08:06:00.225328+0000","last_undegraded":"2024-02-05T08:06:00.225328+0000","last_fullsized":"2024-02-05T08:06:00.225328+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:12.376304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527208+0000","last_change":"2024-02-05T08:05:46.864673+0000","last_active":"2024-02-05T08:05:59.527208+0000","last_peered":"2024-02-05T08:05:59.527208+0000","last_clean":"2024-02-05T08:05:59.527208+0000","last_became_active":"2024-02-05T08:04:45.560815+0000","last_became_peered":"2024-02-05T08:04:45.560815+0000","last_unstale":"2024-02-05T08:05:59.527208+0000","last_undegraded":"2024-02-05T08:05:59.527208+0000","last_fullsized":"2024-02-05T08:05:59.527208+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:46.864589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.864589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.980095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530377+0000","last_change":"2024-02-05T08:05:53.112020+0000","last_active":"2024-02-05T08:05:59.530377+0000","last_peered":"2024-02-05T08:05:59.530377+0000","last_clean":"2024-02-05T08:05:59.530377+0000","last_became_active":"2024-02-05T08:04:41.543255+0000","last_became_peered":"2024-02-05T08:04:41.543255+0000","last_unstale":"2024-02-05T08:05:59.530377+0000","last_undegraded":"2024-02-05T08:05:59.530377+0000","last_fullsized":"2024-02-05T08:05:59.530377+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:53.111931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.111931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:19.887019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530372+0000","last_change":"2024-02-05T08:04:43.554427+0000","last_active":"2024-02-05T08:05:59.530372+0000","last_peered":"2024-02-05T08:05:59.530372+0000","last_clean":"2024-02-05T08:05:59.530372+0000","last_became_active":"2024-02-05T08:04:43.554101+0000","last_became_peered":"2024-02-05T08:04:43.554101+0000","last_unstale":"2024-02-05T08:05:59.530372+0000","last_undegraded":"2024-02-05T08:05:59.530372+0000","last_fullsized":"2024-02-05T08:05:59.530372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:05.485949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528132+0000","last_change":"2024-02-05T08:04:47.584516+0000","last_active":"2024-02-05T08:05:59.528132+0000","last_peered":"2024-02-05T08:05:59.528132+0000","last_clean":"2024-02-05T08:05:59.528132+0000","last_became_active":"2024-02-05T08:04:47.584235+0000","last_became_peered":"2024-02-05T08:04:47.584235+0000","last_unstale":"2024-02-05T08:05:59.528132+0000","last_undegraded":"2024-02-05T08:05:59.528132+0000","last_fullsized":"2024-02-05T08:05:59.528132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:11.950403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269080+0000","last_change":"2024-02-05T08:04:45.559777+0000","last_active":"2024-02-05T08:06:00.269080+0000","last_peered":"2024-02-05T08:06:00.269080+0000","last_clean":"2024-02-05T08:06:00.269080+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:06:00.269080+0000","last_undegraded":"2024-02-05T08:06:00.269080+0000","last_fullsized":"2024-02-05T08:06:00.269080+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.670790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718361+0000","last_change":"2024-02-05T08:05:45.722765+0000","last_active":"2024-02-05T08:05:59.718361+0000","last_peered":"2024-02-05T08:05:59.718361+0000","last_clean":"2024-02-05T08:05:59.718361+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:05:59.718361+0000","last_undegraded":"2024-02-05T08:05:59.718361+0000","last_fullsized":"2024-02-05T08:05:59.718361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:45.722652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:45.722652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.867081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226318+0000","last_change":"2024-02-05T08:04:43.606315+0000","last_active":"2024-02-05T08:06:00.226318+0000","last_peered":"2024-02-05T08:06:00.226318+0000","last_clean":"2024-02-05T08:06:00.226318+0000","last_became_active":"2024-02-05T08:04:43.605981+0000","last_became_peered":"2024-02-05T08:04:43.605981+0000","last_unstale":"2024-02-05T08:06:00.226318+0000","last_undegraded":"2024-02-05T08:06:00.226318+0000","last_fullsized":"2024-02-05T08:06:00.226318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530817+0000","last_change":"2024-02-05T08:04:47.578924+0000","last_active":"2024-02-05T08:05:59.530817+0000","last_peered":"2024-02-05T08:05:59.530817+0000","last_clean":"2024-02-05T08:05:59.530817+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:05:59.530817+0000","last_undegraded":"2024-02-05T08:05:59.530817+0000","last_fullsized":"2024-02-05T08:05:59.530817+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:03.039774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226597+0000","last_change":"2024-02-05T08:04:45.566275+0000","last_active":"2024-02-05T08:06:00.226597+0000","last_peered":"2024-02-05T08:06:00.226597+0000","last_clean":"2024-02-05T08:06:00.226597+0000","last_became_active":"2024-02-05T08:04:45.565240+0000","last_became_peered":"2024-02-05T08:04:45.565240+0000","last_unstale":"2024-02-05T08:06:00.226597+0000","last_undegraded":"2024-02-05T08:06:00.226597+0000","last_fullsized":"2024-02-05T08:06:00.226597+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527536+0000","last_change":"2024-02-05T08:04:41.540528+0000","last_active":"2024-02-05T08:05:59.527536+0000","last_peered":"2024-02-05T08:05:59.527536+0000","last_clean":"2024-02-05T08:05:59.527536+0000","last_became_active":"2024-02-05T08:04:41.540252+0000","last_became_peered":"2024-02-05T08:04:41.540252+0000","last_unstale":"2024-02-05T08:05:59.527536+0000","last_undegraded":"2024-02-05T08:05:59.527536+0000","last_fullsized":"2024-02-05T08:05:59.527536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:02.085874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269241+0000","last_change":"2024-02-05T08:05:56.541620+0000","last_active":"2024-02-05T08:06:00.269241+0000","last_peered":"2024-02-05T08:06:00.269241+0000","last_clean":"2024-02-05T08:06:00.269241+0000","last_became_active":"2024-02-05T08:04:43.606414+0000","last_became_peered":"2024-02-05T08:04:43.606414+0000","last_unstale":"2024-02-05T08:06:00.269241+0000","last_undegraded":"2024-02-05T08:06:00.269241+0000","last_fullsized":"2024-02-05T08:06:00.269241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:05:56.541510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:56.541510+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:04.755122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.545844+0000","last_change":"2024-02-05T08:04:47.578623+0000","last_active":"2024-02-05T08:06:00.545844+0000","last_peered":"2024-02-05T08:06:00.545844+0000","last_clean":"2024-02-05T08:06:00.545844+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:06:00.545844+0000","last_undegraded":"2024-02-05T08:06:00.545844+0000","last_fullsized":"2024-02-05T08:06:00.545844+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.088742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226724+0000","last_change":"2024-02-05T08:04:45.567945+0000","last_active":"2024-02-05T08:06:00.226724+0000","last_peered":"2024-02-05T08:06:00.226724+0000","last_clean":"2024-02-05T08:06:00.226724+0000","last_became_active":"2024-02-05T08:04:45.567622+0000","last_became_peered":"2024-02-05T08:04:45.567622+0000","last_unstale":"2024-02-05T08:06:00.226724+0000","last_undegraded":"2024-02-05T08:06:00.226724+0000","last_fullsized":"2024-02-05T08:06:00.226724+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718248+0000","last_change":"2024-02-05T08:04:41.541120+0000","last_active":"2024-02-05T08:05:59.718248+0000","last_peered":"2024-02-05T08:05:59.718248+0000","last_clean":"2024-02-05T08:05:59.718248+0000","last_became_active":"2024-02-05T08:04:41.540432+0000","last_became_peered":"2024-02-05T08:04:41.540432+0000","last_unstale":"2024-02-05T08:05:59.718248+0000","last_undegraded":"2024-02-05T08:05:59.718248+0000","last_fullsized":"2024-02-05T08:05:59.718248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:10.219126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528495+0000","last_change":"2024-02-05T08:04:43.608826+0000","last_active":"2024-02-05T08:05:59.528495+0000","last_peered":"2024-02-05T08:05:59.528495+0000","last_clean":"2024-02-05T08:05:59.528495+0000","last_became_active":"2024-02-05T08:04:43.608489+0000","last_became_peered":"2024-02-05T08:04:43.608489+0000","last_unstale":"2024-02-05T08:05:59.528495+0000","last_undegraded":"2024-02-05T08:05:59.528495+0000","last_fullsized":"2024-02-05T08:05:59.528495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.929658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718647+0000","last_change":"2024-02-05T08:05:56.721472+0000","last_active":"2024-02-05T08:05:59.718647+0000","last_peered":"2024-02-05T08:05:59.718647+0000","last_clean":"2024-02-05T08:05:59.718647+0000","last_became_active":"2024-02-05T08:04:48.172747+0000","last_became_peered":"2024-02-05T08:04:48.172747+0000","last_unstale":"2024-02-05T08:05:59.718647+0000","last_undegraded":"2024-02-05T08:05:59.718647+0000","last_fullsized":"2024-02-05T08:05:59.718647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:56.721381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:56.721381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.336615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269199+0000","last_change":"2024-02-05T08:04:45.560521+0000","last_active":"2024-02-05T08:06:00.269199+0000","last_peered":"2024-02-05T08:06:00.269199+0000","last_clean":"2024-02-05T08:06:00.269199+0000","last_became_active":"2024-02-05T08:04:45.560241+0000","last_became_peered":"2024-02-05T08:04:45.560241+0000","last_unstale":"2024-02-05T08:06:00.269199+0000","last_undegraded":"2024-02-05T08:06:00.269199+0000","last_fullsized":"2024-02-05T08:06:00.269199+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720257+0000","last_change":"2024-02-05T08:05:49.229502+0000","last_active":"2024-02-05T08:05:59.720257+0000","last_peered":"2024-02-05T08:05:59.720257+0000","last_clean":"2024-02-05T08:05:59.720257+0000","last_became_active":"2024-02-05T08:04:41.537709+0000","last_became_peered":"2024-02-05T08:04:41.537709+0000","last_unstale":"2024-02-05T08:05:59.720257+0000","last_undegraded":"2024-02-05T08:05:59.720257+0000","last_fullsized":"2024-02-05T08:05:59.720257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:49.229415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:49.229415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:03.840509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226736+0000","last_change":"2024-02-05T08:04:43.606964+0000","last_active":"2024-02-05T08:06:00.226736+0000","last_peered":"2024-02-05T08:06:00.226736+0000","last_clean":"2024-02-05T08:06:00.226736+0000","last_became_active":"2024-02-05T08:04:43.606624+0000","last_became_peered":"2024-02-05T08:04:43.606624+0000","last_unstale":"2024-02-05T08:06:00.226736+0000","last_undegraded":"2024-02-05T08:06:00.226736+0000","last_fullsized":"2024-02-05T08:06:00.226736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268004+0000","last_change":"2024-02-05T08:05:59.500065+0000","last_active":"2024-02-05T08:06:00.268004+0000","last_peered":"2024-02-05T08:06:00.268004+0000","last_clean":"2024-02-05T08:06:00.268004+0000","last_became_active":"2024-02-05T08:04:48.170100+0000","last_became_peered":"2024-02-05T08:04:48.170100+0000","last_unstale":"2024-02-05T08:06:00.268004+0000","last_undegraded":"2024-02-05T08:06:00.268004+0000","last_fullsized":"2024-02-05T08:06:00.268004+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:59.499960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:59.499960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:16.082380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718992+0000","last_change":"2024-02-05T08:05:57.687803+0000","last_active":"2024-02-05T08:05:59.718992+0000","last_peered":"2024-02-05T08:05:59.718992+0000","last_clean":"2024-02-05T08:05:59.718992+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:05:59.718992+0000","last_undegraded":"2024-02-05T08:05:59.718992+0000","last_fullsized":"2024-02-05T08:05:59.718992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:05:57.687679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.687679+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:23.035896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268523+0000","last_change":"2024-02-05T08:04:43.608563+0000","last_active":"2024-02-05T08:06:00.268523+0000","last_peered":"2024-02-05T08:06:00.268523+0000","last_clean":"2024-02-05T08:06:00.268523+0000","last_became_active":"2024-02-05T08:04:43.608254+0000","last_became_peered":"2024-02-05T08:04:43.608254+0000","last_unstale":"2024-02-05T08:06:00.268523+0000","last_undegraded":"2024-02-05T08:06:00.268523+0000","last_fullsized":"2024-02-05T08:06:00.268523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.187066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720386+0000","last_change":"2024-02-05T08:05:59.720386+0000","last_active":"2024-02-05T08:05:59.720386+0000","last_peered":"2024-02-05T08:05:59.720386+0000","last_clean":"2024-02-05T08:05:59.720386+0000","last_became_active":"2024-02-05T08:04:45.561044+0000","last_became_peered":"2024-02-05T08:04:45.561044+0000","last_unstale":"2024-02-05T08:05:59.720386+0000","last_undegraded":"2024-02-05T08:05:59.720386+0000","last_fullsized":"2024-02-05T08:05:59.720386+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:05:59.720223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:59.720223+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.562177+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225545+0000","last_change":"2024-02-05T08:04:47.587025+0000","last_active":"2024-02-05T08:06:00.225545+0000","last_peered":"2024-02-05T08:06:00.225545+0000","last_clean":"2024-02-05T08:06:00.225545+0000","last_became_active":"2024-02-05T08:04:47.586044+0000","last_became_peered":"2024-02-05T08:04:47.586044+0000","last_unstale":"2024-02-05T08:06:00.225545+0000","last_undegraded":"2024-02-05T08:06:00.225545+0000","last_fullsized":"2024-02-05T08:06:00.225545+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.700997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.762873+0000","last_change":"2024-02-05T08:06:01.762898+0000","last_active":"2024-02-05T08:06:01.762873+0000","last_peered":"2024-02-05T08:06:01.762873+0000","last_clean":"2024-02-05T08:06:01.762873+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:06:01.762873+0000","last_undegraded":"2024-02-05T08:06:01.762873+0000","last_fullsized":"2024-02-05T08:06:01.762873+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:01.762805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:01.762805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:12.602032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530066+0000","last_change":"2024-02-05T08:04:43.556901+0000","last_active":"2024-02-05T08:05:59.530066+0000","last_peered":"2024-02-05T08:05:59.530066+0000","last_clean":"2024-02-05T08:05:59.530066+0000","last_became_active":"2024-02-05T08:04:43.556580+0000","last_became_peered":"2024-02-05T08:04:43.556580+0000","last_unstale":"2024-02-05T08:05:59.530066+0000","last_undegraded":"2024-02-05T08:05:59.530066+0000","last_fullsized":"2024-02-05T08:05:59.530066+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.567288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546025+0000","last_change":"2024-02-05T08:04:45.569382+0000","last_active":"2024-02-05T08:06:00.546025+0000","last_peered":"2024-02-05T08:06:00.546025+0000","last_clean":"2024-02-05T08:06:00.546025+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:06:00.546025+0000","last_undegraded":"2024-02-05T08:06:00.546025+0000","last_fullsized":"2024-02-05T08:06:00.546025+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.462610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.717934+0000","last_change":"2024-02-05T08:04:47.587320+0000","last_active":"2024-02-05T08:05:59.717934+0000","last_peered":"2024-02-05T08:05:59.717934+0000","last_clean":"2024-02-05T08:05:59.717934+0000","last_became_active":"2024-02-05T08:04:47.587082+0000","last_became_peered":"2024-02-05T08:04:47.587082+0000","last_unstale":"2024-02-05T08:05:59.717934+0000","last_undegraded":"2024-02-05T08:05:59.717934+0000","last_fullsized":"2024-02-05T08:05:59.717934+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546310+0000","last_change":"2024-02-05T08:05:55.596909+0000","last_active":"2024-02-05T08:06:00.546310+0000","last_peered":"2024-02-05T08:06:00.546310+0000","last_clean":"2024-02-05T08:06:00.546310+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:06:00.546310+0000","last_undegraded":"2024-02-05T08:06:00.546310+0000","last_fullsized":"2024-02-05T08:06:00.546310+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:55.596827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.596827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.318812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718966+0000","last_change":"2024-02-05T08:05:53.741209+0000","last_active":"2024-02-05T08:05:59.718966+0000","last_peered":"2024-02-05T08:05:59.718966+0000","last_clean":"2024-02-05T08:05:59.718966+0000","last_became_active":"2024-02-05T08:04:41.540586+0000","last_became_peered":"2024-02-05T08:04:41.540586+0000","last_unstale":"2024-02-05T08:05:59.718966+0000","last_undegraded":"2024-02-05T08:05:59.718966+0000","last_fullsized":"2024-02-05T08:05:59.718966+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:53.741059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.741059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:13.753256+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":112,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720616+0000","last_change":"2024-02-05T08:05:03.797738+0000","last_active":"2024-02-05T08:05:59.720616+0000","last_peered":"2024-02-05T08:05:59.720616+0000","last_clean":"2024-02-05T08:05:59.720616+0000","last_became_active":"2024-02-05T08:05:03.786876+0000","last_became_peered":"2024-02-05T08:05:03.786876+0000","last_unstale":"2024-02-05T08:05:59.720616+0000","last_undegraded":"2024-02-05T08:05:59.720616+0000","last_fullsized":"2024-02-05T08:05:59.720616+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:09.572379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.526787+0000","last_change":"2024-02-05T08:04:47.582355+0000","last_active":"2024-02-05T08:05:59.526787+0000","last_peered":"2024-02-05T08:05:59.526787+0000","last_clean":"2024-02-05T08:05:59.526787+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:05:59.526787+0000","last_undegraded":"2024-02-05T08:05:59.526787+0000","last_fullsized":"2024-02-05T08:05:59.526787+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546233+0000","last_change":"2024-02-05T08:04:45.567684+0000","last_active":"2024-02-05T08:06:00.546233+0000","last_peered":"2024-02-05T08:06:00.546233+0000","last_clean":"2024-02-05T08:06:00.546233+0000","last_became_active":"2024-02-05T08:04:45.567341+0000","last_became_peered":"2024-02-05T08:04:45.567341+0000","last_unstale":"2024-02-05T08:06:00.546233+0000","last_undegraded":"2024-02-05T08:06:00.546233+0000","last_fullsized":"2024-02-05T08:06:00.546233+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:03.017379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268978+0000","last_change":"2024-02-05T08:04:41.540429+0000","last_active":"2024-02-05T08:06:00.268978+0000","last_peered":"2024-02-05T08:06:00.268978+0000","last_clean":"2024-02-05T08:06:00.268978+0000","last_became_active":"2024-02-05T08:04:41.540116+0000","last_became_peered":"2024-02-05T08:04:41.540116+0000","last_unstale":"2024-02-05T08:06:00.268978+0000","last_undegraded":"2024-02-05T08:06:00.268978+0000","last_fullsized":"2024-02-05T08:06:00.268978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.767296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528808+0000","last_change":"2024-02-05T08:04:43.552776+0000","last_active":"2024-02-05T08:05:59.528808+0000","last_peered":"2024-02-05T08:05:59.528808+0000","last_clean":"2024-02-05T08:05:59.528808+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:05:59.528808+0000","last_undegraded":"2024-02-05T08:05:59.528808+0000","last_fullsized":"2024-02-05T08:05:59.528808+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225432+0000","last_change":"2024-02-05T08:04:47.588669+0000","last_active":"2024-02-05T08:06:00.225432+0000","last_peered":"2024-02-05T08:06:00.225432+0000","last_clean":"2024-02-05T08:06:00.225432+0000","last_became_active":"2024-02-05T08:04:47.588395+0000","last_became_peered":"2024-02-05T08:04:47.588395+0000","last_unstale":"2024-02-05T08:06:00.225432+0000","last_undegraded":"2024-02-05T08:06:00.225432+0000","last_fullsized":"2024-02-05T08:06:00.225432+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:16.402122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268703+0000","last_change":"2024-02-05T08:05:52.503758+0000","last_active":"2024-02-05T08:06:00.268703+0000","last_peered":"2024-02-05T08:06:00.268703+0000","last_clean":"2024-02-05T08:06:00.268703+0000","last_became_active":"2024-02-05T08:04:40.527674+0000","last_became_peered":"2024-02-05T08:04:40.527674+0000","last_unstale":"2024-02-05T08:06:00.268703+0000","last_undegraded":"2024-02-05T08:06:00.268703+0000","last_fullsized":"2024-02-05T08:06:00.268703+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.503651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.821538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530022+0000","last_change":"2024-02-05T08:04:41.537767+0000","last_active":"2024-02-05T08:05:59.530022+0000","last_peered":"2024-02-05T08:05:59.530022+0000","last_clean":"2024-02-05T08:05:59.530022+0000","last_became_active":"2024-02-05T08:04:41.537478+0000","last_became_peered":"2024-02-05T08:04:41.537478+0000","last_unstale":"2024-02-05T08:05:59.530022+0000","last_undegraded":"2024-02-05T08:05:59.530022+0000","last_fullsized":"2024-02-05T08:05:59.530022+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"85'38","reported_seq":138,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527944+0000","last_change":"2024-02-05T08:05:43.858932+0000","last_active":"2024-02-05T08:05:59.527944+0000","last_peered":"2024-02-05T08:05:59.527944+0000","last_clean":"2024-02-05T08:05:59.527944+0000","last_became_active":"2024-02-05T08:04:43.554536+0000","last_became_peered":"2024-02-05T08:04:43.554536+0000","last_unstale":"2024-02-05T08:05:59.527944+0000","last_undegraded":"2024-02-05T08:05:59.527944+0000","last_fullsized":"2024-02-05T08:05:59.527944+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:05:43.858841+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:54.322675+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527257+0000","last_change":"2024-02-05T08:04:45.562140+0000","last_active":"2024-02-05T08:05:59.527257+0000","last_peered":"2024-02-05T08:05:59.527257+0000","last_clean":"2024-02-05T08:05:59.527257+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:05:59.527257+0000","last_undegraded":"2024-02-05T08:05:59.527257+0000","last_fullsized":"2024-02-05T08:05:59.527257+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:14.185461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.719032+0000","last_change":"2024-02-05T08:04:48.170226+0000","last_active":"2024-02-05T08:05:59.719032+0000","last_peered":"2024-02-05T08:05:59.719032+0000","last_clean":"2024-02-05T08:05:59.719032+0000","last_became_active":"2024-02-05T08:04:48.169894+0000","last_became_peered":"2024-02-05T08:04:48.169894+0000","last_unstale":"2024-02-05T08:05:59.719032+0000","last_undegraded":"2024-02-05T08:05:59.719032+0000","last_fullsized":"2024-02-05T08:05:59.719032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:15.026925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"67'1","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.526644+0000","last_change":"2024-02-05T08:04:41.536875+0000","last_active":"2024-02-05T08:05:59.526644+0000","last_peered":"2024-02-05T08:05:59.526644+0000","last_clean":"2024-02-05T08:05:59.526644+0000","last_became_active":"2024-02-05T08:04:41.536585+0000","last_became_peered":"2024-02-05T08:04:41.536585+0000","last_unstale":"2024-02-05T08:05:59.526644+0000","last_undegraded":"2024-02-05T08:05:59.526644+0000","last_fullsized":"2024-02-05T08:05:59.526644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.042189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.545805+0000","last_change":"2024-02-05T08:05:41.807274+0000","last_active":"2024-02-05T08:06:00.545805+0000","last_peered":"2024-02-05T08:06:00.545805+0000","last_clean":"2024-02-05T08:06:00.545805+0000","last_became_active":"2024-02-05T08:04:40.529073+0000","last_became_peered":"2024-02-05T08:04:40.529073+0000","last_unstale":"2024-02-05T08:06:00.545805+0000","last_undegraded":"2024-02-05T08:06:00.545805+0000","last_fullsized":"2024-02-05T08:06:00.545805+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:41.807173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:05:41.807173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:48.748767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546072+0000","last_change":"2024-02-05T08:05:45.827248+0000","last_active":"2024-02-05T08:06:00.546072+0000","last_peered":"2024-02-05T08:06:00.546072+0000","last_clean":"2024-02-05T08:06:00.546072+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:06:00.546072+0000","last_undegraded":"2024-02-05T08:06:00.546072+0000","last_fullsized":"2024-02-05T08:06:00.546072+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:05:45.827140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:45.827140+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:14.695181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546608+0000","last_change":"2024-02-05T08:05:58.550125+0000","last_active":"2024-02-05T08:06:00.546608+0000","last_peered":"2024-02-05T08:06:00.546608+0000","last_clean":"2024-02-05T08:06:00.546608+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:06:00.546608+0000","last_undegraded":"2024-02-05T08:06:00.546608+0000","last_fullsized":"2024-02-05T08:06:00.546608+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:58.550049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.550049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.897417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268113+0000","last_change":"2024-02-05T08:04:47.586644+0000","last_active":"2024-02-05T08:06:00.268113+0000","last_peered":"2024-02-05T08:06:00.268113+0000","last_clean":"2024-02-05T08:06:00.268113+0000","last_became_active":"2024-02-05T08:04:47.586358+0000","last_became_peered":"2024-02-05T08:04:47.586358+0000","last_unstale":"2024-02-05T08:06:00.268113+0000","last_undegraded":"2024-02-05T08:06:00.268113+0000","last_fullsized":"2024-02-05T08:06:00.268113+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:13.569831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":316,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718051+0000","last_change":"2024-02-05T08:05:16.657716+0000","last_active":"2024-02-05T08:05:59.718051+0000","last_peered":"2024-02-05T08:05:59.718051+0000","last_clean":"2024-02-05T08:05:59.718051+0000","last_became_active":"2024-02-05T08:04:30.515158+0000","last_became_peered":"2024-02-05T08:04:30.515158+0000","last_unstale":"2024-02-05T08:05:59.718051+0000","last_undegraded":"2024-02-05T08:05:59.718051+0000","last_fullsized":"2024-02-05T08:05:59.718051+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'119","last_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:05:16.657600+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:37.807886+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226170+0000","last_change":"2024-02-05T08:04:41.538282+0000","last_active":"2024-02-05T08:06:00.226170+0000","last_peered":"2024-02-05T08:06:00.226170+0000","last_clean":"2024-02-05T08:06:00.226170+0000","last_became_active":"2024-02-05T08:04:41.538002+0000","last_became_peered":"2024-02-05T08:04:41.538002+0000","last_unstale":"2024-02-05T08:06:00.226170+0000","last_undegraded":"2024-02-05T08:06:00.226170+0000","last_fullsized":"2024-02-05T08:06:00.226170+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.327424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":102,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.304976+0000","last_change":"2024-02-05T08:06:00.304976+0000","last_active":"2024-02-05T08:06:00.304976+0000","last_peered":"2024-02-05T08:06:00.304976+0000","last_clean":"2024-02-05T08:06:00.304976+0000","last_became_active":"2024-02-05T08:04:43.606401+0000","last_became_peered":"2024-02-05T08:04:43.606401+0000","last_unstale":"2024-02-05T08:06:00.304976+0000","last_undegraded":"2024-02-05T08:06:00.304976+0000","last_fullsized":"2024-02-05T08:06:00.304976+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-02-05T08:06:00.304861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:00.304861+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:15.661923+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718680+0000","last_change":"2024-02-05T08:04:45.565211+0000","last_active":"2024-02-05T08:05:59.718680+0000","last_peered":"2024-02-05T08:05:59.718680+0000","last_clean":"2024-02-05T08:05:59.718680+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:05:59.718680+0000","last_undegraded":"2024-02-05T08:05:59.718680+0000","last_fullsized":"2024-02-05T08:05:59.718680+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:10.360421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527679+0000","last_change":"2024-02-05T08:04:47.587068+0000","last_active":"2024-02-05T08:05:59.527679+0000","last_peered":"2024-02-05T08:05:59.527679+0000","last_clean":"2024-02-05T08:05:59.527679+0000","last_became_active":"2024-02-05T08:04:47.586774+0000","last_became_peered":"2024-02-05T08:04:47.586774+0000","last_unstale":"2024-02-05T08:05:59.527679+0000","last_undegraded":"2024-02-05T08:05:59.527679+0000","last_fullsized":"2024-02-05T08:05:59.527679+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:00.755929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527011+0000","last_change":"2024-02-05T08:04:41.538403+0000","last_active":"2024-02-05T08:05:59.527011+0000","last_peered":"2024-02-05T08:05:59.527011+0000","last_clean":"2024-02-05T08:05:59.527011+0000","last_became_active":"2024-02-05T08:04:41.537872+0000","last_became_peered":"2024-02-05T08:04:41.537872+0000","last_unstale":"2024-02-05T08:05:59.527011+0000","last_undegraded":"2024-02-05T08:05:59.527011+0000","last_fullsized":"2024-02-05T08:05:59.527011+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.800257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":154,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.904289+0000","last_change":"2024-02-05T08:05:43.746185+0000","last_active":"2024-02-05T08:06:00.904289+0000","last_peered":"2024-02-05T08:06:00.904289+0000","last_clean":"2024-02-05T08:06:00.904289+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:06:00.904289+0000","last_undegraded":"2024-02-05T08:06:00.904289+0000","last_fullsized":"2024-02-05T08:06:00.904289+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:05:43.746095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:05:43.746095+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:55.186390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528230+0000","last_change":"2024-02-05T08:04:43.550001+0000","last_active":"2024-02-05T08:05:59.528230+0000","last_peered":"2024-02-05T08:05:59.528230+0000","last_clean":"2024-02-05T08:05:59.528230+0000","last_became_active":"2024-02-05T08:04:43.549687+0000","last_became_peered":"2024-02-05T08:04:43.549687+0000","last_unstale":"2024-02-05T08:05:59.528230+0000","last_undegraded":"2024-02-05T08:05:59.528230+0000","last_fullsized":"2024-02-05T08:05:59.528230+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.794328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269362+0000","last_change":"2024-02-05T08:04:45.559137+0000","last_active":"2024-02-05T08:06:00.269362+0000","last_peered":"2024-02-05T08:06:00.269362+0000","last_clean":"2024-02-05T08:06:00.269362+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:06:00.269362+0000","last_undegraded":"2024-02-05T08:06:00.269362+0000","last_fullsized":"2024-02-05T08:06:00.269362+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:12.171151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527562+0000","last_change":"2024-02-05T08:04:48.170100+0000","last_active":"2024-02-05T08:05:59.527562+0000","last_peered":"2024-02-05T08:05:59.527562+0000","last_clean":"2024-02-05T08:05:59.527562+0000","last_became_active":"2024-02-05T08:04:48.169777+0000","last_became_peered":"2024-02-05T08:04:48.169777+0000","last_unstale":"2024-02-05T08:05:59.527562+0000","last_undegraded":"2024-02-05T08:05:59.527562+0000","last_fullsized":"2024-02-05T08:05:59.527562+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527843+0000","last_change":"2024-02-05T08:05:51.880952+0000","last_active":"2024-02-05T08:05:59.527843+0000","last_peered":"2024-02-05T08:05:59.527843+0000","last_clean":"2024-02-05T08:05:59.527843+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:05:59.527843+0000","last_undegraded":"2024-02-05T08:05:59.527843+0000","last_fullsized":"2024-02-05T08:05:59.527843+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:51.880864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:51.880864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:53.033627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":98,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528739+0000","last_change":"2024-02-05T08:05:55.329780+0000","last_active":"2024-02-05T08:05:59.528739+0000","last_peered":"2024-02-05T08:05:59.528739+0000","last_clean":"2024-02-05T08:05:59.528739+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:05:59.528739+0000","last_undegraded":"2024-02-05T08:05:59.528739+0000","last_fullsized":"2024-02-05T08:05:59.528739+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:05:55.329677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.329677+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:07.966593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720746+0000","last_change":"2024-02-05T08:04:45.565899+0000","last_active":"2024-02-05T08:05:59.720746+0000","last_peered":"2024-02-05T08:05:59.720746+0000","last_clean":"2024-02-05T08:05:59.720746+0000","last_became_active":"2024-02-05T08:04:45.565618+0000","last_became_peered":"2024-02-05T08:04:45.565618+0000","last_unstale":"2024-02-05T08:05:59.720746+0000","last_undegraded":"2024-02-05T08:05:59.720746+0000","last_fullsized":"2024-02-05T08:05:59.720746+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.936746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528338+0000","last_change":"2024-02-05T08:05:58.215810+0000","last_active":"2024-02-05T08:05:59.528338+0000","last_peered":"2024-02-05T08:05:59.528338+0000","last_clean":"2024-02-05T08:05:59.528338+0000","last_became_active":"2024-02-05T08:04:47.583866+0000","last_became_peered":"2024-02-05T08:04:47.583866+0000","last_unstale":"2024-02-05T08:05:59.528338+0000","last_undegraded":"2024-02-05T08:05:59.528338+0000","last_fullsized":"2024-02-05T08:05:59.528338+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:58.215737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.215737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.871723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269169+0000","last_change":"2024-02-05T08:04:41.538084+0000","last_active":"2024-02-05T08:06:00.269169+0000","last_peered":"2024-02-05T08:06:00.269169+0000","last_clean":"2024-02-05T08:06:00.269169+0000","last_became_active":"2024-02-05T08:04:41.537818+0000","last_became_peered":"2024-02-05T08:04:41.537818+0000","last_unstale":"2024-02-05T08:06:00.269169+0000","last_undegraded":"2024-02-05T08:06:00.269169+0000","last_fullsized":"2024-02-05T08:06:00.269169+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.324263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527293+0000","last_change":"2024-02-05T08:04:43.552388+0000","last_active":"2024-02-05T08:05:59.527293+0000","last_peered":"2024-02-05T08:05:59.527293+0000","last_clean":"2024-02-05T08:05:59.527293+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:05:59.527293+0000","last_undegraded":"2024-02-05T08:05:59.527293+0000","last_fullsized":"2024-02-05T08:05:59.527293+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:05.174347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528862+0000","last_change":"2024-02-05T08:05:57.266029+0000","last_active":"2024-02-05T08:05:59.528862+0000","last_peered":"2024-02-05T08:05:59.528862+0000","last_clean":"2024-02-05T08:05:59.528862+0000","last_became_active":"2024-02-05T08:04:45.562462+0000","last_became_peered":"2024-02-05T08:04:45.562462+0000","last_unstale":"2024-02-05T08:05:59.528862+0000","last_undegraded":"2024-02-05T08:05:59.528862+0000","last_fullsized":"2024-02-05T08:05:59.528862+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:05:57.265939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.265939+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:03.556401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718573+0000","last_change":"2024-02-05T08:04:48.172841+0000","last_active":"2024-02-05T08:05:59.718573+0000","last_peered":"2024-02-05T08:05:59.718573+0000","last_clean":"2024-02-05T08:05:59.718573+0000","last_became_active":"2024-02-05T08:04:48.172564+0000","last_became_peered":"2024-02-05T08:04:48.172564+0000","last_unstale":"2024-02-05T08:05:59.718573+0000","last_undegraded":"2024-02-05T08:05:59.718573+0000","last_fullsized":"2024-02-05T08:05:59.718573+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:15.681030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528317+0000","last_change":"2024-02-05T08:05:52.283473+0000","last_active":"2024-02-05T08:05:59.528317+0000","last_peered":"2024-02-05T08:05:59.528317+0000","last_clean":"2024-02-05T08:05:59.528317+0000","last_became_active":"2024-02-05T08:04:41.534433+0000","last_became_peered":"2024-02-05T08:04:41.534433+0000","last_unstale":"2024-02-05T08:05:59.528317+0000","last_undegraded":"2024-02-05T08:05:59.528317+0000","last_fullsized":"2024-02-05T08:05:59.528317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:05:52.283383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.283383+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:05.067021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529862+0000","last_change":"2024-02-05T08:04:43.551586+0000","last_active":"2024-02-05T08:05:59.529862+0000","last_peered":"2024-02-05T08:05:59.529862+0000","last_clean":"2024-02-05T08:05:59.529862+0000","last_became_active":"2024-02-05T08:04:43.551184+0000","last_became_peered":"2024-02-05T08:04:43.551184+0000","last_unstale":"2024-02-05T08:05:59.529862+0000","last_undegraded":"2024-02-05T08:05:59.529862+0000","last_fullsized":"2024-02-05T08:05:59.529862+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:05.037825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226052+0000","last_change":"2024-02-05T08:04:45.563011+0000","last_active":"2024-02-05T08:06:00.226052+0000","last_peered":"2024-02-05T08:06:00.226052+0000","last_clean":"2024-02-05T08:06:00.226052+0000","last_became_active":"2024-02-05T08:04:45.562597+0000","last_became_peered":"2024-02-05T08:04:45.562597+0000","last_unstale":"2024-02-05T08:06:00.226052+0000","last_undegraded":"2024-02-05T08:06:00.226052+0000","last_fullsized":"2024-02-05T08:06:00.226052+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.065463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.186855+0000","last_change":"2024-02-05T08:06:01.186878+0000","last_active":"2024-02-05T08:06:01.186855+0000","last_peered":"2024-02-05T08:06:01.186855+0000","last_clean":"2024-02-05T08:06:01.186855+0000","last_became_active":"2024-02-05T08:04:47.582776+0000","last_became_peered":"2024-02-05T08:04:47.582776+0000","last_unstale":"2024-02-05T08:06:01.186855+0000","last_undegraded":"2024-02-05T08:06:01.186855+0000","last_fullsized":"2024-02-05T08:06:01.186855+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:01.186782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:01.186782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.355302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226658+0000","last_change":"2024-02-05T08:05:48.267727+0000","last_active":"2024-02-05T08:06:00.226658+0000","last_peered":"2024-02-05T08:06:00.226658+0000","last_clean":"2024-02-05T08:06:00.226658+0000","last_became_active":"2024-02-05T08:04:41.540818+0000","last_became_peered":"2024-02-05T08:04:41.540818+0000","last_unstale":"2024-02-05T08:06:00.226658+0000","last_undegraded":"2024-02-05T08:06:00.226658+0000","last_fullsized":"2024-02-05T08:06:00.226658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:05:48.267582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.267582+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:15.800342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225869+0000","last_change":"2024-02-05T08:05:52.195604+0000","last_active":"2024-02-05T08:06:00.225869+0000","last_peered":"2024-02-05T08:06:00.225869+0000","last_clean":"2024-02-05T08:06:00.225869+0000","last_became_active":"2024-02-05T08:04:43.607546+0000","last_became_peered":"2024-02-05T08:04:43.607546+0000","last_unstale":"2024-02-05T08:06:00.225869+0000","last_undegraded":"2024-02-05T08:06:00.225869+0000","last_fullsized":"2024-02-05T08:06:00.225869+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-05T08:05:52.195494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.195494+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:16.333300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529710+0000","last_change":"2024-02-05T08:04:45.561907+0000","last_active":"2024-02-05T08:05:59.529710+0000","last_peered":"2024-02-05T08:05:59.529710+0000","last_clean":"2024-02-05T08:05:59.529710+0000","last_became_active":"2024-02-05T08:04:45.561639+0000","last_became_peered":"2024-02-05T08:04:45.561639+0000","last_unstale":"2024-02-05T08:05:59.529710+0000","last_undegraded":"2024-02-05T08:05:59.529710+0000","last_fullsized":"2024-02-05T08:05:59.529710+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.611331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268056+0000","last_change":"2024-02-05T08:05:48.449994+0000","last_active":"2024-02-05T08:06:00.268056+0000","last_peered":"2024-02-05T08:06:00.268056+0000","last_clean":"2024-02-05T08:06:00.268056+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:06:00.268056+0000","last_undegraded":"2024-02-05T08:06:00.268056+0000","last_fullsized":"2024-02-05T08:06:00.268056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:48.449900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.449900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:55.343526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530822+0000","last_change":"2024-02-05T08:04:45.567294+0000","last_active":"2024-02-05T08:05:59.530822+0000","last_peered":"2024-02-05T08:05:59.530822+0000","last_clean":"2024-02-05T08:05:59.530822+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:05:59.530822+0000","last_undegraded":"2024-02-05T08:05:59.530822+0000","last_fullsized":"2024-02-05T08:05:59.530822+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.907702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225833+0000","last_change":"2024-02-05T08:05:59.261588+0000","last_active":"2024-02-05T08:06:00.225833+0000","last_peered":"2024-02-05T08:06:00.225833+0000","last_clean":"2024-02-05T08:06:00.225833+0000","last_became_active":"2024-02-05T08:04:41.541327+0000","last_became_peered":"2024-02-05T08:04:41.541327+0000","last_unstale":"2024-02-05T08:06:00.225833+0000","last_undegraded":"2024-02-05T08:06:00.225833+0000","last_fullsized":"2024-02-05T08:06:00.225833+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:59.261506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:59.261506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:24.020924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.526944+0000","last_change":"2024-02-05T08:05:53.816142+0000","last_active":"2024-02-05T08:05:59.526944+0000","last_peered":"2024-02-05T08:05:59.526944+0000","last_clean":"2024-02-05T08:05:59.526944+0000","last_became_active":"2024-02-05T08:04:43.555028+0000","last_became_peered":"2024-02-05T08:04:43.555028+0000","last_unstale":"2024-02-05T08:05:59.526944+0000","last_undegraded":"2024-02-05T08:05:59.526944+0000","last_fullsized":"2024-02-05T08:05:59.526944+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:05:53.815997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.815997+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.642275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225515+0000","last_change":"2024-02-05T08:04:48.170254+0000","last_active":"2024-02-05T08:06:00.225515+0000","last_peered":"2024-02-05T08:06:00.225515+0000","last_clean":"2024-02-05T08:06:00.225515+0000","last_became_active":"2024-02-05T08:04:48.169762+0000","last_became_peered":"2024-02-05T08:04:48.169762+0000","last_unstale":"2024-02-05T08:06:00.225515+0000","last_undegraded":"2024-02-05T08:06:00.225515+0000","last_fullsized":"2024-02-05T08:06:00.225515+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:13.151401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.561022+0000","last_change":"2024-02-05T08:06:01.561064+0000","last_active":"2024-02-05T08:06:01.561022+0000","last_peered":"2024-02-05T08:06:01.561022+0000","last_clean":"2024-02-05T08:06:01.561022+0000","last_became_active":"2024-02-05T08:04:45.561679+0000","last_became_peered":"2024-02-05T08:04:45.561679+0000","last_unstale":"2024-02-05T08:06:01.561022+0000","last_undegraded":"2024-02-05T08:06:01.561022+0000","last_fullsized":"2024-02-05T08:06:01.561022+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:01.560919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:01.560919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:21.298440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529733+0000","last_change":"2024-02-05T08:05:48.116107+0000","last_active":"2024-02-05T08:05:59.529733+0000","last_peered":"2024-02-05T08:05:59.529733+0000","last_clean":"2024-02-05T08:05:59.529733+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:05:59.529733+0000","last_undegraded":"2024-02-05T08:05:59.529733+0000","last_fullsized":"2024-02-05T08:05:59.529733+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:48.116008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.116008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:06.262952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530647+0000","last_change":"2024-02-05T08:04:43.554584+0000","last_active":"2024-02-05T08:05:59.530647+0000","last_peered":"2024-02-05T08:05:59.530647+0000","last_clean":"2024-02-05T08:05:59.530647+0000","last_became_active":"2024-02-05T08:04:43.554161+0000","last_became_peered":"2024-02-05T08:04:43.554161+0000","last_unstale":"2024-02-05T08:05:59.530647+0000","last_undegraded":"2024-02-05T08:05:59.530647+0000","last_fullsized":"2024-02-05T08:05:59.530647+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.545953+0000","last_change":"2024-02-05T08:05:48.720004+0000","last_active":"2024-02-05T08:06:00.545953+0000","last_peered":"2024-02-05T08:06:00.545953+0000","last_clean":"2024-02-05T08:06:00.545953+0000","last_became_active":"2024-02-05T08:04:47.585962+0000","last_became_peered":"2024-02-05T08:04:47.585962+0000","last_unstale":"2024-02-05T08:06:00.545953+0000","last_undegraded":"2024-02-05T08:06:00.545953+0000","last_fullsized":"2024-02-05T08:06:00.545953+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:05:48.719923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.719923+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:54.016895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528031+0000","last_change":"2024-02-05T08:04:45.559587+0000","last_active":"2024-02-05T08:05:59.528031+0000","last_peered":"2024-02-05T08:05:59.528031+0000","last_clean":"2024-02-05T08:05:59.528031+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:05:59.528031+0000","last_undegraded":"2024-02-05T08:05:59.528031+0000","last_fullsized":"2024-02-05T08:05:59.528031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:02.577883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.721051+0000","last_change":"2024-02-05T08:05:50.230381+0000","last_active":"2024-02-05T08:05:59.721051+0000","last_peered":"2024-02-05T08:05:59.721051+0000","last_clean":"2024-02-05T08:05:59.721051+0000","last_became_active":"2024-02-05T08:04:41.538397+0000","last_became_peered":"2024-02-05T08:04:41.538397+0000","last_unstale":"2024-02-05T08:05:59.721051+0000","last_undegraded":"2024-02-05T08:05:59.721051+0000","last_fullsized":"2024-02-05T08:05:59.721051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:50.230263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:50.230263+0000","last_clean_scrub_stamp":"2024-02-05T08:05:50.230263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:59.311643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530628+0000","last_change":"2024-02-05T08:04:43.555191+0000","last_active":"2024-02-05T08:05:59.530628+0000","last_peered":"2024-02-05T08:05:59.530628+0000","last_clean":"2024-02-05T08:05:59.530628+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:05:59.530628+0000","last_undegraded":"2024-02-05T08:05:59.530628+0000","last_fullsized":"2024-02-05T08:05:59.530628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:12.495416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530787+0000","last_change":"2024-02-05T08:04:47.573584+0000","last_active":"2024-02-05T08:05:59.530787+0000","last_peered":"2024-02-05T08:05:59.530787+0000","last_clean":"2024-02-05T08:05:59.530787+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:05:59.530787+0000","last_undegraded":"2024-02-05T08:05:59.530787+0000","last_fullsized":"2024-02-05T08:05:59.530787+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.290723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546453+0000","last_change":"2024-02-05T08:04:45.566852+0000","last_active":"2024-02-05T08:06:00.546453+0000","last_peered":"2024-02-05T08:06:00.546453+0000","last_clean":"2024-02-05T08:06:00.546453+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:06:00.546453+0000","last_undegraded":"2024-02-05T08:06:00.546453+0000","last_fullsized":"2024-02-05T08:06:00.546453+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226287+0000","last_change":"2024-02-05T08:04:41.534743+0000","last_active":"2024-02-05T08:06:00.226287+0000","last_peered":"2024-02-05T08:06:00.226287+0000","last_clean":"2024-02-05T08:06:00.226287+0000","last_became_active":"2024-02-05T08:04:41.533838+0000","last_became_peered":"2024-02-05T08:04:41.533838+0000","last_unstale":"2024-02-05T08:06:00.226287+0000","last_undegraded":"2024-02-05T08:06:00.226287+0000","last_fullsized":"2024-02-05T08:06:00.226287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.461601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530284+0000","last_change":"2024-02-05T08:04:43.560240+0000","last_active":"2024-02-05T08:05:59.530284+0000","last_peered":"2024-02-05T08:05:59.530284+0000","last_clean":"2024-02-05T08:05:59.530284+0000","last_became_active":"2024-02-05T08:04:43.559922+0000","last_became_peered":"2024-02-05T08:04:43.559922+0000","last_unstale":"2024-02-05T08:05:59.530284+0000","last_undegraded":"2024-02-05T08:05:59.530284+0000","last_fullsized":"2024-02-05T08:05:59.530284+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":412,"ondisk_log_size":412,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":70,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037781,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32660,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712492,"statfs":{"total":95995035648,"available":95961591808,"internally_reserved":0,"allocated":1515520,"data_stored":1027404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.59799999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.58499999999999996}]},{"osd":3,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.48199999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.50800000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.49299999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.434},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.45200000000000001}]}]},{"osd":6,"up_from":51,"seq":219043332123,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1495040,"data_stored":1025877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Mon Feb 5 08:05:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.42799999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:05:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.44900000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:05:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.47799999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:05:01 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.40699999999999997}]},{"osd":5,"last update":"Mon Feb 5 08:05:01 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.57099999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:05:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54800000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":946176,"data_stored":436744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34399999999999997,"15min":0.31},"min":{"1min":0.216,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.37,"15min":0.32500000000000001},"min":{"1min":0.22900000000000001,"5min":0.16,"15min":0.16},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.19500000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.63700000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.41899999999999998,"15min":0.40500000000000003},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.48999999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:05:54 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.434,"15min":0.441},"min":{"1min":0.27200000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.47399999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.41399999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86199999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.45000000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.74399999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.78200000000000003}]},{"osd":5,"last update":"Mon Feb 5 08:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.627,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.752}]},{"osd":6,"last update":"Mon Feb 5 08:05:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.73799999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.85599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28504,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716648,"statfs":{"total":95995035648,"available":95965847552,"internally_reserved":0,"allocated":1519616,"data_stored":1027147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.36199999999999999,"15min":0.33400000000000002},"min":{"1min":0.25,"5min":0.152,"15min":0.152},"max":{"1min":0.501,"5min":0.621,"15min":0.621},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.37,"15min":0.33600000000000002},"min":{"1min":0.26500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.26500000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:05:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38500000000000001,"15min":0.372},"min":{"1min":0.252,"5min":0.218,"15min":0.218},"max":{"1min":0.60899999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40300000000000002,"15min":0.38900000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.28999999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.45200000000000001,"15min":0.45900000000000002},"min":{"1min":0.29599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71899999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.46300000000000002,"15min":0.46800000000000003},"min":{"1min":0.32800000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.58199999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.372}]},{"osd":4,"last update":"Mon Feb 5 08:05:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.79300000000000004,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.52800000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.74399999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60499999999999998,"15min":0.60599999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.71199999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:05:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.58799999999999997}]},{"osd":7,"last update":"Mon Feb 5 08:05:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61499999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32080,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713072,"statfs":{"total":95995035648,"available":95962185728,"internally_reserved":0,"allocated":921600,"data_stored":436186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:24 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38,"15min":0.37},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.57099999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.38200000000000001},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56899999999999995,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.39100000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:05:24 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.36799999999999999,"15min":0.35899999999999999},"min":{"1min":0.248,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.495,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.40200000000000002,"15min":0.38},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.623,"5min":0.63,"15min":0.63},"last":0.35599999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:05:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46999999999999997,"15min":0.46700000000000003},"min":{"1min":0.34499999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.58699999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56599999999999995,"5min":0.746,"15min":0.746},"last":0.374}]},{"osd":4,"last update":"Mon Feb 5 08:05:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.371,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83299999999999996,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.58599999999999997,"15min":0.57999999999999996},"min":{"1min":0.45500000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85699999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.69699999999999995}]},{"osd":5,"last update":"Mon Feb 5 08:05:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.746}]},{"osd":6,"last update":"Mon Feb 5 08:05:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.73299999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.75800000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117034,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":942080,"data_stored":436889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.438,"15min":0.45100000000000001},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.29799999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.46800000000000003,"15min":0.48199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70299999999999996,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.37},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.45000000000000001,"15min":0.46200000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.33200000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49099999999999999,"15min":0.497},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47499999999999998,"15min":0.48399999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.26800000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.55600000000000005}]},{"osd":5,"last update":"Mon Feb 5 08:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.495},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.46400000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.59299999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:05:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.53700000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:05:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.51200000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953509,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":435729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:09 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54600000000000004,"15min":0.54800000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.72099999999999997,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.41999999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:05:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.40600000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Feb 5 08:05:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77700000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:05:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.61299999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:05:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.38800000000000001,"15min":0.372},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.41299999999999998,"15min":0.39900000000000002},"min":{"1min":0.29099999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.56699999999999995}]},{"osd":6,"last update":"Mon Feb 5 08:05:02 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.71599999999999997}]},{"osd":7,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.74299999999999999}]}]},{"osd":5,"up_from":42,"seq":180388626463,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":901120,"data_stored":435436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.33100000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.72899999999999998,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.53200000000000003,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71299999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.47299999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.374,"15min":0.374},"max":{"1min":0.69999999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.35299999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.53100000000000003}]},{"osd":2,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56599999999999995,"15min":0.57099999999999995},"min":{"1min":0.34899999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76200000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.373,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.48599999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55600000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.749,"15min":0.749},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.70999999999999996}]},{"osd":4,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.41999999999999998,"15min":0.41799999999999998},"min":{"1min":0.28199999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60599999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Feb 5 08:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45400000000000001,"15min":0.44600000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.625},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.42299999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.51700000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:05:30 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.59199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:06:07.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:06:07.267 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:07.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:06:07.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: 4.c scrub starts 2024-02-05T08:06:07.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: 4.c scrub ok 2024-02-05T08:06:07.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:06:07.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:06:07.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:06:07.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.644 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T08:06:07.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.645 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:06:07.647 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:06:07.647 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: 4.c scrub starts 2024-02-05T08:06:07.647 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: 4.c scrub ok 2024-02-05T08:06:07.647 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:06:07.647 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:06:07.647 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:06:07.647 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.647 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.648 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.648 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.648 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.648 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.648 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T08:06:07.648 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.648 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.648 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:06:07.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:06:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: 4.c scrub starts 2024-02-05T08:06:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: 4.c scrub ok 2024-02-05T08:06:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:06:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:06:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:06:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T08:06:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:07.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:06:07.979 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:06:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:06:08.527 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[77504]: ts=2024-02-05T08:06:08.240Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000421191s 2024-02-05T08:06:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[30086]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-05T08:06:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[30086]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T08:06:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[30086]: Deploying daemon grafana.a on smithi196 2024-02-05T08:06:08.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[30086]: 3.18 deep-scrub starts 2024-02-05T08:06:08.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[30086]: 3.18 deep-scrub ok 2024-02-05T08:06:08.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[30086]: 5.8 deep-scrub starts 2024-02-05T08:06:08.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[30086]: 5.8 deep-scrub ok 2024-02-05T08:06:08.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[39278]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-05T08:06:08.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[39278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T08:06:08.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[39278]: Deploying daemon grafana.a on smithi196 2024-02-05T08:06:08.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[39278]: 3.18 deep-scrub starts 2024-02-05T08:06:08.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[39278]: 3.18 deep-scrub ok 2024-02-05T08:06:08.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[39278]: 5.8 deep-scrub starts 2024-02-05T08:06:08.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:08 smithi134 ceph-mon[39278]: 5.8 deep-scrub ok 2024-02-05T08:06:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:08 smithi196 ceph-mon[33018]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-05T08:06:08.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:08 smithi196 ceph-mon[33018]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T08:06:08.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:08 smithi196 ceph-mon[33018]: Deploying daemon grafana.a on smithi196 2024-02-05T08:06:08.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:08 smithi196 ceph-mon[33018]: 3.18 deep-scrub starts 2024-02-05T08:06:08.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:08 smithi196 ceph-mon[33018]: 3.18 deep-scrub ok 2024-02-05T08:06:08.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:08 smithi196 ceph-mon[33018]: 5.8 deep-scrub starts 2024-02-05T08:06:08.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:08 smithi196 ceph-mon[33018]: 5.8 deep-scrub ok 2024-02-05T08:06:09.106 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:06:09.108 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:06:09.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:09 smithi134 ceph-mon[30086]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:06:09.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:09 smithi134 ceph-mon[39278]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:06:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:09 smithi196 ceph-mon[33018]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:06:09.865 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-02-05T08:06:07.490546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1013,"num_read_kb":859,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":622,"ondisk_log_size":622,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":245884,"kb_used_data":8956,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749715332,"statfs":{"total":767960285184,"available":767708499968,"internally_reserved":0,"allocated":9170944,"data_stored":5261412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":242535987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002519"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529806+0000","last_change":"2024-02-05T08:05:55.114592+0000","last_active":"2024-02-05T08:05:59.529806+0000","last_peered":"2024-02-05T08:05:59.529806+0000","last_clean":"2024-02-05T08:05:59.529806+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:05:59.529806+0000","last_undegraded":"2024-02-05T08:05:59.529806+0000","last_fullsized":"2024-02-05T08:05:59.529806+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:55.114498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.114498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.383006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527645+0000","last_change":"2024-02-05T08:04:45.560680+0000","last_active":"2024-02-05T08:05:59.527645+0000","last_peered":"2024-02-05T08:05:59.527645+0000","last_clean":"2024-02-05T08:05:59.527645+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:05:59.527645+0000","last_undegraded":"2024-02-05T08:05:59.527645+0000","last_fullsized":"2024-02-05T08:05:59.527645+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527961+0000","last_change":"2024-02-05T08:05:53.302720+0000","last_active":"2024-02-05T08:05:59.527961+0000","last_peered":"2024-02-05T08:05:59.527961+0000","last_clean":"2024-02-05T08:05:59.527961+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:05:59.527961+0000","last_undegraded":"2024-02-05T08:05:59.527961+0000","last_fullsized":"2024-02-05T08:05:59.527961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:53.302642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.302642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:57.391075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"90'13","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529707+0000","last_change":"2024-02-05T08:05:58.103861+0000","last_active":"2024-02-05T08:05:59.529707+0000","last_peered":"2024-02-05T08:05:59.529707+0000","last_clean":"2024-02-05T08:05:59.529707+0000","last_became_active":"2024-02-05T08:04:43.551169+0000","last_became_peered":"2024-02-05T08:04:43.551169+0000","last_unstale":"2024-02-05T08:05:59.529707+0000","last_undegraded":"2024-02-05T08:05:59.529707+0000","last_fullsized":"2024-02-05T08:05:59.529707+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.103749+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:14.381872+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.227351+0000","last_change":"2024-02-05T08:06:00.227371+0000","last_active":"2024-02-05T08:06:00.227351+0000","last_peered":"2024-02-05T08:06:00.227351+0000","last_clean":"2024-02-05T08:06:00.227351+0000","last_became_active":"2024-02-05T08:04:48.172811+0000","last_became_peered":"2024-02-05T08:04:48.172811+0000","last_unstale":"2024-02-05T08:06:00.227351+0000","last_undegraded":"2024-02-05T08:06:00.227351+0000","last_fullsized":"2024-02-05T08:06:00.227351+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:00.227295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:00.227295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:06.714686+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529726+0000","last_change":"2024-02-05T08:04:45.564397+0000","last_active":"2024-02-05T08:05:59.529726+0000","last_peered":"2024-02-05T08:05:59.529726+0000","last_clean":"2024-02-05T08:05:59.529726+0000","last_became_active":"2024-02-05T08:04:45.563795+0000","last_became_peered":"2024-02-05T08:04:45.563795+0000","last_unstale":"2024-02-05T08:05:59.529726+0000","last_undegraded":"2024-02-05T08:05:59.529726+0000","last_fullsized":"2024-02-05T08:05:59.529726+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.204318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226552+0000","last_change":"2024-02-05T08:05:42.259337+0000","last_active":"2024-02-05T08:06:00.226552+0000","last_peered":"2024-02-05T08:06:00.226552+0000","last_clean":"2024-02-05T08:06:00.226552+0000","last_became_active":"2024-02-05T08:04:41.535435+0000","last_became_peered":"2024-02-05T08:04:41.535435+0000","last_unstale":"2024-02-05T08:06:00.226552+0000","last_undegraded":"2024-02-05T08:06:00.226552+0000","last_fullsized":"2024-02-05T08:06:00.226552+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:42.259257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:42.259257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:48.329795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226085+0000","last_change":"2024-02-05T08:04:43.560996+0000","last_active":"2024-02-05T08:06:00.226085+0000","last_peered":"2024-02-05T08:06:00.226085+0000","last_clean":"2024-02-05T08:06:00.226085+0000","last_became_active":"2024-02-05T08:04:43.560660+0000","last_became_peered":"2024-02-05T08:04:43.560660+0000","last_unstale":"2024-02-05T08:06:00.226085+0000","last_undegraded":"2024-02-05T08:06:00.226085+0000","last_fullsized":"2024-02-05T08:06:00.226085+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528432+0000","last_change":"2024-02-05T08:05:56.285954+0000","last_active":"2024-02-05T08:05:59.528432+0000","last_peered":"2024-02-05T08:05:59.528432+0000","last_clean":"2024-02-05T08:05:59.528432+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:05:59.528432+0000","last_undegraded":"2024-02-05T08:05:59.528432+0000","last_fullsized":"2024-02-05T08:05:59.528432+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:56.285876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:56.285876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:18.137707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268950+0000","last_change":"2024-02-05T08:04:45.560281+0000","last_active":"2024-02-05T08:06:00.268950+0000","last_peered":"2024-02-05T08:06:00.268950+0000","last_clean":"2024-02-05T08:06:00.268950+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:06:00.268950+0000","last_undegraded":"2024-02-05T08:06:00.268950+0000","last_fullsized":"2024-02-05T08:06:00.268950+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:14.202262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269001+0000","last_change":"2024-02-05T08:05:55.576535+0000","last_active":"2024-02-05T08:06:00.269001+0000","last_peered":"2024-02-05T08:06:00.269001+0000","last_clean":"2024-02-05T08:06:00.269001+0000","last_became_active":"2024-02-05T08:04:41.540764+0000","last_became_peered":"2024-02-05T08:04:41.540764+0000","last_unstale":"2024-02-05T08:06:00.269001+0000","last_undegraded":"2024-02-05T08:06:00.269001+0000","last_fullsized":"2024-02-05T08:06:00.269001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:55.576441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.576441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.917805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529949+0000","last_change":"2024-02-05T08:04:43.557923+0000","last_active":"2024-02-05T08:05:59.529949+0000","last_peered":"2024-02-05T08:05:59.529949+0000","last_clean":"2024-02-05T08:05:59.529949+0000","last_became_active":"2024-02-05T08:04:43.557652+0000","last_became_peered":"2024-02-05T08:04:43.557652+0000","last_unstale":"2024-02-05T08:05:59.529949+0000","last_undegraded":"2024-02-05T08:05:59.529949+0000","last_fullsized":"2024-02-05T08:05:59.529949+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.987638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.267980+0000","last_change":"2024-02-05T08:04:47.587766+0000","last_active":"2024-02-05T08:06:00.267980+0000","last_peered":"2024-02-05T08:06:00.267980+0000","last_clean":"2024-02-05T08:06:00.267980+0000","last_became_active":"2024-02-05T08:04:47.587515+0000","last_became_peered":"2024-02-05T08:04:47.587515+0000","last_unstale":"2024-02-05T08:06:00.267980+0000","last_undegraded":"2024-02-05T08:06:00.267980+0000","last_fullsized":"2024-02-05T08:06:00.267980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.970614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718175+0000","last_change":"2024-02-05T08:05:52.725804+0000","last_active":"2024-02-05T08:05:59.718175+0000","last_peered":"2024-02-05T08:05:59.718175+0000","last_clean":"2024-02-05T08:05:59.718175+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:05:59.718175+0000","last_undegraded":"2024-02-05T08:05:59.718175+0000","last_fullsized":"2024-02-05T08:05:59.718175+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:52.725679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.725679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.406845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268975+0000","last_change":"2024-02-05T08:05:46.458359+0000","last_active":"2024-02-05T08:06:00.268975+0000","last_peered":"2024-02-05T08:06:00.268975+0000","last_clean":"2024-02-05T08:06:00.268975+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:06:00.268975+0000","last_undegraded":"2024-02-05T08:06:00.268975+0000","last_fullsized":"2024-02-05T08:06:00.268975+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.458244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.877756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530213+0000","last_change":"2024-02-05T08:04:43.557900+0000","last_active":"2024-02-05T08:05:59.530213+0000","last_peered":"2024-02-05T08:05:59.530213+0000","last_clean":"2024-02-05T08:05:59.530213+0000","last_became_active":"2024-02-05T08:04:43.557559+0000","last_became_peered":"2024-02-05T08:04:43.557559+0000","last_unstale":"2024-02-05T08:05:59.530213+0000","last_undegraded":"2024-02-05T08:05:59.530213+0000","last_fullsized":"2024-02-05T08:05:59.530213+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:09.465379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530663+0000","last_change":"2024-02-05T08:05:51.076610+0000","last_active":"2024-02-05T08:05:59.530663+0000","last_peered":"2024-02-05T08:05:59.530663+0000","last_clean":"2024-02-05T08:05:59.530663+0000","last_became_active":"2024-02-05T08:04:48.170756+0000","last_became_peered":"2024-02-05T08:04:48.170756+0000","last_unstale":"2024-02-05T08:05:59.530663+0000","last_undegraded":"2024-02-05T08:05:59.530663+0000","last_fullsized":"2024-02-05T08:05:59.530663+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:51.076514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:51.076514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:11.206152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527244+0000","last_change":"2024-02-05T08:04:45.559908+0000","last_active":"2024-02-05T08:05:59.527244+0000","last_peered":"2024-02-05T08:05:59.527244+0000","last_clean":"2024-02-05T08:05:59.527244+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:05:59.527244+0000","last_undegraded":"2024-02-05T08:05:59.527244+0000","last_fullsized":"2024-02-05T08:05:59.527244+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:11.185330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528706+0000","last_change":"2024-02-05T08:04:41.541812+0000","last_active":"2024-02-05T08:05:59.528706+0000","last_peered":"2024-02-05T08:05:59.528706+0000","last_clean":"2024-02-05T08:05:59.528706+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:05:59.528706+0000","last_undegraded":"2024-02-05T08:05:59.528706+0000","last_fullsized":"2024-02-05T08:05:59.528706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546521+0000","last_change":"2024-02-05T08:05:46.814239+0000","last_active":"2024-02-05T08:06:00.546521+0000","last_peered":"2024-02-05T08:06:00.546521+0000","last_clean":"2024-02-05T08:06:00.546521+0000","last_became_active":"2024-02-05T08:04:43.557496+0000","last_became_peered":"2024-02-05T08:04:43.557496+0000","last_unstale":"2024-02-05T08:06:00.546521+0000","last_undegraded":"2024-02-05T08:06:00.546521+0000","last_fullsized":"2024-02-05T08:06:00.546521+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:05:46.814139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.814139+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:58.081182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225446+0000","last_change":"2024-02-05T08:04:48.170578+0000","last_active":"2024-02-05T08:06:00.225446+0000","last_peered":"2024-02-05T08:06:00.225446+0000","last_clean":"2024-02-05T08:06:00.225446+0000","last_became_active":"2024-02-05T08:04:48.169770+0000","last_became_peered":"2024-02-05T08:04:48.169770+0000","last_unstale":"2024-02-05T08:06:00.225446+0000","last_undegraded":"2024-02-05T08:06:00.225446+0000","last_fullsized":"2024-02-05T08:06:00.225446+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.961379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530140+0000","last_change":"2024-02-05T08:05:57.052122+0000","last_active":"2024-02-05T08:05:59.530140+0000","last_peered":"2024-02-05T08:05:59.530140+0000","last_clean":"2024-02-05T08:05:59.530140+0000","last_became_active":"2024-02-05T08:04:45.563948+0000","last_became_peered":"2024-02-05T08:04:45.563948+0000","last_unstale":"2024-02-05T08:05:59.530140+0000","last_undegraded":"2024-02-05T08:05:59.530140+0000","last_fullsized":"2024-02-05T08:05:59.530140+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.052030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:14.589576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530056+0000","last_change":"2024-02-05T08:04:41.534341+0000","last_active":"2024-02-05T08:05:59.530056+0000","last_peered":"2024-02-05T08:05:59.530056+0000","last_clean":"2024-02-05T08:05:59.530056+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:05:59.530056+0000","last_undegraded":"2024-02-05T08:05:59.530056+0000","last_fullsized":"2024-02-05T08:05:59.530056+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.393385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"90'17","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226406+0000","last_change":"2024-02-05T08:05:58.285391+0000","last_active":"2024-02-05T08:06:00.226406+0000","last_peered":"2024-02-05T08:06:00.226406+0000","last_clean":"2024-02-05T08:06:00.226406+0000","last_became_active":"2024-02-05T08:04:43.557596+0000","last_became_peered":"2024-02-05T08:04:43.557596+0000","last_unstale":"2024-02-05T08:06:00.226406+0000","last_undegraded":"2024-02-05T08:06:00.226406+0000","last_fullsized":"2024-02-05T08:06:00.226406+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.285304+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.927379+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718547+0000","last_change":"2024-02-05T08:05:55.766576+0000","last_active":"2024-02-05T08:05:59.718547+0000","last_peered":"2024-02-05T08:05:59.718547+0000","last_clean":"2024-02-05T08:05:59.718547+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:05:59.718547+0000","last_undegraded":"2024-02-05T08:05:59.718547+0000","last_fullsized":"2024-02-05T08:05:59.718547+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:05:55.766478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.766478+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:59.016044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.600922+0000","last_change":"2024-02-05T08:04:45.565057+0000","last_active":"2024-02-05T08:06:01.600922+0000","last_peered":"2024-02-05T08:06:01.600922+0000","last_clean":"2024-02-05T08:06:01.600922+0000","last_became_active":"2024-02-05T08:04:45.564774+0000","last_became_peered":"2024-02-05T08:04:45.564774+0000","last_unstale":"2024-02-05T08:06:01.600922+0000","last_undegraded":"2024-02-05T08:06:01.600922+0000","last_fullsized":"2024-02-05T08:06:01.600922+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527726+0000","last_change":"2024-02-05T08:04:41.532653+0000","last_active":"2024-02-05T08:05:59.527726+0000","last_peered":"2024-02-05T08:05:59.527726+0000","last_clean":"2024-02-05T08:05:59.527726+0000","last_became_active":"2024-02-05T08:04:41.532339+0000","last_became_peered":"2024-02-05T08:04:41.532339+0000","last_unstale":"2024-02-05T08:05:59.527726+0000","last_undegraded":"2024-02-05T08:05:59.527726+0000","last_fullsized":"2024-02-05T08:05:59.527726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528405+0000","last_change":"2024-02-05T08:04:43.606275+0000","last_active":"2024-02-05T08:05:59.528405+0000","last_peered":"2024-02-05T08:05:59.528405+0000","last_clean":"2024-02-05T08:05:59.528405+0000","last_became_active":"2024-02-05T08:04:43.605952+0000","last_became_peered":"2024-02-05T08:04:43.605952+0000","last_unstale":"2024-02-05T08:05:59.528405+0000","last_undegraded":"2024-02-05T08:05:59.528405+0000","last_fullsized":"2024-02-05T08:05:59.528405+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:09.695654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527368+0000","last_change":"2024-02-05T08:04:47.578164+0000","last_active":"2024-02-05T08:05:59.527368+0000","last_peered":"2024-02-05T08:05:59.527368+0000","last_clean":"2024-02-05T08:05:59.527368+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:05:59.527368+0000","last_undegraded":"2024-02-05T08:05:59.527368+0000","last_fullsized":"2024-02-05T08:05:59.527368+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528636+0000","last_change":"2024-02-05T08:04:45.564760+0000","last_active":"2024-02-05T08:05:59.528636+0000","last_peered":"2024-02-05T08:05:59.528636+0000","last_clean":"2024-02-05T08:05:59.528636+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:05:59.528636+0000","last_undegraded":"2024-02-05T08:05:59.528636+0000","last_fullsized":"2024-02-05T08:05:59.528636+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.404706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:07.137987+0000","last_change":"2024-02-05T08:06:07.138006+0000","last_active":"2024-02-05T08:06:07.137987+0000","last_peered":"2024-02-05T08:06:07.137987+0000","last_clean":"2024-02-05T08:06:07.137987+0000","last_became_active":"2024-02-05T08:04:41.534448+0000","last_became_peered":"2024-02-05T08:04:41.534448+0000","last_unstale":"2024-02-05T08:06:07.137987+0000","last_undegraded":"2024-02-05T08:06:07.137987+0000","last_fullsized":"2024-02-05T08:06:07.137987+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:06:07.137916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:29.601881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.721085+0000","last_change":"2024-02-05T08:05:46.208637+0000","last_active":"2024-02-05T08:05:59.721085+0000","last_peered":"2024-02-05T08:05:59.721085+0000","last_clean":"2024-02-05T08:05:59.721085+0000","last_became_active":"2024-02-05T08:05:03.789351+0000","last_became_peered":"2024-02-05T08:05:03.789351+0000","last_unstale":"2024-02-05T08:05:59.721085+0000","last_undegraded":"2024-02-05T08:05:59.721085+0000","last_fullsized":"2024-02-05T08:05:59.721085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:05:46.208534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.208534+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:00.233101+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.717804+0000","last_change":"2024-02-05T08:04:47.579487+0000","last_active":"2024-02-05T08:05:59.717804+0000","last_peered":"2024-02-05T08:05:59.717804+0000","last_clean":"2024-02-05T08:05:59.717804+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:05:59.717804+0000","last_undegraded":"2024-02-05T08:05:59.717804+0000","last_fullsized":"2024-02-05T08:05:59.717804+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:10.763840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720346+0000","last_change":"2024-02-05T08:05:54.367693+0000","last_active":"2024-02-05T08:05:59.720346+0000","last_peered":"2024-02-05T08:05:59.720346+0000","last_clean":"2024-02-05T08:05:59.720346+0000","last_became_active":"2024-02-05T08:04:45.560874+0000","last_became_peered":"2024-02-05T08:04:45.560874+0000","last_unstale":"2024-02-05T08:05:59.720346+0000","last_undegraded":"2024-02-05T08:05:59.720346+0000","last_fullsized":"2024-02-05T08:05:59.720346+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:05:54.367588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:17.218061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528041+0000","last_change":"2024-02-05T08:04:41.536675+0000","last_active":"2024-02-05T08:05:59.528041+0000","last_peered":"2024-02-05T08:05:59.528041+0000","last_clean":"2024-02-05T08:05:59.528041+0000","last_became_active":"2024-02-05T08:04:41.536019+0000","last_became_peered":"2024-02-05T08:04:41.536019+0000","last_unstale":"2024-02-05T08:05:59.528041+0000","last_undegraded":"2024-02-05T08:05:59.528041+0000","last_fullsized":"2024-02-05T08:05:59.528041+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225759+0000","last_change":"2024-02-05T08:04:43.555567+0000","last_active":"2024-02-05T08:06:00.225759+0000","last_peered":"2024-02-05T08:06:00.225759+0000","last_clean":"2024-02-05T08:06:00.225759+0000","last_became_active":"2024-02-05T08:04:43.552866+0000","last_became_peered":"2024-02-05T08:04:43.552866+0000","last_unstale":"2024-02-05T08:06:00.225759+0000","last_undegraded":"2024-02-05T08:06:00.225759+0000","last_fullsized":"2024-02-05T08:06:00.225759+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:03.755886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:02.190986+0000","last_change":"2024-02-05T08:06:02.191006+0000","last_active":"2024-02-05T08:06:02.190986+0000","last_peered":"2024-02-05T08:06:02.190986+0000","last_clean":"2024-02-05T08:06:02.190986+0000","last_became_active":"2024-02-05T08:04:48.172793+0000","last_became_peered":"2024-02-05T08:04:48.172793+0000","last_unstale":"2024-02-05T08:06:02.190986+0000","last_undegraded":"2024-02-05T08:06:02.190986+0000","last_fullsized":"2024-02-05T08:06:02.190986+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:02.190931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:02.190931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:27.955974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718344+0000","last_change":"2024-02-05T08:05:48.748227+0000","last_active":"2024-02-05T08:05:59.718344+0000","last_peered":"2024-02-05T08:05:59.718344+0000","last_clean":"2024-02-05T08:05:59.718344+0000","last_became_active":"2024-02-05T08:04:45.563610+0000","last_became_peered":"2024-02-05T08:04:45.563610+0000","last_unstale":"2024-02-05T08:05:59.718344+0000","last_undegraded":"2024-02-05T08:05:59.718344+0000","last_fullsized":"2024-02-05T08:05:59.718344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:48.748132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:48.748132+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.748132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:17.563379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:02.532340+0000","last_change":"2024-02-05T08:06:02.532375+0000","last_active":"2024-02-05T08:06:02.532340+0000","last_peered":"2024-02-05T08:06:02.532340+0000","last_clean":"2024-02-05T08:06:02.532340+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:06:02.532340+0000","last_undegraded":"2024-02-05T08:06:02.532340+0000","last_fullsized":"2024-02-05T08:06:02.532340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:02.532248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:02.532248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:05.965893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226059+0000","last_change":"2024-02-05T08:04:43.606219+0000","last_active":"2024-02-05T08:06:00.226059+0000","last_peered":"2024-02-05T08:06:00.226059+0000","last_clean":"2024-02-05T08:06:00.226059+0000","last_became_active":"2024-02-05T08:04:43.605932+0000","last_became_peered":"2024-02-05T08:04:43.605932+0000","last_unstale":"2024-02-05T08:06:00.226059+0000","last_undegraded":"2024-02-05T08:06:00.226059+0000","last_fullsized":"2024-02-05T08:06:00.226059+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:00.517159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:03.183974+0000","last_change":"2024-02-05T08:06:03.184008+0000","last_active":"2024-02-05T08:06:03.183974+0000","last_peered":"2024-02-05T08:06:03.183974+0000","last_clean":"2024-02-05T08:06:03.183974+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:06:03.183974+0000","last_undegraded":"2024-02-05T08:06:03.183974+0000","last_fullsized":"2024-02-05T08:06:03.183974+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:03.183906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:03.183906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:24.748158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226266+0000","last_change":"2024-02-05T08:05:57.270638+0000","last_active":"2024-02-05T08:06:00.226266+0000","last_peered":"2024-02-05T08:06:00.226266+0000","last_clean":"2024-02-05T08:06:00.226266+0000","last_became_active":"2024-02-05T08:04:45.568484+0000","last_became_peered":"2024-02-05T08:04:45.568484+0000","last_unstale":"2024-02-05T08:06:00.226266+0000","last_undegraded":"2024-02-05T08:06:00.226266+0000","last_fullsized":"2024-02-05T08:06:00.226266+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:57.270554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.270554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:03.898076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718661+0000","last_change":"2024-02-05T08:04:41.540005+0000","last_active":"2024-02-05T08:05:59.718661+0000","last_peered":"2024-02-05T08:05:59.718661+0000","last_clean":"2024-02-05T08:05:59.718661+0000","last_became_active":"2024-02-05T08:04:41.539698+0000","last_became_peered":"2024-02-05T08:04:41.539698+0000","last_unstale":"2024-02-05T08:05:59.718661+0000","last_undegraded":"2024-02-05T08:05:59.718661+0000","last_fullsized":"2024-02-05T08:05:59.718661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.105779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527388+0000","last_change":"2024-02-05T08:05:48.910387+0000","last_active":"2024-02-05T08:05:59.527388+0000","last_peered":"2024-02-05T08:05:59.527388+0000","last_clean":"2024-02-05T08:05:59.527388+0000","last_became_active":"2024-02-05T08:04:43.555031+0000","last_became_peered":"2024-02-05T08:04:43.555031+0000","last_unstale":"2024-02-05T08:05:59.527388+0000","last_undegraded":"2024-02-05T08:05:59.527388+0000","last_fullsized":"2024-02-05T08:05:59.527388+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:05:48.910304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.910304+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:49.156858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225328+0000","last_change":"2024-02-05T08:04:47.586079+0000","last_active":"2024-02-05T08:06:00.225328+0000","last_peered":"2024-02-05T08:06:00.225328+0000","last_clean":"2024-02-05T08:06:00.225328+0000","last_became_active":"2024-02-05T08:04:47.585496+0000","last_became_peered":"2024-02-05T08:04:47.585496+0000","last_unstale":"2024-02-05T08:06:00.225328+0000","last_undegraded":"2024-02-05T08:06:00.225328+0000","last_fullsized":"2024-02-05T08:06:00.225328+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:12.376304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527208+0000","last_change":"2024-02-05T08:05:46.864673+0000","last_active":"2024-02-05T08:05:59.527208+0000","last_peered":"2024-02-05T08:05:59.527208+0000","last_clean":"2024-02-05T08:05:59.527208+0000","last_became_active":"2024-02-05T08:04:45.560815+0000","last_became_peered":"2024-02-05T08:04:45.560815+0000","last_unstale":"2024-02-05T08:05:59.527208+0000","last_undegraded":"2024-02-05T08:05:59.527208+0000","last_fullsized":"2024-02-05T08:05:59.527208+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:46.864589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.864589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.980095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530377+0000","last_change":"2024-02-05T08:05:53.112020+0000","last_active":"2024-02-05T08:05:59.530377+0000","last_peered":"2024-02-05T08:05:59.530377+0000","last_clean":"2024-02-05T08:05:59.530377+0000","last_became_active":"2024-02-05T08:04:41.543255+0000","last_became_peered":"2024-02-05T08:04:41.543255+0000","last_unstale":"2024-02-05T08:05:59.530377+0000","last_undegraded":"2024-02-05T08:05:59.530377+0000","last_fullsized":"2024-02-05T08:05:59.530377+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:53.111931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.111931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:19.887019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530372+0000","last_change":"2024-02-05T08:04:43.554427+0000","last_active":"2024-02-05T08:05:59.530372+0000","last_peered":"2024-02-05T08:05:59.530372+0000","last_clean":"2024-02-05T08:05:59.530372+0000","last_became_active":"2024-02-05T08:04:43.554101+0000","last_became_peered":"2024-02-05T08:04:43.554101+0000","last_unstale":"2024-02-05T08:05:59.530372+0000","last_undegraded":"2024-02-05T08:05:59.530372+0000","last_fullsized":"2024-02-05T08:05:59.530372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:05.485949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528132+0000","last_change":"2024-02-05T08:04:47.584516+0000","last_active":"2024-02-05T08:05:59.528132+0000","last_peered":"2024-02-05T08:05:59.528132+0000","last_clean":"2024-02-05T08:05:59.528132+0000","last_became_active":"2024-02-05T08:04:47.584235+0000","last_became_peered":"2024-02-05T08:04:47.584235+0000","last_unstale":"2024-02-05T08:05:59.528132+0000","last_undegraded":"2024-02-05T08:05:59.528132+0000","last_fullsized":"2024-02-05T08:05:59.528132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:11.950403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.601081+0000","last_change":"2024-02-05T08:04:45.559777+0000","last_active":"2024-02-05T08:06:01.601081+0000","last_peered":"2024-02-05T08:06:01.601081+0000","last_clean":"2024-02-05T08:06:01.601081+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:06:01.601081+0000","last_undegraded":"2024-02-05T08:06:01.601081+0000","last_fullsized":"2024-02-05T08:06:01.601081+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718361+0000","last_change":"2024-02-05T08:05:45.722765+0000","last_active":"2024-02-05T08:05:59.718361+0000","last_peered":"2024-02-05T08:05:59.718361+0000","last_clean":"2024-02-05T08:05:59.718361+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:05:59.718361+0000","last_undegraded":"2024-02-05T08:05:59.718361+0000","last_fullsized":"2024-02-05T08:05:59.718361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:45.722652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:45.722652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.867081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226318+0000","last_change":"2024-02-05T08:04:43.606315+0000","last_active":"2024-02-05T08:06:00.226318+0000","last_peered":"2024-02-05T08:06:00.226318+0000","last_clean":"2024-02-05T08:06:00.226318+0000","last_became_active":"2024-02-05T08:04:43.605981+0000","last_became_peered":"2024-02-05T08:04:43.605981+0000","last_unstale":"2024-02-05T08:06:00.226318+0000","last_undegraded":"2024-02-05T08:06:00.226318+0000","last_fullsized":"2024-02-05T08:06:00.226318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530817+0000","last_change":"2024-02-05T08:04:47.578924+0000","last_active":"2024-02-05T08:05:59.530817+0000","last_peered":"2024-02-05T08:05:59.530817+0000","last_clean":"2024-02-05T08:05:59.530817+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:05:59.530817+0000","last_undegraded":"2024-02-05T08:05:59.530817+0000","last_fullsized":"2024-02-05T08:05:59.530817+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:03.039774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.600989+0000","last_change":"2024-02-05T08:04:45.566275+0000","last_active":"2024-02-05T08:06:01.600989+0000","last_peered":"2024-02-05T08:06:01.600989+0000","last_clean":"2024-02-05T08:06:01.600989+0000","last_became_active":"2024-02-05T08:04:45.565240+0000","last_became_peered":"2024-02-05T08:04:45.565240+0000","last_unstale":"2024-02-05T08:06:01.600989+0000","last_undegraded":"2024-02-05T08:06:01.600989+0000","last_fullsized":"2024-02-05T08:06:01.600989+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527536+0000","last_change":"2024-02-05T08:04:41.540528+0000","last_active":"2024-02-05T08:05:59.527536+0000","last_peered":"2024-02-05T08:05:59.527536+0000","last_clean":"2024-02-05T08:05:59.527536+0000","last_became_active":"2024-02-05T08:04:41.540252+0000","last_became_peered":"2024-02-05T08:04:41.540252+0000","last_unstale":"2024-02-05T08:05:59.527536+0000","last_undegraded":"2024-02-05T08:05:59.527536+0000","last_fullsized":"2024-02-05T08:05:59.527536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:02.085874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269241+0000","last_change":"2024-02-05T08:05:56.541620+0000","last_active":"2024-02-05T08:06:00.269241+0000","last_peered":"2024-02-05T08:06:00.269241+0000","last_clean":"2024-02-05T08:06:00.269241+0000","last_became_active":"2024-02-05T08:04:43.606414+0000","last_became_peered":"2024-02-05T08:04:43.606414+0000","last_unstale":"2024-02-05T08:06:00.269241+0000","last_undegraded":"2024-02-05T08:06:00.269241+0000","last_fullsized":"2024-02-05T08:06:00.269241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:05:56.541510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:56.541510+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:04.755122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.545844+0000","last_change":"2024-02-05T08:04:47.578623+0000","last_active":"2024-02-05T08:06:00.545844+0000","last_peered":"2024-02-05T08:06:00.545844+0000","last_clean":"2024-02-05T08:06:00.545844+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:06:00.545844+0000","last_undegraded":"2024-02-05T08:06:00.545844+0000","last_fullsized":"2024-02-05T08:06:00.545844+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.088742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:04.157672+0000","last_change":"2024-02-05T08:06:04.157708+0000","last_active":"2024-02-05T08:06:04.157672+0000","last_peered":"2024-02-05T08:06:04.157672+0000","last_clean":"2024-02-05T08:06:04.157672+0000","last_became_active":"2024-02-05T08:04:45.567622+0000","last_became_peered":"2024-02-05T08:04:45.567622+0000","last_unstale":"2024-02-05T08:06:04.157672+0000","last_undegraded":"2024-02-05T08:06:04.157672+0000","last_fullsized":"2024-02-05T08:06:04.157672+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:04.157598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:04.157598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:15.237706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718248+0000","last_change":"2024-02-05T08:04:41.541120+0000","last_active":"2024-02-05T08:05:59.718248+0000","last_peered":"2024-02-05T08:05:59.718248+0000","last_clean":"2024-02-05T08:05:59.718248+0000","last_became_active":"2024-02-05T08:04:41.540432+0000","last_became_peered":"2024-02-05T08:04:41.540432+0000","last_unstale":"2024-02-05T08:05:59.718248+0000","last_undegraded":"2024-02-05T08:05:59.718248+0000","last_fullsized":"2024-02-05T08:05:59.718248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:10.219126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528495+0000","last_change":"2024-02-05T08:04:43.608826+0000","last_active":"2024-02-05T08:05:59.528495+0000","last_peered":"2024-02-05T08:05:59.528495+0000","last_clean":"2024-02-05T08:05:59.528495+0000","last_became_active":"2024-02-05T08:04:43.608489+0000","last_became_peered":"2024-02-05T08:04:43.608489+0000","last_unstale":"2024-02-05T08:05:59.528495+0000","last_undegraded":"2024-02-05T08:05:59.528495+0000","last_fullsized":"2024-02-05T08:05:59.528495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.929658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718647+0000","last_change":"2024-02-05T08:05:56.721472+0000","last_active":"2024-02-05T08:05:59.718647+0000","last_peered":"2024-02-05T08:05:59.718647+0000","last_clean":"2024-02-05T08:05:59.718647+0000","last_became_active":"2024-02-05T08:04:48.172747+0000","last_became_peered":"2024-02-05T08:04:48.172747+0000","last_unstale":"2024-02-05T08:05:59.718647+0000","last_undegraded":"2024-02-05T08:05:59.718647+0000","last_fullsized":"2024-02-05T08:05:59.718647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:56.721381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:56.721381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.336615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:04.495948+0000","last_change":"2024-02-05T08:06:04.495991+0000","last_active":"2024-02-05T08:06:04.495948+0000","last_peered":"2024-02-05T08:06:04.495948+0000","last_clean":"2024-02-05T08:06:04.495948+0000","last_became_active":"2024-02-05T08:04:45.560241+0000","last_became_peered":"2024-02-05T08:04:45.560241+0000","last_unstale":"2024-02-05T08:06:04.495948+0000","last_undegraded":"2024-02-05T08:06:04.495948+0000","last_fullsized":"2024-02-05T08:06:04.495948+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:04.495861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:04.495861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:12.837854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720257+0000","last_change":"2024-02-05T08:05:49.229502+0000","last_active":"2024-02-05T08:05:59.720257+0000","last_peered":"2024-02-05T08:05:59.720257+0000","last_clean":"2024-02-05T08:05:59.720257+0000","last_became_active":"2024-02-05T08:04:41.537709+0000","last_became_peered":"2024-02-05T08:04:41.537709+0000","last_unstale":"2024-02-05T08:05:59.720257+0000","last_undegraded":"2024-02-05T08:05:59.720257+0000","last_fullsized":"2024-02-05T08:05:59.720257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:49.229415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:49.229415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:03.840509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":63,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226736+0000","last_change":"2024-02-05T08:04:43.606964+0000","last_active":"2024-02-05T08:06:00.226736+0000","last_peered":"2024-02-05T08:06:00.226736+0000","last_clean":"2024-02-05T08:06:00.226736+0000","last_became_active":"2024-02-05T08:04:43.606624+0000","last_became_peered":"2024-02-05T08:04:43.606624+0000","last_unstale":"2024-02-05T08:06:00.226736+0000","last_undegraded":"2024-02-05T08:06:00.226736+0000","last_fullsized":"2024-02-05T08:06:00.226736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268004+0000","last_change":"2024-02-05T08:05:59.500065+0000","last_active":"2024-02-05T08:06:00.268004+0000","last_peered":"2024-02-05T08:06:00.268004+0000","last_clean":"2024-02-05T08:06:00.268004+0000","last_became_active":"2024-02-05T08:04:48.170100+0000","last_became_peered":"2024-02-05T08:04:48.170100+0000","last_unstale":"2024-02-05T08:06:00.268004+0000","last_undegraded":"2024-02-05T08:06:00.268004+0000","last_fullsized":"2024-02-05T08:06:00.268004+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:59.499960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:59.499960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:16.082380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718992+0000","last_change":"2024-02-05T08:05:57.687803+0000","last_active":"2024-02-05T08:05:59.718992+0000","last_peered":"2024-02-05T08:05:59.718992+0000","last_clean":"2024-02-05T08:05:59.718992+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:05:59.718992+0000","last_undegraded":"2024-02-05T08:05:59.718992+0000","last_fullsized":"2024-02-05T08:05:59.718992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:05:57.687679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.687679+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:23.035896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268523+0000","last_change":"2024-02-05T08:04:43.608563+0000","last_active":"2024-02-05T08:06:00.268523+0000","last_peered":"2024-02-05T08:06:00.268523+0000","last_clean":"2024-02-05T08:06:00.268523+0000","last_became_active":"2024-02-05T08:04:43.608254+0000","last_became_peered":"2024-02-05T08:04:43.608254+0000","last_unstale":"2024-02-05T08:06:00.268523+0000","last_undegraded":"2024-02-05T08:06:00.268523+0000","last_fullsized":"2024-02-05T08:06:00.268523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.187066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.600984+0000","last_change":"2024-02-05T08:05:59.720386+0000","last_active":"2024-02-05T08:06:01.600984+0000","last_peered":"2024-02-05T08:06:01.600984+0000","last_clean":"2024-02-05T08:06:01.600984+0000","last_became_active":"2024-02-05T08:04:45.561044+0000","last_became_peered":"2024-02-05T08:04:45.561044+0000","last_unstale":"2024-02-05T08:06:01.600984+0000","last_undegraded":"2024-02-05T08:06:01.600984+0000","last_fullsized":"2024-02-05T08:06:01.600984+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:05:59.720223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:59.720223+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.562177+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225545+0000","last_change":"2024-02-05T08:04:47.587025+0000","last_active":"2024-02-05T08:06:00.225545+0000","last_peered":"2024-02-05T08:06:00.225545+0000","last_clean":"2024-02-05T08:06:00.225545+0000","last_became_active":"2024-02-05T08:04:47.586044+0000","last_became_peered":"2024-02-05T08:04:47.586044+0000","last_unstale":"2024-02-05T08:06:00.225545+0000","last_undegraded":"2024-02-05T08:06:00.225545+0000","last_fullsized":"2024-02-05T08:06:00.225545+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.700997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.762873+0000","last_change":"2024-02-05T08:06:01.762898+0000","last_active":"2024-02-05T08:06:01.762873+0000","last_peered":"2024-02-05T08:06:01.762873+0000","last_clean":"2024-02-05T08:06:01.762873+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:06:01.762873+0000","last_undegraded":"2024-02-05T08:06:01.762873+0000","last_fullsized":"2024-02-05T08:06:01.762873+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:01.762805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:01.762805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:12.602032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530066+0000","last_change":"2024-02-05T08:04:43.556901+0000","last_active":"2024-02-05T08:05:59.530066+0000","last_peered":"2024-02-05T08:05:59.530066+0000","last_clean":"2024-02-05T08:05:59.530066+0000","last_became_active":"2024-02-05T08:04:43.556580+0000","last_became_peered":"2024-02-05T08:04:43.556580+0000","last_unstale":"2024-02-05T08:05:59.530066+0000","last_undegraded":"2024-02-05T08:05:59.530066+0000","last_fullsized":"2024-02-05T08:05:59.530066+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.567288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:07.368791+0000","last_change":"2024-02-05T08:06:07.368811+0000","last_active":"2024-02-05T08:06:07.368791+0000","last_peered":"2024-02-05T08:06:07.368791+0000","last_clean":"2024-02-05T08:06:07.368791+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:06:07.368791+0000","last_undegraded":"2024-02-05T08:06:07.368791+0000","last_fullsized":"2024-02-05T08:06:07.368791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_clean_scrub_stamp":"2024-02-05T08:06:07.368730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:21.064118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:03.741658+0000","last_change":"2024-02-05T08:06:03.741686+0000","last_active":"2024-02-05T08:06:03.741658+0000","last_peered":"2024-02-05T08:06:03.741658+0000","last_clean":"2024-02-05T08:06:03.741658+0000","last_became_active":"2024-02-05T08:04:47.587082+0000","last_became_peered":"2024-02-05T08:04:47.587082+0000","last_unstale":"2024-02-05T08:06:03.741658+0000","last_undegraded":"2024-02-05T08:06:03.741658+0000","last_fullsized":"2024-02-05T08:06:03.741658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:03.741577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:03.741577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:11.425517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546310+0000","last_change":"2024-02-05T08:05:55.596909+0000","last_active":"2024-02-05T08:06:00.546310+0000","last_peered":"2024-02-05T08:06:00.546310+0000","last_clean":"2024-02-05T08:06:00.546310+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:06:00.546310+0000","last_undegraded":"2024-02-05T08:06:00.546310+0000","last_fullsized":"2024-02-05T08:06:00.546310+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:55.596827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.596827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.318812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718966+0000","last_change":"2024-02-05T08:05:53.741209+0000","last_active":"2024-02-05T08:05:59.718966+0000","last_peered":"2024-02-05T08:05:59.718966+0000","last_clean":"2024-02-05T08:05:59.718966+0000","last_became_active":"2024-02-05T08:04:41.540586+0000","last_became_peered":"2024-02-05T08:04:41.540586+0000","last_unstale":"2024-02-05T08:05:59.718966+0000","last_undegraded":"2024-02-05T08:05:59.718966+0000","last_fullsized":"2024-02-05T08:05:59.718966+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:53.741059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.741059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:13.753256+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":112,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720616+0000","last_change":"2024-02-05T08:05:03.797738+0000","last_active":"2024-02-05T08:05:59.720616+0000","last_peered":"2024-02-05T08:05:59.720616+0000","last_clean":"2024-02-05T08:05:59.720616+0000","last_became_active":"2024-02-05T08:05:03.786876+0000","last_became_peered":"2024-02-05T08:05:03.786876+0000","last_unstale":"2024-02-05T08:05:59.720616+0000","last_undegraded":"2024-02-05T08:05:59.720616+0000","last_fullsized":"2024-02-05T08:05:59.720616+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:09.572379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.526787+0000","last_change":"2024-02-05T08:04:47.582355+0000","last_active":"2024-02-05T08:05:59.526787+0000","last_peered":"2024-02-05T08:05:59.526787+0000","last_clean":"2024-02-05T08:05:59.526787+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:05:59.526787+0000","last_undegraded":"2024-02-05T08:05:59.526787+0000","last_fullsized":"2024-02-05T08:05:59.526787+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.369671+0000","last_change":"2024-02-05T08:06:06.369693+0000","last_active":"2024-02-05T08:06:06.369671+0000","last_peered":"2024-02-05T08:06:06.369671+0000","last_clean":"2024-02-05T08:06:06.369671+0000","last_became_active":"2024-02-05T08:04:45.567341+0000","last_became_peered":"2024-02-05T08:04:45.567341+0000","last_unstale":"2024-02-05T08:06:06.369671+0000","last_undegraded":"2024-02-05T08:06:06.369671+0000","last_fullsized":"2024-02-05T08:06:06.369671+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:06.369612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:06.369612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:31.000287+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268978+0000","last_change":"2024-02-05T08:04:41.540429+0000","last_active":"2024-02-05T08:06:00.268978+0000","last_peered":"2024-02-05T08:06:00.268978+0000","last_clean":"2024-02-05T08:06:00.268978+0000","last_became_active":"2024-02-05T08:04:41.540116+0000","last_became_peered":"2024-02-05T08:04:41.540116+0000","last_unstale":"2024-02-05T08:06:00.268978+0000","last_undegraded":"2024-02-05T08:06:00.268978+0000","last_fullsized":"2024-02-05T08:06:00.268978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.767296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:04.215819+0000","last_change":"2024-02-05T08:06:04.215847+0000","last_active":"2024-02-05T08:06:04.215819+0000","last_peered":"2024-02-05T08:06:04.215819+0000","last_clean":"2024-02-05T08:06:04.215819+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:06:04.215819+0000","last_undegraded":"2024-02-05T08:06:04.215819+0000","last_fullsized":"2024-02-05T08:06:04.215819+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:06:04.215758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:04.215758+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:29.117755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225432+0000","last_change":"2024-02-05T08:04:47.588669+0000","last_active":"2024-02-05T08:06:00.225432+0000","last_peered":"2024-02-05T08:06:00.225432+0000","last_clean":"2024-02-05T08:06:00.225432+0000","last_became_active":"2024-02-05T08:04:47.588395+0000","last_became_peered":"2024-02-05T08:04:47.588395+0000","last_unstale":"2024-02-05T08:06:00.225432+0000","last_undegraded":"2024-02-05T08:06:00.225432+0000","last_fullsized":"2024-02-05T08:06:00.225432+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:16.402122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268703+0000","last_change":"2024-02-05T08:05:52.503758+0000","last_active":"2024-02-05T08:06:00.268703+0000","last_peered":"2024-02-05T08:06:00.268703+0000","last_clean":"2024-02-05T08:06:00.268703+0000","last_became_active":"2024-02-05T08:04:40.527674+0000","last_became_peered":"2024-02-05T08:04:40.527674+0000","last_unstale":"2024-02-05T08:06:00.268703+0000","last_undegraded":"2024-02-05T08:06:00.268703+0000","last_fullsized":"2024-02-05T08:06:00.268703+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.503651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.821538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530022+0000","last_change":"2024-02-05T08:04:41.537767+0000","last_active":"2024-02-05T08:05:59.530022+0000","last_peered":"2024-02-05T08:05:59.530022+0000","last_clean":"2024-02-05T08:05:59.530022+0000","last_became_active":"2024-02-05T08:04:41.537478+0000","last_became_peered":"2024-02-05T08:04:41.537478+0000","last_unstale":"2024-02-05T08:05:59.530022+0000","last_undegraded":"2024-02-05T08:05:59.530022+0000","last_fullsized":"2024-02-05T08:05:59.530022+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"85'38","reported_seq":139,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.600758+0000","last_change":"2024-02-05T08:05:43.858932+0000","last_active":"2024-02-05T08:06:01.600758+0000","last_peered":"2024-02-05T08:06:01.600758+0000","last_clean":"2024-02-05T08:06:01.600758+0000","last_became_active":"2024-02-05T08:04:43.554536+0000","last_became_peered":"2024-02-05T08:04:43.554536+0000","last_unstale":"2024-02-05T08:06:01.600758+0000","last_undegraded":"2024-02-05T08:06:01.600758+0000","last_fullsized":"2024-02-05T08:06:01.600758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:05:43.858841+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:54.322675+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527257+0000","last_change":"2024-02-05T08:04:45.562140+0000","last_active":"2024-02-05T08:05:59.527257+0000","last_peered":"2024-02-05T08:05:59.527257+0000","last_clean":"2024-02-05T08:05:59.527257+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:05:59.527257+0000","last_undegraded":"2024-02-05T08:05:59.527257+0000","last_fullsized":"2024-02-05T08:05:59.527257+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:14.185461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.719032+0000","last_change":"2024-02-05T08:04:48.170226+0000","last_active":"2024-02-05T08:05:59.719032+0000","last_peered":"2024-02-05T08:05:59.719032+0000","last_clean":"2024-02-05T08:05:59.719032+0000","last_became_active":"2024-02-05T08:04:48.169894+0000","last_became_peered":"2024-02-05T08:04:48.169894+0000","last_unstale":"2024-02-05T08:05:59.719032+0000","last_undegraded":"2024-02-05T08:05:59.719032+0000","last_fullsized":"2024-02-05T08:05:59.719032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:15.026925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"67'1","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.526644+0000","last_change":"2024-02-05T08:04:41.536875+0000","last_active":"2024-02-05T08:05:59.526644+0000","last_peered":"2024-02-05T08:05:59.526644+0000","last_clean":"2024-02-05T08:05:59.526644+0000","last_became_active":"2024-02-05T08:04:41.536585+0000","last_became_peered":"2024-02-05T08:04:41.536585+0000","last_unstale":"2024-02-05T08:05:59.526644+0000","last_undegraded":"2024-02-05T08:05:59.526644+0000","last_fullsized":"2024-02-05T08:05:59.526644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.042189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.545805+0000","last_change":"2024-02-05T08:05:41.807274+0000","last_active":"2024-02-05T08:06:00.545805+0000","last_peered":"2024-02-05T08:06:00.545805+0000","last_clean":"2024-02-05T08:06:00.545805+0000","last_became_active":"2024-02-05T08:04:40.529073+0000","last_became_peered":"2024-02-05T08:04:40.529073+0000","last_unstale":"2024-02-05T08:06:00.545805+0000","last_undegraded":"2024-02-05T08:06:00.545805+0000","last_fullsized":"2024-02-05T08:06:00.545805+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:41.807173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:05:41.807173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:48.748767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546072+0000","last_change":"2024-02-05T08:05:45.827248+0000","last_active":"2024-02-05T08:06:00.546072+0000","last_peered":"2024-02-05T08:06:00.546072+0000","last_clean":"2024-02-05T08:06:00.546072+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:06:00.546072+0000","last_undegraded":"2024-02-05T08:06:00.546072+0000","last_fullsized":"2024-02-05T08:06:00.546072+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:05:45.827140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:45.827140+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:14.695181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546608+0000","last_change":"2024-02-05T08:05:58.550125+0000","last_active":"2024-02-05T08:06:00.546608+0000","last_peered":"2024-02-05T08:06:00.546608+0000","last_clean":"2024-02-05T08:06:00.546608+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:06:00.546608+0000","last_undegraded":"2024-02-05T08:06:00.546608+0000","last_fullsized":"2024-02-05T08:06:00.546608+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:58.550049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.550049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.897417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268113+0000","last_change":"2024-02-05T08:04:47.586644+0000","last_active":"2024-02-05T08:06:00.268113+0000","last_peered":"2024-02-05T08:06:00.268113+0000","last_clean":"2024-02-05T08:06:00.268113+0000","last_became_active":"2024-02-05T08:04:47.586358+0000","last_became_peered":"2024-02-05T08:04:47.586358+0000","last_unstale":"2024-02-05T08:06:00.268113+0000","last_undegraded":"2024-02-05T08:06:00.268113+0000","last_fullsized":"2024-02-05T08:06:00.268113+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:13.569831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":316,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718051+0000","last_change":"2024-02-05T08:05:16.657716+0000","last_active":"2024-02-05T08:05:59.718051+0000","last_peered":"2024-02-05T08:05:59.718051+0000","last_clean":"2024-02-05T08:05:59.718051+0000","last_became_active":"2024-02-05T08:04:30.515158+0000","last_became_peered":"2024-02-05T08:04:30.515158+0000","last_unstale":"2024-02-05T08:05:59.718051+0000","last_undegraded":"2024-02-05T08:05:59.718051+0000","last_fullsized":"2024-02-05T08:05:59.718051+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'119","last_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:05:16.657600+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:37.807886+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226170+0000","last_change":"2024-02-05T08:04:41.538282+0000","last_active":"2024-02-05T08:06:00.226170+0000","last_peered":"2024-02-05T08:06:00.226170+0000","last_clean":"2024-02-05T08:06:00.226170+0000","last_became_active":"2024-02-05T08:04:41.538002+0000","last_became_peered":"2024-02-05T08:04:41.538002+0000","last_unstale":"2024-02-05T08:06:00.226170+0000","last_undegraded":"2024-02-05T08:06:00.226170+0000","last_fullsized":"2024-02-05T08:06:00.226170+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.327424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":103,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.601225+0000","last_change":"2024-02-05T08:06:00.304976+0000","last_active":"2024-02-05T08:06:01.601225+0000","last_peered":"2024-02-05T08:06:01.601225+0000","last_clean":"2024-02-05T08:06:01.601225+0000","last_became_active":"2024-02-05T08:04:43.606401+0000","last_became_peered":"2024-02-05T08:04:43.606401+0000","last_unstale":"2024-02-05T08:06:01.601225+0000","last_undegraded":"2024-02-05T08:06:01.601225+0000","last_fullsized":"2024-02-05T08:06:01.601225+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-02-05T08:06:00.304861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:00.304861+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:15.661923+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718680+0000","last_change":"2024-02-05T08:04:45.565211+0000","last_active":"2024-02-05T08:05:59.718680+0000","last_peered":"2024-02-05T08:05:59.718680+0000","last_clean":"2024-02-05T08:05:59.718680+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:05:59.718680+0000","last_undegraded":"2024-02-05T08:05:59.718680+0000","last_fullsized":"2024-02-05T08:05:59.718680+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:10.360421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527679+0000","last_change":"2024-02-05T08:04:47.587068+0000","last_active":"2024-02-05T08:05:59.527679+0000","last_peered":"2024-02-05T08:05:59.527679+0000","last_clean":"2024-02-05T08:05:59.527679+0000","last_became_active":"2024-02-05T08:04:47.586774+0000","last_became_peered":"2024-02-05T08:04:47.586774+0000","last_unstale":"2024-02-05T08:05:59.527679+0000","last_undegraded":"2024-02-05T08:05:59.527679+0000","last_fullsized":"2024-02-05T08:05:59.527679+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:00.755929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527011+0000","last_change":"2024-02-05T08:04:41.538403+0000","last_active":"2024-02-05T08:05:59.527011+0000","last_peered":"2024-02-05T08:05:59.527011+0000","last_clean":"2024-02-05T08:05:59.527011+0000","last_became_active":"2024-02-05T08:04:41.537872+0000","last_became_peered":"2024-02-05T08:04:41.537872+0000","last_unstale":"2024-02-05T08:05:59.527011+0000","last_undegraded":"2024-02-05T08:05:59.527011+0000","last_fullsized":"2024-02-05T08:05:59.527011+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.800257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":159,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:05.912571+0000","last_change":"2024-02-05T08:05:43.746185+0000","last_active":"2024-02-05T08:06:05.912571+0000","last_peered":"2024-02-05T08:06:05.912571+0000","last_clean":"2024-02-05T08:06:05.912571+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:06:05.912571+0000","last_undegraded":"2024-02-05T08:06:05.912571+0000","last_fullsized":"2024-02-05T08:06:05.912571+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:05:43.746095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:05:43.746095+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:55.186390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528230+0000","last_change":"2024-02-05T08:04:43.550001+0000","last_active":"2024-02-05T08:05:59.528230+0000","last_peered":"2024-02-05T08:05:59.528230+0000","last_clean":"2024-02-05T08:05:59.528230+0000","last_became_active":"2024-02-05T08:04:43.549687+0000","last_became_peered":"2024-02-05T08:04:43.549687+0000","last_unstale":"2024-02-05T08:05:59.528230+0000","last_undegraded":"2024-02-05T08:05:59.528230+0000","last_fullsized":"2024-02-05T08:05:59.528230+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.794328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269362+0000","last_change":"2024-02-05T08:04:45.559137+0000","last_active":"2024-02-05T08:06:00.269362+0000","last_peered":"2024-02-05T08:06:00.269362+0000","last_clean":"2024-02-05T08:06:00.269362+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:06:00.269362+0000","last_undegraded":"2024-02-05T08:06:00.269362+0000","last_fullsized":"2024-02-05T08:06:00.269362+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:12.171151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527562+0000","last_change":"2024-02-05T08:04:48.170100+0000","last_active":"2024-02-05T08:05:59.527562+0000","last_peered":"2024-02-05T08:05:59.527562+0000","last_clean":"2024-02-05T08:05:59.527562+0000","last_became_active":"2024-02-05T08:04:48.169777+0000","last_became_peered":"2024-02-05T08:04:48.169777+0000","last_unstale":"2024-02-05T08:05:59.527562+0000","last_undegraded":"2024-02-05T08:05:59.527562+0000","last_fullsized":"2024-02-05T08:05:59.527562+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527843+0000","last_change":"2024-02-05T08:05:51.880952+0000","last_active":"2024-02-05T08:05:59.527843+0000","last_peered":"2024-02-05T08:05:59.527843+0000","last_clean":"2024-02-05T08:05:59.527843+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:05:59.527843+0000","last_undegraded":"2024-02-05T08:05:59.527843+0000","last_fullsized":"2024-02-05T08:05:59.527843+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:51.880864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:51.880864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:53.033627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":98,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528739+0000","last_change":"2024-02-05T08:05:55.329780+0000","last_active":"2024-02-05T08:05:59.528739+0000","last_peered":"2024-02-05T08:05:59.528739+0000","last_clean":"2024-02-05T08:05:59.528739+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:05:59.528739+0000","last_undegraded":"2024-02-05T08:05:59.528739+0000","last_fullsized":"2024-02-05T08:05:59.528739+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:05:55.329677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.329677+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:07.966593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720746+0000","last_change":"2024-02-05T08:04:45.565899+0000","last_active":"2024-02-05T08:05:59.720746+0000","last_peered":"2024-02-05T08:05:59.720746+0000","last_clean":"2024-02-05T08:05:59.720746+0000","last_became_active":"2024-02-05T08:04:45.565618+0000","last_became_peered":"2024-02-05T08:04:45.565618+0000","last_unstale":"2024-02-05T08:05:59.720746+0000","last_undegraded":"2024-02-05T08:05:59.720746+0000","last_fullsized":"2024-02-05T08:05:59.720746+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.936746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528338+0000","last_change":"2024-02-05T08:05:58.215810+0000","last_active":"2024-02-05T08:05:59.528338+0000","last_peered":"2024-02-05T08:05:59.528338+0000","last_clean":"2024-02-05T08:05:59.528338+0000","last_became_active":"2024-02-05T08:04:47.583866+0000","last_became_peered":"2024-02-05T08:04:47.583866+0000","last_unstale":"2024-02-05T08:05:59.528338+0000","last_undegraded":"2024-02-05T08:05:59.528338+0000","last_fullsized":"2024-02-05T08:05:59.528338+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:58.215737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.215737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.871723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269169+0000","last_change":"2024-02-05T08:04:41.538084+0000","last_active":"2024-02-05T08:06:00.269169+0000","last_peered":"2024-02-05T08:06:00.269169+0000","last_clean":"2024-02-05T08:06:00.269169+0000","last_became_active":"2024-02-05T08:04:41.537818+0000","last_became_peered":"2024-02-05T08:04:41.537818+0000","last_unstale":"2024-02-05T08:06:00.269169+0000","last_undegraded":"2024-02-05T08:06:00.269169+0000","last_fullsized":"2024-02-05T08:06:00.269169+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.324263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527293+0000","last_change":"2024-02-05T08:04:43.552388+0000","last_active":"2024-02-05T08:05:59.527293+0000","last_peered":"2024-02-05T08:05:59.527293+0000","last_clean":"2024-02-05T08:05:59.527293+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:05:59.527293+0000","last_undegraded":"2024-02-05T08:05:59.527293+0000","last_fullsized":"2024-02-05T08:05:59.527293+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:05.174347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.600735+0000","last_change":"2024-02-05T08:05:57.266029+0000","last_active":"2024-02-05T08:06:01.600735+0000","last_peered":"2024-02-05T08:06:01.600735+0000","last_clean":"2024-02-05T08:06:01.600735+0000","last_became_active":"2024-02-05T08:04:45.562462+0000","last_became_peered":"2024-02-05T08:04:45.562462+0000","last_unstale":"2024-02-05T08:06:01.600735+0000","last_undegraded":"2024-02-05T08:06:01.600735+0000","last_fullsized":"2024-02-05T08:06:01.600735+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:05:57.265939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.265939+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:03.556401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718573+0000","last_change":"2024-02-05T08:04:48.172841+0000","last_active":"2024-02-05T08:05:59.718573+0000","last_peered":"2024-02-05T08:05:59.718573+0000","last_clean":"2024-02-05T08:05:59.718573+0000","last_became_active":"2024-02-05T08:04:48.172564+0000","last_became_peered":"2024-02-05T08:04:48.172564+0000","last_unstale":"2024-02-05T08:05:59.718573+0000","last_undegraded":"2024-02-05T08:05:59.718573+0000","last_fullsized":"2024-02-05T08:05:59.718573+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:15.681030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528317+0000","last_change":"2024-02-05T08:05:52.283473+0000","last_active":"2024-02-05T08:05:59.528317+0000","last_peered":"2024-02-05T08:05:59.528317+0000","last_clean":"2024-02-05T08:05:59.528317+0000","last_became_active":"2024-02-05T08:04:41.534433+0000","last_became_peered":"2024-02-05T08:04:41.534433+0000","last_unstale":"2024-02-05T08:05:59.528317+0000","last_undegraded":"2024-02-05T08:05:59.528317+0000","last_fullsized":"2024-02-05T08:05:59.528317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:05:52.283383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.283383+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:05.067021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529862+0000","last_change":"2024-02-05T08:04:43.551586+0000","last_active":"2024-02-05T08:05:59.529862+0000","last_peered":"2024-02-05T08:05:59.529862+0000","last_clean":"2024-02-05T08:05:59.529862+0000","last_became_active":"2024-02-05T08:04:43.551184+0000","last_became_peered":"2024-02-05T08:04:43.551184+0000","last_unstale":"2024-02-05T08:05:59.529862+0000","last_undegraded":"2024-02-05T08:05:59.529862+0000","last_fullsized":"2024-02-05T08:05:59.529862+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:05.037825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.600735+0000","last_change":"2024-02-05T08:04:45.563011+0000","last_active":"2024-02-05T08:06:01.600735+0000","last_peered":"2024-02-05T08:06:01.600735+0000","last_clean":"2024-02-05T08:06:01.600735+0000","last_became_active":"2024-02-05T08:04:45.562597+0000","last_became_peered":"2024-02-05T08:04:45.562597+0000","last_unstale":"2024-02-05T08:06:01.600735+0000","last_undegraded":"2024-02-05T08:06:01.600735+0000","last_fullsized":"2024-02-05T08:06:01.600735+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.186855+0000","last_change":"2024-02-05T08:06:01.186878+0000","last_active":"2024-02-05T08:06:01.186855+0000","last_peered":"2024-02-05T08:06:01.186855+0000","last_clean":"2024-02-05T08:06:01.186855+0000","last_became_active":"2024-02-05T08:04:47.582776+0000","last_became_peered":"2024-02-05T08:04:47.582776+0000","last_unstale":"2024-02-05T08:06:01.186855+0000","last_undegraded":"2024-02-05T08:06:01.186855+0000","last_fullsized":"2024-02-05T08:06:01.186855+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:01.186782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:01.186782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.355302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226658+0000","last_change":"2024-02-05T08:05:48.267727+0000","last_active":"2024-02-05T08:06:00.226658+0000","last_peered":"2024-02-05T08:06:00.226658+0000","last_clean":"2024-02-05T08:06:00.226658+0000","last_became_active":"2024-02-05T08:04:41.540818+0000","last_became_peered":"2024-02-05T08:04:41.540818+0000","last_unstale":"2024-02-05T08:06:00.226658+0000","last_undegraded":"2024-02-05T08:06:00.226658+0000","last_fullsized":"2024-02-05T08:06:00.226658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:05:48.267582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.267582+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:15.800342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225869+0000","last_change":"2024-02-05T08:05:52.195604+0000","last_active":"2024-02-05T08:06:00.225869+0000","last_peered":"2024-02-05T08:06:00.225869+0000","last_clean":"2024-02-05T08:06:00.225869+0000","last_became_active":"2024-02-05T08:04:43.607546+0000","last_became_peered":"2024-02-05T08:04:43.607546+0000","last_unstale":"2024-02-05T08:06:00.225869+0000","last_undegraded":"2024-02-05T08:06:00.225869+0000","last_fullsized":"2024-02-05T08:06:00.225869+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-05T08:05:52.195494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.195494+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:16.333300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529710+0000","last_change":"2024-02-05T08:04:45.561907+0000","last_active":"2024-02-05T08:05:59.529710+0000","last_peered":"2024-02-05T08:05:59.529710+0000","last_clean":"2024-02-05T08:05:59.529710+0000","last_became_active":"2024-02-05T08:04:45.561639+0000","last_became_peered":"2024-02-05T08:04:45.561639+0000","last_unstale":"2024-02-05T08:05:59.529710+0000","last_undegraded":"2024-02-05T08:05:59.529710+0000","last_fullsized":"2024-02-05T08:05:59.529710+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.611331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268056+0000","last_change":"2024-02-05T08:05:48.449994+0000","last_active":"2024-02-05T08:06:00.268056+0000","last_peered":"2024-02-05T08:06:00.268056+0000","last_clean":"2024-02-05T08:06:00.268056+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:06:00.268056+0000","last_undegraded":"2024-02-05T08:06:00.268056+0000","last_fullsized":"2024-02-05T08:06:00.268056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:48.449900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.449900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:55.343526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.600906+0000","last_change":"2024-02-05T08:04:45.567294+0000","last_active":"2024-02-05T08:06:01.600906+0000","last_peered":"2024-02-05T08:06:01.600906+0000","last_clean":"2024-02-05T08:06:01.600906+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:06:01.600906+0000","last_undegraded":"2024-02-05T08:06:01.600906+0000","last_fullsized":"2024-02-05T08:06:01.600906+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225833+0000","last_change":"2024-02-05T08:05:59.261588+0000","last_active":"2024-02-05T08:06:00.225833+0000","last_peered":"2024-02-05T08:06:00.225833+0000","last_clean":"2024-02-05T08:06:00.225833+0000","last_became_active":"2024-02-05T08:04:41.541327+0000","last_became_peered":"2024-02-05T08:04:41.541327+0000","last_unstale":"2024-02-05T08:06:00.225833+0000","last_undegraded":"2024-02-05T08:06:00.225833+0000","last_fullsized":"2024-02-05T08:06:00.225833+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:59.261506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:59.261506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:24.020924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.526944+0000","last_change":"2024-02-05T08:05:53.816142+0000","last_active":"2024-02-05T08:05:59.526944+0000","last_peered":"2024-02-05T08:05:59.526944+0000","last_clean":"2024-02-05T08:05:59.526944+0000","last_became_active":"2024-02-05T08:04:43.555028+0000","last_became_peered":"2024-02-05T08:04:43.555028+0000","last_unstale":"2024-02-05T08:05:59.526944+0000","last_undegraded":"2024-02-05T08:05:59.526944+0000","last_fullsized":"2024-02-05T08:05:59.526944+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:05:53.815997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.815997+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.642275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225515+0000","last_change":"2024-02-05T08:04:48.170254+0000","last_active":"2024-02-05T08:06:00.225515+0000","last_peered":"2024-02-05T08:06:00.225515+0000","last_clean":"2024-02-05T08:06:00.225515+0000","last_became_active":"2024-02-05T08:04:48.169762+0000","last_became_peered":"2024-02-05T08:04:48.169762+0000","last_unstale":"2024-02-05T08:06:00.225515+0000","last_undegraded":"2024-02-05T08:06:00.225515+0000","last_fullsized":"2024-02-05T08:06:00.225515+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:13.151401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.561022+0000","last_change":"2024-02-05T08:06:01.561064+0000","last_active":"2024-02-05T08:06:01.561022+0000","last_peered":"2024-02-05T08:06:01.561022+0000","last_clean":"2024-02-05T08:06:01.561022+0000","last_became_active":"2024-02-05T08:04:45.561679+0000","last_became_peered":"2024-02-05T08:04:45.561679+0000","last_unstale":"2024-02-05T08:06:01.561022+0000","last_undegraded":"2024-02-05T08:06:01.561022+0000","last_fullsized":"2024-02-05T08:06:01.561022+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:01.560919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:01.560919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:21.298440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529733+0000","last_change":"2024-02-05T08:05:48.116107+0000","last_active":"2024-02-05T08:05:59.529733+0000","last_peered":"2024-02-05T08:05:59.529733+0000","last_clean":"2024-02-05T08:05:59.529733+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:05:59.529733+0000","last_undegraded":"2024-02-05T08:05:59.529733+0000","last_fullsized":"2024-02-05T08:05:59.529733+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:48.116008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.116008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:06.262952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530647+0000","last_change":"2024-02-05T08:04:43.554584+0000","last_active":"2024-02-05T08:05:59.530647+0000","last_peered":"2024-02-05T08:05:59.530647+0000","last_clean":"2024-02-05T08:05:59.530647+0000","last_became_active":"2024-02-05T08:04:43.554161+0000","last_became_peered":"2024-02-05T08:04:43.554161+0000","last_unstale":"2024-02-05T08:05:59.530647+0000","last_undegraded":"2024-02-05T08:05:59.530647+0000","last_fullsized":"2024-02-05T08:05:59.530647+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.545953+0000","last_change":"2024-02-05T08:05:48.720004+0000","last_active":"2024-02-05T08:06:00.545953+0000","last_peered":"2024-02-05T08:06:00.545953+0000","last_clean":"2024-02-05T08:06:00.545953+0000","last_became_active":"2024-02-05T08:04:47.585962+0000","last_became_peered":"2024-02-05T08:04:47.585962+0000","last_unstale":"2024-02-05T08:06:00.545953+0000","last_undegraded":"2024-02-05T08:06:00.545953+0000","last_fullsized":"2024-02-05T08:06:00.545953+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:05:48.719923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.719923+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:54.016895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528031+0000","last_change":"2024-02-05T08:04:45.559587+0000","last_active":"2024-02-05T08:05:59.528031+0000","last_peered":"2024-02-05T08:05:59.528031+0000","last_clean":"2024-02-05T08:05:59.528031+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:05:59.528031+0000","last_undegraded":"2024-02-05T08:05:59.528031+0000","last_fullsized":"2024-02-05T08:05:59.528031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:02.577883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.721051+0000","last_change":"2024-02-05T08:05:50.230381+0000","last_active":"2024-02-05T08:05:59.721051+0000","last_peered":"2024-02-05T08:05:59.721051+0000","last_clean":"2024-02-05T08:05:59.721051+0000","last_became_active":"2024-02-05T08:04:41.538397+0000","last_became_peered":"2024-02-05T08:04:41.538397+0000","last_unstale":"2024-02-05T08:05:59.721051+0000","last_undegraded":"2024-02-05T08:05:59.721051+0000","last_fullsized":"2024-02-05T08:05:59.721051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:50.230263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:50.230263+0000","last_clean_scrub_stamp":"2024-02-05T08:05:50.230263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:59.311643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530628+0000","last_change":"2024-02-05T08:04:43.555191+0000","last_active":"2024-02-05T08:05:59.530628+0000","last_peered":"2024-02-05T08:05:59.530628+0000","last_clean":"2024-02-05T08:05:59.530628+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:05:59.530628+0000","last_undegraded":"2024-02-05T08:05:59.530628+0000","last_fullsized":"2024-02-05T08:05:59.530628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:12.495416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530787+0000","last_change":"2024-02-05T08:04:47.573584+0000","last_active":"2024-02-05T08:05:59.530787+0000","last_peered":"2024-02-05T08:05:59.530787+0000","last_clean":"2024-02-05T08:05:59.530787+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:05:59.530787+0000","last_undegraded":"2024-02-05T08:05:59.530787+0000","last_fullsized":"2024-02-05T08:05:59.530787+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.290723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:04.443720+0000","last_change":"2024-02-05T08:06:04.443720+0000","last_active":"2024-02-05T08:06:04.443720+0000","last_peered":"2024-02-05T08:06:04.443720+0000","last_clean":"2024-02-05T08:06:04.443720+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:06:04.443720+0000","last_undegraded":"2024-02-05T08:06:04.443720+0000","last_fullsized":"2024-02-05T08:06:04.443720+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:06:04.443624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:04.443624+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:19.854071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226287+0000","last_change":"2024-02-05T08:04:41.534743+0000","last_active":"2024-02-05T08:06:00.226287+0000","last_peered":"2024-02-05T08:06:00.226287+0000","last_clean":"2024-02-05T08:06:00.226287+0000","last_became_active":"2024-02-05T08:04:41.533838+0000","last_became_peered":"2024-02-05T08:04:41.533838+0000","last_unstale":"2024-02-05T08:06:00.226287+0000","last_undegraded":"2024-02-05T08:06:00.226287+0000","last_fullsized":"2024-02-05T08:06:00.226287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.461601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.080113+0000","last_change":"2024-02-05T08:06:06.080146+0000","last_active":"2024-02-05T08:06:06.080113+0000","last_peered":"2024-02-05T08:06:06.080113+0000","last_clean":"2024-02-05T08:06:06.080113+0000","last_became_active":"2024-02-05T08:04:43.559922+0000","last_became_peered":"2024-02-05T08:04:43.559922+0000","last_unstale":"2024-02-05T08:06:06.080113+0000","last_undegraded":"2024-02-05T08:06:06.080113+0000","last_fullsized":"2024-02-05T08:06:06.080113+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:06:06.080040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:06.080040+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:25.229905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":412,"ondisk_log_size":412,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037782,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32656,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712496,"statfs":{"total":95995035648,"available":95961595904,"internally_reserved":0,"allocated":1515520,"data_stored":1027404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.49099999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.435},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.57999999999999996}]},{"osd":2,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.59799999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.68000000000000005}]},{"osd":5,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.63600000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.72599999999999998}]}]},{"osd":6,"up_from":51,"seq":219043332124,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1495040,"data_stored":1025877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.74299999999999999,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.74199999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60399999999999998,"15min":0.60099999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.97099999999999997,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40500000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80600000000000005,"5min":0.871,"15min":0.871},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.78900000000000003,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51400000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.436},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.48199999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.44400000000000001,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.44400000000000001},"min":{"1min":0.313,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.70499999999999996,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.39200000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.373,"5min":0.309,"15min":0.309},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.47099999999999997,"15min":0.46300000000000002},"min":{"1min":0.35199999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73699999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.503}]},{"osd":7,"last update":"Mon Feb 5 08:05:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.41999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":436744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34399999999999997,"15min":0.31},"min":{"1min":0.216,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.37,"15min":0.32500000000000001},"min":{"1min":0.22900000000000001,"5min":0.16,"15min":0.16},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.26100000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.19500000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.63700000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.315},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.41899999999999998,"15min":0.40500000000000003},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.29899999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:05:54 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.434,"15min":0.441},"min":{"1min":0.27200000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.34599999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.41399999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86199999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.443},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.45000000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.74399999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.46999999999999997}]},{"osd":5,"last update":"Mon Feb 5 08:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.42999999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:06:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82299999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.45200000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.41099999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.53700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1519616,"data_stored":1027147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.36199999999999999,"15min":0.33400000000000002},"min":{"1min":0.25,"5min":0.152,"15min":0.152},"max":{"1min":0.501,"5min":0.621,"15min":0.621},"last":0.255},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.37,"15min":0.33600000000000002},"min":{"1min":0.26500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.35199999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:05:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38500000000000001,"15min":0.372},"min":{"1min":0.252,"5min":0.218,"15min":0.218},"max":{"1min":0.60899999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40300000000000002,"15min":0.38900000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.373}]},{"osd":3,"last update":"Mon Feb 5 08:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.45200000000000001,"15min":0.45900000000000002},"min":{"1min":0.29599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71899999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.46300000000000002,"15min":0.46800000000000003},"min":{"1min":0.32800000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.58199999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.32300000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:05:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.79300000000000004,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.52800000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.438}]},{"osd":5,"last update":"Mon Feb 5 08:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60499999999999998,"15min":0.60599999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.46400000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.621,"15min":0.621},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.56000000000000005}]},{"osd":7,"last update":"Mon Feb 5 08:05:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.54100000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":436186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:24 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38,"15min":0.37},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.57099999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.38200000000000001},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56899999999999995,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.46999999999999997}]},{"osd":1,"last update":"Mon Feb 5 08:05:24 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.36799999999999999,"15min":0.35899999999999999},"min":{"1min":0.248,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.495,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.40200000000000002,"15min":0.38},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.623,"5min":0.63,"15min":0.63},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Feb 5 08:05:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46999999999999997,"15min":0.46700000000000003},"min":{"1min":0.34499999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.58699999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56599999999999995,"5min":0.746,"15min":0.746},"last":0.501}]},{"osd":4,"last update":"Mon Feb 5 08:05:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.371,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83299999999999996,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.58599999999999997,"15min":0.57999999999999996},"min":{"1min":0.45500000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85699999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.57499999999999996}]},{"osd":5,"last update":"Mon Feb 5 08:05:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.58699999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63400000000000001,"15min":0.63600000000000001},"min":{"1min":0.435,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80600000000000005,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63500000000000001,"15min":0.63900000000000001},"min":{"1min":0.42299999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83899999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.66600000000000004}]},{"osd":7,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.621},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.63900000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117035,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":436889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.438,"15min":0.45100000000000001},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.44500000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.46800000000000003,"15min":0.48199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70299999999999996,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.45000000000000001,"15min":0.46200000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.46100000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49099999999999999,"15min":0.497},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.502},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47499999999999998,"15min":0.48399999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.39900000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.68899999999999995}]},{"osd":5,"last update":"Mon Feb 5 08:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.46400000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.64600000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.47599999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58499999999999996,"15min":0.58099999999999996},"min":{"1min":0.48899999999999999,"5min":0.438,"15min":0.438},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.55600000000000005}]},{"osd":7,"last update":"Mon Feb 5 08:05:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.70899999999999996}]}]},{"osd":4,"up_from":32,"seq":137438953510,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":435729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:09 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54600000000000004,"15min":0.54800000000000004},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.72099999999999997,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.442}]},{"osd":1,"last update":"Mon Feb 5 08:05:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.40600000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.375}]},{"osd":2,"last update":"Mon Feb 5 08:05:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.504},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77700000000000002,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.35699999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:05:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.54700000000000004}]},{"osd":5,"last update":"Mon Feb 5 08:05:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.38800000000000001,"15min":0.372},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.41299999999999998,"15min":0.39900000000000002},"min":{"1min":0.29099999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.40200000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.311,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69899999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.443,"15min":0.42499999999999999},"min":{"1min":0.35199999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.59899999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.64500000000000002}]}]},{"osd":5,"up_from":42,"seq":180388626464,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":901120,"data_stored":435436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.33100000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.72899999999999998,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.53200000000000003,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71299999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.374,"15min":0.374},"max":{"1min":0.69999999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.498},{"interface":"front","average":{"1min":0.504,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.35299999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56599999999999995,"15min":0.57099999999999995},"min":{"1min":0.34899999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76200000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.442},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.373,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.56899999999999995}]},{"osd":3,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55600000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.749,"15min":0.749},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.623}]},{"osd":4,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.41999999999999998,"15min":0.41799999999999998},"min":{"1min":0.28199999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60599999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.68100000000000005}]},{"osd":6,"last update":"Mon Feb 5 08:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45400000000000001,"15min":0.44600000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.42299999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.63700000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:05:30 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.66400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:06:09.872 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-05T08:06:09.872 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-05T08:06:09.872 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-05T08:06:09.872 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph health --format=json 2024-02-05T08:06:10.111 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:10.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:10 smithi134 ceph-mon[39278]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:06:10.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:10 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:06:10.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:10 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:06:10.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:10 smithi134 ceph-mon[39278]: 6.1d scrub starts 2024-02-05T08:06:10.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:10 smithi134 ceph-mon[39278]: 6.1d scrub ok 2024-02-05T08:06:10.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:10 smithi134 ceph-mon[30086]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:06:10.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:10 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:06:10.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:10 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:06:10.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:10 smithi134 ceph-mon[30086]: 6.1d scrub starts 2024-02-05T08:06:10.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:10 smithi134 ceph-mon[30086]: 6.1d scrub ok 2024-02-05T08:06:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:10 smithi196 ceph-mon[33018]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:06:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:10 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:06:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:10 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:06:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:10 smithi196 ceph-mon[33018]: 6.1d scrub starts 2024-02-05T08:06:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:10 smithi196 ceph-mon[33018]: 6.1d scrub ok 2024-02-05T08:06:11.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[30086]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:06:11.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[30086]: 4.e scrub starts 2024-02-05T08:06:11.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[30086]: 4.e scrub ok 2024-02-05T08:06:11.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[30086]: 5.14 scrub starts 2024-02-05T08:06:11.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[30086]: 5.14 scrub ok 2024-02-05T08:06:11.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[30086]: 3.17 scrub starts 2024-02-05T08:06:11.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[30086]: 3.17 scrub ok 2024-02-05T08:06:11.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[39278]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:06:11.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[39278]: 4.e scrub starts 2024-02-05T08:06:11.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[39278]: 4.e scrub ok 2024-02-05T08:06:11.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[39278]: 5.14 scrub starts 2024-02-05T08:06:11.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[39278]: 5.14 scrub ok 2024-02-05T08:06:11.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[39278]: 3.17 scrub starts 2024-02-05T08:06:11.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:11 smithi134 ceph-mon[39278]: 3.17 scrub ok 2024-02-05T08:06:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:11 smithi196 ceph-mon[33018]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:06:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:11 smithi196 ceph-mon[33018]: 4.e scrub starts 2024-02-05T08:06:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:11 smithi196 ceph-mon[33018]: 4.e scrub ok 2024-02-05T08:06:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:11 smithi196 ceph-mon[33018]: 5.14 scrub starts 2024-02-05T08:06:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:11 smithi196 ceph-mon[33018]: 5.14 scrub ok 2024-02-05T08:06:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:11 smithi196 ceph-mon[33018]: 3.17 scrub starts 2024-02-05T08:06:11.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:11 smithi196 ceph-mon[33018]: 3.17 scrub ok 2024-02-05T08:06:11.999 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:06:11.999 INFO:teuthology.orchestra.run.smithi134.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-05T08:06:12.570 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[30086]: 3.3 deep-scrub starts 2024-02-05T08:06:12.570 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[30086]: 3.3 deep-scrub ok 2024-02-05T08:06:12.570 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[30086]: 4.17 scrub starts 2024-02-05T08:06:12.570 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[30086]: 4.17 scrub ok 2024-02-05T08:06:12.570 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[30086]: 3.19 scrub starts 2024-02-05T08:06:12.570 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[30086]: 3.19 scrub ok 2024-02-05T08:06:12.570 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1869558842' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-05T08:06:12.570 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[39278]: 3.3 deep-scrub starts 2024-02-05T08:06:12.571 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[39278]: 3.3 deep-scrub ok 2024-02-05T08:06:12.571 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[39278]: 4.17 scrub starts 2024-02-05T08:06:12.571 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[39278]: 4.17 scrub ok 2024-02-05T08:06:12.571 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[39278]: 3.19 scrub starts 2024-02-05T08:06:12.571 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[39278]: 3.19 scrub ok 2024-02-05T08:06:12.571 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:12 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1869558842' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-05T08:06:12.596 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-05T08:06:12.596 INFO:tasks.cephadm:Setup complete, yielding 2024-02-05T08:06:12.597 INFO:teuthology.run_tasks:Running task thrashosds... 2024-02-05T08:06:12.608 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-05T08:06:12.609 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-testing-3-2024-02-04-2254'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-guits-testing-3-2024-02-04-2254', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'}} 2024-02-05T08:06:12.609 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-05T08:06:12.609 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-02-05T08:06:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:12 smithi196 ceph-mon[33018]: 3.3 deep-scrub starts 2024-02-05T08:06:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:12 smithi196 ceph-mon[33018]: 3.3 deep-scrub ok 2024-02-05T08:06:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:12 smithi196 ceph-mon[33018]: 4.17 scrub starts 2024-02-05T08:06:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:12 smithi196 ceph-mon[33018]: 4.17 scrub ok 2024-02-05T08:06:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:12 smithi196 ceph-mon[33018]: 3.19 scrub starts 2024-02-05T08:06:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:12 smithi196 ceph-mon[33018]: 3.19 scrub ok 2024-02-05T08:06:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:12 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1869558842' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-05T08:06:12.849 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:13.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:06:12 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:06:13.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:13 smithi196 ceph-mon[33018]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-02-05T08:06:13.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:13 smithi196 ceph-mon[33018]: 6.9 scrub starts 2024-02-05T08:06:13.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:13 smithi196 ceph-mon[33018]: 6.9 scrub ok 2024-02-05T08:06:13.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:13 smithi196 ceph-mon[33018]: 3.c scrub starts 2024-02-05T08:06:13.611 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:13 smithi196 ceph-mon[33018]: 3.c scrub ok 2024-02-05T08:06:13.659 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:13 smithi134 ceph-mon[30086]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-02-05T08:06:13.659 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:13 smithi134 ceph-mon[30086]: 6.9 scrub starts 2024-02-05T08:06:13.659 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:13 smithi134 ceph-mon[30086]: 6.9 scrub ok 2024-02-05T08:06:13.659 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:13 smithi134 ceph-mon[30086]: 3.c scrub starts 2024-02-05T08:06:13.659 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:13 smithi134 ceph-mon[30086]: 3.c scrub ok 2024-02-05T08:06:13.659 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:13 smithi134 ceph-mon[39278]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 855 B/s rd, 0 op/s 2024-02-05T08:06:13.659 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:13 smithi134 ceph-mon[39278]: 6.9 scrub starts 2024-02-05T08:06:13.660 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:13 smithi134 ceph-mon[39278]: 6.9 scrub ok 2024-02-05T08:06:13.660 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:13 smithi134 ceph-mon[39278]: 3.c scrub starts 2024-02-05T08:06:13.660 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:13 smithi134 ceph-mon[39278]: 3.c scrub ok 2024-02-05T08:06:14.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:14 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:14.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:14 smithi134 ceph-mon[30086]: 4.14 scrub starts 2024-02-05T08:06:14.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:14 smithi134 ceph-mon[30086]: 4.14 scrub ok 2024-02-05T08:06:14.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:14 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:14.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:14 smithi134 ceph-mon[39278]: 4.14 scrub starts 2024-02-05T08:06:14.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:14 smithi134 ceph-mon[39278]: 4.14 scrub ok 2024-02-05T08:06:14.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:14 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:14.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:14 smithi196 ceph-mon[33018]: 4.14 scrub starts 2024-02-05T08:06:14.661 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:14 smithi196 ceph-mon[33018]: 4.14 scrub ok 2024-02-05T08:06:15.016 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:14 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:06:14] ENGINE Bus STOPPING 2024-02-05T08:06:15.016 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:14 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:06:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T08:06:15.016 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:14 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:06:14] ENGINE Bus STOPPED 2024-02-05T08:06:15.016 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:14 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:06:14] ENGINE Bus STARTING 2024-02-05T08:06:15.017 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:14 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:06:14] ENGINE Serving on http://:::9283 2024-02-05T08:06:15.017 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:14 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:06:14] ENGINE Bus STARTED 2024-02-05T08:06:15.404 INFO:tasks.thrashosds:Beginning thrashosds... 2024-02-05T08:06:15.405 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-05T08:06:15.405 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:06:15.647 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:15.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:15 smithi134 ceph-mon[30086]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:15.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:15 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:06:15.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:15 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:06:15.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:15 smithi134 ceph-mon[30086]: 3.5 scrub starts 2024-02-05T08:06:15.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:15 smithi134 ceph-mon[30086]: 3.5 scrub ok 2024-02-05T08:06:15.768 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:15 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1154698321' entity='client.admin' 2024-02-05T08:06:15.769 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:15 smithi134 ceph-mon[39278]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:15.769 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:15 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:06:15.769 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:15 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:06:15.769 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:15 smithi134 ceph-mon[39278]: 3.5 scrub starts 2024-02-05T08:06:15.769 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:15 smithi134 ceph-mon[39278]: 3.5 scrub ok 2024-02-05T08:06:15.769 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:15 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1154698321' entity='client.admin' 2024-02-05T08:06:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:15 smithi196 ceph-mon[33018]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:15 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:06:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:15 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:06:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:15 smithi196 ceph-mon[33018]: 3.5 scrub starts 2024-02-05T08:06:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:15 smithi196 ceph-mon[33018]: 3.5 scrub ok 2024-02-05T08:06:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:15 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1154698321' entity='client.admin' 2024-02-05T08:06:16.277 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:06:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[77504]: ts=2024-02-05T08:06:16.243Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002786452s 2024-02-05T08:06:16.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:16 smithi134 ceph-mon[30086]: 5.15 scrub starts 2024-02-05T08:06:16.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:16 smithi134 ceph-mon[30086]: 5.15 scrub ok 2024-02-05T08:06:16.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:16 smithi134 ceph-mon[30086]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:16.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:16 smithi134 ceph-mon[30086]: 5.19 scrub starts 2024-02-05T08:06:16.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:16 smithi134 ceph-mon[30086]: 5.19 scrub ok 2024-02-05T08:06:16.775 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:16 smithi134 ceph-mon[39278]: 5.15 scrub starts 2024-02-05T08:06:16.775 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:16 smithi134 ceph-mon[39278]: 5.15 scrub ok 2024-02-05T08:06:16.775 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:16 smithi134 ceph-mon[39278]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:16.775 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:16 smithi134 ceph-mon[39278]: 5.19 scrub starts 2024-02-05T08:06:16.775 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:16 smithi134 ceph-mon[39278]: 5.19 scrub ok 2024-02-05T08:06:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:16 smithi196 ceph-mon[33018]: 5.15 scrub starts 2024-02-05T08:06:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:16 smithi196 ceph-mon[33018]: 5.15 scrub ok 2024-02-05T08:06:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:16 smithi196 ceph-mon[33018]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:16 smithi196 ceph-mon[33018]: 5.19 scrub starts 2024-02-05T08:06:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:16 smithi196 ceph-mon[33018]: 5.19 scrub ok 2024-02-05T08:06:17.480 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:06:17.483 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:06:17.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:17 smithi134 ceph-mon[30086]: 3.15 scrub starts 2024-02-05T08:06:17.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:17 smithi134 ceph-mon[30086]: 3.15 scrub ok 2024-02-05T08:06:17.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:17 smithi134 ceph-mon[30086]: 4.19 scrub starts 2024-02-05T08:06:17.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:17 smithi134 ceph-mon[30086]: 4.19 scrub ok 2024-02-05T08:06:17.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:17 smithi134 ceph-mon[39278]: 3.15 scrub starts 2024-02-05T08:06:17.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:17 smithi134 ceph-mon[39278]: 3.15 scrub ok 2024-02-05T08:06:17.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:17 smithi134 ceph-mon[39278]: 4.19 scrub starts 2024-02-05T08:06:17.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:17 smithi134 ceph-mon[39278]: 4.19 scrub ok 2024-02-05T08:06:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:17 smithi196 ceph-mon[33018]: 3.15 scrub starts 2024-02-05T08:06:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:17 smithi196 ceph-mon[33018]: 3.15 scrub ok 2024-02-05T08:06:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:17 smithi196 ceph-mon[33018]: 4.19 scrub starts 2024-02-05T08:06:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:17 smithi196 ceph-mon[33018]: 4.19 scrub ok 2024-02-05T08:06:18.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:06:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:06:18.241 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-02-05T08:06:15.492564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1018,"num_read_kb":864,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":622,"ondisk_log_size":622,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":245884,"kb_used_data":8956,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749715332,"statfs":{"total":767960285184,"available":767708499968,"internally_reserved":0,"allocated":9170944,"data_stored":5261412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":242535987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003072"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529806+0000","last_change":"2024-02-05T08:05:55.114592+0000","last_active":"2024-02-05T08:05:59.529806+0000","last_peered":"2024-02-05T08:05:59.529806+0000","last_clean":"2024-02-05T08:05:59.529806+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:05:59.529806+0000","last_undegraded":"2024-02-05T08:05:59.529806+0000","last_fullsized":"2024-02-05T08:05:59.529806+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:55.114498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.114498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.383006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527645+0000","last_change":"2024-02-05T08:04:45.560680+0000","last_active":"2024-02-05T08:05:59.527645+0000","last_peered":"2024-02-05T08:05:59.527645+0000","last_clean":"2024-02-05T08:05:59.527645+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:05:59.527645+0000","last_undegraded":"2024-02-05T08:05:59.527645+0000","last_fullsized":"2024-02-05T08:05:59.527645+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527961+0000","last_change":"2024-02-05T08:05:53.302720+0000","last_active":"2024-02-05T08:05:59.527961+0000","last_peered":"2024-02-05T08:05:59.527961+0000","last_clean":"2024-02-05T08:05:59.527961+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:05:59.527961+0000","last_undegraded":"2024-02-05T08:05:59.527961+0000","last_fullsized":"2024-02-05T08:05:59.527961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:53.302642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.302642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:57.391075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"90'13","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529707+0000","last_change":"2024-02-05T08:05:58.103861+0000","last_active":"2024-02-05T08:05:59.529707+0000","last_peered":"2024-02-05T08:05:59.529707+0000","last_clean":"2024-02-05T08:05:59.529707+0000","last_became_active":"2024-02-05T08:04:43.551169+0000","last_became_peered":"2024-02-05T08:04:43.551169+0000","last_unstale":"2024-02-05T08:05:59.529707+0000","last_undegraded":"2024-02-05T08:05:59.529707+0000","last_fullsized":"2024-02-05T08:05:59.529707+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.103749+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:14.381872+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.227351+0000","last_change":"2024-02-05T08:06:00.227371+0000","last_active":"2024-02-05T08:06:00.227351+0000","last_peered":"2024-02-05T08:06:00.227351+0000","last_clean":"2024-02-05T08:06:00.227351+0000","last_became_active":"2024-02-05T08:04:48.172811+0000","last_became_peered":"2024-02-05T08:04:48.172811+0000","last_unstale":"2024-02-05T08:06:00.227351+0000","last_undegraded":"2024-02-05T08:06:00.227351+0000","last_fullsized":"2024-02-05T08:06:00.227351+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:00.227295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:00.227295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:06.714686+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529726+0000","last_change":"2024-02-05T08:04:45.564397+0000","last_active":"2024-02-05T08:05:59.529726+0000","last_peered":"2024-02-05T08:05:59.529726+0000","last_clean":"2024-02-05T08:05:59.529726+0000","last_became_active":"2024-02-05T08:04:45.563795+0000","last_became_peered":"2024-02-05T08:04:45.563795+0000","last_unstale":"2024-02-05T08:05:59.529726+0000","last_undegraded":"2024-02-05T08:05:59.529726+0000","last_fullsized":"2024-02-05T08:05:59.529726+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.204318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226552+0000","last_change":"2024-02-05T08:05:42.259337+0000","last_active":"2024-02-05T08:06:00.226552+0000","last_peered":"2024-02-05T08:06:00.226552+0000","last_clean":"2024-02-05T08:06:00.226552+0000","last_became_active":"2024-02-05T08:04:41.535435+0000","last_became_peered":"2024-02-05T08:04:41.535435+0000","last_unstale":"2024-02-05T08:06:00.226552+0000","last_undegraded":"2024-02-05T08:06:00.226552+0000","last_fullsized":"2024-02-05T08:06:00.226552+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:42.259257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:42.259257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:48.329795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226085+0000","last_change":"2024-02-05T08:04:43.560996+0000","last_active":"2024-02-05T08:06:00.226085+0000","last_peered":"2024-02-05T08:06:00.226085+0000","last_clean":"2024-02-05T08:06:00.226085+0000","last_became_active":"2024-02-05T08:04:43.560660+0000","last_became_peered":"2024-02-05T08:04:43.560660+0000","last_unstale":"2024-02-05T08:06:00.226085+0000","last_undegraded":"2024-02-05T08:06:00.226085+0000","last_fullsized":"2024-02-05T08:06:00.226085+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528432+0000","last_change":"2024-02-05T08:05:56.285954+0000","last_active":"2024-02-05T08:05:59.528432+0000","last_peered":"2024-02-05T08:05:59.528432+0000","last_clean":"2024-02-05T08:05:59.528432+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:05:59.528432+0000","last_undegraded":"2024-02-05T08:05:59.528432+0000","last_fullsized":"2024-02-05T08:05:59.528432+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:56.285876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:56.285876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:18.137707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268950+0000","last_change":"2024-02-05T08:04:45.560281+0000","last_active":"2024-02-05T08:06:00.268950+0000","last_peered":"2024-02-05T08:06:00.268950+0000","last_clean":"2024-02-05T08:06:00.268950+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:06:00.268950+0000","last_undegraded":"2024-02-05T08:06:00.268950+0000","last_fullsized":"2024-02-05T08:06:00.268950+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:14.202262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269001+0000","last_change":"2024-02-05T08:05:55.576535+0000","last_active":"2024-02-05T08:06:00.269001+0000","last_peered":"2024-02-05T08:06:00.269001+0000","last_clean":"2024-02-05T08:06:00.269001+0000","last_became_active":"2024-02-05T08:04:41.540764+0000","last_became_peered":"2024-02-05T08:04:41.540764+0000","last_unstale":"2024-02-05T08:06:00.269001+0000","last_undegraded":"2024-02-05T08:06:00.269001+0000","last_fullsized":"2024-02-05T08:06:00.269001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:55.576441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.576441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.917805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529949+0000","last_change":"2024-02-05T08:04:43.557923+0000","last_active":"2024-02-05T08:05:59.529949+0000","last_peered":"2024-02-05T08:05:59.529949+0000","last_clean":"2024-02-05T08:05:59.529949+0000","last_became_active":"2024-02-05T08:04:43.557652+0000","last_became_peered":"2024-02-05T08:04:43.557652+0000","last_unstale":"2024-02-05T08:05:59.529949+0000","last_undegraded":"2024-02-05T08:05:59.529949+0000","last_fullsized":"2024-02-05T08:05:59.529949+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.987638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.267980+0000","last_change":"2024-02-05T08:04:47.587766+0000","last_active":"2024-02-05T08:06:00.267980+0000","last_peered":"2024-02-05T08:06:00.267980+0000","last_clean":"2024-02-05T08:06:00.267980+0000","last_became_active":"2024-02-05T08:04:47.587515+0000","last_became_peered":"2024-02-05T08:04:47.587515+0000","last_unstale":"2024-02-05T08:06:00.267980+0000","last_undegraded":"2024-02-05T08:06:00.267980+0000","last_fullsized":"2024-02-05T08:06:00.267980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.970614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718175+0000","last_change":"2024-02-05T08:05:52.725804+0000","last_active":"2024-02-05T08:05:59.718175+0000","last_peered":"2024-02-05T08:05:59.718175+0000","last_clean":"2024-02-05T08:05:59.718175+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:05:59.718175+0000","last_undegraded":"2024-02-05T08:05:59.718175+0000","last_fullsized":"2024-02-05T08:05:59.718175+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:52.725679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.725679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.406845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268975+0000","last_change":"2024-02-05T08:05:46.458359+0000","last_active":"2024-02-05T08:06:00.268975+0000","last_peered":"2024-02-05T08:06:00.268975+0000","last_clean":"2024-02-05T08:06:00.268975+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:06:00.268975+0000","last_undegraded":"2024-02-05T08:06:00.268975+0000","last_fullsized":"2024-02-05T08:06:00.268975+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.458244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.877756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530213+0000","last_change":"2024-02-05T08:04:43.557900+0000","last_active":"2024-02-05T08:05:59.530213+0000","last_peered":"2024-02-05T08:05:59.530213+0000","last_clean":"2024-02-05T08:05:59.530213+0000","last_became_active":"2024-02-05T08:04:43.557559+0000","last_became_peered":"2024-02-05T08:04:43.557559+0000","last_unstale":"2024-02-05T08:05:59.530213+0000","last_undegraded":"2024-02-05T08:05:59.530213+0000","last_fullsized":"2024-02-05T08:05:59.530213+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:09.465379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530663+0000","last_change":"2024-02-05T08:05:51.076610+0000","last_active":"2024-02-05T08:05:59.530663+0000","last_peered":"2024-02-05T08:05:59.530663+0000","last_clean":"2024-02-05T08:05:59.530663+0000","last_became_active":"2024-02-05T08:04:48.170756+0000","last_became_peered":"2024-02-05T08:04:48.170756+0000","last_unstale":"2024-02-05T08:05:59.530663+0000","last_undegraded":"2024-02-05T08:05:59.530663+0000","last_fullsized":"2024-02-05T08:05:59.530663+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:51.076514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:51.076514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:11.206152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527244+0000","last_change":"2024-02-05T08:04:45.559908+0000","last_active":"2024-02-05T08:05:59.527244+0000","last_peered":"2024-02-05T08:05:59.527244+0000","last_clean":"2024-02-05T08:05:59.527244+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:05:59.527244+0000","last_undegraded":"2024-02-05T08:05:59.527244+0000","last_fullsized":"2024-02-05T08:05:59.527244+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:11.185330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528706+0000","last_change":"2024-02-05T08:04:41.541812+0000","last_active":"2024-02-05T08:05:59.528706+0000","last_peered":"2024-02-05T08:05:59.528706+0000","last_clean":"2024-02-05T08:05:59.528706+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:05:59.528706+0000","last_undegraded":"2024-02-05T08:05:59.528706+0000","last_fullsized":"2024-02-05T08:05:59.528706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546521+0000","last_change":"2024-02-05T08:05:46.814239+0000","last_active":"2024-02-05T08:06:00.546521+0000","last_peered":"2024-02-05T08:06:00.546521+0000","last_clean":"2024-02-05T08:06:00.546521+0000","last_became_active":"2024-02-05T08:04:43.557496+0000","last_became_peered":"2024-02-05T08:04:43.557496+0000","last_unstale":"2024-02-05T08:06:00.546521+0000","last_undegraded":"2024-02-05T08:06:00.546521+0000","last_fullsized":"2024-02-05T08:06:00.546521+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:05:46.814139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.814139+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:58.081182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225446+0000","last_change":"2024-02-05T08:04:48.170578+0000","last_active":"2024-02-05T08:06:00.225446+0000","last_peered":"2024-02-05T08:06:00.225446+0000","last_clean":"2024-02-05T08:06:00.225446+0000","last_became_active":"2024-02-05T08:04:48.169770+0000","last_became_peered":"2024-02-05T08:04:48.169770+0000","last_unstale":"2024-02-05T08:06:00.225446+0000","last_undegraded":"2024-02-05T08:06:00.225446+0000","last_fullsized":"2024-02-05T08:06:00.225446+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.961379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530140+0000","last_change":"2024-02-05T08:05:57.052122+0000","last_active":"2024-02-05T08:05:59.530140+0000","last_peered":"2024-02-05T08:05:59.530140+0000","last_clean":"2024-02-05T08:05:59.530140+0000","last_became_active":"2024-02-05T08:04:45.563948+0000","last_became_peered":"2024-02-05T08:04:45.563948+0000","last_unstale":"2024-02-05T08:05:59.530140+0000","last_undegraded":"2024-02-05T08:05:59.530140+0000","last_fullsized":"2024-02-05T08:05:59.530140+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.052030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:14.589576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530056+0000","last_change":"2024-02-05T08:04:41.534341+0000","last_active":"2024-02-05T08:05:59.530056+0000","last_peered":"2024-02-05T08:05:59.530056+0000","last_clean":"2024-02-05T08:05:59.530056+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:05:59.530056+0000","last_undegraded":"2024-02-05T08:05:59.530056+0000","last_fullsized":"2024-02-05T08:05:59.530056+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.393385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"90'17","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226406+0000","last_change":"2024-02-05T08:05:58.285391+0000","last_active":"2024-02-05T08:06:00.226406+0000","last_peered":"2024-02-05T08:06:00.226406+0000","last_clean":"2024-02-05T08:06:00.226406+0000","last_became_active":"2024-02-05T08:04:43.557596+0000","last_became_peered":"2024-02-05T08:04:43.557596+0000","last_unstale":"2024-02-05T08:06:00.226406+0000","last_undegraded":"2024-02-05T08:06:00.226406+0000","last_fullsized":"2024-02-05T08:06:00.226406+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.285304+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.927379+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718547+0000","last_change":"2024-02-05T08:05:55.766576+0000","last_active":"2024-02-05T08:05:59.718547+0000","last_peered":"2024-02-05T08:05:59.718547+0000","last_clean":"2024-02-05T08:05:59.718547+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:05:59.718547+0000","last_undegraded":"2024-02-05T08:05:59.718547+0000","last_fullsized":"2024-02-05T08:05:59.718547+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:05:55.766478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.766478+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:59.016044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.601853+0000","last_change":"2024-02-05T08:04:45.565057+0000","last_active":"2024-02-05T08:06:06.601853+0000","last_peered":"2024-02-05T08:06:06.601853+0000","last_clean":"2024-02-05T08:06:06.601853+0000","last_became_active":"2024-02-05T08:04:45.564774+0000","last_became_peered":"2024-02-05T08:04:45.564774+0000","last_unstale":"2024-02-05T08:06:06.601853+0000","last_undegraded":"2024-02-05T08:06:06.601853+0000","last_fullsized":"2024-02-05T08:06:06.601853+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:11.775078+0000","last_change":"2024-02-05T08:06:11.775125+0000","last_active":"2024-02-05T08:06:11.775078+0000","last_peered":"2024-02-05T08:06:11.775078+0000","last_clean":"2024-02-05T08:06:11.775078+0000","last_became_active":"2024-02-05T08:04:41.532339+0000","last_became_peered":"2024-02-05T08:04:41.532339+0000","last_unstale":"2024-02-05T08:06:11.775078+0000","last_undegraded":"2024-02-05T08:06:11.775078+0000","last_fullsized":"2024-02-05T08:06:11.775078+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:06:11.774993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:11.774993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.286950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528405+0000","last_change":"2024-02-05T08:04:43.606275+0000","last_active":"2024-02-05T08:05:59.528405+0000","last_peered":"2024-02-05T08:05:59.528405+0000","last_clean":"2024-02-05T08:05:59.528405+0000","last_became_active":"2024-02-05T08:04:43.605952+0000","last_became_peered":"2024-02-05T08:04:43.605952+0000","last_unstale":"2024-02-05T08:05:59.528405+0000","last_undegraded":"2024-02-05T08:05:59.528405+0000","last_fullsized":"2024-02-05T08:05:59.528405+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:09.695654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:09.715125+0000","last_change":"2024-02-05T08:06:09.715146+0000","last_active":"2024-02-05T08:06:09.715125+0000","last_peered":"2024-02-05T08:06:09.715125+0000","last_clean":"2024-02-05T08:06:09.715125+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:06:09.715125+0000","last_undegraded":"2024-02-05T08:06:09.715125+0000","last_fullsized":"2024-02-05T08:06:09.715125+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:09.715067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:09.715067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:24.602489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528636+0000","last_change":"2024-02-05T08:04:45.564760+0000","last_active":"2024-02-05T08:05:59.528636+0000","last_peered":"2024-02-05T08:05:59.528636+0000","last_clean":"2024-02-05T08:05:59.528636+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:05:59.528636+0000","last_undegraded":"2024-02-05T08:05:59.528636+0000","last_fullsized":"2024-02-05T08:05:59.528636+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.404706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:07.137987+0000","last_change":"2024-02-05T08:06:07.138006+0000","last_active":"2024-02-05T08:06:07.137987+0000","last_peered":"2024-02-05T08:06:07.137987+0000","last_clean":"2024-02-05T08:06:07.137987+0000","last_became_active":"2024-02-05T08:04:41.534448+0000","last_became_peered":"2024-02-05T08:04:41.534448+0000","last_unstale":"2024-02-05T08:06:07.137987+0000","last_undegraded":"2024-02-05T08:06:07.137987+0000","last_fullsized":"2024-02-05T08:06:07.137987+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:06:07.137916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:29.601881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.721085+0000","last_change":"2024-02-05T08:05:46.208637+0000","last_active":"2024-02-05T08:05:59.721085+0000","last_peered":"2024-02-05T08:05:59.721085+0000","last_clean":"2024-02-05T08:05:59.721085+0000","last_became_active":"2024-02-05T08:05:03.789351+0000","last_became_peered":"2024-02-05T08:05:03.789351+0000","last_unstale":"2024-02-05T08:05:59.721085+0000","last_undegraded":"2024-02-05T08:05:59.721085+0000","last_fullsized":"2024-02-05T08:05:59.721085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:05:46.208534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.208534+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:00.233101+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.717804+0000","last_change":"2024-02-05T08:04:47.579487+0000","last_active":"2024-02-05T08:05:59.717804+0000","last_peered":"2024-02-05T08:05:59.717804+0000","last_clean":"2024-02-05T08:05:59.717804+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:05:59.717804+0000","last_undegraded":"2024-02-05T08:05:59.717804+0000","last_fullsized":"2024-02-05T08:05:59.717804+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:10.763840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720346+0000","last_change":"2024-02-05T08:05:54.367693+0000","last_active":"2024-02-05T08:05:59.720346+0000","last_peered":"2024-02-05T08:05:59.720346+0000","last_clean":"2024-02-05T08:05:59.720346+0000","last_became_active":"2024-02-05T08:04:45.560874+0000","last_became_peered":"2024-02-05T08:04:45.560874+0000","last_unstale":"2024-02-05T08:05:59.720346+0000","last_undegraded":"2024-02-05T08:05:59.720346+0000","last_fullsized":"2024-02-05T08:05:59.720346+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:05:54.367588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:17.218061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:10.193331+0000","last_change":"2024-02-05T08:06:10.193352+0000","last_active":"2024-02-05T08:06:10.193331+0000","last_peered":"2024-02-05T08:06:10.193331+0000","last_clean":"2024-02-05T08:06:10.193331+0000","last_became_active":"2024-02-05T08:04:41.536019+0000","last_became_peered":"2024-02-05T08:04:41.536019+0000","last_unstale":"2024-02-05T08:06:10.193331+0000","last_undegraded":"2024-02-05T08:06:10.193331+0000","last_fullsized":"2024-02-05T08:06:10.193331+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:10.193272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:10.193272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:33.575164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225759+0000","last_change":"2024-02-05T08:04:43.555567+0000","last_active":"2024-02-05T08:06:00.225759+0000","last_peered":"2024-02-05T08:06:00.225759+0000","last_clean":"2024-02-05T08:06:00.225759+0000","last_became_active":"2024-02-05T08:04:43.552866+0000","last_became_peered":"2024-02-05T08:04:43.552866+0000","last_unstale":"2024-02-05T08:06:00.225759+0000","last_undegraded":"2024-02-05T08:06:00.225759+0000","last_fullsized":"2024-02-05T08:06:00.225759+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:03.755886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:02.190986+0000","last_change":"2024-02-05T08:06:02.191006+0000","last_active":"2024-02-05T08:06:02.190986+0000","last_peered":"2024-02-05T08:06:02.190986+0000","last_clean":"2024-02-05T08:06:02.190986+0000","last_became_active":"2024-02-05T08:04:48.172793+0000","last_became_peered":"2024-02-05T08:04:48.172793+0000","last_unstale":"2024-02-05T08:06:02.190986+0000","last_undegraded":"2024-02-05T08:06:02.190986+0000","last_fullsized":"2024-02-05T08:06:02.190986+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:02.190931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:02.190931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:27.955974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718344+0000","last_change":"2024-02-05T08:05:48.748227+0000","last_active":"2024-02-05T08:05:59.718344+0000","last_peered":"2024-02-05T08:05:59.718344+0000","last_clean":"2024-02-05T08:05:59.718344+0000","last_became_active":"2024-02-05T08:04:45.563610+0000","last_became_peered":"2024-02-05T08:04:45.563610+0000","last_unstale":"2024-02-05T08:05:59.718344+0000","last_undegraded":"2024-02-05T08:05:59.718344+0000","last_fullsized":"2024-02-05T08:05:59.718344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:48.748132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:48.748132+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.748132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:17.563379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:02.532340+0000","last_change":"2024-02-05T08:06:02.532375+0000","last_active":"2024-02-05T08:06:02.532340+0000","last_peered":"2024-02-05T08:06:02.532340+0000","last_clean":"2024-02-05T08:06:02.532340+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:06:02.532340+0000","last_undegraded":"2024-02-05T08:06:02.532340+0000","last_fullsized":"2024-02-05T08:06:02.532340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:02.532248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:02.532248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:05.965893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226059+0000","last_change":"2024-02-05T08:04:43.606219+0000","last_active":"2024-02-05T08:06:00.226059+0000","last_peered":"2024-02-05T08:06:00.226059+0000","last_clean":"2024-02-05T08:06:00.226059+0000","last_became_active":"2024-02-05T08:04:43.605932+0000","last_became_peered":"2024-02-05T08:04:43.605932+0000","last_unstale":"2024-02-05T08:06:00.226059+0000","last_undegraded":"2024-02-05T08:06:00.226059+0000","last_fullsized":"2024-02-05T08:06:00.226059+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:00.517159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:03.183974+0000","last_change":"2024-02-05T08:06:03.184008+0000","last_active":"2024-02-05T08:06:03.183974+0000","last_peered":"2024-02-05T08:06:03.183974+0000","last_clean":"2024-02-05T08:06:03.183974+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:06:03.183974+0000","last_undegraded":"2024-02-05T08:06:03.183974+0000","last_fullsized":"2024-02-05T08:06:03.183974+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:03.183906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:03.183906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:24.748158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226266+0000","last_change":"2024-02-05T08:05:57.270638+0000","last_active":"2024-02-05T08:06:00.226266+0000","last_peered":"2024-02-05T08:06:00.226266+0000","last_clean":"2024-02-05T08:06:00.226266+0000","last_became_active":"2024-02-05T08:04:45.568484+0000","last_became_peered":"2024-02-05T08:04:45.568484+0000","last_unstale":"2024-02-05T08:06:00.226266+0000","last_undegraded":"2024-02-05T08:06:00.226266+0000","last_fullsized":"2024-02-05T08:06:00.226266+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:57.270554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.270554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:03.898076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718661+0000","last_change":"2024-02-05T08:04:41.540005+0000","last_active":"2024-02-05T08:05:59.718661+0000","last_peered":"2024-02-05T08:05:59.718661+0000","last_clean":"2024-02-05T08:05:59.718661+0000","last_became_active":"2024-02-05T08:04:41.539698+0000","last_became_peered":"2024-02-05T08:04:41.539698+0000","last_unstale":"2024-02-05T08:05:59.718661+0000","last_undegraded":"2024-02-05T08:05:59.718661+0000","last_fullsized":"2024-02-05T08:05:59.718661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.105779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527388+0000","last_change":"2024-02-05T08:05:48.910387+0000","last_active":"2024-02-05T08:05:59.527388+0000","last_peered":"2024-02-05T08:05:59.527388+0000","last_clean":"2024-02-05T08:05:59.527388+0000","last_became_active":"2024-02-05T08:04:43.555031+0000","last_became_peered":"2024-02-05T08:04:43.555031+0000","last_unstale":"2024-02-05T08:05:59.527388+0000","last_undegraded":"2024-02-05T08:05:59.527388+0000","last_fullsized":"2024-02-05T08:05:59.527388+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:05:48.910304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.910304+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:49.156858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225328+0000","last_change":"2024-02-05T08:04:47.586079+0000","last_active":"2024-02-05T08:06:00.225328+0000","last_peered":"2024-02-05T08:06:00.225328+0000","last_clean":"2024-02-05T08:06:00.225328+0000","last_became_active":"2024-02-05T08:04:47.585496+0000","last_became_peered":"2024-02-05T08:04:47.585496+0000","last_unstale":"2024-02-05T08:06:00.225328+0000","last_undegraded":"2024-02-05T08:06:00.225328+0000","last_fullsized":"2024-02-05T08:06:00.225328+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:12.376304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527208+0000","last_change":"2024-02-05T08:05:46.864673+0000","last_active":"2024-02-05T08:05:59.527208+0000","last_peered":"2024-02-05T08:05:59.527208+0000","last_clean":"2024-02-05T08:05:59.527208+0000","last_became_active":"2024-02-05T08:04:45.560815+0000","last_became_peered":"2024-02-05T08:04:45.560815+0000","last_unstale":"2024-02-05T08:05:59.527208+0000","last_undegraded":"2024-02-05T08:05:59.527208+0000","last_fullsized":"2024-02-05T08:05:59.527208+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:46.864589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.864589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.980095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530377+0000","last_change":"2024-02-05T08:05:53.112020+0000","last_active":"2024-02-05T08:05:59.530377+0000","last_peered":"2024-02-05T08:05:59.530377+0000","last_clean":"2024-02-05T08:05:59.530377+0000","last_became_active":"2024-02-05T08:04:41.543255+0000","last_became_peered":"2024-02-05T08:04:41.543255+0000","last_unstale":"2024-02-05T08:05:59.530377+0000","last_undegraded":"2024-02-05T08:05:59.530377+0000","last_fullsized":"2024-02-05T08:05:59.530377+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:53.111931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.111931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:19.887019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530372+0000","last_change":"2024-02-05T08:04:43.554427+0000","last_active":"2024-02-05T08:05:59.530372+0000","last_peered":"2024-02-05T08:05:59.530372+0000","last_clean":"2024-02-05T08:05:59.530372+0000","last_became_active":"2024-02-05T08:04:43.554101+0000","last_became_peered":"2024-02-05T08:04:43.554101+0000","last_unstale":"2024-02-05T08:05:59.530372+0000","last_undegraded":"2024-02-05T08:05:59.530372+0000","last_fullsized":"2024-02-05T08:05:59.530372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:05.485949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528132+0000","last_change":"2024-02-05T08:04:47.584516+0000","last_active":"2024-02-05T08:05:59.528132+0000","last_peered":"2024-02-05T08:05:59.528132+0000","last_clean":"2024-02-05T08:05:59.528132+0000","last_became_active":"2024-02-05T08:04:47.584235+0000","last_became_peered":"2024-02-05T08:04:47.584235+0000","last_unstale":"2024-02-05T08:05:59.528132+0000","last_undegraded":"2024-02-05T08:05:59.528132+0000","last_fullsized":"2024-02-05T08:05:59.528132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:11.950403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.601874+0000","last_change":"2024-02-05T08:04:45.559777+0000","last_active":"2024-02-05T08:06:06.601874+0000","last_peered":"2024-02-05T08:06:06.601874+0000","last_clean":"2024-02-05T08:06:06.601874+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:06:06.601874+0000","last_undegraded":"2024-02-05T08:06:06.601874+0000","last_fullsized":"2024-02-05T08:06:06.601874+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718361+0000","last_change":"2024-02-05T08:05:45.722765+0000","last_active":"2024-02-05T08:05:59.718361+0000","last_peered":"2024-02-05T08:05:59.718361+0000","last_clean":"2024-02-05T08:05:59.718361+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:05:59.718361+0000","last_undegraded":"2024-02-05T08:05:59.718361+0000","last_fullsized":"2024-02-05T08:05:59.718361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:45.722652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:45.722652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.867081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226318+0000","last_change":"2024-02-05T08:04:43.606315+0000","last_active":"2024-02-05T08:06:00.226318+0000","last_peered":"2024-02-05T08:06:00.226318+0000","last_clean":"2024-02-05T08:06:00.226318+0000","last_became_active":"2024-02-05T08:04:43.605981+0000","last_became_peered":"2024-02-05T08:04:43.605981+0000","last_unstale":"2024-02-05T08:06:00.226318+0000","last_undegraded":"2024-02-05T08:06:00.226318+0000","last_fullsized":"2024-02-05T08:06:00.226318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530817+0000","last_change":"2024-02-05T08:04:47.578924+0000","last_active":"2024-02-05T08:05:59.530817+0000","last_peered":"2024-02-05T08:05:59.530817+0000","last_clean":"2024-02-05T08:05:59.530817+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:05:59.530817+0000","last_undegraded":"2024-02-05T08:05:59.530817+0000","last_fullsized":"2024-02-05T08:05:59.530817+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:03.039774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:10.047557+0000","last_change":"2024-02-05T08:06:10.047557+0000","last_active":"2024-02-05T08:06:10.047557+0000","last_peered":"2024-02-05T08:06:10.047557+0000","last_clean":"2024-02-05T08:06:10.047557+0000","last_became_active":"2024-02-05T08:04:45.565240+0000","last_became_peered":"2024-02-05T08:04:45.565240+0000","last_unstale":"2024-02-05T08:06:10.047557+0000","last_undegraded":"2024-02-05T08:06:10.047557+0000","last_fullsized":"2024-02-05T08:06:10.047557+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:06:10.047432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:10.047432+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:11.831172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527536+0000","last_change":"2024-02-05T08:04:41.540528+0000","last_active":"2024-02-05T08:05:59.527536+0000","last_peered":"2024-02-05T08:05:59.527536+0000","last_clean":"2024-02-05T08:05:59.527536+0000","last_became_active":"2024-02-05T08:04:41.540252+0000","last_became_peered":"2024-02-05T08:04:41.540252+0000","last_unstale":"2024-02-05T08:05:59.527536+0000","last_undegraded":"2024-02-05T08:05:59.527536+0000","last_fullsized":"2024-02-05T08:05:59.527536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:02.085874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269241+0000","last_change":"2024-02-05T08:05:56.541620+0000","last_active":"2024-02-05T08:06:00.269241+0000","last_peered":"2024-02-05T08:06:00.269241+0000","last_clean":"2024-02-05T08:06:00.269241+0000","last_became_active":"2024-02-05T08:04:43.606414+0000","last_became_peered":"2024-02-05T08:04:43.606414+0000","last_unstale":"2024-02-05T08:06:00.269241+0000","last_undegraded":"2024-02-05T08:06:00.269241+0000","last_fullsized":"2024-02-05T08:06:00.269241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:05:56.541510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:56.541510+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:04.755122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:09.355522+0000","last_change":"2024-02-05T08:06:09.355544+0000","last_active":"2024-02-05T08:06:09.355522+0000","last_peered":"2024-02-05T08:06:09.355522+0000","last_clean":"2024-02-05T08:06:09.355522+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:06:09.355522+0000","last_undegraded":"2024-02-05T08:06:09.355522+0000","last_fullsized":"2024-02-05T08:06:09.355522+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:09.355465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:09.355465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.216149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:04.157672+0000","last_change":"2024-02-05T08:06:04.157708+0000","last_active":"2024-02-05T08:06:04.157672+0000","last_peered":"2024-02-05T08:06:04.157672+0000","last_clean":"2024-02-05T08:06:04.157672+0000","last_became_active":"2024-02-05T08:04:45.567622+0000","last_became_peered":"2024-02-05T08:04:45.567622+0000","last_unstale":"2024-02-05T08:06:04.157672+0000","last_undegraded":"2024-02-05T08:06:04.157672+0000","last_fullsized":"2024-02-05T08:06:04.157672+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:04.157598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:04.157598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:15.237706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718248+0000","last_change":"2024-02-05T08:04:41.541120+0000","last_active":"2024-02-05T08:05:59.718248+0000","last_peered":"2024-02-05T08:05:59.718248+0000","last_clean":"2024-02-05T08:05:59.718248+0000","last_became_active":"2024-02-05T08:04:41.540432+0000","last_became_peered":"2024-02-05T08:04:41.540432+0000","last_unstale":"2024-02-05T08:05:59.718248+0000","last_undegraded":"2024-02-05T08:05:59.718248+0000","last_fullsized":"2024-02-05T08:05:59.718248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:10.219126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528495+0000","last_change":"2024-02-05T08:04:43.608826+0000","last_active":"2024-02-05T08:05:59.528495+0000","last_peered":"2024-02-05T08:05:59.528495+0000","last_clean":"2024-02-05T08:05:59.528495+0000","last_became_active":"2024-02-05T08:04:43.608489+0000","last_became_peered":"2024-02-05T08:04:43.608489+0000","last_unstale":"2024-02-05T08:05:59.528495+0000","last_undegraded":"2024-02-05T08:05:59.528495+0000","last_fullsized":"2024-02-05T08:05:59.528495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.929658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718647+0000","last_change":"2024-02-05T08:05:56.721472+0000","last_active":"2024-02-05T08:05:59.718647+0000","last_peered":"2024-02-05T08:05:59.718647+0000","last_clean":"2024-02-05T08:05:59.718647+0000","last_became_active":"2024-02-05T08:04:48.172747+0000","last_became_peered":"2024-02-05T08:04:48.172747+0000","last_unstale":"2024-02-05T08:05:59.718647+0000","last_undegraded":"2024-02-05T08:05:59.718647+0000","last_fullsized":"2024-02-05T08:05:59.718647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:56.721381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:56.721381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.336615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:04.495948+0000","last_change":"2024-02-05T08:06:04.495991+0000","last_active":"2024-02-05T08:06:04.495948+0000","last_peered":"2024-02-05T08:06:04.495948+0000","last_clean":"2024-02-05T08:06:04.495948+0000","last_became_active":"2024-02-05T08:04:45.560241+0000","last_became_peered":"2024-02-05T08:04:45.560241+0000","last_unstale":"2024-02-05T08:06:04.495948+0000","last_undegraded":"2024-02-05T08:06:04.495948+0000","last_fullsized":"2024-02-05T08:06:04.495948+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:04.495861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:04.495861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:12.837854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720257+0000","last_change":"2024-02-05T08:05:49.229502+0000","last_active":"2024-02-05T08:05:59.720257+0000","last_peered":"2024-02-05T08:05:59.720257+0000","last_clean":"2024-02-05T08:05:59.720257+0000","last_became_active":"2024-02-05T08:04:41.537709+0000","last_became_peered":"2024-02-05T08:04:41.537709+0000","last_unstale":"2024-02-05T08:05:59.720257+0000","last_undegraded":"2024-02-05T08:05:59.720257+0000","last_fullsized":"2024-02-05T08:05:59.720257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:49.229415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:49.229415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:03.840509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:11.050373+0000","last_change":"2024-02-05T08:06:11.050399+0000","last_active":"2024-02-05T08:06:11.050373+0000","last_peered":"2024-02-05T08:06:11.050373+0000","last_clean":"2024-02-05T08:06:11.050373+0000","last_became_active":"2024-02-05T08:04:43.606624+0000","last_became_peered":"2024-02-05T08:04:43.606624+0000","last_unstale":"2024-02-05T08:06:11.050373+0000","last_undegraded":"2024-02-05T08:06:11.050373+0000","last_fullsized":"2024-02-05T08:06:11.050373+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-05T08:06:11.050310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:11.050310+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:27.559791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268004+0000","last_change":"2024-02-05T08:05:59.500065+0000","last_active":"2024-02-05T08:06:00.268004+0000","last_peered":"2024-02-05T08:06:00.268004+0000","last_clean":"2024-02-05T08:06:00.268004+0000","last_became_active":"2024-02-05T08:04:48.170100+0000","last_became_peered":"2024-02-05T08:04:48.170100+0000","last_unstale":"2024-02-05T08:06:00.268004+0000","last_undegraded":"2024-02-05T08:06:00.268004+0000","last_fullsized":"2024-02-05T08:06:00.268004+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:59.499960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:59.499960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:16.082380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718992+0000","last_change":"2024-02-05T08:05:57.687803+0000","last_active":"2024-02-05T08:05:59.718992+0000","last_peered":"2024-02-05T08:05:59.718992+0000","last_clean":"2024-02-05T08:05:59.718992+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:05:59.718992+0000","last_undegraded":"2024-02-05T08:05:59.718992+0000","last_fullsized":"2024-02-05T08:05:59.718992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:05:57.687679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.687679+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:23.035896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268523+0000","last_change":"2024-02-05T08:04:43.608563+0000","last_active":"2024-02-05T08:06:00.268523+0000","last_peered":"2024-02-05T08:06:00.268523+0000","last_clean":"2024-02-05T08:06:00.268523+0000","last_became_active":"2024-02-05T08:04:43.608254+0000","last_became_peered":"2024-02-05T08:04:43.608254+0000","last_unstale":"2024-02-05T08:06:00.268523+0000","last_undegraded":"2024-02-05T08:06:00.268523+0000","last_fullsized":"2024-02-05T08:06:00.268523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.187066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.601904+0000","last_change":"2024-02-05T08:05:59.720386+0000","last_active":"2024-02-05T08:06:06.601904+0000","last_peered":"2024-02-05T08:06:06.601904+0000","last_clean":"2024-02-05T08:06:06.601904+0000","last_became_active":"2024-02-05T08:04:45.561044+0000","last_became_peered":"2024-02-05T08:04:45.561044+0000","last_unstale":"2024-02-05T08:06:06.601904+0000","last_undegraded":"2024-02-05T08:06:06.601904+0000","last_fullsized":"2024-02-05T08:06:06.601904+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:05:59.720223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:59.720223+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.562177+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225545+0000","last_change":"2024-02-05T08:04:47.587025+0000","last_active":"2024-02-05T08:06:00.225545+0000","last_peered":"2024-02-05T08:06:00.225545+0000","last_clean":"2024-02-05T08:06:00.225545+0000","last_became_active":"2024-02-05T08:04:47.586044+0000","last_became_peered":"2024-02-05T08:04:47.586044+0000","last_unstale":"2024-02-05T08:06:00.225545+0000","last_undegraded":"2024-02-05T08:06:00.225545+0000","last_fullsized":"2024-02-05T08:06:00.225545+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.700997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.762873+0000","last_change":"2024-02-05T08:06:01.762898+0000","last_active":"2024-02-05T08:06:01.762873+0000","last_peered":"2024-02-05T08:06:01.762873+0000","last_clean":"2024-02-05T08:06:01.762873+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:06:01.762873+0000","last_undegraded":"2024-02-05T08:06:01.762873+0000","last_fullsized":"2024-02-05T08:06:01.762873+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:01.762805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:01.762805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:12.602032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530066+0000","last_change":"2024-02-05T08:04:43.556901+0000","last_active":"2024-02-05T08:05:59.530066+0000","last_peered":"2024-02-05T08:05:59.530066+0000","last_clean":"2024-02-05T08:05:59.530066+0000","last_became_active":"2024-02-05T08:04:43.556580+0000","last_became_peered":"2024-02-05T08:04:43.556580+0000","last_unstale":"2024-02-05T08:05:59.530066+0000","last_undegraded":"2024-02-05T08:05:59.530066+0000","last_fullsized":"2024-02-05T08:05:59.530066+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.567288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:07.368791+0000","last_change":"2024-02-05T08:06:07.368811+0000","last_active":"2024-02-05T08:06:07.368791+0000","last_peered":"2024-02-05T08:06:07.368791+0000","last_clean":"2024-02-05T08:06:07.368791+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:06:07.368791+0000","last_undegraded":"2024-02-05T08:06:07.368791+0000","last_fullsized":"2024-02-05T08:06:07.368791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_clean_scrub_stamp":"2024-02-05T08:06:07.368730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:21.064118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:03.741658+0000","last_change":"2024-02-05T08:06:03.741686+0000","last_active":"2024-02-05T08:06:03.741658+0000","last_peered":"2024-02-05T08:06:03.741658+0000","last_clean":"2024-02-05T08:06:03.741658+0000","last_became_active":"2024-02-05T08:04:47.587082+0000","last_became_peered":"2024-02-05T08:04:47.587082+0000","last_unstale":"2024-02-05T08:06:03.741658+0000","last_undegraded":"2024-02-05T08:06:03.741658+0000","last_fullsized":"2024-02-05T08:06:03.741658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:03.741577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:03.741577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:11.425517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546310+0000","last_change":"2024-02-05T08:05:55.596909+0000","last_active":"2024-02-05T08:06:00.546310+0000","last_peered":"2024-02-05T08:06:00.546310+0000","last_clean":"2024-02-05T08:06:00.546310+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:06:00.546310+0000","last_undegraded":"2024-02-05T08:06:00.546310+0000","last_fullsized":"2024-02-05T08:06:00.546310+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:55.596827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.596827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.318812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718966+0000","last_change":"2024-02-05T08:05:53.741209+0000","last_active":"2024-02-05T08:05:59.718966+0000","last_peered":"2024-02-05T08:05:59.718966+0000","last_clean":"2024-02-05T08:05:59.718966+0000","last_became_active":"2024-02-05T08:04:41.540586+0000","last_became_peered":"2024-02-05T08:04:41.540586+0000","last_unstale":"2024-02-05T08:05:59.718966+0000","last_undegraded":"2024-02-05T08:05:59.718966+0000","last_fullsized":"2024-02-05T08:05:59.718966+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:53.741059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.741059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:13.753256+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":112,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720616+0000","last_change":"2024-02-05T08:05:03.797738+0000","last_active":"2024-02-05T08:05:59.720616+0000","last_peered":"2024-02-05T08:05:59.720616+0000","last_clean":"2024-02-05T08:05:59.720616+0000","last_became_active":"2024-02-05T08:05:03.786876+0000","last_became_peered":"2024-02-05T08:05:03.786876+0000","last_unstale":"2024-02-05T08:05:59.720616+0000","last_undegraded":"2024-02-05T08:05:59.720616+0000","last_fullsized":"2024-02-05T08:05:59.720616+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:09.572379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:12.271918+0000","last_change":"2024-02-05T08:06:12.271939+0000","last_active":"2024-02-05T08:06:12.271918+0000","last_peered":"2024-02-05T08:06:12.271918+0000","last_clean":"2024-02-05T08:06:12.271918+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:06:12.271918+0000","last_undegraded":"2024-02-05T08:06:12.271918+0000","last_fullsized":"2024-02-05T08:06:12.271918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:12.271856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:12.271856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:31.305936+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.369671+0000","last_change":"2024-02-05T08:06:06.369693+0000","last_active":"2024-02-05T08:06:06.369671+0000","last_peered":"2024-02-05T08:06:06.369671+0000","last_clean":"2024-02-05T08:06:06.369671+0000","last_became_active":"2024-02-05T08:04:45.567341+0000","last_became_peered":"2024-02-05T08:04:45.567341+0000","last_unstale":"2024-02-05T08:06:06.369671+0000","last_undegraded":"2024-02-05T08:06:06.369671+0000","last_fullsized":"2024-02-05T08:06:06.369671+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:06.369612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:06.369612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:31.000287+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268978+0000","last_change":"2024-02-05T08:04:41.540429+0000","last_active":"2024-02-05T08:06:00.268978+0000","last_peered":"2024-02-05T08:06:00.268978+0000","last_clean":"2024-02-05T08:06:00.268978+0000","last_became_active":"2024-02-05T08:04:41.540116+0000","last_became_peered":"2024-02-05T08:04:41.540116+0000","last_unstale":"2024-02-05T08:06:00.268978+0000","last_undegraded":"2024-02-05T08:06:00.268978+0000","last_fullsized":"2024-02-05T08:06:00.268978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.767296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:04.215819+0000","last_change":"2024-02-05T08:06:04.215847+0000","last_active":"2024-02-05T08:06:04.215819+0000","last_peered":"2024-02-05T08:06:04.215819+0000","last_clean":"2024-02-05T08:06:04.215819+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:06:04.215819+0000","last_undegraded":"2024-02-05T08:06:04.215819+0000","last_fullsized":"2024-02-05T08:06:04.215819+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:06:04.215758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:04.215758+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:29.117755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225432+0000","last_change":"2024-02-05T08:04:47.588669+0000","last_active":"2024-02-05T08:06:00.225432+0000","last_peered":"2024-02-05T08:06:00.225432+0000","last_clean":"2024-02-05T08:06:00.225432+0000","last_became_active":"2024-02-05T08:04:47.588395+0000","last_became_peered":"2024-02-05T08:04:47.588395+0000","last_unstale":"2024-02-05T08:06:00.225432+0000","last_undegraded":"2024-02-05T08:06:00.225432+0000","last_fullsized":"2024-02-05T08:06:00.225432+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:16.402122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268703+0000","last_change":"2024-02-05T08:05:52.503758+0000","last_active":"2024-02-05T08:06:00.268703+0000","last_peered":"2024-02-05T08:06:00.268703+0000","last_clean":"2024-02-05T08:06:00.268703+0000","last_became_active":"2024-02-05T08:04:40.527674+0000","last_became_peered":"2024-02-05T08:04:40.527674+0000","last_unstale":"2024-02-05T08:06:00.268703+0000","last_undegraded":"2024-02-05T08:06:00.268703+0000","last_fullsized":"2024-02-05T08:06:00.268703+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.503651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.821538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:10.946963+0000","last_change":"2024-02-05T08:06:10.946983+0000","last_active":"2024-02-05T08:06:10.946963+0000","last_peered":"2024-02-05T08:06:10.946963+0000","last_clean":"2024-02-05T08:06:10.946963+0000","last_became_active":"2024-02-05T08:04:41.537478+0000","last_became_peered":"2024-02-05T08:04:41.537478+0000","last_unstale":"2024-02-05T08:06:10.946963+0000","last_undegraded":"2024-02-05T08:06:10.946963+0000","last_fullsized":"2024-02-05T08:06:10.946963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:06:10.946904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.448870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"85'38","reported_seq":140,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.601729+0000","last_change":"2024-02-05T08:05:43.858932+0000","last_active":"2024-02-05T08:06:06.601729+0000","last_peered":"2024-02-05T08:06:06.601729+0000","last_clean":"2024-02-05T08:06:06.601729+0000","last_became_active":"2024-02-05T08:04:43.554536+0000","last_became_peered":"2024-02-05T08:04:43.554536+0000","last_unstale":"2024-02-05T08:06:06.601729+0000","last_undegraded":"2024-02-05T08:06:06.601729+0000","last_fullsized":"2024-02-05T08:06:06.601729+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:05:43.858841+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:54.322675+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527257+0000","last_change":"2024-02-05T08:04:45.562140+0000","last_active":"2024-02-05T08:05:59.527257+0000","last_peered":"2024-02-05T08:05:59.527257+0000","last_clean":"2024-02-05T08:05:59.527257+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:05:59.527257+0000","last_undegraded":"2024-02-05T08:05:59.527257+0000","last_fullsized":"2024-02-05T08:05:59.527257+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:14.185461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.719032+0000","last_change":"2024-02-05T08:04:48.170226+0000","last_active":"2024-02-05T08:05:59.719032+0000","last_peered":"2024-02-05T08:05:59.719032+0000","last_clean":"2024-02-05T08:05:59.719032+0000","last_became_active":"2024-02-05T08:04:48.169894+0000","last_became_peered":"2024-02-05T08:04:48.169894+0000","last_unstale":"2024-02-05T08:05:59.719032+0000","last_undegraded":"2024-02-05T08:05:59.719032+0000","last_fullsized":"2024-02-05T08:05:59.719032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:15.026925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"67'1","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.526644+0000","last_change":"2024-02-05T08:04:41.536875+0000","last_active":"2024-02-05T08:05:59.526644+0000","last_peered":"2024-02-05T08:05:59.526644+0000","last_clean":"2024-02-05T08:05:59.526644+0000","last_became_active":"2024-02-05T08:04:41.536585+0000","last_became_peered":"2024-02-05T08:04:41.536585+0000","last_unstale":"2024-02-05T08:05:59.526644+0000","last_undegraded":"2024-02-05T08:05:59.526644+0000","last_fullsized":"2024-02-05T08:05:59.526644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.042189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.545805+0000","last_change":"2024-02-05T08:05:41.807274+0000","last_active":"2024-02-05T08:06:00.545805+0000","last_peered":"2024-02-05T08:06:00.545805+0000","last_clean":"2024-02-05T08:06:00.545805+0000","last_became_active":"2024-02-05T08:04:40.529073+0000","last_became_peered":"2024-02-05T08:04:40.529073+0000","last_unstale":"2024-02-05T08:06:00.545805+0000","last_undegraded":"2024-02-05T08:06:00.545805+0000","last_fullsized":"2024-02-05T08:06:00.545805+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:41.807173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:05:41.807173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:48.748767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546072+0000","last_change":"2024-02-05T08:05:45.827248+0000","last_active":"2024-02-05T08:06:00.546072+0000","last_peered":"2024-02-05T08:06:00.546072+0000","last_clean":"2024-02-05T08:06:00.546072+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:06:00.546072+0000","last_undegraded":"2024-02-05T08:06:00.546072+0000","last_fullsized":"2024-02-05T08:06:00.546072+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:05:45.827140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:45.827140+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:14.695181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546608+0000","last_change":"2024-02-05T08:05:58.550125+0000","last_active":"2024-02-05T08:06:00.546608+0000","last_peered":"2024-02-05T08:06:00.546608+0000","last_clean":"2024-02-05T08:06:00.546608+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:06:00.546608+0000","last_undegraded":"2024-02-05T08:06:00.546608+0000","last_fullsized":"2024-02-05T08:06:00.546608+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:58.550049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.550049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.897417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268113+0000","last_change":"2024-02-05T08:04:47.586644+0000","last_active":"2024-02-05T08:06:00.268113+0000","last_peered":"2024-02-05T08:06:00.268113+0000","last_clean":"2024-02-05T08:06:00.268113+0000","last_became_active":"2024-02-05T08:04:47.586358+0000","last_became_peered":"2024-02-05T08:04:47.586358+0000","last_unstale":"2024-02-05T08:06:00.268113+0000","last_undegraded":"2024-02-05T08:06:00.268113+0000","last_fullsized":"2024-02-05T08:06:00.268113+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:13.569831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":316,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718051+0000","last_change":"2024-02-05T08:05:16.657716+0000","last_active":"2024-02-05T08:05:59.718051+0000","last_peered":"2024-02-05T08:05:59.718051+0000","last_clean":"2024-02-05T08:05:59.718051+0000","last_became_active":"2024-02-05T08:04:30.515158+0000","last_became_peered":"2024-02-05T08:04:30.515158+0000","last_unstale":"2024-02-05T08:05:59.718051+0000","last_undegraded":"2024-02-05T08:05:59.718051+0000","last_fullsized":"2024-02-05T08:05:59.718051+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'119","last_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:05:16.657600+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:37.807886+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226170+0000","last_change":"2024-02-05T08:04:41.538282+0000","last_active":"2024-02-05T08:06:00.226170+0000","last_peered":"2024-02-05T08:06:00.226170+0000","last_clean":"2024-02-05T08:06:00.226170+0000","last_became_active":"2024-02-05T08:04:41.538002+0000","last_became_peered":"2024-02-05T08:04:41.538002+0000","last_unstale":"2024-02-05T08:06:00.226170+0000","last_undegraded":"2024-02-05T08:06:00.226170+0000","last_fullsized":"2024-02-05T08:06:00.226170+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.327424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":104,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.601875+0000","last_change":"2024-02-05T08:06:00.304976+0000","last_active":"2024-02-05T08:06:06.601875+0000","last_peered":"2024-02-05T08:06:06.601875+0000","last_clean":"2024-02-05T08:06:06.601875+0000","last_became_active":"2024-02-05T08:04:43.606401+0000","last_became_peered":"2024-02-05T08:04:43.606401+0000","last_unstale":"2024-02-05T08:06:06.601875+0000","last_undegraded":"2024-02-05T08:06:06.601875+0000","last_fullsized":"2024-02-05T08:06:06.601875+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-02-05T08:06:00.304861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:00.304861+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:15.661923+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718680+0000","last_change":"2024-02-05T08:04:45.565211+0000","last_active":"2024-02-05T08:05:59.718680+0000","last_peered":"2024-02-05T08:05:59.718680+0000","last_clean":"2024-02-05T08:05:59.718680+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:05:59.718680+0000","last_undegraded":"2024-02-05T08:05:59.718680+0000","last_fullsized":"2024-02-05T08:05:59.718680+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:10.360421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527679+0000","last_change":"2024-02-05T08:04:47.587068+0000","last_active":"2024-02-05T08:05:59.527679+0000","last_peered":"2024-02-05T08:05:59.527679+0000","last_clean":"2024-02-05T08:05:59.527679+0000","last_became_active":"2024-02-05T08:04:47.586774+0000","last_became_peered":"2024-02-05T08:04:47.586774+0000","last_unstale":"2024-02-05T08:05:59.527679+0000","last_undegraded":"2024-02-05T08:05:59.527679+0000","last_fullsized":"2024-02-05T08:05:59.527679+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:00.755929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527011+0000","last_change":"2024-02-05T08:04:41.538403+0000","last_active":"2024-02-05T08:05:59.527011+0000","last_peered":"2024-02-05T08:05:59.527011+0000","last_clean":"2024-02-05T08:05:59.527011+0000","last_became_active":"2024-02-05T08:04:41.537872+0000","last_became_peered":"2024-02-05T08:04:41.537872+0000","last_unstale":"2024-02-05T08:05:59.527011+0000","last_undegraded":"2024-02-05T08:05:59.527011+0000","last_fullsized":"2024-02-05T08:05:59.527011+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.800257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":164,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:10.918680+0000","last_change":"2024-02-05T08:05:43.746185+0000","last_active":"2024-02-05T08:06:10.918680+0000","last_peered":"2024-02-05T08:06:10.918680+0000","last_clean":"2024-02-05T08:06:10.918680+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:06:10.918680+0000","last_undegraded":"2024-02-05T08:06:10.918680+0000","last_fullsized":"2024-02-05T08:06:10.918680+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:05:43.746095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:05:43.746095+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:55.186390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":80,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528230+0000","last_change":"2024-02-05T08:04:43.550001+0000","last_active":"2024-02-05T08:05:59.528230+0000","last_peered":"2024-02-05T08:05:59.528230+0000","last_clean":"2024-02-05T08:05:59.528230+0000","last_became_active":"2024-02-05T08:04:43.549687+0000","last_became_peered":"2024-02-05T08:04:43.549687+0000","last_unstale":"2024-02-05T08:05:59.528230+0000","last_undegraded":"2024-02-05T08:05:59.528230+0000","last_fullsized":"2024-02-05T08:05:59.528230+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.794328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269362+0000","last_change":"2024-02-05T08:04:45.559137+0000","last_active":"2024-02-05T08:06:00.269362+0000","last_peered":"2024-02-05T08:06:00.269362+0000","last_clean":"2024-02-05T08:06:00.269362+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:06:00.269362+0000","last_undegraded":"2024-02-05T08:06:00.269362+0000","last_fullsized":"2024-02-05T08:06:00.269362+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:12.171151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527562+0000","last_change":"2024-02-05T08:04:48.170100+0000","last_active":"2024-02-05T08:05:59.527562+0000","last_peered":"2024-02-05T08:05:59.527562+0000","last_clean":"2024-02-05T08:05:59.527562+0000","last_became_active":"2024-02-05T08:04:48.169777+0000","last_became_peered":"2024-02-05T08:04:48.169777+0000","last_unstale":"2024-02-05T08:05:59.527562+0000","last_undegraded":"2024-02-05T08:05:59.527562+0000","last_fullsized":"2024-02-05T08:05:59.527562+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527843+0000","last_change":"2024-02-05T08:05:51.880952+0000","last_active":"2024-02-05T08:05:59.527843+0000","last_peered":"2024-02-05T08:05:59.527843+0000","last_clean":"2024-02-05T08:05:59.527843+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:05:59.527843+0000","last_undegraded":"2024-02-05T08:05:59.527843+0000","last_fullsized":"2024-02-05T08:05:59.527843+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:51.880864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:51.880864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:53.033627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":98,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528739+0000","last_change":"2024-02-05T08:05:55.329780+0000","last_active":"2024-02-05T08:05:59.528739+0000","last_peered":"2024-02-05T08:05:59.528739+0000","last_clean":"2024-02-05T08:05:59.528739+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:05:59.528739+0000","last_undegraded":"2024-02-05T08:05:59.528739+0000","last_fullsized":"2024-02-05T08:05:59.528739+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:05:55.329677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.329677+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:07.966593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720746+0000","last_change":"2024-02-05T08:04:45.565899+0000","last_active":"2024-02-05T08:05:59.720746+0000","last_peered":"2024-02-05T08:05:59.720746+0000","last_clean":"2024-02-05T08:05:59.720746+0000","last_became_active":"2024-02-05T08:04:45.565618+0000","last_became_peered":"2024-02-05T08:04:45.565618+0000","last_unstale":"2024-02-05T08:05:59.720746+0000","last_undegraded":"2024-02-05T08:05:59.720746+0000","last_fullsized":"2024-02-05T08:05:59.720746+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.936746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528338+0000","last_change":"2024-02-05T08:05:58.215810+0000","last_active":"2024-02-05T08:05:59.528338+0000","last_peered":"2024-02-05T08:05:59.528338+0000","last_clean":"2024-02-05T08:05:59.528338+0000","last_became_active":"2024-02-05T08:04:47.583866+0000","last_became_peered":"2024-02-05T08:04:47.583866+0000","last_unstale":"2024-02-05T08:05:59.528338+0000","last_undegraded":"2024-02-05T08:05:59.528338+0000","last_fullsized":"2024-02-05T08:05:59.528338+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:58.215737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.215737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.871723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269169+0000","last_change":"2024-02-05T08:04:41.538084+0000","last_active":"2024-02-05T08:06:00.269169+0000","last_peered":"2024-02-05T08:06:00.269169+0000","last_clean":"2024-02-05T08:06:00.269169+0000","last_became_active":"2024-02-05T08:04:41.537818+0000","last_became_peered":"2024-02-05T08:04:41.537818+0000","last_unstale":"2024-02-05T08:06:00.269169+0000","last_undegraded":"2024-02-05T08:06:00.269169+0000","last_fullsized":"2024-02-05T08:06:00.269169+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.324263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527293+0000","last_change":"2024-02-05T08:04:43.552388+0000","last_active":"2024-02-05T08:05:59.527293+0000","last_peered":"2024-02-05T08:05:59.527293+0000","last_clean":"2024-02-05T08:05:59.527293+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:05:59.527293+0000","last_undegraded":"2024-02-05T08:05:59.527293+0000","last_fullsized":"2024-02-05T08:05:59.527293+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:05.174347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":85,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.601747+0000","last_change":"2024-02-05T08:05:57.266029+0000","last_active":"2024-02-05T08:06:06.601747+0000","last_peered":"2024-02-05T08:06:06.601747+0000","last_clean":"2024-02-05T08:06:06.601747+0000","last_became_active":"2024-02-05T08:04:45.562462+0000","last_became_peered":"2024-02-05T08:04:45.562462+0000","last_unstale":"2024-02-05T08:06:06.601747+0000","last_undegraded":"2024-02-05T08:06:06.601747+0000","last_fullsized":"2024-02-05T08:06:06.601747+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:05:57.265939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.265939+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:03.556401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718573+0000","last_change":"2024-02-05T08:04:48.172841+0000","last_active":"2024-02-05T08:05:59.718573+0000","last_peered":"2024-02-05T08:05:59.718573+0000","last_clean":"2024-02-05T08:05:59.718573+0000","last_became_active":"2024-02-05T08:04:48.172564+0000","last_became_peered":"2024-02-05T08:04:48.172564+0000","last_unstale":"2024-02-05T08:05:59.718573+0000","last_undegraded":"2024-02-05T08:05:59.718573+0000","last_fullsized":"2024-02-05T08:05:59.718573+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:15.681030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528317+0000","last_change":"2024-02-05T08:05:52.283473+0000","last_active":"2024-02-05T08:05:59.528317+0000","last_peered":"2024-02-05T08:05:59.528317+0000","last_clean":"2024-02-05T08:05:59.528317+0000","last_became_active":"2024-02-05T08:04:41.534433+0000","last_became_peered":"2024-02-05T08:04:41.534433+0000","last_unstale":"2024-02-05T08:05:59.528317+0000","last_undegraded":"2024-02-05T08:05:59.528317+0000","last_fullsized":"2024-02-05T08:05:59.528317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:05:52.283383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.283383+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:05.067021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529862+0000","last_change":"2024-02-05T08:04:43.551586+0000","last_active":"2024-02-05T08:05:59.529862+0000","last_peered":"2024-02-05T08:05:59.529862+0000","last_clean":"2024-02-05T08:05:59.529862+0000","last_became_active":"2024-02-05T08:04:43.551184+0000","last_became_peered":"2024-02-05T08:04:43.551184+0000","last_unstale":"2024-02-05T08:05:59.529862+0000","last_undegraded":"2024-02-05T08:05:59.529862+0000","last_fullsized":"2024-02-05T08:05:59.529862+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:05.037825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.601729+0000","last_change":"2024-02-05T08:04:45.563011+0000","last_active":"2024-02-05T08:06:06.601729+0000","last_peered":"2024-02-05T08:06:06.601729+0000","last_clean":"2024-02-05T08:06:06.601729+0000","last_became_active":"2024-02-05T08:04:45.562597+0000","last_became_peered":"2024-02-05T08:04:45.562597+0000","last_unstale":"2024-02-05T08:06:06.601729+0000","last_undegraded":"2024-02-05T08:06:06.601729+0000","last_fullsized":"2024-02-05T08:06:06.601729+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.186855+0000","last_change":"2024-02-05T08:06:01.186878+0000","last_active":"2024-02-05T08:06:01.186855+0000","last_peered":"2024-02-05T08:06:01.186855+0000","last_clean":"2024-02-05T08:06:01.186855+0000","last_became_active":"2024-02-05T08:04:47.582776+0000","last_became_peered":"2024-02-05T08:04:47.582776+0000","last_unstale":"2024-02-05T08:06:01.186855+0000","last_undegraded":"2024-02-05T08:06:01.186855+0000","last_fullsized":"2024-02-05T08:06:01.186855+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:01.186782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:01.186782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.355302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226658+0000","last_change":"2024-02-05T08:05:48.267727+0000","last_active":"2024-02-05T08:06:00.226658+0000","last_peered":"2024-02-05T08:06:00.226658+0000","last_clean":"2024-02-05T08:06:00.226658+0000","last_became_active":"2024-02-05T08:04:41.540818+0000","last_became_peered":"2024-02-05T08:04:41.540818+0000","last_unstale":"2024-02-05T08:06:00.226658+0000","last_undegraded":"2024-02-05T08:06:00.226658+0000","last_fullsized":"2024-02-05T08:06:00.226658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:05:48.267582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.267582+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:15.800342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225869+0000","last_change":"2024-02-05T08:05:52.195604+0000","last_active":"2024-02-05T08:06:00.225869+0000","last_peered":"2024-02-05T08:06:00.225869+0000","last_clean":"2024-02-05T08:06:00.225869+0000","last_became_active":"2024-02-05T08:04:43.607546+0000","last_became_peered":"2024-02-05T08:04:43.607546+0000","last_unstale":"2024-02-05T08:06:00.225869+0000","last_undegraded":"2024-02-05T08:06:00.225869+0000","last_fullsized":"2024-02-05T08:06:00.225869+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-05T08:05:52.195494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.195494+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:16.333300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529710+0000","last_change":"2024-02-05T08:04:45.561907+0000","last_active":"2024-02-05T08:05:59.529710+0000","last_peered":"2024-02-05T08:05:59.529710+0000","last_clean":"2024-02-05T08:05:59.529710+0000","last_became_active":"2024-02-05T08:04:45.561639+0000","last_became_peered":"2024-02-05T08:04:45.561639+0000","last_unstale":"2024-02-05T08:05:59.529710+0000","last_undegraded":"2024-02-05T08:05:59.529710+0000","last_fullsized":"2024-02-05T08:05:59.529710+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.611331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268056+0000","last_change":"2024-02-05T08:05:48.449994+0000","last_active":"2024-02-05T08:06:00.268056+0000","last_peered":"2024-02-05T08:06:00.268056+0000","last_clean":"2024-02-05T08:06:00.268056+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:06:00.268056+0000","last_undegraded":"2024-02-05T08:06:00.268056+0000","last_fullsized":"2024-02-05T08:06:00.268056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:48.449900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.449900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:55.343526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.601858+0000","last_change":"2024-02-05T08:04:45.567294+0000","last_active":"2024-02-05T08:06:06.601858+0000","last_peered":"2024-02-05T08:06:06.601858+0000","last_clean":"2024-02-05T08:06:06.601858+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:06:06.601858+0000","last_undegraded":"2024-02-05T08:06:06.601858+0000","last_fullsized":"2024-02-05T08:06:06.601858+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225833+0000","last_change":"2024-02-05T08:05:59.261588+0000","last_active":"2024-02-05T08:06:00.225833+0000","last_peered":"2024-02-05T08:06:00.225833+0000","last_clean":"2024-02-05T08:06:00.225833+0000","last_became_active":"2024-02-05T08:04:41.541327+0000","last_became_peered":"2024-02-05T08:04:41.541327+0000","last_unstale":"2024-02-05T08:06:00.225833+0000","last_undegraded":"2024-02-05T08:06:00.225833+0000","last_fullsized":"2024-02-05T08:06:00.225833+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:59.261506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:59.261506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:24.020924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.526944+0000","last_change":"2024-02-05T08:05:53.816142+0000","last_active":"2024-02-05T08:05:59.526944+0000","last_peered":"2024-02-05T08:05:59.526944+0000","last_clean":"2024-02-05T08:05:59.526944+0000","last_became_active":"2024-02-05T08:04:43.555028+0000","last_became_peered":"2024-02-05T08:04:43.555028+0000","last_unstale":"2024-02-05T08:05:59.526944+0000","last_undegraded":"2024-02-05T08:05:59.526944+0000","last_fullsized":"2024-02-05T08:05:59.526944+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:05:53.815997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.815997+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.642275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225515+0000","last_change":"2024-02-05T08:04:48.170254+0000","last_active":"2024-02-05T08:06:00.225515+0000","last_peered":"2024-02-05T08:06:00.225515+0000","last_clean":"2024-02-05T08:06:00.225515+0000","last_became_active":"2024-02-05T08:04:48.169762+0000","last_became_peered":"2024-02-05T08:04:48.169762+0000","last_unstale":"2024-02-05T08:06:00.225515+0000","last_undegraded":"2024-02-05T08:06:00.225515+0000","last_fullsized":"2024-02-05T08:06:00.225515+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:13.151401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.561022+0000","last_change":"2024-02-05T08:06:01.561064+0000","last_active":"2024-02-05T08:06:01.561022+0000","last_peered":"2024-02-05T08:06:01.561022+0000","last_clean":"2024-02-05T08:06:01.561022+0000","last_became_active":"2024-02-05T08:04:45.561679+0000","last_became_peered":"2024-02-05T08:04:45.561679+0000","last_unstale":"2024-02-05T08:06:01.561022+0000","last_undegraded":"2024-02-05T08:06:01.561022+0000","last_fullsized":"2024-02-05T08:06:01.561022+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:01.560919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:01.560919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:21.298440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529733+0000","last_change":"2024-02-05T08:05:48.116107+0000","last_active":"2024-02-05T08:05:59.529733+0000","last_peered":"2024-02-05T08:05:59.529733+0000","last_clean":"2024-02-05T08:05:59.529733+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:05:59.529733+0000","last_undegraded":"2024-02-05T08:05:59.529733+0000","last_fullsized":"2024-02-05T08:05:59.529733+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:48.116008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.116008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:06.262952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:09.984233+0000","last_change":"2024-02-05T08:06:09.984289+0000","last_active":"2024-02-05T08:06:09.984233+0000","last_peered":"2024-02-05T08:06:09.984233+0000","last_clean":"2024-02-05T08:06:09.984233+0000","last_became_active":"2024-02-05T08:04:43.554161+0000","last_became_peered":"2024-02-05T08:04:43.554161+0000","last_unstale":"2024-02-05T08:06:09.984233+0000","last_undegraded":"2024-02-05T08:06:09.984233+0000","last_fullsized":"2024-02-05T08:06:09.984233+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:06:09.984141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:09.984141+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.006301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.545953+0000","last_change":"2024-02-05T08:05:48.720004+0000","last_active":"2024-02-05T08:06:00.545953+0000","last_peered":"2024-02-05T08:06:00.545953+0000","last_clean":"2024-02-05T08:06:00.545953+0000","last_became_active":"2024-02-05T08:04:47.585962+0000","last_became_peered":"2024-02-05T08:04:47.585962+0000","last_unstale":"2024-02-05T08:06:00.545953+0000","last_undegraded":"2024-02-05T08:06:00.545953+0000","last_fullsized":"2024-02-05T08:06:00.545953+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:05:48.719923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.719923+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:54.016895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528031+0000","last_change":"2024-02-05T08:04:45.559587+0000","last_active":"2024-02-05T08:05:59.528031+0000","last_peered":"2024-02-05T08:05:59.528031+0000","last_clean":"2024-02-05T08:05:59.528031+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:05:59.528031+0000","last_undegraded":"2024-02-05T08:05:59.528031+0000","last_fullsized":"2024-02-05T08:05:59.528031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:02.577883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.721051+0000","last_change":"2024-02-05T08:05:50.230381+0000","last_active":"2024-02-05T08:05:59.721051+0000","last_peered":"2024-02-05T08:05:59.721051+0000","last_clean":"2024-02-05T08:05:59.721051+0000","last_became_active":"2024-02-05T08:04:41.538397+0000","last_became_peered":"2024-02-05T08:04:41.538397+0000","last_unstale":"2024-02-05T08:05:59.721051+0000","last_undegraded":"2024-02-05T08:05:59.721051+0000","last_fullsized":"2024-02-05T08:05:59.721051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:50.230263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:50.230263+0000","last_clean_scrub_stamp":"2024-02-05T08:05:50.230263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:59.311643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530628+0000","last_change":"2024-02-05T08:04:43.555191+0000","last_active":"2024-02-05T08:05:59.530628+0000","last_peered":"2024-02-05T08:05:59.530628+0000","last_clean":"2024-02-05T08:05:59.530628+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:05:59.530628+0000","last_undegraded":"2024-02-05T08:05:59.530628+0000","last_fullsized":"2024-02-05T08:05:59.530628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:12.495416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530787+0000","last_change":"2024-02-05T08:04:47.573584+0000","last_active":"2024-02-05T08:05:59.530787+0000","last_peered":"2024-02-05T08:05:59.530787+0000","last_clean":"2024-02-05T08:05:59.530787+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:05:59.530787+0000","last_undegraded":"2024-02-05T08:05:59.530787+0000","last_fullsized":"2024-02-05T08:05:59.530787+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.290723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.601739+0000","last_change":"2024-02-05T08:06:04.443720+0000","last_active":"2024-02-05T08:06:06.601739+0000","last_peered":"2024-02-05T08:06:06.601739+0000","last_clean":"2024-02-05T08:06:06.601739+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:06:06.601739+0000","last_undegraded":"2024-02-05T08:06:06.601739+0000","last_fullsized":"2024-02-05T08:06:06.601739+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:06:04.443624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:04.443624+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:19.854071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226287+0000","last_change":"2024-02-05T08:04:41.534743+0000","last_active":"2024-02-05T08:06:00.226287+0000","last_peered":"2024-02-05T08:06:00.226287+0000","last_clean":"2024-02-05T08:06:00.226287+0000","last_became_active":"2024-02-05T08:04:41.533838+0000","last_became_peered":"2024-02-05T08:04:41.533838+0000","last_unstale":"2024-02-05T08:06:00.226287+0000","last_undegraded":"2024-02-05T08:06:00.226287+0000","last_fullsized":"2024-02-05T08:06:00.226287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.461601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.080113+0000","last_change":"2024-02-05T08:06:06.080146+0000","last_active":"2024-02-05T08:06:06.080113+0000","last_peered":"2024-02-05T08:06:06.080113+0000","last_clean":"2024-02-05T08:06:06.080113+0000","last_became_active":"2024-02-05T08:04:43.559922+0000","last_became_peered":"2024-02-05T08:04:43.559922+0000","last_unstale":"2024-02-05T08:06:06.080113+0000","last_undegraded":"2024-02-05T08:06:06.080113+0000","last_fullsized":"2024-02-05T08:06:06.080113+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:06:06.080040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:06.080040+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:25.229905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":412,"ondisk_log_size":412,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":80,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037783,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32656,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712496,"statfs":{"total":95995035648,"available":95961595904,"internally_reserved":0,"allocated":1515520,"data_stored":1027404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.442},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.48699999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.55200000000000005}]},{"osd":2,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.51400000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.53800000000000003}]},{"osd":4,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.42699999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.502}]},{"osd":6,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.56299999999999994}]}]},{"osd":6,"up_from":51,"seq":219043332125,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1495040,"data_stored":1025877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.74299999999999999,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.74199999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.53300000000000003}]},{"osd":1,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60399999999999998,"15min":0.60099999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.97099999999999997,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40500000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80600000000000005,"5min":0.871,"15min":0.871},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.78900000000000003,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51400000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.436},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.48199999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.44400000000000001,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.44400000000000001},"min":{"1min":0.313,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.70499999999999996,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.39200000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.373,"5min":0.309,"15min":0.309},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.47099999999999997,"15min":0.46300000000000002},"min":{"1min":0.35199999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73699999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.503}]},{"osd":7,"last update":"Mon Feb 5 08:05:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.41999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":436744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34399999999999997,"15min":0.31},"min":{"1min":0.216,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.37,"15min":0.32500000000000001},"min":{"1min":0.22900000000000001,"5min":0.16,"15min":0.16},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.19500000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.63700000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.41899999999999998,"15min":0.40500000000000003},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.28599999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:05:54 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.434,"15min":0.441},"min":{"1min":0.27200000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.24199999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.41399999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.86199999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.375},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.45000000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.74399999999999999,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.38600000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.44400000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:06:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82299999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.45200000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.42299999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.436}]}]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1519616,"data_stored":1027147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.36199999999999999,"15min":0.33400000000000002},"min":{"1min":0.25,"5min":0.152,"15min":0.152},"max":{"1min":0.501,"5min":0.621,"15min":0.621},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.37,"15min":0.33600000000000002},"min":{"1min":0.26500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.28000000000000003}]},{"osd":2,"last update":"Mon Feb 5 08:05:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38500000000000001,"15min":0.372},"min":{"1min":0.252,"5min":0.218,"15min":0.218},"max":{"1min":0.60899999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40300000000000002,"15min":0.38900000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.34000000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.45200000000000001,"15min":0.45900000000000002},"min":{"1min":0.29599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71899999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.46300000000000002,"15min":0.46800000000000003},"min":{"1min":0.32800000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.58199999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.30199999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:05:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56799999999999995,"15min":0.56599999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.79300000000000004,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.52800000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.39800000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60499999999999998,"15min":0.60599999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.375},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.437}]},{"osd":6,"last update":"Mon Feb 5 08:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.621,"15min":0.621},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.40699999999999997}]},{"osd":7,"last update":"Mon Feb 5 08:05:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.51400000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":436186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:24 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38,"15min":0.37},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.57099999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.38200000000000001},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56899999999999995,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.40100000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:05:24 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.36799999999999999,"15min":0.35899999999999999},"min":{"1min":0.248,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.495,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.40200000000000002,"15min":0.38},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.623,"5min":0.63,"15min":0.63},"last":0.27200000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:05:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46999999999999997,"15min":0.46700000000000003},"min":{"1min":0.34499999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.58699999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56599999999999995,"5min":0.746,"15min":0.746},"last":0.245}]},{"osd":4,"last update":"Mon Feb 5 08:05:12 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.371,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83299999999999996,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.374},{"interface":"front","average":{"1min":0.62,"5min":0.58599999999999997,"15min":0.57999999999999996},"min":{"1min":0.45500000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85699999999999998,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.43099999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:05:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.47699999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63400000000000001,"15min":0.63600000000000001},"min":{"1min":0.435,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80600000000000005,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.5},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63500000000000001,"15min":0.63900000000000001},"min":{"1min":0.42299999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83899999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.36099999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.54400000000000004}]}]},{"osd":3,"up_from":27,"seq":115964117036,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":436889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.31},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.438,"15min":0.45100000000000001},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.35499999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.46800000000000003,"15min":0.48199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70299999999999996,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.45000000000000001,"15min":0.46200000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.47599999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49099999999999999,"15min":0.497},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.373},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47499999999999998,"15min":0.48399999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.439},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.42699999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.46400000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.39600000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.47599999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58499999999999996,"15min":0.58099999999999996},"min":{"1min":0.48899999999999999,"5min":0.438,"15min":0.438},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.44900000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:05:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.51200000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953511,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":435729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.748,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53700000000000003,"15min":0.54400000000000004},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.45300000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.38400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.38400000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.40699999999999997,"5min":0.376,"15min":0.376},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:05:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.38800000000000001,"15min":0.372},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.41299999999999998,"15min":0.39900000000000002},"min":{"1min":0.29099999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.51000000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.311,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69899999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.443,"15min":0.42499999999999999},"min":{"1min":0.35199999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.441}]},{"osd":7,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.496},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.59199999999999997}]}]},{"osd":5,"up_from":42,"seq":180388626465,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":901120,"data_stored":435436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.33100000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.72899999999999998,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.53200000000000003,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71299999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.498}]},{"osd":1,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.374,"15min":0.374},"max":{"1min":0.69999999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.35299999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60999999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56599999999999995,"15min":0.57099999999999995},"min":{"1min":0.34899999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76200000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.373,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.64000000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55600000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.749,"15min":0.749},"last":0.626},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.54500000000000004}]},{"osd":4,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.434,"5min":0.41999999999999998,"15min":0.41799999999999998},"min":{"1min":0.28199999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60599999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.47499999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45400000000000001,"15min":0.44600000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.42299999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56200000000000006}]},{"osd":7,"last update":"Mon Feb 5 08:05:30 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.439}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:06:18.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:06:18.497 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:18.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:18 smithi134 ceph-mon[39278]: 3.11 scrub starts 2024-02-05T08:06:18.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:18 smithi134 ceph-mon[39278]: 3.11 scrub ok 2024-02-05T08:06:18.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:18 smithi134 ceph-mon[39278]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:06:18.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:18 smithi134 ceph-mon[39278]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:18.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:18 smithi134 ceph-mon[30086]: 3.11 scrub starts 2024-02-05T08:06:18.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:18 smithi134 ceph-mon[30086]: 3.11 scrub ok 2024-02-05T08:06:18.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:18 smithi134 ceph-mon[30086]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:06:18.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:18 smithi134 ceph-mon[30086]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:18 smithi196 ceph-mon[33018]: 3.11 scrub starts 2024-02-05T08:06:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:18 smithi196 ceph-mon[33018]: 3.11 scrub ok 2024-02-05T08:06:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:18 smithi196 ceph-mon[33018]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:06:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:18 smithi196 ceph-mon[33018]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:19.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:19 smithi134 ceph-mon[30086]: 5.4 scrub starts 2024-02-05T08:06:19.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:19 smithi134 ceph-mon[30086]: 5.4 scrub ok 2024-02-05T08:06:19.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:19 smithi134 ceph-mon[30086]: 6.1 scrub starts 2024-02-05T08:06:19.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:19 smithi134 ceph-mon[30086]: 6.1 scrub ok 2024-02-05T08:06:19.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:19 smithi134 ceph-mon[39278]: 5.4 scrub starts 2024-02-05T08:06:19.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:19 smithi134 ceph-mon[39278]: 5.4 scrub ok 2024-02-05T08:06:19.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:19 smithi134 ceph-mon[39278]: 6.1 scrub starts 2024-02-05T08:06:19.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:19 smithi134 ceph-mon[39278]: 6.1 scrub ok 2024-02-05T08:06:19.779 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:19 smithi196 ceph-mon[33018]: 5.4 scrub starts 2024-02-05T08:06:19.780 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:19 smithi196 ceph-mon[33018]: 5.4 scrub ok 2024-02-05T08:06:19.780 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:19 smithi196 ceph-mon[33018]: 6.1 scrub starts 2024-02-05T08:06:19.780 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:19 smithi196 ceph-mon[33018]: 6.1 scrub ok 2024-02-05T08:06:20.370 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:06:20.372 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:06:20.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:20 smithi134 ceph-mon[30086]: 4.8 scrub starts 2024-02-05T08:06:20.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:20 smithi134 ceph-mon[30086]: 4.8 scrub ok 2024-02-05T08:06:20.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:20 smithi134 ceph-mon[30086]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:20.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:20 smithi134 ceph-mon[30086]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:06:20.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:20 smithi134 ceph-mon[39278]: 4.8 scrub starts 2024-02-05T08:06:20.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:20 smithi134 ceph-mon[39278]: 4.8 scrub ok 2024-02-05T08:06:20.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:20 smithi134 ceph-mon[39278]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:20.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:20 smithi134 ceph-mon[39278]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:06:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:20 smithi196 ceph-mon[33018]: 4.8 scrub starts 2024-02-05T08:06:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:20 smithi196 ceph-mon[33018]: 4.8 scrub ok 2024-02-05T08:06:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:20 smithi196 ceph-mon[33018]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:20.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:20 smithi196 ceph-mon[33018]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:06:21.173 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-02-05T08:06:19.493585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1023,"num_read_kb":869,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":622,"ondisk_log_size":622,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":245884,"kb_used_data":8956,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749715332,"statfs":{"total":767960285184,"available":767708499968,"internally_reserved":0,"allocated":9170944,"data_stored":5261412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":242535987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003039"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529806+0000","last_change":"2024-02-05T08:05:55.114592+0000","last_active":"2024-02-05T08:05:59.529806+0000","last_peered":"2024-02-05T08:05:59.529806+0000","last_clean":"2024-02-05T08:05:59.529806+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:05:59.529806+0000","last_undegraded":"2024-02-05T08:05:59.529806+0000","last_fullsized":"2024-02-05T08:05:59.529806+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:55.114498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.114498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.383006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:15.730257+0000","last_change":"2024-02-05T08:06:15.730288+0000","last_active":"2024-02-05T08:06:15.730257+0000","last_peered":"2024-02-05T08:06:15.730257+0000","last_clean":"2024-02-05T08:06:15.730257+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:06:15.730257+0000","last_undegraded":"2024-02-05T08:06:15.730257+0000","last_fullsized":"2024-02-05T08:06:15.730257+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:15.730171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:15.730171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:18.523768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527961+0000","last_change":"2024-02-05T08:05:53.302720+0000","last_active":"2024-02-05T08:05:59.527961+0000","last_peered":"2024-02-05T08:05:59.527961+0000","last_clean":"2024-02-05T08:05:59.527961+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:05:59.527961+0000","last_undegraded":"2024-02-05T08:05:59.527961+0000","last_fullsized":"2024-02-05T08:05:59.527961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:53.302642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.302642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:57.391075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"90'13","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529707+0000","last_change":"2024-02-05T08:05:58.103861+0000","last_active":"2024-02-05T08:05:59.529707+0000","last_peered":"2024-02-05T08:05:59.529707+0000","last_clean":"2024-02-05T08:05:59.529707+0000","last_became_active":"2024-02-05T08:04:43.551169+0000","last_became_peered":"2024-02-05T08:04:43.551169+0000","last_unstale":"2024-02-05T08:05:59.529707+0000","last_undegraded":"2024-02-05T08:05:59.529707+0000","last_fullsized":"2024-02-05T08:05:59.529707+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.103749+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:14.381872+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.227351+0000","last_change":"2024-02-05T08:06:00.227371+0000","last_active":"2024-02-05T08:06:00.227351+0000","last_peered":"2024-02-05T08:06:00.227351+0000","last_clean":"2024-02-05T08:06:00.227351+0000","last_became_active":"2024-02-05T08:04:48.172811+0000","last_became_peered":"2024-02-05T08:04:48.172811+0000","last_unstale":"2024-02-05T08:06:00.227351+0000","last_undegraded":"2024-02-05T08:06:00.227351+0000","last_fullsized":"2024-02-05T08:06:00.227351+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:00.227295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:00.227295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:06.714686+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529726+0000","last_change":"2024-02-05T08:04:45.564397+0000","last_active":"2024-02-05T08:05:59.529726+0000","last_peered":"2024-02-05T08:05:59.529726+0000","last_clean":"2024-02-05T08:05:59.529726+0000","last_became_active":"2024-02-05T08:04:45.563795+0000","last_became_peered":"2024-02-05T08:04:45.563795+0000","last_unstale":"2024-02-05T08:05:59.529726+0000","last_undegraded":"2024-02-05T08:05:59.529726+0000","last_fullsized":"2024-02-05T08:05:59.529726+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.204318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226552+0000","last_change":"2024-02-05T08:05:42.259337+0000","last_active":"2024-02-05T08:06:00.226552+0000","last_peered":"2024-02-05T08:06:00.226552+0000","last_clean":"2024-02-05T08:06:00.226552+0000","last_became_active":"2024-02-05T08:04:41.535435+0000","last_became_peered":"2024-02-05T08:04:41.535435+0000","last_unstale":"2024-02-05T08:06:00.226552+0000","last_undegraded":"2024-02-05T08:06:00.226552+0000","last_fullsized":"2024-02-05T08:06:00.226552+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:42.259257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:42.259257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:48.329795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:16.010525+0000","last_change":"2024-02-05T08:06:16.010552+0000","last_active":"2024-02-05T08:06:16.010525+0000","last_peered":"2024-02-05T08:06:16.010525+0000","last_clean":"2024-02-05T08:06:16.010525+0000","last_became_active":"2024-02-05T08:04:43.560660+0000","last_became_peered":"2024-02-05T08:04:43.560660+0000","last_unstale":"2024-02-05T08:06:16.010525+0000","last_undegraded":"2024-02-05T08:06:16.010525+0000","last_fullsized":"2024-02-05T08:06:16.010525+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:06:16.010469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:16.010469+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:28.967684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528432+0000","last_change":"2024-02-05T08:05:56.285954+0000","last_active":"2024-02-05T08:05:59.528432+0000","last_peered":"2024-02-05T08:05:59.528432+0000","last_clean":"2024-02-05T08:05:59.528432+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:05:59.528432+0000","last_undegraded":"2024-02-05T08:05:59.528432+0000","last_fullsized":"2024-02-05T08:05:59.528432+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:56.285876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:56.285876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:18.137707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268950+0000","last_change":"2024-02-05T08:04:45.560281+0000","last_active":"2024-02-05T08:06:00.268950+0000","last_peered":"2024-02-05T08:06:00.268950+0000","last_clean":"2024-02-05T08:06:00.268950+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:06:00.268950+0000","last_undegraded":"2024-02-05T08:06:00.268950+0000","last_fullsized":"2024-02-05T08:06:00.268950+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:14.202262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269001+0000","last_change":"2024-02-05T08:05:55.576535+0000","last_active":"2024-02-05T08:06:00.269001+0000","last_peered":"2024-02-05T08:06:00.269001+0000","last_clean":"2024-02-05T08:06:00.269001+0000","last_became_active":"2024-02-05T08:04:41.540764+0000","last_became_peered":"2024-02-05T08:04:41.540764+0000","last_unstale":"2024-02-05T08:06:00.269001+0000","last_undegraded":"2024-02-05T08:06:00.269001+0000","last_fullsized":"2024-02-05T08:06:00.269001+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:55.576441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.576441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.917805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529949+0000","last_change":"2024-02-05T08:04:43.557923+0000","last_active":"2024-02-05T08:05:59.529949+0000","last_peered":"2024-02-05T08:05:59.529949+0000","last_clean":"2024-02-05T08:05:59.529949+0000","last_became_active":"2024-02-05T08:04:43.557652+0000","last_became_peered":"2024-02-05T08:04:43.557652+0000","last_unstale":"2024-02-05T08:05:59.529949+0000","last_undegraded":"2024-02-05T08:05:59.529949+0000","last_fullsized":"2024-02-05T08:05:59.529949+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.987638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.267980+0000","last_change":"2024-02-05T08:04:47.587766+0000","last_active":"2024-02-05T08:06:00.267980+0000","last_peered":"2024-02-05T08:06:00.267980+0000","last_clean":"2024-02-05T08:06:00.267980+0000","last_became_active":"2024-02-05T08:04:47.587515+0000","last_became_peered":"2024-02-05T08:04:47.587515+0000","last_unstale":"2024-02-05T08:06:00.267980+0000","last_undegraded":"2024-02-05T08:06:00.267980+0000","last_fullsized":"2024-02-05T08:06:00.267980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.970614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718175+0000","last_change":"2024-02-05T08:05:52.725804+0000","last_active":"2024-02-05T08:05:59.718175+0000","last_peered":"2024-02-05T08:05:59.718175+0000","last_clean":"2024-02-05T08:05:59.718175+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:05:59.718175+0000","last_undegraded":"2024-02-05T08:05:59.718175+0000","last_fullsized":"2024-02-05T08:05:59.718175+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:52.725679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.725679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.406845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268975+0000","last_change":"2024-02-05T08:05:46.458359+0000","last_active":"2024-02-05T08:06:00.268975+0000","last_peered":"2024-02-05T08:06:00.268975+0000","last_clean":"2024-02-05T08:06:00.268975+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:06:00.268975+0000","last_undegraded":"2024-02-05T08:06:00.268975+0000","last_fullsized":"2024-02-05T08:06:00.268975+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.458244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.877756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":64,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530213+0000","last_change":"2024-02-05T08:04:43.557900+0000","last_active":"2024-02-05T08:05:59.530213+0000","last_peered":"2024-02-05T08:05:59.530213+0000","last_clean":"2024-02-05T08:05:59.530213+0000","last_became_active":"2024-02-05T08:04:43.557559+0000","last_became_peered":"2024-02-05T08:04:43.557559+0000","last_unstale":"2024-02-05T08:05:59.530213+0000","last_undegraded":"2024-02-05T08:05:59.530213+0000","last_fullsized":"2024-02-05T08:05:59.530213+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:09.465379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530663+0000","last_change":"2024-02-05T08:05:51.076610+0000","last_active":"2024-02-05T08:05:59.530663+0000","last_peered":"2024-02-05T08:05:59.530663+0000","last_clean":"2024-02-05T08:05:59.530663+0000","last_became_active":"2024-02-05T08:04:48.170756+0000","last_became_peered":"2024-02-05T08:04:48.170756+0000","last_unstale":"2024-02-05T08:05:59.530663+0000","last_undegraded":"2024-02-05T08:05:59.530663+0000","last_fullsized":"2024-02-05T08:05:59.530663+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:51.076514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:51.076514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:11.206152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527244+0000","last_change":"2024-02-05T08:04:45.559908+0000","last_active":"2024-02-05T08:05:59.527244+0000","last_peered":"2024-02-05T08:05:59.527244+0000","last_clean":"2024-02-05T08:05:59.527244+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:05:59.527244+0000","last_undegraded":"2024-02-05T08:05:59.527244+0000","last_fullsized":"2024-02-05T08:05:59.527244+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:11.185330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528706+0000","last_change":"2024-02-05T08:04:41.541812+0000","last_active":"2024-02-05T08:05:59.528706+0000","last_peered":"2024-02-05T08:05:59.528706+0000","last_clean":"2024-02-05T08:05:59.528706+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:05:59.528706+0000","last_undegraded":"2024-02-05T08:05:59.528706+0000","last_fullsized":"2024-02-05T08:05:59.528706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546521+0000","last_change":"2024-02-05T08:05:46.814239+0000","last_active":"2024-02-05T08:06:00.546521+0000","last_peered":"2024-02-05T08:06:00.546521+0000","last_clean":"2024-02-05T08:06:00.546521+0000","last_became_active":"2024-02-05T08:04:43.557496+0000","last_became_peered":"2024-02-05T08:04:43.557496+0000","last_unstale":"2024-02-05T08:06:00.546521+0000","last_undegraded":"2024-02-05T08:06:00.546521+0000","last_fullsized":"2024-02-05T08:06:00.546521+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:05:46.814139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.814139+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:58.081182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225446+0000","last_change":"2024-02-05T08:04:48.170578+0000","last_active":"2024-02-05T08:06:00.225446+0000","last_peered":"2024-02-05T08:06:00.225446+0000","last_clean":"2024-02-05T08:06:00.225446+0000","last_became_active":"2024-02-05T08:04:48.169770+0000","last_became_peered":"2024-02-05T08:04:48.169770+0000","last_unstale":"2024-02-05T08:06:00.225446+0000","last_undegraded":"2024-02-05T08:06:00.225446+0000","last_fullsized":"2024-02-05T08:06:00.225446+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.961379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530140+0000","last_change":"2024-02-05T08:05:57.052122+0000","last_active":"2024-02-05T08:05:59.530140+0000","last_peered":"2024-02-05T08:05:59.530140+0000","last_clean":"2024-02-05T08:05:59.530140+0000","last_became_active":"2024-02-05T08:04:45.563948+0000","last_became_peered":"2024-02-05T08:04:45.563948+0000","last_unstale":"2024-02-05T08:05:59.530140+0000","last_undegraded":"2024-02-05T08:05:59.530140+0000","last_fullsized":"2024-02-05T08:05:59.530140+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.052030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:14.589576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530056+0000","last_change":"2024-02-05T08:04:41.534341+0000","last_active":"2024-02-05T08:05:59.530056+0000","last_peered":"2024-02-05T08:05:59.530056+0000","last_clean":"2024-02-05T08:05:59.530056+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:05:59.530056+0000","last_undegraded":"2024-02-05T08:05:59.530056+0000","last_fullsized":"2024-02-05T08:05:59.530056+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.393385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"90'17","reported_seq":90,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226406+0000","last_change":"2024-02-05T08:05:58.285391+0000","last_active":"2024-02-05T08:06:00.226406+0000","last_peered":"2024-02-05T08:06:00.226406+0000","last_clean":"2024-02-05T08:06:00.226406+0000","last_became_active":"2024-02-05T08:04:43.557596+0000","last_became_peered":"2024-02-05T08:04:43.557596+0000","last_unstale":"2024-02-05T08:06:00.226406+0000","last_undegraded":"2024-02-05T08:06:00.226406+0000","last_fullsized":"2024-02-05T08:06:00.226406+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.285304+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.927379+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718547+0000","last_change":"2024-02-05T08:05:55.766576+0000","last_active":"2024-02-05T08:05:59.718547+0000","last_peered":"2024-02-05T08:05:59.718547+0000","last_clean":"2024-02-05T08:05:59.718547+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:05:59.718547+0000","last_undegraded":"2024-02-05T08:05:59.718547+0000","last_fullsized":"2024-02-05T08:05:59.718547+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:05:55.766478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.766478+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:59.016044+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":93,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:14.384387+0000","last_change":"2024-02-05T08:06:14.384387+0000","last_active":"2024-02-05T08:06:14.384387+0000","last_peered":"2024-02-05T08:06:14.384387+0000","last_clean":"2024-02-05T08:06:14.384387+0000","last_became_active":"2024-02-05T08:04:45.564774+0000","last_became_peered":"2024-02-05T08:04:45.564774+0000","last_unstale":"2024-02-05T08:06:14.384387+0000","last_undegraded":"2024-02-05T08:06:14.384387+0000","last_fullsized":"2024-02-05T08:06:14.384387+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:06:14.384219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:14.384219+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:41.883934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":68,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:11.775078+0000","last_change":"2024-02-05T08:06:11.775125+0000","last_active":"2024-02-05T08:06:11.775078+0000","last_peered":"2024-02-05T08:06:11.775078+0000","last_clean":"2024-02-05T08:06:11.775078+0000","last_became_active":"2024-02-05T08:04:41.532339+0000","last_became_peered":"2024-02-05T08:04:41.532339+0000","last_unstale":"2024-02-05T08:06:11.775078+0000","last_undegraded":"2024-02-05T08:06:11.775078+0000","last_fullsized":"2024-02-05T08:06:11.775078+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:06:11.774993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:11.774993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.286950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528405+0000","last_change":"2024-02-05T08:04:43.606275+0000","last_active":"2024-02-05T08:05:59.528405+0000","last_peered":"2024-02-05T08:05:59.528405+0000","last_clean":"2024-02-05T08:05:59.528405+0000","last_became_active":"2024-02-05T08:04:43.605952+0000","last_became_peered":"2024-02-05T08:04:43.605952+0000","last_unstale":"2024-02-05T08:05:59.528405+0000","last_undegraded":"2024-02-05T08:05:59.528405+0000","last_fullsized":"2024-02-05T08:05:59.528405+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:09.695654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:09.715125+0000","last_change":"2024-02-05T08:06:09.715146+0000","last_active":"2024-02-05T08:06:09.715125+0000","last_peered":"2024-02-05T08:06:09.715125+0000","last_clean":"2024-02-05T08:06:09.715125+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:06:09.715125+0000","last_undegraded":"2024-02-05T08:06:09.715125+0000","last_fullsized":"2024-02-05T08:06:09.715125+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:09.715067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:09.715067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:24.602489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528636+0000","last_change":"2024-02-05T08:04:45.564760+0000","last_active":"2024-02-05T08:05:59.528636+0000","last_peered":"2024-02-05T08:05:59.528636+0000","last_clean":"2024-02-05T08:05:59.528636+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:05:59.528636+0000","last_undegraded":"2024-02-05T08:05:59.528636+0000","last_fullsized":"2024-02-05T08:05:59.528636+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.404706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:07.137987+0000","last_change":"2024-02-05T08:06:07.138006+0000","last_active":"2024-02-05T08:06:07.137987+0000","last_peered":"2024-02-05T08:06:07.137987+0000","last_clean":"2024-02-05T08:06:07.137987+0000","last_became_active":"2024-02-05T08:04:41.534448+0000","last_became_peered":"2024-02-05T08:04:41.534448+0000","last_unstale":"2024-02-05T08:06:07.137987+0000","last_undegraded":"2024-02-05T08:06:07.137987+0000","last_fullsized":"2024-02-05T08:06:07.137987+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:06:07.137916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:29.601881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.721085+0000","last_change":"2024-02-05T08:05:46.208637+0000","last_active":"2024-02-05T08:05:59.721085+0000","last_peered":"2024-02-05T08:05:59.721085+0000","last_clean":"2024-02-05T08:05:59.721085+0000","last_became_active":"2024-02-05T08:05:03.789351+0000","last_became_peered":"2024-02-05T08:05:03.789351+0000","last_unstale":"2024-02-05T08:05:59.721085+0000","last_undegraded":"2024-02-05T08:05:59.721085+0000","last_fullsized":"2024-02-05T08:05:59.721085+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:05:46.208534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.208534+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:00.233101+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.717804+0000","last_change":"2024-02-05T08:04:47.579487+0000","last_active":"2024-02-05T08:05:59.717804+0000","last_peered":"2024-02-05T08:05:59.717804+0000","last_clean":"2024-02-05T08:05:59.717804+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:05:59.717804+0000","last_undegraded":"2024-02-05T08:05:59.717804+0000","last_fullsized":"2024-02-05T08:05:59.717804+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:10.763840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720346+0000","last_change":"2024-02-05T08:05:54.367693+0000","last_active":"2024-02-05T08:05:59.720346+0000","last_peered":"2024-02-05T08:05:59.720346+0000","last_clean":"2024-02-05T08:05:59.720346+0000","last_became_active":"2024-02-05T08:04:45.560874+0000","last_became_peered":"2024-02-05T08:04:45.560874+0000","last_unstale":"2024-02-05T08:05:59.720346+0000","last_undegraded":"2024-02-05T08:05:59.720346+0000","last_fullsized":"2024-02-05T08:05:59.720346+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:05:54.367588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:17.218061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:10.193331+0000","last_change":"2024-02-05T08:06:10.193352+0000","last_active":"2024-02-05T08:06:10.193331+0000","last_peered":"2024-02-05T08:06:10.193331+0000","last_clean":"2024-02-05T08:06:10.193331+0000","last_became_active":"2024-02-05T08:04:41.536019+0000","last_became_peered":"2024-02-05T08:04:41.536019+0000","last_unstale":"2024-02-05T08:06:10.193331+0000","last_undegraded":"2024-02-05T08:06:10.193331+0000","last_fullsized":"2024-02-05T08:06:10.193331+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:10.193272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:10.193272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:33.575164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":60,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225759+0000","last_change":"2024-02-05T08:04:43.555567+0000","last_active":"2024-02-05T08:06:00.225759+0000","last_peered":"2024-02-05T08:06:00.225759+0000","last_clean":"2024-02-05T08:06:00.225759+0000","last_became_active":"2024-02-05T08:04:43.552866+0000","last_became_peered":"2024-02-05T08:04:43.552866+0000","last_unstale":"2024-02-05T08:06:00.225759+0000","last_undegraded":"2024-02-05T08:06:00.225759+0000","last_fullsized":"2024-02-05T08:06:00.225759+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:03.755886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:02.190986+0000","last_change":"2024-02-05T08:06:02.191006+0000","last_active":"2024-02-05T08:06:02.190986+0000","last_peered":"2024-02-05T08:06:02.190986+0000","last_clean":"2024-02-05T08:06:02.190986+0000","last_became_active":"2024-02-05T08:04:48.172793+0000","last_became_peered":"2024-02-05T08:04:48.172793+0000","last_unstale":"2024-02-05T08:06:02.190986+0000","last_undegraded":"2024-02-05T08:06:02.190986+0000","last_fullsized":"2024-02-05T08:06:02.190986+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:02.190931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:02.190931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:27.955974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718344+0000","last_change":"2024-02-05T08:05:48.748227+0000","last_active":"2024-02-05T08:05:59.718344+0000","last_peered":"2024-02-05T08:05:59.718344+0000","last_clean":"2024-02-05T08:05:59.718344+0000","last_became_active":"2024-02-05T08:04:45.563610+0000","last_became_peered":"2024-02-05T08:04:45.563610+0000","last_unstale":"2024-02-05T08:05:59.718344+0000","last_undegraded":"2024-02-05T08:05:59.718344+0000","last_fullsized":"2024-02-05T08:05:59.718344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:48.748132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:48.748132+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.748132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:17.563379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:02.532340+0000","last_change":"2024-02-05T08:06:02.532375+0000","last_active":"2024-02-05T08:06:02.532340+0000","last_peered":"2024-02-05T08:06:02.532340+0000","last_clean":"2024-02-05T08:06:02.532340+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:06:02.532340+0000","last_undegraded":"2024-02-05T08:06:02.532340+0000","last_fullsized":"2024-02-05T08:06:02.532340+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:02.532248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:02.532248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:05.965893+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226059+0000","last_change":"2024-02-05T08:04:43.606219+0000","last_active":"2024-02-05T08:06:00.226059+0000","last_peered":"2024-02-05T08:06:00.226059+0000","last_clean":"2024-02-05T08:06:00.226059+0000","last_became_active":"2024-02-05T08:04:43.605932+0000","last_became_peered":"2024-02-05T08:04:43.605932+0000","last_unstale":"2024-02-05T08:06:00.226059+0000","last_undegraded":"2024-02-05T08:06:00.226059+0000","last_fullsized":"2024-02-05T08:06:00.226059+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:00.517159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:03.183974+0000","last_change":"2024-02-05T08:06:03.184008+0000","last_active":"2024-02-05T08:06:03.183974+0000","last_peered":"2024-02-05T08:06:03.183974+0000","last_clean":"2024-02-05T08:06:03.183974+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:06:03.183974+0000","last_undegraded":"2024-02-05T08:06:03.183974+0000","last_fullsized":"2024-02-05T08:06:03.183974+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:03.183906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:03.183906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:24.748158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226266+0000","last_change":"2024-02-05T08:05:57.270638+0000","last_active":"2024-02-05T08:06:00.226266+0000","last_peered":"2024-02-05T08:06:00.226266+0000","last_clean":"2024-02-05T08:06:00.226266+0000","last_became_active":"2024-02-05T08:04:45.568484+0000","last_became_peered":"2024-02-05T08:04:45.568484+0000","last_unstale":"2024-02-05T08:06:00.226266+0000","last_undegraded":"2024-02-05T08:06:00.226266+0000","last_fullsized":"2024-02-05T08:06:00.226266+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:57.270554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.270554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:03.898076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:15.889872+0000","last_change":"2024-02-05T08:06:15.889891+0000","last_active":"2024-02-05T08:06:15.889872+0000","last_peered":"2024-02-05T08:06:15.889872+0000","last_clean":"2024-02-05T08:06:15.889872+0000","last_became_active":"2024-02-05T08:04:41.539698+0000","last_became_peered":"2024-02-05T08:04:41.539698+0000","last_unstale":"2024-02-05T08:06:15.889872+0000","last_undegraded":"2024-02-05T08:06:15.889872+0000","last_fullsized":"2024-02-05T08:06:15.889872+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:15.889817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:15.889817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:27.988275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527388+0000","last_change":"2024-02-05T08:05:48.910387+0000","last_active":"2024-02-05T08:05:59.527388+0000","last_peered":"2024-02-05T08:05:59.527388+0000","last_clean":"2024-02-05T08:05:59.527388+0000","last_became_active":"2024-02-05T08:04:43.555031+0000","last_became_peered":"2024-02-05T08:04:43.555031+0000","last_unstale":"2024-02-05T08:05:59.527388+0000","last_undegraded":"2024-02-05T08:05:59.527388+0000","last_fullsized":"2024-02-05T08:05:59.527388+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:05:48.910304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.910304+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:49.156858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225328+0000","last_change":"2024-02-05T08:04:47.586079+0000","last_active":"2024-02-05T08:06:00.225328+0000","last_peered":"2024-02-05T08:06:00.225328+0000","last_clean":"2024-02-05T08:06:00.225328+0000","last_became_active":"2024-02-05T08:04:47.585496+0000","last_became_peered":"2024-02-05T08:04:47.585496+0000","last_unstale":"2024-02-05T08:06:00.225328+0000","last_undegraded":"2024-02-05T08:06:00.225328+0000","last_fullsized":"2024-02-05T08:06:00.225328+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:12.376304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527208+0000","last_change":"2024-02-05T08:05:46.864673+0000","last_active":"2024-02-05T08:05:59.527208+0000","last_peered":"2024-02-05T08:05:59.527208+0000","last_clean":"2024-02-05T08:05:59.527208+0000","last_became_active":"2024-02-05T08:04:45.560815+0000","last_became_peered":"2024-02-05T08:04:45.560815+0000","last_unstale":"2024-02-05T08:05:59.527208+0000","last_undegraded":"2024-02-05T08:05:59.527208+0000","last_fullsized":"2024-02-05T08:05:59.527208+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:46.864589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:46.864589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.980095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530377+0000","last_change":"2024-02-05T08:05:53.112020+0000","last_active":"2024-02-05T08:05:59.530377+0000","last_peered":"2024-02-05T08:05:59.530377+0000","last_clean":"2024-02-05T08:05:59.530377+0000","last_became_active":"2024-02-05T08:04:41.543255+0000","last_became_peered":"2024-02-05T08:04:41.543255+0000","last_unstale":"2024-02-05T08:05:59.530377+0000","last_undegraded":"2024-02-05T08:05:59.530377+0000","last_fullsized":"2024-02-05T08:05:59.530377+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:53.111931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.111931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:19.887019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530372+0000","last_change":"2024-02-05T08:04:43.554427+0000","last_active":"2024-02-05T08:05:59.530372+0000","last_peered":"2024-02-05T08:05:59.530372+0000","last_clean":"2024-02-05T08:05:59.530372+0000","last_became_active":"2024-02-05T08:04:43.554101+0000","last_became_peered":"2024-02-05T08:04:43.554101+0000","last_unstale":"2024-02-05T08:05:59.530372+0000","last_undegraded":"2024-02-05T08:05:59.530372+0000","last_fullsized":"2024-02-05T08:05:59.530372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:05.485949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528132+0000","last_change":"2024-02-05T08:04:47.584516+0000","last_active":"2024-02-05T08:05:59.528132+0000","last_peered":"2024-02-05T08:05:59.528132+0000","last_clean":"2024-02-05T08:05:59.528132+0000","last_became_active":"2024-02-05T08:04:47.584235+0000","last_became_peered":"2024-02-05T08:04:47.584235+0000","last_unstale":"2024-02-05T08:05:59.528132+0000","last_undegraded":"2024-02-05T08:05:59.528132+0000","last_fullsized":"2024-02-05T08:05:59.528132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:11.950403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:15.421421+0000","last_change":"2024-02-05T08:06:15.421421+0000","last_active":"2024-02-05T08:06:15.421421+0000","last_peered":"2024-02-05T08:06:15.421421+0000","last_clean":"2024-02-05T08:06:15.421421+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:06:15.421421+0000","last_undegraded":"2024-02-05T08:06:15.421421+0000","last_fullsized":"2024-02-05T08:06:15.421421+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:06:15.421301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:15.421301+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:28.607979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718361+0000","last_change":"2024-02-05T08:05:45.722765+0000","last_active":"2024-02-05T08:05:59.718361+0000","last_peered":"2024-02-05T08:05:59.718361+0000","last_clean":"2024-02-05T08:05:59.718361+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:05:59.718361+0000","last_undegraded":"2024-02-05T08:05:59.718361+0000","last_fullsized":"2024-02-05T08:05:59.718361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:45.722652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:45.722652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.867081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:13.023052+0000","last_change":"2024-02-05T08:06:13.023108+0000","last_active":"2024-02-05T08:06:13.023052+0000","last_peered":"2024-02-05T08:06:13.023052+0000","last_clean":"2024-02-05T08:06:13.023052+0000","last_became_active":"2024-02-05T08:04:43.605981+0000","last_became_peered":"2024-02-05T08:04:43.605981+0000","last_unstale":"2024-02-05T08:06:13.023052+0000","last_undegraded":"2024-02-05T08:06:13.023052+0000","last_fullsized":"2024-02-05T08:06:13.023052+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:06:13.022959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:13.022959+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:31.842367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530817+0000","last_change":"2024-02-05T08:04:47.578924+0000","last_active":"2024-02-05T08:05:59.530817+0000","last_peered":"2024-02-05T08:05:59.530817+0000","last_clean":"2024-02-05T08:05:59.530817+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:05:59.530817+0000","last_undegraded":"2024-02-05T08:05:59.530817+0000","last_fullsized":"2024-02-05T08:05:59.530817+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:03.039774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:11.602928+0000","last_change":"2024-02-05T08:06:10.047557+0000","last_active":"2024-02-05T08:06:11.602928+0000","last_peered":"2024-02-05T08:06:11.602928+0000","last_clean":"2024-02-05T08:06:11.602928+0000","last_became_active":"2024-02-05T08:04:45.565240+0000","last_became_peered":"2024-02-05T08:04:45.565240+0000","last_unstale":"2024-02-05T08:06:11.602928+0000","last_undegraded":"2024-02-05T08:06:11.602928+0000","last_fullsized":"2024-02-05T08:06:11.602928+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:06:10.047432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:10.047432+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:11.831172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527536+0000","last_change":"2024-02-05T08:04:41.540528+0000","last_active":"2024-02-05T08:05:59.527536+0000","last_peered":"2024-02-05T08:05:59.527536+0000","last_clean":"2024-02-05T08:05:59.527536+0000","last_became_active":"2024-02-05T08:04:41.540252+0000","last_became_peered":"2024-02-05T08:04:41.540252+0000","last_unstale":"2024-02-05T08:05:59.527536+0000","last_undegraded":"2024-02-05T08:05:59.527536+0000","last_fullsized":"2024-02-05T08:05:59.527536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:02.085874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269241+0000","last_change":"2024-02-05T08:05:56.541620+0000","last_active":"2024-02-05T08:06:00.269241+0000","last_peered":"2024-02-05T08:06:00.269241+0000","last_clean":"2024-02-05T08:06:00.269241+0000","last_became_active":"2024-02-05T08:04:43.606414+0000","last_became_peered":"2024-02-05T08:04:43.606414+0000","last_unstale":"2024-02-05T08:06:00.269241+0000","last_undegraded":"2024-02-05T08:06:00.269241+0000","last_fullsized":"2024-02-05T08:06:00.269241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:05:56.541510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:56.541510+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:04.755122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:09.355522+0000","last_change":"2024-02-05T08:06:09.355544+0000","last_active":"2024-02-05T08:06:09.355522+0000","last_peered":"2024-02-05T08:06:09.355522+0000","last_clean":"2024-02-05T08:06:09.355522+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:06:09.355522+0000","last_undegraded":"2024-02-05T08:06:09.355522+0000","last_fullsized":"2024-02-05T08:06:09.355522+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:09.355465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:09.355465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.216149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:04.157672+0000","last_change":"2024-02-05T08:06:04.157708+0000","last_active":"2024-02-05T08:06:04.157672+0000","last_peered":"2024-02-05T08:06:04.157672+0000","last_clean":"2024-02-05T08:06:04.157672+0000","last_became_active":"2024-02-05T08:04:45.567622+0000","last_became_peered":"2024-02-05T08:04:45.567622+0000","last_unstale":"2024-02-05T08:06:04.157672+0000","last_undegraded":"2024-02-05T08:06:04.157672+0000","last_fullsized":"2024-02-05T08:06:04.157672+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:04.157598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:04.157598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:15.237706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:16.874002+0000","last_change":"2024-02-05T08:06:16.874026+0000","last_active":"2024-02-05T08:06:16.874002+0000","last_peered":"2024-02-05T08:06:16.874002+0000","last_clean":"2024-02-05T08:06:16.874002+0000","last_became_active":"2024-02-05T08:04:41.540432+0000","last_became_peered":"2024-02-05T08:04:41.540432+0000","last_unstale":"2024-02-05T08:06:16.874002+0000","last_undegraded":"2024-02-05T08:06:16.874002+0000","last_fullsized":"2024-02-05T08:06:16.874002+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:16.873935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:16.873935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:44.498997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528495+0000","last_change":"2024-02-05T08:04:43.608826+0000","last_active":"2024-02-05T08:05:59.528495+0000","last_peered":"2024-02-05T08:05:59.528495+0000","last_clean":"2024-02-05T08:05:59.528495+0000","last_became_active":"2024-02-05T08:04:43.608489+0000","last_became_peered":"2024-02-05T08:04:43.608489+0000","last_unstale":"2024-02-05T08:05:59.528495+0000","last_undegraded":"2024-02-05T08:05:59.528495+0000","last_fullsized":"2024-02-05T08:05:59.528495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.929658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718647+0000","last_change":"2024-02-05T08:05:56.721472+0000","last_active":"2024-02-05T08:05:59.718647+0000","last_peered":"2024-02-05T08:05:59.718647+0000","last_clean":"2024-02-05T08:05:59.718647+0000","last_became_active":"2024-02-05T08:04:48.172747+0000","last_became_peered":"2024-02-05T08:04:48.172747+0000","last_unstale":"2024-02-05T08:05:59.718647+0000","last_undegraded":"2024-02-05T08:05:59.718647+0000","last_fullsized":"2024-02-05T08:05:59.718647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:56.721381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:56.721381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.336615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:04.495948+0000","last_change":"2024-02-05T08:06:04.495991+0000","last_active":"2024-02-05T08:06:04.495948+0000","last_peered":"2024-02-05T08:06:04.495948+0000","last_clean":"2024-02-05T08:06:04.495948+0000","last_became_active":"2024-02-05T08:04:45.560241+0000","last_became_peered":"2024-02-05T08:04:45.560241+0000","last_unstale":"2024-02-05T08:06:04.495948+0000","last_undegraded":"2024-02-05T08:06:04.495948+0000","last_fullsized":"2024-02-05T08:06:04.495948+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:04.495861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:04.495861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:12.837854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720257+0000","last_change":"2024-02-05T08:05:49.229502+0000","last_active":"2024-02-05T08:05:59.720257+0000","last_peered":"2024-02-05T08:05:59.720257+0000","last_clean":"2024-02-05T08:05:59.720257+0000","last_became_active":"2024-02-05T08:04:41.537709+0000","last_became_peered":"2024-02-05T08:04:41.537709+0000","last_unstale":"2024-02-05T08:05:59.720257+0000","last_undegraded":"2024-02-05T08:05:59.720257+0000","last_fullsized":"2024-02-05T08:05:59.720257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:49.229415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:49.229415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:03.840509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:11.050373+0000","last_change":"2024-02-05T08:06:11.050399+0000","last_active":"2024-02-05T08:06:11.050373+0000","last_peered":"2024-02-05T08:06:11.050373+0000","last_clean":"2024-02-05T08:06:11.050373+0000","last_became_active":"2024-02-05T08:04:43.606624+0000","last_became_peered":"2024-02-05T08:04:43.606624+0000","last_unstale":"2024-02-05T08:06:11.050373+0000","last_undegraded":"2024-02-05T08:06:11.050373+0000","last_fullsized":"2024-02-05T08:06:11.050373+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-02-05T08:06:11.050310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:11.050310+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:27.559791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268004+0000","last_change":"2024-02-05T08:05:59.500065+0000","last_active":"2024-02-05T08:06:00.268004+0000","last_peered":"2024-02-05T08:06:00.268004+0000","last_clean":"2024-02-05T08:06:00.268004+0000","last_became_active":"2024-02-05T08:04:48.170100+0000","last_became_peered":"2024-02-05T08:04:48.170100+0000","last_unstale":"2024-02-05T08:06:00.268004+0000","last_undegraded":"2024-02-05T08:06:00.268004+0000","last_fullsized":"2024-02-05T08:06:00.268004+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:59.499960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:59.499960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:16.082380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718992+0000","last_change":"2024-02-05T08:05:57.687803+0000","last_active":"2024-02-05T08:05:59.718992+0000","last_peered":"2024-02-05T08:05:59.718992+0000","last_clean":"2024-02-05T08:05:59.718992+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:05:59.718992+0000","last_undegraded":"2024-02-05T08:05:59.718992+0000","last_fullsized":"2024-02-05T08:05:59.718992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:05:57.687679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.687679+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:23.035896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":79,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268523+0000","last_change":"2024-02-05T08:04:43.608563+0000","last_active":"2024-02-05T08:06:00.268523+0000","last_peered":"2024-02-05T08:06:00.268523+0000","last_clean":"2024-02-05T08:06:00.268523+0000","last_became_active":"2024-02-05T08:04:43.608254+0000","last_became_peered":"2024-02-05T08:04:43.608254+0000","last_unstale":"2024-02-05T08:06:00.268523+0000","last_undegraded":"2024-02-05T08:06:00.268523+0000","last_fullsized":"2024-02-05T08:06:00.268523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.187066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:11.602758+0000","last_change":"2024-02-05T08:05:59.720386+0000","last_active":"2024-02-05T08:06:11.602758+0000","last_peered":"2024-02-05T08:06:11.602758+0000","last_clean":"2024-02-05T08:06:11.602758+0000","last_became_active":"2024-02-05T08:04:45.561044+0000","last_became_peered":"2024-02-05T08:04:45.561044+0000","last_unstale":"2024-02-05T08:06:11.602758+0000","last_undegraded":"2024-02-05T08:06:11.602758+0000","last_fullsized":"2024-02-05T08:06:11.602758+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:05:59.720223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:59.720223+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.562177+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225545+0000","last_change":"2024-02-05T08:04:47.587025+0000","last_active":"2024-02-05T08:06:00.225545+0000","last_peered":"2024-02-05T08:06:00.225545+0000","last_clean":"2024-02-05T08:06:00.225545+0000","last_became_active":"2024-02-05T08:04:47.586044+0000","last_became_peered":"2024-02-05T08:04:47.586044+0000","last_unstale":"2024-02-05T08:06:00.225545+0000","last_undegraded":"2024-02-05T08:06:00.225545+0000","last_fullsized":"2024-02-05T08:06:00.225545+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.700997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.762873+0000","last_change":"2024-02-05T08:06:01.762898+0000","last_active":"2024-02-05T08:06:01.762873+0000","last_peered":"2024-02-05T08:06:01.762873+0000","last_clean":"2024-02-05T08:06:01.762873+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:06:01.762873+0000","last_undegraded":"2024-02-05T08:06:01.762873+0000","last_fullsized":"2024-02-05T08:06:01.762873+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:01.762805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:01.762805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:12.602032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530066+0000","last_change":"2024-02-05T08:04:43.556901+0000","last_active":"2024-02-05T08:05:59.530066+0000","last_peered":"2024-02-05T08:05:59.530066+0000","last_clean":"2024-02-05T08:05:59.530066+0000","last_became_active":"2024-02-05T08:04:43.556580+0000","last_became_peered":"2024-02-05T08:04:43.556580+0000","last_unstale":"2024-02-05T08:05:59.530066+0000","last_undegraded":"2024-02-05T08:05:59.530066+0000","last_fullsized":"2024-02-05T08:05:59.530066+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:06.567288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:07.368791+0000","last_change":"2024-02-05T08:06:07.368811+0000","last_active":"2024-02-05T08:06:07.368791+0000","last_peered":"2024-02-05T08:06:07.368791+0000","last_clean":"2024-02-05T08:06:07.368791+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:06:07.368791+0000","last_undegraded":"2024-02-05T08:06:07.368791+0000","last_fullsized":"2024-02-05T08:06:07.368791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_clean_scrub_stamp":"2024-02-05T08:06:07.368730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:21.064118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":55,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:03.741658+0000","last_change":"2024-02-05T08:06:03.741686+0000","last_active":"2024-02-05T08:06:03.741658+0000","last_peered":"2024-02-05T08:06:03.741658+0000","last_clean":"2024-02-05T08:06:03.741658+0000","last_became_active":"2024-02-05T08:04:47.587082+0000","last_became_peered":"2024-02-05T08:04:47.587082+0000","last_unstale":"2024-02-05T08:06:03.741658+0000","last_undegraded":"2024-02-05T08:06:03.741658+0000","last_fullsized":"2024-02-05T08:06:03.741658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:03.741577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:03.741577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:11.425517+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546310+0000","last_change":"2024-02-05T08:05:55.596909+0000","last_active":"2024-02-05T08:06:00.546310+0000","last_peered":"2024-02-05T08:06:00.546310+0000","last_clean":"2024-02-05T08:06:00.546310+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:06:00.546310+0000","last_undegraded":"2024-02-05T08:06:00.546310+0000","last_fullsized":"2024-02-05T08:06:00.546310+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:55.596827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.596827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.318812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718966+0000","last_change":"2024-02-05T08:05:53.741209+0000","last_active":"2024-02-05T08:05:59.718966+0000","last_peered":"2024-02-05T08:05:59.718966+0000","last_clean":"2024-02-05T08:05:59.718966+0000","last_became_active":"2024-02-05T08:04:41.540586+0000","last_became_peered":"2024-02-05T08:04:41.540586+0000","last_unstale":"2024-02-05T08:05:59.718966+0000","last_undegraded":"2024-02-05T08:05:59.718966+0000","last_fullsized":"2024-02-05T08:05:59.718966+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:53.741059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.741059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:13.753256+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":112,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720616+0000","last_change":"2024-02-05T08:05:03.797738+0000","last_active":"2024-02-05T08:05:59.720616+0000","last_peered":"2024-02-05T08:05:59.720616+0000","last_clean":"2024-02-05T08:05:59.720616+0000","last_became_active":"2024-02-05T08:05:03.786876+0000","last_became_peered":"2024-02-05T08:05:03.786876+0000","last_unstale":"2024-02-05T08:05:59.720616+0000","last_undegraded":"2024-02-05T08:05:59.720616+0000","last_fullsized":"2024-02-05T08:05:59.720616+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:09.572379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:12.271918+0000","last_change":"2024-02-05T08:06:12.271939+0000","last_active":"2024-02-05T08:06:12.271918+0000","last_peered":"2024-02-05T08:06:12.271918+0000","last_clean":"2024-02-05T08:06:12.271918+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:06:12.271918+0000","last_undegraded":"2024-02-05T08:06:12.271918+0000","last_fullsized":"2024-02-05T08:06:12.271918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:12.271856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:12.271856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:31.305936+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.369671+0000","last_change":"2024-02-05T08:06:06.369693+0000","last_active":"2024-02-05T08:06:06.369671+0000","last_peered":"2024-02-05T08:06:06.369671+0000","last_clean":"2024-02-05T08:06:06.369671+0000","last_became_active":"2024-02-05T08:04:45.567341+0000","last_became_peered":"2024-02-05T08:04:45.567341+0000","last_unstale":"2024-02-05T08:06:06.369671+0000","last_undegraded":"2024-02-05T08:06:06.369671+0000","last_fullsized":"2024-02-05T08:06:06.369671+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:06.369612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:06.369612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:31.000287+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:12.516191+0000","last_change":"2024-02-05T08:06:12.516228+0000","last_active":"2024-02-05T08:06:12.516191+0000","last_peered":"2024-02-05T08:06:12.516191+0000","last_clean":"2024-02-05T08:06:12.516191+0000","last_became_active":"2024-02-05T08:04:41.540116+0000","last_became_peered":"2024-02-05T08:04:41.540116+0000","last_unstale":"2024-02-05T08:06:12.516191+0000","last_undegraded":"2024-02-05T08:06:12.516191+0000","last_fullsized":"2024-02-05T08:06:12.516191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:12.516100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:12.516100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:15.796144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:04.215819+0000","last_change":"2024-02-05T08:06:04.215847+0000","last_active":"2024-02-05T08:06:04.215819+0000","last_peered":"2024-02-05T08:06:04.215819+0000","last_clean":"2024-02-05T08:06:04.215819+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:06:04.215819+0000","last_undegraded":"2024-02-05T08:06:04.215819+0000","last_fullsized":"2024-02-05T08:06:04.215819+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:06:04.215758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:04.215758+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:29.117755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225432+0000","last_change":"2024-02-05T08:04:47.588669+0000","last_active":"2024-02-05T08:06:00.225432+0000","last_peered":"2024-02-05T08:06:00.225432+0000","last_clean":"2024-02-05T08:06:00.225432+0000","last_became_active":"2024-02-05T08:04:47.588395+0000","last_became_peered":"2024-02-05T08:04:47.588395+0000","last_unstale":"2024-02-05T08:06:00.225432+0000","last_undegraded":"2024-02-05T08:06:00.225432+0000","last_fullsized":"2024-02-05T08:06:00.225432+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:16.402122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268703+0000","last_change":"2024-02-05T08:05:52.503758+0000","last_active":"2024-02-05T08:06:00.268703+0000","last_peered":"2024-02-05T08:06:00.268703+0000","last_clean":"2024-02-05T08:06:00.268703+0000","last_became_active":"2024-02-05T08:04:40.527674+0000","last_became_peered":"2024-02-05T08:04:40.527674+0000","last_unstale":"2024-02-05T08:06:00.268703+0000","last_undegraded":"2024-02-05T08:06:00.268703+0000","last_fullsized":"2024-02-05T08:06:00.268703+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.503651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.821538+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:10.946963+0000","last_change":"2024-02-05T08:06:10.946983+0000","last_active":"2024-02-05T08:06:10.946963+0000","last_peered":"2024-02-05T08:06:10.946963+0000","last_clean":"2024-02-05T08:06:10.946963+0000","last_became_active":"2024-02-05T08:04:41.537478+0000","last_became_peered":"2024-02-05T08:04:41.537478+0000","last_unstale":"2024-02-05T08:06:10.946963+0000","last_undegraded":"2024-02-05T08:06:10.946963+0000","last_fullsized":"2024-02-05T08:06:10.946963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:06:10.946904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.448870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"85'38","reported_seq":141,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:11.602841+0000","last_change":"2024-02-05T08:05:43.858932+0000","last_active":"2024-02-05T08:06:11.602841+0000","last_peered":"2024-02-05T08:06:11.602841+0000","last_clean":"2024-02-05T08:06:11.602841+0000","last_became_active":"2024-02-05T08:04:43.554536+0000","last_became_peered":"2024-02-05T08:04:43.554536+0000","last_unstale":"2024-02-05T08:06:11.602841+0000","last_undegraded":"2024-02-05T08:06:11.602841+0000","last_fullsized":"2024-02-05T08:06:11.602841+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:05:43.858841+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:54.322675+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527257+0000","last_change":"2024-02-05T08:04:45.562140+0000","last_active":"2024-02-05T08:05:59.527257+0000","last_peered":"2024-02-05T08:05:59.527257+0000","last_clean":"2024-02-05T08:05:59.527257+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:05:59.527257+0000","last_undegraded":"2024-02-05T08:05:59.527257+0000","last_fullsized":"2024-02-05T08:05:59.527257+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:14.185461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.719032+0000","last_change":"2024-02-05T08:04:48.170226+0000","last_active":"2024-02-05T08:05:59.719032+0000","last_peered":"2024-02-05T08:05:59.719032+0000","last_clean":"2024-02-05T08:05:59.719032+0000","last_became_active":"2024-02-05T08:04:48.169894+0000","last_became_peered":"2024-02-05T08:04:48.169894+0000","last_unstale":"2024-02-05T08:05:59.719032+0000","last_undegraded":"2024-02-05T08:05:59.719032+0000","last_fullsized":"2024-02-05T08:05:59.719032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:15.026925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"67'1","reported_seq":65,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.526644+0000","last_change":"2024-02-05T08:04:41.536875+0000","last_active":"2024-02-05T08:05:59.526644+0000","last_peered":"2024-02-05T08:05:59.526644+0000","last_clean":"2024-02-05T08:05:59.526644+0000","last_became_active":"2024-02-05T08:04:41.536585+0000","last_became_peered":"2024-02-05T08:04:41.536585+0000","last_unstale":"2024-02-05T08:05:59.526644+0000","last_undegraded":"2024-02-05T08:05:59.526644+0000","last_fullsized":"2024-02-05T08:05:59.526644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.042189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.545805+0000","last_change":"2024-02-05T08:05:41.807274+0000","last_active":"2024-02-05T08:06:00.545805+0000","last_peered":"2024-02-05T08:06:00.545805+0000","last_clean":"2024-02-05T08:06:00.545805+0000","last_became_active":"2024-02-05T08:04:40.529073+0000","last_became_peered":"2024-02-05T08:04:40.529073+0000","last_unstale":"2024-02-05T08:06:00.545805+0000","last_undegraded":"2024-02-05T08:06:00.545805+0000","last_fullsized":"2024-02-05T08:06:00.545805+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:41.807173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:05:41.807173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:48.748767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546072+0000","last_change":"2024-02-05T08:05:45.827248+0000","last_active":"2024-02-05T08:06:00.546072+0000","last_peered":"2024-02-05T08:06:00.546072+0000","last_clean":"2024-02-05T08:06:00.546072+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:06:00.546072+0000","last_undegraded":"2024-02-05T08:06:00.546072+0000","last_fullsized":"2024-02-05T08:06:00.546072+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:05:45.827140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:45.827140+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:14.695181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":59,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.546608+0000","last_change":"2024-02-05T08:05:58.550125+0000","last_active":"2024-02-05T08:06:00.546608+0000","last_peered":"2024-02-05T08:06:00.546608+0000","last_clean":"2024-02-05T08:06:00.546608+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:06:00.546608+0000","last_undegraded":"2024-02-05T08:06:00.546608+0000","last_fullsized":"2024-02-05T08:06:00.546608+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:58.550049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.550049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.897417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268113+0000","last_change":"2024-02-05T08:04:47.586644+0000","last_active":"2024-02-05T08:06:00.268113+0000","last_peered":"2024-02-05T08:06:00.268113+0000","last_clean":"2024-02-05T08:06:00.268113+0000","last_became_active":"2024-02-05T08:04:47.586358+0000","last_became_peered":"2024-02-05T08:04:47.586358+0000","last_unstale":"2024-02-05T08:06:00.268113+0000","last_undegraded":"2024-02-05T08:06:00.268113+0000","last_fullsized":"2024-02-05T08:06:00.268113+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:13.569831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":316,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718051+0000","last_change":"2024-02-05T08:05:16.657716+0000","last_active":"2024-02-05T08:05:59.718051+0000","last_peered":"2024-02-05T08:05:59.718051+0000","last_clean":"2024-02-05T08:05:59.718051+0000","last_became_active":"2024-02-05T08:04:30.515158+0000","last_became_peered":"2024-02-05T08:04:30.515158+0000","last_unstale":"2024-02-05T08:05:59.718051+0000","last_undegraded":"2024-02-05T08:05:59.718051+0000","last_fullsized":"2024-02-05T08:05:59.718051+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'119","last_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:05:16.657600+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:37.807886+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226170+0000","last_change":"2024-02-05T08:04:41.538282+0000","last_active":"2024-02-05T08:06:00.226170+0000","last_peered":"2024-02-05T08:06:00.226170+0000","last_clean":"2024-02-05T08:06:00.226170+0000","last_became_active":"2024-02-05T08:04:41.538002+0000","last_became_peered":"2024-02-05T08:04:41.538002+0000","last_unstale":"2024-02-05T08:06:00.226170+0000","last_undegraded":"2024-02-05T08:06:00.226170+0000","last_fullsized":"2024-02-05T08:06:00.226170+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.327424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":105,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:11.602774+0000","last_change":"2024-02-05T08:06:00.304976+0000","last_active":"2024-02-05T08:06:11.602774+0000","last_peered":"2024-02-05T08:06:11.602774+0000","last_clean":"2024-02-05T08:06:11.602774+0000","last_became_active":"2024-02-05T08:04:43.606401+0000","last_became_peered":"2024-02-05T08:04:43.606401+0000","last_unstale":"2024-02-05T08:06:11.602774+0000","last_undegraded":"2024-02-05T08:06:11.602774+0000","last_fullsized":"2024-02-05T08:06:11.602774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-02-05T08:06:00.304861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:00.304861+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:15.661923+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718680+0000","last_change":"2024-02-05T08:04:45.565211+0000","last_active":"2024-02-05T08:05:59.718680+0000","last_peered":"2024-02-05T08:05:59.718680+0000","last_clean":"2024-02-05T08:05:59.718680+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:05:59.718680+0000","last_undegraded":"2024-02-05T08:05:59.718680+0000","last_fullsized":"2024-02-05T08:05:59.718680+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:10.360421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527679+0000","last_change":"2024-02-05T08:04:47.587068+0000","last_active":"2024-02-05T08:05:59.527679+0000","last_peered":"2024-02-05T08:05:59.527679+0000","last_clean":"2024-02-05T08:05:59.527679+0000","last_became_active":"2024-02-05T08:04:47.586774+0000","last_became_peered":"2024-02-05T08:04:47.586774+0000","last_unstale":"2024-02-05T08:05:59.527679+0000","last_undegraded":"2024-02-05T08:05:59.527679+0000","last_fullsized":"2024-02-05T08:05:59.527679+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:00.755929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527011+0000","last_change":"2024-02-05T08:04:41.538403+0000","last_active":"2024-02-05T08:05:59.527011+0000","last_peered":"2024-02-05T08:05:59.527011+0000","last_clean":"2024-02-05T08:05:59.527011+0000","last_became_active":"2024-02-05T08:04:41.537872+0000","last_became_peered":"2024-02-05T08:04:41.537872+0000","last_unstale":"2024-02-05T08:05:59.527011+0000","last_undegraded":"2024-02-05T08:05:59.527011+0000","last_fullsized":"2024-02-05T08:05:59.527011+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:07.800257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":169,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:15.923743+0000","last_change":"2024-02-05T08:05:43.746185+0000","last_active":"2024-02-05T08:06:15.923743+0000","last_peered":"2024-02-05T08:06:15.923743+0000","last_clean":"2024-02-05T08:06:15.923743+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:06:15.923743+0000","last_undegraded":"2024-02-05T08:06:15.923743+0000","last_fullsized":"2024-02-05T08:06:15.923743+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:05:43.746095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:05:43.746095+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:55.186390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":90,"num_read_kb":85,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528230+0000","last_change":"2024-02-05T08:04:43.550001+0000","last_active":"2024-02-05T08:05:59.528230+0000","last_peered":"2024-02-05T08:05:59.528230+0000","last_clean":"2024-02-05T08:05:59.528230+0000","last_became_active":"2024-02-05T08:04:43.549687+0000","last_became_peered":"2024-02-05T08:04:43.549687+0000","last_unstale":"2024-02-05T08:05:59.528230+0000","last_undegraded":"2024-02-05T08:05:59.528230+0000","last_fullsized":"2024-02-05T08:05:59.528230+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.794328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.269362+0000","last_change":"2024-02-05T08:04:45.559137+0000","last_active":"2024-02-05T08:06:00.269362+0000","last_peered":"2024-02-05T08:06:00.269362+0000","last_clean":"2024-02-05T08:06:00.269362+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:06:00.269362+0000","last_undegraded":"2024-02-05T08:06:00.269362+0000","last_fullsized":"2024-02-05T08:06:00.269362+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:12.171151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527562+0000","last_change":"2024-02-05T08:04:48.170100+0000","last_active":"2024-02-05T08:05:59.527562+0000","last_peered":"2024-02-05T08:05:59.527562+0000","last_clean":"2024-02-05T08:05:59.527562+0000","last_became_active":"2024-02-05T08:04:48.169777+0000","last_became_peered":"2024-02-05T08:04:48.169777+0000","last_unstale":"2024-02-05T08:05:59.527562+0000","last_undegraded":"2024-02-05T08:05:59.527562+0000","last_fullsized":"2024-02-05T08:05:59.527562+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527843+0000","last_change":"2024-02-05T08:05:51.880952+0000","last_active":"2024-02-05T08:05:59.527843+0000","last_peered":"2024-02-05T08:05:59.527843+0000","last_clean":"2024-02-05T08:05:59.527843+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:05:59.527843+0000","last_undegraded":"2024-02-05T08:05:59.527843+0000","last_fullsized":"2024-02-05T08:05:59.527843+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:51.880864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:51.880864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:53.033627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":98,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528739+0000","last_change":"2024-02-05T08:05:55.329780+0000","last_active":"2024-02-05T08:05:59.528739+0000","last_peered":"2024-02-05T08:05:59.528739+0000","last_clean":"2024-02-05T08:05:59.528739+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:05:59.528739+0000","last_undegraded":"2024-02-05T08:05:59.528739+0000","last_fullsized":"2024-02-05T08:05:59.528739+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:05:55.329677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:55.329677+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:07.966593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.720746+0000","last_change":"2024-02-05T08:04:45.565899+0000","last_active":"2024-02-05T08:05:59.720746+0000","last_peered":"2024-02-05T08:05:59.720746+0000","last_clean":"2024-02-05T08:05:59.720746+0000","last_became_active":"2024-02-05T08:04:45.565618+0000","last_became_peered":"2024-02-05T08:04:45.565618+0000","last_unstale":"2024-02-05T08:05:59.720746+0000","last_undegraded":"2024-02-05T08:05:59.720746+0000","last_fullsized":"2024-02-05T08:05:59.720746+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:04.936746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528338+0000","last_change":"2024-02-05T08:05:58.215810+0000","last_active":"2024-02-05T08:05:59.528338+0000","last_peered":"2024-02-05T08:05:59.528338+0000","last_clean":"2024-02-05T08:05:59.528338+0000","last_became_active":"2024-02-05T08:04:47.583866+0000","last_became_peered":"2024-02-05T08:04:47.583866+0000","last_unstale":"2024-02-05T08:05:59.528338+0000","last_undegraded":"2024-02-05T08:05:59.528338+0000","last_fullsized":"2024-02-05T08:05:59.528338+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:58.215737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:58.215737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:08.871723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:14.454416+0000","last_change":"2024-02-05T08:06:14.454452+0000","last_active":"2024-02-05T08:06:14.454416+0000","last_peered":"2024-02-05T08:06:14.454416+0000","last_clean":"2024-02-05T08:06:14.454416+0000","last_became_active":"2024-02-05T08:04:41.537818+0000","last_became_peered":"2024-02-05T08:04:41.537818+0000","last_unstale":"2024-02-05T08:06:14.454416+0000","last_undegraded":"2024-02-05T08:06:14.454416+0000","last_fullsized":"2024-02-05T08:06:14.454416+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:14.454340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:06:14.454340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:26.230751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":70,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.527293+0000","last_change":"2024-02-05T08:04:43.552388+0000","last_active":"2024-02-05T08:05:59.527293+0000","last_peered":"2024-02-05T08:05:59.527293+0000","last_clean":"2024-02-05T08:05:59.527293+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:05:59.527293+0000","last_undegraded":"2024-02-05T08:05:59.527293+0000","last_fullsized":"2024-02-05T08:05:59.527293+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:05.174347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:11.602802+0000","last_change":"2024-02-05T08:05:57.266029+0000","last_active":"2024-02-05T08:06:11.602802+0000","last_peered":"2024-02-05T08:06:11.602802+0000","last_clean":"2024-02-05T08:06:11.602802+0000","last_became_active":"2024-02-05T08:04:45.562462+0000","last_became_peered":"2024-02-05T08:04:45.562462+0000","last_unstale":"2024-02-05T08:06:11.602802+0000","last_undegraded":"2024-02-05T08:06:11.602802+0000","last_fullsized":"2024-02-05T08:06:11.602802+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:05:57.265939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:05:57.265939+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:03.556401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":47,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.718573+0000","last_change":"2024-02-05T08:04:48.172841+0000","last_active":"2024-02-05T08:05:59.718573+0000","last_peered":"2024-02-05T08:05:59.718573+0000","last_clean":"2024-02-05T08:05:59.718573+0000","last_became_active":"2024-02-05T08:04:48.172564+0000","last_became_peered":"2024-02-05T08:04:48.172564+0000","last_unstale":"2024-02-05T08:05:59.718573+0000","last_undegraded":"2024-02-05T08:05:59.718573+0000","last_fullsized":"2024-02-05T08:05:59.718573+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:15.681030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528317+0000","last_change":"2024-02-05T08:05:52.283473+0000","last_active":"2024-02-05T08:05:59.528317+0000","last_peered":"2024-02-05T08:05:59.528317+0000","last_clean":"2024-02-05T08:05:59.528317+0000","last_became_active":"2024-02-05T08:04:41.534433+0000","last_became_peered":"2024-02-05T08:04:41.534433+0000","last_unstale":"2024-02-05T08:05:59.528317+0000","last_undegraded":"2024-02-05T08:05:59.528317+0000","last_fullsized":"2024-02-05T08:05:59.528317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:05:52.283383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.283383+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:05.067021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":75,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529862+0000","last_change":"2024-02-05T08:04:43.551586+0000","last_active":"2024-02-05T08:05:59.529862+0000","last_peered":"2024-02-05T08:05:59.529862+0000","last_clean":"2024-02-05T08:05:59.529862+0000","last_became_active":"2024-02-05T08:04:43.551184+0000","last_became_peered":"2024-02-05T08:04:43.551184+0000","last_unstale":"2024-02-05T08:05:59.529862+0000","last_undegraded":"2024-02-05T08:05:59.529862+0000","last_fullsized":"2024-02-05T08:05:59.529862+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:05.037825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:11.602788+0000","last_change":"2024-02-05T08:04:45.563011+0000","last_active":"2024-02-05T08:06:11.602788+0000","last_peered":"2024-02-05T08:06:11.602788+0000","last_clean":"2024-02-05T08:06:11.602788+0000","last_became_active":"2024-02-05T08:04:45.562597+0000","last_became_peered":"2024-02-05T08:04:45.562597+0000","last_unstale":"2024-02-05T08:06:11.602788+0000","last_undegraded":"2024-02-05T08:06:11.602788+0000","last_fullsized":"2024-02-05T08:06:11.602788+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.186855+0000","last_change":"2024-02-05T08:06:01.186878+0000","last_active":"2024-02-05T08:06:01.186855+0000","last_peered":"2024-02-05T08:06:01.186855+0000","last_clean":"2024-02-05T08:06:01.186855+0000","last_became_active":"2024-02-05T08:04:47.582776+0000","last_became_peered":"2024-02-05T08:04:47.582776+0000","last_unstale":"2024-02-05T08:06:01.186855+0000","last_undegraded":"2024-02-05T08:06:01.186855+0000","last_fullsized":"2024-02-05T08:06:01.186855+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:01.186782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:06:01.186782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:02.355302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226658+0000","last_change":"2024-02-05T08:05:48.267727+0000","last_active":"2024-02-05T08:06:00.226658+0000","last_peered":"2024-02-05T08:06:00.226658+0000","last_clean":"2024-02-05T08:06:00.226658+0000","last_became_active":"2024-02-05T08:04:41.540818+0000","last_became_peered":"2024-02-05T08:04:41.540818+0000","last_unstale":"2024-02-05T08:06:00.226658+0000","last_undegraded":"2024-02-05T08:06:00.226658+0000","last_fullsized":"2024-02-05T08:06:00.226658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:05:48.267582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.267582+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:15.800342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":94,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225869+0000","last_change":"2024-02-05T08:05:52.195604+0000","last_active":"2024-02-05T08:06:00.225869+0000","last_peered":"2024-02-05T08:06:00.225869+0000","last_clean":"2024-02-05T08:06:00.225869+0000","last_became_active":"2024-02-05T08:04:43.607546+0000","last_became_peered":"2024-02-05T08:04:43.607546+0000","last_unstale":"2024-02-05T08:06:00.225869+0000","last_undegraded":"2024-02-05T08:06:00.225869+0000","last_fullsized":"2024-02-05T08:06:00.225869+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-05T08:05:52.195494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:52.195494+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:16.333300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":50,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529710+0000","last_change":"2024-02-05T08:04:45.561907+0000","last_active":"2024-02-05T08:05:59.529710+0000","last_peered":"2024-02-05T08:05:59.529710+0000","last_clean":"2024-02-05T08:05:59.529710+0000","last_became_active":"2024-02-05T08:04:45.561639+0000","last_became_peered":"2024-02-05T08:04:45.561639+0000","last_unstale":"2024-02-05T08:05:59.529710+0000","last_undegraded":"2024-02-05T08:05:59.529710+0000","last_fullsized":"2024-02-05T08:05:59.529710+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:08.611331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":54,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.268056+0000","last_change":"2024-02-05T08:05:48.449994+0000","last_active":"2024-02-05T08:06:00.268056+0000","last_peered":"2024-02-05T08:06:00.268056+0000","last_clean":"2024-02-05T08:06:00.268056+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:06:00.268056+0000","last_undegraded":"2024-02-05T08:06:00.268056+0000","last_fullsized":"2024-02-05T08:06:00.268056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:48.449900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.449900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:55.343526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":78,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:11.602738+0000","last_change":"2024-02-05T08:04:45.567294+0000","last_active":"2024-02-05T08:06:11.602738+0000","last_peered":"2024-02-05T08:06:11.602738+0000","last_clean":"2024-02-05T08:06:11.602738+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:06:11.602738+0000","last_undegraded":"2024-02-05T08:06:11.602738+0000","last_fullsized":"2024-02-05T08:06:11.602738+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225833+0000","last_change":"2024-02-05T08:05:59.261588+0000","last_active":"2024-02-05T08:06:00.225833+0000","last_peered":"2024-02-05T08:06:00.225833+0000","last_clean":"2024-02-05T08:06:00.225833+0000","last_became_active":"2024-02-05T08:04:41.541327+0000","last_became_peered":"2024-02-05T08:04:41.541327+0000","last_unstale":"2024-02-05T08:06:00.225833+0000","last_undegraded":"2024-02-05T08:06:00.225833+0000","last_fullsized":"2024-02-05T08:06:00.225833+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:59.261506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:59.261506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:24.020924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":88,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.526944+0000","last_change":"2024-02-05T08:05:53.816142+0000","last_active":"2024-02-05T08:05:59.526944+0000","last_peered":"2024-02-05T08:05:59.526944+0000","last_clean":"2024-02-05T08:05:59.526944+0000","last_became_active":"2024-02-05T08:04:43.555028+0000","last_became_peered":"2024-02-05T08:04:43.555028+0000","last_unstale":"2024-02-05T08:05:59.526944+0000","last_undegraded":"2024-02-05T08:05:59.526944+0000","last_fullsized":"2024-02-05T08:05:59.526944+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:05:53.815997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:05:53.815997+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:10.642275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.225515+0000","last_change":"2024-02-05T08:04:48.170254+0000","last_active":"2024-02-05T08:06:00.225515+0000","last_peered":"2024-02-05T08:06:00.225515+0000","last_clean":"2024-02-05T08:06:00.225515+0000","last_became_active":"2024-02-05T08:04:48.169762+0000","last_became_peered":"2024-02-05T08:04:48.169762+0000","last_unstale":"2024-02-05T08:06:00.225515+0000","last_undegraded":"2024-02-05T08:06:00.225515+0000","last_fullsized":"2024-02-05T08:06:00.225515+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:13.151401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:01.561022+0000","last_change":"2024-02-05T08:06:01.561064+0000","last_active":"2024-02-05T08:06:01.561022+0000","last_peered":"2024-02-05T08:06:01.561022+0000","last_clean":"2024-02-05T08:06:01.561022+0000","last_became_active":"2024-02-05T08:04:45.561679+0000","last_became_peered":"2024-02-05T08:04:45.561679+0000","last_unstale":"2024-02-05T08:06:01.561022+0000","last_undegraded":"2024-02-05T08:06:01.561022+0000","last_fullsized":"2024-02-05T08:06:01.561022+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:06:01.560919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:01.560919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:21.298440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.529733+0000","last_change":"2024-02-05T08:05:48.116107+0000","last_active":"2024-02-05T08:05:59.529733+0000","last_peered":"2024-02-05T08:05:59.529733+0000","last_clean":"2024-02-05T08:05:59.529733+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:05:59.529733+0000","last_undegraded":"2024-02-05T08:05:59.529733+0000","last_fullsized":"2024-02-05T08:05:59.529733+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:48.116008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.116008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:06.262952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":81,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:09.984233+0000","last_change":"2024-02-05T08:06:09.984289+0000","last_active":"2024-02-05T08:06:09.984233+0000","last_peered":"2024-02-05T08:06:09.984233+0000","last_clean":"2024-02-05T08:06:09.984233+0000","last_became_active":"2024-02-05T08:04:43.554161+0000","last_became_peered":"2024-02-05T08:04:43.554161+0000","last_unstale":"2024-02-05T08:06:09.984233+0000","last_undegraded":"2024-02-05T08:06:09.984233+0000","last_fullsized":"2024-02-05T08:06:09.984233+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:06:09.984141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:09.984141+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:20.006301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":56,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.545953+0000","last_change":"2024-02-05T08:05:48.720004+0000","last_active":"2024-02-05T08:06:00.545953+0000","last_peered":"2024-02-05T08:06:00.545953+0000","last_clean":"2024-02-05T08:06:00.545953+0000","last_became_active":"2024-02-05T08:04:47.585962+0000","last_became_peered":"2024-02-05T08:04:47.585962+0000","last_unstale":"2024-02-05T08:06:00.545953+0000","last_undegraded":"2024-02-05T08:06:00.545953+0000","last_fullsized":"2024-02-05T08:06:00.545953+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:05:48.719923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:05:48.719923+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:54.016895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":51,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.528031+0000","last_change":"2024-02-05T08:04:45.559587+0000","last_active":"2024-02-05T08:05:59.528031+0000","last_peered":"2024-02-05T08:05:59.528031+0000","last_clean":"2024-02-05T08:05:59.528031+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:05:59.528031+0000","last_undegraded":"2024-02-05T08:05:59.528031+0000","last_fullsized":"2024-02-05T08:05:59.528031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:04:44.536163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:02.577883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.721051+0000","last_change":"2024-02-05T08:05:50.230381+0000","last_active":"2024-02-05T08:05:59.721051+0000","last_peered":"2024-02-05T08:05:59.721051+0000","last_clean":"2024-02-05T08:05:59.721051+0000","last_became_active":"2024-02-05T08:04:41.538397+0000","last_became_peered":"2024-02-05T08:04:41.538397+0000","last_unstale":"2024-02-05T08:05:59.721051+0000","last_undegraded":"2024-02-05T08:05:59.721051+0000","last_fullsized":"2024-02-05T08:05:59.721051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:05:50.230263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:50.230263+0000","last_clean_scrub_stamp":"2024-02-05T08:05:50.230263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:59.311643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530628+0000","last_change":"2024-02-05T08:04:43.555191+0000","last_active":"2024-02-05T08:05:59.530628+0000","last_peered":"2024-02-05T08:05:59.530628+0000","last_clean":"2024-02-05T08:05:59.530628+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:05:59.530628+0000","last_undegraded":"2024-02-05T08:05:59.530628+0000","last_fullsized":"2024-02-05T08:05:59.530628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:04:42.523363+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:12.495416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":46,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:05:59.530787+0000","last_change":"2024-02-05T08:04:47.573584+0000","last_active":"2024-02-05T08:05:59.530787+0000","last_peered":"2024-02-05T08:05:59.530787+0000","last_clean":"2024-02-05T08:05:59.530787+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:05:59.530787+0000","last_undegraded":"2024-02-05T08:05:59.530787+0000","last_fullsized":"2024-02-05T08:05:59.530787+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:04:46.546354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.290723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:11.603053+0000","last_change":"2024-02-05T08:06:04.443720+0000","last_active":"2024-02-05T08:06:11.603053+0000","last_peered":"2024-02-05T08:06:11.603053+0000","last_clean":"2024-02-05T08:06:11.603053+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:06:11.603053+0000","last_undegraded":"2024-02-05T08:06:11.603053+0000","last_fullsized":"2024-02-05T08:06:11.603053+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:06:04.443624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:06:04.443624+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:19.854071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:00.226287+0000","last_change":"2024-02-05T08:04:41.534743+0000","last_active":"2024-02-05T08:06:00.226287+0000","last_peered":"2024-02-05T08:06:00.226287+0000","last_clean":"2024-02-05T08:06:00.226287+0000","last_became_active":"2024-02-05T08:04:41.533838+0000","last_became_peered":"2024-02-05T08:04:41.533838+0000","last_unstale":"2024-02-05T08:06:00.226287+0000","last_undegraded":"2024-02-05T08:06:00.226287+0000","last_fullsized":"2024-02-05T08:06:00.226287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:04:40.511631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:06:01.461601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":77,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-02-05T08:06:06.080113+0000","last_change":"2024-02-05T08:06:06.080146+0000","last_active":"2024-02-05T08:06:06.080113+0000","last_peered":"2024-02-05T08:06:06.080113+0000","last_clean":"2024-02-05T08:06:06.080113+0000","last_became_active":"2024-02-05T08:04:43.559922+0000","last_became_peered":"2024-02-05T08:04:43.559922+0000","last_unstale":"2024-02-05T08:06:06.080113+0000","last_undegraded":"2024-02-05T08:06:06.080113+0000","last_fullsized":"2024-02-05T08:06:06.080113+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:06:06.080040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:06:06.080040+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:07:25.229905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":697,"num_read_kb":454,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":412,"ondisk_log_size":412,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":85,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037784,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32656,"kb_used_data":1480,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712496,"statfs":{"total":95995035648,"available":95961595904,"internally_reserved":0,"allocated":1515520,"data_stored":1027404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.48299999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.44400000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.59699999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.373}]},{"osd":5,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.42099999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:05:31 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.40999999999999998}]}]},{"osd":6,"up_from":51,"seq":219043332126,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1495040,"data_stored":1025877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.74299999999999999,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.74199999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.55500000000000005}]},{"osd":1,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.60399999999999998,"15min":0.60099999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.97099999999999997,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.55300000000000005,"15min":0.55100000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.439}]},{"osd":2,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40500000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80600000000000005,"5min":0.871,"15min":0.871},"last":0.504},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.57999999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.78900000000000003,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.53900000000000003}]},{"osd":3,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57899999999999996,"15min":0.57699999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.56100000000000005}]},{"osd":4,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.44400000000000001,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.45100000000000001,"15min":0.44400000000000001},"min":{"1min":0.313,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.70499999999999996,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.41899999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.373,"5min":0.309,"15min":0.309},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.47099999999999997,"15min":0.46300000000000002},"min":{"1min":0.35199999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73699999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.48999999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:05:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.36299999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":436744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:57 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.34399999999999997,"15min":0.31},"min":{"1min":0.216,"5min":0.14699999999999999,"15min":0.14699999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.37,"15min":0.32500000000000001},"min":{"1min":0.22900000000000001,"5min":0.16,"15min":0.16},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.36499999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.374},"min":{"1min":0.19500000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.63700000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.41899999999999998,"15min":0.40500000000000003},"min":{"1min":0.31900000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.40799999999999997}]},{"osd":3,"last update":"Mon Feb 5 08:05:54 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.434,"15min":0.441},"min":{"1min":0.27200000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.38200000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.59099999999999997,"15min":0.58799999999999997},"min":{"1min":0.375,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.44},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78500000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Feb 5 08:05:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.55000000000000004},"min":{"1min":0.433,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.56999999999999995,"15min":0.56000000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.45000000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:06:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.438,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82299999999999995,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63700000000000001,"15min":0.63600000000000001},"min":{"1min":0.45200000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.47899999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.58299999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28500,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716652,"statfs":{"total":95995035648,"available":95965851648,"internally_reserved":0,"allocated":1519616,"data_stored":1027147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:06:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.36199999999999999,"15min":0.33400000000000002},"min":{"1min":0.25,"5min":0.152,"15min":0.152},"max":{"1min":0.501,"5min":0.621,"15min":0.621},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.37,"15min":0.33600000000000002},"min":{"1min":0.26500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:05:28 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38500000000000001,"15min":0.372},"min":{"1min":0.252,"5min":0.218,"15min":0.218},"max":{"1min":0.60899999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40300000000000002,"15min":0.38900000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.35099999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:05:55 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.45200000000000001,"15min":0.45900000000000002},"min":{"1min":0.29599999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71899999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.46300000000000002,"15min":0.46800000000000003},"min":{"1min":0.32800000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.58199999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.38600000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:06:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56299999999999994,"15min":0.56399999999999995},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.71299999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.754,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56200000000000006}]},{"osd":5,"last update":"Mon Feb 5 08:05:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60499999999999998,"15min":0.60599999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.61299999999999999,"15min":0.621},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.41599999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.621,"15min":0.621},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.50800000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:05:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52000000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":436186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:24 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.38,"15min":0.37},"min":{"1min":0.27200000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.57099999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.40799999999999997,"5min":0.39900000000000002,"15min":0.38200000000000001},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56899999999999995,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.34799999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:05:24 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.36799999999999999,"15min":0.35899999999999999},"min":{"1min":0.248,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.495,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.40200000000000002,"15min":0.38},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.623,"5min":0.63,"15min":0.63},"last":0.315}]},{"osd":3,"last update":"Mon Feb 5 08:05:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46999999999999997,"15min":0.46700000000000003},"min":{"1min":0.34499999999999997,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.58699999999999997,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.378},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.56599999999999995,"5min":0.746,"15min":0.746},"last":0.36199999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:06:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.374,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71599999999999997,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.43099999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.69999999999999996,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.436}]},{"osd":5,"last update":"Mon Feb 5 08:05:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.57699999999999996,"15min":0.57399999999999995},"min":{"1min":0.42399999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.46100000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.63400000000000001,"15min":0.63600000000000001},"min":{"1min":0.435,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.80600000000000005,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63500000000000001,"15min":0.63900000000000001},"min":{"1min":0.42299999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83899999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.47199999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.52800000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117037,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":436889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":31914441},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40899999999999997,"15min":0.41099999999999998},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.347,"5min":1.347,"15min":1.347},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.438,"15min":0.45100000000000001},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.44900000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.46800000000000003,"15min":0.48199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.70299999999999996,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.25},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.45000000000000001,"15min":0.46200000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.30599999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:05:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49099999999999999,"15min":0.497},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.377},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.47499999999999998,"15min":0.48399999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.52,"5min":1.52,"15min":1.52},"last":0.42799999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:05:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.41299999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.61499999999999999,"15min":0.60499999999999998},"min":{"1min":0.46400000000000002,"5min":0.439,"15min":0.439},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":0.36099999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:06:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.47599999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58499999999999996,"15min":0.58099999999999996},"min":{"1min":0.48899999999999999,"5min":0.438,"15min":0.438},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.54500000000000004}]},{"osd":7,"last update":"Mon Feb 5 08:05:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.503}]}]},{"osd":4,"up_from":32,"seq":137438953512,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":435729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.748,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.629}]},{"osd":1,"last update":"Mon Feb 5 08:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53700000000000003,"15min":0.54400000000000004},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.37}]},{"osd":2,"last update":"Mon Feb 5 08:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54000000000000004,"15min":0.54300000000000004},"min":{"1min":0.38400000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.56399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Feb 5 08:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.40699999999999997,"5min":0.376,"15min":0.376},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.442}]},{"osd":5,"last update":"Mon Feb 5 08:05:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.38800000000000001,"15min":0.372},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.495},{"interface":"front","average":{"1min":0.497,"5min":0.41299999999999998,"15min":0.39900000000000002},"min":{"1min":0.29099999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.58899999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:06:06 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.443,"15min":0.42899999999999999},"min":{"1min":0.311,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69899999999999995,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.443,"15min":0.42499999999999999},"min":{"1min":0.35199999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.56200000000000006}]},{"osd":7,"last update":"Mon Feb 5 08:05:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.51600000000000001}]}]},{"osd":5,"up_from":42,"seq":180388626466,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":901120,"data_stored":435436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.33100000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.72899999999999998,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.53200000000000003,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71299999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.502}]},{"osd":1,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.56599999999999995,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.374,"15min":0.374},"max":{"1min":0.69999999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.35299999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.376}]},{"osd":2,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56599999999999995,"15min":0.57099999999999995},"min":{"1min":0.34899999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76200000000000001,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.373,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.46400000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55600000000000005,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70099999999999996,"5min":0.749,"15min":0.749},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.59099999999999997},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.41999999999999998,"15min":0.41799999999999998},"min":{"1min":0.28199999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.60599999999999998,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.35799999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:06:01 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45400000000000001,"15min":0.44600000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.42299999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.51000000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:05:30 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.442},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:06:21.180 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-05T08:06:21.180 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd dump 2024-02-05T08:06:21.419 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:21 smithi196 ceph-mon[33018]: 4.11 scrub starts 2024-02-05T08:06:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:21 smithi196 ceph-mon[33018]: 4.11 scrub ok 2024-02-05T08:06:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:21 smithi196 ceph-mon[33018]: 6.19 scrub starts 2024-02-05T08:06:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:21 smithi196 ceph-mon[33018]: 6.19 scrub ok 2024-02-05T08:06:21.816 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:21 smithi134 ceph-mon[30086]: 4.11 scrub starts 2024-02-05T08:06:21.816 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:21 smithi134 ceph-mon[30086]: 4.11 scrub ok 2024-02-05T08:06:21.816 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:21 smithi134 ceph-mon[30086]: 6.19 scrub starts 2024-02-05T08:06:21.816 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:21 smithi134 ceph-mon[30086]: 6.19 scrub ok 2024-02-05T08:06:21.816 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:21 smithi134 ceph-mon[39278]: 4.11 scrub starts 2024-02-05T08:06:21.816 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:21 smithi134 ceph-mon[39278]: 4.11 scrub ok 2024-02-05T08:06:21.816 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:21 smithi134 ceph-mon[39278]: 6.19 scrub starts 2024-02-05T08:06:21.817 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:21 smithi134 ceph-mon[39278]: 6.19 scrub ok 2024-02-05T08:06:22.540 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:22 smithi196 systemd[1]: Starting Ceph grafana.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:06:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:22 smithi196 ceph-mon[33018]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:06:22.814 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:22 smithi196 podman[66657]: 2024-02-05 08:06:22.569573187 +0000 UTC m=+0.017875082 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-02-05T08:06:22.815 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:22 smithi196 podman[66657]: 2024-02-05 08:06:22.677485065 +0000 UTC m=+0.125786966 container create d8ab63d57982625ccf5a94ae5003cf52815b91117fe62b7feaff9058531a98b9 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a, distribution-scope=public, io.openshift.tags=base rhel8, architecture=x86_64, maintainer=Paul Cuzner , io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, vcs-type=git, description=Ceph Grafana Container, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc.) 2024-02-05T08:06:22.872 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:22 smithi134 ceph-mon[30086]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:22.872 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:06:22.872 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:22 smithi134 ceph-mon[39278]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:22.873 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:06:23.137 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:06:22 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:06:23.138 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:22 smithi196 podman[66657]: 2024-02-05 08:06:22.912969055 +0000 UTC m=+0.361270943 container init d8ab63d57982625ccf5a94ae5003cf52815b91117fe62b7feaff9058531a98b9 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, architecture=x86_64, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, build-date=2022-04-29T11:48:11.055897, vcs-type=git, name=ubi8, release=239.1651231664, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container) 2024-02-05T08:06:23.138 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:22 smithi196 podman[66657]: 2024-02-05 08:06:22.918235217 +0000 UTC m=+0.366537114 container start d8ab63d57982625ccf5a94ae5003cf52815b91117fe62b7feaff9058531a98b9 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, architecture=x86_64, io.buildah.version=1.32.0, maintainer=Paul Cuzner , name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, release=239.1651231664) 2024-02-05T08:06:23.138 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 bash[66657]: d8ab63d57982625ccf5a94ae5003cf52815b91117fe62b7feaff9058531a98b9 2024-02-05T08:06:23.138 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 systemd[1]: Started Ceph grafana.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:06:23.138 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074648859Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-02-05T08:06:23.138 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074847834Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-02-05T08:06:23.138 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074865043Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-02-05T08:06:23.138 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074871986Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-02-05T08:06:23.138 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074878715Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-05T08:06:23.138 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074885272Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-05T08:06:23.138 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074890825Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074896455Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074902504Z level=info msg="Path Home" path=/usr/share/grafana 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074907853Z level=info msg="Path Data" path=/var/lib/grafana 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074913195Z level=info msg="Path Logs" path=/var/log/grafana 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074918452Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074924032Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=settings t=2024-02-05T08:06:23.074929561Z level=info msg="App mode production" 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=sqlstore t=2024-02-05T08:06:23.07497207Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=sqlstore t=2024-02-05T08:06:23.074987008Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.075860824Z level=info msg="Starting DB migrations" 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.075936233Z level=info msg="Executing migration" id="create migration_log table" 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.076438669Z level=info msg="Executing migration" id="create user table" 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.076869483Z level=info msg="Executing migration" id="add unique index user.login" 2024-02-05T08:06:23.139 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.077724803Z level=info msg="Executing migration" id="add unique index user.email" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.078201618Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.079035005Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.079854455Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.080572691Z level=info msg="Executing migration" id="create user table v2" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.080973122Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.081351292Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.08178996Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.082119907Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.082490701Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.082913833Z level=info msg="Executing migration" id="Update user table charset" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.083192463Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.083653012Z level=info msg="Executing migration" id="Add missing user data" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.084022667Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-02-05T08:06:23.140 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.084422751Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-02-05T08:06:23.141 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.084841702Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-02-05T08:06:23.141 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.085246432Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-02-05T08:06:23.141 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.086480728Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-02-05T08:06:23.141 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.08692088Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-02-05T08:06:23.141 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.087375196Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-02-05T08:06:23.141 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.087780033Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-02-05T08:06:23.141 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.08818372Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-02-05T08:06:23.141 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.088644781Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-02-05T08:06:23.142 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.088955098Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-02-05T08:06:23.142 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.089312299Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-02-05T08:06:23.142 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.089736507Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-02-05T08:06:23.142 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.09009042Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-02-05T08:06:23.142 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.090489737Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-05T08:06:23.142 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.091116391Z level=info msg="Executing migration" id="create temp_user v2" 2024-02-05T08:06:23.142 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.091528046Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-02-05T08:06:23.142 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.091930153Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-02-05T08:06:23.142 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.092332386Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-02-05T08:06:23.142 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.092769461Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-02-05T08:06:23.142 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.0931734Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.093520141Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.093868212Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.094176362Z level=info msg="Executing migration" id="create star table" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.094557491Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.094965723Z level=info msg="Executing migration" id="create org table v1" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.09537161Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.095784515Z level=info msg="Executing migration" id="create org_user table v1" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.096158614Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.096593188Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.097005591Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.097399181Z level=info msg="Executing migration" id="Update org table charset" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.097692049Z level=info msg="Executing migration" id="Update org_user table charset" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.09796644Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-02-05T08:06:23.143 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.098252196Z level=info msg="Executing migration" id="create dashboard table" 2024-02-05T08:06:23.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.09863489Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-02-05T08:06:23.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.099027827Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-02-05T08:06:23.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.099428193Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-02-05T08:06:23.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.099810004Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-02-05T08:06:23.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.100827452Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-05T08:06:23.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.101543155Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-02-05T08:06:23.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.102410928Z level=info msg="Executing migration" id="create dashboard v2" 2024-02-05T08:06:23.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.103282355Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-02-05T08:06:23.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.104122967Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-02-05T08:06:23.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.104985078Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-02-05T08:06:23.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.10583714Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-02-05T08:06:23.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.106406595Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-02-05T08:06:23.144 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.107227067Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.10774964Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.108612522Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.109477497Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.110316309Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.111105741Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.112088764Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.112866746Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.113173859Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.113486792Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.11400432Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.114833885Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.115376654Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.118235395Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-02-05T08:06:23.145 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.121041904Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-02-05T08:06:23.146 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.124098248Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-02-05T08:06:23.146 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.126678163Z level=info msg="Executing migration" id="Update dashboard title length" 2024-02-05T08:06:23.146 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.129256868Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-05T08:06:23.146 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.132049375Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-02-05T08:06:23.146 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.135005799Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-05T08:06:23.146 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.137578049Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-02-05T08:06:23.297 INFO:teuthology.orchestra.run.smithi134.stdout:epoch 91 2024-02-05T08:06:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T08:06:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:created 2024-02-05T07:59:09.321262+0000 2024-02-05T08:06:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:modified 2024-02-05T08:05:59.522143+0000 2024-02-05T08:06:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-05T08:06:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:crush_version 18 2024-02-05T08:06:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:full_ratio 0.95 2024-02-05T08:06:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:backfillfull_ratio 0.9 2024-02-05T08:06:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:nearfull_ratio 0.85 2024-02-05T08:06:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:require_min_compat_client luminous 2024-02-05T08:06:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:min_compat_client luminous 2024-02-05T08:06:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:require_osd_release squid 2024-02-05T08:06:23.298 INFO:teuthology.orchestra.run.smithi134.stdout:stretch_mode_enabled false 2024-02-05T08:06:23.299 INFO:teuthology.orchestra.run.smithi134.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-02-05T08:06:23.299 INFO:teuthology.orchestra.run.smithi134.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 70 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-02-05T08:06:23.299 INFO:teuthology.orchestra.run.smithi134.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-02-05T08:06:23.299 INFO:teuthology.orchestra.run.smithi134.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-02-05T08:06:23.299 INFO:teuthology.orchestra.run.smithi134.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 72 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-02-05T08:06:23.299 INFO:teuthology.orchestra.run.smithi134.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-02-05T08:06:23.299 INFO:teuthology.orchestra.run.smithi134.stdout:max_osd 8 2024-02-05T08:06:23.299 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6802/2540271081,v1:172.21.15.134:6803/2540271081] [v2:172.21.15.134:6804/2540271081,v1:172.21.15.134:6805/2540271081] exists,up d90a5ea0-9a19-452e-9564-ca7b7a7c45e4 2024-02-05T08:06:23.299 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960] [v2:172.21.15.134:6812/2032802960,v1:172.21.15.134:6813/2032802960] exists,up 4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4 2024-02-05T08:06:23.299 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2 up in weight 1 up_from 18 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6818/153827987,v1:172.21.15.134:6819/153827987] [v2:172.21.15.134:6820/153827987,v1:172.21.15.134:6821/153827987] exists,up 078e578b-855b-49ff-83bd-0548a9c5a708 2024-02-05T08:06:23.299 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3 up in weight 1 up_from 27 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6826/1527507056,v1:172.21.15.134:6827/1527507056] [v2:172.21.15.134:6828/1527507056,v1:172.21.15.134:6829/1527507056] exists,up 6774e330-3202-4671-894e-172cc2f6189d 2024-02-05T08:06:23.299 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4 up in weight 1 up_from 32 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6800/3005156808,v1:172.21.15.196:6801/3005156808] [v2:172.21.15.196:6802/3005156808,v1:172.21.15.196:6803/3005156808] exists,up 52f70cfa-ab0e-4751-a5f8-1045c61c1693 2024-02-05T08:06:23.299 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5 up in weight 1 up_from 42 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6808/1603224792,v1:172.21.15.196:6809/1603224792] [v2:172.21.15.196:6810/1603224792,v1:172.21.15.196:6811/1603224792] exists,up b4b21769-d40f-43fd-b7e9-2477b74d3955 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6 up in weight 1 up_from 51 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6816/3727413143,v1:172.21.15.196:6817/3727413143] [v2:172.21.15.196:6818/3727413143,v1:172.21.15.196:6819/3727413143] exists,up af1cced4-0015-47f5-b105-27641bfe4f4a 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7 up in weight 1 up_from 60 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6824/4206571010,v1:172.21.15.196:6825/4206571010] [v2:172.21.15.196:6826/4206571010,v1:172.21.15.196:6827/4206571010] exists,up c54beeae-9f7c-4723-a369-13bb531b4057 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 4.7 [1,2] 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 4.a [1,2] 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:pg_upmap_items 4.1f [1,2] 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/788812699 expires 2024-02-06T08:05:21.465844+0000 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/1046500326 expires 2024-02-06T07:59:36.275039+0000 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/464208089 expires 2024-02-06T07:59:36.275039+0000 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/301941854 expires 2024-02-06T07:59:36.275039+0000 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:6800/3607168099 expires 2024-02-06T08:00:02.709280+0000 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:6800/673876301 expires 2024-02-06T07:59:36.275039+0000 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/4225724311 expires 2024-02-06T08:00:02.709280+0000 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:6800/1525062017 expires 2024-02-06T08:05:21.465844+0000 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:6801/3607168099 expires 2024-02-06T08:00:02.709280+0000 2024-02-05T08:06:23.300 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:6801/673876301 expires 2024-02-06T07:59:36.275039+0000 2024-02-05T08:06:23.301 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/3200884805 expires 2024-02-06T08:05:21.465844+0000 2024-02-05T08:06:23.301 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/2952444615 expires 2024-02-06T08:00:02.709280+0000 2024-02-05T08:06:23.301 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/3325835814 expires 2024-02-06T08:05:21.465844+0000 2024-02-05T08:06:23.301 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/3161126612 expires 2024-02-06T08:00:02.709280+0000 2024-02-05T08:06:23.301 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/69097091 expires 2024-02-06T08:05:21.465844+0000 2024-02-05T08:06:23.301 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:6801/1525062017 expires 2024-02-06T08:05:21.465844+0000 2024-02-05T08:06:23.301 INFO:teuthology.orchestra.run.smithi134.stdout:blocklist 172.21.15.134:0/1645475987 expires 2024-02-06T08:05:21.465844+0000 2024-02-05T08:06:23.388 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.140451767Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-05T08:06:23.388 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.143404619Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-05T08:06:23.388 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.146202439Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-02-05T08:06:23.388 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.148892017Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.151590331Z level=info msg="Executing migration" id="Add check_sum column" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.154213957Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.156988063Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.159843511Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.162381612Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.165246156Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.168111671Z level=info msg="Executing migration" id="create data_source table" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.170813387Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.173318446Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.173803625Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.176441139Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.176876357Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.17918493Z level=info msg="Executing migration" id="create data_source table v2" 2024-02-05T08:06:23.389 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.18190812Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-02-05T08:06:23.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.184451294Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-02-05T08:06:23.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.187373368Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-02-05T08:06:23.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.189867705Z level=info msg="Executing migration" id="Add column with_credentials" 2024-02-05T08:06:23.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.192683327Z level=info msg="Executing migration" id="Add secure json data column" 2024-02-05T08:06:23.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.195411193Z level=info msg="Executing migration" id="Update data_source table charset" 2024-02-05T08:06:23.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.198149554Z level=info msg="Executing migration" id="Update initial version to 1" 2024-02-05T08:06:23.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.200791626Z level=info msg="Executing migration" id="Add read_only data column" 2024-02-05T08:06:23.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.203633304Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-02-05T08:06:23.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.20629069Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-02-05T08:06:23.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.208909574Z level=info msg="Executing migration" id="Add uid column" 2024-02-05T08:06:23.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.209757241Z level=info msg="Executing migration" id="Update uid value" 2024-02-05T08:06:23.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.210687313Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-02-05T08:06:23.390 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.211408807Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-02-05T08:06:23.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.212416088Z level=info msg="Executing migration" id="create api_key table" 2024-02-05T08:06:23.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.213055278Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-02-05T08:06:23.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.213873403Z level=info msg="Executing migration" id="add index api_key.key" 2024-02-05T08:06:23.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.214728742Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-02-05T08:06:23.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.215553939Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-02-05T08:06:23.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.216517593Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-02-05T08:06:23.391 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.217206996Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-02-05T08:06:23.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.218006347Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-02-05T08:06:23.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.219085239Z level=info msg="Executing migration" id="create api_key table v2" 2024-02-05T08:06:23.392 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.219983436Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.220660998Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.221438979Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.22187995Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.222200935Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.222688324Z level=info msg="Executing migration" id="Update api_key table charset" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.223493835Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.22436278Z level=info msg="Executing migration" id="Add service account foreign key" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.224947349Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.225748296Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.226374125Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.226960109Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.227754773Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.228167762Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-02-05T08:06:23.393 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.228573063Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.229630938Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.230283618Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.231125331Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.231407423Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.231704955Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.23225892Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.232829296Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.233128701Z level=info msg="Executing migration" id="create quota table v1" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.233545428Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.234017629Z level=info msg="Executing migration" id="Update quota table charset" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.234769157Z level=info msg="Executing migration" id="create plugin_setting table" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.235212918Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.235643018Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-02-05T08:06:23.394 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.236473252Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-02-05T08:06:23.395 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.237466841Z level=info msg="Executing migration" id="create session table" 2024-02-05T08:06:23.395 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.238100368Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-02-05T08:06:23.395 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.238795282Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-02-05T08:06:23.395 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.239113546Z level=info msg="Executing migration" id="create playlist table v2" 2024-02-05T08:06:23.395 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.23961294Z level=info msg="Executing migration" id="create playlist item table v2" 2024-02-05T08:06:23.395 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.240577504Z level=info msg="Executing migration" id="Update playlist table charset" 2024-02-05T08:06:23.395 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.240906043Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-02-05T08:06:23.395 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.241732928Z level=info msg="Executing migration" id="drop preferences table v2" 2024-02-05T08:06:23.395 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.2420613Z level=info msg="Executing migration" id="drop preferences table v3" 2024-02-05T08:06:23.395 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.242577776Z level=info msg="Executing migration" id="create preferences table v3" 2024-02-05T08:06:23.395 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.243011826Z level=info msg="Executing migration" id="Update preferences table charset" 2024-02-05T08:06:23.395 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.243468574Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-02-05T08:06:23.395 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.244076848Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.244943841Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.245798052Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.246420054Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.247407583Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.248100116Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.248565293Z level=info msg="Executing migration" id="create alert table v1" 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.249398016Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.250240156Z level=info msg="Executing migration" id="add index alert state" 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.251169747Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.251802794Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.252739368Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.25558204Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.258097632Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-05T08:06:23.396 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.25952806Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.262415967Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.265128305Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.267833041Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.268216827Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.271129524Z level=info msg="Executing migration" id="Add column is_default" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.273665255Z level=info msg="Executing migration" id="Add column frequency" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.276520247Z level=info msg="Executing migration" id="Add column send_reminder" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.277322042Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.280367743Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.283202426Z level=info msg="Executing migration" id="Update alert table charset" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.285746328Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.286092801Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.288896858Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-05T08:06:23.397 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.291786042Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.294460701Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.294861886Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.297571163Z level=info msg="Executing migration" id="Add for to alert table" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.298280895Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.300270789Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.30290954Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.30336006Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.306194492Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.308784968Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.311497843Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.312064703Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.314768758Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.31765795Z level=info msg="Executing migration" id="create annotation table v5" 2024-02-05T08:06:23.398 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.320208636Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.320671602Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.323205771Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.326050115Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.328724183Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.329348973Z level=info msg="Executing migration" id="Update annotation table charset" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.330121736Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.330973243Z level=info msg="Executing migration" id="Drop category_id index" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.331823995Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.33254152Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.333537338Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.334153459Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.335115409Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.336896674Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-02-05T08:06:23.399 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.337795287Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.338449884Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.339284387Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.339679908Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.340511715Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.341391367Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.342086202Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.342897236Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.343368797Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.344217804Z level=info msg="Executing migration" id="Add epoch_end column" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.345038158Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.345894492Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.34623807Z level=info msg="Executing migration" id="Move region to single row" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.347092814Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-02-05T08:06:23.400 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.348031811Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.348750732Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.349198613Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.34996722Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.350811031Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.351642152Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.351980952Z level=info msg="Executing migration" id="create test_data table" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.35285761Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.353645902Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.354496118Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.35505038Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.355867407Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.356233587Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.358816127Z level=info msg="Executing migration" id="create team table" 2024-02-05T08:06:23.401 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.361660954Z level=info msg="Executing migration" id="add index team.org_id" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.362294433Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.365083206Z level=info msg="Executing migration" id="create team member table" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.367964342Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.370538404Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.371000235Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.373793625Z level=info msg="Executing migration" id="Add column email to team table" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.376426611Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.379204108Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.380046894Z level=info msg="Executing migration" id="create dashboard acl table" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.382922515Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.38543016Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.388233292Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.388739206Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-02-05T08:06:23.402 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.391596538Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-02-05T08:06:23.627 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[30086]: 5.7 scrub starts 2024-02-05T08:06:23.627 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[30086]: 5.7 scrub ok 2024-02-05T08:06:23.627 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:23.627 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:23.627 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:23.627 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:23.627 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:23.627 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[30086]: Deploying daemon node-exporter.a on smithi134 2024-02-05T08:06:23.627 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3543537374' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-05T08:06:23.627 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[39278]: 5.7 scrub starts 2024-02-05T08:06:23.628 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[39278]: 5.7 scrub ok 2024-02-05T08:06:23.628 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:23.628 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:23.628 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:23.628 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:23.628 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:23.628 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[39278]: Deploying daemon node-exporter.a on smithi134 2024-02-05T08:06:23.628 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:23 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3543537374' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-05T08:06:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-mon[33018]: 5.7 scrub starts 2024-02-05T08:06:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-mon[33018]: 5.7 scrub ok 2024-02-05T08:06:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-mon[33018]: Deploying daemon node-exporter.a on smithi134 2024-02-05T08:06:23.638 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3543537374' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-05T08:06:23.639 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.394384401Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-02-05T08:06:23.639 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.397144643Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-02-05T08:06:23.639 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.399872908Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-02-05T08:06:23.639 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.402623173Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-02-05T08:06:23.639 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.40526026Z level=info msg="Executing migration" id="create tag table" 2024-02-05T08:06:23.639 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.407773647Z level=info msg="Executing migration" id="add index tag.key_value" 2024-02-05T08:06:23.640 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.410419887Z level=info msg="Executing migration" id="create login attempt table" 2024-02-05T08:06:23.640 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.413215055Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-02-05T08:06:23.640 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.415904523Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-02-05T08:06:23.640 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.418441798Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-05T08:06:23.640 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.421079212Z level=info msg="Executing migration" id="create login_attempt v2" 2024-02-05T08:06:23.640 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.423594355Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-02-05T08:06:23.640 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.426409734Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-02-05T08:06:23.640 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.428971584Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-02-05T08:06:23.640 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.431785963Z level=info msg="Executing migration" id="create user auth table" 2024-02-05T08:06:23.641 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.434371541Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-05T08:06:23.641 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.437215295Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-02-05T08:06:23.641 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.439859943Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-02-05T08:06:23.641 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.440603319Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-02-05T08:06:23.641 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.443456699Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-02-05T08:06:23.641 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.446098083Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-02-05T08:06:23.641 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.448969894Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-02-05T08:06:23.641 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.451550798Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-02-05T08:06:23.641 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.454265425Z level=info msg="Executing migration" id="create server_lock table" 2024-02-05T08:06:23.642 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.45693398Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-02-05T08:06:23.642 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.459724307Z level=info msg="Executing migration" id="create user auth token table" 2024-02-05T08:06:23.642 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.462238634Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-02-05T08:06:23.642 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.46492348Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-02-05T08:06:23.642 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.46785269Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-02-05T08:06:23.642 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.470563372Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-02-05T08:06:23.642 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.473008813Z level=info msg="Executing migration" id="create cache_data table" 2024-02-05T08:06:23.642 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.475666414Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-02-05T08:06:23.642 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.478524775Z level=info msg="Executing migration" id="create short_url table v1" 2024-02-05T08:06:23.642 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.481042003Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-02-05T08:06:23.642 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.48380161Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-02-05T08:06:23.642 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.486312776Z level=info msg="Executing migration" id="delete alert_definition table" 2024-02-05T08:06:23.643 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.486652054Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-02-05T08:06:23.643 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.487116722Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-02-05T08:06:23.643 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.489799222Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-02-05T08:06:23.643 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.490257121Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-02-05T08:06:23.643 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.490555022Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-02-05T08:06:23.643 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.49097001Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-02-05T08:06:23.644 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.491364867Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-02-05T08:06:23.644 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.491787146Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-02-05T08:06:23.644 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.492210697Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-02-05T08:06:23.644 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.493019973Z level=info msg="Executing migration" id="drop alert_definition table" 2024-02-05T08:06:23.644 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.493447736Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-02-05T08:06:23.644 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.493747036Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-02-05T08:06:23.645 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.494128486Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-05T08:06:23.645 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.494563948Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-05T08:06:23.645 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.494987681Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-05T08:06:23.645 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.495246887Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-02-05T08:06:23.645 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.495683718Z level=info msg="Executing migration" id="create alert_instance table" 2024-02-05T08:06:23.646 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.496082675Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-05T08:06:23.646 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.496563241Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-05T08:06:23.648 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.496978671Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-02-05T08:06:23.648 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.497796553Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-05T08:06:23.648 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.498200423Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-02-05T08:06:23.648 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.498608646Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-02-05T08:06:23.648 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.501963829Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-02-05T08:06:23.648 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.505924379Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-05T08:06:23.648 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.506357961Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-02-05T08:06:23.648 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.506800748Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-02-05T08:06:23.648 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.507631983Z level=info msg="Executing migration" id="create alert_rule table" 2024-02-05T08:06:23.648 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.50803558Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-02-05T08:06:23.649 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.508446327Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-02-05T08:06:23.649 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.508865653Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-05T08:06:23.649 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.509266997Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-02-05T08:06:23.649 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.509562171Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-02-05T08:06:23.649 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.510342174Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-02-05T08:06:23.649 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.511128126Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-02-05T08:06:23.649 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.511978913Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-02-05T08:06:23.649 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.512386574Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-05T08:06:23.649 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.512802507Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-02-05T08:06:23.650 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.513616951Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-02-05T08:06:23.650 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.514419871Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-05T08:06:23.650 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.514844346Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-02-05T08:06:23.650 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.515651295Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-02-05T08:06:23.650 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.516440992Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-02-05T08:06:23.650 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.516753292Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-02-05T08:06:23.650 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.517184503Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-05T08:06:23.651 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.517609604Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-05T08:06:23.651 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.518028185Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-05T08:06:23.651 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.518297369Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-02-05T08:06:23.651 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.519158194Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-02-05T08:06:23.651 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.520007388Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-02-05T08:06:23.651 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.520862932Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-02-05T08:06:23.651 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.521716654Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-02-05T08:06:23.651 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.522540944Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-02-05T08:06:23.651 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.52285875Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-02-05T08:06:23.651 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.523234943Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-02-05T08:06:23.651 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.524081325Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-05T08:06:23.652 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.524378998Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-02-05T08:06:23.652 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.525253629Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-02-05T08:06:23.652 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.525690158Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-02-05T08:06:23.652 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.526532566Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-02-05T08:06:23.652 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.527499225Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-02-05T08:06:23.652 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.528311194Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-02-05T08:06:23.652 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.529319276Z level=info msg="Executing migration" id="create provenance_type table" 2024-02-05T08:06:23.652 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.530344953Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-02-05T08:06:23.652 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.531021444Z level=info msg="Executing migration" id="create alert_image table" 2024-02-05T08:06:23.652 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.531859897Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-02-05T08:06:23.653 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.532280997Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-02-05T08:06:23.653 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.533061702Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-02-05T08:06:23.653 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.533541573Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-02-05T08:06:23.653 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.534319107Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-02-05T08:06:23.653 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.534410461Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-02-05T08:06:23.653 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.534683878Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-02-05T08:06:23.653 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.535470913Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-02-05T08:06:23.653 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.535900265Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-02-05T08:06:23.653 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.5360182Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-02-05T08:06:23.653 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.536123604Z level=info msg="alerts found to migrate" alerts=0 2024-02-05T08:06:23.653 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.536524878Z level=info msg="Executing migration" id="create library_element table v1" 2024-02-05T08:06:23.653 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.537475357Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-02-05T08:06:23.654 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.538065235Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-02-05T08:06:23.654 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.539033031Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-02-05T08:06:23.654 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.53974573Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-02-05T08:06:23.654 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.540572408Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-02-05T08:06:23.654 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.540865232Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-02-05T08:06:23.654 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.541728222Z level=info msg="Executing migration" id="create data_keys table" 2024-02-05T08:06:23.654 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.542155247Z level=info msg="Executing migration" id="create secrets table" 2024-02-05T08:06:23.654 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.542606187Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-02-05T08:06:23.654 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.546930299Z level=info msg="Executing migration" id="add name column into data_keys" 2024-02-05T08:06:23.654 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.547847509Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-02-05T08:06:23.654 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.548680098Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-02-05T08:06:23.654 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.553025454Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-02-05T08:06:23.655 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.557291376Z level=info msg="Executing migration" id="create kv_store table v1" 2024-02-05T08:06:23.655 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.558202138Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-02-05T08:06:23.655 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.558916921Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-02-05T08:06:23.655 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.559792795Z level=info msg="Executing migration" id="create permission table" 2024-02-05T08:06:23.655 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.560452795Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-02-05T08:06:23.655 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.560878683Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-02-05T08:06:23.655 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.561735695Z level=info msg="Executing migration" id="create role table" 2024-02-05T08:06:23.655 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.564601503Z level=info msg="Executing migration" id="add column display_name" 2024-02-05T08:06:23.655 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.567308327Z level=info msg="Executing migration" id="add column group_name" 2024-02-05T08:06:23.655 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.568278984Z level=info msg="Executing migration" id="add index role.org_id" 2024-02-05T08:06:23.655 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.571003144Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-02-05T08:06:23.655 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.573590869Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-02-05T08:06:23.655 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.576297084Z level=info msg="Executing migration" id="create team role table" 2024-02-05T08:06:23.656 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.576742922Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-02-05T08:06:23.656 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.579443031Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-02-05T08:06:23.656 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.582244371Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-02-05T08:06:23.656 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.585043024Z level=info msg="Executing migration" id="create user role table" 2024-02-05T08:06:23.656 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.587931795Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-02-05T08:06:23.656 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.59065591Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-02-05T08:06:23.656 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.593454722Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-02-05T08:06:23.656 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.596168366Z level=info msg="Executing migration" id="create builtin role table" 2024-02-05T08:06:23.656 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.596678107Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-02-05T08:06:23.656 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.599352891Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-02-05T08:06:23.656 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.602062552Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-02-05T08:06:23.656 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.604711022Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-02-05T08:06:23.656 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.605243033Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.607894069Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.610307386Z level=info msg="Executing migration" id="add unique index role.uid" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.613150755Z level=info msg="Executing migration" id="create seed assignment table" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.613708749Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.616348162Z level=info msg="Executing migration" id="add column hidden to role table" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.619081918Z level=info msg="Executing migration" id="create query_history table v1" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.621573789Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.622002267Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.624737817Z level=info msg="Executing migration" id="teams permissions migration" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.62533463Z level=info msg="Executing migration" id="dashboard permissions" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.625790019Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.626578854Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.626888665Z level=info msg="Executing migration" id="alerting notification permissions" 2024-02-05T08:06:23.657 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.627219176Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-02-05T08:06:23.658 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.627617881Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-02-05T08:06:23.658 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.628087685Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-02-05T08:06:23.658 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.629134248Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-02-05T08:06:23.658 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.629932123Z level=info msg="Executing migration" id="create correlation table v1" 2024-02-05T08:06:23.658 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.630895921Z level=info msg="Executing migration" id="add index correlations.uid" 2024-02-05T08:06:23.658 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.631535009Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-02-05T08:06:23.658 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.632371795Z level=info msg="Executing migration" id="add correlation config column" 2024-02-05T08:06:23.658 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.633354991Z level=info msg="Executing migration" id="create entity_events table" 2024-02-05T08:06:23.658 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.633773584Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-02-05T08:06:23.658 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.634552737Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-05T08:06:23.658 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.63463741Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-05T08:06:23.658 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.635404321Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-05T08:06:23.658 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.635518832Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-05T08:06:23.659 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.636174218Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-02-05T08:06:23.659 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.636637995Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-02-05T08:06:23.659 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.637629099Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-02-05T08:06:23.659 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.638367471Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-05T08:06:23.659 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.639215232Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-02-05T08:06:23.659 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.639671497Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-05T08:06:23.659 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.64052497Z level=info msg="Executing migration" id="Drop public config table" 2024-02-05T08:06:23.659 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.641310998Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-02-05T08:06:23.659 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.642162626Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-02-05T08:06:23.659 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.642661745Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-05T08:06:23.659 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.643452524Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-02-05T08:06:23.939 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6802/2540271081,v1:172.21.15.134:6803/2540271081] [v2:172.21.15.134:6804/2540271081,v1:172.21.15.134:6805/2540271081] exists,up d90a5ea0-9a19-452e-9564-ca7b7a7c45e4', 'osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6810/2032802960,v1:172.21.15.134:6811/2032802960] [v2:172.21.15.134:6812/2032802960,v1:172.21.15.134:6813/2032802960] exists,up 4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4', 'osd.2 up in weight 1 up_from 18 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6818/153827987,v1:172.21.15.134:6819/153827987] [v2:172.21.15.134:6820/153827987,v1:172.21.15.134:6821/153827987] exists,up 078e578b-855b-49ff-83bd-0548a9c5a708', 'osd.3 up in weight 1 up_from 27 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.134:6826/1527507056,v1:172.21.15.134:6827/1527507056] [v2:172.21.15.134:6828/1527507056,v1:172.21.15.134:6829/1527507056] exists,up 6774e330-3202-4671-894e-172cc2f6189d', 'osd.4 up in weight 1 up_from 32 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6800/3005156808,v1:172.21.15.196:6801/3005156808] [v2:172.21.15.196:6802/3005156808,v1:172.21.15.196:6803/3005156808] exists,up 52f70cfa-ab0e-4751-a5f8-1045c61c1693', 'osd.5 up in weight 1 up_from 42 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6808/1603224792,v1:172.21.15.196:6809/1603224792] [v2:172.21.15.196:6810/1603224792,v1:172.21.15.196:6811/1603224792] exists,up b4b21769-d40f-43fd-b7e9-2477b74d3955', 'osd.6 up in weight 1 up_from 51 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6816/3727413143,v1:172.21.15.196:6817/3727413143] [v2:172.21.15.196:6818/3727413143,v1:172.21.15.196:6819/3727413143] exists,up af1cced4-0015-47f5-b105-27641bfe4f4a', 'osd.7 up in weight 1 up_from 60 up_thru 72 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6824/4206571010,v1:172.21.15.196:6825/4206571010] [v2:172.21.15.196:6826/4206571010,v1:172.21.15.196:6827/4206571010] exists,up c54beeae-9f7c-4723-a369-13bb531b4057'] 2024-02-05T08:06:23.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon mon.a config show 2024-02-05T08:06:24.065 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.644302232Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-02-05T08:06:24.065 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.647318318Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-02-05T08:06:24.065 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.648359226Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-02-05T08:06:24.065 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.649379438Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-02-05T08:06:24.065 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.650229454Z level=info msg="Executing migration" id="add share column" 2024-02-05T08:06:24.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.65123791Z level=info msg="Executing migration" id="create default alerting folders" 2024-02-05T08:06:24.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.651706668Z level=info msg="Executing migration" id="create file table" 2024-02-05T08:06:24.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.652689784Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-02-05T08:06:24.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.655520909Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-02-05T08:06:24.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.65826038Z level=info msg="Executing migration" id="create file_meta table" 2024-02-05T08:06:24.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.65876475Z level=info msg="Executing migration" id="file table idx: path key" 2024-02-05T08:06:24.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.661521939Z level=info msg="Executing migration" id="set path collation in file table" 2024-02-05T08:06:24.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.664126615Z level=info msg="Executing migration" id="managed permissions migration" 2024-02-05T08:06:24.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.666964943Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-02-05T08:06:24.066 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.667370476Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-02-05T08:06:24.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.670184766Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-02-05T08:06:24.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.672947115Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-02-05T08:06:24.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.675530355Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-02-05T08:06:24.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.67833564Z level=info msg="Executing migration" id="update group index for alert rules" 2024-02-05T08:06:24.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.681070046Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-02-05T08:06:24.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.683827758Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-02-05T08:06:24.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.686500913Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-02-05T08:06:24.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.687568931Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-02-05T08:06:24.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.690270592Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-02-05T08:06:24.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.692983109Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-02-05T08:06:24.067 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.702665335Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-02-05T08:06:24.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.70314397Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-02-05T08:06:24.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.705780057Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-02-05T08:06:24.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.709334973Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-02-05T08:06:24.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.712067202Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-02-05T08:06:24.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.714764396Z level=info msg="Executing migration" id="create folder table" 2024-02-05T08:06:24.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.71729962Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-02-05T08:06:24.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.719910184Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-02-05T08:06:24.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.722605244Z level=info msg="Executing migration" id="Update folder title length" 2024-02-05T08:06:24.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.725207021Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-02-05T08:06:24.068 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=migrator t=2024-02-05T08:06:23.727740467Z level=info msg="migrations completed" performed=485 skipped=0 duration=651.825025ms 2024-02-05T08:06:24.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=sqlstore t=2024-02-05T08:06:23.728165483Z level=info msg="Created default organization" 2024-02-05T08:06:24.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=plugin.loader t=2024-02-05T08:06:23.785653117Z level=info msg="Plugin registered" pluginID=input 2024-02-05T08:06:24.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=plugin.loader t=2024-02-05T08:06:23.81501748Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-02-05T08:06:24.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=plugin.loader t=2024-02-05T08:06:23.815045466Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-02-05T08:06:24.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=secrets t=2024-02-05T08:06:23.815082611Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-02-05T08:06:24.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=query_data t=2024-02-05T08:06:23.817218885Z level=info msg="Query Service initialization" 2024-02-05T08:06:24.069 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=live.push_http t=2024-02-05T08:06:23.819683647Z level=info msg="Live Push Gateway initialization" 2024-02-05T08:06:24.194 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:24.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:06:24.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:06:24.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-mon[33018]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:24.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-mon[33018]: 6.12 scrub starts 2024-02-05T08:06:24.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-mon[33018]: 6.12 scrub ok 2024-02-05T08:06:24.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-mon[33018]: osdmap e92: 8 total, 8 up, 8 in 2024-02-05T08:06:24.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:06:24.711 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:06:24.712 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=infra.usagestats.collector t=2024-02-05T08:06:24.446604112Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-02-05T08:06:24.712 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=provisioning.datasources t=2024-02-05T08:06:24.447680936Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-05T08:06:24.712 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=provisioning.datasources t=2024-02-05T08:06:24.461083623Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-02-05T08:06:24.712 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=provisioning.alerting t=2024-02-05T08:06:24.471957969Z level=info msg="starting to provision alerting" 2024-02-05T08:06:24.712 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=provisioning.alerting t=2024-02-05T08:06:24.471977242Z level=info msg="finished to provision alerting" 2024-02-05T08:06:24.712 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=ngalert.state.manager t=2024-02-05T08:06:24.472078541Z level=info msg="Warming state cache for startup" 2024-02-05T08:06:24.712 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=grafanaStorageLogger t=2024-02-05T08:06:24.472118428Z level=info msg="storage starting" 2024-02-05T08:06:24.712 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=http.server t=2024-02-05T08:06:24.475091245Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-02-05T08:06:24.712 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=ngalert.state.manager t=2024-02-05T08:06:24.483622418Z level=info msg="State cache has been initialized" states=0 duration=11.543119ms 2024-02-05T08:06:24.712 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=ticker t=2024-02-05T08:06:24.483668399Z level=info msg=starting first_tick=2024-02-05T08:06:30Z 2024-02-05T08:06:24.713 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=ngalert.multiorg.alertmanager t=2024-02-05T08:06:24.483682592Z level=info msg="starting MultiOrg Alertmanager" 2024-02-05T08:06:24.713 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:06:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=sqlstore.transactions t=2024-02-05T08:06:24.494091258Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-05T08:06:24.969 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:06:24.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:06:24.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[30086]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:24.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[30086]: 6.12 scrub starts 2024-02-05T08:06:24.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[30086]: 6.12 scrub ok 2024-02-05T08:06:24.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[30086]: osdmap e92: 8 total, 8 up, 8 in 2024-02-05T08:06:24.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:06:24.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:06:24.970 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:06:24.970 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:06:24.970 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[39278]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:24.971 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[39278]: 6.12 scrub starts 2024-02-05T08:06:24.971 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[39278]: 6.12 scrub ok 2024-02-05T08:06:24.971 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[39278]: osdmap e92: 8 total, 8 up, 8 in 2024-02-05T08:06:24.971 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:06:24.971 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:24 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:06:25.635 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:25 smithi134 ceph-mon[30086]: 6.4 scrub starts 2024-02-05T08:06:25.635 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:25 smithi134 ceph-mon[30086]: 6.4 scrub ok 2024-02-05T08:06:25.635 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:25 smithi134 ceph-mon[30086]: osdmap e93: 8 total, 8 up, 8 in 2024-02-05T08:06:25.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:25 smithi134 ceph-mon[39278]: 6.4 scrub starts 2024-02-05T08:06:25.636 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:25 smithi134 ceph-mon[39278]: 6.4 scrub ok 2024-02-05T08:06:25.636 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:25 smithi134 ceph-mon[39278]: osdmap e93: 8 total, 8 up, 8 in 2024-02-05T08:06:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:25 smithi196 ceph-mon[33018]: 6.4 scrub starts 2024-02-05T08:06:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:25 smithi196 ceph-mon[33018]: 6.4 scrub ok 2024-02-05T08:06:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:25 smithi196 ceph-mon[33018]: osdmap e93: 8 total, 8 up, 8 in 2024-02-05T08:06:25.892 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:06:25.892 INFO:teuthology.orchestra.run.smithi134.stdout: "name": "mon.a", 2024-02-05T08:06:25.892 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster": "ceph", 2024-02-05T08:06:25.892 INFO:teuthology.orchestra.run.smithi134.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-02-05T08:06:25.892 INFO:teuthology.orchestra.run.smithi134.stdout: "admin_socket_mode": "", 2024-02-05T08:06:25.892 INFO:teuthology.orchestra.run.smithi134.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-02-05T08:06:25.892 INFO:teuthology.orchestra.run.smithi134.stdout: "auth_client_required": "cephx, none", 2024-02-05T08:06:25.892 INFO:teuthology.orchestra.run.smithi134.stdout: "auth_cluster_required": "cephx", 2024-02-05T08:06:25.892 INFO:teuthology.orchestra.run.smithi134.stdout: "auth_debug": "true", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "auth_service_required": "cephx", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "auth_service_ticket_ttl": "240.000000", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "auth_supported": "", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_aio": "true", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_aio_max_queue_depth": "1024", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_aio_poll_ms": "250", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_aio_reap_max": "16", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_async_discard": "false", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_block_size": "4096", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_debug_aio": "false", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-02-05T08:06:25.893 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_debug_inflight_ios": "false", 2024-02-05T08:06:25.894 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_enable_discard": "false", 2024-02-05T08:06:25.894 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_flock_retry": "3", 2024-02-05T08:06:25.894 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_flock_retry_interval": "0.100000", 2024-02-05T08:06:25.894 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_inject_crash": "0", 2024-02-05T08:06:25.894 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_inject_crash_flush_delay": "2", 2024-02-05T08:06:25.894 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_ioring": "false", 2024-02-05T08:06:25.894 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_ioring_hipri": "false", 2024-02-05T08:06:25.894 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_ioring_sqthread_poll": "false", 2024-02-05T08:06:25.894 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-02-05T08:06:25.894 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_read_buffer_alignment": "4096", 2024-02-05T08:06:25.894 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-02-05T08:06:25.894 INFO:teuthology.orchestra.run.smithi134.stdout: "bdev_type": "", 2024-02-05T08:06:25.894 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_alloc_size": "1048576", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_allocator": "hybrid", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_buffered_io": "true", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_check_for_zeros": "false", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_check_volume_selector_often": "false", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_compact_log_sync": "false", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_log_compact_min_size": "16777216", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_log_replay_check_allocations": "true", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_max_log_runway": "4194304", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_max_prefetch": "1048576", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_min_flush_size": "524288", 2024-02-05T08:06:25.895 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_min_log_runway": "1048576", 2024-02-05T08:06:25.896 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_replay_recovery": "false", 2024-02-05T08:06:25.896 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-02-05T08:06:25.896 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_shared_alloc_size": "65536", 2024-02-05T08:06:25.896 INFO:teuthology.orchestra.run.smithi134.stdout: "bluefs_sync_write": "false", 2024-02-05T08:06:25.896 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-02-05T08:06:25.896 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-02-05T08:06:25.896 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-02-05T08:06:25.896 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_allocation_from_file": "true", 2024-02-05T08:06:25.896 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_allocator": "hybrid", 2024-02-05T08:06:25.896 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-02-05T08:06:25.896 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-02-05T08:06:25.897 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-02-05T08:06:25.897 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-02-05T08:06:25.897 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-02-05T08:06:25.897 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-02-05T08:06:25.897 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_blobid_prealloc": "10240", 2024-02-05T08:06:25.897 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_block_create": "true", 2024-02-05T08:06:25.897 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_block_db_create": "false", 2024-02-05T08:06:25.897 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_block_db_path": "", 2024-02-05T08:06:25.897 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_block_db_size": "0", 2024-02-05T08:06:25.897 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_block_path": "", 2024-02-05T08:06:25.898 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_block_preallocate_file": "false", 2024-02-05T08:06:25.898 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_block_size": "107374182400", 2024-02-05T08:06:25.898 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_block_wal_create": "false", 2024-02-05T08:06:25.898 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_block_wal_path": "", 2024-02-05T08:06:25.898 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_block_wal_size": "100663296", 2024-02-05T08:06:25.898 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_bluefs": "true", 2024-02-05T08:06:25.898 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-02-05T08:06:25.898 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_bluefs_env_mirror": "false", 2024-02-05T08:06:25.898 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-02-05T08:06:25.898 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-02-05T08:06:25.898 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-02-05T08:06:25.898 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-02-05T08:06:25.898 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-02-05T08:06:25.899 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-02-05T08:06:25.899 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_autotune": "true", 2024-02-05T08:06:25.899 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-02-05T08:06:25.899 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-02-05T08:06:25.899 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-02-05T08:06:25.899 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-02-05T08:06:25.899 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_size": "0", 2024-02-05T08:06:25.899 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-02-05T08:06:25.899 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-02-05T08:06:25.899 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-02-05T08:06:25.899 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-02-05T08:06:25.900 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cache_type": "2q", 2024-02-05T08:06:25.900 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-02-05T08:06:25.900 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_clone_cow": "true", 2024-02-05T08:06:25.900 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_compression_algorithm": "snappy", 2024-02-05T08:06:25.900 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_compression_max_blob_size": "0", 2024-02-05T08:06:25.900 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-02-05T08:06:25.900 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-02-05T08:06:25.900 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_compression_min_blob_size": "0", 2024-02-05T08:06:25.900 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-02-05T08:06:25.900 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-02-05T08:06:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_compression_mode": "none", 2024-02-05T08:06:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-02-05T08:06:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_csum_type": "crc32c", 2024-02-05T08:06:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_enforce_settings": "default", 2024-02-05T08:06:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_freelist": "false", 2024-02-05T08:06:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_fsck_abort": "false", 2024-02-05T08:06:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-02-05T08:06:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-02-05T08:06:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_inject_read_err": "false", 2024-02-05T08:06:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_legacy_omap": "false", 2024-02-05T08:06:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-02-05T08:06:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-02-05T08:06:25.901 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_prefill": "0.000000", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_small_allocations": "0", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_default_buffered_read": "true", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_default_buffered_write": "false", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_deferred_batch_ops": "0", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_elastic_shared_blobs": "true", 2024-02-05T08:06:25.902 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fail_eio": "false", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_on_mkfs": "true", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_on_mount": "false", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-02-05T08:06:25.903 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_on_umount": "false", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_ignore_data_csum": "false", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_kvbackend": "rocksdb", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-02-05T08:06:25.904 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_log_op_age": "5.000000", 2024-02-05T08:06:25.905 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_max_alloc_size": "0", 2024-02-05T08:06:25.905 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_max_blob_size": "0", 2024-02-05T08:06:25.905 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-02-05T08:06:25.905 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-02-05T08:06:25.905 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_max_defer_interval": "3.000000", 2024-02-05T08:06:25.905 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_max_deferred_txc": "32", 2024-02-05T08:06:25.905 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_min_alloc_size": "0", 2024-02-05T08:06:25.905 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-02-05T08:06:25.905 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-02-05T08:06:25.905 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_nid_prealloc": "1024", 2024-02-05T08:06:25.906 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_prefer_deferred_size": "0", 2024-02-05T08:06:25.906 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-02-05T08:06:25.906 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-02-05T08:06:25.906 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_qfsck_on_mount": "true", 2024-02-05T08:06:25.906 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_retry_disk_reads": "3", 2024-02-05T08:06:25.906 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_rocksdb_cf": "true", 2024-02-05T08:06:25.906 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-02-05T08:06:25.907 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-02-05T08:06:25.907 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_rocksdb_options_annex": "", 2024-02-05T08:06:25.907 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_spdk_coremask": "0x1", 2024-02-05T08:06:25.907 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_spdk_io_sleep": "5", 2024-02-05T08:06:25.907 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_spdk_max_io_completion": "0", 2024-02-05T08:06:25.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_spdk_mem": "512", 2024-02-05T08:06:25.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_sync_submit_transaction": "false", 2024-02-05T08:06:25.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_throttle_bytes": "67108864", 2024-02-05T08:06:25.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_throttle_cost_per_io": "0", 2024-02-05T08:06:25.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-02-05T08:06:25.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-02-05T08:06:25.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-02-05T08:06:25.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-02-05T08:06:25.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_tracing": "false", 2024-02-05T08:06:25.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-02-05T08:06:25.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-02-05T08:06:25.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_volume_selection_reserved": "0", 2024-02-05T08:06:25.908 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "bluestore_zero_block_detection": "false", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "cephfs_mirror_action_update_interval": "2", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "cephfs_mirror_mount_timeout": "10", 2024-02-05T08:06:25.909 INFO:teuthology.orchestra.run.smithi134.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "cephx_cluster_require_signatures": "false", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "cephx_cluster_require_version": "2", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "cephx_require_signatures": "false", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "cephx_require_version": "2", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "cephx_service_require_signatures": "false", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "cephx_service_require_version": "2", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "cephx_sign_messages": "true", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "chdir": "", 2024-02-05T08:06:25.910 INFO:teuthology.orchestra.run.smithi134.stdout: "client_acl_type": "", 2024-02-05T08:06:25.911 INFO:teuthology.orchestra.run.smithi134.stdout: "client_asio_thread_count": "2", 2024-02-05T08:06:25.911 INFO:teuthology.orchestra.run.smithi134.stdout: "client_cache_mid": "0.750000", 2024-02-05T08:06:25.911 INFO:teuthology.orchestra.run.smithi134.stdout: "client_cache_size": "16384", 2024-02-05T08:06:25.911 INFO:teuthology.orchestra.run.smithi134.stdout: "client_caps_release_delay": "5", 2024-02-05T08:06:25.911 INFO:teuthology.orchestra.run.smithi134.stdout: "client_check_pool_perm": "true", 2024-02-05T08:06:25.911 INFO:teuthology.orchestra.run.smithi134.stdout: "client_collect_and_send_global_metrics": "false", 2024-02-05T08:06:25.911 INFO:teuthology.orchestra.run.smithi134.stdout: "client_debug_force_sync_read": "false", 2024-02-05T08:06:25.911 INFO:teuthology.orchestra.run.smithi134.stdout: "client_debug_getattr_caps": "false", 2024-02-05T08:06:25.911 INFO:teuthology.orchestra.run.smithi134.stdout: "client_debug_inject_tick_delay": "0", 2024-02-05T08:06:25.911 INFO:teuthology.orchestra.run.smithi134.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-02-05T08:06:25.911 INFO:teuthology.orchestra.run.smithi134.stdout: "client_die_on_failed_remount": "false", 2024-02-05T08:06:25.911 INFO:teuthology.orchestra.run.smithi134.stdout: "client_dirsize_rbytes": "true", 2024-02-05T08:06:25.911 INFO:teuthology.orchestra.run.smithi134.stdout: "client_force_lazyio": "false", 2024-02-05T08:06:25.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_fs": "", 2024-02-05T08:06:25.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_inject_fixed_oldest_tid": "false", 2024-02-05T08:06:25.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_inject_release_failure": "false", 2024-02-05T08:06:25.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_max_inline_size": "4096", 2024-02-05T08:06:25.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_max_retries_on_remount_failure": "5", 2024-02-05T08:06:25.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_mds_namespace": "", 2024-02-05T08:06:25.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_metadata": "", 2024-02-05T08:06:25.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_mount_gid": "-1", 2024-02-05T08:06:25.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_mount_timeout": "300", 2024-02-05T08:06:25.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_mount_uid": "-1", 2024-02-05T08:06:25.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_mountpoint": "/", 2024-02-05T08:06:25.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_notify_timeout": "10", 2024-02-05T08:06:25.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_oc": "true", 2024-02-05T08:06:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: "client_oc_max_dirty": "104857600", 2024-02-05T08:06:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: "client_oc_max_dirty_age": "5.000000", 2024-02-05T08:06:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: "client_oc_max_objects": "1000", 2024-02-05T08:06:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: "client_oc_size": "209715200", 2024-02-05T08:06:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: "client_oc_target_dirty": "8388608", 2024-02-05T08:06:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: "client_permissions": "true", 2024-02-05T08:06:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: "client_quota": "true", 2024-02-05T08:06:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: "client_quota_df": "true", 2024-02-05T08:06:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: "client_readahead_max_bytes": "0", 2024-02-05T08:06:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: "client_readahead_max_periods": "4", 2024-02-05T08:06:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: "client_readahead_min": "131072", 2024-02-05T08:06:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: "client_reconnect_stale": "false", 2024-02-05T08:06:25.913 INFO:teuthology.orchestra.run.smithi134.stdout: "client_shutdown_timeout": "30", 2024-02-05T08:06:25.914 INFO:teuthology.orchestra.run.smithi134.stdout: "client_snapdir": ".snap", 2024-02-05T08:06:25.914 INFO:teuthology.orchestra.run.smithi134.stdout: "client_tick_interval": "1", 2024-02-05T08:06:25.914 INFO:teuthology.orchestra.run.smithi134.stdout: "client_trace": "", 2024-02-05T08:06:25.914 INFO:teuthology.orchestra.run.smithi134.stdout: "client_try_dentry_invalidate": "false", 2024-02-05T08:06:25.914 INFO:teuthology.orchestra.run.smithi134.stdout: "client_use_faked_inos": "false", 2024-02-05T08:06:25.914 INFO:teuthology.orchestra.run.smithi134.stdout: "client_use_random_mds": "false", 2024-02-05T08:06:25.914 INFO:teuthology.orchestra.run.smithi134.stdout: "clog_to_graylog": "false", 2024-02-05T08:06:25.914 INFO:teuthology.orchestra.run.smithi134.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-02-05T08:06:25.914 INFO:teuthology.orchestra.run.smithi134.stdout: "clog_to_graylog_port": "12201", 2024-02-05T08:06:25.914 INFO:teuthology.orchestra.run.smithi134.stdout: "clog_to_monitors": "default=true", 2024-02-05T08:06:25.914 INFO:teuthology.orchestra.run.smithi134.stdout: "clog_to_syslog": "false", 2024-02-05T08:06:25.914 INFO:teuthology.orchestra.run.smithi134.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-02-05T08:06:25.915 INFO:teuthology.orchestra.run.smithi134.stdout: "clog_to_syslog_level": "info", 2024-02-05T08:06:25.915 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "-", 2024-02-05T08:06:25.915 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_network": "", 2024-02-05T08:06:25.915 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_network_interface": "", 2024-02-05T08:06:25.915 INFO:teuthology.orchestra.run.smithi134.stdout: "compressor_zlib_isal": "false", 2024-02-05T08:06:25.915 INFO:teuthology.orchestra.run.smithi134.stdout: "compressor_zlib_level": "5", 2024-02-05T08:06:25.915 INFO:teuthology.orchestra.run.smithi134.stdout: "compressor_zlib_winsize": "-15", 2024-02-05T08:06:25.915 INFO:teuthology.orchestra.run.smithi134.stdout: "compressor_zstd_level": "1", 2024-02-05T08:06:25.915 INFO:teuthology.orchestra.run.smithi134.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0", 2024-02-05T08:06:25.915 INFO:teuthology.orchestra.run.smithi134.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-02-05T08:06:25.915 INFO:teuthology.orchestra.run.smithi134.stdout: "crimson_alien_op_num_threads": "6", 2024-02-05T08:06:25.915 INFO:teuthology.orchestra.run.smithi134.stdout: "crimson_alien_thread_cpu_cores": "", 2024-02-05T08:06:25.915 INFO:teuthology.orchestra.run.smithi134.stdout: "crimson_osd_obc_lru_size": "512", 2024-02-05T08:06:25.916 INFO:teuthology.orchestra.run.smithi134.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-02-05T08:06:25.916 INFO:teuthology.orchestra.run.smithi134.stdout: "crimson_seastar_cpu_cores": "", 2024-02-05T08:06:25.916 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_location": "", 2024-02-05T08:06:25.916 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_location_hook": "", 2024-02-05T08:06:25.916 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_location_hook_timeout": "10", 2024-02-05T08:06:25.916 INFO:teuthology.orchestra.run.smithi134.stdout: "daemonize": "false", 2024-02-05T08:06:25.916 INFO:teuthology.orchestra.run.smithi134.stdout: "daos_pool": "tank", 2024-02-05T08:06:25.916 INFO:teuthology.orchestra.run.smithi134.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-02-05T08:06:25.916 INFO:teuthology.orchestra.run.smithi134.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-02-05T08:06:25.916 INFO:teuthology.orchestra.run.smithi134.stdout: "dbstore_db_name_prefix": "dbstore", 2024-02-05T08:06:25.916 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_alienstore": "0/5", 2024-02-05T08:06:25.916 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_allow_any_pool_priority": "false", 2024-02-05T08:06:25.916 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_asok": "1/5", 2024-02-05T08:06:25.917 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_asok_assert_abort": "false", 2024-02-05T08:06:25.917 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_asserts_on_shutdown": "true", 2024-02-05T08:06:25.917 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_auth": "1/5", 2024-02-05T08:06:25.917 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_bdev": "1/3", 2024-02-05T08:06:25.917 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_bluefs": "1/5", 2024-02-05T08:06:25.917 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_bluestore": "1/5", 2024-02-05T08:06:25.917 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_buffer": "0/1", 2024-02-05T08:06:25.917 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_ceph_exporter": "1/5", 2024-02-05T08:06:25.917 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_cephfs_mirror": "0/5", 2024-02-05T08:06:25.917 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_cephsqlite": "0/5", 2024-02-05T08:06:25.918 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_client": "0/5", 2024-02-05T08:06:25.918 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_compressor": "1/5", 2024-02-05T08:06:25.918 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_context": "0/1", 2024-02-05T08:06:25.918 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_crimson_interrupt": "0/5", 2024-02-05T08:06:25.918 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_crush": "1/1", 2024-02-05T08:06:25.918 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_crypto": "1/5", 2024-02-05T08:06:25.918 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_cyanstore": "0/5", 2024-02-05T08:06:25.918 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_deliberately_leak_memory": "false", 2024-02-05T08:06:25.918 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_disable_randomized_ping": "false", 2024-02-05T08:06:25.918 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_dpdk": "1/5", 2024-02-05T08:06:25.918 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_eventtrace": "1/5", 2024-02-05T08:06:25.918 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_filer": "0/1", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_filestore": "1/3", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_finisher": "1/1", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_fuse": "1/5", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_heartbeat_testing_span": "0", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_heartbeatmap": "1/5", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_immutable_obj_cache": "0/5", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_javaclient": "1/5", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_journal": "1/3", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_journaler": "0/5", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_kstore": "1/5", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_lockdep": "0/1", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_mclock": "1/5", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_mds": "1/5", 2024-02-05T08:06:25.919 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_mds_balancer": "1/5", 2024-02-05T08:06:25.920 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_mds_locker": "1/5", 2024-02-05T08:06:25.920 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_mds_log": "1/5", 2024-02-05T08:06:25.920 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_mds_log_expire": "1/5", 2024-02-05T08:06:25.920 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_mds_migrator": "1/5", 2024-02-05T08:06:25.920 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_memstore": "1/5", 2024-02-05T08:06:25.920 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_mgr": "2/5", 2024-02-05T08:06:25.920 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_mgrc": "1/5", 2024-02-05T08:06:25.920 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_mon": "20/20", 2024-02-05T08:06:25.920 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_monc": "0/10", 2024-02-05T08:06:25.920 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_ms": "1/1", 2024-02-05T08:06:25.920 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_none": "0/5", 2024-02-05T08:06:25.921 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_objclass": "0/5", 2024-02-05T08:06:25.921 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_objectcacher": "0/5", 2024-02-05T08:06:25.921 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_objecter": "0/1", 2024-02-05T08:06:25.921 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_optracker": "0/5", 2024-02-05T08:06:25.921 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_osd": "1/5", 2024-02-05T08:06:25.921 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_paxos": "20/20", 2024-02-05T08:06:25.921 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_perfcounter": "1/5", 2024-02-05T08:06:25.921 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_prioritycache": "1/5", 2024-02-05T08:06:25.921 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_rados": "0/5", 2024-02-05T08:06:25.921 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_rbd": "0/5", 2024-02-05T08:06:25.921 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_rbd_mirror": "0/5", 2024-02-05T08:06:25.921 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_rbd_pwl": "0/5", 2024-02-05T08:06:25.921 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_rbd_replay": "0/5", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_refs": "0/0", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_reserver": "1/1", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_rgw": "1/5", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_rgw_access": "1/5", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_rgw_datacache": "1/5", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_rgw_dbstore": "1/5", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_rgw_flight": "1/5", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_rgw_lifecycle": "1/5", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_rgw_sync": "1/5", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_rocksdb": "4/5", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore": "0/5", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore_backref": "0/5", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore_cache": "0/5", 2024-02-05T08:06:25.922 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore_cleaner": "0/5", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore_device": "0/5", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore_epm": "0/5", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore_journal": "0/5", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore_lba": "0/5", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore_odata": "0/5", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore_omap": "0/5", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore_onode": "0/5", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore_t": "0/5", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_seastore_tm": "0/5", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_striper": "0/1", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_test": "0/5", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_throttle": "1/1", 2024-02-05T08:06:25.923 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_timer": "0/1", 2024-02-05T08:06:25.924 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_tp": "0/5", 2024-02-05T08:06:25.924 INFO:teuthology.orchestra.run.smithi134.stdout: "debug_trace": "1/5", 2024-02-05T08:06:25.924 INFO:teuthology.orchestra.run.smithi134.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-02-05T08:06:25.924 INFO:teuthology.orchestra.run.smithi134.stdout: "device_failure_prediction_mode": "none", 2024-02-05T08:06:25.924 INFO:teuthology.orchestra.run.smithi134.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-02-05T08:06:25.924 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-02-05T08:06:25.924 INFO:teuthology.orchestra.run.smithi134.stdout: "err_to_graylog": "false", 2024-02-05T08:06:25.924 INFO:teuthology.orchestra.run.smithi134.stdout: "err_to_journald": "false", 2024-02-05T08:06:25.924 INFO:teuthology.orchestra.run.smithi134.stdout: "err_to_stderr": "true", 2024-02-05T08:06:25.924 INFO:teuthology.orchestra.run.smithi134.stdout: "err_to_syslog": "false", 2024-02-05T08:06:25.924 INFO:teuthology.orchestra.run.smithi134.stdout: "event_tracing": "false", 2024-02-05T08:06:25.924 INFO:teuthology.orchestra.run.smithi134.stdout: "exporter_addr": "0.0.0.0", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "exporter_http_port": "9926", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "exporter_prio_limit": "5", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "exporter_sort_metrics": "true", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "exporter_stats_period": "5", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "fake_statfs_for_testing": "0", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "fatal_signal_handlers": "true", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "filer_max_purge_ops": "10", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "filer_max_truncate_ops": "128", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_apply_finisher_threads": "1", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_blackhole": "false", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_btrfs_clone_range": "true", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_btrfs_snap": "true", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_caller_concurrency": "10", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_collect_device_partition_information": "true", 2024-02-05T08:06:25.925 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_commit_timeout": "600.000000", 2024-02-05T08:06:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_debug_inject_read_err": "false", 2024-02-05T08:06:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_debug_omap_check": "false", 2024-02-05T08:06:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_debug_random_read_err": "0.000000", 2024-02-05T08:06:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_debug_verify_split": "false", 2024-02-05T08:06:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_dump_file": "", 2024-02-05T08:06:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-02-05T08:06:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-02-05T08:06:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_fadvise": "true", 2024-02-05T08:06:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_fail_eio": "true", 2024-02-05T08:06:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_fd_cache_shards": "16", 2024-02-05T08:06:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_fd_cache_size": "128", 2024-02-05T08:06:25.926 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_fiemap": "false", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_fiemap_threshold": "4096", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_index_retry_probability": "0.000000", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_inject_stall": "0", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_journal_parallel": "false", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_journal_trailing": "false", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_journal_writeahead": "false", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_kill_at": "0", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_inline_xattr_size": "0", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-02-05T08:06:25.927 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_inline_xattrs": "0", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_inline_xattrs_other": "2", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_sync_interval": "5.000000", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_xattr_value_size": "0", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_merge_threshold": "-10", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_min_sync_interval": "0.010000", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_odsync_write": "false", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_omap_backend": "rocksdb", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_omap_backend_path": "", 2024-02-05T08:06:25.928 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_omap_header_cache_size": "1024", 2024-02-05T08:06:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_ondisk_finisher_threads": "1", 2024-02-05T08:06:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-02-05T08:06:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_op_thread_timeout": "60", 2024-02-05T08:06:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_op_threads": "2", 2024-02-05T08:06:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_punch_hole": "false", 2024-02-05T08:06:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-02-05T08:06:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-02-05T08:06:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-02-05T08:06:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-02-05T08:06:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-02-05T08:06:25.929 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_queue_max_bytes": "104857600", 2024-02-05T08:06:25.930 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-02-05T08:06:25.930 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-02-05T08:06:25.930 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-02-05T08:06:25.930 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_queue_max_ops": "50", 2024-02-05T08:06:25.930 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-02-05T08:06:25.930 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_seek_data_hole": "false", 2024-02-05T08:06:25.930 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_sloppy_crc": "false", 2024-02-05T08:06:25.930 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-02-05T08:06:25.930 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_splice": "false", 2024-02-05T08:06:25.930 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_split_multiple": "2", 2024-02-05T08:06:25.930 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_split_rand_factor": "20", 2024-02-05T08:06:25.930 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_update_to": "1000", 2024-02-05T08:06:25.931 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-02-05T08:06:25.931 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-02-05T08:06:25.931 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-02-05T08:06:25.931 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-02-05T08:06:25.931 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-02-05T08:06:25.931 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-02-05T08:06:25.931 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_wbthrottle_enable": "true", 2024-02-05T08:06:25.931 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-02-05T08:06:25.931 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-02-05T08:06:25.931 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-02-05T08:06:25.931 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-02-05T08:06:25.931 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-02-05T08:06:25.931 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-02-05T08:06:25.932 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_xfs_extsize": "false", 2024-02-05T08:06:25.932 INFO:teuthology.orchestra.run.smithi134.stdout: "filestore_zfs_snap": "false", 2024-02-05T08:06:25.932 INFO:teuthology.orchestra.run.smithi134.stdout: "fio_dir": "/tmp/fio", 2024-02-05T08:06:25.932 INFO:teuthology.orchestra.run.smithi134.stdout: "fsid": "43e13aa4-c3fc-11ee-95b6-87774f69a715", 2024-02-05T08:06:25.932 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_allow_other": "true", 2024-02-05T08:06:25.932 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_atomic_o_trunc": "true", 2024-02-05T08:06:25.932 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_big_writes": "true", 2024-02-05T08:06:25.932 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_debug": "false", 2024-02-05T08:06:25.932 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_default_permissions": "false", 2024-02-05T08:06:25.932 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_disable_pagecache": "false", 2024-02-05T08:06:25.933 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_max_write": "0", 2024-02-05T08:06:25.933 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_multithreaded": "true", 2024-02-05T08:06:25.933 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_require_active_mds": "true", 2024-02-05T08:06:25.933 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_set_user_groups": "true", 2024-02-05T08:06:25.933 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_splice_move": "true", 2024-02-05T08:06:25.933 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_splice_read": "true", 2024-02-05T08:06:25.933 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_splice_write": "true", 2024-02-05T08:06:25.933 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_syncfs_on_mksnap": "true", 2024-02-05T08:06:25.933 INFO:teuthology.orchestra.run.smithi134.stdout: "fuse_use_invalidate_cb": "true", 2024-02-05T08:06:25.933 INFO:teuthology.orchestra.run.smithi134.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-02-05T08:06:25.933 INFO:teuthology.orchestra.run.smithi134.stdout: "gss_target_name": "ceph", 2024-02-05T08:06:25.933 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_file": "", 2024-02-05T08:06:25.934 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_inject_failure": "0", 2024-02-05T08:06:25.934 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_interval": "5", 2024-02-05T08:06:25.934 INFO:teuthology.orchestra.run.smithi134.stdout: "host": "smithi134", 2024-02-05T08:06:25.934 INFO:teuthology.orchestra.run.smithi134.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-02-05T08:06:25.934 INFO:teuthology.orchestra.run.smithi134.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-02-05T08:06:25.934 INFO:teuthology.orchestra.run.smithi134.stdout: "immutable_object_cache_max_size": "1073741824", 2024-02-05T08:06:25.934 INFO:teuthology.orchestra.run.smithi134.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-02-05T08:06:25.934 INFO:teuthology.orchestra.run.smithi134.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-02-05T08:06:25.934 INFO:teuthology.orchestra.run.smithi134.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-02-05T08:06:25.934 INFO:teuthology.orchestra.run.smithi134.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-02-05T08:06:25.934 INFO:teuthology.orchestra.run.smithi134.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-02-05T08:06:25.934 INFO:teuthology.orchestra.run.smithi134.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-02-05T08:06:25.935 INFO:teuthology.orchestra.run.smithi134.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-02-05T08:06:25.935 INFO:teuthology.orchestra.run.smithi134.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-02-05T08:06:25.935 INFO:teuthology.orchestra.run.smithi134.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-02-05T08:06:25.935 INFO:teuthology.orchestra.run.smithi134.stdout: "immutable_object_cache_watermark": "0.900000", 2024-02-05T08:06:25.935 INFO:teuthology.orchestra.run.smithi134.stdout: "inject_early_sigterm": "false", 2024-02-05T08:06:25.935 INFO:teuthology.orchestra.run.smithi134.stdout: "jaeger_agent_port": "6799", 2024-02-05T08:06:25.935 INFO:teuthology.orchestra.run.smithi134.stdout: "jaeger_tracing_enable": "false", 2024-02-05T08:06:25.935 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_aio": "true", 2024-02-05T08:06:25.935 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_align_min_size": "65536", 2024-02-05T08:06:25.935 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_block_align": "true", 2024-02-05T08:06:25.935 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_block_size": "4096", 2024-02-05T08:06:25.935 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_dio": "true", 2024-02-05T08:06:25.936 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_discard": "false", 2024-02-05T08:06:25.936 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_force_aio": "false", 2024-02-05T08:06:25.936 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_ignore_corruption": "false", 2024-02-05T08:06:25.936 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_max_write_bytes": "10485760", 2024-02-05T08:06:25.936 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_max_write_entries": "100", 2024-02-05T08:06:25.936 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_replay_from": "0", 2024-02-05T08:06:25.936 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_throttle_high_multiple": "0.000000", 2024-02-05T08:06:25.936 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-02-05T08:06:25.936 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-02-05T08:06:25.936 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_throttle_max_multiple": "0.000000", 2024-02-05T08:06:25.936 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_write_header_frequency": "0", 2024-02-05T08:06:25.936 INFO:teuthology.orchestra.run.smithi134.stdout: "journal_zero_on_create": "false", 2024-02-05T08:06:25.937 INFO:teuthology.orchestra.run.smithi134.stdout: "journaler_prefetch_periods": "10", 2024-02-05T08:06:25.937 INFO:teuthology.orchestra.run.smithi134.stdout: "journaler_prezero_periods": "5", 2024-02-05T08:06:25.937 INFO:teuthology.orchestra.run.smithi134.stdout: "journaler_write_head_interval": "15", 2024-02-05T08:06:25.937 INFO:teuthology.orchestra.run.smithi134.stdout: "key": "", 2024-02-05T08:06:25.937 INFO:teuthology.orchestra.run.smithi134.stdout: "keyfile": "", 2024-02-05T08:06:25.937 INFO:teuthology.orchestra.run.smithi134.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-02-05T08:06:25.937 INFO:teuthology.orchestra.run.smithi134.stdout: "kstore_backend": "rocksdb", 2024-02-05T08:06:25.937 INFO:teuthology.orchestra.run.smithi134.stdout: "kstore_default_stripe_size": "65536", 2024-02-05T08:06:25.937 INFO:teuthology.orchestra.run.smithi134.stdout: "kstore_fsck_on_mount": "false", 2024-02-05T08:06:25.937 INFO:teuthology.orchestra.run.smithi134.stdout: "kstore_fsck_on_mount_deep": "true", 2024-02-05T08:06:25.937 INFO:teuthology.orchestra.run.smithi134.stdout: "kstore_max_bytes": "67108864", 2024-02-05T08:06:25.937 INFO:teuthology.orchestra.run.smithi134.stdout: "kstore_max_ops": "512", 2024-02-05T08:06:25.938 INFO:teuthology.orchestra.run.smithi134.stdout: "kstore_nid_prealloc": "1024", 2024-02-05T08:06:25.938 INFO:teuthology.orchestra.run.smithi134.stdout: "kstore_onode_map_size": "1024", 2024-02-05T08:06:25.938 INFO:teuthology.orchestra.run.smithi134.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-02-05T08:06:25.938 INFO:teuthology.orchestra.run.smithi134.stdout: "kstore_sync_submit_transaction": "false", 2024-02-05T08:06:25.938 INFO:teuthology.orchestra.run.smithi134.stdout: "kstore_sync_transaction": "false", 2024-02-05T08:06:25.938 INFO:teuthology.orchestra.run.smithi134.stdout: "librados_thread_count": "2", 2024-02-05T08:06:25.938 INFO:teuthology.orchestra.run.smithi134.stdout: "lockdep": "false", 2024-02-05T08:06:25.938 INFO:teuthology.orchestra.run.smithi134.stdout: "lockdep_force_backtrace": "false", 2024-02-05T08:06:25.938 INFO:teuthology.orchestra.run.smithi134.stdout: "log_coarse_timestamps": "true", 2024-02-05T08:06:25.938 INFO:teuthology.orchestra.run.smithi134.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-02-05T08:06:25.938 INFO:teuthology.orchestra.run.smithi134.stdout: "log_flush_on_exit": "false", 2024-02-05T08:06:25.938 INFO:teuthology.orchestra.run.smithi134.stdout: "log_graylog_host": "127.0.0.1", 2024-02-05T08:06:25.938 INFO:teuthology.orchestra.run.smithi134.stdout: "log_graylog_port": "12201", 2024-02-05T08:06:25.939 INFO:teuthology.orchestra.run.smithi134.stdout: "log_max_new": "1000", 2024-02-05T08:06:25.939 INFO:teuthology.orchestra.run.smithi134.stdout: "log_max_recent": "10000", 2024-02-05T08:06:25.939 INFO:teuthology.orchestra.run.smithi134.stdout: "log_stderr_prefix": "", 2024-02-05T08:06:25.939 INFO:teuthology.orchestra.run.smithi134.stdout: "log_stop_at_utilization": "0.970000", 2024-02-05T08:06:25.939 INFO:teuthology.orchestra.run.smithi134.stdout: "log_to_file": "true", 2024-02-05T08:06:25.939 INFO:teuthology.orchestra.run.smithi134.stdout: "log_to_graylog": "false", 2024-02-05T08:06:25.939 INFO:teuthology.orchestra.run.smithi134.stdout: "log_to_journald": "false", 2024-02-05T08:06:25.939 INFO:teuthology.orchestra.run.smithi134.stdout: "log_to_stderr": "false", 2024-02-05T08:06:25.939 INFO:teuthology.orchestra.run.smithi134.stdout: "log_to_syslog": "false", 2024-02-05T08:06:25.939 INFO:teuthology.orchestra.run.smithi134.stdout: "mandatory_topic_permissions": "false", 2024-02-05T08:06:25.939 INFO:teuthology.orchestra.run.smithi134.stdout: "max_rotating_auth_attempts": "10", 2024-02-05T08:06:25.940 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-02-05T08:06:25.940 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_action_on_write_error": "1", 2024-02-05T08:06:25.940 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_alternate_name_max": "8192", 2024-02-05T08:06:25.940 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_asio_thread_count": "2", 2024-02-05T08:06:25.940 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_export_pin": "true", 2024-02-05T08:06:25.940 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_fragment_dirs": "true", 2024-02-05T08:06:25.940 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-02-05T08:06:25.940 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_fragment_interval": "5", 2024-02-05T08:06:25.940 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_fragment_size_max": "100000", 2024-02-05T08:06:25.940 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_idle_threshold": "0.000000", 2024-02-05T08:06:25.940 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_interval": "10", 2024-02-05T08:06:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_max": "-1", 2024-02-05T08:06:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_max_until": "-1", 2024-02-05T08:06:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_merge_size": "50", 2024-02-05T08:06:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_midchunk": "0.300000", 2024-02-05T08:06:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_min_rebalance": "0.100000", 2024-02-05T08:06:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_min_start": "0.200000", 2024-02-05T08:06:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_minchunk": "0.001000", 2024-02-05T08:06:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_mode": "0", 2024-02-05T08:06:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_need_max": "1.200000", 2024-02-05T08:06:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_need_min": "0.800000", 2024-02-05T08:06:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_overload_epochs": "2", 2024-02-05T08:06:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-02-05T08:06:25.941 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_sample_interval": "3.000000", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_split_bits": "3", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_split_rd": "25000.000000", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_split_size": "10000", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_split_wr": "10000.000000", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_target_decay": "10.000000", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_beacon_grace": "15.000000", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_beacon_interval": "4.000000", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_beacon_mon_down_grace": "60", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_cache_memory_limit": "4294967296", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_cache_mid": "0.700000", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_cache_release_free_interval": "10", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_cache_reservation": "0.050000", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_cache_trim_interval": "1", 2024-02-05T08:06:25.942 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_cache_trim_threshold": "262144", 2024-02-05T08:06:25.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-02-05T08:06:25.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-02-05T08:06:25.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_client_delegate_inos_pct": "50", 2024-02-05T08:06:25.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_client_prealloc_inos": "1000", 2024-02-05T08:06:25.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-02-05T08:06:25.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_connect_bootstrapping": "false", 2024-02-05T08:06:25.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_damage_table_max_entries": "10000", 2024-02-05T08:06:25.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-02-05T08:06:25.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_debug_auth_pins": "false", 2024-02-05T08:06:25.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_debug_frag": "false", 2024-02-05T08:06:25.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_debug_scatterstat": "false", 2024-02-05T08:06:25.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_debug_subtrees": "false", 2024-02-05T08:06:25.943 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_decay_halflife": "5.000000", 2024-02-05T08:06:25.944 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_default_dir_hash": "2", 2024-02-05T08:06:25.944 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_defer_session_stale": "true", 2024-02-05T08:06:25.944 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_deny_all_reconnect": "false", 2024-02-05T08:06:25.944 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_dir_keys_per_op": "16384", 2024-02-05T08:06:25.944 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_dir_max_commit_size": "10", 2024-02-05T08:06:25.944 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_dir_max_entries": "0", 2024-02-05T08:06:25.944 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_dir_prefetch": "true", 2024-02-05T08:06:25.944 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_dirstat_min_interval": "1.000000", 2024-02-05T08:06:25.944 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_dump_cache_after_rejoin": "false", 2024-02-05T08:06:25.944 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_dump_cache_on_map": "false", 2024-02-05T08:06:25.944 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_dump_cache_threshold_file": "0", 2024-02-05T08:06:25.944 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-02-05T08:06:25.944 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_early_reply": "true", 2024-02-05T08:06:25.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_enable_op_tracker": "true", 2024-02-05T08:06:25.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_enforce_unique_name": "true", 2024-02-05T08:06:25.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_export_ephemeral_distributed": "true", 2024-02-05T08:06:25.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-02-05T08:06:25.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_export_ephemeral_random": "true", 2024-02-05T08:06:25.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-02-05T08:06:25.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-02-05T08:06:25.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_forward_all_requests_to_auth": "false", 2024-02-05T08:06:25.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-02-05T08:06:25.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-02-05T08:06:25.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-02-05T08:06:25.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-02-05T08:06:25.945 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_health_cache_threshold": "1.500000", 2024-02-05T08:06:25.946 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_health_summarize_threshold": "10", 2024-02-05T08:06:25.946 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_heartbeat_grace": "15.000000", 2024-02-05T08:06:25.946 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_heartbeat_reset_grace": "1000", 2024-02-05T08:06:25.946 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_inject_health_dummy": "false", 2024-02-05T08:06:25.946 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-02-05T08:06:25.946 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_inject_migrator_session_race": "false", 2024-02-05T08:06:25.946 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-02-05T08:06:25.946 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-02-05T08:06:25.946 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-02-05T08:06:25.946 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_join_fs": "", 2024-02-05T08:06:25.946 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_journal_format": "1", 2024-02-05T08:06:25.946 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_kill_create_at": "0", 2024-02-05T08:06:25.946 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_kill_export_at": "0", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_kill_import_at": "0", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_kill_journal_at": "0", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_kill_journal_expire_at": "0", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_kill_journal_replay_at": "0", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_kill_link_at": "0", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_kill_mdstable_at": "0", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_kill_openc_at": "0", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_kill_rename_at": "0", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_kill_shutdown_at": "0", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log_event_large_threshold": "524288", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log_events_per_segment": "1024", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log_major_segment_event_ratio": "12", 2024-02-05T08:06:25.947 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log_max_events": "-1", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log_max_segments": "128", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log_pause": "false", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log_segment_size": "0", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log_skip_corrupt_events": "false", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log_skip_unbounded_events": "false", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log_trim_threshold": "128", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_log_warn_factor": "2.000000", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_max_caps_per_client": "1048576", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_max_completed_flushes": "100000", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_max_completed_requests": "100000", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_max_export_size": "20971520", 2024-02-05T08:06:25.948 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_max_file_recover": "32", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_max_purge_files": "64", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_max_purge_ops": "8192", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_max_snaps_per_dir": "100", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_metrics_update_interval": "2", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_min_caps_per_client": "100", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_min_caps_working_set": "10000", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_numa_node": "-1", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_op_complaint_time": "30.000000", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_op_history_duration": "600", 2024-02-05T08:06:25.949 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_op_history_size": "20", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_op_history_slow_op_size": "20", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_op_history_slow_op_threshold": "10", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_op_log_threshold": "5", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_ping_grace": "15", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_ping_interval": "5", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_recall_max_caps": "30000", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_recall_max_decay_threshold": "131072", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_recall_warning_threshold": "262144", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_reconnect_timeout": "45.000000", 2024-02-05T08:06:25.950 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_replay_interval": "1.000000", 2024-02-05T08:06:25.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-02-05T08:06:25.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-02-05T08:06:25.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_root_ino_gid": "0", 2024-02-05T08:06:25.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_root_ino_uid": "0", 2024-02-05T08:06:25.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-02-05T08:06:25.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_session_blocklist_on_evict": "true", 2024-02-05T08:06:25.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_session_blocklist_on_timeout": "true", 2024-02-05T08:06:25.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-02-05T08:06:25.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-02-05T08:06:25.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-02-05T08:06:25.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-02-05T08:06:25.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-02-05T08:06:25.951 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_session_metadata_threshold": "16777216", 2024-02-05T08:06:25.952 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-02-05T08:06:25.952 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_shutdown_check": "0", 2024-02-05T08:06:25.952 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_skip_ino": "0", 2024-02-05T08:06:25.952 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_sleep_rank_change": "0.000000", 2024-02-05T08:06:25.952 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_snap_max_uid": "4294967294", 2024-02-05T08:06:25.952 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_snap_min_uid": "0", 2024-02-05T08:06:25.952 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_snap_rstat": "false", 2024-02-05T08:06:25.952 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_standby_replay_damaged": "false", 2024-02-05T08:06:25.952 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_symlink_recovery": "true", 2024-02-05T08:06:25.952 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_task_status_update_interval": "2.000000", 2024-02-05T08:06:25.952 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_thrash_exports": "0", 2024-02-05T08:06:25.952 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_thrash_fragments": "0", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_tick_interval": "5.000000", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_valgrind_exit": "false", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_verify_backtrace": "1", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_verify_scatter": "false", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_wipe_ino_prealloc": "false", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "mds_wipe_sessions": "false", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "mempool_debug": "false", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "memstore_debug_omit_block_device_write": "false", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "memstore_device_bytes": "1073741824", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "memstore_page_set": "false", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "memstore_page_size": "65536", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_client_bytes": "134217728", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_client_messages": "512", 2024-02-05T08:06:25.953 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-02-05T08:06:25.954 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_connect_retry_interval": "1.000000", 2024-02-05T08:06:25.954 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-02-05T08:06:25.954 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-02-05T08:06:25.954 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_disabled_modules": "", 2024-02-05T08:06:25.954 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-02-05T08:06:25.954 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_max_pg_num_change": "128", 2024-02-05T08:06:25.954 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_mds_bytes": "134217728", 2024-02-05T08:06:25.954 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_mds_messages": "128", 2024-02-05T08:06:25.954 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-02-05T08:06:25.954 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_mon_bytes": "134217728", 2024-02-05T08:06:25.956 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_mon_messages": "128", 2024-02-05T08:06:25.956 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_osd_bytes": "536870912", 2024-02-05T08:06:25.956 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_osd_messages": "8192", 2024-02-05T08:06:25.956 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_pool": "true", 2024-02-05T08:06:25.956 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_service_beacon_grace": "60.000000", 2024-02-05T08:06:25.956 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_standby_modules": "true", 2024-02-05T08:06:25.956 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_stats_period": "5", 2024-02-05T08:06:25.956 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_stats_threshold": "5", 2024-02-05T08:06:25.956 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_tick_period": "2", 2024-02-05T08:06:25.956 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-02-05T08:06:25.956 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_accept_timeout_factor": "2.000000", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_allow_pool_delete": "true", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_allow_pool_size_one": "false", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_auth_validate_all_caps": "true", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_client_bytes": "104857600", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_client_directed_command_retry": "2", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_client_hunt_interval": "3.000000", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_client_hunt_parallel": "3", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_client_log_interval": "1.000000", 2024-02-05T08:06:25.957 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_client_ping_interval": "10.000000", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_client_ping_timeout": "30.000000", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_client_target_rank": "-1", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_clock_drift_allowed": "1.000000", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_cluster_log_file_level": "debug", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_cluster_log_to_file": "true", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_cluster_log_to_graylog": "false", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_cluster_log_to_journald": "true", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_cluster_log_to_stderr": "false", 2024-02-05T08:06:25.958 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-02-05T08:06:25.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-02-05T08:06:25.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-02-05T08:06:25.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_compact_on_bootstrap": "false", 2024-02-05T08:06:25.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_compact_on_start": "false", 2024-02-05T08:06:25.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_compact_on_trim": "true", 2024-02-05T08:06:25.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_con_tracker_persist_interval": "10", 2024-02-05T08:06:25.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_con_tracker_score_halflife": "43200", 2024-02-05T08:06:25.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_config_key_max_entry_size": "65536", 2024-02-05T08:06:25.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_cpu_threads": "4", 2024-02-05T08:06:25.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_crush_min_required_version": "hammer", 2024-02-05T08:06:25.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_daemon_bytes": "419430400", 2024-02-05T08:06:25.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-02-05T08:06:25.959 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_data_avail_crit": "5", 2024-02-05T08:06:25.960 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_data_avail_warn": "5", 2024-02-05T08:06:25.960 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_data_size_warn": "16106127360", 2024-02-05T08:06:25.960 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_debug_block_osdmap_trim": "false", 2024-02-05T08:06:25.960 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-02-05T08:06:25.960 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_debug_dump_json": "false", 2024-02-05T08:06:25.960 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-02-05T08:06:25.960 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_debug_dump_transactions": "false", 2024-02-05T08:06:25.960 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_debug_extra_checks": "false", 2024-02-05T08:06:25.960 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-02-05T08:06:25.960 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-02-05T08:06:25.960 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_debug_no_require_reef": "false", 2024-02-05T08:06:25.960 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_debug_no_require_squid": "false", 2024-02-05T08:06:25.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-02-05T08:06:25.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_delta_reset_interval": "10.000000", 2024-02-05T08:06:25.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_dns_srv_name": "ceph-mon", 2024-02-05T08:06:25.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_down_mkfs_grace": "60", 2024-02-05T08:06:25.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_election_default_strategy": "1", 2024-02-05T08:06:25.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_election_timeout": "5.000000", 2024-02-05T08:06:25.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-02-05T08:06:25.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_elector_ping_divisor": "2", 2024-02-05T08:06:25.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_elector_ping_timeout": "2.000000", 2024-02-05T08:06:25.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_enable_op_tracker": "true", 2024-02-05T08:06:25.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_fake_pool_delete": "false", 2024-02-05T08:06:25.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_force_quorum_join": "false", 2024-02-05T08:06:25.961 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_globalid_prealloc": "10000", 2024-02-05T08:06:25.962 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_health_detail_to_clog": "true", 2024-02-05T08:06:25.962 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_health_log_update_period": "5", 2024-02-05T08:06:25.962 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_health_max_detail": "50", 2024-02-05T08:06:25.962 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_health_to_clog": "true", 2024-02-05T08:06:25.962 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_health_to_clog_interval": "600", 2024-02-05T08:06:25.962 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-02-05T08:06:25.962 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_host": "[v2:172.21.15.134:3300/0,v1:172.21.15.134:6789/0]", 2024-02-05T08:06:25.962 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_host_override": "", 2024-02-05T08:06:25.962 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_initial_members": "", 2024-02-05T08:06:25.962 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-02-05T08:06:25.962 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-02-05T08:06:25.962 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-02-05T08:06:25.962 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-02-05T08:06:25.963 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_keyvaluedb": "rocksdb", 2024-02-05T08:06:25.963 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_lease": "5.000000", 2024-02-05T08:06:25.963 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-02-05T08:06:25.963 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-02-05T08:06:25.963 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_log_full_interval": "50", 2024-02-05T08:06:25.963 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_log_max": "10000", 2024-02-05T08:06:25.963 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_log_max_summary": "50", 2024-02-05T08:06:25.963 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_max_log_entries_per_event": "4096", 2024-02-05T08:06:25.963 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_max_log_epochs": "500", 2024-02-05T08:06:25.963 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_max_mdsmap_epochs": "500", 2024-02-05T08:06:25.963 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_max_mgrmap_epochs": "500", 2024-02-05T08:06:25.963 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_max_osd": "10000", 2024-02-05T08:06:25.963 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_max_pg_per_osd": "10000", 2024-02-05T08:06:25.964 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_max_pool_pg_num": "65536", 2024-02-05T08:06:25.964 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-02-05T08:06:25.964 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-02-05T08:06:25.964 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_mds_force_trim_to": "0", 2024-02-05T08:06:25.964 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_mds_skip_sanity": "false", 2024-02-05T08:06:25.964 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_memory_autotune": "true", 2024-02-05T08:06:25.964 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_memory_target": "2147483648", 2024-02-05T08:06:25.964 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_mgr_beacon_grace": "30", 2024-02-05T08:06:25.964 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-02-05T08:06:25.964 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_mgr_digest_period": "5", 2024-02-05T08:06:25.964 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_mgr_inactive_grace": "60", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_mgr_mkfs_grace": "240", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_min_osdmap_epochs": "50", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_op_complaint_time": "30", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_op_history_duration": "600", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_op_history_size": "20", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_op_history_slow_op_size": "20", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_op_history_slow_op_threshold": "10", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_op_log_threshold": "5", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_auto_mark_in": "false", 2024-02-05T08:06:25.965 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_auto_mark_new_in": "true", 2024-02-05T08:06:25.966 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-02-05T08:06:25.966 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-02-05T08:06:25.966 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_cache_size": "500", 2024-02-05T08:06:25.966 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_cache_size_min": "134217728", 2024-02-05T08:06:25.966 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_crush_smoke_test": "true", 2024-02-05T08:06:25.966 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_destroyed_out_interval": "600", 2024-02-05T08:06:25.966 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_down_out_interval": "600", 2024-02-05T08:06:25.966 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-02-05T08:06:25.966 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-02-05T08:06:25.966 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_force_trim_to": "0", 2024-02-05T08:06:25.966 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_full_ratio": "0.950000", 2024-02-05T08:06:25.966 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-02-05T08:06:25.966 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_laggy_halflife": "3600", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_laggy_max_interval": "300", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_laggy_weight": "0.300000", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_max_creating_pgs": "1024", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_max_initial_pgs": "1024", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_min_down_reporters": "2", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_prime_pg_temp": "true", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_report_timeout": "900", 2024-02-05T08:06:25.967 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_warn_num_repaired": "10", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osd_warn_op_age": "32.000000", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osdmap_full_prune_interval": "2", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osdmap_full_prune_min": "15", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_pg_stuck_threshold": "60", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_pg_warn_min_objects": "10000", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_pg_warn_min_per_osd": "0", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-02-05T08:06:25.968 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_pool_quota_crit_threshold": "0", 2024-02-05T08:06:25.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_pool_quota_warn_threshold": "0", 2024-02-05T08:06:25.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_probe_timeout": "2.000000", 2024-02-05T08:06:25.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_reweight_max_change": "0.050000", 2024-02-05T08:06:25.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_reweight_max_osds": "4", 2024-02-05T08:06:25.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-02-05T08:06:25.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-02-05T08:06:25.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-02-05T08:06:25.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-02-05T08:06:25.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-02-05T08:06:25.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_scrub_interval": "86400", 2024-02-05T08:06:25.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_scrub_max_keys": "100", 2024-02-05T08:06:25.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_scrub_timeout": "300", 2024-02-05T08:06:25.969 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_session_timeout": "300", 2024-02-05T08:06:25.970 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_smart_report_timeout": "5", 2024-02-05T08:06:25.970 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_stat_smooth_intervals": "6", 2024-02-05T08:06:25.970 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-02-05T08:06:25.970 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_stretch_pool_min_size": "2", 2024-02-05T08:06:25.970 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_stretch_pool_size": "4", 2024-02-05T08:06:25.970 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-02-05T08:06:25.970 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_subscribe_interval": "86400.000000", 2024-02-05T08:06:25.970 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_sync_debug": "false", 2024-02-05T08:06:25.970 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_sync_max_payload_keys": "2000", 2024-02-05T08:06:25.970 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_sync_max_payload_size": "1048576", 2024-02-05T08:06:25.970 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_sync_provider_kill_at": "0", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_sync_requester_kill_at": "0", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_sync_timeout": "60.000000", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_target_pg_per_osd": "100", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_tick_interval": "5", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_timecheck_interval": "300.000000", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_use_min_delay_socket": "false", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_older_version_delay": "604800", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_filestore_osds": "true", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-02-05T08:06:25.971 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-02-05T08:06:25.972 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_misplaced": "false", 2024-02-05T08:06:25.972 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-02-05T08:06:25.972 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_older_version": "true", 2024-02-05T08:06:25.972 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-02-05T08:06:25.972 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_pool_no_app": "true", 2024-02-05T08:06:25.972 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-02-05T08:06:25.972 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-02-05T08:06:25.972 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-02-05T08:06:25.972 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-02-05T08:06:25.972 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_on_too_few_osds": "false", 2024-02-05T08:06:25.972 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-02-05T08:06:25.972 INFO:teuthology.orchestra.run.smithi134.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-02-05T08:06:25.972 INFO:teuthology.orchestra.run.smithi134.stdout: "monmap": "", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "motr_tracing_enabled": "false", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_op_threads": "3", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_buffer_size": "131072", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_cm": "false", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_device_name": "", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_dscp": "96", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_gid_idx": "0", 2024-02-05T08:06:25.973 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_local_gid": "", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_polling_us": "1000", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_port_num": "1", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_roce_ver": "1", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_send_buffers": "1024", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_sl": "3", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_support_srq": "true", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_rdma_type": "ib", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_async_reap_threshold": "5", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_bind_before_connect": "false", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_bind_ipv4": "true", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_bind_ipv6": "false", 2024-02-05T08:06:25.974 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_bind_msgr1": "true", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_bind_msgr2": "true", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_bind_port_max": "7568", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_bind_port_min": "6800", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_bind_prefer_ipv4": "false", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_bind_retry_count": "3", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_bind_retry_delay": "5", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_blackhole_client": "false", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_blackhole_mds": "false", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_blackhole_mgr": "false", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_blackhole_mon": "false", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_blackhole_osd": "false", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_client_mode": "crc secure", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_cluster_mode": "crc secure", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_cluster_type": "", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_compress_secure": "false", 2024-02-05T08:06:25.975 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_connection_idle_timeout": "900", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_connection_ready_timeout": "10", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_crc_data": "true", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_crc_header": "true", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_die_on_bad_msg": "false", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_die_on_bug": "true", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_die_on_old_message": "true", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_die_on_skipped_message": "false", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_die_on_unhandled_msg": "false", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_coremask": "0xF", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_devs_allowlist": "", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_enable_tso": "true", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-02-05T08:06:25.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_hugepages": "", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_hw_flow_control": "true", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_lro": "true", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_memory_channel": "4", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_pmd": "", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_port_id": "0", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dump_corrupt_message_level": "1", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_dump_on_send": "false", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_initial_backoff": "0.200000", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_inject_delay_max": "1.000000", 2024-02-05T08:06:25.977 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_inject_delay_probability": "0.000000", 2024-02-05T08:06:25.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_inject_delay_type": "", 2024-02-05T08:06:25.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_inject_internal_delays": "0.000000", 2024-02-05T08:06:25.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_inject_network_congestion": "0", 2024-02-05T08:06:25.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_inject_socket_failures": "0", 2024-02-05T08:06:25.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_learn_addr_from_peer": "true", 2024-02-05T08:06:25.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_max_accept_failures": "4", 2024-02-05T08:06:25.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_max_backoff": "15.000000", 2024-02-05T08:06:25.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_mon_client_mode": "secure crc", 2024-02-05T08:06:25.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_mon_cluster_mode": "secure crc", 2024-02-05T08:06:25.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_mon_service_mode": "secure crc", 2024-02-05T08:06:25.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_osd_compress_min_size": "1024", 2024-02-05T08:06:25.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_osd_compress_mode": "none", 2024-02-05T08:06:25.978 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_osd_compression_algorithm": "snappy", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_pq_min_cost": "65536", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_public_type": "", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_service_mode": "crc secure", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_tcp_listen_backlog": "512", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_tcp_nodelay": "true", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_tcp_rcvbuf": "0", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "ms_type": "async+posix", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "no_config_file": "false", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "objecter_completion_locks_per_session": "32", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "objecter_debug_inject_relock_delay": "false", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "objecter_inflight_op_bytes": "104857600", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "objecter_inflight_ops": "1024", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "objecter_inject_no_watch_ping": "false", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-02-05T08:06:25.979 INFO:teuthology.orchestra.run.smithi134.stdout: "objecter_tick_interval": "5.000000", 2024-02-05T08:06:25.980 INFO:teuthology.orchestra.run.smithi134.stdout: "objecter_timeout": "10.000000", 2024-02-05T08:06:25.980 INFO:teuthology.orchestra.run.smithi134.stdout: "objectstore_blackhole": "false", 2024-02-05T08:06:25.980 INFO:teuthology.orchestra.run.smithi134.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-02-05T08:06:25.980 INFO:teuthology.orchestra.run.smithi134.stdout: "openssl_engine_opts": "", 2024-02-05T08:06:25.980 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_agent_delay_time": "5.000000", 2024-02-05T08:06:25.980 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_agent_hist_halflife": "1000", 2024-02-05T08:06:25.980 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_agent_max_low_ops": "2", 2024-02-05T08:06:25.980 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_agent_max_ops": "4", 2024-02-05T08:06:25.980 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-02-05T08:06:25.980 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_agent_quantize_effort": "0.100000", 2024-02-05T08:06:25.980 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_agent_slop": "0.020000", 2024-02-05T08:06:25.980 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-02-05T08:06:25.980 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_allow_recovery_below_min_size": "true", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_asio_thread_count": "2", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_async_recovery_min_cost": "100", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_auto_mark_unfound_lost": "false", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_backfill_retry_interval": "30.000000", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_backfill_scan_max": "512", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_backfill_scan_min": "64", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_backoff_on_degraded": "false", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_backoff_on_peering": "false", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_backoff_on_unfound": "true", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_beacon_report_interval": "300", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_bench_duration": "30", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_bench_max_block_size": "67108864", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_bench_small_size_max_iops": "100", 2024-02-05T08:06:25.981 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_blkin_trace_all": "false", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_blocked_scrub_grace_period": "120", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_check_for_log_corruption": "false", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_class_update_on_start": "true", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_client_message_cap": "256", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_client_message_size_cap": "524288000", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_client_op_priority": "63", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_client_watch_timeout": "30", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_command_max_records": "256", 2024-02-05T08:06:25.982 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_compact_on_start": "false", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_crush_chooseleaf_type": "1", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_crush_initial_weight": "-1.000000", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_crush_update_on_start": "true", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_crush_update_weight_set": "true", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_drop_ping_duration": "0", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_feed_pullee": "-1", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-02-05T08:06:25.983 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_misdirected_ops": "false", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_no_acting_change": "false", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_no_purge_strays": "false", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_op_order": "false", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_pg_log_writeout": "false", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_pretend_recovery_active": "false", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_shutdown": "false", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_trim_objects": "false", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_verify_cached_snaps": "false", 2024-02-05T08:06:25.984 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_verify_missing_on_start": "false", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_verify_snaps": "false", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_deep_scrub_keys": "1024", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_deep_scrub_stride": "524288", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_default_data_pool_replay_window": "45", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_default_notify_timeout": "30", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_delete_sleep": "0.000000", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-02-05T08:06:25.985 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_discard_disconnected_ops": "true", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_enable_op_tracker": "true", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_extblkdev_plugins": "vdo", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_fast_info": "true", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_fast_shutdown": "true", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_fast_shutdown_timeout": "15", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_function_tracing": "false", 2024-02-05T08:06:25.986 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_heartbeat_grace": "20", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_heartbeat_interval": "6", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_heartbeat_min_peers": "10", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_heartbeat_min_size": "2000", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_heartbeat_stale": "600", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_hit_set_max_size": "100000", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_hit_set_min_size": "1000", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-02-05T08:06:25.987 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_journal_flush_on_shutdown": "true", 2024-02-05T08:06:25.988 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_journal_size": "5120", 2024-02-05T08:06:25.988 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_kill_backfill_at": "0", 2024-02-05T08:06:25.988 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_loop_before_reset_tphandle": "64", 2024-02-05T08:06:25.988 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_map_cache_size": "50", 2024-02-05T08:06:25.988 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_map_dedup": "true", 2024-02-05T08:06:25.988 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_map_message_max": "40", 2024-02-05T08:06:25.988 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_map_message_max_bytes": "10485760", 2024-02-05T08:06:25.988 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_map_share_max_epochs": "40", 2024-02-05T08:06:25.988 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_attr_name_len": "100", 2024-02-05T08:06:25.988 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_attr_size": "0", 2024-02-05T08:06:25.988 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_backfills": "1", 2024-02-05T08:06:25.988 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_markdown_count": "5", 2024-02-05T08:06:25.988 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_markdown_period": "600", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_object_name_len": "2048", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_object_namespace_len": "256", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_object_size": "134217728", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_omap_entries_per_request": "1024", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_pg_blocked_by": "16", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_pg_log_entries": "10000", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_pgls": "1024", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_push_cost": "8388608", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_push_objects": "10", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_scrubs": "3", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_trimming_pgs": "2", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_write_op_reply_len": "64", 2024-02-05T08:06:25.989 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_max_write_size": "90", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_override_recovery_settings": "false", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_profile": "balanced", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-02-05T08:06:25.990 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-02-05T08:06:25.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-02-05T08:06:25.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-02-05T08:06:25.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-02-05T08:06:25.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-02-05T08:06:25.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-02-05T08:06:25.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mclock_skip_benchmark": "false", 2024-02-05T08:06:25.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_memory_base": "805306368", 2024-02-05T08:06:25.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_memory_cache_min": "134217728", 2024-02-05T08:06:25.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-02-05T08:06:25.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-02-05T08:06:25.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_memory_target": "4294967296", 2024-02-05T08:06:25.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_memory_target_autotune": "false", 2024-02-05T08:06:25.991 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-02-05T08:06:25.992 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_min_pg_log_entries": "250", 2024-02-05T08:06:25.992 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_min_recovery_priority": "0", 2024-02-05T08:06:25.992 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mon_heartbeat_interval": "30", 2024-02-05T08:06:25.992 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-02-05T08:06:25.992 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mon_report_interval": "5", 2024-02-05T08:06:25.992 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mon_report_max_in_flight": "2", 2024-02-05T08:06:25.992 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-02-05T08:06:25.992 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_num_cache_shards": "32", 2024-02-05T08:06:25.992 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_num_op_tracker_shard": "32", 2024-02-05T08:06:25.992 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_numa_auto_affinity": "true", 2024-02-05T08:06:25.992 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_numa_node": "-1", 2024-02-05T08:06:25.992 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_numa_prefer_iface": "true", 2024-02-05T08:06:25.992 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-02-05T08:06:25.993 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_objecter_finishers": "1", 2024-02-05T08:06:25.993 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_objectstore": "bluestore", 2024-02-05T08:06:25.993 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_objectstore_fuse": "false", 2024-02-05T08:06:25.993 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_objectstore_tracing": "false", 2024-02-05T08:06:25.993 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_complaint_time": "30.000000", 2024-02-05T08:06:25.993 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_history_duration": "600", 2024-02-05T08:06:25.993 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_history_size": "20", 2024-02-05T08:06:25.993 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_history_slow_op_size": "20", 2024-02-05T08:06:25.993 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-02-05T08:06:25.993 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_log_threshold": "5", 2024-02-05T08:06:25.993 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_num_shards": "0", 2024-02-05T08:06:25.993 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_num_shards_hdd": "5", 2024-02-05T08:06:25.993 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_num_shards_ssd": "8", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_num_threads_per_shard": "0", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_pq_min_cost": "65536", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_queue": "mclock_scheduler", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_queue_cut_off": "high", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_thread_suicide_timeout": "150", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_op_thread_timeout": "15", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_open_classes_on_start": "true", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_os_flags": "0", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_peering_op_priority": "255", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pg_delete_cost": "1048576", 2024-02-05T08:06:25.994 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pg_delete_priority": "5", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pg_log_dups_tracked": "3000", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pg_log_trim_max": "10000", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pg_log_trim_min": "100", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pg_object_context_cache_count": "64", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pg_stat_report_interval_max_seconds": "500", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-02-05T08:06:25.995 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-02-05T08:06:25.996 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-02-05T08:06:25.996 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_crimson": "false", 2024-02-05T08:06:25.996 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_crush_rule": "-1", 2024-02-05T08:06:25.996 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_ec_fast_read": "false", 2024-02-05T08:06:25.996 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-02-05T08:06:25.996 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_flag_bulk": "false", 2024-02-05T08:06:25.996 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-02-05T08:06:25.996 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_flag_nodelete": "false", 2024-02-05T08:06:25.996 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-02-05T08:06:25.996 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-02-05T08:06:25.996 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_flags": "0", 2024-02-05T08:06:25.996 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-02-05T08:06:25.996 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_min_size": "0", 2024-02-05T08:06:25.997 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-02-05T08:06:25.997 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_pg_num": "32", 2024-02-05T08:06:25.997 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_pgp_num": "0", 2024-02-05T08:06:25.997 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-02-05T08:06:25.997 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_read_ratio": "70", 2024-02-05T08:06:25.997 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_size": "3", 2024-02-05T08:06:25.997 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_default_type": "replicated", 2024-02-05T08:06:25.997 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-02-05T08:06:25.997 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_pool_use_gmt_hitset": "true", 2024-02-05T08:06:25.997 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_push_per_object_cost": "1000", 2024-02-05T08:06:25.997 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_read_ec_check_for_errors": "false", 2024-02-05T08:06:25.997 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recover_clone_overlap": "true", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recover_clone_overlap_limit": "10", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_cost": "20971520", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_delay_start": "0.000000", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_max_active": "0", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_max_active_hdd": "3", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_max_active_ssd": "10", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_max_chunk": "8388608", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_max_single_start": "1", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_op_priority": "3", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_op_warn_multiple": "16", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_priority": "5", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_retry_interval": "30.000000", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_sleep": "0.000000", 2024-02-05T08:06:25.998 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-02-05T08:06:25.999 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-02-05T08:06:25.999 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-02-05T08:06:25.999 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_repair_during_recovery": "false", 2024-02-05T08:06:25.999 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_requested_scrub_priority": "120", 2024-02-05T08:06:25.999 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-02-05T08:06:25.999 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_rollback_to_cluster_snap": "", 2024-02-05T08:06:25.999 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_auto_repair": "false", 2024-02-05T08:06:25.999 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-02-05T08:06:25.999 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-02-05T08:06:25.999 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_begin_hour": "0", 2024-02-05T08:06:25.999 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_begin_week_day": "0", 2024-02-05T08:06:25.999 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_chunk_max": "25", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_chunk_min": "5", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_cost": "52428800", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_during_recovery": "false", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_end_hour": "0", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_end_week_day": "0", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_event_cost": "4096", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_invalid_stats": "true", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_load_threshold": "0.500000", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_max_interval": "604800.000000", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_max_preemptions": "5", 2024-02-05T08:06:26.000 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_min_interval": "86400.000000", 2024-02-05T08:06:26.001 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_priority": "5", 2024-02-05T08:06:26.001 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_reservation_timeout": "300000", 2024-02-05T08:06:26.001 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_sleep": "0.000000", 2024-02-05T08:06:26.001 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-02-05T08:06:26.001 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-02-05T08:06:26.001 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-02-05T08:06:26.001 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_shutdown_pgref_assert": "false", 2024-02-05T08:06:26.001 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_skip_data_digest": "false", 2024-02-05T08:06:26.001 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_smart_report_timeout": "5", 2024-02-05T08:06:26.001 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_snap_trim_cost": "1048576", 2024-02-05T08:06:26.001 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_snap_trim_priority": "5", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_snap_trim_sleep": "0.000000", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_stats_update_period_scrubbing": "15", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_target_transaction_size": "30", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-02-05T08:06:26.002 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_tier_default_cache_mode": "writeback", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_tracing": "false", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_use_stale_snap": "false", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "osdc_blkin_trace_all": "false", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "paxos_kill_at": "0", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "paxos_max_join_drift": "10", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "paxos_min": "500", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "paxos_min_wait": "0.050000", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "paxos_propose_interval": "1.000000", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "paxos_service_trim_max": "500", 2024-02-05T08:06:26.003 INFO:teuthology.orchestra.run.smithi134.stdout: "paxos_service_trim_max_multiplier": "20", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "paxos_service_trim_min": "10", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "paxos_stash_full_interval": "25", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "paxos_trim_max": "500", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "paxos_trim_min": "250", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "perf": "true", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "pid_file": "", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "-", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrv": "", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "public_bind_addr": "-", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "public_network": "", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "public_network_interface": "", 2024-02-05T08:06:26.004 INFO:teuthology.orchestra.run.smithi134.stdout: "qat_compressor_enabled": "false", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "qat_compressor_session_max_number": "256", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rados_mon_op_timeout": "0", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rados_osd_op_timeout": "0", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rados_tracing": "false", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_atime_update_interval": "60", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_balance_parent_reads": "false", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_balance_snap_reads": "false", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_blkin_trace_all": "false", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_blocklist_expire_seconds": "0", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_blocklist_on_break_lock": "true", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_cache": "true", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_cache_block_writes_upfront": "false", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_cache_max_dirty": "25165824", 2024-02-05T08:06:26.005 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_cache_max_dirty_object": "0", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_cache_policy": "writearound", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_cache_size": "33554432", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_cache_target_dirty": "16777216", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_clone_copy_on_read": "false", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_compression_hint": "none", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_concurrent_management_ops": "10", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_default_clone_format": "auto", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_default_data_pool": "", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_default_features": "61", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_default_format": "2", 2024-02-05T08:06:26.006 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_default_map_options": "", 2024-02-05T08:06:26.007 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_default_order": "22", 2024-02-05T08:06:26.007 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_default_pool": "rbd", 2024-02-05T08:06:26.007 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-02-05T08:06:26.007 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_default_stripe_count": "0", 2024-02-05T08:06:26.007 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_default_stripe_unit": "0", 2024-02-05T08:06:26.007 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_disable_zero_copy_writes": "true", 2024-02-05T08:06:26.007 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_discard_granularity_bytes": "65536", 2024-02-05T08:06:26.007 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-02-05T08:06:26.007 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_enable_alloc_hint": "true", 2024-02-05T08:06:26.007 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-02-05T08:06:26.007 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_io_scheduler": "simple", 2024-02-05T08:06:26.007 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-02-05T08:06:26.007 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_journal_commit_age": "5.000000", 2024-02-05T08:06:26.008 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-02-05T08:06:26.008 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-02-05T08:06:26.008 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-02-05T08:06:26.008 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-02-05T08:06:26.008 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_journal_object_flush_interval": "0", 2024-02-05T08:06:26.008 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-02-05T08:06:26.008 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-02-05T08:06:26.008 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_journal_order": "24", 2024-02-05T08:06:26.008 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_journal_pool": "", 2024-02-05T08:06:26.008 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_journal_splay_width": "4", 2024-02-05T08:06:26.008 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_localize_parent_reads": "false", 2024-02-05T08:06:26.008 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_localize_snap_reads": "false", 2024-02-05T08:06:26.008 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_image_policy_type": "simple", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-02-05T08:06:26.009 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_memory_autotune": "true", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_memory_base": "805306368", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_memory_target": "4294967296", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirroring_delete_delay": "0", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirroring_replay_delay": "0", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-02-05T08:06:26.010 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_move_to_trash_on_remove": "false", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_mtime_update_interval": "60", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_non_blocking_aio": "true", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_op_thread_timeout": "60", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_op_threads": "1", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_parent_cache_enabled": "false", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_persistent_cache_mode": "disabled", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_persistent_cache_path": "/tmp", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_persistent_cache_size": "1073741824", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_plugins": "", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_bps_burst": "0", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-02-05T08:06:26.011 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_bps_limit": "0", 2024-02-05T08:06:26.012 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_exclude_ops": "0", 2024-02-05T08:06:26.012 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_iops_burst": "0", 2024-02-05T08:06:26.012 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-02-05T08:06:26.012 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_iops_limit": "0", 2024-02-05T08:06:26.012 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_read_bps_burst": "0", 2024-02-05T08:06:26.012 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-02-05T08:06:26.012 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_read_bps_limit": "0", 2024-02-05T08:06:26.012 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_read_iops_burst": "0", 2024-02-05T08:06:26.012 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-02-05T08:06:26.012 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_read_iops_limit": "0", 2024-02-05T08:06:26.012 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_schedule_tick_min": "50", 2024-02-05T08:06:26.012 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_write_bps_burst": "0", 2024-02-05T08:06:26.012 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-02-05T08:06:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_write_bps_limit": "0", 2024-02-05T08:06:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_write_iops_burst": "0", 2024-02-05T08:06:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-02-05T08:06:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_qos_write_iops_limit": "0", 2024-02-05T08:06:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_quiesce_notification_attempts": "10", 2024-02-05T08:06:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_read_from_replica_policy": "default", 2024-02-05T08:06:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-02-05T08:06:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_readahead_max_bytes": "524288", 2024-02-05T08:06:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_readahead_trigger_requests": "10", 2024-02-05T08:06:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_request_timed_out_seconds": "30", 2024-02-05T08:06:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_skip_partial_discard": "true", 2024-02-05T08:06:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-02-05T08:06:26.013 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_tracing": "false", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_validate_names": "true", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd_validate_pool": "true", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "restapi_base_url": "", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "restapi_log_level": "", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_acl_grants_max_num": "100", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_admin_entry": "admin", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_backend_store": "rados", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_barbican_url": "", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_beast_enable_async": "true", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_bucket_counters_cache": "false", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-02-05T08:06:26.014 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_bucket_index_max_aio": "128", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_bucket_quota_ttl": "600", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_cache_enabled": "true", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_cache_expiry_interval": "900", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_cache_lru_size": "10000", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_config_store": "rados", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_content_length_compat": "false", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_copy_obj_progress": "true", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_cors_rules_max_num": "100", 2024-02-05T08:06:26.015 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_cross_domain_policy": "", 2024-02-05T08:06:26.016 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_default_encryption_key": "", 2024-02-05T08:06:26.016 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_kmip_addr": "", 2024-02-05T08:06:26.016 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_kmip_ca_path": "", 2024-02-05T08:06:26.016 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_kmip_client_cert": "", 2024-02-05T08:06:26.016 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_kmip_client_key": "", 2024-02-05T08:06:26.016 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-02-05T08:06:26.016 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_kmip_password": "", 2024-02-05T08:06:26.016 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-02-05T08:06:26.016 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_kmip_username": "", 2024-02-05T08:06:26.016 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_require_ssl": "true", 2024-02-05T08:06:26.016 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-02-05T08:06:26.016 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-02-05T08:06:26.017 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-02-05T08:06:26.017 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-02-05T08:06:26.017 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-02-05T08:06:26.017 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-02-05T08:06:26.017 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-02-05T08:06:26.017 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-02-05T08:06:26.017 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-02-05T08:06:26.017 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-02-05T08:06:26.017 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-02-05T08:06:26.017 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-02-05T08:06:26.017 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-02-05T08:06:26.017 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-02-05T08:06:26.017 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_suppress_logs": "true", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_vault_addr": "", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_vault_auth": "token", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_vault_namespace": "", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_vault_prefix": "", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_vault_token_file": "", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_curl_buffersize": "524288", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_curl_low_speed_limit": "1024", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_curl_low_speed_time": "300", 2024-02-05T08:06:26.018 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_curl_tcp_keepalive": "0", 2024-02-05T08:06:26.019 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-02-05T08:06:26.019 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-02-05T08:06:26.019 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-02-05T08:06:26.019 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-02-05T08:06:26.019 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-02-05T08:06:26.019 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_d3n_l1_fadvise": "4", 2024-02-05T08:06:26.019 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-02-05T08:06:26.019 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-02-05T08:06:26.019 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-02-05T08:06:26.019 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_d4n_host": "127.0.0.1", 2024-02-05T08:06:26.019 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_d4n_port": "6379", 2024-02-05T08:06:26.019 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-02-05T08:06:26.020 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_data_log_changes_size": "1000", 2024-02-05T08:06:26.020 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_data_log_num_shards": "128", 2024-02-05T08:06:26.020 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-02-05T08:06:26.020 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_data_log_window": "30", 2024-02-05T08:06:26.020 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_data_notify_interval_msec": "0", 2024-02-05T08:06:26.020 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_data_sync_poll_interval": "20", 2024-02-05T08:06:26.020 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_data_sync_spawn_window": "20", 2024-02-05T08:06:26.020 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-02-05T08:06:26.020 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-02-05T08:06:26.020 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-02-05T08:06:26.020 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_default_data_log_backing": "fifo", 2024-02-05T08:06:26.020 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-02-05T08:06:26.021 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_default_region_info_oid": "default.region", 2024-02-05T08:06:26.021 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-02-05T08:06:26.021 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-02-05T08:06:26.021 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_defer_to_bucket_acls": "", 2024-02-05T08:06:26.021 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-02-05T08:06:26.021 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-02-05T08:06:26.021 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-02-05T08:06:26.021 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-02-05T08:06:26.021 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-02-05T08:06:26.021 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-02-05T08:06:26.021 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-02-05T08:06:26.022 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-02-05T08:06:26.022 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dmclock_data_res": "500.000000", 2024-02-05T08:06:26.022 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-02-05T08:06:26.022 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-02-05T08:06:26.022 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-02-05T08:06:26.022 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-02-05T08:06:26.022 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dns_name": "", 2024-02-05T08:06:26.022 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dns_s3website_name": "", 2024-02-05T08:06:26.022 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_dynamic_resharding": "true", 2024-02-05T08:06:26.022 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-02-05T08:06:26.022 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_enable_gc_threads": "true", 2024-02-05T08:06:26.022 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_enable_lc_threads": "true", 2024-02-05T08:06:26.022 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_enable_ops_log": "false", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_enable_quota_threads": "true", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_enable_static_website": "false", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_enable_usage_log": "false", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_enforce_swift_acls": "true", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_exit_timeout_secs": "120", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_expose_bucket": "false", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_extended_http_attrs": "", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_filter": "none", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_frontends": "beast port=7480", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_gc_max_concurrent_io": "10", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_gc_max_deferred": "50", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-02-05T08:06:26.023 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_gc_max_objs": "32", 2024-02-05T08:06:26.024 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_gc_max_queue_size": "134213632", 2024-02-05T08:06:26.024 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_gc_max_trim_chunk": "16", 2024-02-05T08:06:26.024 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_gc_obj_min_wait": "7200", 2024-02-05T08:06:26.024 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_gc_processor_max_time": "3600", 2024-02-05T08:06:26.024 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_gc_processor_period": "3600", 2024-02-05T08:06:26.024 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-02-05T08:06:26.024 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_get_obj_window_size": "16777216", 2024-02-05T08:06:26.024 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_healthcheck_disabling_path": "", 2024-02-05T08:06:26.024 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_ignore_get_invalid_range": "false", 2024-02-05T08:06:26.025 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_init_timeout": "300", 2024-02-05T08:06:26.025 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-02-05T08:06:26.025 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-02-05T08:06:26.025 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_kafka_connection_idle": "30", 2024-02-05T08:06:26.025 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_kafka_sleep_timeout": "10", 2024-02-05T08:06:26.025 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-02-05T08:06:26.025 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-02-05T08:06:26.025 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_admin_domain": "", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_admin_password": "", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_admin_password_path": "", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_admin_project": "", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_admin_tenant": "", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_admin_token": "", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_admin_token_path": "", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_admin_user": "", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_api_version": "2", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_barbican_domain": "", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_barbican_password": "", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_barbican_project": "", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_barbican_tenant": "", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_barbican_user": "", 2024-02-05T08:06:26.026 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-02-05T08:06:26.027 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_implicit_tenants": "false", 2024-02-05T08:06:26.027 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-02-05T08:06:26.027 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_service_token_enabled": "false", 2024-02-05T08:06:26.027 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_token_cache_size": "10000", 2024-02-05T08:06:26.027 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_url": "", 2024-02-05T08:06:26.027 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_keystone_verify_ssl": "true", 2024-02-05T08:06:26.027 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_lc_debug_interval": "-1", 2024-02-05T08:06:26.027 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_lc_lock_max_time": "90", 2024-02-05T08:06:26.027 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_lc_max_objs": "32", 2024-02-05T08:06:26.027 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_lc_max_rules": "1000", 2024-02-05T08:06:26.027 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_lc_max_worker": "3", 2024-02-05T08:06:26.027 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_lc_max_wp_worker": "3", 2024-02-05T08:06:26.027 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_lc_thread_delay": "0", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_ldap_dnattr": "uid", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_ldap_searchfilter": "", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_ldap_uri": "ldaps://", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_log_http_headers": "", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_log_nonexistent_bucket": "false", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_log_object_name_utc": "false", 2024-02-05T08:06:26.028 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-02-05T08:06:26.029 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-02-05T08:06:26.029 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_max_attr_name_len": "0", 2024-02-05T08:06:26.029 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_max_attr_size": "0", 2024-02-05T08:06:26.029 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_max_attrs_num_in_req": "0", 2024-02-05T08:06:26.029 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_max_chunk_size": "4194304", 2024-02-05T08:06:26.029 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_max_concurrent_requests": "1024", 2024-02-05T08:06:26.029 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-02-05T08:06:26.029 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_max_dynamic_shards": "1999", 2024-02-05T08:06:26.029 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_max_listing_results": "1000", 2024-02-05T08:06:26.029 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_max_notify_retries": "10", 2024-02-05T08:06:26.029 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_max_objs_per_shard": "100000", 2024-02-05T08:06:26.029 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_max_put_param_size": "1048576", 2024-02-05T08:06:26.029 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_max_put_size": "5368709120", 2024-02-05T08:06:26.030 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_max_slo_entries": "1000", 2024-02-05T08:06:26.030 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_md_log_max_shards": "64", 2024-02-05T08:06:26.030 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_md_notify_interval_msec": "200", 2024-02-05T08:06:26.030 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_meta_sync_poll_interval": "20", 2024-02-05T08:06:26.030 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_meta_sync_spawn_window": "20", 2024-02-05T08:06:26.030 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-02-05T08:06:26.030 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_mp_lock_max_time": "600", 2024-02-05T08:06:26.030 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-02-05T08:06:26.030 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_multipart_min_part_size": "5242880", 2024-02-05T08:06:26.030 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-02-05T08:06:26.030 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-02-05T08:06:26.030 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_nfs_fhcache_size": "2017", 2024-02-05T08:06:26.030 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_nfs_lru_lanes": "5", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_nfs_max_gc": "300", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_nfs_run_gc_threads": "false", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_nfs_run_lc_threads": "false", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_nfs_run_quota_threads": "false", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_nfs_run_sync_thread": "false", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_num_async_rados_threads": "32", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_num_control_oids": "8", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_numa_node": "-1", 2024-02-05T08:06:26.031 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_obj_stripe_size": "4194304", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_objexp_chunk_size": "100", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_objexp_gc_interval": "600", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_objexp_hints_num_shards": "127", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_op_thread_timeout": "600", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_op_tracing": "false", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_opa_token": "", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_opa_url": "", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_opa_verify_ssl": "true", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-02-05T08:06:26.032 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_ops_log_rados": "false", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_ops_log_socket_path": "", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_period_push_interval": "2.000000", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_period_push_interval_max": "30.000000", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_period_root_pool": ".rgw.root", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_posix_cache_lanes": "3", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_posix_cache_max_buckets": "100", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_posix_cache_partitions": "3", 2024-02-05T08:06:26.033 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-02-05T08:06:26.034 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_print_continue": "true", 2024-02-05T08:06:26.034 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_print_prohibited_content_length": "false", 2024-02-05T08:06:26.034 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-02-05T08:06:26.034 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-02-05T08:06:26.034 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-02-05T08:06:26.034 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-02-05T08:06:26.034 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_rados_tracing": "false", 2024-02-05T08:06:26.034 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_realm": "", 2024-02-05T08:06:26.034 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_realm_id": "", 2024-02-05T08:06:26.034 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-02-05T08:06:26.034 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_region": "", 2024-02-05T08:06:26.034 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_region_root_pool": ".rgw.root", 2024-02-05T08:06:26.034 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_relaxed_region_enforcement": "false", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_request_uri": "", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_reshard_batch_size": "64", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_reshard_max_aio": "128", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_reshard_num_logs": "16", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_reshard_thread_interval": "600", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_resolve_cname": "false", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_rest_getusage_op_compat": "false", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_run_sync_thread": "true", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_s3_auth_use_keystone": "false", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_s3_auth_use_ldap": "false", 2024-02-05T08:06:26.035 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_s3_auth_use_rados": "true", 2024-02-05T08:06:26.036 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_s3_auth_use_sts": "false", 2024-02-05T08:06:26.036 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-02-05T08:06:26.036 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_s3_success_create_obj_status": "0", 2024-02-05T08:06:26.036 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-02-05T08:06:26.036 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_scheduler_type": "throttler", 2024-02-05T08:06:26.036 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_script_uri": "", 2024-02-05T08:06:26.036 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_service_provider_name": "", 2024-02-05T08:06:26.036 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-02-05T08:06:26.036 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sts_client_id": "", 2024-02-05T08:06:26.036 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sts_client_secret": "", 2024-02-05T08:06:26.036 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sts_entry": "sts", 2024-02-05T08:06:26.036 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sts_key": "sts", 2024-02-05T08:06:26.036 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sts_max_session_duration": "43200", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sts_min_session_duration": "900", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sts_token_introspection_url": "", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_swift_account_in_url": "false", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_swift_auth_entry": "auth", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_swift_auth_url": "", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_swift_custom_header": "", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_swift_enforce_content_length": "false", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_swift_need_stats": "true", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_swift_tenant_name": "", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_swift_token_expiration": "86400", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_swift_url": "", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_swift_url_prefix": "swift", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_swift_versioning_enabled": "false", 2024-02-05T08:06:26.037 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-02-05T08:06:26.038 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-02-05T08:06:26.038 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sync_lease_period": "120", 2024-02-05T08:06:26.038 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-02-05T08:06:26.038 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sync_log_trim_interval": "1200", 2024-02-05T08:06:26.038 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-02-05T08:06:26.038 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-02-05T08:06:26.038 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-02-05T08:06:26.038 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sync_obj_etag_verify": "false", 2024-02-05T08:06:26.038 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sync_trace_history_size": "4096", 2024-02-05T08:06:26.038 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-02-05T08:06:26.038 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-02-05T08:06:26.038 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_thread_pool_size": "512", 2024-02-05T08:06:26.038 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_topic_persistency_max_retries": "0", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_torrent_comment": "", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_torrent_createby": "", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_torrent_encoding": "", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_torrent_flag": "false", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_torrent_max_size": "5368709120", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_torrent_origin": "", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_torrent_sha_unit": "524288", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_torrent_tracker": "", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_trust_forwarded_https": "false", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_usage_log_tick_interval": "30", 2024-02-05T08:06:26.039 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_usage_max_shards": "32", 2024-02-05T08:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_usage_max_user_shards": "1", 2024-02-05T08:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_use_opa_authz": "false", 2024-02-05T08:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_user_counters_cache": "false", 2024-02-05T08:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_user_counters_cache_size": "10000", 2024-02-05T08:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-02-05T08:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_user_default_quota_max_size": "-1", 2024-02-05T08:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_user_max_buckets": "1000", 2024-02-05T08:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_user_policies_max_num": "100", 2024-02-05T08:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-02-05T08:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-02-05T08:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_user_quota_sync_interval": "86400", 2024-02-05T08:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-02-05T08:06:26.040 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_user_unique_email": "true", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_verify_ssl": "true", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_website_routing_rules_max_num": "50", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_zone": "", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_zone_id": "", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_zonegroup": "", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_zonegroup_id": "", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rgwlc_auto_session_clear": "true", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rgwlc_skip_bucket_step": "false", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_block_size": "4096", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-02-05T08:06:26.041 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-02-05T08:06:26.042 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_cache_shard_bits": "4", 2024-02-05T08:06:26.042 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_cache_size": "536870912", 2024-02-05T08:06:26.042 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_cache_type": "binned_lru", 2024-02-05T08:06:26.042 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-02-05T08:06:26.042 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-02-05T08:06:26.042 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-02-05T08:06:26.042 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_collect_compaction_stats": "false", 2024-02-05T08:06:26.042 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_collect_extended_stats": "false", 2024-02-05T08:06:26.042 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_collect_memory_stats": "false", 2024-02-05T08:06:26.042 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-02-05T08:06:26.042 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_index_type": "binary_search", 2024-02-05T08:06:26.042 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_log_to_ceph_log": "true", 2024-02-05T08:06:26.042 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_metadata_block_size": "4096", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_partition_filters": "false", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_perf": "false", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "rotating_keys_renewal_timeout": "10", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "run_dir": "/var/run/ceph", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_block_create": "true", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_cache_lru_size": "67108864", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_cbjournal_size": "5368709120", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_data_delta_based_overwrite": "0", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_default_max_object_size": "16777216", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_device_size": "53687091200", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_journal_batch_capacity": "16", 2024-02-05T08:06:26.043 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_journal_iodepth_limit": "5", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_main_device_type": "SSD", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_max_concurrent_transactions": "8", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "seastore_segment_size": "67108864", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "set_keepcaps": "false", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "setgroup": "ceph", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "setuser": "ceph", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "setuser_match_path": "", 2024-02-05T08:06:26.044 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_misplaced_ratio": "0.050000", 2024-02-05T08:06:26.045 INFO:teuthology.orchestra.run.smithi134.stdout: "thp": "false", 2024-02-05T08:06:26.045 INFO:teuthology.orchestra.run.smithi134.stdout: "threadpool_default_timeout": "60", 2024-02-05T08:06:26.045 INFO:teuthology.orchestra.run.smithi134.stdout: "threadpool_empty_queue_max_wait": "2", 2024-02-05T08:06:26.045 INFO:teuthology.orchestra.run.smithi134.stdout: "throttler_perf_counter": "true" 2024-02-05T08:06:26.045 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:06:26.251 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:25 smithi134 bash[79271]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-05T08:06:26.527 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:26 smithi134 bash[79271]: Getting image source signatures 2024-02-05T08:06:26.527 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:26 smithi134 bash[79271]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-05T08:06:26.527 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:26 smithi134 bash[79271]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-05T08:06:26.527 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:26 smithi134 bash[79271]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-05T08:06:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:26 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:06:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:26 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:06:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:26 smithi196 ceph-mon[33018]: 3.b scrub starts 2024-02-05T08:06:26.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:26 smithi196 ceph-mon[33018]: 3.b scrub ok 2024-02-05T08:06:26.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:26 smithi196 ceph-mon[33018]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:06:26.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:26 smithi196 ceph-mon[33018]: 6.5 scrub starts 2024-02-05T08:06:26.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:26 smithi196 ceph-mon[33018]: 6.5 scrub ok 2024-02-05T08:06:26.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:26 smithi196 ceph-mon[33018]: 3.1b scrub starts 2024-02-05T08:06:26.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:26 smithi196 ceph-mon[33018]: 3.1b scrub ok 2024-02-05T08:06:26.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:06:26.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:06:26.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[30086]: 3.b scrub starts 2024-02-05T08:06:26.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[30086]: 3.b scrub ok 2024-02-05T08:06:26.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[30086]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:06:26.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[30086]: 6.5 scrub starts 2024-02-05T08:06:26.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[30086]: 6.5 scrub ok 2024-02-05T08:06:26.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[30086]: 3.1b scrub starts 2024-02-05T08:06:26.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[30086]: 3.1b scrub ok 2024-02-05T08:06:26.982 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:06:26.983 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:06:26.983 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[39278]: 3.b scrub starts 2024-02-05T08:06:26.983 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[39278]: 3.b scrub ok 2024-02-05T08:06:26.983 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[39278]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:06:26.983 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[39278]: 6.5 scrub starts 2024-02-05T08:06:26.983 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[39278]: 6.5 scrub ok 2024-02-05T08:06:26.983 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[39278]: 3.1b scrub starts 2024-02-05T08:06:26.983 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:26 smithi134 ceph-mon[39278]: 3.1b scrub ok 2024-02-05T08:06:27.012 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-02-05T08:06:27.271 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:28.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:06:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:06:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[30086]: 4.10 scrub starts 2024-02-05T08:06:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[30086]: 4.10 scrub ok 2024-02-05T08:06:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[30086]: 5.18 scrub starts 2024-02-05T08:06:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[30086]: 5.18 scrub ok 2024-02-05T08:06:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:28.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[30086]: 5.2 deep-scrub starts 2024-02-05T08:06:28.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[30086]: 5.2 deep-scrub ok 2024-02-05T08:06:28.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[30086]: 6.3 scrub starts 2024-02-05T08:06:28.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[30086]: 6.3 scrub ok 2024-02-05T08:06:28.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[30086]: 3.12 scrub starts 2024-02-05T08:06:28.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[30086]: 3.12 scrub ok 2024-02-05T08:06:28.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[39278]: 4.10 scrub starts 2024-02-05T08:06:28.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[39278]: 4.10 scrub ok 2024-02-05T08:06:28.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[39278]: 5.18 scrub starts 2024-02-05T08:06:28.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[39278]: 5.18 scrub ok 2024-02-05T08:06:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[39278]: 5.2 deep-scrub starts 2024-02-05T08:06:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[39278]: 5.2 deep-scrub ok 2024-02-05T08:06:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[39278]: 6.3 scrub starts 2024-02-05T08:06:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[39278]: 6.3 scrub ok 2024-02-05T08:06:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[39278]: 3.12 scrub starts 2024-02-05T08:06:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:27 smithi134 ceph-mon[39278]: 3.12 scrub ok 2024-02-05T08:06:28.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:27 smithi196 ceph-mon[33018]: 4.10 scrub starts 2024-02-05T08:06:28.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:27 smithi196 ceph-mon[33018]: 4.10 scrub ok 2024-02-05T08:06:28.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:27 smithi196 ceph-mon[33018]: 5.18 scrub starts 2024-02-05T08:06:28.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:27 smithi196 ceph-mon[33018]: 5.18 scrub ok 2024-02-05T08:06:28.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:27 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:28.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:27 smithi196 ceph-mon[33018]: 5.2 deep-scrub starts 2024-02-05T08:06:28.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:27 smithi196 ceph-mon[33018]: 5.2 deep-scrub ok 2024-02-05T08:06:28.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:27 smithi196 ceph-mon[33018]: 6.3 scrub starts 2024-02-05T08:06:28.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:27 smithi196 ceph-mon[33018]: 6.3 scrub ok 2024-02-05T08:06:28.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:27 smithi196 ceph-mon[33018]: 3.12 scrub starts 2024-02-05T08:06:28.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:27 smithi196 ceph-mon[33018]: 3.12 scrub ok 2024-02-05T08:06:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:28 smithi134 ceph-mon[39278]: 5.0 scrub starts 2024-02-05T08:06:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:28 smithi134 ceph-mon[39278]: 5.0 scrub ok 2024-02-05T08:06:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:28 smithi134 ceph-mon[39278]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:28 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:06:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:28 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:06:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:28 smithi134 ceph-mon[30086]: 5.0 scrub starts 2024-02-05T08:06:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:28 smithi134 ceph-mon[30086]: 5.0 scrub ok 2024-02-05T08:06:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:28 smithi134 ceph-mon[30086]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:28 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:06:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:28 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:06:28.778 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:28 smithi134 bash[79271]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-05T08:06:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:28 smithi196 ceph-mon[33018]: 5.0 scrub starts 2024-02-05T08:06:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:28 smithi196 ceph-mon[33018]: 5.0 scrub ok 2024-02-05T08:06:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:28 smithi196 ceph-mon[33018]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:28 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:06:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:28 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:06:29.073 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:28 smithi134 bash[79271]: Writing manifest to image destination 2024-02-05T08:06:29.527 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:29 smithi134 podman[79271]: 2024-02-05 08:06:29.071778874 +0000 UTC m=+3.403253738 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-05T08:06:29.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:29 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:06:29.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:29 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:06:29.825 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:29 smithi134 ceph-mon[30086]: osdmap e94: 8 total, 8 up, 8 in 2024-02-05T08:06:29.825 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:29 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:06:29.825 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:29 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:06:29.825 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:29 smithi134 ceph-mon[39278]: osdmap e94: 8 total, 8 up, 8 in 2024-02-05T08:06:29.826 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:29 smithi134 podman[79271]: 2024-02-05 08:06:29.714030182 +0000 UTC m=+4.045505042 container create 419ba33757dc8c37bd99ba56768944cca2f553b47fbbc91980fc388667072997 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T08:06:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:29 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:06:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:29 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:06:30.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:29 smithi196 ceph-mon[33018]: osdmap e94: 8 total, 8 up, 8 in 2024-02-05T08:06:30.194 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 podman[79271]: 2024-02-05 08:06:30.110653982 +0000 UTC m=+4.442128841 container init 419ba33757dc8c37bd99ba56768944cca2f553b47fbbc91980fc388667072997 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T08:06:30.194 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 podman[79271]: 2024-02-05 08:06:30.116704396 +0000 UTC m=+4.448179256 container start 419ba33757dc8c37bd99ba56768944cca2f553b47fbbc91980fc388667072997 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T08:06:30.194 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.120Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-05T08:06:30.194 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.120Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-05T08:06:30.194 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-05T08:06:30.194 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-05T08:06:30.194 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-05T08:06:30.194 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-05T08:06:30.194 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-05T08:06:30.195 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=arp 2024-02-05T08:06:30.195 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-05T08:06:30.195 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-05T08:06:30.195 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-05T08:06:30.195 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-05T08:06:30.195 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-05T08:06:30.195 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-05T08:06:30.195 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-05T08:06:30.196 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-05T08:06:30.196 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=edac 2024-02-05T08:06:30.196 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-05T08:06:30.196 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-05T08:06:30.196 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-05T08:06:30.196 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-05T08:06:30.196 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-05T08:06:30.196 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-05T08:06:30.196 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-05T08:06:30.196 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-05T08:06:30.196 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-05T08:06:30.197 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-05T08:06:30.197 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-05T08:06:30.197 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-05T08:06:30.197 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-05T08:06:30.197 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-05T08:06:30.197 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-05T08:06:30.197 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-05T08:06:30.197 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=os 2024-02-05T08:06:30.197 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-05T08:06:30.197 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-05T08:06:30.197 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-05T08:06:30.197 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-05T08:06:30.197 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-05T08:06:30.198 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-05T08:06:30.198 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-05T08:06:30.198 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=stat 2024-02-05T08:06:30.198 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-05T08:06:30.198 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-05T08:06:30.198 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-05T08:06:30.198 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=time 2024-02-05T08:06:30.198 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-05T08:06:30.198 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=uname 2024-02-05T08:06:30.198 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-05T08:06:30.198 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-05T08:06:30.198 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.121Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-05T08:06:30.198 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.122Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-05T08:06:30.199 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a[79492]: ts=2024-02-05T08:06:30.122Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-05T08:06:30.199 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 bash[79271]: 419ba33757dc8c37bd99ba56768944cca2f553b47fbbc91980fc388667072997 2024-02-05T08:06:30.427 INFO:teuthology.orchestra.run.smithi134.stdout:mon.a: {} 2024-02-05T08:06:30.437 INFO:teuthology.orchestra.run.smithi134.stdout:mon.b: {} 2024-02-05T08:06:30.447 INFO:teuthology.orchestra.run.smithi134.stdout:mon.c: {} 2024-02-05T08:06:30.447 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:06:30 smithi134 systemd[1]: Started Ceph node-exporter.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:06:30.448 INFO:teuthology.orchestra.run.smithi134.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T08:06:30.448 INFO:teuthology.orchestra.run.smithi134.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T08:06:30.448 INFO:teuthology.orchestra.run.smithi134.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T08:06:30.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: 6.1f scrub starts 2024-02-05T08:06:30.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: 6.1f scrub ok 2024-02-05T08:06:30.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: 6.e deep-scrub starts 2024-02-05T08:06:30.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: 6.e deep-scrub ok 2024-02-05T08:06:30.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:30.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: 4.6 scrub starts 2024-02-05T08:06:30.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: 4.6 scrub ok 2024-02-05T08:06:30.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: 4.1a scrub starts 2024-02-05T08:06:30.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: 4.1a scrub ok 2024-02-05T08:06:30.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:30.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:30.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:30.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: Deploying daemon node-exporter.b on smithi196 2024-02-05T08:06:30.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2309345256' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:06:30.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: 6.1f scrub starts 2024-02-05T08:06:30.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: 6.1f scrub ok 2024-02-05T08:06:30.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: 6.e deep-scrub starts 2024-02-05T08:06:30.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: 6.e deep-scrub ok 2024-02-05T08:06:30.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:30.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: 4.6 scrub starts 2024-02-05T08:06:30.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: 4.6 scrub ok 2024-02-05T08:06:30.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: 4.1a scrub starts 2024-02-05T08:06:30.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: 4.1a scrub ok 2024-02-05T08:06:30.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:30.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:30.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:30.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: Deploying daemon node-exporter.b on smithi196 2024-02-05T08:06:30.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:30 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2309345256' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:06:30.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: 6.1f scrub starts 2024-02-05T08:06:30.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: 6.1f scrub ok 2024-02-05T08:06:30.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: 6.e deep-scrub starts 2024-02-05T08:06:30.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: 6.e deep-scrub ok 2024-02-05T08:06:30.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:30.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: 4.6 scrub starts 2024-02-05T08:06:30.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: 4.6 scrub ok 2024-02-05T08:06:30.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: 4.1a scrub starts 2024-02-05T08:06:30.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: 4.1a scrub ok 2024-02-05T08:06:30.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:30.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:30.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:30.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: Deploying daemon node-exporter.b on smithi196 2024-02-05T08:06:30.895 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:30 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2309345256' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:06:31.176 INFO:teuthology.run_tasks:Running task full_sequential... 2024-02-05T08:06:31.187 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T08:06:31.213 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T08:06:31.213 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-02-05T08:06:31.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-02-05T08:06:31.214 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-02-05T08:06:31.214 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:06:31.214 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:06:31.214 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:06:31.214 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd out 6 2024-02-05T08:06:31.214 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-02-05T08:06:31.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:31.215 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-02-05T08:06:31.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:06:31.215 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-02-05T08:06:31.216 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T08:06:31.216 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-02-05T08:06:31.216 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:06:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:31.248 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:31.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:31.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:31.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:31.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:31.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:31.478 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:31.518 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:31.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:31.536 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:31.536 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:31.539 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:31.564 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:31 smithi196 bash[66961]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-05T08:06:31.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:31.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:31.644 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:31.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:31.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:31.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:31.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:31.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:31.872 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:31.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:31.897 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:31.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:31.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:31 smithi196 ceph-mon[33018]: 5.c scrub starts 2024-02-05T08:06:31.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:31 smithi196 ceph-mon[33018]: 5.c scrub ok 2024-02-05T08:06:31.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:31 smithi196 ceph-mon[33018]: 6.b scrub starts 2024-02-05T08:06:31.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:31 smithi196 ceph-mon[33018]: 6.b scrub ok 2024-02-05T08:06:31.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:31 smithi196 ceph-mon[33018]: 4.16 scrub starts 2024-02-05T08:06:31.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:31 smithi196 ceph-mon[33018]: 4.16 scrub ok 2024-02-05T08:06:31.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:31 smithi134 ceph-mon[30086]: 5.c scrub starts 2024-02-05T08:06:31.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:31 smithi134 ceph-mon[30086]: 5.c scrub ok 2024-02-05T08:06:31.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:31 smithi134 ceph-mon[30086]: 6.b scrub starts 2024-02-05T08:06:31.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:31 smithi134 ceph-mon[30086]: 6.b scrub ok 2024-02-05T08:06:31.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:31 smithi134 ceph-mon[30086]: 4.16 scrub starts 2024-02-05T08:06:31.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:31 smithi134 ceph-mon[30086]: 4.16 scrub ok 2024-02-05T08:06:31.952 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:31 smithi134 ceph-mon[39278]: 5.c scrub starts 2024-02-05T08:06:31.952 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:31 smithi134 ceph-mon[39278]: 5.c scrub ok 2024-02-05T08:06:31.952 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:31 smithi134 ceph-mon[39278]: 6.b scrub starts 2024-02-05T08:06:31.952 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:31 smithi134 ceph-mon[39278]: 6.b scrub ok 2024-02-05T08:06:31.953 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:31 smithi134 ceph-mon[39278]: 4.16 scrub starts 2024-02-05T08:06:31.953 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:31 smithi134 ceph-mon[39278]: 4.16 scrub ok 2024-02-05T08:06:31.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:32.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:32.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:32.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:32.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:32.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:32.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:32.250 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:32.274 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:32.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:32.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:32.314 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:31 smithi196 bash[66961]: Getting image source signatures 2024-02-05T08:06:32.314 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:31 smithi196 bash[66961]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-05T08:06:32.314 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:31 smithi196 bash[66961]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-05T08:06:32.314 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:31 smithi196 bash[66961]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-05T08:06:32.376 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:32.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:32.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:32.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:32.529 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:32.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:32.631 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:32.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:32.655 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:32.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:32.756 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:32 smithi134 ceph-mon[30086]: 6.7 scrub starts 2024-02-05T08:06:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:32 smithi134 ceph-mon[30086]: 6.7 scrub ok 2024-02-05T08:06:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:32 smithi134 ceph-mon[30086]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-02-05T08:06:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:32 smithi134 ceph-mon[39278]: 6.7 scrub starts 2024-02-05T08:06:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:32 smithi134 ceph-mon[39278]: 6.7 scrub ok 2024-02-05T08:06:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:32 smithi134 ceph-mon[39278]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-02-05T08:06:32.779 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:32.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:32.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:32.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:32 smithi196 ceph-mon[33018]: 6.7 scrub starts 2024-02-05T08:06:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:32 smithi196 ceph-mon[33018]: 6.7 scrub ok 2024-02-05T08:06:32.896 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:32 smithi196 ceph-mon[33018]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 694 B/s rd, 0 op/s 2024-02-05T08:06:32.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:32.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:33.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:33.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:33.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:33.055 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:33.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:33.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:33.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:33.262 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:33.286 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:33.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:33.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:06:32 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:06:33.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:33.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:33.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:33.513 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:33.538 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:33.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:33.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:33.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:33.644 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:33.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:33.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:33.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:33.789 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:33.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:33 smithi196 ceph-mon[33018]: 3.1 scrub starts 2024-02-05T08:06:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:33 smithi196 ceph-mon[33018]: 3.1 scrub ok 2024-02-05T08:06:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:33 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:33 smithi196 ceph-mon[33018]: 3.2 deep-scrub starts 2024-02-05T08:06:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:33 smithi196 ceph-mon[33018]: 3.2 deep-scrub ok 2024-02-05T08:06:33.815 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:33 smithi196 bash[66961]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-05T08:06:33.815 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:33 smithi196 bash[66961]: Writing manifest to image destination 2024-02-05T08:06:33.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:33.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:33.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:34.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:33 smithi134 ceph-mon[30086]: 3.1 scrub starts 2024-02-05T08:06:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:33 smithi134 ceph-mon[30086]: 3.1 scrub ok 2024-02-05T08:06:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:33 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:33 smithi134 ceph-mon[30086]: 3.2 deep-scrub starts 2024-02-05T08:06:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:33 smithi134 ceph-mon[30086]: 3.2 deep-scrub ok 2024-02-05T08:06:34.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:33 smithi134 ceph-mon[39278]: 3.1 scrub starts 2024-02-05T08:06:34.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:33 smithi134 ceph-mon[39278]: 3.1 scrub ok 2024-02-05T08:06:34.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:33 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:34.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:33 smithi134 ceph-mon[39278]: 3.2 deep-scrub starts 2024-02-05T08:06:34.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:33 smithi134 ceph-mon[39278]: 3.2 deep-scrub ok 2024-02-05T08:06:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:34.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:34.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:34.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:34.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:34.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:34.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:34.251 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:33 smithi196 podman[66961]: 2024-02-05 08:06:33.874368192 +0000 UTC m=+2.499919538 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-05T08:06:34.251 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:33 smithi196 podman[66961]: 2024-02-05 08:06:33.981837347 +0000 UTC m=+2.607388686 container create 779d88c4e760247af80687dc57457c29f3dab7ebc3a394ddc4c7ed1bc4b3c10f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T08:06:34.251 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 podman[66961]: 2024-02-05 08:06:34.251303093 +0000 UTC m=+2.876854437 container init 779d88c4e760247af80687dc57457c29f3dab7ebc3a394ddc4c7ed1bc4b3c10f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T08:06:34.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:34.291 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:34.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:34.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:34.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:34.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:34.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:34.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:34.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:34.545 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:34.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:34.565 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 podman[66961]: 2024-02-05 08:06:34.2567211 +0000 UTC m=+2.882272438 container start 779d88c4e760247af80687dc57457c29f3dab7ebc3a394ddc4c7ed1bc4b3c10f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T08:06:34.565 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.260Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-05T08:06:34.565 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.260Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-05T08:06:34.565 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.260Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-05T08:06:34.566 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.260Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-05T08:06:34.566 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-05T08:06:34.566 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-05T08:06:34.566 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-05T08:06:34.566 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=arp 2024-02-05T08:06:34.566 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-05T08:06:34.566 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-05T08:06:34.567 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-05T08:06:34.567 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-05T08:06:34.567 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-05T08:06:34.567 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-05T08:06:34.567 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-05T08:06:34.567 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-05T08:06:34.567 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=edac 2024-02-05T08:06:34.567 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-05T08:06:34.567 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-05T08:06:34.568 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-05T08:06:34.568 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-05T08:06:34.568 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-05T08:06:34.568 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-05T08:06:34.568 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-05T08:06:34.568 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-05T08:06:34.569 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-05T08:06:34.569 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-05T08:06:34.569 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-05T08:06:34.569 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-05T08:06:34.569 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-05T08:06:34.569 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-05T08:06:34.569 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-05T08:06:34.569 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-05T08:06:34.569 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=os 2024-02-05T08:06:34.570 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-05T08:06:34.570 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-05T08:06:34.570 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-05T08:06:34.570 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-05T08:06:34.570 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-05T08:06:34.570 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-05T08:06:34.570 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-05T08:06:34.570 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=stat 2024-02-05T08:06:34.571 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-05T08:06:34.571 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-05T08:06:34.571 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-05T08:06:34.571 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=time 2024-02-05T08:06:34.571 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-05T08:06:34.571 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=uname 2024-02-05T08:06:34.571 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-05T08:06:34.571 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-05T08:06:34.571 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.261Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-05T08:06:34.571 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.262Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-05T08:06:34.571 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b[67216]: ts=2024-02-05T08:06:34.262Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-05T08:06:34.571 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 bash[66961]: 779d88c4e760247af80687dc57457c29f3dab7ebc3a394ddc4c7ed1bc4b3c10f 2024-02-05T08:06:34.571 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:06:34 smithi196 systemd[1]: Started Ceph node-exporter.b for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:06:34.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:34.671 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:34.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:34.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:34.773 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:34.798 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:34.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:34.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:34.850 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[30086]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:34.850 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[30086]: 4.2 scrub starts 2024-02-05T08:06:34.850 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[30086]: 4.2 scrub ok 2024-02-05T08:06:34.850 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[30086]: 5.1e scrub starts 2024-02-05T08:06:34.850 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[30086]: 5.1e scrub ok 2024-02-05T08:06:34.851 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[30086]: 6.11 deep-scrub starts 2024-02-05T08:06:34.851 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[30086]: 6.11 deep-scrub ok 2024-02-05T08:06:34.851 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:34.851 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:34.851 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:34.851 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:34.851 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:34.851 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:06:34.851 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[39278]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:34.851 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[39278]: 4.2 scrub starts 2024-02-05T08:06:34.851 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[39278]: 4.2 scrub ok 2024-02-05T08:06:34.851 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[39278]: 5.1e scrub starts 2024-02-05T08:06:34.852 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[39278]: 5.1e scrub ok 2024-02-05T08:06:34.852 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[39278]: 6.11 deep-scrub starts 2024-02-05T08:06:34.852 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[39278]: 6.11 deep-scrub ok 2024-02-05T08:06:34.852 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:34.852 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:34.852 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:34.852 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:34.852 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:34.852 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:34 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:06:34.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:34.924 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:34.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:35.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:35.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-mon[33018]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:06:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-mon[33018]: 4.2 scrub starts 2024-02-05T08:06:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-mon[33018]: 4.2 scrub ok 2024-02-05T08:06:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-mon[33018]: 5.1e scrub starts 2024-02-05T08:06:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-mon[33018]: 5.1e scrub ok 2024-02-05T08:06:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-mon[33018]: 6.11 deep-scrub starts 2024-02-05T08:06:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-mon[33018]: 6.11 deep-scrub ok 2024-02-05T08:06:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:34 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:06:35.152 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:35.176 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:35.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:35.278 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:35.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:35.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:35.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:35.403 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:35.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:35.529 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:35.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:35.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:35.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:35.678 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:35.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:35.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:35.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:35.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:35.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:35.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:35.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:35.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:35.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:35.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:35.964 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:35 smithi134 ceph-mon[30086]: 3.0 deep-scrub starts 2024-02-05T08:06:35.964 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:35 smithi134 ceph-mon[30086]: 3.0 deep-scrub ok 2024-02-05T08:06:35.964 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:35 smithi134 ceph-mon[30086]: 4.1e scrub starts 2024-02-05T08:06:35.964 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:35 smithi134 ceph-mon[30086]: 4.1e scrub ok 2024-02-05T08:06:35.964 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:35 smithi134 ceph-mon[39278]: 3.0 deep-scrub starts 2024-02-05T08:06:35.964 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:35 smithi134 ceph-mon[39278]: 3.0 deep-scrub ok 2024-02-05T08:06:35.965 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:35 smithi134 ceph-mon[39278]: 4.1e scrub starts 2024-02-05T08:06:35.965 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:35 smithi134 ceph-mon[39278]: 4.1e scrub ok 2024-02-05T08:06:36.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:36.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:36.057 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:36.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:35 smithi196 ceph-mon[33018]: 3.0 deep-scrub starts 2024-02-05T08:06:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:35 smithi196 ceph-mon[33018]: 3.0 deep-scrub ok 2024-02-05T08:06:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:35 smithi196 ceph-mon[33018]: 4.1e scrub starts 2024-02-05T08:06:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:35 smithi196 ceph-mon[33018]: 4.1e scrub ok 2024-02-05T08:06:36.159 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:36.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:36.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:36.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:36.284 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:36.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:36.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:36.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:36.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:36.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:36.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:36.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:36.559 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:36.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:36.618 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'unique_pool_0' created 2024-02-05T08:06:36.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:36.684 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:36.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:36.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:36.810 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:36.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:36.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:36.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[30086]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:06:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[30086]: 5.1d scrub starts 2024-02-05T08:06:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[30086]: 5.1d scrub ok 2024-02-05T08:06:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[30086]: 6.16 scrub starts 2024-02-05T08:06:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[30086]: 6.16 scrub ok 2024-02-05T08:06:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/742072620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T08:06:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T08:06:36.924 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:36.924 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[39278]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:06:36.924 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[39278]: 5.1d scrub starts 2024-02-05T08:06:36.925 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[39278]: 5.1d scrub ok 2024-02-05T08:06:36.925 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[39278]: 6.16 scrub starts 2024-02-05T08:06:36.925 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[39278]: 6.16 scrub ok 2024-02-05T08:06:36.925 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/742072620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T08:06:36.925 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T08:06:36.925 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:36 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:36.939 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:36.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:36.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:37.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:36 smithi196 ceph-mon[33018]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:06:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:36 smithi196 ceph-mon[33018]: 5.1d scrub starts 2024-02-05T08:06:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:36 smithi196 ceph-mon[33018]: 5.1d scrub ok 2024-02-05T08:06:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:36 smithi196 ceph-mon[33018]: 6.16 scrub starts 2024-02-05T08:06:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:36 smithi196 ceph-mon[33018]: 6.16 scrub ok 2024-02-05T08:06:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/742072620' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T08:06:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:36 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T08:06:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:36 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:37.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:37.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:37.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:37.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:37.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:37.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:37.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:37.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:37.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:37.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:37.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:37.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:37.443 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:37.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:37.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:37.570 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:37.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:37.625 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:06:37.673 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:37.697 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:37.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:37.799 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:37.821 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:06:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:06:37.821 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-05T08:06:37.821 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[30086]: osdmap e95: 8 total, 8 up, 8 in 2024-02-05T08:06:37.821 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:06:37.821 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[30086]: 6.c deep-scrub starts 2024-02-05T08:06:37.821 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[30086]: 6.c deep-scrub ok 2024-02-05T08:06:37.821 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:06:37.821 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:06:37.822 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1947469970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:06:37.822 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:06:37.822 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:06:37.822 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:06:37.822 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[30086]: osdmap e96: 8 total, 8 up, 8 in 2024-02-05T08:06:37.822 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-05T08:06:37.822 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[39278]: osdmap e95: 8 total, 8 up, 8 in 2024-02-05T08:06:37.822 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:06:37.822 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[39278]: 6.c deep-scrub starts 2024-02-05T08:06:37.822 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[39278]: 6.c deep-scrub ok 2024-02-05T08:06:37.822 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:06:37.823 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:06:37.823 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1947469970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:06:37.823 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:06:37.823 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:06:37.823 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:06:37.823 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:37 smithi134 ceph-mon[39278]: osdmap e96: 8 total, 8 up, 8 in 2024-02-05T08:06:37.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:37.825 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:37.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:37.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:37.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:37.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:37.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:37.991 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:37 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-05T08:06:37.991 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:37 smithi196 ceph-mon[33018]: osdmap e95: 8 total, 8 up, 8 in 2024-02-05T08:06:37.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:37 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:06:37.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:37 smithi196 ceph-mon[33018]: 6.c deep-scrub starts 2024-02-05T08:06:37.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:37 smithi196 ceph-mon[33018]: 6.c deep-scrub ok 2024-02-05T08:06:37.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:37 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:06:37.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:37 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:06:37.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:37 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1947469970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:06:37.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:37 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:06:37.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:37 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:06:37.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:37 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:06:37.993 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:37 smithi196 ceph-mon[33018]: osdmap e96: 8 total, 8 up, 8 in 2024-02-05T08:06:38.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:38.077 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:38.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:38.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:38.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:38.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:38.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:38.303 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:38.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:38.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:38.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:38.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:38.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:38.454 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:38.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:38.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:38.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:38.580 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:38.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:38.625 INFO:teuthology.orchestra.run.smithi134.stderr:marked out osd.6. 2024-02-05T08:06:38.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:38.705 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:38.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:38.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a[30063]: 2024-02-05T08:06:38.457+0000 7f4523bf3700 -1 mon.a@0(leader).osd e96 definitely_dead 0 2024-02-05T08:06:38.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[30086]: 5.1b deep-scrub starts 2024-02-05T08:06:38.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[30086]: 5.1b deep-scrub ok 2024-02-05T08:06:38.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[30086]: pgmap v54: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:06:38.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[30086]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:06:38.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3097672409' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-05T08:06:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[30086]: Client client.admin marked osd.6 out, while it was still marked up 2024-02-05T08:06:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3097672409' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-05T08:06:38.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[30086]: osdmap e97: 8 total, 8 up, 7 in 2024-02-05T08:06:38.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[39278]: 5.1b deep-scrub starts 2024-02-05T08:06:38.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[39278]: 5.1b deep-scrub ok 2024-02-05T08:06:38.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[39278]: pgmap v54: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:06:38.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[39278]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:06:38.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3097672409' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-05T08:06:38.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[39278]: Client client.admin marked osd.6 out, while it was still marked up 2024-02-05T08:06:38.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3097672409' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-05T08:06:38.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:38 smithi134 ceph-mon[39278]: osdmap e97: 8 total, 8 up, 7 in 2024-02-05T08:06:38.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:38.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:38.831 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:38.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:38.862 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-02-05T08:06:38.912 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:38 smithi196 ceph-mon[33018]: 5.1b deep-scrub starts 2024-02-05T08:06:38.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:38 smithi196 ceph-mon[33018]: 5.1b deep-scrub ok 2024-02-05T08:06:38.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:38 smithi196 ceph-mon[33018]: pgmap v54: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:06:38.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:38 smithi196 ceph-mon[33018]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:06:38.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:38 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3097672409' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-02-05T08:06:38.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:38 smithi196 ceph-mon[33018]: Client client.admin marked osd.6 out, while it was still marked up 2024-02-05T08:06:38.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:38 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3097672409' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-02-05T08:06:38.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:38 smithi196 ceph-mon[33018]: osdmap e97: 8 total, 8 up, 7 in 2024-02-05T08:06:38.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:38.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:38.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:38.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:39.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:39.081 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:39.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:39.084 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:06:39.084 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:06:39.084 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-02-05T08:06:39.085 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:06:39.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:39.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:39.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:39.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:39.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:39.332 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:39.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:39.434 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:39.458 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:39.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:39.560 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:39.589 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:39.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:39.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:39.691 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:39.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:39.716 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:39.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:39.817 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:39.841 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:39.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:39.943 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:39.967 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:39.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:40.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:40.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:40.093 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:40.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:40.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:40.220 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:40.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:40.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:40.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:40 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:40.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:40 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:40.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:40 smithi134 ceph-mon[30086]: osdmap e98: 8 total, 8 up, 7 in 2024-02-05T08:06:40.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:40 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:40.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:40 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:40.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:40 smithi134 ceph-mon[39278]: osdmap e98: 8 total, 8 up, 7 in 2024-02-05T08:06:40.322 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:40.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:40.346 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:40.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:40.447 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:40.473 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:40.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:40 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:40 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:40 smithi196 ceph-mon[33018]: osdmap e98: 8 total, 8 up, 7 in 2024-02-05T08:06:40.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:40.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:40.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:40.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:40.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:40.724 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:40.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:40.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:40.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:40.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:40.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:40.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T08:06:40.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:40.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:40.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:40.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:41.078 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:41.102 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:41.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:41.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:41.204 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:41.228 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:41.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:41.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:41.355 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:41.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:41.411 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:41 smithi134 ceph-mon[30086]: pgmap v57: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:41.411 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:41 smithi134 ceph-mon[30086]: osdmap e99: 8 total, 8 up, 7 in 2024-02-05T08:06:41.412 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:41 smithi134 ceph-mon[39278]: pgmap v57: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:41.412 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:41 smithi134 ceph-mon[39278]: osdmap e99: 8 total, 8 up, 7 in 2024-02-05T08:06:41.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:41.480 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:41.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:41.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:41 smithi196 ceph-mon[33018]: pgmap v57: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:06:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:41 smithi196 ceph-mon[33018]: osdmap e99: 8 total, 8 up, 7 in 2024-02-05T08:06:41.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:41.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:41.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:41.708 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:41.732 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:41.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:41.834 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:41.858 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:41.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:41.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:41.960 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:41.984 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:41.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:41.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:42.086 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:42.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:42.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:42.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:42.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:42.235 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:42.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:42.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:42.306 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:06:42.308 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:42.324 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:42.334 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:06:42.337 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:42.361 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:06:42.364 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:42.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:42.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:42.381 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:06:42.401 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:06:42.421 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:06:42.443 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:06:42.460 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:06:42.461 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:06:42.461 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:06:42.461 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:06:42.461 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:06:42.461 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:06:42.462 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:06:42.462 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:06:42.462 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:06:42.466 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:42.491 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:42.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:42.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:42.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:42.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:42.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:42.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:42.661 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:42 smithi134 ceph-mon[30086]: pgmap v60: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:06:42.661 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:42 smithi134 ceph-mon[30086]: osdmap e100: 8 total, 8 up, 7 in 2024-02-05T08:06:42.661 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:42 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3089860075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:06:42.662 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:42 smithi134 ceph-mon[39278]: pgmap v60: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:06:42.662 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:42 smithi134 ceph-mon[39278]: osdmap e100: 8 total, 8 up, 7 in 2024-02-05T08:06:42.662 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:42 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3089860075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:06:42.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:42.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:42.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:42.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:42.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:42 smithi196 ceph-mon[33018]: pgmap v60: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 211 MiB used, 626 GiB / 626 GiB avail 2024-02-05T08:06:42.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:42 smithi196 ceph-mon[33018]: osdmap e100: 8 total, 8 up, 7 in 2024-02-05T08:06:42.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:42 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3089860075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:06:42.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:42.869 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:42.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:42.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:42.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:42.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:42.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:43.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:43.123 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:43.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:43.225 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:43.249 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:43.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:43.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:06:42 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:06:43.352 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:43.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:43.377 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:43.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:43.478 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:43.503 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:43.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:43.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:43 smithi196 ceph-mon[33018]: osdmap e101: 8 total, 8 up, 7 in 2024-02-05T08:06:43.579 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:43 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:43.605 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:43.628 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:43.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:43.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:43.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:43.755 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:43.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:43.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:43 smithi134 ceph-mon[30086]: osdmap e101: 8 total, 8 up, 7 in 2024-02-05T08:06:43.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:43 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:43.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:43 smithi134 ceph-mon[39278]: osdmap e101: 8 total, 8 up, 7 in 2024-02-05T08:06:43.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:43 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:43.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:06:43.856 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:43.897 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:43.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:44.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:44.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:44.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:44.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:44.128 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:44.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:44.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:44.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:44.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:44.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:44.279 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:44.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:44.380 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:44.404 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:44.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:44.505 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:44.531 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:44.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:44.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:44.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:44.657 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:44.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:44.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:44.767 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:06:44.768 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:06:44.768 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-02-05T08:06:44.768 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.0 reset_purged_snaps_last 2024-02-05T08:06:44.782 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:44.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:44.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:44 smithi196 ceph-mon[33018]: pgmap v63: 148 pgs: 1 peering, 147 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 767 B/s rd, 0 op/s; 144 KiB/s, 14 objects/s recovering 2024-02-05T08:06:44.884 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:44.909 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:44.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:45.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:44 smithi134 ceph-mon[30086]: pgmap v63: 148 pgs: 1 peering, 147 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 767 B/s rd, 0 op/s; 144 KiB/s, 14 objects/s recovering 2024-02-05T08:06:45.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:44 smithi134 ceph-mon[39278]: pgmap v63: 148 pgs: 1 peering, 147 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 767 B/s rd, 0 op/s; 144 KiB/s, 14 objects/s recovering 2024-02-05T08:06:45.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:45.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:45.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:45.161 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:45.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:45.262 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:45.286 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:45.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:45.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:45.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:45.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:45.513 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:45.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:45.538 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:45.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:45.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:45.663 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:45.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:45.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:45.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:45.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:45.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:45.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:45.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:45.916 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:45.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:45.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:45.957 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:06:46.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:46.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:46.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:46.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:46.148 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:46.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:46.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:46.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:46.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:46.303 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:46.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:46.382 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:06:46.383 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:06:46.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:46.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:46.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:46.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:46.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:46.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:46.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:46.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:46.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:46.692 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:46.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:46.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:46.818 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:46.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:46.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:46.864 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:46 smithi134 ceph-mon[30086]: pgmap v64: 148 pgs: 1 peering, 147 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 515 B/s rd, 0 op/s; 97 KiB/s, 9 objects/s recovering 2024-02-05T08:06:46.864 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:46 smithi134 ceph-mon[39278]: pgmap v64: 148 pgs: 1 peering, 147 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 515 B/s rd, 0 op/s; 97 KiB/s, 9 objects/s recovering 2024-02-05T08:06:46.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:46.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:46.945 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:46.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:47.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:46 smithi196 ceph-mon[33018]: pgmap v64: 148 pgs: 1 peering, 147 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 515 B/s rd, 0 op/s; 97 KiB/s, 9 objects/s recovering 2024-02-05T08:06:47.069 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:47.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:47.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:47.171 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:47.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:47.195 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:47.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:47.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:47.319 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:47.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:47.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:47.421 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:47.445 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:47.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:47.547 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:47.573 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:47.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:47.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:47.675 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:47.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:47.699 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:47.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:47.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T08:06:47.801 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:47.829 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:47.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:47.930 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:47.954 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:47.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:47.999 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:06:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:06:48.055 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:48.080 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:48.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:48.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:48.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:48.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:48.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:48.216 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:48.216 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:48.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:48.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:48.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:48.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:48.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:48.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:48.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:48.568 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:48.588 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-02-05T08:06:48.596 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:48.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:48.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:48.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:48.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:48.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:48.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:48.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:48.853 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:48.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:48.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:48.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:48.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:48.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:49.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:48 smithi134 ceph-mon[30086]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 83 KiB/s, 8 objects/s recovering 2024-02-05T08:06:49.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:48 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4267442404' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:06:49.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:48 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:06:49.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:48 smithi134 ceph-mon[39278]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 83 KiB/s, 8 objects/s recovering 2024-02-05T08:06:49.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:48 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4267442404' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:06:49.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:48 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:06:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:48 smithi196 ceph-mon[33018]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 83 KiB/s, 8 objects/s recovering 2024-02-05T08:06:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:48 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4267442404' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:06:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:48 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:06:49.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:49.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:49.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:49.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:49.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:49.237 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:49.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:49.339 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:49.364 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:49.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:49.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:49.494 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:49.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:49.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:49.596 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:49.619 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:49.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:49.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:49.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:49.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:49.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:49.875 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:49.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:49.909 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:49 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:06:49.909 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:49 smithi134 ceph-mon[30086]: osdmap e102: 8 total, 8 up, 7 in 2024-02-05T08:06:49.909 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:49 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:06:49.909 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:49 smithi134 ceph-mon[39278]: osdmap e102: 8 total, 8 up, 7 in 2024-02-05T08:06:49.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:50.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:50.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:50.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:49 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:06:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:49 smithi196 ceph-mon[33018]: osdmap e102: 8 total, 8 up, 7 in 2024-02-05T08:06:50.107 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:50.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:50.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:50.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:50.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:50.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:50.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:50.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:50.336 DEBUG:teuthology.orchestra.run.smithi134:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-02-05T08:06:50.338 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T08:06:50.338 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T08:06:50.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:50.385 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:50.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:50.424 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:hints = 1 2024-02-05T08:06:50.424 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T08:06:50.424 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_83016 2024-02-05T08:06:50.424 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:06:50.424 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-02-05T08:06:50.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:50.535 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:50.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:50.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:50.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:50.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:50.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:50.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:50.810 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:50.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:50.853 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:50 smithi134 ceph-mon[30086]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 73 KiB/s, 7 objects/s recovering 2024-02-05T08:06:50.853 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:50 smithi134 ceph-mon[30086]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:06:50.853 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:50 smithi134 ceph-mon[39278]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 73 KiB/s, 7 objects/s recovering 2024-02-05T08:06:50.853 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:50 smithi134 ceph-mon[39278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:06:50.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:50.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:50.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:50.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:50.971 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:06:51.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:50 smithi196 ceph-mon[33018]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 73 KiB/s, 7 objects/s recovering 2024-02-05T08:06:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:50 smithi196 ceph-mon[33018]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:06:51.069 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:51.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:51.171 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:51.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:51.214 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:51.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:51.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:51.357 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:51.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:51.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:51.424 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 1 15 2805 2790 174.365 174.375 0.00176506 0.0056886 2024-02-05T08:06:51.460 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:51.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:51.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:51.599 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:51.605 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:06:51.649 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:51.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:51.751 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:51.800 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:51.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:51.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:51.928 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:51.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:52.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:51 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/745180177' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:06:52.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:51 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:06:52.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:51 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/745180177' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:06:52.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:51 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:06:52.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:52.057 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:52.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:51 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/745180177' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:06:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:51 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:06:52.159 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:52.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:52.207 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:52.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:52.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:52.349 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:52.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:52.424 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 2 16 5451 5435 169.832 165.312 0.00321436 0.00587467 2024-02-05T08:06:52.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:52.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:52.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:52.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:52.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:52.622 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:52.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:52.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:52.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:52.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:52.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:52.885 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:52 smithi134 ceph-mon[30086]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 572 B/s rd, 0 op/s 2024-02-05T08:06:52.885 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:52 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:06:52.885 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:52 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:06:52.885 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:52 smithi134 ceph-mon[30086]: osdmap e103: 8 total, 8 up, 7 in 2024-02-05T08:06:52.885 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:52 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:52.885 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:52 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:06:52.886 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:52 smithi134 ceph-mon[39278]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 572 B/s rd, 0 op/s 2024-02-05T08:06:52.886 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:52 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:06:52.886 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:52 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:06:52.886 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:52 smithi134 ceph-mon[39278]: osdmap e103: 8 total, 8 up, 7 in 2024-02-05T08:06:52.886 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:52 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:52.886 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:52 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:06:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:52.887 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:52.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:52.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:52 smithi196 ceph-mon[33018]: pgmap v68: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 572 B/s rd, 0 op/s 2024-02-05T08:06:52.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:52 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:06:52.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:52 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:06:52.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:52 smithi196 ceph-mon[33018]: osdmap e103: 8 total, 8 up, 7 in 2024-02-05T08:06:52.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:52 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:06:52.918 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:52 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:06:52.989 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:53.034 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:53.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:53.136 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:53.176 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:53.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:53.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:53.278 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:53.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:06:52 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:06:53.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:53.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:53.424 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 3 16 8097 8081 168.341 165.375 0.00865732 0.0059305 2024-02-05T08:06:53.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:53.455 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:53.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:53.557 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:53.599 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:53.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:53.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:53.746 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:53.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:53.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:53 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:53.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:53 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:53.848 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:53.889 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:53.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:53.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:54.021 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:54.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:54.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:53 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:06:54.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:54.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:54.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:54.250 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:54.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:54.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:54.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:54.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:54.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:54.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:54.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:54.424 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 4 16 10628 10612 165.799 158.188 0.00750082 0.00602225 2024-02-05T08:06:54.508 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:54.550 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:54.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:54.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:54.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:54.696 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:54.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:54.727 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.1 reset_purged_snaps_last 2024-02-05T08:06:54.799 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:54.858 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:54.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:54.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:54.986 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:54.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:54 smithi134 ceph-mon[39278]: pgmap v70: 148 pgs: 148 active+clean; 353 MiB data, 1009 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 704 op/s 2024-02-05T08:06:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:54 smithi134 ceph-mon[30086]: pgmap v70: 148 pgs: 148 active+clean; 353 MiB data, 1009 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 704 op/s 2024-02-05T08:06:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:54 smithi196 ceph-mon[33018]: pgmap v70: 148 pgs: 148 active+clean; 353 MiB data, 1009 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 704 op/s 2024-02-05T08:06:55.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:06:55.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:55.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:06:55.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:55.282 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:55.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:55.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:55.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:55.424 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 5 16 12908 12892 161.137 142.5 0.00425502 0.00619492 2024-02-05T08:06:55.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:55.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:55.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:55.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:55.555 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:55.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:55.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:55.657 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:55.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:55.712 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:55.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:55.813 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:55.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:55.841 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:55.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:55.942 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:55.969 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:55.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:56.070 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:56.112 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:56.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:56.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:56.214 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:56.258 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:56.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:56.360 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:56.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:56.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:56.424 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 6 16 15160 15144 157.737 140.75 0.00807749 0.00633063 2024-02-05T08:06:56.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:56.517 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:56.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:56.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:56.619 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:56.646 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:56.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:56.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:56.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:56.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:56.798 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:56.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:56.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:56.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:56.924 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:56.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:57.014 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:56 smithi134 ceph-mon[30086]: pgmap v71: 148 pgs: 148 active+clean; 353 MiB data, 1009 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 44 MiB/s wr, 704 op/s 2024-02-05T08:06:57.015 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:56 smithi134 ceph-mon[39278]: pgmap v71: 148 pgs: 148 active+clean; 353 MiB data, 1009 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 44 MiB/s wr, 704 op/s 2024-02-05T08:06:57.025 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:06:57.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:56 smithi196 ceph-mon[33018]: pgmap v71: 148 pgs: 148 active+clean; 353 MiB data, 1009 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 44 MiB/s wr, 704 op/s 2024-02-05T08:06:57.069 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:57.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:57.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:06:57.171 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:57.214 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:57.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:57.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:57.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:57.346 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:57.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:57.425 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 7 15 16986 16971 151.514 114.188 0.00937215 0.00659129 2024-02-05T08:06:57.447 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:57.475 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:57.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:57.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:57.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:57.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:57.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:57.696 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:06:57.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:57.732 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:57.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:57.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:57.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:57.921 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:06:57.921 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:06:57.921 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:06:57.921 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:06:57.922 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:06:57.922 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:06:57.922 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:06:57.922 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:06:57.922 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:06:57.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:06:57.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:06:57.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:06:57.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:06:57.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:06:57.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:06:57.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:06:57.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:57.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:58.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:58.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:06:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:06:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:06:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:57 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3086340555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:06:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:57 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:06:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:57 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:06:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:57 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3086340555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:06:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:57 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:06:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:57 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:06:58.051 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:58.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:57 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3086340555' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:06:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:57 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:06:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:57 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:06:58.154 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:58.197 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:58.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:58.299 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:58.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:58.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:58.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:58.425 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 8 15 18795 18780 146.706 113.062 0.00432277 0.00680925 2024-02-05T08:06:58.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:58.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:58.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:58.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:58.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:58.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:58.686 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:58.709 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:58.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:58.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:06:58.839 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:58.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:58.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:06:58.942 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:58.967 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:58.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:58.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:58 smithi134 ceph-mon[30086]: pgmap v72: 148 pgs: 148 active+clean; 1.0 GiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 120 MiB/s wr, 1.92k op/s 2024-02-05T08:06:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:06:58 smithi134 ceph-mon[30086]: osdmap e104: 8 total, 8 up, 7 in 2024-02-05T08:06:59.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:58 smithi134 ceph-mon[39278]: pgmap v72: 148 pgs: 148 active+clean; 1.0 GiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 120 MiB/s wr, 1.92k op/s 2024-02-05T08:06:59.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:06:58 smithi134 ceph-mon[39278]: osdmap e104: 8 total, 8 up, 7 in 2024-02-05T08:06:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:58 smithi196 ceph-mon[33018]: pgmap v72: 148 pgs: 148 active+clean; 1.0 GiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.1 KiB/s rd, 120 MiB/s wr, 1.92k op/s 2024-02-05T08:06:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:06:58 smithi196 ceph-mon[33018]: osdmap e104: 8 total, 8 up, 7 in 2024-02-05T08:06:59.069 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:59.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:59.110 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:59.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:59.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:06:59.254 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:59.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:59.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:06:59.356 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:59.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:59.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:59.425 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 9 16 20343 20327 141.148 96.6875 0.00537185 0.00707806 2024-02-05T08:06:59.485 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:06:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:59.512 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:06:59.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:06:59.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:06:59.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:59.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:59.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:06:59.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:06:59.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:59.767 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:59.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:06:59.868 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:06:59.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:06:59.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:06:59.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:06:59.998 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:00.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:00.041 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:00.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:00.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:00.169 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:00.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:00.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:00.271 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:00.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:00.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:00.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:00.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:00.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:00.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:00.425 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 10 16 21746 21730 135.8 87.6875 0.00657582 0.00728655 2024-02-05T08:07:00.524 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:00.552 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:00.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:00.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:00.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:00.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:00.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:00.809 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:00.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:00.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:00.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:00.951 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:00.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:00 smithi134 ceph-mon[30086]: pgmap v74: 148 pgs: 148 active+clean; 1.0 GiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-05T08:07:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:00 smithi134 ceph-mon[39278]: pgmap v74: 148 pgs: 148 active+clean; 1.0 GiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-05T08:07:01.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:01.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:00 smithi196 ceph-mon[33018]: pgmap v74: 148 pgs: 148 active+clean; 1.0 GiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-05T08:07:01.082 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:01.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:01.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:01.183 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:01.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:01.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:01.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:01.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:01.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:01.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:01.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:01.425 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 11 16 23427 23411 133.004 105.062 0.00182088 0.00750041 2024-02-05T08:07:01.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:01.485 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:01.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:01.586 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:01.626 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:01.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:01.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:01.754 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:01.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:01.856 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:01.897 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:01.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:01.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:02.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:02.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:02.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:02.151 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:02.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:02.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:02.291 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:02.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:02.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:02.393 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:02.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:02.417 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:02.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:02.425 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 12 16 25163 25147 130.962 108.5 0.00244887 0.0076066 2024-02-05T08:07:02.518 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:02.545 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:02.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:02.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:02.647 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:02.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:02.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:02.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:02.778 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:02.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:02.820 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:02.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:02.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:02.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:02 smithi196 ceph-mon[33018]: pgmap v75: 148 pgs: 148 active+clean; 1.0 GiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-02-05T08:07:02.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:02.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:02.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:03.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:02 smithi134 ceph-mon[30086]: pgmap v75: 148 pgs: 148 active+clean; 1.0 GiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-02-05T08:07:03.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:02 smithi134 ceph-mon[39278]: pgmap v75: 148 pgs: 148 active+clean; 1.0 GiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.0 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-02-05T08:07:03.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:03.076 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:03.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:03.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:03.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:03.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:03.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:03.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:03.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:03.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:07:02 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:07:03.322 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:03.374 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:03.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:03.425 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 13 16 26700 26684 128.274 96.0625 0.00788544 0.00778959 2024-02-05T08:07:03.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:03.519 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:03.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:03.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:03.621 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:03.663 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:03.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:03.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:03.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:03.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:03.812 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:03.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:03.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:03.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:03.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:03.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:03.980 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:07:03.980 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:07:03.980 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:07:03.980 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:07:03.980 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.980 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:19156 7.f 7:f07da8a0:::benchmark_data_smithi134_83016_object19155:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:07:03.980 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.637666+0000", 2024-02-05T08:07:03.980 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 5.3372683409999997, 2024-02-05T08:07:03.980 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14367015799999999, 2024-02-05T08:07:03.980 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 19156 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.637666+0000", 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.981 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.637666+0000", 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.637667+0000", 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.048e-06 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.637687+0000", 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9792999999999998e-05 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.982 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.637695+0000", 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1300000000000001e-06 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.637700+0000", 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.322e-06 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.640284+0000", 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002583784 2024-02-05T08:07:03.983 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.984 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.984 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:03.984 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.640390+0000", 2024-02-05T08:07:03.984 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000106376 2024-02-05T08:07:03.984 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.984 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.984 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:07:03.985 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.640510+0000", 2024-02-05T08:07:03.985 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011983499999999999 2024-02-05T08:07:03.985 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.985 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.985 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:07:03.985 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.641696+0000", 2024-02-05T08:07:03.985 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011860659999999999 2024-02-05T08:07:03.985 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.985 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644278+0000", 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0025819150000000002 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.781111+0000", 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13683325499999999 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.781296+0000", 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018474999999999999 2024-02-05T08:07:03.986 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.781336+0000", 2024-02-05T08:07:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9884e-05 2024-02-05T08:07:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:19169 7.6 7:609d1232:::benchmark_data_smithi134_83016_object19168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:07:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.642725+0000", 2024-02-05T08:07:03.987 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 5.3322091269999996, 2024-02-05T08:07:03.988 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14115056400000001, 2024-02-05T08:07:03.988 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:03.988 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:03.988 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:07:03.988 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:07:03.988 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:07:03.988 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 19169 2024-02-05T08:07:03.988 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.988 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:03.988 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.988 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642725+0000", 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642725+0000", 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642726+0000", 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1999999999999999e-06 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.989 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642751+0000", 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4997000000000001e-05 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642762+0000", 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1158000000000001e-05 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642770+0000", 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2779999999999996e-06 2024-02-05T08:07:03.990 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642952+0000", 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018238799999999999 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.643075+0000", 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012345600000000001 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.643234+0000", 2024-02-05T08:07:03.991 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015835099999999999 2024-02-05T08:07:03.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.992 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.992 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:07:03.992 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644310+0000", 2024-02-05T08:07:03.992 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001076385 2024-02-05T08:07:03.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.992 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.992 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:03.992 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.645841+0000", 2024-02-05T08:07:03.992 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015305189999999999 2024-02-05T08:07:03.992 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.992 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.992 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:03.993 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.783788+0000", 2024-02-05T08:07:03.993 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.137947819 2024-02-05T08:07:03.993 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.993 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.993 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:03.993 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.783841+0000", 2024-02-05T08:07:03.993 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2308000000000001e-05 2024-02-05T08:07:03.993 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.993 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.993 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:03.993 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.783875+0000", 2024-02-05T08:07:03.993 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4705000000000001e-05 2024-02-05T08:07:03.993 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:03.994 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:03.994 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:03.994 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.994 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.994 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:19182 7.f 7:ff37928f:::benchmark_data_smithi134_83016_object19181:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:07:03.994 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.649284+0000", 2024-02-05T08:07:03.994 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 5.3256495519999998, 2024-02-05T08:07:03.994 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13518719100000001, 2024-02-05T08:07:03.994 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:03.994 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:03.994 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:07:03.994 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:07:03.995 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:03 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:03.995 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:03 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:03.996 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:07:03.996 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 19182 2024-02-05T08:07:03.996 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.996 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:03.996 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.996 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:03.996 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649284+0000", 2024-02-05T08:07:03.996 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:03.996 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.996 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.996 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:03.996 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649284+0000", 2024-02-05T08:07:03.996 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649286+0000", 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.062e-06 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649329+0000", 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3838000000000003e-05 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:03.997 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649344+0000", 2024-02-05T08:07:03.998 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4491000000000001e-05 2024-02-05T08:07:03.998 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.998 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.998 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:03.998 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649351+0000", 2024-02-05T08:07:03.998 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5379999999999997e-06 2024-02-05T08:07:03.998 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.998 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.998 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:03.998 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649405+0000", 2024-02-05T08:07:03.998 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3498999999999997e-05 2024-02-05T08:07:03.998 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.998 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.999 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:03.999 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649519+0000", 2024-02-05T08:07:03.999 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000113973 2024-02-05T08:07:03.999 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.999 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.999 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:07:03.999 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649620+0000", 2024-02-05T08:07:03.999 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000100898 2024-02-05T08:07:03.999 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:03.999 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:03.999 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:07:03.999 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.650351+0000", 2024-02-05T08:07:03.999 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00073095899999999999 2024-02-05T08:07:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.650566+0000", 2024-02-05T08:07:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000214797 2024-02-05T08:07:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.784387+0000", 2024-02-05T08:07:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.133821577 2024-02-05T08:07:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.000 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.784437+0000", 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0284999999999997e-05 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.784472+0000", 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4273999999999997e-05 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20666 7.7 7:e8712a05:::benchmark_data_smithi134_83016_object20665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:07:04.001 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.663704+0000", 2024-02-05T08:07:04.002 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.3112301369999999, 2024-02-05T08:07:04.002 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15447154699999999, 2024-02-05T08:07:04.002 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.002 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:04.002 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:07:04.002 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:07:04.002 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:07:04.002 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20666 2024-02-05T08:07:04.002 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.002 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.002 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.002 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.002 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.663704+0000", 2024-02-05T08:07:04.003 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.003 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.003 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.003 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.003 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.663704+0000", 2024-02-05T08:07:04.003 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.003 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.003 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.003 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.003 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.663705+0000", 2024-02-05T08:07:04.003 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.54e-07 2024-02-05T08:07:04.003 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.003 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.004 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.004 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.663727+0000", 2024-02-05T08:07:04.004 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2719000000000001e-05 2024-02-05T08:07:04.004 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.004 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.004 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.004 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.663737+0000", 2024-02-05T08:07:04.004 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.312e-06 2024-02-05T08:07:04.004 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.004 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.005 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.005 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.663744+0000", 2024-02-05T08:07:04.005 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5610000000000001e-06 2024-02-05T08:07:04.005 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.005 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.005 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.005 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.665902+0000", 2024-02-05T08:07:04.005 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00215796 2024-02-05T08:07:04.005 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.006 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.006 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.006 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666007+0000", 2024-02-05T08:07:04.006 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010508 2024-02-05T08:07:04.006 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.006 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.006 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:07:04.006 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666127+0000", 2024-02-05T08:07:04.006 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011996400000000001 2024-02-05T08:07:04.006 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.006 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.007 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:07:04.007 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669365+0000", 2024-02-05T08:07:04.007 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032372899999999999 2024-02-05T08:07:04.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.007 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.007 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.007 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670222+0000", 2024-02-05T08:07:04.007 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00085750099999999999 2024-02-05T08:07:04.007 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.007 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.008 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.008 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818094+0000", 2024-02-05T08:07:04.008 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14787227999999999 2024-02-05T08:07:04.008 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.008 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.008 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:04.008 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818142+0000", 2024-02-05T08:07:04.008 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7299999999999998e-05 2024-02-05T08:07:04.008 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.008 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.008 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.008 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818175+0000", 2024-02-05T08:07:04.008 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3825999999999998e-05 2024-02-05T08:07:04.009 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.009 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.009 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.009 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.009 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.009 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20673 7.f 7:f0d5ee1c:::benchmark_data_smithi134_83016_object20672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:07:04.009 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.666508+0000", 2024-02-05T08:07:04.009 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.3084264909999996, 2024-02-05T08:07:04.009 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.149946568, 2024-02-05T08:07:04.009 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.010 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:04.010 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:07:04.010 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:07:04.010 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:07:04.010 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20673 2024-02-05T08:07:04.010 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.010 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.010 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.010 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.010 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666508+0000", 2024-02-05T08:07:04.010 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.010 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666508+0000", 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666509+0000", 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5600000000000004e-07 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666532+0000", 2024-02-05T08:07:04.011 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3519e-05 2024-02-05T08:07:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666543+0000", 2024-02-05T08:07:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0713000000000001e-05 2024-02-05T08:07:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666550+0000", 2024-02-05T08:07:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0779999999999998e-06 2024-02-05T08:07:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667364+0000", 2024-02-05T08:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000814506 2024-02-05T08:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667478+0000", 2024-02-05T08:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011346899999999999 2024-02-05T08:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667603+0000", 2024-02-05T08:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012499200000000001 2024-02-05T08:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669370+0000", 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017667609999999999 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816310+0000", 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14694006400000001 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816351+0000", 2024-02-05T08:07:04.014 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1686e-05 2024-02-05T08:07:04.015 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.015 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.015 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:04.015 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816413+0000", 2024-02-05T08:07:04.015 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2018000000000002e-05 2024-02-05T08:07:04.015 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.015 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.015 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.015 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816454+0000", 2024-02-05T08:07:04.015 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0806000000000002e-05 2024-02-05T08:07:04.015 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.015 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.015 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.016 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.016 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.016 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20676 7.6 7:66971f6f:::benchmark_data_smithi134_83016_object20675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:07:04.016 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.667124+0000", 2024-02-05T08:07:04.016 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.307810269, 2024-02-05T08:07:04.016 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14974601900000001, 2024-02-05T08:07:04.016 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.016 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:04.016 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:07:04.016 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:07:04.016 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:07:04.017 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20676 2024-02-05T08:07:04.017 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.017 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.017 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.017 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.017 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667124+0000", 2024-02-05T08:07:04.017 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.017 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.017 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.017 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.018 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667124+0000", 2024-02-05T08:07:04.018 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.018 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.018 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.018 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.018 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667125+0000", 2024-02-05T08:07:04.018 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8199999999999999e-07 2024-02-05T08:07:04.018 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.018 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.018 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.019 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667145+0000", 2024-02-05T08:07:04.019 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0829999999999999e-05 2024-02-05T08:07:04.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.019 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.019 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.019 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667155+0000", 2024-02-05T08:07:04.019 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6989999999999998e-06 2024-02-05T08:07:04.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.019 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.019 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.019 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667162+0000", 2024-02-05T08:07:04.019 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0380000000000002e-06 2024-02-05T08:07:04.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.020 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.020 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.020 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668327+0000", 2024-02-05T08:07:04.020 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011653309999999999 2024-02-05T08:07:04.020 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.020 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.020 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.020 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668440+0000", 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011294900000000001 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668571+0000", 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000130801 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672190+0000", 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036185119999999999 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.021 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816650+0000", 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14446054799999999 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816783+0000", 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013320000000000001 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816835+0000", 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1891000000000002e-05 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.022 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816870+0000", 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4437999999999998e-05 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20678 7.6 7:6bafa7b0:::benchmark_data_smithi134_83016_object20677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.669255+0000", 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.305679262, 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.147953636, 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:07:04.023 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:07:04.024 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:07:04.024 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20678 2024-02-05T08:07:04.024 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.024 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.024 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.024 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.024 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669255+0000", 2024-02-05T08:07:04.024 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.024 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.024 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.024 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.024 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669255+0000", 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669256+0000", 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.212e-06 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669276+0000", 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9670999999999999e-05 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.025 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.026 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669286+0000", 2024-02-05T08:07:04.026 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0563e-05 2024-02-05T08:07:04.026 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.026 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.026 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.026 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669294+0000", 2024-02-05T08:07:04.026 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6029999999999999e-06 2024-02-05T08:07:04.026 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.026 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.027 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.027 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669393+0000", 2024-02-05T08:07:04.027 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9204999999999996e-05 2024-02-05T08:07:04.027 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.028 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.028 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.028 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669513+0000", 2024-02-05T08:07:04.028 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012022 2024-02-05T08:07:04.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.028 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.028 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:07:04.028 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671056+0000", 2024-02-05T08:07:04.028 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015427990000000001 2024-02-05T08:07:04.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.028 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.029 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:07:04.029 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816671+0000", 2024-02-05T08:07:04.029 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14561491500000001 2024-02-05T08:07:04.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.029 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.029 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.029 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816705+0000", 2024-02-05T08:07:04.029 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4248000000000001e-05 2024-02-05T08:07:04.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.029 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.029 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.029 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817079+0000", 2024-02-05T08:07:04.030 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00037375199999999998 2024-02-05T08:07:04.030 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.030 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.030 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:04.030 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817168+0000", 2024-02-05T08:07:04.030 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8786e-05 2024-02-05T08:07:04.030 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.030 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.030 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.030 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817208+0000", 2024-02-05T08:07:04.030 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0661999999999998e-05 2024-02-05T08:07:04.030 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20681 7.7 7:ea75bb27:::benchmark_data_smithi134_83016_object20680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:07:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.670167+0000", 2024-02-05T08:07:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.3047672879999999, 2024-02-05T08:07:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.148127287, 2024-02-05T08:07:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:07:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:07:04.031 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:07:04.032 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20681 2024-02-05T08:07:04.032 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.032 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.032 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.032 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.032 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670167+0000", 2024-02-05T08:07:04.032 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.032 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.032 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.032 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.032 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670167+0000", 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670168+0000", 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7599999999999996e-07 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670195+0000", 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7202000000000001e-05 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670204+0000", 2024-02-05T08:07:04.033 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9910000000000008e-06 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670529+0000", 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00032530200000000002 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670654+0000", 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012481600000000001 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670757+0000", 2024-02-05T08:07:04.034 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010313 2024-02-05T08:07:04.035 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.035 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.035 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:07:04.035 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670878+0000", 2024-02-05T08:07:04.035 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000120541 2024-02-05T08:07:04.035 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.035 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.035 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.035 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818057+0000", 2024-02-05T08:07:04.035 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14717940500000001 2024-02-05T08:07:04.035 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.035 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.035 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:07:04.036 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818068+0000", 2024-02-05T08:07:04.036 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0699e-05 2024-02-05T08:07:04.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.036 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.036 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.036 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818216+0000", 2024-02-05T08:07:04.036 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000148088 2024-02-05T08:07:04.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.036 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.036 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:04.036 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818257+0000", 2024-02-05T08:07:04.037 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1600000000000002e-05 2024-02-05T08:07:04.037 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.037 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.037 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.037 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818294+0000", 2024-02-05T08:07:04.037 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6637000000000002e-05 2024-02-05T08:07:04.037 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.037 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.037 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.037 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.037 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.037 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20682 7.7 7:ec8770ce:::benchmark_data_smithi134_83016_object20681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:07:04.038 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.670551+0000", 2024-02-05T08:07:04.038 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.3043833920000001, 2024-02-05T08:07:04.038 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.153080151, 2024-02-05T08:07:04.038 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.038 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:04.038 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:07:04.038 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:07:04.038 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:07:04.038 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20682 2024-02-05T08:07:04.038 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670551+0000", 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670551+0000", 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670551+0000", 2024-02-05T08:07:04.039 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.99e-07 2024-02-05T08:07:04.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.040 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.040 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.040 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670579+0000", 2024-02-05T08:07:04.040 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7994999999999999e-05 2024-02-05T08:07:04.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.040 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.040 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.040 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670589+0000", 2024-02-05T08:07:04.040 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4080000000000004e-06 2024-02-05T08:07:04.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.040 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.040 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.041 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670596+0000", 2024-02-05T08:07:04.041 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8020000000000003e-06 2024-02-05T08:07:04.041 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.041 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.041 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.041 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816434+0000", 2024-02-05T08:07:04.041 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14583796900000001 2024-02-05T08:07:04.041 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.041 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.041 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.042 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816554+0000", 2024-02-05T08:07:04.042 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000120381 2024-02-05T08:07:04.042 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.042 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.042 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:07:04.042 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816699+0000", 2024-02-05T08:07:04.042 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014480800000000001 2024-02-05T08:07:04.042 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.042 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.042 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:07:04.042 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.820595+0000", 2024-02-05T08:07:04.042 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0038963589999999998 2024-02-05T08:07:04.042 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.822835+0000", 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0022401370000000001 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.823546+0000", 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071045400000000001 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.823595+0000", 2024-02-05T08:07:04.043 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9200000000000003e-05 2024-02-05T08:07:04.044 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.044 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.044 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.044 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.823631+0000", 2024-02-05T08:07:04.044 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5939000000000003e-05 2024-02-05T08:07:04.044 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.044 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.044 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.044 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.044 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.044 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:20673 7.f e104/95 ondisk, result = 0)", 2024-02-05T08:07:04.044 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.670931+0000", 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.3040029650000005, 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.145386667, 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670931+0000", 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670931+0000", 2024-02-05T08:07:04.045 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670932+0000", 2024-02-05T08:07:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7300000000000004e-07 2024-02-05T08:07:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670936+0000", 2024-02-05T08:07:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3939999999999998e-06 2024-02-05T08:07:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.046 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.047 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.047 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670947+0000", 2024-02-05T08:07:04.047 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0305999999999999e-05 2024-02-05T08:07:04.047 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.047 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.047 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.047 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670953+0000", 2024-02-05T08:07:04.047 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6329999999999999e-06 2024-02-05T08:07:04.047 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.048 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.048 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.048 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816293+0000", 2024-02-05T08:07:04.048 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.145339473 2024-02-05T08:07:04.048 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.048 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.048 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.048 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816303+0000", 2024-02-05T08:07:04.048 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.012e-05 2024-02-05T08:07:04.048 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.048 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.048 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.048 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816318+0000", 2024-02-05T08:07:04.049 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4768e-05 2024-02-05T08:07:04.049 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.049 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.049 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.049 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.049 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.049 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:20676 7.6 e104/95 ondisk, result = 0)", 2024-02-05T08:07:04.049 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.671365+0000", 2024-02-05T08:07:04.049 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.3035694810000003, 2024-02-05T08:07:04.049 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.145297906, 2024-02-05T08:07:04.050 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.050 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:07:04.050 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.050 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.050 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.050 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671365+0000", 2024-02-05T08:07:04.050 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.050 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.050 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.050 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.050 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671365+0000", 2024-02-05T08:07:04.051 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.051 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.051 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.051 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.051 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671365+0000", 2024-02-05T08:07:04.051 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1399999999999997e-07 2024-02-05T08:07:04.051 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.051 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671368+0000", 2024-02-05T08:07:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.309e-06 2024-02-05T08:07:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671377+0000", 2024-02-05T08:07:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4069999999999993e-06 2024-02-05T08:07:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.052 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671382+0000", 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6400000000000002e-06 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816635+0000", 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14525242199999999 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816644+0000", 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7809999999999994e-06 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.053 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816662+0000", 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8833000000000001e-05 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:20673 7.f e104/95 ondisk, result = 0)", 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.671934+0000", 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.3030002600000001, 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.144526025, 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:07:04.054 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671934+0000", 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671934+0000", 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671934+0000", 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4899999999999995e-07 2024-02-05T08:07:04.055 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.056 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.056 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.056 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671938+0000", 2024-02-05T08:07:04.056 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6330000000000002e-06 2024-02-05T08:07:04.056 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.056 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.056 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.056 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671950+0000", 2024-02-05T08:07:04.056 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2138e-05 2024-02-05T08:07:04.056 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.056 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.056 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.057 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671957+0000", 2024-02-05T08:07:04.057 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5780000000000004e-06 2024-02-05T08:07:04.057 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.057 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.057 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.057 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816340+0000", 2024-02-05T08:07:04.057 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14438348600000001 2024-02-05T08:07:04.057 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.057 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.057 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.057 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816347+0000", 2024-02-05T08:07:04.057 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4640000000000003e-06 2024-02-05T08:07:04.058 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.058 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.058 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.058 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816460+0000", 2024-02-05T08:07:04.058 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000113077 2024-02-05T08:07:04.058 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.058 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.058 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.058 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.058 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.058 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:20678 7.6 e104/95 ondisk, result = 0)", 2024-02-05T08:07:04.058 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.672005+0000", 2024-02-05T08:07:04.058 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.3029286019999997, 2024-02-05T08:07:04.059 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14470676399999999, 2024-02-05T08:07:04.059 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.059 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:07:04.059 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.059 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.059 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.059 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672005+0000", 2024-02-05T08:07:04.059 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.059 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.059 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.060 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.060 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672005+0000", 2024-02-05T08:07:04.060 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.060 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.060 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.060 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.060 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672006+0000", 2024-02-05T08:07:04.060 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7399999999999998e-07 2024-02-05T08:07:04.060 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.060 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.060 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.060 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672009+0000", 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1580000000000001e-06 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672017+0000", 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0749999999999998e-06 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672022+0000", 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3240000000000002e-06 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.061 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816694+0000", 2024-02-05T08:07:04.062 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.144671833 2024-02-05T08:07:04.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.062 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.062 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816700+0000", 2024-02-05T08:07:04.062 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1319999999999999e-06 2024-02-05T08:07:04.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.062 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816712+0000", 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1768e-05 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:20676 7.6 e104/95 ondisk, result = 0)", 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.672729+0000", 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.3022046840000003, 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14414776600000001, 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:07:04.063 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.064 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.064 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.064 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672729+0000", 2024-02-05T08:07:04.064 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.064 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.064 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.064 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.064 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672729+0000", 2024-02-05T08:07:04.064 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.064 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.064 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.064 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.064 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672730+0000", 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3499999999999996e-07 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672734+0000", 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.963e-06 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816648+0000", 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14391443000000001 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.065 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816658+0000", 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.251e-06 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816771+0000", 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011333200000000001 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816778+0000", 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0940000000000002e-06 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.066 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.067 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816877+0000", 2024-02-05T08:07:04.067 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9061000000000006e-05 2024-02-05T08:07:04.067 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.067 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.067 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.067 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.067 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.067 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:20686 7.5 e104/95 7:ae49ae8c:::benchmark_data_smithi134_83016_object20685:head v 104'1244, mlcod=104'1242)", 2024-02-05T08:07:04.067 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.673008+0000", 2024-02-05T08:07:04.067 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4.3019261450000004, 2024-02-05T08:07:04.067 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14735541999999999, 2024-02-05T08:07:04.067 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.068 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:04.068 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.068 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.068 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.068 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673008+0000", 2024-02-05T08:07:04.068 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.068 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.068 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.068 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673008+0000", 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673009+0000", 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8899999999999999e-07 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673168+0000", 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015927700000000001 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.069 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.070 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.070 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816728+0000", 2024-02-05T08:07:04.070 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14355969099999999 2024-02-05T08:07:04.070 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.070 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.070 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.070 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816734+0000", 2024-02-05T08:07:04.070 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8020000000000003e-06 2024-02-05T08:07:04.070 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817049+0000", 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031493299999999999 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817070+0000", 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1016999999999999e-05 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.820256+0000", 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031859319999999998 2024-02-05T08:07:04.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.072 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.072 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.072 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.820363+0000", 2024-02-05T08:07:04.072 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107079 2024-02-05T08:07:04.072 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.072 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.072 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.072 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.072 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21728 7.f 7:f122ef99:::benchmark_data_smithi134_83016_object21727:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:07:04.072 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.322403+0000", 2024-02-05T08:07:04.072 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 3.6525307059999998, 2024-02-05T08:07:04.072 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13493356500000001, 2024-02-05T08:07:04.073 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.073 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:04.073 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:07:04.073 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:07:04.073 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:07:04.073 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21728 2024-02-05T08:07:04.073 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.073 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.073 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.073 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322403+0000", 2024-02-05T08:07:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322403+0000", 2024-02-05T08:07:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322405+0000", 2024-02-05T08:07:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4139999999999999e-06 2024-02-05T08:07:04.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.075 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:04.076 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:03 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:04.077 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.077 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.077 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322425+0000", 2024-02-05T08:07:04.077 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0528999999999999e-05 2024-02-05T08:07:04.077 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.077 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.077 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.077 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322437+0000", 2024-02-05T08:07:04.078 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1516e-05 2024-02-05T08:07:04.078 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.078 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.078 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.078 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322444+0000", 2024-02-05T08:07:04.078 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3640000000000001e-06 2024-02-05T08:07:04.078 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.078 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.078 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.079 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.323349+0000", 2024-02-05T08:07:04.079 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00090460099999999999 2024-02-05T08:07:04.079 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.079 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.079 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.079 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.323461+0000", 2024-02-05T08:07:04.079 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011223299999999999 2024-02-05T08:07:04.079 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.079 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.079 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:07:04.080 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.323578+0000", 2024-02-05T08:07:04.080 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011699799999999999 2024-02-05T08:07:04.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.080 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.080 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:07:04.080 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325229+0000", 2024-02-05T08:07:04.080 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0016505650000000001 2024-02-05T08:07:04.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.080 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.080 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.080 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325811+0000", 2024-02-05T08:07:04.081 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00058246600000000002 2024-02-05T08:07:04.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.081 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.081 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.081 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.457245+0000", 2024-02-05T08:07:04.081 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.131433507 2024-02-05T08:07:04.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.081 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.081 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:04.081 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.457300+0000", 2024-02-05T08:07:04.081 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4993e-05 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.457337+0000", 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7379000000000002e-05 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:21730 7.d e104/97 7:b1aeee65:::benchmark_data_smithi134_83016_object21729:head v 104'1403, mlcod=104'1400)", 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.325103+0000", 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 3.6498315140000002, 2024-02-05T08:07:04.082 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13322199400000001, 2024-02-05T08:07:04.083 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.083 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:04.083 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.083 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.083 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.083 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325103+0000", 2024-02-05T08:07:04.083 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.083 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.083 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.083 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325103+0000", 2024-02-05T08:07:04.084 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.084 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.084 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325103+0000", 2024-02-05T08:07:04.084 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0100000000000004e-07 2024-02-05T08:07:04.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.084 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.084 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325125+0000", 2024-02-05T08:07:04.085 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1359999999999999e-05 2024-02-05T08:07:04.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.085 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.085 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.085 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325135+0000", 2024-02-05T08:07:04.085 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0876999999999999e-05 2024-02-05T08:07:04.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.085 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.085 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.085 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325143+0000", 2024-02-05T08:07:04.085 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0509999999999999e-06 2024-02-05T08:07:04.086 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.086 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.086 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.086 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326388+0000", 2024-02-05T08:07:04.086 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001245257 2024-02-05T08:07:04.086 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.086 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.086 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.086 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326400+0000", 2024-02-05T08:07:04.086 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2034e-05 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.458288+0000", 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13188823199999999 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.458325+0000", 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6482000000000001e-05 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.088 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21737 7.9 7:94ff69c1:::benchmark_data_smithi134_83016_object21736:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:07:04.088 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.326361+0000", 2024-02-05T08:07:04.088 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 3.648573082, 2024-02-05T08:07:04.088 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13403157700000001, 2024-02-05T08:07:04.088 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.088 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:04.088 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:07:04.088 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:07:04.089 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:07:04.089 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21737 2024-02-05T08:07:04.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.089 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.089 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.089 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326361+0000", 2024-02-05T08:07:04.089 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.090 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.090 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326361+0000", 2024-02-05T08:07:04.090 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.090 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.090 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.090 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326362+0000", 2024-02-05T08:07:04.090 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2979999999999999e-06 2024-02-05T08:07:04.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.091 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.091 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326412+0000", 2024-02-05T08:07:04.091 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9719999999999998e-05 2024-02-05T08:07:04.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.091 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.091 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326434+0000", 2024-02-05T08:07:04.092 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2439999999999999e-05 2024-02-05T08:07:04.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.092 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.092 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326442+0000", 2024-02-05T08:07:04.092 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.515e-06 2024-02-05T08:07:04.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.093 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.093 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326491+0000", 2024-02-05T08:07:04.093 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9438999999999999e-05 2024-02-05T08:07:04.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.093 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.093 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326599+0000", 2024-02-05T08:07:04.094 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107643 2024-02-05T08:07:04.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.094 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.094 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:07:04.094 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326723+0000", 2024-02-05T08:07:04.094 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012374700000000001 2024-02-05T08:07:04.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.094 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.095 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.095 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.460112+0000", 2024-02-05T08:07:04.095 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13338972499999999 2024-02-05T08:07:04.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.095 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:07:04.095 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.460125+0000", 2024-02-05T08:07:04.095 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2300000000000001e-05 2024-02-05T08:07:04.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.096 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.096 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.096 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.460158+0000", 2024-02-05T08:07:04.096 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2792e-05 2024-02-05T08:07:04.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.096 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.097 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:04.097 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.460349+0000", 2024-02-05T08:07:04.097 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019101200000000001 2024-02-05T08:07:04.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.097 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.097 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.097 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.460393+0000", 2024-02-05T08:07:04.097 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3946000000000002e-05 2024-02-05T08:07:04.098 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.098 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.098 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.098 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.098 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21741 7.9 7:9929f208:::benchmark_data_smithi134_83016_object21740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:07:04.098 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.327604+0000", 2024-02-05T08:07:04.098 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 3.6473304770000001, 2024-02-05T08:07:04.098 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13433927100000001, 2024-02-05T08:07:04.099 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.099 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:04.099 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:07:04.099 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:07:04.099 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:07:04.099 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21741 2024-02-05T08:07:04.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.099 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.100 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.100 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327604+0000", 2024-02-05T08:07:04.100 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.100 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.100 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.100 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.100 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327604+0000", 2024-02-05T08:07:04.100 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.101 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.101 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.101 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327605+0000", 2024-02-05T08:07:04.101 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.111e-06 2024-02-05T08:07:04.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.101 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.101 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.102 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327638+0000", 2024-02-05T08:07:04.102 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2975000000000002e-05 2024-02-05T08:07:04.102 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.102 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.102 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.102 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327671+0000", 2024-02-05T08:07:04.102 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2855000000000003e-05 2024-02-05T08:07:04.102 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.102 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.103 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.103 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327695+0000", 2024-02-05T08:07:04.103 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.421e-05 2024-02-05T08:07:04.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.103 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.103 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.103 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.457002+0000", 2024-02-05T08:07:04.103 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12930754899999999 2024-02-05T08:07:04.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.103 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.104 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.104 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.457153+0000", 2024-02-05T08:07:04.104 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015092299999999999 2024-02-05T08:07:04.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.104 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.104 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:07:04.104 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.457319+0000", 2024-02-05T08:07:04.104 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016542400000000001 2024-02-05T08:07:04.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.104 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.104 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.104 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.460440+0000", 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031209060000000001 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.460476+0000", 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5998999999999999e-05 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.461770+0000", 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0012941599999999999 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.105 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.461835+0000", 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4950000000000007e-05 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.461943+0000", 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010820899999999999 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21742 7.9 7:9a982efe:::benchmark_data_smithi134_83016_object21741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.327752+0000", 2024-02-05T08:07:04.106 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 3.6471819330000002, 2024-02-05T08:07:04.107 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13573534500000001, 2024-02-05T08:07:04.107 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:07:04.107 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:07:04.107 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:07:04.107 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:07:04.107 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:07:04.107 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21742 2024-02-05T08:07:04.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.107 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:07:04.107 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.107 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:07:04.107 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327752+0000", 2024-02-05T08:07:04.107 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.108 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:07:04.108 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327752+0000", 2024-02-05T08:07:04.108 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:07:04.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.108 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:07:04.108 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327753+0000", 2024-02-05T08:07:04.108 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8299999999999996e-07 2024-02-05T08:07:04.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.108 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327789+0000", 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5932000000000001e-05 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327798+0000", 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5289999999999997e-06 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327805+0000", 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9890000000000003e-06 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.109 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:07:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.460504+0000", 2024-02-05T08:07:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13269827100000001 2024-02-05T08:07:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:07:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.460710+0000", 2024-02-05T08:07:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020684499999999999 2024-02-05T08:07:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:07:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.460839+0000", 2024-02-05T08:07:04.110 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012876000000000001 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.463215+0000", 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023763259999999998 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.463245+0000", 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9924e-05 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.463372+0000", 2024-02-05T08:07:04.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000126907 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.463461+0000", 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8586999999999997e-05 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.463487+0000", 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6591999999999999e-05 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:07:04.112 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:07:04.113 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:07:04.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:04.114 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:04.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:04.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:04.243 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:04.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:04.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:04.384 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:04.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:04.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:04.426 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 14 15 28414 28399 126.761 107.188 0.00695146 0.00787796 2024-02-05T08:07:04.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:04.532 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:04.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:04.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:04.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:04.661 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:04.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:04.763 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:04.806 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:04.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:04.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:04.908 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:04.945 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:04.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:04.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:05.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:04 smithi134 ceph-mon[30086]: pgmap v76: 148 pgs: 148 active+clean; 1.5 GiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 123 MiB/s wr, 1.96k op/s 2024-02-05T08:07:05.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:04 smithi134 ceph-mon[39278]: pgmap v76: 148 pgs: 148 active+clean; 1.5 GiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 123 MiB/s wr, 1.96k op/s 2024-02-05T08:07:05.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:05.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:04 smithi196 ceph-mon[33018]: pgmap v76: 148 pgs: 148 active+clean; 1.5 GiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 123 MiB/s wr, 1.96k op/s 2024-02-05T08:07:05.075 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:05.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:05.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:05.219 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:05.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:05.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:05.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:05.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:05.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:05.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:05.426 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 15 16 30183 30167 125.677 110.5 0.0057878 0.00795111 2024-02-05T08:07:05.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:05.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:05.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:05.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:05.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:05.618 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:05.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:05.720 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:05.763 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:05.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:05.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:05.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T08:07:05.864 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:05.950 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:05.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:05.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:06.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:06.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:06.107 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:06.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:06.208 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:06.259 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:06.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:06.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:06.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:06.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:06.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:06.427 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 16 16 31809 31793 124.172 101.625 0.00632742 0.00804721 2024-02-05T08:07:06.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:07:06.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:06.515 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:06.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:06.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:06.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:06.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:06.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:06.746 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:06.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:06 smithi134 ceph-mon[39278]: pgmap v77: 148 pgs: 148 active+clean; 1.5 GiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 123 MiB/s wr, 1.96k op/s 2024-02-05T08:07:06.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:06 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:07:06.779 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:06 smithi134 ceph-mon[30086]: pgmap v77: 148 pgs: 148 active+clean; 1.5 GiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 123 MiB/s wr, 1.96k op/s 2024-02-05T08:07:06.779 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:06 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:07:06.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:06.813 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:06.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:06.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:06.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:06.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:07.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:06 smithi196 ceph-mon[33018]: pgmap v77: 148 pgs: 148 active+clean; 1.5 GiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 123 MiB/s wr, 1.96k op/s 2024-02-05T08:07:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:06 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:07:07.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:07.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:07.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:07.169 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:07.195 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:07.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:07.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:07.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:07.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:07.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:07.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:07.429 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 17 16 33163 33147 121.846 84.625 0.00662395 0.00820023 2024-02-05T08:07:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:07.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:07.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:07.557 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:07.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:07.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:07.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:07.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:07.712 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:07.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:07.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:07.877 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:07.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:07.907 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:07 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:07:07.907 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:07 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:07:07.907 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:07 smithi134 ceph-mon[30086]: osdmap e105: 8 total, 8 up, 7 in 2024-02-05T08:07:07.908 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:07:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:07:07.908 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:07 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:07:07.908 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:07 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:07:07.908 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:07 smithi134 ceph-mon[39278]: osdmap e105: 8 total, 8 up, 7 in 2024-02-05T08:07:07.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:08.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:08.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:08.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:07 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:07:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:07 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:07:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:07 smithi196 ceph-mon[33018]: osdmap e105: 8 total, 8 up, 7 in 2024-02-05T08:07:08.107 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:08.154 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:08.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:08.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:08.300 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:08.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:08.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:08.427 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 18 16 34777 34761 120.68 100.875 0.00858334 0.00827951 2024-02-05T08:07:08.428 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:08.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:08.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:08.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:08.557 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:08.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:08.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:08.611 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:08.616 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:08.658 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:08.709 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:08 smithi134 ceph-mon[39278]: pgmap v79: 148 pgs: 148 active+clean; 1.9 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T08:07:08.709 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:08 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:07:08.709 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:08 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:07:08.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:08.710 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:08.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:08.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:08.860 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:08.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:08.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:09.009 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:09.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:09.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:08 smithi134 ceph-mon[30086]: pgmap v79: 148 pgs: 148 active+clean; 1.9 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T08:07:09.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:08 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:07:09.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:08 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:07:09.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:08 smithi134 ceph-mon[30086]: osdmap e106: 8 total, 8 up, 7 in 2024-02-05T08:07:09.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:09.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:09.021 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:08 smithi134 ceph-mon[39278]: osdmap e106: 8 total, 8 up, 7 in 2024-02-05T08:07:09.021 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:09.021 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:08 smithi196 ceph-mon[33018]: pgmap v79: 148 pgs: 148 active+clean; 1.9 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T08:07:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:08 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:07:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:08 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:07:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:08 smithi196 ceph-mon[33018]: osdmap e106: 8 total, 8 up, 7 in 2024-02-05T08:07:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:09.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:09.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:09.170 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:09.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:09.271 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:09.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:09.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:09.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:09.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:09.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:09.427 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 19 16 36104 36088 118.693 82.9375 0.012812 0.00841876 2024-02-05T08:07:09.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:09.581 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:09.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:09.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:09.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:09.709 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:09.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:09.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:09.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:09.838 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:09.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:09.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:09.874 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:09.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:09.988 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:09.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:09.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:10.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:10.115 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:10.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:10.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:10.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:10.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:10.244 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:10.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:10.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:10.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:10.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:10.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:10.427 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:07:10.427924+0000 min lat: 0.00118368 max lat: 0.159692 avg lat: 0.00852817 2024-02-05T08:07:10.427 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:07:10.427 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 20 16 37508 37492 117.146 87.75 0.0217478 0.00852817 2024-02-05T08:07:10.439 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.2 reset_purged_snaps_last 2024-02-05T08:07:10.474 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:10.557 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:10.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:10.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:10.659 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[30086]: pgmap v81: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-02-05T08:07:10.659 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[30086]: osdmap e107: 8 total, 8 up, 7 in 2024-02-05T08:07:10.659 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:07:10.659 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:07:10.659 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:10.659 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[30086]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-05T08:07:10.659 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[30086]: Reconfiguring daemon alertmanager.a on smithi134 2024-02-05T08:07:10.659 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[39278]: pgmap v81: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-02-05T08:07:10.659 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[39278]: osdmap e107: 8 total, 8 up, 7 in 2024-02-05T08:07:10.659 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:07:10.659 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:07:10.660 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:10.660 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[39278]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-05T08:07:10.660 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:10 smithi134 ceph-mon[39278]: Reconfiguring daemon alertmanager.a on smithi134 2024-02-05T08:07:10.684 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:10.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:10.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:10.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:10.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:10.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:10.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:10 smithi196 ceph-mon[33018]: pgmap v81: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-02-05T08:07:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:10 smithi196 ceph-mon[33018]: osdmap e107: 8 total, 8 up, 7 in 2024-02-05T08:07:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:10 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:07:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:10 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:07:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:10 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:10 smithi196 ceph-mon[33018]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-05T08:07:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:10 smithi196 ceph-mon[33018]: Reconfiguring daemon alertmanager.a on smithi134 2024-02-05T08:07:10.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:10.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:10.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:10.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:11.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:11.069 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:11.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:11.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:11.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:11.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:11.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:11.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:11.298 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:11.327 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:11.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:11.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:11.427 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 21 16 38905 38889 115.725 87.3125 0.0084881 0.00863522 2024-02-05T08:07:11.430 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:11.470 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:11.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:11.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:11.572 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:11.613 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:11.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:11.714 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:11.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:11.760 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:11.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:11.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:11.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:11.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:11.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:11.986 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:12.023 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:12.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:12.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:12.124 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:12.151 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:12.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:12.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:12.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:12.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:12.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:12.381 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:12.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:12.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:12.427 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 22 15 40318 40303 114.481 88.375 0.0156537 0.00872944 2024-02-05T08:07:12.521 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:12.557 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:12.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:12.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:12.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:12.701 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:12.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:12.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:12.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:12 smithi196 ceph-mon[33018]: pgmap v83: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-02-05T08:07:12.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:12.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:12.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:12.931 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:12 smithi134 ceph-mon[30086]: pgmap v83: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-02-05T08:07:12.931 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:12 smithi134 ceph-mon[39278]: pgmap v83: 148 pgs: 148 active+clean; 2.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-02-05T08:07:12.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:12.998 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:12.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:13.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:13.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:13.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:13.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:13.226 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:13.251 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:13.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:13.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:13.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:07:12 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:07:13.353 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:13.393 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:13.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:13.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:13.427 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 23 16 41803 41787 113.536 92.75 0.00556633 0.00880188 2024-02-05T08:07:13.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:13.522 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:13.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:13.624 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:13.650 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:13.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:13.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:13.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:13.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:13.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:13.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:13.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:13.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:13.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:13.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:14.008 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:13 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:14.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:13 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:14.048 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:14.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:14.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:13 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:14.150 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:14.176 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:14.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:14.278 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:14.321 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:14.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:14.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:14.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:14.428 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 24 16 43302 43286 112.709 93.6875 0.0151793 0.00886591 2024-02-05T08:07:14.475 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:14.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:14.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:14.634 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:14.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:14.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:14.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:14 smithi134 ceph-mon[30086]: pgmap v84: 148 pgs: 148 active+clean; 2.5 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.5 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-02-05T08:07:14.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:14 smithi134 ceph-mon[30086]: osdmap e108: 8 total, 8 up, 7 in 2024-02-05T08:07:14.680 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:14 smithi134 ceph-mon[39278]: pgmap v84: 148 pgs: 148 active+clean; 2.5 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.5 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-02-05T08:07:14.680 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:14 smithi134 ceph-mon[39278]: osdmap e108: 8 total, 8 up, 7 in 2024-02-05T08:07:14.736 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:14.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:14.783 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:14.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:14.884 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:14.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:14.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:14.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:15.012 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:15.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:15.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:15.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:15.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:14 smithi196 ceph-mon[33018]: pgmap v84: 148 pgs: 148 active+clean; 2.5 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.5 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-02-05T08:07:15.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:14 smithi196 ceph-mon[33018]: osdmap e108: 8 total, 8 up, 7 in 2024-02-05T08:07:15.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:15.186 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:15.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:15.288 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:15.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:15.329 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:15.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:15.427 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 25 16 44483 44467 111.153 73.8125 0.0103846 0.0089879 2024-02-05T08:07:15.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:15.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:15.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:15.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:15.558 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:15.600 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:07:15.602 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:15.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:15.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:15.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:15.730 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:15.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:15.832 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:15.846 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:15 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4011146405' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:07:15.846 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:15 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:07:15.846 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:15 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4011146405' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:07:15.846 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:15 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:07:15.881 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:15.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:15.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:16.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:16.029 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:16.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:16.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:15 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4011146405' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:07:16.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:15 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:07:16.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:16.174 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:16.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:16.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:16.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:16.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:16.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:16.403 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:16.428 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 26 16 45720 45704 109.851 77.3125 0.008084 0.00909736 2024-02-05T08:07:16.430 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:16.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:16.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:16.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:16.570 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:16.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:16.671 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:16.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:16.718 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:16.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:16.819 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:16.859 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:16.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:16.926 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:07:16.926 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[30086]: pgmap v86: 148 pgs: 148 active+clean; 2.5 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 655 B/s rd, 57 MiB/s wr, 916 op/s 2024-02-05T08:07:16.927 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:07:16.927 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:07:16.927 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[30086]: osdmap e109: 8 total, 8 up, 7 in 2024-02-05T08:07:16.927 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[30086]: 4.7 scrub starts 2024-02-05T08:07:16.927 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[30086]: 4.7 scrub ok 2024-02-05T08:07:16.927 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[30086]: 6.d scrub starts 2024-02-05T08:07:16.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[30086]: 6.d scrub ok 2024-02-05T08:07:16.928 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[39278]: pgmap v86: 148 pgs: 148 active+clean; 2.5 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 655 B/s rd, 57 MiB/s wr, 916 op/s 2024-02-05T08:07:16.928 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:07:16.929 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:07:16.929 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[39278]: osdmap e109: 8 total, 8 up, 7 in 2024-02-05T08:07:16.929 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[39278]: 4.7 scrub starts 2024-02-05T08:07:16.929 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[39278]: 4.7 scrub ok 2024-02-05T08:07:16.929 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[39278]: 6.d scrub starts 2024-02-05T08:07:16.929 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:16 smithi134 ceph-mon[39278]: 6.d scrub ok 2024-02-05T08:07:16.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:16.971 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:07:16.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:16.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:17.021 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:07:17.063 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:07:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:16 smithi196 ceph-mon[33018]: pgmap v86: 148 pgs: 148 active+clean; 2.5 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 655 B/s rd, 57 MiB/s wr, 916 op/s 2024-02-05T08:07:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:16 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:07:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:16 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:07:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:16 smithi196 ceph-mon[33018]: osdmap e109: 8 total, 8 up, 7 in 2024-02-05T08:07:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:16 smithi196 ceph-mon[33018]: 4.7 scrub starts 2024-02-05T08:07:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:16 smithi196 ceph-mon[33018]: 4.7 scrub ok 2024-02-05T08:07:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:16 smithi196 ceph-mon[33018]: 6.d scrub starts 2024-02-05T08:07:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:16 smithi196 ceph-mon[33018]: 6.d scrub ok 2024-02-05T08:07:17.089 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:17.102 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:07:17.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:17.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:17.143 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:07:17.180 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:07:17.222 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:07:17.227 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:07:17.228 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:07:17.228 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:07:17.228 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:07:17.228 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:07:17.228 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:07:17.228 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:07:17.228 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:07:17.236 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:17.276 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:17.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:17.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:17.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:17.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:17.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:17.428 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 27 15 46907 46892 108.532 74.25 0.0151635 0.00920767 2024-02-05T08:07:17.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:17.564 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:17.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:17.666 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:17.706 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:17.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:17.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:17.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:17.835 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:17.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:17.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:17.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:17.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:17.978 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:17.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:17 smithi134 ceph-mon[30086]: 6.1c scrub starts 2024-02-05T08:07:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:17 smithi134 ceph-mon[30086]: 6.1c scrub ok 2024-02-05T08:07:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:17 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/602621699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:07:18.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:07:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:07:18.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:17 smithi134 ceph-mon[39278]: 6.1c scrub starts 2024-02-05T08:07:18.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:17 smithi134 ceph-mon[39278]: 6.1c scrub ok 2024-02-05T08:07:18.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:17 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/602621699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:07:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:17 smithi196 ceph-mon[33018]: 6.1c scrub starts 2024-02-05T08:07:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:17 smithi196 ceph-mon[33018]: 6.1c scrub ok 2024-02-05T08:07:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:17 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/602621699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:07:18.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:18.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:18.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:18.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:18.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:18.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:18.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:18.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:18.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:18.428 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 28 16 48244 48228 107.638 83.5 0.00428401 0.00928464 2024-02-05T08:07:18.460 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:18.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:18.504 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:18.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:18.606 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:18.635 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:07:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:18.644 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:18.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:18.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:18.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:18.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:18.872 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:18.924 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:18.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:18.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:19.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:19.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:18 smithi134 ceph-mon[30086]: 5.17 scrub starts 2024-02-05T08:07:19.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:18 smithi134 ceph-mon[30086]: 5.17 scrub ok 2024-02-05T08:07:19.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:18 smithi134 ceph-mon[30086]: pgmap v88: 148 pgs: 148 active+clean; 2.6 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T08:07:19.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:18 smithi134 ceph-mon[30086]: 4.13 scrub starts 2024-02-05T08:07:19.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:18 smithi134 ceph-mon[30086]: 4.13 scrub ok 2024-02-05T08:07:19.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:18 smithi134 ceph-mon[39278]: 5.17 scrub starts 2024-02-05T08:07:19.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:18 smithi134 ceph-mon[39278]: 5.17 scrub ok 2024-02-05T08:07:19.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:18 smithi134 ceph-mon[39278]: pgmap v88: 148 pgs: 148 active+clean; 2.6 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T08:07:19.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:18 smithi134 ceph-mon[39278]: 4.13 scrub starts 2024-02-05T08:07:19.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:18 smithi134 ceph-mon[39278]: 4.13 scrub ok 2024-02-05T08:07:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:18 smithi196 ceph-mon[33018]: 5.17 scrub starts 2024-02-05T08:07:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:18 smithi196 ceph-mon[33018]: 5.17 scrub ok 2024-02-05T08:07:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:18 smithi196 ceph-mon[33018]: pgmap v88: 148 pgs: 148 active+clean; 2.6 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T08:07:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:18 smithi196 ceph-mon[33018]: 4.13 scrub starts 2024-02-05T08:07:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:18 smithi196 ceph-mon[33018]: 4.13 scrub ok 2024-02-05T08:07:19.070 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:19.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:19.172 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:19.197 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:19.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:19.299 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:19.339 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:19.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:19.428 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 29 16 49462 49446 106.551 76.125 0.078125 0.00936632 2024-02-05T08:07:19.441 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:19.476 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:19.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:19.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:19.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:19.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:19.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:19.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:19.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:19.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:19.835 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:19.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:19.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:19.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:19.963 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:19.990 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:19.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[30086]: 3.7 scrub starts 2024-02-05T08:07:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[30086]: 3.7 scrub ok 2024-02-05T08:07:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[30086]: 2.1 scrub starts 2024-02-05T08:07:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[30086]: 2.1 scrub ok 2024-02-05T08:07:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[30086]: 6.17 scrub starts 2024-02-05T08:07:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[30086]: 6.17 scrub ok 2024-02-05T08:07:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[30086]: 5.5 scrub starts 2024-02-05T08:07:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[30086]: 5.5 scrub ok 2024-02-05T08:07:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[30086]: osdmap e110: 8 total, 8 up, 7 in 2024-02-05T08:07:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[39278]: 3.7 scrub starts 2024-02-05T08:07:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[39278]: 3.7 scrub ok 2024-02-05T08:07:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[39278]: 2.1 scrub starts 2024-02-05T08:07:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[39278]: 2.1 scrub ok 2024-02-05T08:07:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[39278]: 6.17 scrub starts 2024-02-05T08:07:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[39278]: 6.17 scrub ok 2024-02-05T08:07:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[39278]: 5.5 scrub starts 2024-02-05T08:07:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[39278]: 5.5 scrub ok 2024-02-05T08:07:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:19 smithi134 ceph-mon[39278]: osdmap e110: 8 total, 8 up, 7 in 2024-02-05T08:07:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:19 smithi196 ceph-mon[33018]: 3.7 scrub starts 2024-02-05T08:07:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:19 smithi196 ceph-mon[33018]: 3.7 scrub ok 2024-02-05T08:07:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:19 smithi196 ceph-mon[33018]: 2.1 scrub starts 2024-02-05T08:07:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:19 smithi196 ceph-mon[33018]: 2.1 scrub ok 2024-02-05T08:07:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:19 smithi196 ceph-mon[33018]: 6.17 scrub starts 2024-02-05T08:07:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:19 smithi196 ceph-mon[33018]: 6.17 scrub ok 2024-02-05T08:07:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:19 smithi196 ceph-mon[33018]: 5.5 scrub starts 2024-02-05T08:07:20.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:19 smithi196 ceph-mon[33018]: 5.5 scrub ok 2024-02-05T08:07:20.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:19 smithi196 ceph-mon[33018]: osdmap e110: 8 total, 8 up, 7 in 2024-02-05T08:07:20.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:20.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:20.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:20.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:20.222 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:20.248 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:20.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:20.350 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:20.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:20.405 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:20.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:20.424 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:20.428 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 30 15 50814 50799 105.818 84.5625 0.00902692 0.00944354 2024-02-05T08:07:20.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:20.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:20.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:20.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:20.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:20.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:20.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:20.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:20 smithi134 ceph-mon[30086]: pgmap v89: 148 pgs: 148 active+clean; 2.9 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-02-05T08:07:20.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:20 smithi134 ceph-mon[30086]: 3.4 scrub starts 2024-02-05T08:07:20.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:20 smithi134 ceph-mon[30086]: 3.4 scrub ok 2024-02-05T08:07:20.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:20 smithi134 ceph-mon[30086]: 3.1f scrub starts 2024-02-05T08:07:20.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:20 smithi134 ceph-mon[30086]: 3.1f scrub ok 2024-02-05T08:07:20.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:20 smithi134 ceph-mon[39278]: pgmap v89: 148 pgs: 148 active+clean; 2.9 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-02-05T08:07:20.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:20 smithi134 ceph-mon[39278]: 3.4 scrub starts 2024-02-05T08:07:20.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:20 smithi134 ceph-mon[39278]: 3.4 scrub ok 2024-02-05T08:07:20.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:20 smithi134 ceph-mon[39278]: 3.1f scrub starts 2024-02-05T08:07:20.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:20 smithi134 ceph-mon[39278]: 3.1f scrub ok 2024-02-05T08:07:20.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:20.830 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:20.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:20.933 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:20.958 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:20.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:20.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:21.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:20 smithi196 ceph-mon[33018]: pgmap v89: 148 pgs: 148 active+clean; 2.9 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-02-05T08:07:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:20 smithi196 ceph-mon[33018]: 3.4 scrub starts 2024-02-05T08:07:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:20 smithi196 ceph-mon[33018]: 3.4 scrub ok 2024-02-05T08:07:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:20 smithi196 ceph-mon[33018]: 3.1f scrub starts 2024-02-05T08:07:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:20 smithi196 ceph-mon[33018]: 3.1f scrub ok 2024-02-05T08:07:21.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:21.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:21.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:21.215 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:21.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:21.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:21.351 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:07:21.351 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:07:21.357 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:21.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:21.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:21.428 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 31 16 52071 52055 104.937 78.5 0.007871 0.00952385 2024-02-05T08:07:21.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:21.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:21.502 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:21.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:21.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:21.631 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:21.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:21.733 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:21.771 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:21.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:21.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:21 smithi134 ceph-mon[30086]: 6.6 scrub starts 2024-02-05T08:07:21.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:21 smithi134 ceph-mon[30086]: 6.6 scrub ok 2024-02-05T08:07:21.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:21 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]: dispatch 2024-02-05T08:07:21.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:21 smithi134 ceph-mon[39278]: 6.6 scrub starts 2024-02-05T08:07:21.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:21 smithi134 ceph-mon[39278]: 6.6 scrub ok 2024-02-05T08:07:21.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:21 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]: dispatch 2024-02-05T08:07:21.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:21.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:21.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:22.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:22.045 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:22.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:21 smithi196 ceph-mon[33018]: 6.6 scrub starts 2024-02-05T08:07:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:21 smithi196 ceph-mon[33018]: 6.6 scrub ok 2024-02-05T08:07:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:21 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]: dispatch 2024-02-05T08:07:22.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:22.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:22.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:22.276 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:22.304 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:22.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:22.405 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:22.428 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 32 16 53368 53352 104.19 81.0625 0.0109436 0.00959219 2024-02-05T08:07:22.431 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:22.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:22.481 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:07:22.533 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:22.563 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T08:07:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:22.588 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:22.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:22.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:22.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:22.764 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:22.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:22.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[39278]: pgmap v91: 148 pgs: 148 active+clean; 2.9 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 735 B/s rd, 59 MiB/s wr, 945 op/s 2024-02-05T08:07:22.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]': finished 2024-02-05T08:07:22.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[39278]: osdmap e111: 8 total, 8 up, 7 in 2024-02-05T08:07:22.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:07:22.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[39278]: 1.0 scrub starts 2024-02-05T08:07:22.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[39278]: 1.0 scrub ok 2024-02-05T08:07:22.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:07:22.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:07:22.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[39278]: 5.e scrub starts 2024-02-05T08:07:22.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[39278]: 5.e scrub ok 2024-02-05T08:07:22.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[39278]: 4.4 scrub starts 2024-02-05T08:07:22.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[39278]: 4.4 scrub ok 2024-02-05T08:07:22.780 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[30086]: pgmap v91: 148 pgs: 148 active+clean; 2.9 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 735 B/s rd, 59 MiB/s wr, 945 op/s 2024-02-05T08:07:22.780 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]': finished 2024-02-05T08:07:22.780 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[30086]: osdmap e111: 8 total, 8 up, 7 in 2024-02-05T08:07:22.780 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:07:22.780 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[30086]: 1.0 scrub starts 2024-02-05T08:07:22.781 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[30086]: 1.0 scrub ok 2024-02-05T08:07:22.781 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:07:22.781 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:07:22.781 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[30086]: 5.e scrub starts 2024-02-05T08:07:22.781 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[30086]: 5.e scrub ok 2024-02-05T08:07:22.781 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[30086]: 4.4 scrub starts 2024-02-05T08:07:22.782 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:22 smithi134 ceph-mon[30086]: 4.4 scrub ok 2024-02-05T08:07:22.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:22.892 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:22.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:22.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:22 smithi196 ceph-mon[33018]: pgmap v91: 148 pgs: 148 active+clean; 2.9 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 735 B/s rd, 59 MiB/s wr, 945 op/s 2024-02-05T08:07:22.950 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1a", "id": [3, 2]}]': finished 2024-02-05T08:07:22.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:22 smithi196 ceph-mon[33018]: osdmap e111: 8 total, 8 up, 7 in 2024-02-05T08:07:22.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:22 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:07:22.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:22 smithi196 ceph-mon[33018]: 1.0 scrub starts 2024-02-05T08:07:22.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:22 smithi196 ceph-mon[33018]: 1.0 scrub ok 2024-02-05T08:07:22.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:22 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:07:22.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:07:22.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:22 smithi196 ceph-mon[33018]: 5.e scrub starts 2024-02-05T08:07:22.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:22 smithi196 ceph-mon[33018]: 5.e scrub ok 2024-02-05T08:07:22.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:22 smithi196 ceph-mon[33018]: 4.4 scrub starts 2024-02-05T08:07:22.951 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:22 smithi196 ceph-mon[33018]: 4.4 scrub ok 2024-02-05T08:07:22.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:23.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:23.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:23.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:23.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:23.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:23.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:23.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:23.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:23.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:23.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:07:22 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:07:23.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:23.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:23.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:23.428 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 33 15 54504 54489 103.186 71.0625 0.0124578 0.00968527 2024-02-05T08:07:23.520 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:23.560 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:23.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:23.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:23.706 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:23.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:23.755 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:23 smithi134 ceph-mon[30086]: osdmap e112: 8 total, 8 up, 7 in 2024-02-05T08:07:23.755 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:23 smithi134 ceph-mon[30086]: 2.0 scrub starts 2024-02-05T08:07:23.755 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:23 smithi134 ceph-mon[30086]: 2.0 scrub ok 2024-02-05T08:07:23.755 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:23 smithi134 ceph-mon[30086]: 4.12 scrub starts 2024-02-05T08:07:23.755 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:23 smithi134 ceph-mon[30086]: 4.12 scrub ok 2024-02-05T08:07:23.755 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:23 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:23.755 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:23 smithi134 systemd[1]: Stopping Ceph alertmanager.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:07:23.756 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:23 smithi134 ceph-mon[39278]: osdmap e112: 8 total, 8 up, 7 in 2024-02-05T08:07:23.756 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:23 smithi134 ceph-mon[39278]: 2.0 scrub starts 2024-02-05T08:07:23.756 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:23 smithi134 ceph-mon[39278]: 2.0 scrub ok 2024-02-05T08:07:23.756 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:23 smithi134 ceph-mon[39278]: 4.12 scrub starts 2024-02-05T08:07:23.756 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:23 smithi134 ceph-mon[39278]: 4.12 scrub ok 2024-02-05T08:07:23.756 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:23 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:23.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:23.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:23.887 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:23.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:23.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:24.012 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:24.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:23 smithi196 ceph-mon[33018]: osdmap e112: 8 total, 8 up, 7 in 2024-02-05T08:07:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:23 smithi196 ceph-mon[33018]: 2.0 scrub starts 2024-02-05T08:07:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:23 smithi196 ceph-mon[33018]: 2.0 scrub ok 2024-02-05T08:07:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:23 smithi196 ceph-mon[33018]: 4.12 scrub starts 2024-02-05T08:07:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:23 smithi196 ceph-mon[33018]: 4.12 scrub ok 2024-02-05T08:07:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:23 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:24.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:24.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:24.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:24.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:24.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:24.294 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:24.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:24.395 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:24.429 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 34 16 55658 55642 102.271 72.0625 0.0112717 0.00977145 2024-02-05T08:07:24.435 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:24.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:24.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:24.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:24.568 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:24.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:24.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:24.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:24.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:24.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:24.825 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:24.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:24.927 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:24.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:24.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:25.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:24 smithi134 ceph-mon[30086]: pgmap v94: 148 pgs: 1 peering, 147 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-02-05T08:07:25.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:24 smithi134 ceph-mon[30086]: 4.1c scrub starts 2024-02-05T08:07:25.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:24 smithi134 ceph-mon[30086]: 4.1c scrub ok 2024-02-05T08:07:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:24 smithi134 ceph-mon[39278]: pgmap v94: 148 pgs: 1 peering, 147 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-02-05T08:07:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:24 smithi134 ceph-mon[39278]: 4.1c scrub starts 2024-02-05T08:07:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:24 smithi134 ceph-mon[39278]: 4.1c scrub ok 2024-02-05T08:07:25.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:25.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:24 smithi196 ceph-mon[33018]: pgmap v94: 148 pgs: 1 peering, 147 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-02-05T08:07:25.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:24 smithi196 ceph-mon[33018]: 4.1c scrub starts 2024-02-05T08:07:25.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:24 smithi196 ceph-mon[33018]: 4.1c scrub ok 2024-02-05T08:07:25.098 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:25.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:25.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:25.243 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:25.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:25.346 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:25.371 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:25.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:25.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:25.428 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 35 16 56833 56817 101.447 73.4375 0.0110201 0.009851 2024-02-05T08:07:25.473 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:25.507 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:25.516 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:25.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:25.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:25.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:25.675 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:25.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:25.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:25.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:25.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:25.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:25.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:25.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:26.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:25 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:07:26.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:25 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:07:26.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:25 smithi134 ceph-mon[30086]: 6.a scrub starts 2024-02-05T08:07:26.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:25 smithi134 ceph-mon[30086]: 6.a scrub ok 2024-02-05T08:07:26.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:25 smithi134 ceph-mon[30086]: 3.6 scrub starts 2024-02-05T08:07:26.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:25 smithi134 ceph-mon[30086]: 3.6 scrub ok 2024-02-05T08:07:26.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:25 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:07:26.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:25 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:07:26.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:25 smithi134 ceph-mon[39278]: 6.a scrub starts 2024-02-05T08:07:26.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:25 smithi134 ceph-mon[39278]: 6.a scrub ok 2024-02-05T08:07:26.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:25 smithi134 ceph-mon[39278]: 3.6 scrub starts 2024-02-05T08:07:26.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:25 smithi134 ceph-mon[39278]: 3.6 scrub ok 2024-02-05T08:07:26.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:26.060 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:26.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:25 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:07:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:25 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:07:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:25 smithi196 ceph-mon[33018]: 6.a scrub starts 2024-02-05T08:07:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:25 smithi196 ceph-mon[33018]: 6.a scrub ok 2024-02-05T08:07:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:25 smithi196 ceph-mon[33018]: 3.6 scrub starts 2024-02-05T08:07:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:25 smithi196 ceph-mon[33018]: 3.6 scrub ok 2024-02-05T08:07:26.121 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:26.162 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:26.189 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:26.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:26.250 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:26.290 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:26.344 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:26.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:26.430 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 36 16 57980 57964 100.618 71.6875 0.0103615 0.00993336 2024-02-05T08:07:26.445 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:26.496 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:26.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:26.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:26.635 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:26.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:26.737 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:26.781 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:26.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:26.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[30086]: 4.1d scrub starts 2024-02-05T08:07:26.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[30086]: 4.1d scrub ok 2024-02-05T08:07:26.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[30086]: pgmap v95: 148 pgs: 1 peering, 147 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-02-05T08:07:26.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[30086]: 6.15 scrub starts 2024-02-05T08:07:26.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[30086]: 6.15 scrub ok 2024-02-05T08:07:26.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[30086]: 5.13 scrub starts 2024-02-05T08:07:26.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[30086]: 5.13 scrub ok 2024-02-05T08:07:26.875 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[39278]: 4.1d scrub starts 2024-02-05T08:07:26.876 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[39278]: 4.1d scrub ok 2024-02-05T08:07:26.876 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[39278]: pgmap v95: 148 pgs: 1 peering, 147 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-02-05T08:07:26.876 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[39278]: 6.15 scrub starts 2024-02-05T08:07:26.876 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[39278]: 6.15 scrub ok 2024-02-05T08:07:26.876 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[39278]: 5.13 scrub starts 2024-02-05T08:07:26.876 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:26 smithi134 ceph-mon[39278]: 5.13 scrub ok 2024-02-05T08:07:26.883 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:26.932 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:26.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:27.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:27.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:27.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:27.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:26 smithi196 ceph-mon[33018]: 4.1d scrub starts 2024-02-05T08:07:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:26 smithi196 ceph-mon[33018]: 4.1d scrub ok 2024-02-05T08:07:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:26 smithi196 ceph-mon[33018]: pgmap v95: 148 pgs: 1 peering, 147 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-02-05T08:07:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:26 smithi196 ceph-mon[33018]: 6.15 scrub starts 2024-02-05T08:07:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:26 smithi196 ceph-mon[33018]: 6.15 scrub ok 2024-02-05T08:07:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:26 smithi196 ceph-mon[33018]: 5.13 scrub starts 2024-02-05T08:07:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:26 smithi196 ceph-mon[33018]: 5.13 scrub ok 2024-02-05T08:07:27.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:27.188 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:27.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:27.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:27.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:27.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:27.429 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 37 16 59141 59125 99.8593 72.5625 0.0179215 0.010008 2024-02-05T08:07:27.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:27.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:27.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:27.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:27.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:27.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:27.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:27.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:27.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:27.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:27.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:27 smithi134 ceph-mon[30086]: 3.1a scrub starts 2024-02-05T08:07:27.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:27 smithi134 ceph-mon[30086]: 3.1a scrub ok 2024-02-05T08:07:27.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:27 smithi134 ceph-mon[39278]: 3.1a scrub starts 2024-02-05T08:07:27.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:27 smithi134 ceph-mon[39278]: 3.1a scrub ok 2024-02-05T08:07:27.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:27.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:27.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:27.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:27.972 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:27.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:27.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:28.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:07:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:07:28.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:27 smithi196 ceph-mon[33018]: 3.1a scrub starts 2024-02-05T08:07:28.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:27 smithi196 ceph-mon[33018]: 3.1a scrub ok 2024-02-05T08:07:28.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:28.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:28.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:28.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:28.201 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:28.239 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:28.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:28.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:28.341 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:28.366 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:28.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:28.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:28.430 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 38 16 60346 60330 99.2124 75.3125 0.0105877 0.0100742 2024-02-05T08:07:28.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:28.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:28.516 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:28.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:28.617 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:28.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:28.854 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:28.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:28.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:28.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:28.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:29.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[39278]: pgmap v96: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.3 KiB/s rd, 59 MiB/s wr, 945 op/s; 13 B/s, 0 objects/s recovering 2024-02-05T08:07:29.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[39278]: 5.9 scrub starts 2024-02-05T08:07:29.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[39278]: 5.9 scrub ok 2024-02-05T08:07:29.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[39278]: 5.b scrub starts 2024-02-05T08:07:29.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[39278]: 5.b scrub ok 2024-02-05T08:07:29.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[39278]: 3.1e scrub starts 2024-02-05T08:07:29.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[39278]: 3.1e scrub ok 2024-02-05T08:07:29.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[39278]: 4.15 scrub starts 2024-02-05T08:07:29.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[39278]: 4.15 scrub ok 2024-02-05T08:07:29.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[30086]: pgmap v96: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.3 KiB/s rd, 59 MiB/s wr, 945 op/s; 13 B/s, 0 objects/s recovering 2024-02-05T08:07:29.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[30086]: 5.9 scrub starts 2024-02-05T08:07:29.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[30086]: 5.9 scrub ok 2024-02-05T08:07:29.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[30086]: 5.b scrub starts 2024-02-05T08:07:29.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[30086]: 5.b scrub ok 2024-02-05T08:07:29.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[30086]: 3.1e scrub starts 2024-02-05T08:07:29.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[30086]: 3.1e scrub ok 2024-02-05T08:07:29.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[30086]: 4.15 scrub starts 2024-02-05T08:07:29.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:28 smithi134 ceph-mon[30086]: 4.15 scrub ok 2024-02-05T08:07:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:28 smithi196 ceph-mon[33018]: pgmap v96: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.3 KiB/s rd, 59 MiB/s wr, 945 op/s; 13 B/s, 0 objects/s recovering 2024-02-05T08:07:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:28 smithi196 ceph-mon[33018]: 5.9 scrub starts 2024-02-05T08:07:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:28 smithi196 ceph-mon[33018]: 5.9 scrub ok 2024-02-05T08:07:29.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:28 smithi196 ceph-mon[33018]: 5.b scrub starts 2024-02-05T08:07:29.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:28 smithi196 ceph-mon[33018]: 5.b scrub ok 2024-02-05T08:07:29.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:28 smithi196 ceph-mon[33018]: 3.1e scrub starts 2024-02-05T08:07:29.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:28 smithi196 ceph-mon[33018]: 3.1e scrub ok 2024-02-05T08:07:29.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:28 smithi196 ceph-mon[33018]: 4.15 scrub starts 2024-02-05T08:07:29.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:28 smithi196 ceph-mon[33018]: 4.15 scrub ok 2024-02-05T08:07:29.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:29.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:29.114 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:29.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:29.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:29.251 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:29.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:29.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:29.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:29.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:29.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:29.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:29.430 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 39 16 61414 61398 98.3799 66.75 0.00529526 0.0101596 2024-02-05T08:07:29.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:29.519 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:29.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:29.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:29.621 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:29.660 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:29.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:29.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:29.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:29.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:29.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:29.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:29.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:29.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:29.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:30.016 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:07:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:07:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[39278]: 6.13 scrub starts 2024-02-05T08:07:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[39278]: 6.13 scrub ok 2024-02-05T08:07:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[39278]: osdmap e113: 8 total, 8 up, 7 in 2024-02-05T08:07:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[39278]: 3.16 scrub starts 2024-02-05T08:07:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[39278]: 3.16 scrub ok 2024-02-05T08:07:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[39278]: osdmap e114: 8 total, 8 up, 7 in 2024-02-05T08:07:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[39278]: 5.14 scrub starts 2024-02-05T08:07:30.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:07:30.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:07:30.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[30086]: 6.13 scrub starts 2024-02-05T08:07:30.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[30086]: 6.13 scrub ok 2024-02-05T08:07:30.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[30086]: osdmap e113: 8 total, 8 up, 7 in 2024-02-05T08:07:30.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[30086]: 3.16 scrub starts 2024-02-05T08:07:30.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[30086]: 3.16 scrub ok 2024-02-05T08:07:30.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[30086]: osdmap e114: 8 total, 8 up, 7 in 2024-02-05T08:07:30.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:29 smithi134 ceph-mon[30086]: 5.14 scrub starts 2024-02-05T08:07:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:30.039 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:30.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:29 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:07:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:29 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:07:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:29 smithi196 ceph-mon[33018]: 6.13 scrub starts 2024-02-05T08:07:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:29 smithi196 ceph-mon[33018]: 6.13 scrub ok 2024-02-05T08:07:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:29 smithi196 ceph-mon[33018]: osdmap e113: 8 total, 8 up, 7 in 2024-02-05T08:07:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:29 smithi196 ceph-mon[33018]: 3.16 scrub starts 2024-02-05T08:07:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:29 smithi196 ceph-mon[33018]: 3.16 scrub ok 2024-02-05T08:07:30.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:29 smithi196 ceph-mon[33018]: osdmap e114: 8 total, 8 up, 7 in 2024-02-05T08:07:30.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:29 smithi196 ceph-mon[33018]: 5.14 scrub starts 2024-02-05T08:07:30.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:30.167 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:30.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:30.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:30.269 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:30.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:30.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:30.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:30.430 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:07:30.430868+0000 min lat: 0.00118368 max lat: 0.159692 avg lat: 0.0102351 2024-02-05T08:07:30.430 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:07:30.430 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 40 16 62521 62505 97.65 69.1875 0.00709448 0.0102351 2024-02-05T08:07:30.435 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:30.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:30.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:30.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:30.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:30.577 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:30.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:30.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:30.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:30.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:30.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:30.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:30.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:30.861 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:30.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:30.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:31.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:31.003 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:31.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[30086]: pgmap v98: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.3 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T08:07:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[30086]: 3.d scrub starts 2024-02-05T08:07:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[30086]: 3.d scrub ok 2024-02-05T08:07:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[30086]: 5.14 scrub ok 2024-02-05T08:07:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[30086]: 5.d scrub starts 2024-02-05T08:07:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[30086]: 5.d scrub ok 2024-02-05T08:07:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[30086]: 4.9 scrub starts 2024-02-05T08:07:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[30086]: 4.9 scrub ok 2024-02-05T08:07:31.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[39278]: pgmap v98: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.3 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T08:07:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[39278]: 3.d scrub starts 2024-02-05T08:07:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[39278]: 3.d scrub ok 2024-02-05T08:07:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[39278]: 5.14 scrub ok 2024-02-05T08:07:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[39278]: 5.d scrub starts 2024-02-05T08:07:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[39278]: 5.d scrub ok 2024-02-05T08:07:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[39278]: 4.9 scrub starts 2024-02-05T08:07:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:30 smithi134 ceph-mon[39278]: 4.9 scrub ok 2024-02-05T08:07:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:30 smithi196 ceph-mon[33018]: pgmap v98: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.3 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T08:07:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:30 smithi196 ceph-mon[33018]: 3.d scrub starts 2024-02-05T08:07:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:30 smithi196 ceph-mon[33018]: 3.d scrub ok 2024-02-05T08:07:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:30 smithi196 ceph-mon[33018]: 5.14 scrub ok 2024-02-05T08:07:31.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:30 smithi196 ceph-mon[33018]: 5.d scrub starts 2024-02-05T08:07:31.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:30 smithi196 ceph-mon[33018]: 5.d scrub ok 2024-02-05T08:07:31.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:30 smithi196 ceph-mon[33018]: 4.9 scrub starts 2024-02-05T08:07:31.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:30 smithi196 ceph-mon[33018]: 4.9 scrub ok 2024-02-05T08:07:31.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:31.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:31.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:31.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:31.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:31.295 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:31.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:31.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:31.368 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.3 reset_purged_snaps_last 2024-02-05T08:07:31.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:31.421 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:31.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:31.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:31.431 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 41 16 63674 63658 97.0239 72.0625 0.01013 0.0103007 2024-02-05T08:07:31.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:31.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:31.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:31.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:31.651 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:31.678 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:31.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:31.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:31.805 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:31.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:31.907 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:31.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:31 smithi134 ceph-mon[30086]: 3.13 scrub starts 2024-02-05T08:07:31.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:31 smithi134 ceph-mon[30086]: 3.13 scrub ok 2024-02-05T08:07:31.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:31 smithi134 ceph-mon[30086]: 3.1d scrub starts 2024-02-05T08:07:31.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:31 smithi134 ceph-mon[30086]: 3.1d scrub ok 2024-02-05T08:07:31.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:31 smithi134 ceph-mon[30086]: 6.1b scrub starts 2024-02-05T08:07:31.932 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:31 smithi134 ceph-mon[39278]: 3.13 scrub starts 2024-02-05T08:07:31.932 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:31 smithi134 ceph-mon[39278]: 3.13 scrub ok 2024-02-05T08:07:31.932 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:31 smithi134 ceph-mon[39278]: 3.1d scrub starts 2024-02-05T08:07:31.932 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:31 smithi134 ceph-mon[39278]: 3.1d scrub ok 2024-02-05T08:07:31.932 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:31 smithi134 ceph-mon[39278]: 6.1b scrub starts 2024-02-05T08:07:31.932 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:31 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[77504]: ts=2024-02-05T08:07:31.691Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-05T08:07:31.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:31.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:32.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:32.060 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:32.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:31 smithi196 ceph-mon[33018]: 3.13 scrub starts 2024-02-05T08:07:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:31 smithi196 ceph-mon[33018]: 3.13 scrub ok 2024-02-05T08:07:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:31 smithi196 ceph-mon[33018]: 3.1d scrub starts 2024-02-05T08:07:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:31 smithi196 ceph-mon[33018]: 3.1d scrub ok 2024-02-05T08:07:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:31 smithi196 ceph-mon[33018]: 6.1b scrub starts 2024-02-05T08:07:32.163 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:32.189 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:32.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:32.291 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:32.334 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:32.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:32.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:32.431 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 42 16 64750 64734 96.3149 67.25 0.0204917 0.0103764 2024-02-05T08:07:32.435 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:32.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:32.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:32.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:32.562 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:32.598 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:32.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:32.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:32.745 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:32.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:32.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:32.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:32.883 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:32.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:32.984 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:33.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:33.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:33.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:33.024 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:32 smithi134 ceph-mon[30086]: pgmap v100: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 45 MiB/s wr, 723 op/s; 13 B/s, 0 objects/s recovering 2024-02-05T08:07:33.024 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:32 smithi134 ceph-mon[30086]: 6.1b scrub ok 2024-02-05T08:07:33.024 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:32 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1564120417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:07:33.025 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:32 smithi134 ceph-mon[30086]: 4.0 scrub starts 2024-02-05T08:07:33.025 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:32 smithi134 ceph-mon[39278]: pgmap v100: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 45 MiB/s wr, 723 op/s; 13 B/s, 0 objects/s recovering 2024-02-05T08:07:33.025 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:32 smithi134 ceph-mon[39278]: 6.1b scrub ok 2024-02-05T08:07:33.025 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:32 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1564120417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:07:33.025 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:32 smithi134 ceph-mon[39278]: 4.0 scrub starts 2024-02-05T08:07:33.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:07:32 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:07:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:32 smithi196 ceph-mon[33018]: pgmap v100: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 45 MiB/s wr, 723 op/s; 13 B/s, 0 objects/s recovering 2024-02-05T08:07:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:32 smithi196 ceph-mon[33018]: 6.1b scrub ok 2024-02-05T08:07:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:32 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1564120417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:07:33.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:32 smithi196 ceph-mon[33018]: 4.0 scrub starts 2024-02-05T08:07:33.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:33.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:33.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:33.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:33.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:33.282 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:33.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:33.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:33.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:33.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:33.431 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 43 16 65799 65783 95.5996 65.5625 0.00887433 0.010455 2024-02-05T08:07:33.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:33.554 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:33.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:33.655 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:33.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:33.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:33.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:33.784 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:33.812 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:33.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:33.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:33.957 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:33.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:33.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:33 smithi134 ceph-mon[30086]: 4.0 scrub ok 2024-02-05T08:07:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:33 smithi134 ceph-mon[30086]: 5.12 scrub starts 2024-02-05T08:07:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:33 smithi134 ceph-mon[30086]: 5.12 scrub ok 2024-02-05T08:07:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:33 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:33 smithi134 ceph-mon[30086]: 4.3 scrub starts 2024-02-05T08:07:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:33 smithi134 ceph-mon[30086]: 4.3 scrub ok 2024-02-05T08:07:34.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:33 smithi134 ceph-mon[39278]: 4.0 scrub ok 2024-02-05T08:07:34.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:33 smithi134 ceph-mon[39278]: 5.12 scrub starts 2024-02-05T08:07:34.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:33 smithi134 ceph-mon[39278]: 5.12 scrub ok 2024-02-05T08:07:34.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:33 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:34.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:33 smithi134 ceph-mon[39278]: 4.3 scrub starts 2024-02-05T08:07:34.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:33 smithi134 ceph-mon[39278]: 4.3 scrub ok 2024-02-05T08:07:34.059 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:33 smithi196 ceph-mon[33018]: 4.0 scrub ok 2024-02-05T08:07:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:33 smithi196 ceph-mon[33018]: 5.12 scrub starts 2024-02-05T08:07:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:33 smithi196 ceph-mon[33018]: 5.12 scrub ok 2024-02-05T08:07:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:33 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:33 smithi196 ceph-mon[33018]: 4.3 scrub starts 2024-02-05T08:07:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:33 smithi196 ceph-mon[33018]: 4.3 scrub ok 2024-02-05T08:07:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:34.099 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:34.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:34.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:34.245 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:34.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:34.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:34.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:34.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:34.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:34.431 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 44 16 66878 66862 94.9595 67.4375 0.00938923 0.0105236 2024-02-05T08:07:34.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:34.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:34.521 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:34.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:34.526 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:34 smithi134 podman[87148]: 2024-02-05 08:07:34.290285472 +0000 UTC m=+8.714515821 container died 6843e7bbe75accc92cdc239d5e3a71215e76d7a9c70662bb801beb035db43e83 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T08:07:34.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:34.647 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:34.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:34.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:34.749 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:34.774 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:34.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:34.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:34.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:34 smithi134 ceph-mon[30086]: pgmap v101: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T08:07:34.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:34 smithi134 ceph-mon[30086]: 4.f scrub starts 2024-02-05T08:07:34.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:34 smithi134 ceph-mon[30086]: 4.f scrub ok 2024-02-05T08:07:34.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:34 smithi134 ceph-mon[30086]: 6.0 scrub starts 2024-02-05T08:07:34.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:34 smithi134 ceph-mon[30086]: 6.0 scrub ok 2024-02-05T08:07:34.890 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:34 smithi134 podman[87148]: 2024-02-05 08:07:34.704702374 +0000 UTC m=+9.128932727 container cleanup 6843e7bbe75accc92cdc239d5e3a71215e76d7a9c70662bb801beb035db43e83 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T08:07:34.890 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:34 smithi134 bash[87148]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a 2024-02-05T08:07:34.891 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:34 smithi134 ceph-mon[39278]: pgmap v101: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T08:07:34.891 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:34 smithi134 ceph-mon[39278]: 4.f scrub starts 2024-02-05T08:07:34.891 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:34 smithi134 ceph-mon[39278]: 4.f scrub ok 2024-02-05T08:07:34.891 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:34 smithi134 ceph-mon[39278]: 6.0 scrub starts 2024-02-05T08:07:34.891 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:34 smithi134 ceph-mon[39278]: 6.0 scrub ok 2024-02-05T08:07:34.903 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:34.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:34.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:35.005 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:35.054 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:35.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:34 smithi196 ceph-mon[33018]: pgmap v101: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T08:07:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:34 smithi196 ceph-mon[33018]: 4.f scrub starts 2024-02-05T08:07:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:34 smithi196 ceph-mon[33018]: 4.f scrub ok 2024-02-05T08:07:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:34 smithi196 ceph-mon[33018]: 6.0 scrub starts 2024-02-05T08:07:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:34 smithi196 ceph-mon[33018]: 6.0 scrub ok 2024-02-05T08:07:35.108 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:07:35.109 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:07:35.109 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:07:35.109 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:07:35.109 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:07:35.109 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:07:35.109 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:07:35.109 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:07:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:07:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:07:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:07:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:07:35.109 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:07:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:07:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:07:35.110 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:07:35.155 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:35.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:35.183 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:35.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:35.277 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:34 smithi134 podman[87791]: 2024-02-05 08:07:34.889738335 +0000 UTC m=+3.130395560 container remove 6843e7bbe75accc92cdc239d5e3a71215e76d7a9c70662bb801beb035db43e83 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T08:07:35.277 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:34 smithi134 podman[87791]: 2024-02-05 08:07:34.991008572 +0000 UTC m=+3.231665806 volume remove 8039c1e931cc0ea1a0b3ba3b84cf8ef30a213fbbba1acbc24c7993147ddfe2f5 2024-02-05T08:07:35.284 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:35.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:35.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:35.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:35.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:35.431 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 45 16 67996 67980 94.4018 69.875 0.0144308 0.0105881 2024-02-05T08:07:35.446 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:35.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:35.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:35.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:35.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:35.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:35.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:35.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:35.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:35.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:35.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:35.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:35.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:35.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:36.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:36.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:07:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:07:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:07:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:07:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[30086]: 4.1 scrub starts 2024-02-05T08:07:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[30086]: 4.1 scrub ok 2024-02-05T08:07:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[30086]: 5.19 scrub starts 2024-02-05T08:07:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[30086]: 5.19 scrub ok 2024-02-05T08:07:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[30086]: osdmap e115: 8 total, 8 up, 7 in 2024-02-05T08:07:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[30086]: 4.11 scrub starts 2024-02-05T08:07:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[30086]: 4.11 scrub ok 2024-02-05T08:07:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:07:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:07:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:07:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:07:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[39278]: 4.1 scrub starts 2024-02-05T08:07:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[39278]: 4.1 scrub ok 2024-02-05T08:07:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[39278]: 5.19 scrub starts 2024-02-05T08:07:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[39278]: 5.19 scrub ok 2024-02-05T08:07:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[39278]: osdmap e115: 8 total, 8 up, 7 in 2024-02-05T08:07:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[39278]: 4.11 scrub starts 2024-02-05T08:07:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:35 smithi134 ceph-mon[39278]: 4.11 scrub ok 2024-02-05T08:07:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:35 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:07:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:35 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:07:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:35 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:07:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:35 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:07:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:35 smithi196 ceph-mon[33018]: 4.1 scrub starts 2024-02-05T08:07:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:35 smithi196 ceph-mon[33018]: 4.1 scrub ok 2024-02-05T08:07:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:35 smithi196 ceph-mon[33018]: 5.19 scrub starts 2024-02-05T08:07:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:35 smithi196 ceph-mon[33018]: 5.19 scrub ok 2024-02-05T08:07:36.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:35 smithi196 ceph-mon[33018]: osdmap e115: 8 total, 8 up, 7 in 2024-02-05T08:07:36.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:35 smithi196 ceph-mon[33018]: 4.11 scrub starts 2024-02-05T08:07:36.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:35 smithi196 ceph-mon[33018]: 4.11 scrub ok 2024-02-05T08:07:36.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:36.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:36.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:36.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:36.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:36.257 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:36.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:36.358 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:36.399 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:36.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:36.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:36.431 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 46 16 69003 68987 93.7178 62.9375 0.0103068 0.0106647 2024-02-05T08:07:36.501 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:36.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:36.542 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:36.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:36.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:36.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:36.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:36.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:36.768 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:36.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:36.837 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:36.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:36.888 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:36 smithi134 ceph-mon[30086]: pgmap v102: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T08:07:36.889 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:36 smithi134 ceph-mon[30086]: 3.19 scrub starts 2024-02-05T08:07:36.889 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:36 smithi134 ceph-mon[30086]: 3.19 scrub ok 2024-02-05T08:07:36.889 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:36 smithi134 ceph-mon[30086]: 3.8 scrub starts 2024-02-05T08:07:36.889 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:36 smithi134 ceph-mon[30086]: 3.8 scrub ok 2024-02-05T08:07:36.889 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:36 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:07:36.889 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:36 smithi134 ceph-mon[39278]: pgmap v102: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T08:07:36.889 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:36 smithi134 ceph-mon[39278]: 3.19 scrub starts 2024-02-05T08:07:36.889 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:36 smithi134 ceph-mon[39278]: 3.19 scrub ok 2024-02-05T08:07:36.890 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:36 smithi134 ceph-mon[39278]: 3.8 scrub starts 2024-02-05T08:07:36.890 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:36 smithi134 ceph-mon[39278]: 3.8 scrub ok 2024-02-05T08:07:36.890 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:36 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:07:36.938 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:36.985 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:36.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:36 smithi196 ceph-mon[33018]: pgmap v102: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T08:07:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:36 smithi196 ceph-mon[33018]: 3.19 scrub starts 2024-02-05T08:07:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:36 smithi196 ceph-mon[33018]: 3.19 scrub ok 2024-02-05T08:07:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:36 smithi196 ceph-mon[33018]: 3.8 scrub starts 2024-02-05T08:07:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:36 smithi196 ceph-mon[33018]: 3.8 scrub ok 2024-02-05T08:07:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:36 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:07:37.087 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:37.125 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:37.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:37.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:37.227 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:37.268 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:37.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:37.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:37.395 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:37.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:37.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:37.431 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 47 16 69988 69972 93.0335 61.5625 0.0144239 0.010742 2024-02-05T08:07:37.497 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:37.522 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:37.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:37.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:37.594 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:07:37.594 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:07:37.594 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-02-05T08:07:37.594 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-02-05T08:07:37.594 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:07:37.624 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:37.651 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:37.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:37.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:37.756 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:37.795 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:37.822 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:37.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:37.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:37.925 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:37.982 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:37.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:38.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[30086]: 3.10 scrub starts 2024-02-05T08:07:38.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[30086]: 5.1a scrub starts 2024-02-05T08:07:38.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[30086]: 5.1a scrub ok 2024-02-05T08:07:38.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[30086]: 3.10 scrub ok 2024-02-05T08:07:38.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[30086]: 5.1 scrub starts 2024-02-05T08:07:38.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[30086]: 5.1 scrub ok 2024-02-05T08:07:38.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[30086]: 4.1f scrub starts 2024-02-05T08:07:38.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[30086]: 4.1f scrub ok 2024-02-05T08:07:38.008 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:07:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:07:38.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[39278]: 3.10 scrub starts 2024-02-05T08:07:38.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[39278]: 5.1a scrub starts 2024-02-05T08:07:38.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[39278]: 5.1a scrub ok 2024-02-05T08:07:38.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[39278]: 3.10 scrub ok 2024-02-05T08:07:38.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[39278]: 5.1 scrub starts 2024-02-05T08:07:38.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[39278]: 5.1 scrub ok 2024-02-05T08:07:38.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[39278]: 4.1f scrub starts 2024-02-05T08:07:38.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:37 smithi134 ceph-mon[39278]: 4.1f scrub ok 2024-02-05T08:07:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:37 smithi196 ceph-mon[33018]: 3.10 scrub starts 2024-02-05T08:07:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:37 smithi196 ceph-mon[33018]: 5.1a scrub starts 2024-02-05T08:07:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:37 smithi196 ceph-mon[33018]: 5.1a scrub ok 2024-02-05T08:07:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:37 smithi196 ceph-mon[33018]: 3.10 scrub ok 2024-02-05T08:07:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:37 smithi196 ceph-mon[33018]: 5.1 scrub starts 2024-02-05T08:07:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:37 smithi196 ceph-mon[33018]: 5.1 scrub ok 2024-02-05T08:07:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:37 smithi196 ceph-mon[33018]: 4.1f scrub starts 2024-02-05T08:07:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:37 smithi196 ceph-mon[33018]: 4.1f scrub ok 2024-02-05T08:07:38.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:38.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:38.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:38.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:07:38.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:38.264 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:38.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:38.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:38.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:38.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:38.433 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 48 16 70950 70934 92.3453 60.125 0.0192183 0.0108235 2024-02-05T08:07:38.456 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T08:07:38.493 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:38.518 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:38.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:38.619 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:38.644 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:38.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:38.746 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:38.773 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:38 smithi134 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-05T08:07:38.773 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:38 smithi134 systemd[1]: Stopped Ceph alertmanager.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:07:38.773 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:38 smithi134 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@alertmanager.a.service: Consumed 1.313s CPU time. 2024-02-05T08:07:38.773 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:38 smithi134 systemd[1]: Starting Ceph alertmanager.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:07:38.773 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:07:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:38.818 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:38.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:38.919 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:38.966 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:38.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:39.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[30086]: pgmap v104: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 822 op/s 2024-02-05T08:07:39.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[30086]: 4.1b scrub starts 2024-02-05T08:07:39.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[30086]: 4.1b scrub ok 2024-02-05T08:07:39.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/467228589' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:07:39.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:07:39.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[30086]: 6.1d scrub starts 2024-02-05T08:07:39.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[30086]: 6.1d scrub ok 2024-02-05T08:07:39.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[39278]: pgmap v104: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 822 op/s 2024-02-05T08:07:39.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[39278]: 4.1b scrub starts 2024-02-05T08:07:39.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[39278]: 4.1b scrub ok 2024-02-05T08:07:39.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/467228589' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:07:39.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:07:39.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[39278]: 6.1d scrub starts 2024-02-05T08:07:39.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:38 smithi134 ceph-mon[39278]: 6.1d scrub ok 2024-02-05T08:07:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:38 smithi196 ceph-mon[33018]: pgmap v104: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 822 op/s 2024-02-05T08:07:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:38 smithi196 ceph-mon[33018]: 4.1b scrub starts 2024-02-05T08:07:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:38 smithi196 ceph-mon[33018]: 4.1b scrub ok 2024-02-05T08:07:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:38 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/467228589' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:07:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:38 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:07:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:38 smithi196 ceph-mon[33018]: 6.1d scrub starts 2024-02-05T08:07:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:38 smithi196 ceph-mon[33018]: 6.1d scrub ok 2024-02-05T08:07:39.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:39.094 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:39.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:39.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:39.196 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:39.233 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:39.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:39.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:39.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:39.377 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:39.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:39.433 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 49 16 71928 71912 91.7075 61.125 0.00977393 0.0108986 2024-02-05T08:07:39.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:39.556 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:39.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:39.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:39.658 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:39.707 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:39.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:39.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:39.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:39.848 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:39.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:39.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:39.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:39.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:39.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:39.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[30086]: 6.18 scrub starts 2024-02-05T08:07:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[30086]: 6.18 scrub ok 2024-02-05T08:07:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[30086]: 5.8 scrub starts 2024-02-05T08:07:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[30086]: 5.8 scrub ok 2024-02-05T08:07:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:07:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:07:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:07:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[30086]: osdmap e116: 8 total, 8 up, 7 in 2024-02-05T08:07:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[30086]: 4.d scrub starts 2024-02-05T08:07:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[30086]: 4.d scrub ok 2024-02-05T08:07:40.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[39278]: 6.18 scrub starts 2024-02-05T08:07:40.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[39278]: 6.18 scrub ok 2024-02-05T08:07:40.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[39278]: 5.8 scrub starts 2024-02-05T08:07:40.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[39278]: 5.8 scrub ok 2024-02-05T08:07:40.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:07:40.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:07:40.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:07:40.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[39278]: osdmap e116: 8 total, 8 up, 7 in 2024-02-05T08:07:40.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[39278]: 4.d scrub starts 2024-02-05T08:07:40.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:39 smithi134 ceph-mon[39278]: 4.d scrub ok 2024-02-05T08:07:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:39 smithi196 ceph-mon[33018]: 6.18 scrub starts 2024-02-05T08:07:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:39 smithi196 ceph-mon[33018]: 6.18 scrub ok 2024-02-05T08:07:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:39 smithi196 ceph-mon[33018]: 5.8 scrub starts 2024-02-05T08:07:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:39 smithi196 ceph-mon[33018]: 5.8 scrub ok 2024-02-05T08:07:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:39 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:07:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:39 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:07:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:39 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:07:40.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:39 smithi196 ceph-mon[33018]: osdmap e116: 8 total, 8 up, 7 in 2024-02-05T08:07:40.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:39 smithi196 ceph-mon[33018]: 4.d scrub starts 2024-02-05T08:07:40.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:39 smithi196 ceph-mon[33018]: 4.d scrub ok 2024-02-05T08:07:40.079 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:40.124 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:40.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:40.226 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:40.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:40.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:40.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:40.354 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:40.397 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:40.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:40.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:40.433 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 50 16 72985 72969 91.1945 66.0625 0.00925332 0.0109602 2024-02-05T08:07:40.499 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:40.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:40.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:40.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:40.575 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:40.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:40.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:40.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:40.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:40.750 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:40.755 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:40.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:40.822 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:40.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:40.923 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:40.981 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:40.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:40.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[30086]: pgmap v106: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-02-05T08:07:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:07:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:07:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[30086]: 3.1c scrub starts 2024-02-05T08:07:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[30086]: 3.1c scrub ok 2024-02-05T08:07:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[30086]: 6.1e deep-scrub starts 2024-02-05T08:07:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[30086]: 6.1e deep-scrub ok 2024-02-05T08:07:41.028 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:40 smithi134 podman[88864]: 2024-02-05 08:07:40.849824412 +0000 UTC m=+0.099567196 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-05T08:07:41.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[39278]: pgmap v106: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-02-05T08:07:41.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:07:41.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:07:41.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[39278]: 3.1c scrub starts 2024-02-05T08:07:41.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[39278]: 3.1c scrub ok 2024-02-05T08:07:41.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[39278]: 6.1e deep-scrub starts 2024-02-05T08:07:41.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:40 smithi134 ceph-mon[39278]: 6.1e deep-scrub ok 2024-02-05T08:07:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:40 smithi196 ceph-mon[33018]: pgmap v106: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 85 MiB/s wr, 1.35k op/s 2024-02-05T08:07:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:40 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:07:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:40 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:07:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:40 smithi196 ceph-mon[33018]: 3.1c scrub starts 2024-02-05T08:07:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:40 smithi196 ceph-mon[33018]: 3.1c scrub ok 2024-02-05T08:07:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:40 smithi196 ceph-mon[33018]: 6.1e deep-scrub starts 2024-02-05T08:07:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:40 smithi196 ceph-mon[33018]: 6.1e deep-scrub ok 2024-02-05T08:07:41.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:41.126 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:41.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:41.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:41.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:41.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:41.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:41.383 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:41.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:41.433 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 51 16 73947 73931 90.5852 60.125 0.0040198 0.0110341 2024-02-05T08:07:41.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:41.510 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:41.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:41.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:41.612 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:41.660 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:41.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:41.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:41.789 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:41.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:41.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:41.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:41.919 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:41.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:42.021 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:42.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:07:42.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:07:42.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[30086]: 6.8 scrub starts 2024-02-05T08:07:42.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[30086]: 6.8 scrub ok 2024-02-05T08:07:42.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[30086]: 2.2 scrub starts 2024-02-05T08:07:42.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[30086]: 2.2 scrub ok 2024-02-05T08:07:42.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[30086]: 6.2 scrub starts 2024-02-05T08:07:42.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[30086]: 6.2 scrub ok 2024-02-05T08:07:42.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[30086]: 6.1 scrub starts 2024-02-05T08:07:42.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[30086]: 6.1 scrub ok 2024-02-05T08:07:42.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:07:42.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:07:42.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[39278]: 6.8 scrub starts 2024-02-05T08:07:42.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[39278]: 6.8 scrub ok 2024-02-05T08:07:42.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[39278]: 2.2 scrub starts 2024-02-05T08:07:42.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[39278]: 2.2 scrub ok 2024-02-05T08:07:42.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[39278]: 6.2 scrub starts 2024-02-05T08:07:42.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[39278]: 6.2 scrub ok 2024-02-05T08:07:42.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[39278]: 6.1 scrub starts 2024-02-05T08:07:42.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:41 smithi134 ceph-mon[39278]: 6.1 scrub ok 2024-02-05T08:07:42.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:42.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:42.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:41 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:07:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:41 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:07:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:41 smithi196 ceph-mon[33018]: 6.8 scrub starts 2024-02-05T08:07:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:41 smithi196 ceph-mon[33018]: 6.8 scrub ok 2024-02-05T08:07:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:41 smithi196 ceph-mon[33018]: 2.2 scrub starts 2024-02-05T08:07:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:41 smithi196 ceph-mon[33018]: 2.2 scrub ok 2024-02-05T08:07:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:41 smithi196 ceph-mon[33018]: 6.2 scrub starts 2024-02-05T08:07:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:41 smithi196 ceph-mon[33018]: 6.2 scrub ok 2024-02-05T08:07:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:41 smithi196 ceph-mon[33018]: 6.1 scrub starts 2024-02-05T08:07:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:41 smithi196 ceph-mon[33018]: 6.1 scrub ok 2024-02-05T08:07:42.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:07:42.149 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:42.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:42.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:42.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:42.279 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:42.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:42.305 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:42.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:42.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:42.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:42.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:42.434 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 52 16 74958 74942 90.0581 63.1875 0.0212629 0.0110985 2024-02-05T08:07:42.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:42.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:42.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:42.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:42.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:42.692 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:42.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:42.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:42.843 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:42.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:42.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:42.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:42.972 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:42.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:43.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:42 smithi134 ceph-mon[30086]: pgmap v107: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 41 MiB/s wr, 653 op/s 2024-02-05T08:07:43.027 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:42 smithi134 podman[88864]: 2024-02-05 08:07:42.875451142 +0000 UTC m=+2.125193916 volume create c6d40b6d621dec42873ee7e6c937c3e19a943f4770b141a709174cc33b64e672 2024-02-05T08:07:43.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:42 smithi134 ceph-mon[39278]: pgmap v107: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 41 MiB/s wr, 653 op/s 2024-02-05T08:07:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:42 smithi196 ceph-mon[33018]: pgmap v107: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 41 MiB/s wr, 653 op/s 2024-02-05T08:07:43.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:07:42 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:07:43.074 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:43.115 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:43.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:43.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:43.216 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:43.265 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:43.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:43.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:43.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:43.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:43.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:43.434 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 53 15 75981 75966 89.5664 64 0.0182537 0.0111595 2024-02-05T08:07:43.494 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:43.537 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:43.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:43.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:43.663 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:43.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:43.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:43.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:43.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:43.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:43.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:43.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:43.920 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:43.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:44.021 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:44.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:43 smithi134 ceph-mon[30086]: 4.17 deep-scrub starts 2024-02-05T08:07:44.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:43 smithi134 ceph-mon[30086]: 4.17 deep-scrub ok 2024-02-05T08:07:44.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:43 smithi134 ceph-mon[30086]: 3.9 scrub starts 2024-02-05T08:07:44.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:43 smithi134 ceph-mon[30086]: 3.9 scrub ok 2024-02-05T08:07:44.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:43 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:43 smithi134 ceph-mon[39278]: 4.17 deep-scrub starts 2024-02-05T08:07:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:43 smithi134 ceph-mon[39278]: 4.17 deep-scrub ok 2024-02-05T08:07:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:43 smithi134 ceph-mon[39278]: 3.9 scrub starts 2024-02-05T08:07:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:43 smithi134 ceph-mon[39278]: 3.9 scrub ok 2024-02-05T08:07:44.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:43 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:44.063 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:44.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:43 smithi196 ceph-mon[33018]: 4.17 deep-scrub starts 2024-02-05T08:07:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:43 smithi196 ceph-mon[33018]: 4.17 deep-scrub ok 2024-02-05T08:07:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:43 smithi196 ceph-mon[33018]: 3.9 scrub starts 2024-02-05T08:07:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:43 smithi196 ceph-mon[33018]: 3.9 scrub ok 2024-02-05T08:07:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:43 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:44.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:44.190 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:44.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:44.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:44.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:44.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:44.345 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:44.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:44.434 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 54 16 77027 77011 89.1171 65.3125 0.0148817 0.0112156 2024-02-05T08:07:44.446 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:44.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:44.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:44.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:44.527 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:44 smithi134 podman[88864]: 2024-02-05 08:07:44.327545949 +0000 UTC m=+3.577288724 container create 23561823c3c74d573473e7ac7c93e49aa2be690c18448be8ced26fde644935fc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T08:07:44.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:44.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:44.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:44.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:44.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:44.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:44.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:44.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:44.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:44.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:44.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:44.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:44.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:44.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:44.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:45.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:45.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:45.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:45.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:45.205 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:07:45.205 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:07:45.205 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[30086]: pgmap v108: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T08:07:45.205 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[30086]: osdmap e117: 8 total, 8 up, 7 in 2024-02-05T08:07:45.205 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[30086]: 6.4 scrub starts 2024-02-05T08:07:45.206 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[30086]: 6.4 scrub ok 2024-02-05T08:07:45.206 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[30086]: osdmap e118: 8 total, 8 up, 7 in 2024-02-05T08:07:45.206 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:07:45.206 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:07:45.206 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[39278]: pgmap v108: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T08:07:45.206 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[39278]: osdmap e117: 8 total, 8 up, 7 in 2024-02-05T08:07:45.206 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[39278]: 6.4 scrub starts 2024-02-05T08:07:45.206 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[39278]: 6.4 scrub ok 2024-02-05T08:07:45.206 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:44 smithi134 ceph-mon[39278]: osdmap e118: 8 total, 8 up, 7 in 2024-02-05T08:07:45.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:45.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:45.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:45.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:45.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.4 reset_purged_snaps_last 2024-02-05T08:07:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:44 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:07:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:44 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:07:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:44 smithi196 ceph-mon[33018]: pgmap v108: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T08:07:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:44 smithi196 ceph-mon[33018]: osdmap e117: 8 total, 8 up, 7 in 2024-02-05T08:07:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:44 smithi196 ceph-mon[33018]: 6.4 scrub starts 2024-02-05T08:07:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:44 smithi196 ceph-mon[33018]: 6.4 scrub ok 2024-02-05T08:07:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:44 smithi196 ceph-mon[33018]: osdmap e118: 8 total, 8 up, 7 in 2024-02-05T08:07:45.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:45.365 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:45.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:45.435 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 55 16 78006 77990 88.6089 61.1875 0.0184732 0.0112798 2024-02-05T08:07:45.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:45.522 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:45.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:45.623 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:45.675 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:45.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:45.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:45.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:45.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:45.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:45.904 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:45.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:45.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:45.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:45.977 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[30086]: 5.16 scrub starts 2024-02-05T08:07:45.977 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[30086]: 5.16 scrub ok 2024-02-05T08:07:45.977 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[30086]: 7.b scrub starts 2024-02-05T08:07:45.977 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[30086]: 6.1a scrub starts 2024-02-05T08:07:45.977 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[30086]: 6.1a scrub ok 2024-02-05T08:07:45.977 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[30086]: 6.10 scrub starts 2024-02-05T08:07:45.977 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[30086]: 6.10 scrub ok 2024-02-05T08:07:45.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[39278]: 5.16 scrub starts 2024-02-05T08:07:45.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[39278]: 5.16 scrub ok 2024-02-05T08:07:45.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[39278]: 7.b scrub starts 2024-02-05T08:07:45.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[39278]: 6.1a scrub starts 2024-02-05T08:07:45.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[39278]: 6.1a scrub ok 2024-02-05T08:07:45.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[39278]: 6.10 scrub starts 2024-02-05T08:07:45.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:45 smithi134 ceph-mon[39278]: 6.10 scrub ok 2024-02-05T08:07:46.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:46.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:46.084 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:46.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:46.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:46.221 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:46.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:45 smithi196 ceph-mon[33018]: 5.16 scrub starts 2024-02-05T08:07:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:45 smithi196 ceph-mon[33018]: 5.16 scrub ok 2024-02-05T08:07:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:45 smithi196 ceph-mon[33018]: 7.b scrub starts 2024-02-05T08:07:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:45 smithi196 ceph-mon[33018]: 6.1a scrub starts 2024-02-05T08:07:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:45 smithi196 ceph-mon[33018]: 6.1a scrub ok 2024-02-05T08:07:46.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:45 smithi196 ceph-mon[33018]: 6.10 scrub starts 2024-02-05T08:07:46.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:45 smithi196 ceph-mon[33018]: 6.10 scrub ok 2024-02-05T08:07:46.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:46.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:46.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:46.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:46.369 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:07:46.370 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:07:46.370 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:07:46.370 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:07:46.370 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:07:46.370 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:07:46.408 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:07:46.434 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 56 16 78781 78765 87.8915 48.4375 0.0389294 0.0113712 2024-02-05T08:07:46.449 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:07:46.449 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:46.456 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:07:46.456 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:07:46.456 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:07:46.456 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:07:46.456 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:07:46.456 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:07:46.456 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:07:46.456 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:07:46.488 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:46.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:46.590 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:46.632 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:46.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:46.734 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:46.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:46.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:46.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:46.887 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:46.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:46.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:47.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:47.014 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:47.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:47.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[30086]: pgmap v111: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 757 B/s rd, 46 MiB/s wr, 740 op/s 2024-02-05T08:07:47.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[30086]: 3.c scrub starts 2024-02-05T08:07:47.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[30086]: 3.c scrub ok 2024-02-05T08:07:47.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[30086]: 4.19 deep-scrub starts 2024-02-05T08:07:47.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[30086]: 4.19 deep-scrub ok 2024-02-05T08:07:47.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[30086]: 7.d scrub starts 2024-02-05T08:07:47.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2135312605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:07:47.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[39278]: pgmap v111: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 757 B/s rd, 46 MiB/s wr, 740 op/s 2024-02-05T08:07:47.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[39278]: 3.c scrub starts 2024-02-05T08:07:47.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[39278]: 3.c scrub ok 2024-02-05T08:07:47.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[39278]: 4.19 deep-scrub starts 2024-02-05T08:07:47.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[39278]: 4.19 deep-scrub ok 2024-02-05T08:07:47.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[39278]: 7.d scrub starts 2024-02-05T08:07:47.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:46 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2135312605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:07:47.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:47.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:47.153 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:47.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:47.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:47.295 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:47.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:47.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:46 smithi196 ceph-mon[33018]: pgmap v111: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 757 B/s rd, 46 MiB/s wr, 740 op/s 2024-02-05T08:07:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:46 smithi196 ceph-mon[33018]: 3.c scrub starts 2024-02-05T08:07:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:46 smithi196 ceph-mon[33018]: 3.c scrub ok 2024-02-05T08:07:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:46 smithi196 ceph-mon[33018]: 4.19 deep-scrub starts 2024-02-05T08:07:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:46 smithi196 ceph-mon[33018]: 4.19 deep-scrub ok 2024-02-05T08:07:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:46 smithi196 ceph-mon[33018]: 7.d scrub starts 2024-02-05T08:07:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:46 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2135312605' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:07:47.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:47.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:47.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:47.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:47.434 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 57 16 79519 79503 87.1588 46.125 0.00632911 0.0114672 2024-02-05T08:07:47.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:47.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:47.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:47.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:47.674 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:47.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:47.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:47.776 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:47.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:47.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:47.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:47.903 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:47.939 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:47.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:48.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:07:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:07:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[30086]: 3.e deep-scrub starts 2024-02-05T08:07:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[30086]: 3.e deep-scrub ok 2024-02-05T08:07:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[30086]: 5.f scrub starts 2024-02-05T08:07:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[30086]: 5.f scrub ok 2024-02-05T08:07:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[30086]: 7.b scrub ok 2024-02-05T08:07:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[30086]: 3.14 deep-scrub starts 2024-02-05T08:07:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[30086]: 3.14 deep-scrub ok 2024-02-05T08:07:48.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[39278]: 3.e deep-scrub starts 2024-02-05T08:07:48.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[39278]: 3.e deep-scrub ok 2024-02-05T08:07:48.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[39278]: 5.f scrub starts 2024-02-05T08:07:48.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[39278]: 5.f scrub ok 2024-02-05T08:07:48.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[39278]: 7.b scrub ok 2024-02-05T08:07:48.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[39278]: 3.14 deep-scrub starts 2024-02-05T08:07:48.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:47 smithi134 ceph-mon[39278]: 3.14 deep-scrub ok 2024-02-05T08:07:48.041 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:48.076 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:48.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:48.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:48.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:48.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:48.305 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:47 smithi196 ceph-mon[33018]: 3.e deep-scrub starts 2024-02-05T08:07:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:47 smithi196 ceph-mon[33018]: 3.e deep-scrub ok 2024-02-05T08:07:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:47 smithi196 ceph-mon[33018]: 5.f scrub starts 2024-02-05T08:07:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:47 smithi196 ceph-mon[33018]: 5.f scrub ok 2024-02-05T08:07:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:47 smithi196 ceph-mon[33018]: 7.b scrub ok 2024-02-05T08:07:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:47 smithi196 ceph-mon[33018]: 3.14 deep-scrub starts 2024-02-05T08:07:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:47 smithi196 ceph-mon[33018]: 3.14 deep-scrub ok 2024-02-05T08:07:48.351 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:48.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:48.434 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 58 16 80357 80341 86.559 52.375 0.0155755 0.0115475 2024-02-05T08:07:48.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:48.480 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:48.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:48.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:48.527 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:48 smithi134 podman[88864]: 2024-02-05 08:07:48.261815674 +0000 UTC m=+7.511558446 container init 23561823c3c74d573473e7ac7c93e49aa2be690c18448be8ced26fde644935fc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T08:07:48.527 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:48 smithi134 podman[88864]: 2024-02-05 08:07:48.278805764 +0000 UTC m=+7.528548540 container start 23561823c3c74d573473e7ac7c93e49aa2be690c18448be8ced26fde644935fc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T08:07:48.528 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[89215]: ts=2024-02-05T08:07:48.308Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-05T08:07:48.528 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[89215]: ts=2024-02-05T08:07:48.308Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-05T08:07:48.528 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[89215]: ts=2024-02-05T08:07:48.309Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.134 port=9094 2024-02-05T08:07:48.528 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[89215]: ts=2024-02-05T08:07:48.313Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-05T08:07:48.528 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[89215]: ts=2024-02-05T08:07:48.364Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-05T08:07:48.528 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[89215]: ts=2024-02-05T08:07:48.364Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-05T08:07:48.528 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[89215]: ts=2024-02-05T08:07:48.368Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-05T08:07:48.529 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[89215]: ts=2024-02-05T08:07:48.368Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-05T08:07:48.529 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:48 smithi134 bash[88864]: 23561823c3c74d573473e7ac7c93e49aa2be690c18448be8ced26fde644935fc 2024-02-05T08:07:48.529 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:48 smithi134 systemd[1]: Started Ceph alertmanager.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:07:48.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:48.621 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:48.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:48.723 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:48.762 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:48.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:48.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:48.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:48.893 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:48.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:48.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:49.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:49.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:49.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:49.144 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:49.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:49.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[30086]: pgmap v112: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 742 op/s 2024-02-05T08:07:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[30086]: 6.19 scrub starts 2024-02-05T08:07:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[30086]: 6.19 scrub ok 2024-02-05T08:07:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[30086]: 5.0 scrub starts 2024-02-05T08:07:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[30086]: 5.0 scrub ok 2024-02-05T08:07:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:07:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:07:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[30086]: 4.2 scrub starts 2024-02-05T08:07:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[30086]: 4.2 scrub ok 2024-02-05T08:07:49.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:49.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[39278]: pgmap v112: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 742 op/s 2024-02-05T08:07:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[39278]: 6.19 scrub starts 2024-02-05T08:07:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[39278]: 6.19 scrub ok 2024-02-05T08:07:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[39278]: 5.0 scrub starts 2024-02-05T08:07:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[39278]: 5.0 scrub ok 2024-02-05T08:07:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:07:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:07:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[39278]: 4.2 scrub starts 2024-02-05T08:07:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[39278]: 4.2 scrub ok 2024-02-05T08:07:49.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:49.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:48 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:49.288 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:49.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:48 smithi196 ceph-mon[33018]: pgmap v112: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 742 op/s 2024-02-05T08:07:49.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:48 smithi196 ceph-mon[33018]: 6.19 scrub starts 2024-02-05T08:07:49.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:48 smithi196 ceph-mon[33018]: 6.19 scrub ok 2024-02-05T08:07:49.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:48 smithi196 ceph-mon[33018]: 5.0 scrub starts 2024-02-05T08:07:49.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:48 smithi196 ceph-mon[33018]: 5.0 scrub ok 2024-02-05T08:07:49.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:48 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:07:49.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:48 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:07:49.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:48 smithi196 ceph-mon[33018]: 4.2 scrub starts 2024-02-05T08:07:49.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:48 smithi196 ceph-mon[33018]: 4.2 scrub ok 2024-02-05T08:07:49.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:48 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:49.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:48 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:49.318 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:49.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:49.388 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:49.410 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:49.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:49.434 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 59 16 81333 81317 86.1258 61 0.0148141 0.011605 2024-02-05T08:07:49.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:49.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:49.546 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:49.615 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:49.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:49.716 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:49.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:49.764 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:49.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:49.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:49.890 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:49.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:49.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:49.992 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:50.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[39278]: 7.d scrub ok 2024-02-05T08:07:50.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[39278]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-05T08:07:50.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[39278]: 5.3 scrub starts 2024-02-05T08:07:50.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[39278]: 5.3 scrub ok 2024-02-05T08:07:50.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[39278]: Reconfiguring daemon prometheus.a on smithi196 2024-02-05T08:07:50.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[39278]: osdmap e119: 8 total, 8 up, 7 in 2024-02-05T08:07:50.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[39278]: 5.1d deep-scrub starts 2024-02-05T08:07:50.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[39278]: 5.1d deep-scrub ok 2024-02-05T08:07:50.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[39278]: osdmap e120: 8 total, 8 up, 7 in 2024-02-05T08:07:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[30086]: 7.d scrub ok 2024-02-05T08:07:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[30086]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-05T08:07:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[30086]: 5.3 scrub starts 2024-02-05T08:07:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[30086]: 5.3 scrub ok 2024-02-05T08:07:50.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[30086]: Reconfiguring daemon prometheus.a on smithi196 2024-02-05T08:07:50.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[30086]: osdmap e119: 8 total, 8 up, 7 in 2024-02-05T08:07:50.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[30086]: 5.1d deep-scrub starts 2024-02-05T08:07:50.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[30086]: 5.1d deep-scrub ok 2024-02-05T08:07:50.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:49 smithi134 ceph-mon[30086]: osdmap e120: 8 total, 8 up, 7 in 2024-02-05T08:07:50.031 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:50.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:50.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:49 smithi196 ceph-mon[33018]: 7.d scrub ok 2024-02-05T08:07:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:49 smithi196 ceph-mon[33018]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-05T08:07:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:49 smithi196 ceph-mon[33018]: 5.3 scrub starts 2024-02-05T08:07:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:49 smithi196 ceph-mon[33018]: 5.3 scrub ok 2024-02-05T08:07:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:49 smithi196 ceph-mon[33018]: Reconfiguring daemon prometheus.a on smithi196 2024-02-05T08:07:50.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:49 smithi196 ceph-mon[33018]: osdmap e119: 8 total, 8 up, 7 in 2024-02-05T08:07:50.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:49 smithi196 ceph-mon[33018]: 5.1d deep-scrub starts 2024-02-05T08:07:50.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:49 smithi196 ceph-mon[33018]: 5.1d deep-scrub ok 2024-02-05T08:07:50.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:49 smithi196 ceph-mon[33018]: osdmap e120: 8 total, 8 up, 7 in 2024-02-05T08:07:50.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:50.162 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:50.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:50.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:50.303 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:50.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:50.405 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:50.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:50.435 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:07:50.435485+0000 min lat: 0.00118368 max lat: 0.159692 avg lat: 0.0116658 2024-02-05T08:07:50.435 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:07:50.435 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 60 16 82282 82266 85.6789 59.3125 0.00871114 0.0116658 2024-02-05T08:07:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:50.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:50.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:50.577 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:50.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:50.679 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:50.724 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:50.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:50.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:50.777 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:50 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[89215]: ts=2024-02-05T08:07:50.313Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000354955s 2024-02-05T08:07:50.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:50.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:50.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:50.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:50.941 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:07:50.957 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:51.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:50 smithi196 ceph-mon[33018]: pgmap v114: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 47 MiB/s wr, 751 op/s 2024-02-05T08:07:51.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:50 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:07:51.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:50 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:07:51.017 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:50 smithi196 ceph-mon[33018]: osdmap e121: 8 total, 8 up, 7 in 2024-02-05T08:07:51.033 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:51.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:51.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:50 smithi134 ceph-mon[30086]: pgmap v114: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 47 MiB/s wr, 751 op/s 2024-02-05T08:07:51.091 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:50 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:07:51.091 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:50 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:07:51.091 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:50 smithi134 ceph-mon[30086]: osdmap e121: 8 total, 8 up, 7 in 2024-02-05T08:07:51.091 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:50 smithi134 ceph-mon[39278]: pgmap v114: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 47 MiB/s wr, 751 op/s 2024-02-05T08:07:51.091 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:50 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:07:51.091 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:50 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:07:51.091 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:50 smithi134 ceph-mon[39278]: osdmap e121: 8 total, 8 up, 7 in 2024-02-05T08:07:51.135 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:51.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:51.254 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:51.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:51.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:51.383 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:51.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:51.434 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 61 16 83270 83254 85.2866 61.75 0.0134761 0.0117193 2024-02-05T08:07:51.484 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:51.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:51.539 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:51.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:51.640 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:51.682 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:51.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:51.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:51.825 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:51.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:51.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:51.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:51.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:51.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:52.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:51 smithi196 ceph-mon[33018]: 4.b scrub starts 2024-02-05T08:07:52.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:51 smithi196 ceph-mon[33018]: 4.b scrub ok 2024-02-05T08:07:52.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:51 smithi196 ceph-mon[33018]: 3.18 scrub starts 2024-02-05T08:07:52.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:51 smithi196 ceph-mon[33018]: 3.18 scrub ok 2024-02-05T08:07:52.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:51 smithi196 ceph-mon[33018]: 5.11 scrub starts 2024-02-05T08:07:52.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:51 smithi196 ceph-mon[33018]: 5.11 scrub ok 2024-02-05T08:07:52.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:51 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:52.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:51 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:07:52.055 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:52.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:52.095 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:52.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:52.196 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:52.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:52.239 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:52.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[30086]: 4.b scrub starts 2024-02-05T08:07:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[30086]: 4.b scrub ok 2024-02-05T08:07:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[30086]: 3.18 scrub starts 2024-02-05T08:07:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[30086]: 3.18 scrub ok 2024-02-05T08:07:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[30086]: 5.11 scrub starts 2024-02-05T08:07:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[30086]: 5.11 scrub ok 2024-02-05T08:07:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:07:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[39278]: 4.b scrub starts 2024-02-05T08:07:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[39278]: 4.b scrub ok 2024-02-05T08:07:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[39278]: 3.18 scrub starts 2024-02-05T08:07:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[39278]: 3.18 scrub ok 2024-02-05T08:07:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[39278]: 5.11 scrub starts 2024-02-05T08:07:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[39278]: 5.11 scrub ok 2024-02-05T08:07:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:51 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:07:52.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:52.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:52.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:52.435 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 62 16 84206 84190 84.8545 58.5 0.0194593 0.0117794 2024-02-05T08:07:52.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:52.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:52.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:52.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:52.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:52.639 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:52.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:52.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:52.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:52.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:52.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:52.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:52.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:52.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:52.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:52.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:52.998 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:53.039 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:53.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:53.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:53.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:07:52 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:07:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:52 smithi196 ceph-mon[33018]: pgmap v117: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 47 MiB/s wr, 751 op/s 2024-02-05T08:07:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:52 smithi196 ceph-mon[33018]: 5.10 scrub starts 2024-02-05T08:07:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:52 smithi196 ceph-mon[33018]: 5.10 scrub ok 2024-02-05T08:07:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:52 smithi196 ceph-mon[33018]: 3.5 scrub starts 2024-02-05T08:07:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:52 smithi196 ceph-mon[33018]: 3.5 scrub ok 2024-02-05T08:07:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:52 smithi196 ceph-mon[33018]: 4.5 scrub starts 2024-02-05T08:07:53.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:52 smithi196 ceph-mon[33018]: 4.5 scrub ok 2024-02-05T08:07:53.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:53.169 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:53.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:53.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:53.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[30086]: pgmap v117: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 47 MiB/s wr, 751 op/s 2024-02-05T08:07:53.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[30086]: 5.10 scrub starts 2024-02-05T08:07:53.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[30086]: 5.10 scrub ok 2024-02-05T08:07:53.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[30086]: 3.5 scrub starts 2024-02-05T08:07:53.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[30086]: 3.5 scrub ok 2024-02-05T08:07:53.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[30086]: 4.5 scrub starts 2024-02-05T08:07:53.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[30086]: 4.5 scrub ok 2024-02-05T08:07:53.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[39278]: pgmap v117: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 47 MiB/s wr, 751 op/s 2024-02-05T08:07:53.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[39278]: 5.10 scrub starts 2024-02-05T08:07:53.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[39278]: 5.10 scrub ok 2024-02-05T08:07:53.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[39278]: 3.5 scrub starts 2024-02-05T08:07:53.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[39278]: 3.5 scrub ok 2024-02-05T08:07:53.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[39278]: 4.5 scrub starts 2024-02-05T08:07:53.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:52 smithi134 ceph-mon[39278]: 4.5 scrub ok 2024-02-05T08:07:53.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:53.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:53.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:53.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:53.435 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 63 16 85265 85249 84.5581 66.1875 0.010182 0.0118207 2024-02-05T08:07:53.455 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:53.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:53.559 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:53.597 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:53.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:53.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:53.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:53.741 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:53.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:53.814 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 systemd[1]: Stopping Ceph prometheus.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:07:53.842 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:53.882 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:53.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:53.984 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:54.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:54.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:54.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:54.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:54.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:54.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:54.225 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-mon[33018]: 5.15 scrub starts 2024-02-05T08:07:54.225 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-mon[33018]: 5.15 scrub ok 2024-02-05T08:07:54.225 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:54.226 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:07:53.838Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-05T08:07:54.226 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:07:53.838Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-05T08:07:54.226 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:07:53.838Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-05T08:07:54.226 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:07:53.838Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-05T08:07:54.226 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:07:53.838Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-05T08:07:54.226 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:07:53.838Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-05T08:07:54.226 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:07:53.838Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-05T08:07:54.226 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:07:53.838Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-05T08:07:54.226 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:07:53.838Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-05T08:07:54.226 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:07:53.838Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-05T08:07:54.226 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:07:53.838Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-05T08:07:54.226 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[64611]: ts=2024-02-05T08:07:53.838Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-05T08:07:54.226 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:53 smithi196 podman[73356]: 2024-02-05 08:07:53.848787886 +0000 UTC m=+0.142184101 container died b621d9607e33a0ac0960a40881f88137acf9b2abd8619dbe349fc1b68b5d1221 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T08:07:54.240 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:07:54.240 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:07:54.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:53 smithi134 ceph-mon[30086]: 5.15 scrub starts 2024-02-05T08:07:54.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:53 smithi134 ceph-mon[30086]: 5.15 scrub ok 2024-02-05T08:07:54.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:53 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:54.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:53 smithi134 ceph-mon[39278]: 5.15 scrub starts 2024-02-05T08:07:54.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:53 smithi134 ceph-mon[39278]: 5.15 scrub ok 2024-02-05T08:07:54.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:53 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:07:54.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:54.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:54.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:54.367 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:54.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:54.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:54.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:54.435 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 64 15 86242 86227 84.1919 61.125 0.0242436 0.0118723 2024-02-05T08:07:54.481 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:54 smithi196 podman[73356]: 2024-02-05 08:07:54.469144969 +0000 UTC m=+0.762541217 container cleanup b621d9607e33a0ac0960a40881f88137acf9b2abd8619dbe349fc1b68b5d1221 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T08:07:54.481 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:54 smithi196 bash[73356]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a 2024-02-05T08:07:54.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:54.536 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:54.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:54.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:54.675 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:54.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:54.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:54.777 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:54.819 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:54.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:54.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:54.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:54.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:54.962 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:54 smithi196 podman[73375]: 2024-02-05 08:07:54.704304891 +0000 UTC m=+0.858143789 container remove b621d9607e33a0ac0960a40881f88137acf9b2abd8619dbe349fc1b68b5d1221 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T08:07:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:54 smithi196 ceph-mon[33018]: pgmap v118: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T08:07:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:54 smithi196 ceph-mon[33018]: 3.a deep-scrub starts 2024-02-05T08:07:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:54 smithi196 ceph-mon[33018]: 3.a deep-scrub ok 2024-02-05T08:07:54.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:54 smithi196 ceph-mon[33018]: osdmap e122: 8 total, 8 up, 7 in 2024-02-05T08:07:55.050 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:55.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:55.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:55.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:55.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:55.219 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:55.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:55.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:55.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:54 smithi134 ceph-mon[30086]: pgmap v118: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T08:07:55.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:54 smithi134 ceph-mon[30086]: 3.a deep-scrub starts 2024-02-05T08:07:55.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:54 smithi134 ceph-mon[30086]: 3.a deep-scrub ok 2024-02-05T08:07:55.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:54 smithi134 ceph-mon[30086]: osdmap e122: 8 total, 8 up, 7 in 2024-02-05T08:07:55.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:54 smithi134 ceph-mon[39278]: pgmap v118: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T08:07:55.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:54 smithi134 ceph-mon[39278]: 3.a deep-scrub starts 2024-02-05T08:07:55.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:54 smithi134 ceph-mon[39278]: 3.a deep-scrub ok 2024-02-05T08:07:55.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:54 smithi134 ceph-mon[39278]: osdmap e122: 8 total, 8 up, 7 in 2024-02-05T08:07:55.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:54 smithi196 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-05T08:07:55.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:54 smithi196 systemd[1]: Stopped Ceph prometheus.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:07:55.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:54 smithi196 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@prometheus.a.service: Consumed 1.324s CPU time. 2024-02-05T08:07:55.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:54 smithi196 systemd[1]: Starting Ceph prometheus.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:07:55.314 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:55 smithi196 podman[73555]: 2024-02-05 08:07:55.111748143 +0000 UTC m=+0.022130570 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-02-05T08:07:55.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:55.369 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:55.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:55.435 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 65 16 87177 87161 83.7947 58.375 0.0204677 0.0119287 2024-02-05T08:07:55.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:55.519 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:55.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:55.621 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:55.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:55.660 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:55.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:55.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:55.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:55.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:55.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:55.906 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:55 smithi196 podman[73555]: 2024-02-05 08:07:55.630649977 +0000 UTC m=+0.541032404 container create 07dc4e8cf9965ae5d21a0121af7cd6cd0379b9142a06f34106fb51913f97a6a1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T08:07:55.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:55.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:55.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:55.934 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:07:56.016 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:56.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:55 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/491520577' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:07:56.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:55 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:07:56.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:55 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/491520577' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:07:56.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:55 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:07:56.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:56.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:56.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:56.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:56.186 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:56.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:56.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:56.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:55 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/491520577' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:07:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:55 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:07:56.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:56.328 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:56.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:56.431 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:56.435 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 66 16 88129 88113 83.4265 59.5 0.0120591 0.0119814 2024-02-05T08:07:56.473 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:56.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:56.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:56.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:56.602 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:56.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:56.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:56.751 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:56.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:56.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:56.815 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 podman[73555]: 2024-02-05 08:07:56.492559415 +0000 UTC m=+1.402941859 container init 07dc4e8cf9965ae5d21a0121af7cd6cd0379b9142a06f34106fb51913f97a6a1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T08:07:56.815 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 podman[73555]: 2024-02-05 08:07:56.498897107 +0000 UTC m=+1.409279540 container start 07dc4e8cf9965ae5d21a0121af7cd6cd0379b9142a06f34106fb51913f97a6a1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T08:07:56.815 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.529Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-02-05T08:07:56.815 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.529Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-02-05T08:07:56.815 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.529Z caller=main.go:570 level=info host_details="(Linux 5.14.0-416.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jan 31 22:52:48 UTC 2024 x86_64 smithi196 (none))" 2024-02-05T08:07:56.815 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.529Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-05T08:07:56.815 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.529Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-05T08:07:56.815 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.530Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-02-05T08:07:56.815 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.531Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-02-05T08:07:56.815 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.532Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-02-05T08:07:56.816 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.533Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-02-05T08:07:56.816 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.534Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-05T08:07:56.816 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.534Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.309µs 2024-02-05T08:07:56.816 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.534Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-05T08:07:56.816 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.535Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-02-05T08:07:56.816 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.536Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-02-05T08:07:56.816 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.536Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=29.45µs wal_replay_duration=1.193112ms wbl_replay_duration=140ns total_replay_duration=1.25117ms 2024-02-05T08:07:56.816 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.536Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-02-05T08:07:56.816 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.536Z caller=main.go:1029 level=info msg="TSDB started" 2024-02-05T08:07:56.816 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.536Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-05T08:07:56.816 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.558Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.10252ms db_storage=806ns remote_storage=1.645µs web_handler=377ns query_engine=710ns scrape=420.472µs scrape_sd=84.693µs notify=20.565µs notify_sd=17.681µs rules=20.280087ms tracing=6.687µs 2024-02-05T08:07:56.816 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.558Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-02-05T08:07:56.816 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:07:56.558Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-02-05T08:07:56.817 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 bash[73555]: 07dc4e8cf9965ae5d21a0121af7cd6cd0379b9142a06f34106fb51913f97a6a1 2024-02-05T08:07:56.817 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:07:56 smithi196 systemd[1]: Started Ceph prometheus.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:07:56.853 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:07:56.908 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:56.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:56.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:07:56.969 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: 3.f scrub starts 2024-02-05T08:07:56.969 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: 3.f scrub ok 2024-02-05T08:07:56.969 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: pgmap v120: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 50 MiB/s wr, 795 op/s 2024-02-05T08:07:56.969 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: 6.9 scrub starts 2024-02-05T08:07:56.969 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: 6.9 scrub ok 2024-02-05T08:07:56.969 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: 5.1c scrub starts 2024-02-05T08:07:56.969 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: 5.1c scrub ok 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: osdmap e123: 8 total, 8 up, 7 in 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi134.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:56.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:07:56.971 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE Bus STOPPING 2024-02-05T08:07:56.971 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T08:07:56.971 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE Bus STOPPED 2024-02-05T08:07:56.971 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE Bus STARTING 2024-02-05T08:07:56.971 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE Serving on http://:::9283 2024-02-05T08:07:56.971 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE Bus STARTED 2024-02-05T08:07:56.971 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE Bus STOPPING 2024-02-05T08:07:56.971 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T08:07:56.971 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE Bus STOPPED 2024-02-05T08:07:56.971 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE Bus STARTING 2024-02-05T08:07:56.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: 3.f scrub starts 2024-02-05T08:07:56.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: 3.f scrub ok 2024-02-05T08:07:56.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: pgmap v120: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 50 MiB/s wr, 795 op/s 2024-02-05T08:07:56.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: 6.9 scrub starts 2024-02-05T08:07:56.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: 6.9 scrub ok 2024-02-05T08:07:56.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: 5.1c scrub starts 2024-02-05T08:07:56.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: 5.1c scrub ok 2024-02-05T08:07:56.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:07:56.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:07:56.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: osdmap e123: 8 total, 8 up, 7 in 2024-02-05T08:07:56.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:56.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:56.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T08:07:56.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi134.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T08:07:56.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:56.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T08:07:56.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T08:07:56.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:56.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T08:07:56.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T08:07:56.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:56.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:07:57.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:07:57.038 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:57.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:57.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:07:57.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:57.167 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:57.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:57.269 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:57.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE Serving on http://:::9283 2024-02-05T08:07:57.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE Bus STARTED 2024-02-05T08:07:57.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE Bus STOPPING 2024-02-05T08:07:57.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T08:07:57.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE Bus STOPPED 2024-02-05T08:07:57.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:56 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:56] ENGINE Bus STARTING 2024-02-05T08:07:57.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:57] ENGINE Serving on http://:::9283 2024-02-05T08:07:57.277 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:07:57] ENGINE Bus STARTED 2024-02-05T08:07:57.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:57.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:57.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: 3.f scrub starts 2024-02-05T08:07:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: 3.f scrub ok 2024-02-05T08:07:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: pgmap v120: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 50 MiB/s wr, 795 op/s 2024-02-05T08:07:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: 6.9 scrub starts 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: 6.9 scrub ok 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: 5.1c scrub starts 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: 5.1c scrub ok 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: osdmap e123: 8 total, 8 up, 7 in 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi134.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T08:07:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:57.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T08:07:57.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T08:07:57.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:07:57.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:07:57.374 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.5 reset_purged_snaps_last 2024-02-05T08:07:57.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:57.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:57.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:57.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:57.436 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 67 16 89037 89021 83.0273 56.75 0.0179062 0.0120384 2024-02-05T08:07:57.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:57.605 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:57.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:57.707 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:57.729 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:57.805 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:57.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:57.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T08:07:57.907 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:57.964 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:57.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:57.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-mon[30086]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T08:07:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-mon[30086]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi134.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T08:07:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-mon[30086]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T08:07:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-mon[30086]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T08:07:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-mon[30086]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T08:07:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-mon[30086]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T08:07:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-mon[39278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T08:07:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-mon[39278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi134.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T08:07:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-mon[39278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T08:07:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-mon[39278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T08:07:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-mon[39278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T08:07:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:57 smithi134 ceph-mon[39278]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T08:07:58.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:58.095 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:58.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:58.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:58.253 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:58.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:57 smithi196 ceph-mon[33018]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T08:07:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:57 smithi196 ceph-mon[33018]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi134.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T08:07:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:57 smithi196 ceph-mon[33018]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T08:07:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:57 smithi196 ceph-mon[33018]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi196.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T08:07:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:57 smithi196 ceph-mon[33018]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T08:07:58.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:57 smithi196 ceph-mon[33018]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi196.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T08:07:58.354 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:58.399 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:58.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:58.436 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 68 16 89881 89865 82.582 52.75 0.0139572 0.012104 2024-02-05T08:07:58.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:07:58.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:58.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:07:58.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:58.652 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:07:58 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[89215]: ts=2024-02-05T08:07:58.317Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.004174885s 2024-02-05T08:07:58.668 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:58.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:58.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:58.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:07:58.771 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:58.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:58.822 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:58.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:58.923 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:07:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:58.998 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:58.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:07:59.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:07:59.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:59.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:07:59.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:59.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:59.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:07:59.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:07:58 smithi134 ceph-mon[30086]: pgmap v122: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 51 MiB/s wr, 813 op/s 2024-02-05T08:07:59.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:07:58 smithi134 ceph-mon[39278]: pgmap v122: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 51 MiB/s wr, 813 op/s 2024-02-05T08:07:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:07:58 smithi196 ceph-mon[33018]: pgmap v122: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 51 MiB/s wr, 813 op/s 2024-02-05T08:07:59.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:07:59.368 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:07:59.440 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 69 16 90817 90801 82.2285 58.5 0.0243426 0.0121547 2024-02-05T08:07:59.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:59.468 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:59.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:07:59.569 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:59.633 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:59.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:59.737 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:07:59.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:07:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:59.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:07:59.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:07:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:07:59.920 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:07:59.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:00.021 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:00.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:00.067 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:00.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:00.168 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:00.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:00.216 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:00.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:00.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:00.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:00.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:00.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:00.440 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 70 16 91752 91736 81.8886 58.4375 0.00774482 0.0122063 2024-02-05T08:08:00.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:00.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:00.480 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:00.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:00.509 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:00.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:00.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:00.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:00.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:00.636 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:08:00.636 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:08:00.636 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:08:00.636 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:08:00.636 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:08:00.637 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:08:00.681 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:08:00.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:00.722 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:08:00.732 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:08:00.732 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:08:00.732 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:08:00.732 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:08:00.732 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:08:00.732 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:08:00.732 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:08:00.733 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:08:00.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:00.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:00.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:00 smithi196 ceph-mon[33018]: pgmap v123: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.19k op/s 2024-02-05T08:08:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:00 smithi196 ceph-mon[33018]: osdmap e124: 8 total, 8 up, 7 in 2024-02-05T08:08:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:00 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3137620053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:08:00.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:00.883 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:00.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:00.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:00.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:01.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:01.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:00 smithi134 ceph-mon[30086]: pgmap v123: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.19k op/s 2024-02-05T08:08:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:00 smithi134 ceph-mon[30086]: osdmap e124: 8 total, 8 up, 7 in 2024-02-05T08:08:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:00 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3137620053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:08:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:00 smithi134 ceph-mon[39278]: pgmap v123: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.19k op/s 2024-02-05T08:08:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:00 smithi134 ceph-mon[39278]: osdmap e124: 8 total, 8 up, 7 in 2024-02-05T08:08:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:00 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3137620053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:08:01.113 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:01.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:01.166 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:01.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:01.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:01.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:01.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:01.395 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:01.440 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 71 16 92741 92725 81.6056 61.8125 0.0114572 0.012248 2024-02-05T08:08:01.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:01.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:01.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:01.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:01.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:01.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:01.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:01.699 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:01.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:01.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:01.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:01.857 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:01.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:01.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:01.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:01.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:02.099 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:02.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:02.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:02.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:02.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:02.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:02.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:02.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:02.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:02.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:02.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:02.440 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 72 16 93710 93694 81.3133 60.5625 0.00785936 0.0122931 2024-02-05T08:08:02.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:02.540 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:02.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:02.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:02.642 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:02.684 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:02.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:02.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:02.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:02.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:02.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:02.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:02.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:02.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:02.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:03.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:03.090 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:03.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:03.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:03.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:03.234 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:03.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:03.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:08:02 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:08:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:03 smithi196 ceph-mon[33018]: pgmap v125: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 734 B/s rd, 43 MiB/s wr, 686 op/s 2024-02-05T08:08:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:03 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:03 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:03.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:03.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:03.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:03.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:03.440 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 73 16 94709 94693 81.0543 62.4375 0.0102109 0.0123325 2024-02-05T08:08:03.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:03.509 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:03.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:03 smithi134 ceph-mon[30086]: pgmap v125: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 734 B/s rd, 43 MiB/s wr, 686 op/s 2024-02-05T08:08:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:03 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:03 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:03.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:03 smithi134 ceph-mon[39278]: pgmap v125: 148 pgs: 148 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 734 B/s rd, 43 MiB/s wr, 686 op/s 2024-02-05T08:08:03.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:03 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:03.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:03 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:03.611 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:03.676 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:03.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:03.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:03.778 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:03.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:03.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:03.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:03.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:03.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:03.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:03.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:04.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:04.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:04.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:04.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:04.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:04.228 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:04.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:04 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:04 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:04.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:04.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:04.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:04.441 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 74 16 95667 95651 80.7681 59.875 0.0208409 0.0123758 2024-02-05T08:08:04.456 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:04.482 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:04.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:04.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:04.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:04 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:04.584 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:04.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:04.619 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:04.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:04.720 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:04.761 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:04.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:04.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:04.863 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:04.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:04.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:04.990 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:05.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:05.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:05.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:05.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:05.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:05.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:05.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:05.245 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:05.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:05.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:05.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:05.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:05.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:05.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:05.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:05.441 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 75 16 96585 96569 80.4562 57.375 0.0126895 0.0124239 2024-02-05T08:08:05.498 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:05.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:05 smithi134 ceph-mon[30086]: pgmap v126: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-05T08:08:05.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:05 smithi134 ceph-mon[39278]: pgmap v126: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-05T08:08:05.541 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:05.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:05.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:05 smithi196 ceph-mon[33018]: pgmap v126: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-05T08:08:05.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:05.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:05.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:05.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:05.715 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:05.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:05.833 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:05.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:05.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:05.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:05.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:06.064 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:06.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:08:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:06.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:06.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:06.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:06.237 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:06.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:06.260 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:08:06.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:06.365 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:06.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:06.441 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 76 16 97426 97410 80.0891 52.5625 0.00379946 0.0124778 2024-02-05T08:08:06.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:06.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:06.492 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:06.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:06.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:06 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3001089141' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:08:06.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:06 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:08:06.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:06 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3001089141' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:08:06.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:06 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:08:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:06 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3001089141' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:08:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:06 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:08:06.593 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:06.637 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:06.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:06.738 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:06.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:06.764 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:06.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:06.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:06.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:06.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:06.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:06.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:07.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:07.034 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:07.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:07.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:07.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:07.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:07.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:07.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:07.309 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:07.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:07.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:07.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:07.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:07.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:07.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:07.442 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 77 16 98382 98366 79.825 59.75 0.0112243 0.012522 2024-02-05T08:08:07.455 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:07 smithi134 ceph-mon[30086]: pgmap v127: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 62 MiB/s wr, 989 op/s 2024-02-05T08:08:07.455 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:07 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:08:07.455 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:07 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:08:07.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:07 smithi134 ceph-mon[30086]: osdmap e125: 8 total, 8 up, 7 in 2024-02-05T08:08:07.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:07.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:08:07.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:07 smithi134 ceph-mon[39278]: pgmap v127: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 62 MiB/s wr, 989 op/s 2024-02-05T08:08:07.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:07 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:08:07.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:07 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:08:07.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:07 smithi134 ceph-mon[39278]: osdmap e125: 8 total, 8 up, 7 in 2024-02-05T08:08:07.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:07.456 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:08:07.505 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.7 reset_purged_snaps_last 2024-02-05T08:08:07.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:07 smithi196 ceph-mon[33018]: pgmap v127: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 62 MiB/s wr, 989 op/s 2024-02-05T08:08:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:07 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:08:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:07 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:08:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:07 smithi196 ceph-mon[33018]: osdmap e125: 8 total, 8 up, 7 in 2024-02-05T08:08:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:08:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:07.565 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:07.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:07.666 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:07.692 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:07.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:07.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:07.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:07.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:07.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:07.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:07.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:07.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:07.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:08.030 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:08:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:08:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:08:08.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:08.100 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:08.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:08.201 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:08.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:08.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:08.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:08.357 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:08.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:08.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:08 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:08:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:08 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:08:08.421 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:08 smithi134 ceph-mon[30086]: osdmap e126: 8 total, 8 up, 7 in 2024-02-05T08:08:08.421 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:08 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:08:08.421 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:08 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:08:08.422 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:08 smithi134 ceph-mon[39278]: osdmap e126: 8 total, 8 up, 7 in 2024-02-05T08:08:08.442 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 78 16 99347 99331 79.5739 60.3125 0.0167304 0.0125615 2024-02-05T08:08:08.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:08.484 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:08.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:08 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:08:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:08 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:08:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:08 smithi196 ceph-mon[33018]: osdmap e126: 8 total, 8 up, 7 in 2024-02-05T08:08:08.586 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:08.635 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:08.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:08.736 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:08.764 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:08.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:08.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:08.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:08.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:08.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:09.018 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:09.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:09.118 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:09.161 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:09.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:09.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:09.303 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:09.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:09.405 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:09.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:09.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:09.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 79 16 100174 100158 79.2194 51.6875 0.0194808 0.0126179 2024-02-05T08:08:09.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:09.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:09 smithi196 ceph-mon[33018]: pgmap v130: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 44 MiB/s wr, 699 op/s 2024-02-05T08:08:09.587 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:09.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:09.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:09.617 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:09 smithi134 ceph-mon[30086]: pgmap v130: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 44 MiB/s wr, 699 op/s 2024-02-05T08:08:09.617 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:09 smithi134 ceph-mon[39278]: pgmap v130: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 44 MiB/s wr, 699 op/s 2024-02-05T08:08:09.634 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:09.635 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:09.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:09.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:09.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:09.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:09.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:09.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:09.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:09.946 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:10.001 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:10.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:10.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:10.153 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:10.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:10.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:10.295 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:10.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:10.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:10.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:08:10.444731+0000 min lat: 0.00118368 max lat: 0.159692 avg lat: 0.0126579 2024-02-05T08:08:10.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:08:10.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 80 16 101123 101107 78.9705 59.3125 0.0221172 0.0126579 2024-02-05T08:08:10.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:10.453 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:10.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:10.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:10.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:10.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:10.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:10.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:10.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:10.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:10.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:10.775 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:08:10.775 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:08:10.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.776 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:101423 7.e 7:7f6b02d4:::benchmark_data_smithi134_83016_object101422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:10.776 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:10.761001+0000", 2024-02-05T08:08:10.776 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.012970951999999999, 2024-02-05T08:08:10.776 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.013011157000000001, 2024-02-05T08:08:10.776 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:10.776 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:08:10.776 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:10.776 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:10.776 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:10.776 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 101423 2024-02-05T08:08:10.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.776 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.761001+0000", 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.761001+0000", 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.761002+0000", 2024-02-05T08:08:10.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2020000000000001e-06 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.761024+0000", 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1432000000000001e-05 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.761037+0000", 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3871e-05 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:10.778 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.761045+0000", 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5650000000000004e-06 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.764061+0000", 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030164369999999999 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.764198+0000", 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013626799999999999 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.779 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:10.780 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.764340+0000", 2024-02-05T08:08:10.780 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000142764 2024-02-05T08:08:10.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.780 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:10.780 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.768573+0000", 2024-02-05T08:08:10.780 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042321800000000003 2024-02-05T08:08:10.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.781 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.781 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:10.781 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.770286+0000", 2024-02-05T08:08:10.781 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017129109999999999 2024-02-05T08:08:10.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.781 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.781 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:10.781 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773974+0000", 2024-02-05T08:08:10.781 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036879769999999998 2024-02-05T08:08:10.781 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.781 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:10.782 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.782 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.782 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.782 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:101426 7.2 e126/95 7:49eac148:::benchmark_data_smithi134_83016_object101425:head v 126'6361, mlcod=126'6357)", 2024-02-05T08:08:10.782 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:10.767387+0000", 2024-02-05T08:08:10.782 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0065849020000000001, 2024-02-05T08:08:10.782 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0068294940000000002, 2024-02-05T08:08:10.782 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:10.782 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.767387+0000", 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.767387+0000", 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:10.783 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.767388+0000", 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3200000000000004e-07 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.767415+0000", 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7288999999999999e-05 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.767425+0000", 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0318e-05 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.784 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:10.785 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.767432+0000", 2024-02-05T08:08:10.785 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8340000000000001e-06 2024-02-05T08:08:10.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.785 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.785 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:10.785 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.770465+0000", 2024-02-05T08:08:10.785 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030332549999999999 2024-02-05T08:08:10.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.785 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.770485+0000", 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9720000000000001e-05 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:101425 7.c e126/97 7:39be4d9f:::benchmark_data_smithi134_83016_object101424:head v 126'6307, mlcod=126'6304)", 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:10.770783+0000", 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.003188551, 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0035005790000000002, 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:08:10.786 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.770783+0000", 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.770783+0000", 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.770784+0000", 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8999999999999996e-07 2024-02-05T08:08:10.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.770810+0000", 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5868e-05 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.770823+0000", 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.292e-05 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.770831+0000", 2024-02-05T08:08:10.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6860000000000006e-06 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.770938+0000", 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107373 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.770957+0000", 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9069e-05 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.790 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:101427 7.a e126/95 7:5c38cd24:::benchmark_data_smithi134_83016_object101426:head v 126'6163, mlcod=126'6161)", 2024-02-05T08:08:10.790 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:10.772950+0000", 2024-02-05T08:08:10.790 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0010220400000000001, 2024-02-05T08:08:10.790 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001411692, 2024-02-05T08:08:10.790 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:10.790 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:08:10.790 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:10.790 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.790 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:10.790 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.772950+0000", 2024-02-05T08:08:10.791 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.791 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.791 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:10.791 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.772950+0000", 2024-02-05T08:08:10.791 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.791 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.791 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:10.792 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.772951+0000", 2024-02-05T08:08:10.792 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8100000000000001e-07 2024-02-05T08:08:10.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.792 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.792 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:10.792 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.772979+0000", 2024-02-05T08:08:10.792 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8274000000000001e-05 2024-02-05T08:08:10.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.792 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.792 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:10.792 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.772990+0000", 2024-02-05T08:08:10.792 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1037e-05 2024-02-05T08:08:10.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.772998+0000", 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1950000000000003e-06 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773073+0000", 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4697000000000007e-05 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773092+0000", 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8906000000000002e-05 2024-02-05T08:08:10.793 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:101430 7.c e126/97 7:358fe50b:::benchmark_data_smithi134_83016_object101429:head v 126'6308, mlcod=126'6304)", 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:10.773205+0000", 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.000766539, 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001617425, 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:10.794 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773205+0000", 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773205+0000", 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773206+0000", 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8299999999999996e-07 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:10.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773229+0000", 2024-02-05T08:08:10.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3377999999999999e-05 2024-02-05T08:08:10.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.796 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:10.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773239+0000", 2024-02-05T08:08:10.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8600000000000005e-06 2024-02-05T08:08:10.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.796 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:10.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773246+0000", 2024-02-05T08:08:10.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7290000000000003e-06 2024-02-05T08:08:10.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773600+0000", 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00035428499999999997 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773621+0000", 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0599999999999999e-05 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:101432 7.e 7:70b27852:::benchmark_data_smithi134_83016_object101431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:10.797 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:10.773698+0000", 2024-02-05T08:08:10.798 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.00027359900000000002, 2024-02-05T08:08:10.798 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0012225789999999999, 2024-02-05T08:08:10.798 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:10.798 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:08:10.798 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:10.798 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:10.798 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:10.798 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 101432 2024-02-05T08:08:10.798 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.798 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:10.798 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773698+0000", 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773698+0000", 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773700+0000", 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9580000000000001e-06 2024-02-05T08:08:10.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773785+0000", 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4999000000000004e-05 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773797+0000", 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2096000000000001e-05 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773805+0000", 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2790000000000001e-06 2024-02-05T08:08:10.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.774362+0000", 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00055718200000000003 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.774542+0000", 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018000899999999999 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.774674+0000", 2024-02-05T08:08:10.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000131934 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:101433 7.8 7:144e8aeb:::benchmark_data_smithi134_83016_object101432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:10.773826+0000", 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.000145709, 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011764740000000001, 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:10.802 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 101433 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773826+0000", 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773826+0000", 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773827+0000", 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2200000000000004e-07 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773845+0000", 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7963999999999999e-05 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773852+0000", 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2450000000000001e-06 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.804 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.773858+0000", 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6389999999999997e-06 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.774152+0000", 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000294782 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.774332+0000", 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000180144 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:10.805 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:10.806 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,7", 2024-02-05T08:08:10.806 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:10.774461+0000", 2024-02-05T08:08:10.806 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012854699999999999 2024-02-05T08:08:10.806 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.806 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:10.806 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:08:10.807 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 7 2024-02-05T08:08:10.807 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:08:10.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:10.832 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:10.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:10.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:10.982 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:10.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:10.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:11.083 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:11.126 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:11.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:11.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:11.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:11.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:11.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:11.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:11.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:11.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:11.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:11.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 81 16 102106 102090 78.754 61.4375 0.00617088 0.0126926 2024-02-05T08:08:11.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:11.509 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:11.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:11.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:11.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:11 smithi134 ceph-mon[30086]: pgmap v131: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-05T08:08:11.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:11 smithi134 ceph-mon[39278]: pgmap v131: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-05T08:08:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:11 smithi196 ceph-mon[33018]: pgmap v131: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-05T08:08:11.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:11.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:11.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:11.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:08:11.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:11.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:11.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:11.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:11.904 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:11.955 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:11.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:12.056 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:12.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:12.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:12.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:12.227 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:12.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:12.329 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:12.372 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:12.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:12.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:12.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 82 16 102997 102981 78.4727 55.6875 0.0167819 0.012738 2024-02-05T08:08:12.474 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:12.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:12.522 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:12.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:12.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:12.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:12.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:12.750 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:12.796 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:12.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:12.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:12.936 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:12.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:12.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:13.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:13.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:13.078 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:13.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:13.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T08:08:13.180 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:13.260 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:13.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:13.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:13.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:08:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:08:13.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:13.389 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:13.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:13.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:13.447 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 83 16 103950 103934 78.2448 59.5625 0.0137229 0.0127745 2024-02-05T08:08:13.467 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:13 smithi134 ceph-mon[30086]: pgmap v132: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 36 MiB/s wr, 583 op/s 2024-02-05T08:08:13.468 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:13 smithi134 ceph-mon[39278]: pgmap v132: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 36 MiB/s wr, 583 op/s 2024-02-05T08:08:13.491 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:13.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:13.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:13.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:13 smithi196 ceph-mon[33018]: pgmap v132: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 36 MiB/s wr, 583 op/s 2024-02-05T08:08:13.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:13.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:13.650 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:13.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:13.751 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:13.804 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:13.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:13.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:13.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:13.935 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:13.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:13.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:14.037 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:14.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:14.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:14.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:14.216 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:14.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:14.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:14.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:14.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:14 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:14.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:14.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:14.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:14.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 84 16 104899 104883 78.0194 59.3125 0.00668967 0.0128123 2024-02-05T08:08:14.446 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:14.474 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:14.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:14.575 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:14.607 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:14 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:14.616 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:14.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:14.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:14.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:14.744 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:14.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:14.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:14 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:14.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:14.873 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:14.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:14.975 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:15.017 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:15.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:15.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:15.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:15.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:15.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:15.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:15.247 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:15.289 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:15.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:15.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:15.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:15.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:15.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:15.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 85 16 105872 105856 77.8169 60.8125 0.00910874 0.012845 2024-02-05T08:08:15.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:15.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:15.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:15.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:15.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:15.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:15.670 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:15.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:15.772 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:15.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:15 smithi134 ceph-mon[30086]: pgmap v133: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-05T08:08:15.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:15 smithi134 ceph-mon[39278]: pgmap v133: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-05T08:08:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:15 smithi196 ceph-mon[33018]: pgmap v133: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-05T08:08:15.818 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:15.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:15.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:15.943 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:15.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:15.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:16.045 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:16.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:16.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:16.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:16.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:16.218 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:16.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:16.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:16.320 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:16.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:16.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:16.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 86 16 106511 106495 77.3765 39.9375 0.00905491 0.0129183 2024-02-05T08:08:16.444 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:16.485 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:16.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:16.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:16.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:16.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:16.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:16.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:16.716 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:16.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:16.743 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:16.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:16.845 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:16.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:16.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:16.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:16.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:16.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:16.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:16.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:17.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:17.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:17.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:17.226 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:17.265 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:17.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:17.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:17.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:17.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:17.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:17.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 87 16 107209 107193 76.9885 43.625 0.0163552 0.0129839 2024-02-05T08:08:17.492 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:17.526 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:17.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:17.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:17.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:17.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:17.652 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:17.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:17.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:17 smithi134 ceph-mon[30086]: pgmap v134: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-02-05T08:08:17.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:17 smithi134 ceph-mon[39278]: pgmap v134: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-02-05T08:08:17.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:17.777 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:17.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:17.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:17 smithi196 ceph-mon[33018]: pgmap v134: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.00k op/s 2024-02-05T08:08:17.879 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:17.906 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:17.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:17.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:18.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:18.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:08:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:08:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:08:18.031 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:18.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:18.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:18.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:18.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:18.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:18.261 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:18.285 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:18.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:18.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:18.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:18.431 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:18.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:18.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:18.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 88 16 108025 108009 76.6932 51 0.0125617 0.0130342 2024-02-05T08:08:18.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:18.559 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:18.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:18.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:18.661 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:18.686 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:18.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:18.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:18.788 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:18.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:18.838 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:18.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:18.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:18.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:18.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:18.980 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:08:19.022 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:08:19.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:19.068 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:08:19.093 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:19.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:19.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:19.110 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:08:19.148 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:08:19.189 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:08:19.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:19.226 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:08:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:19.238 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:19.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:19.268 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:08:19.271 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:08:19.271 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:08:19.271 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:08:19.271 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:08:19.271 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:08:19.272 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:08:19.272 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:08:19.272 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:08:19.338 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:19.385 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:19.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:19.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 89 16 108963 108947 76.4901 58.625 0.0135772 0.0130686 2024-02-05T08:08:19.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:19.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:19 smithi134 ceph-mon[39278]: pgmap v135: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.5 KiB/s rd, 61 MiB/s wr, 982 op/s 2024-02-05T08:08:19.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:19 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4212449551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:08:19.531 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:19 smithi134 ceph-mon[30086]: pgmap v135: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.5 KiB/s rd, 61 MiB/s wr, 982 op/s 2024-02-05T08:08:19.532 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:19 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4212449551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:08:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:19.545 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:19.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:19.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:19.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:19.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:19.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:19.773 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:19 smithi196 ceph-mon[33018]: pgmap v135: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.5 KiB/s rd, 61 MiB/s wr, 982 op/s 2024-02-05T08:08:19.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:19 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4212449551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:08:19.820 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:19.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:19.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:19.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:19.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:19.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:19.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:20.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:20.094 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:20.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:20.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:20.196 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:20.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:20.222 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:20.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:20.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:20.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:20.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:20.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:20.407 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:20.437 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:20.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Total time run: 90.0126 2024-02-05T08:08:20.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Total writes made: 109931 2024-02-05T08:08:20.444 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Write size: 65536 2024-02-05T08:08:20.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Object size: 65536 2024-02-05T08:08:20.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Bandwidth (MB/sec): 76.3303 2024-02-05T08:08:20.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev Bandwidth: 27.652 2024-02-05T08:08:20.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max bandwidth (MB/sec): 174.375 2024-02-05T08:08:20.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min bandwidth (MB/sec): 39.9375 2024-02-05T08:08:20.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Average IOPS: 1221 2024-02-05T08:08:20.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev IOPS: 442.432 2024-02-05T08:08:20.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max IOPS: 2790 2024-02-05T08:08:20.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min IOPS: 639 2024-02-05T08:08:20.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Average Latency(s): 0.013096 2024-02-05T08:08:20.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev Latency(s): 0.00944946 2024-02-05T08:08:20.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max latency(s): 0.159692 2024-02-05T08:08:20.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min latency(s): 0.00118368 2024-02-05T08:08:20.450 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T08:08:20.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:20.476 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:20.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:20.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:20.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:20.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:20.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:20.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:20.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:20.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:20.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:20.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:20.869 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:20.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:20.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:20.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:20.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:20.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:20.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:21.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:21.146 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:21.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:21.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:21.285 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:21.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:21.386 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:21.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:21.441 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:21.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:21.541 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:21.593 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:21.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:21.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:21.664 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:21 smithi134 ceph-mon[30086]: pgmap v136: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T08:08:21.664 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:21 smithi134 ceph-mon[39278]: pgmap v136: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T08:08:21.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:21.720 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:21.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:21.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:21 smithi196 ceph-mon[33018]: pgmap v136: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T08:08:21.822 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:21.880 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:21.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:21.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:22.025 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:22.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:22.126 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:22.170 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:22.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:22.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:22.272 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:22.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:22.325 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:22.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:22.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:22.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:08:22.479 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:22.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:22.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:22.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:22.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:22.707 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:22.766 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:22.767 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:22.767 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:22.767 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:22 smithi134 ceph-mon[30086]: pgmap v137: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 46 MiB/s wr, 735 op/s 2024-02-05T08:08:22.767 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:08:22.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:22.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:22.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:22 smithi134 ceph-mon[39278]: pgmap v137: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 46 MiB/s wr, 735 op/s 2024-02-05T08:08:22.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:08:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:22.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:22 smithi196 ceph-mon[33018]: pgmap v137: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 46 MiB/s wr, 735 op/s 2024-02-05T08:08:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:08:22.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:22.894 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:22.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:22.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:23.021 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:23.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:23.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:23.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:23.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:23.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:23.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:23.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:23.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:23.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:08:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:08:23.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:23.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:23.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:23.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:23.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:23.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:23.633 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:23.675 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:23.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:23.776 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:23.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:08:23.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:08:23.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:23.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:23 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:23.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:08:23.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:08:23.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:23.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:23 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:23.822 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:23.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:23.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:23.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:23.969 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:23.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:23.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:08:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:08:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:23 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:24.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:24.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:24.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:24.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:24.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:24.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:24.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:24.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:24.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:24.352 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:24.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:24.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:24.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:24.478 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:24.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:24.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:24.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:24.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:24.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:24.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:24.725 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:24 smithi134 ceph-mon[39278]: pgmap v138: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 59 MiB/s wr, 1.13k op/s 2024-02-05T08:08:24.726 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:24 smithi134 ceph-mon[30086]: pgmap v138: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 59 MiB/s wr, 1.13k op/s 2024-02-05T08:08:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:24.749 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:24.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:24.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:24.896 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:24.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:24.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:24.897 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:24 smithi196 ceph-mon[33018]: pgmap v138: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 59 MiB/s wr, 1.13k op/s 2024-02-05T08:08:24.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:25.012 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:08:25.012 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:08:25.013 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-02-05T08:08:25.013 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-05T08:08:25.013 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-05T08:08:25.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-02-05T08:08:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:25.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:25.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:25.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:25.156 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:25.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:25.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:25.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:25.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:25.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:25.302 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:25.309 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:08:25.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:25.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:25.435 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:25.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:25.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:25.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:25.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:25.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:25.650 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:08:25.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:25.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:25.691 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:25.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:25.703 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:08:25.703 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:08:25.765 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:25 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/583437313' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:08:25.765 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:25 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:08:25.792 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:25.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:25.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:25.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:25.894 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:25 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/583437313' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:08:25.894 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:25 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:08:25.895 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:25 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/583437313' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:08:25.895 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:25 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:08:25.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:25.947 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:25.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:26.049 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:26.089 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:26.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:26.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:26.190 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:26.229 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:26.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:26.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:26.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:26.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:26.460 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:26.499 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:26.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:26.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:26.601 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:26.628 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:26.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:26.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:26.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:26.774 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:26.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:26.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:26.780 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:26 smithi134 ceph-mon[30086]: pgmap v139: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 937 B/s rd, 35 MiB/s wr, 740 op/s 2024-02-05T08:08:26.780 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:26 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:08:26.780 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:26 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:08:26.780 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:26 smithi134 ceph-mon[30086]: osdmap e127: 8 total, 8 up, 7 in 2024-02-05T08:08:26.780 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:26 smithi134 ceph-mon[30086]: 4.13 scrub starts 2024-02-05T08:08:26.780 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:26 smithi134 ceph-mon[30086]: 4.13 scrub ok 2024-02-05T08:08:26.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:26 smithi134 ceph-mon[39278]: pgmap v139: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 937 B/s rd, 35 MiB/s wr, 740 op/s 2024-02-05T08:08:26.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:26 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:08:26.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:26 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:08:26.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:26 smithi134 ceph-mon[39278]: osdmap e127: 8 total, 8 up, 7 in 2024-02-05T08:08:26.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:26 smithi134 ceph-mon[39278]: 4.13 scrub starts 2024-02-05T08:08:26.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:26 smithi134 ceph-mon[39278]: 4.13 scrub ok 2024-02-05T08:08:26.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:26.914 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:26.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:26.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:26.940 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:26 smithi196 ceph-mon[33018]: pgmap v139: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 937 B/s rd, 35 MiB/s wr, 740 op/s 2024-02-05T08:08:26.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:26 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:08:26.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:26 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:08:26.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:26 smithi196 ceph-mon[33018]: osdmap e127: 8 total, 8 up, 7 in 2024-02-05T08:08:26.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:26 smithi196 ceph-mon[33018]: 4.13 scrub starts 2024-02-05T08:08:26.941 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:26 smithi196 ceph-mon[33018]: 4.13 scrub ok 2024-02-05T08:08:27.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:27.058 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:27.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:27.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:27.160 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:27.199 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:27.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:27.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T08:08:27.301 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:27.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:27.351 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:27.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:27.407 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:08:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-05T08:08:27.408 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:08:27.451 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:27.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:27.492 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:27.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:27.594 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:27.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:27.635 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:27.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:27.736 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:27.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:27.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:27.867 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:27.910 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:27.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:28.004 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:08:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:08:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:08:28.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:28.039 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:28.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:28.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:28.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:28.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:28.268 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:28.294 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:28.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:28.396 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:28.435 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:28.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:28.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:28.562 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:28.564 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:28.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:28.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:28.666 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:28.672 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:28 smithi134 ceph-mon[30086]: pgmap v141: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 38 MiB/s wr, 913 op/s 2024-02-05T08:08:28.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:28 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:08:28.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:28 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:08:28.673 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:28 smithi134 ceph-mon[39278]: pgmap v141: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 38 MiB/s wr, 913 op/s 2024-02-05T08:08:28.673 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:28 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:08:28.673 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:28 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:08:28.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:28.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:28.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:28.796 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:28.853 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:28.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:28.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:28.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:28.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:28.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:28 smithi196 ceph-mon[33018]: pgmap v141: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 38 MiB/s wr, 913 op/s 2024-02-05T08:08:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:28 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:08:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:28 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:08:29.099 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:29.144 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:29.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:29.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:29.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:29.271 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:29.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:29.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:29.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:29.420 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:29.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:29.468 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:08:29.513 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:08:29.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:29.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:29.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:29.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:29.553 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:08:29.602 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:08:29.642 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:08:29.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:29.682 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:08:29.695 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:29.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:29.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:29.723 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:08:29.764 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:08:29.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[30086]: 7.8 scrub starts 2024-02-05T08:08:29.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[30086]: 3.1b scrub starts 2024-02-05T08:08:29.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[30086]: 3.1b scrub ok 2024-02-05T08:08:29.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:08:29.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:08:29.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[30086]: osdmap e128: 8 total, 8 up, 7 in 2024-02-05T08:08:29.765 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[30086]: 5.c scrub starts 2024-02-05T08:08:29.765 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[30086]: 5.c scrub ok 2024-02-05T08:08:29.765 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2079381893' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:08:29.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[39278]: 7.8 scrub starts 2024-02-05T08:08:29.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[39278]: 3.1b scrub starts 2024-02-05T08:08:29.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[39278]: 3.1b scrub ok 2024-02-05T08:08:29.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:08:29.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:08:29.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[39278]: osdmap e128: 8 total, 8 up, 7 in 2024-02-05T08:08:29.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[39278]: 5.c scrub starts 2024-02-05T08:08:29.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[39278]: 5.c scrub ok 2024-02-05T08:08:29.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:29 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2079381893' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:08:29.770 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:08:29.770 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:08:29.770 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:08:29.770 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:08:29.770 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:08:29.770 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:08:29.770 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:08:29.770 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:08:29.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:29.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:29.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:29.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:29.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:29.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:29.952 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:29.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:30.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:08:30.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:29 smithi196 ceph-mon[33018]: 7.8 scrub starts 2024-02-05T08:08:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:29 smithi196 ceph-mon[33018]: 3.1b scrub starts 2024-02-05T08:08:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:29 smithi196 ceph-mon[33018]: 3.1b scrub ok 2024-02-05T08:08:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:29 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:08:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:29 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:08:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:29 smithi196 ceph-mon[33018]: osdmap e128: 8 total, 8 up, 7 in 2024-02-05T08:08:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:29 smithi196 ceph-mon[33018]: 5.c scrub starts 2024-02-05T08:08:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:29 smithi196 ceph-mon[33018]: 5.c scrub ok 2024-02-05T08:08:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:29 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2079381893' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:08:30.079 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:30.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:30.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:30.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:30.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:30.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:30.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:30.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:30.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:30.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:30.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:30.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:30.562 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:30.600 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:30.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:30.702 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:30.756 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:30.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:30.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:30.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:30.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:30.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:30 smithi134 ceph-mon[30086]: pgmap v143: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 20 MiB/s wr, 1.23k op/s 2024-02-05T08:08:30.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:30 smithi134 ceph-mon[30086]: 5.7 scrub starts 2024-02-05T08:08:30.928 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:30 smithi134 ceph-mon[30086]: 5.7 scrub ok 2024-02-05T08:08:30.928 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:30 smithi134 ceph-mon[39278]: pgmap v143: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 20 MiB/s wr, 1.23k op/s 2024-02-05T08:08:30.928 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:30 smithi134 ceph-mon[39278]: 5.7 scrub starts 2024-02-05T08:08:30.928 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:30 smithi134 ceph-mon[39278]: 5.7 scrub ok 2024-02-05T08:08:30.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:31.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:31.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:31.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:30 smithi196 ceph-mon[33018]: pgmap v143: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 20 MiB/s wr, 1.23k op/s 2024-02-05T08:08:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:30 smithi196 ceph-mon[33018]: 5.7 scrub starts 2024-02-05T08:08:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:30 smithi196 ceph-mon[33018]: 5.7 scrub ok 2024-02-05T08:08:31.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:31.157 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:31.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:31.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:31.301 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:31.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:31.403 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:31.453 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:31.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:31.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:31.485 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:08:31.485 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:08:31.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:08:31.485 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:08:31.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.485 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:19184 7.8 7:1698bfb9:::benchmark_data_smithi134_83016_object19183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:08:31.485 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.649970+0000", 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 92.831890630999993, 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12589771399999999, 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 19184 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649970+0000", 2024-02-05T08:08:31.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649970+0000", 2024-02-05T08:08:31.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649971+0000", 2024-02-05T08:08:31.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3900000000000003e-07 2024-02-05T08:08:31.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649989+0000", 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8097999999999999e-05 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649999+0000", 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0023999999999999e-05 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.650005+0000", 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7200000000000003e-06 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.650045+0000", 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0070000000000001e-05 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.650118+0000", 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2837999999999996e-05 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,7", 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.650209+0000", 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1215e-05 2024-02-05T08:08:31.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.650951+0000", 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00074151200000000005 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651227+0000", 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00027632399999999999 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651273+0000", 2024-02-05T08:08:31.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6313999999999997e-05 2024-02-05T08:08:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.775807+0000", 2024-02-05T08:08:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12453357800000001 2024-02-05T08:08:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.775868+0000", 2024-02-05T08:08:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1081999999999996e-05 2024-02-05T08:08:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.491 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.492 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:19184 7.8 e104/95 ondisk, result = 0)", 2024-02-05T08:08:31.492 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.651219+0000", 2024-02-05T08:08:31.492 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 92.830641526999997, 2024-02-05T08:08:31.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.124660887, 2024-02-05T08:08:31.492 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.492 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:08:31.492 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651219+0000", 2024-02-05T08:08:31.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651219+0000", 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651220+0000", 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5700000000000002e-07 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.493 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651224+0000", 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.732e-06 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651232+0000", 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4859999999999997e-06 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651237+0000", 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0930000000000002e-06 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651266+0000", 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8612999999999999e-05 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651270+0000", 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3499999999999999e-06 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.775880+0000", 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12460995599999999 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.495 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.496 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21740 7.e 7:702731a7:::benchmark_data_smithi134_83016_object21739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:08:31.496 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.327781+0000", 2024-02-05T08:08:31.496 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 91.154079843999995, 2024-02-05T08:08:31.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15303014300000001, 2024-02-05T08:08:31.496 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.496 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.496 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.496 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.496 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.496 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21740 2024-02-05T08:08:31.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.497 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.497 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327781+0000", 2024-02-05T08:08:31.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.497 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327781+0000", 2024-02-05T08:08:31.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327783+0000", 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.688e-06 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327805+0000", 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2300999999999999e-05 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327817+0000", 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2233e-05 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327825+0000", 2024-02-05T08:08:31.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1450000000000008e-06 2024-02-05T08:08:31.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.500 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327885+0000", 2024-02-05T08:08:31.500 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9546999999999998e-05 2024-02-05T08:08:31.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.500 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.500 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.328002+0000", 2024-02-05T08:08:31.500 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000117374 2024-02-05T08:08:31.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.500 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.500 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.328125+0000", 2024-02-05T08:08:31.500 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012256800000000001 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329306+0000", 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001181423 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330032+0000", 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00072612800000000004 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480676+0000", 2024-02-05T08:08:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15064364599999999 2024-02-05T08:08:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480734+0000", 2024-02-05T08:08:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7960000000000001e-05 2024-02-05T08:08:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480811+0000", 2024-02-05T08:08:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7130000000000002e-05 2024-02-05T08:08:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.502 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.503 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.503 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:21738 7.a e104/95 7:5f5817b2:::benchmark_data_smithi134_83016_object21737:head v 104'1353, mlcod=104'1351)", 2024-02-05T08:08:31.503 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.329123+0000", 2024-02-05T08:08:31.503 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 91.152737630999994, 2024-02-05T08:08:31.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15121406700000001, 2024-02-05T08:08:31.503 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.503 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.503 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329123+0000", 2024-02-05T08:08:31.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329123+0000", 2024-02-05T08:08:31.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329124+0000", 2024-02-05T08:08:31.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2500000000000001e-07 2024-02-05T08:08:31.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329156+0000", 2024-02-05T08:08:31.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2563e-05 2024-02-05T08:08:31.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329166+0000", 2024-02-05T08:08:31.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6069999999999997e-06 2024-02-05T08:08:31.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329172+0000", 2024-02-05T08:08:31.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5339999999999996e-06 2024-02-05T08:08:31.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329270+0000", 2024-02-05T08:08:31.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7374999999999995e-05 2024-02-05T08:08:31.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329289+0000", 2024-02-05T08:08:31.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9258000000000001e-05 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330442+0000", 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00115318 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480337+0000", 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14989512499999999 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21746 7.8 7:1fd2f339:::benchmark_data_smithi134_83016_object21745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.329211+0000", 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 91.152650156000007, 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15186722999999999, 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21746 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329211+0000", 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329211+0000", 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329212+0000", 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1200000000000001e-06 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329233+0000", 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1157000000000001e-05 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329245+0000", 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2095e-05 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329253+0000", 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0059999999999992e-06 2024-02-05T08:08:31.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329312+0000", 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8953000000000003e-05 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329460+0000", 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014786199999999999 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,7", 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329592+0000", 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013251199999999999 2024-02-05T08:08:31.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330443+0000", 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00085042699999999995 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480847+0000", 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15040416000000001 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480896+0000", 2024-02-05T08:08:31.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9152e-05 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.481044+0000", 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014736299999999999 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.481078+0000", 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4422999999999999e-05 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:21746 7.8 e104/95 ondisk, result = 0)", 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.330823+0000", 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 91.151038029000006, 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15025973000000001, 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330823+0000", 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330823+0000", 2024-02-05T08:08:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330823+0000", 2024-02-05T08:08:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7100000000000002e-07 2024-02-05T08:08:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330827+0000", 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7249999999999999e-06 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480582+0000", 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14975533799999999 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480591+0000", 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8789999999999992e-06 2024-02-05T08:08:31.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480886+0000", 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00029442999999999998 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480892+0000", 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1240000000000002e-06 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.481083+0000", 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019066299999999999 2024-02-05T08:08:31.517 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:21746 7.8 e104/95 ondisk, result = 0)", 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.330832+0000", 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 91.151028960000005, 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.150026044, 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.518 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330832+0000", 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330832+0000", 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330832+0000", 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3799999999999997e-07 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330835+0000", 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.971e-06 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480208+0000", 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.149372539 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480221+0000", 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2695e-05 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480323+0000", 2024-02-05T08:08:31.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010194799999999999 2024-02-05T08:08:31.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480331+0000", 2024-02-05T08:08:31.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3499999999999997e-06 2024-02-05T08:08:31.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480858+0000", 2024-02-05T08:08:31.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052700299999999995 2024-02-05T08:08:31.521 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:99784 7.e 7:78fc162f:::benchmark_data_smithi134_83016_object99783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:08.939664+0000", 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 22.542196952000001, 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13795306500000001, 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 99784 2024-02-05T08:08:31.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.523 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939664+0000", 2024-02-05T08:08:31.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939664+0000", 2024-02-05T08:08:31.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939664+0000", 2024-02-05T08:08:31.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7e-07 2024-02-05T08:08:31.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939688+0000", 2024-02-05T08:08:31.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3336000000000002e-05 2024-02-05T08:08:31.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939695+0000", 2024-02-05T08:08:31.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5959999999999998e-06 2024-02-05T08:08:31.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939701+0000", 2024-02-05T08:08:31.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6119999999999998e-06 2024-02-05T08:08:31.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939767+0000", 2024-02-05T08:08:31.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6088999999999997e-05 2024-02-05T08:08:31.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939882+0000", 2024-02-05T08:08:31.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011451900000000001 2024-02-05T08:08:31.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.940010+0000", 2024-02-05T08:08:31.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012856599999999999 2024-02-05T08:08:31.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.943195+0000", 2024-02-05T08:08:31.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003185121 2024-02-05T08:08:31.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947687+0000", 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.004491292 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.077504+0000", 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12981711600000001 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.077570+0000", 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6507999999999994e-05 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.077617+0000", 2024-02-05T08:08:31.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6839999999999999e-05 2024-02-05T08:08:31.528 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.528 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.528 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.529 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106208 7.e 7:79fec55e:::benchmark_data_smithi134_83016_object106207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:31.529 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:15.904619+0000", 2024-02-05T08:08:31.529 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 15.577242283, 2024-02-05T08:08:31.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.120579906, 2024-02-05T08:08:31.529 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.529 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.529 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.529 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.529 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.529 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106208 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904619+0000", 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904619+0000", 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904620+0000", 2024-02-05T08:08:31.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7579999999999999e-06 2024-02-05T08:08:31.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.531 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904647+0000", 2024-02-05T08:08:31.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6169e-05 2024-02-05T08:08:31.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.531 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904661+0000", 2024-02-05T08:08:31.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4321000000000001e-05 2024-02-05T08:08:31.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.532 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904669+0000", 2024-02-05T08:08:31.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7800000000000001e-06 2024-02-05T08:08:31.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.532 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909475+0000", 2024-02-05T08:08:31.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0048067120000000003 2024-02-05T08:08:31.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.532 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909614+0000", 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013891899999999999 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909949+0000", 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00033502599999999999 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.922098+0000", 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012149139 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.922137+0000", 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9004000000000001e-05 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025064+0000", 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.102926244 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025154+0000", 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0464000000000007e-05 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025198+0000", 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4369999999999997e-05 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106209 7.e 7:78b87070:::benchmark_data_smithi134_83016_object106208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:15.904686+0000", 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 15.577174492999999, 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.120675377, 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.535 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106209 2024-02-05T08:08:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904686+0000", 2024-02-05T08:08:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904686+0000", 2024-02-05T08:08:31.536 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904687+0000", 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7e-07 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904707+0000", 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0615000000000002e-05 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904715+0000", 2024-02-05T08:08:31.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3690000000000001e-06 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904720+0000", 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5550000000000002e-06 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.915304+0000", 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010583884 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.915436+0000", 2024-02-05T08:08:31.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013168000000000001 2024-02-05T08:08:31.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.915587+0000", 2024-02-05T08:08:31.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000150815 2024-02-05T08:08:31.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.922149+0000", 2024-02-05T08:08:31.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0065619090000000003 2024-02-05T08:08:31.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.017938+0000", 2024-02-05T08:08:31.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.095789183 2024-02-05T08:08:31.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025240+0000", 2024-02-05T08:08:31.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0073026159999999996 2024-02-05T08:08:31.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025325+0000", 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4917e-05 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025362+0000", 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6363999999999999e-05 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106211 7.e 7:7bf49e27:::benchmark_data_smithi134_83016_object106210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:15.906247+0000", 2024-02-05T08:08:31.541 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 15.575614249999999, 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12878605000000001, 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106211 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.906247+0000", 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.906247+0000", 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.906248+0000", 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.06e-06 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.906268+0000", 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0755999999999999e-05 2024-02-05T08:08:31.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.906278+0000", 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8030000000000008e-06 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.906285+0000", 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0929999999999997e-06 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.918969+0000", 2024-02-05T08:08:31.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012684140999999999 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.919104+0000", 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013453299999999999 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.919253+0000", 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014935 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.545 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.017948+0000", 2024-02-05T08:08:31.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.098694226999999995 2024-02-05T08:08:31.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.546 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.017981+0000", 2024-02-05T08:08:31.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3278000000000002e-05 2024-02-05T08:08:31.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.546 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.034565+0000", 2024-02-05T08:08:31.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.016584216999999998 2024-02-05T08:08:31.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.034974+0000", 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00040871100000000002 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.035033+0000", 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8881000000000001e-05 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.548 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106217 7.e 7:74288038:::benchmark_data_smithi134_83016_object106216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:31.548 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:15.909843+0000", 2024-02-05T08:08:31.548 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 15.572018367, 2024-02-05T08:08:31.548 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12792978999999999, 2024-02-05T08:08:31.548 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.548 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.548 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.548 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.548 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.548 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106217 2024-02-05T08:08:31.548 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.548 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909843+0000", 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909843+0000", 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909844+0000", 2024-02-05T08:08:31.549 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.206e-06 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909867+0000", 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3632999999999999e-05 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909876+0000", 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8219999999999996e-06 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909884+0000", 2024-02-05T08:08:31.550 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3370000000000002e-06 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.018007+0000", 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.108123313 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.018147+0000", 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014017599999999999 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.551 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.018392+0000", 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00024491200000000002 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025444+0000", 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.007052221 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025502+0000", 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8044000000000002e-05 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.552 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.035086+0000", 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0095837630000000004 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.037726+0000", 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0026399639999999999 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.037772+0000", 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6399000000000001e-05 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.553 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106225 7.e 7:7e4a02c0:::benchmark_data_smithi134_83016_object106224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:15.917206+0000", 2024-02-05T08:08:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 15.564655010999999, 2024-02-05T08:08:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.120823756, 2024-02-05T08:08:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106225 2024-02-05T08:08:31.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.917206+0000", 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.917206+0000", 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.917207+0000", 2024-02-05T08:08:31.555 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5990000000000001e-06 2024-02-05T08:08:31.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.556 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:31.557 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.557 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.917233+0000", 2024-02-05T08:08:31.557 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5848000000000001e-05 2024-02-05T08:08:31.557 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.557 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.558 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.558 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.917243+0000", 2024-02-05T08:08:31.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7040000000000006e-06 2024-02-05T08:08:31.558 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.558 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.558 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.917249+0000", 2024-02-05T08:08:31.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4649999999999999e-06 2024-02-05T08:08:31.558 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.558 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.558 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.021506+0000", 2024-02-05T08:08:31.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.104256852 2024-02-05T08:08:31.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.559 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.559 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.021656+0000", 2024-02-05T08:08:31.559 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015012700000000001 2024-02-05T08:08:31.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.559 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.559 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.021817+0000", 2024-02-05T08:08:31.559 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016057500000000001 2024-02-05T08:08:31.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.559 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025489+0000", 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036717680000000002 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.029370+0000", 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00388128 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.037825+0000", 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0084547530000000006 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.037992+0000", 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016755700000000001 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.038030+0000", 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7228000000000003e-05 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.561 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106460 7.e 7:7b65a89c:::benchmark_data_smithi134_83016_object106459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.272760+0000", 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 15.209100875000001, 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14252445999999999, 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106460 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272760+0000", 2024-02-05T08:08:31.562 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272760+0000", 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272761+0000", 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3820000000000001e-06 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272786+0000", 2024-02-05T08:08:31.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4306000000000001e-05 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272798+0000", 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2347999999999999e-05 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272807+0000", 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6130000000000004e-06 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272875+0000", 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8631000000000001e-05 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.273003+0000", 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012771200000000001 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.273168+0000", 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016530000000000001 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.276443+0000", 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032742489999999999 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.278626+0000", 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0021837380000000002 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.415201+0000", 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13657491199999999 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.415250+0000", 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8547000000000001e-05 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.415284+0000", 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4721999999999997e-05 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106678 7.e 7:7f897189:::benchmark_data_smithi134_83016_object106677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.632287+0000", 2024-02-05T08:08:31.567 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 14.849573909, 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14517614000000001, 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106678 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632287+0000", 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632287+0000", 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632288+0000", 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3e-06 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632314+0000", 2024-02-05T08:08:31.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5622e-05 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632326+0000", 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1901e-05 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632333+0000", 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.08e-06 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633044+0000", 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071079200000000002 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633761+0000", 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071729699999999999 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633976+0000", 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021524099999999999 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650870+0000", 2024-02-05T08:08:31.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.016893551 2024-02-05T08:08:31.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650906+0000", 2024-02-05T08:08:31.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6427000000000002e-05 2024-02-05T08:08:31.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777225+0000", 2024-02-05T08:08:31.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.126318758 2024-02-05T08:08:31.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777321+0000", 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6436999999999998e-05 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777463+0000", 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000141734 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.573 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106682 7.e 7:7cfa27eb:::benchmark_data_smithi134_83016_object106681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.633782+0000", 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 14.84807898, 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14385479700000001, 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106682 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633782+0000", 2024-02-05T08:08:31.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633782+0000", 2024-02-05T08:08:31.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633783+0000", 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1090000000000001e-06 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633813+0000", 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9856999999999999e-05 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633823+0000", 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0222e-05 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633830+0000", 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0639999999999996e-06 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642537+0000", 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0087064080000000005 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642683+0000", 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000146527 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642858+0000", 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017454499999999999 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650945+0000", 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0080870500000000001 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.769282+0000", 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.118337331 2024-02-05T08:08:31.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777475+0000", 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0081926900000000007 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777545+0000", 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0735000000000006e-05 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777637+0000", 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1259000000000002e-05 2024-02-05T08:08:31.579 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106688 7.e 7:758dcfa0:::benchmark_data_smithi134_83016_object106687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.642683+0000", 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 14.839177436, 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13519677799999999, 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.580 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106688 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642683+0000", 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642683+0000", 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642685+0000", 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.279e-06 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642707+0000", 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2138999999999999e-05 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642717+0000", 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0385e-05 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642725+0000", 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5660000000000001e-06 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650974+0000", 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.008249269 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.651086+0000", 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111517 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.651232+0000", 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000146699 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.762167+0000", 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.110934479 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777649+0000", 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.015482666000000001 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777716+0000", 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6764e-05 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777816+0000", 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010012 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777880+0000", 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3894999999999997e-05 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.585 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106691 7.e 7:7474d3c9:::benchmark_data_smithi134_83016_object106690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.647361+0000", 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 14.834499437, 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14090623099999999, 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106691 2024-02-05T08:08:31.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.647361+0000", 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.647361+0000", 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.647363+0000", 2024-02-05T08:08:31.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.336e-06 2024-02-05T08:08:31.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.647484+0000", 2024-02-05T08:08:31.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012124 2024-02-05T08:08:31.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.647495+0000", 2024-02-05T08:08:31.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0502999999999999e-05 2024-02-05T08:08:31.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.647501+0000", 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8990000000000004e-06 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.655409+0000", 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0079076919999999992 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.655529+0000", 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000120269 2024-02-05T08:08:31.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.655641+0000", 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111978 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.762224+0000", 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.10658280000000001 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777656+0000", 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.015431469 2024-02-05T08:08:31.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.788030+0000", 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010374089 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.788223+0000", 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019300599999999999 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.788268+0000", 2024-02-05T08:08:31.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4950000000000002e-05 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106698 7.e 7:7004ec6e:::benchmark_data_smithi134_83016_object106697:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.650657+0000", 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 14.831203692999999, 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.140741798, 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.592 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106698 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650657+0000", 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650657+0000", 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650658+0000", 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2810000000000001e-06 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650680+0000", 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1571000000000001e-05 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650689+0000", 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3780000000000007e-06 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650696+0000", 2024-02-05T08:08:31.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1430000000000003e-06 2024-02-05T08:08:31.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.762260+0000", 2024-02-05T08:08:31.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.11156439999999999 2024-02-05T08:08:31.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.762424+0000", 2024-02-05T08:08:31.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016437999999999999 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.762591+0000", 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016713700000000001 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.769322+0000", 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0067301870000000003 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777662+0000", 2024-02-05T08:08:31.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0083403160000000004 2024-02-05T08:08:31.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.791304+0000", 2024-02-05T08:08:31.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.013641893 2024-02-05T08:08:31.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.791362+0000", 2024-02-05T08:08:31.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7669e-05 2024-02-05T08:08:31.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.598 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.791399+0000", 2024-02-05T08:08:31.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7443e-05 2024-02-05T08:08:31.598 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.598 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:107150 7.e 7:7956e5e6:::benchmark_data_smithi134_83016_object107149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:17.274871+0000", 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 14.206990064999999, 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13561259000000001, 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 107150 2024-02-05T08:08:31.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.600 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:31.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:31.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274871+0000", 2024-02-05T08:08:31.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:31.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274871+0000", 2024-02-05T08:08:31.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:31.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274873+0000", 2024-02-05T08:08:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7269999999999999e-06 2024-02-05T08:08:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274895+0000", 2024-02-05T08:08:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2886000000000001e-05 2024-02-05T08:08:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274905+0000", 2024-02-05T08:08:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1800000000000002e-06 2024-02-05T08:08:31.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274911+0000", 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.674e-06 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274978+0000", 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6403999999999996e-05 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.275118+0000", 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014021699999999999 2024-02-05T08:08:31.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.275286+0000", 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016760899999999999 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.278162+0000", 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00287601 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.278262+0000", 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000100324 2024-02-05T08:08:31.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.604 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:31.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.410091+0000", 2024-02-05T08:08:31.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13182898700000001 2024-02-05T08:08:31.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.604 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:08:31.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.410282+0000", 2024-02-05T08:08:31.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000191215 2024-02-05T08:08:31.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:31.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:31.604 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:08:31.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.410483+0000", 2024-02-05T08:08:31.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020135699999999999 2024-02-05T08:08:31.605 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.605 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.605 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.605 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:31.605 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:31.605 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:08:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:31.625 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:31.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:31.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:31.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:31.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:31.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[30086]: 7.8 scrub ok 2024-02-05T08:08:31.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[30086]: 3.17 scrub starts 2024-02-05T08:08:31.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[30086]: 3.17 scrub ok 2024-02-05T08:08:31.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[30086]: 4.18 scrub starts 2024-02-05T08:08:31.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[30086]: 4.18 scrub ok 2024-02-05T08:08:31.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[30086]: 6.12 scrub starts 2024-02-05T08:08:31.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[30086]: 6.12 scrub ok 2024-02-05T08:08:31.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[39278]: 7.8 scrub ok 2024-02-05T08:08:31.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[39278]: 3.17 scrub starts 2024-02-05T08:08:31.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[39278]: 3.17 scrub ok 2024-02-05T08:08:31.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[39278]: 4.18 scrub starts 2024-02-05T08:08:31.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[39278]: 4.18 scrub ok 2024-02-05T08:08:31.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[39278]: 6.12 scrub starts 2024-02-05T08:08:31.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:31 smithi134 ceph-mon[39278]: 6.12 scrub ok 2024-02-05T08:08:31.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:31.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:31.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:31.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:31.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:32.022 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:32.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:31 smithi196 ceph-mon[33018]: 7.8 scrub ok 2024-02-05T08:08:32.114 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:31 smithi196 ceph-mon[33018]: 3.17 scrub starts 2024-02-05T08:08:32.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:31 smithi196 ceph-mon[33018]: 3.17 scrub ok 2024-02-05T08:08:32.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:31 smithi196 ceph-mon[33018]: 4.18 scrub starts 2024-02-05T08:08:32.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:31 smithi196 ceph-mon[33018]: 4.18 scrub ok 2024-02-05T08:08:32.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:31 smithi196 ceph-mon[33018]: 6.12 scrub starts 2024-02-05T08:08:32.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:31 smithi196 ceph-mon[33018]: 6.12 scrub ok 2024-02-05T08:08:32.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:32.170 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:32.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:32.271 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:32.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:32.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:32.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:32.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:32.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:32.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:32.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:32.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:32.555 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:32.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:32.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:32.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:32.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:32.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:32 smithi134 ceph-mon[30086]: pgmap v144: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 636 op/s 2024-02-05T08:08:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:32 smithi134 ceph-mon[30086]: 5.2 scrub starts 2024-02-05T08:08:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:32 smithi134 ceph-mon[30086]: 5.2 scrub ok 2024-02-05T08:08:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:32 smithi134 ceph-mon[30086]: 4.e scrub starts 2024-02-05T08:08:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:32 smithi134 ceph-mon[30086]: 4.e scrub ok 2024-02-05T08:08:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:32 smithi134 ceph-mon[39278]: pgmap v144: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 636 op/s 2024-02-05T08:08:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:32 smithi134 ceph-mon[39278]: 5.2 scrub starts 2024-02-05T08:08:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:32 smithi134 ceph-mon[39278]: 5.2 scrub ok 2024-02-05T08:08:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:32 smithi134 ceph-mon[39278]: 4.e scrub starts 2024-02-05T08:08:32.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:32 smithi134 ceph-mon[39278]: 4.e scrub ok 2024-02-05T08:08:32.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:32.825 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:32.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:32.896 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:32.928 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:32.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T08:08:32.971 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:32.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:32.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:33.025 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:32 smithi196 ceph-mon[33018]: pgmap v144: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 636 op/s 2024-02-05T08:08:33.025 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:32 smithi196 ceph-mon[33018]: 5.2 scrub starts 2024-02-05T08:08:33.025 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:32 smithi196 ceph-mon[33018]: 5.2 scrub ok 2024-02-05T08:08:33.025 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:32 smithi196 ceph-mon[33018]: 4.e scrub starts 2024-02-05T08:08:33.025 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:32 smithi196 ceph-mon[33018]: 4.e scrub ok 2024-02-05T08:08:33.074 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:33.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:33.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:33.264 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:08:33.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 0 1 2024-02-05T08:08:33.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:33.295 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:33.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:33.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:08:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:08:33.395 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:33.439 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:33.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:33.541 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:33.586 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:33.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:33.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:33.687 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:33.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:33.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:33.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:33.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:33.867 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:33.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:33.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:33.969 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:33.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:33 smithi134 ceph-mon[30086]: 3.15 scrub starts 2024-02-05T08:08:33.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:33 smithi134 ceph-mon[30086]: 3.15 scrub ok 2024-02-05T08:08:33.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:33 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:33.970 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:33 smithi134 ceph-mon[39278]: 3.15 scrub starts 2024-02-05T08:08:33.970 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:33 smithi134 ceph-mon[39278]: 3.15 scrub ok 2024-02-05T08:08:33.970 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:33 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:34.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:08:34.023 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:34.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:33 smithi196 ceph-mon[33018]: 3.15 scrub starts 2024-02-05T08:08:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:33 smithi196 ceph-mon[33018]: 3.15 scrub ok 2024-02-05T08:08:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:33 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:34.125 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:34.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:34.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:34.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:34.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:34.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:34.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:34.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:34.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:34.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:34.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:34.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:34.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:34.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:34.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:34.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:34.716 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:34.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:34.789 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:34.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:34.859 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:34.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:34.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:34.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:34.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:34.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:35.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:34 smithi134 ceph-mon[30086]: 4.8 scrub starts 2024-02-05T08:08:35.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:34 smithi134 ceph-mon[30086]: 4.8 scrub ok 2024-02-05T08:08:35.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:34 smithi134 ceph-mon[30086]: pgmap v145: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T08:08:35.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:34 smithi134 ceph-mon[30086]: 3.3 scrub starts 2024-02-05T08:08:35.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:34 smithi134 ceph-mon[30086]: 3.3 scrub ok 2024-02-05T08:08:35.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:34 smithi134 ceph-mon[39278]: 4.8 scrub starts 2024-02-05T08:08:35.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:34 smithi134 ceph-mon[39278]: 4.8 scrub ok 2024-02-05T08:08:35.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:34 smithi134 ceph-mon[39278]: pgmap v145: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T08:08:35.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:34 smithi134 ceph-mon[39278]: 3.3 scrub starts 2024-02-05T08:08:35.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:34 smithi134 ceph-mon[39278]: 3.3 scrub ok 2024-02-05T08:08:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:34 smithi196 ceph-mon[33018]: 4.8 scrub starts 2024-02-05T08:08:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:34 smithi196 ceph-mon[33018]: 4.8 scrub ok 2024-02-05T08:08:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:34 smithi196 ceph-mon[33018]: pgmap v145: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T08:08:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:34 smithi196 ceph-mon[33018]: 3.3 scrub starts 2024-02-05T08:08:35.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:34 smithi196 ceph-mon[33018]: 3.3 scrub ok 2024-02-05T08:08:35.092 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:35.137 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:35.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:35.239 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:35.292 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:35.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:35.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:35.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:35.422 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:35.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:35.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:35.568 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:35.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:35.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:35.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:35.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:35.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:35.833 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:35 smithi134 ceph-mon[30086]: 6.7 scrub starts 2024-02-05T08:08:35.833 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:35 smithi134 ceph-mon[30086]: 6.7 scrub ok 2024-02-05T08:08:35.833 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:35 smithi134 ceph-mon[30086]: 4.c scrub starts 2024-02-05T08:08:35.833 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:35 smithi134 ceph-mon[30086]: 4.c scrub ok 2024-02-05T08:08:35.834 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:35 smithi134 ceph-mon[30086]: 3.11 scrub starts 2024-02-05T08:08:35.834 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:35 smithi134 ceph-mon[30086]: 3.11 scrub ok 2024-02-05T08:08:35.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:35 smithi134 ceph-mon[39278]: 6.7 scrub starts 2024-02-05T08:08:35.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:35 smithi134 ceph-mon[39278]: 6.7 scrub ok 2024-02-05T08:08:35.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:35 smithi134 ceph-mon[39278]: 4.c scrub starts 2024-02-05T08:08:35.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:35 smithi134 ceph-mon[39278]: 4.c scrub ok 2024-02-05T08:08:35.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:35 smithi134 ceph-mon[39278]: 3.11 scrub starts 2024-02-05T08:08:35.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:35 smithi134 ceph-mon[39278]: 3.11 scrub ok 2024-02-05T08:08:35.850 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:35.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:35.954 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:36.012 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:36.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:35 smithi196 ceph-mon[33018]: 6.7 scrub starts 2024-02-05T08:08:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:35 smithi196 ceph-mon[33018]: 6.7 scrub ok 2024-02-05T08:08:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:35 smithi196 ceph-mon[33018]: 4.c scrub starts 2024-02-05T08:08:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:35 smithi196 ceph-mon[33018]: 4.c scrub ok 2024-02-05T08:08:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:35 smithi196 ceph-mon[33018]: 3.11 scrub starts 2024-02-05T08:08:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:35 smithi196 ceph-mon[33018]: 3.11 scrub ok 2024-02-05T08:08:36.113 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:36.160 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:36.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:36.261 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:36.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:36.303 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:36.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:36.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:36.429 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:36.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:36.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:36.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:36.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:36.579 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:36.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:36.680 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:36.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:36.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:36.756 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:08:36.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:36.856 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:36.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:36.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:36.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:37.003 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:37.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:37.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:36 smithi134 ceph-mon[30086]: 5.1b scrub starts 2024-02-05T08:08:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:36 smithi134 ceph-mon[30086]: 5.1b scrub ok 2024-02-05T08:08:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:36 smithi134 ceph-mon[30086]: pgmap v146: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T08:08:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3519455633' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:08:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:36 smithi134 ceph-mon[39278]: 5.1b scrub starts 2024-02-05T08:08:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:36 smithi134 ceph-mon[39278]: 5.1b scrub ok 2024-02-05T08:08:37.032 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:36 smithi134 ceph-mon[39278]: pgmap v146: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T08:08:37.032 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3519455633' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:08:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:36 smithi196 ceph-mon[33018]: 5.1b scrub starts 2024-02-05T08:08:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:36 smithi196 ceph-mon[33018]: 5.1b scrub ok 2024-02-05T08:08:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:36 smithi196 ceph-mon[33018]: pgmap v146: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T08:08:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3519455633' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:08:37.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:37.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:37.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:37.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:37.161 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:37.162 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:37.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:37.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:37.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:37.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:37.382 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:37.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:37.440 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:37.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:37.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:37.586 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:37.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:37.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:37.740 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:37.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:37.842 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:37.845 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[30086]: 7.e scrub starts 2024-02-05T08:08:37.845 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[30086]: 6.16 scrub starts 2024-02-05T08:08:37.845 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[30086]: 6.16 scrub ok 2024-02-05T08:08:37.845 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:08:37.845 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:08:37.845 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3519455633' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:08:37.846 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[30086]: osdmap e129: 8 total, 8 up, 7 in 2024-02-05T08:08:37.846 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:37.846 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:08:37.846 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:08:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:08:37.846 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[39278]: 7.e scrub starts 2024-02-05T08:08:37.846 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[39278]: 6.16 scrub starts 2024-02-05T08:08:37.846 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[39278]: 6.16 scrub ok 2024-02-05T08:08:37.847 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:08:37.847 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:08:37.847 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3519455633' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:08:37.847 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[39278]: osdmap e129: 8 total, 8 up, 7 in 2024-02-05T08:08:37.847 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:37.847 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:37 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:08:37.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:37.895 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:37.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:37.996 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:38.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:38.045 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:38.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:37 smithi196 ceph-mon[33018]: 7.e scrub starts 2024-02-05T08:08:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:37 smithi196 ceph-mon[33018]: 6.16 scrub starts 2024-02-05T08:08:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:37 smithi196 ceph-mon[33018]: 6.16 scrub ok 2024-02-05T08:08:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:37 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:08:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:37 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:08:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:37 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3519455633' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:08:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:37 smithi196 ceph-mon[33018]: osdmap e129: 8 total, 8 up, 7 in 2024-02-05T08:08:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:37 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:37 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:08:38.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:38.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:38.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:38.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:38.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:38.314 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:38.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:38.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:38.416 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:38.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:38.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:38.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:38.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:38.585 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:38.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:38.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:38.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:38.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:38.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:38.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:38.853 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:38.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:38.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:38.880 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[30086]: 4.14 deep-scrub starts 2024-02-05T08:08:38.880 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[30086]: 4.14 deep-scrub ok 2024-02-05T08:08:38.880 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[30086]: pgmap v148: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 669 op/s 2024-02-05T08:08:38.880 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[30086]: 4.7 scrub starts 2024-02-05T08:08:38.880 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[30086]: 4.7 scrub ok 2024-02-05T08:08:38.881 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[30086]: 4.1a scrub starts 2024-02-05T08:08:38.881 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[30086]: 4.1a scrub ok 2024-02-05T08:08:38.881 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[30086]: 5.4 scrub starts 2024-02-05T08:08:38.881 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[30086]: 5.4 scrub ok 2024-02-05T08:08:38.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[39278]: 4.14 deep-scrub starts 2024-02-05T08:08:38.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[39278]: 4.14 deep-scrub ok 2024-02-05T08:08:38.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[39278]: pgmap v148: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 669 op/s 2024-02-05T08:08:38.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[39278]: 4.7 scrub starts 2024-02-05T08:08:38.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[39278]: 4.7 scrub ok 2024-02-05T08:08:38.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[39278]: 4.1a scrub starts 2024-02-05T08:08:38.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[39278]: 4.1a scrub ok 2024-02-05T08:08:38.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[39278]: 5.4 scrub starts 2024-02-05T08:08:38.882 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:38 smithi134 ceph-mon[39278]: 5.4 scrub ok 2024-02-05T08:08:38.885 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:08:38.885 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:08:38.885 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.885 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:128188 7.a 7:552605fa:::benchmark_data_smithi134_83016_object18254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-05T08:08:38.885 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:38.869694+0000", 2024-02-05T08:08:38.885 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.014202019, 2024-02-05T08:08:38.885 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.014238271, 2024-02-05T08:08:38.885 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 128188 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.869694+0000", 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.886 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.869694+0000", 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.869696+0000", 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.389e-06 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.869701+0000", 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7500000000000003e-06 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.887 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.888 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:38.888 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.869711+0000", 2024-02-05T08:08:38.888 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.083e-05 2024-02-05T08:08:38.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.888 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.888 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:38.888 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.869719+0000", 2024-02-05T08:08:38.888 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8290000000000008e-06 2024-02-05T08:08:38.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.888 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.889 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:38.889 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.872787+0000", 2024-02-05T08:08:38.889 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030681760000000001 2024-02-05T08:08:38.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.889 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.889 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:38.889 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.872890+0000", 2024-02-05T08:08:38.889 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000102989 2024-02-05T08:08:38.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.889 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.889 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-02-05T08:08:38.889 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.872995+0000", 2024-02-05T08:08:38.889 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104319 2024-02-05T08:08:38.890 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.890 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.890 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:38.890 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.881933+0000", 2024-02-05T08:08:38.890 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0089386650000000002 2024-02-05T08:08:38.890 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.890 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:38.890 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.890 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.890 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.890 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:128189 7.a 7:5b89c146:::benchmark_data_smithi134_83016_object18255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:38.869738+0000", 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.014158178, 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.014288168, 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 128189 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:38.891 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.869738+0000", 2024-02-05T08:08:38.892 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.892 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.892 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.892 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:38.892 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.869738+0000", 2024-02-05T08:08:38.892 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.892 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.892 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.892 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:38.892 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.869739+0000", 2024-02-05T08:08:38.892 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9100000000000004e-07 2024-02-05T08:08:38.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.893 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.893 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:38.893 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.869742+0000", 2024-02-05T08:08:38.893 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7669999999999999e-06 2024-02-05T08:08:38.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.893 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.893 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:38.893 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.869749+0000", 2024-02-05T08:08:38.893 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2579999999999998e-06 2024-02-05T08:08:38.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.869755+0000", 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9900000000000002e-06 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.877615+0000", 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0078600110000000001 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.877725+0000", 2024-02-05T08:08:38.894 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011004499999999999 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.877835+0000", 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109751 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.881976+0000", 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0041409970000000004 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:128194 7.2 7:4eae0484:::benchmark_data_smithi134_83016_object18260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:38.870080+0000", 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.013816874, 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.014029956999999999, 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 128194 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.896 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.870080+0000", 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.870080+0000", 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.870080+0000", 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7400000000000002e-07 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.897 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.870085+0000", 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4360000000000004e-06 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.870100+0000", 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4637000000000001e-05 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.870108+0000", 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9980000000000003e-06 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.898 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:38.899 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.879920+0000", 2024-02-05T08:08:38.899 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0098128810000000007 2024-02-05T08:08:38.899 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.899 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.899 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:38.899 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.880021+0000", 2024-02-05T08:08:38.899 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000100647 2024-02-05T08:08:38.899 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.899 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.899 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:08:38.899 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.880119+0000", 2024-02-05T08:08:38.900 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7837000000000005e-05 2024-02-05T08:08:38.900 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.900 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:38.900 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.900 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.900 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.900 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:128190 7.d e129/97 7:bd604448:::benchmark_data_smithi134_83016_object18256:head v 129'8118, mlcod=129'8116)", 2024-02-05T08:08:38.900 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:38.874882+0000", 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0090147590000000007, 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0093220400000000002, 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.874882+0000", 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.874882+0000", 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.874882+0000", 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0800000000000004e-07 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.874888+0000", 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7699999999999998e-06 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.874896+0000", 2024-02-05T08:08:38.902 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5700000000000004e-06 2024-02-05T08:08:38.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.903 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.903 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:38.903 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.874901+0000", 2024-02-05T08:08:38.903 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0699999999999997e-06 2024-02-05T08:08:38.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.903 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.903 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.878224+0000", 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003323257 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.878243+0000", 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9318e-05 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:128191 7.d e129/97)", 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:38.878157+0000", 2024-02-05T08:08:38.904 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0057392750000000003, 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0061103570000000003, 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.878157+0000", 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.878157+0000", 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.905 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.878158+0000", 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1399999999999996e-07 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.878170+0000", 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2285e-05 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.878185+0000", 2024-02-05T08:08:38.906 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4823e-05 2024-02-05T08:08:38.907 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.907 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.907 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:38.907 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.878192+0000", 2024-02-05T08:08:38.907 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0260000000000002e-06 2024-02-05T08:08:38.907 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.907 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:38.907 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.907 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.907 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.907 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:128197 7.7 e129/97 7:e65d70b4:::benchmark_data_smithi134_83016_object18263:head v 129'7973, mlcod=129'7970)", 2024-02-05T08:08:38.908 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:38.879091+0000", 2024-02-05T08:08:38.908 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0048049809999999998, 2024-02-05T08:08:38.908 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0052333299999999996, 2024-02-05T08:08:38.908 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:38.908 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:08:38.908 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:38.908 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.908 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:38.908 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.879091+0000", 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.879091+0000", 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.879092+0000", 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7400000000000002e-07 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:38.909 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.879098+0000", 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7119999999999997e-06 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.879112+0000", 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3542e-05 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.879118+0000", 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5760000000000002e-06 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.910 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.882085+0000", 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0029667489999999999 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.882104+0000", 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9096000000000001e-05 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:128199 7.a 7:53dc5f99:::benchmark_data_smithi134_83016_object18265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:38.881950+0000", 2024-02-05T08:08:38.911 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0019462679999999999, 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002442928, 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 128199 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.881950+0000", 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.912 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.913 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:38.913 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.881950+0000", 2024-02-05T08:08:38.913 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.913 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.913 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.913 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:38.913 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.881952+0000", 2024-02-05T08:08:38.913 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.341e-06 2024-02-05T08:08:38.913 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.913 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.914 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:38.914 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.881956+0000", 2024-02-05T08:08:38.914 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4129999999999999e-06 2024-02-05T08:08:38.914 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.914 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.914 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:38.914 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.881967+0000", 2024-02-05T08:08:38.914 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.133e-05 2024-02-05T08:08:38.914 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.914 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.914 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:38.914 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.881976+0000", 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8480000000000007e-06 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.882068+0000", 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2120000000000006e-05 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.882193+0000", 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012473399999999999 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.915 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.882302+0000", 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109249 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:128200 7.1 e129/97 7:8b333a83:::benchmark_data_smithi134_83016_object18266:head v 129'8050, mlcod=129'8048)", 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:38.882008+0000", 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0018889379999999999, 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0024695960000000001, 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:08:38.916 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:38.917 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.917 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:38.917 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.882008+0000", 2024-02-05T08:08:38.917 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.917 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.917 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.917 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:38.917 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.882008+0000", 2024-02-05T08:08:38.917 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.917 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.917 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.917 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:38.917 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.882008+0000", 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1200000000000003e-07 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.882013+0000", 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5070000000000002e-06 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.882020+0000", 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7710000000000006e-06 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:38.918 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.882026+0000", 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6690000000000003e-06 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.882489+0000", 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00046268200000000001 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.882507+0000", 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8236999999999998e-05 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.919 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:128199 7.a e129/95)", 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:38.884086+0000", 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4294967295.9998107, 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00044872400000000001, 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.884086+0000", 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.920 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.884086+0000", 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.884087+0000", 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1399999999999995e-07 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.884091+0000", 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6709999999999998e-06 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.921 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.884101+0000", 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9089999999999995e-06 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.884108+0000", 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8410000000000002e-06 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:128176 7.7 e129/97 7:e9ed06a7:::benchmark_data_smithi134_83016_object18242:head v 129'7972, mlcod=129'7966)", 2024-02-05T08:08:38.922 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:38.863417+0000", 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.020479079000000001, 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.021165329, 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.863417+0000", 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.863417+0000", 2024-02-05T08:08:38.923 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:08:38.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.924 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.924 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:08:38.924 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.863418+0000", 2024-02-05T08:08:38.924 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0999999999999999e-07 2024-02-05T08:08:38.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.924 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.924 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:08:38.924 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.863422+0000", 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1269999999999996e-06 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.863429+0000", 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7780000000000002e-06 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.863434+0000", 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3650000000000003e-06 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.925 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.867317+0000", 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003882801 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:38.867337+0000", 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0234e-05 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 10 2024-02-05T08:08:38.926 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:08:38.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:38.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:38.998 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:38.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:38 smithi196 ceph-mon[33018]: 4.14 deep-scrub starts 2024-02-05T08:08:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:38 smithi196 ceph-mon[33018]: 4.14 deep-scrub ok 2024-02-05T08:08:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:38 smithi196 ceph-mon[33018]: pgmap v148: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 669 op/s 2024-02-05T08:08:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:38 smithi196 ceph-mon[33018]: 4.7 scrub starts 2024-02-05T08:08:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:38 smithi196 ceph-mon[33018]: 4.7 scrub ok 2024-02-05T08:08:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:38 smithi196 ceph-mon[33018]: 4.1a scrub starts 2024-02-05T08:08:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:38 smithi196 ceph-mon[33018]: 4.1a scrub ok 2024-02-05T08:08:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:38 smithi196 ceph-mon[33018]: 5.4 scrub starts 2024-02-05T08:08:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:38 smithi196 ceph-mon[33018]: 5.4 scrub ok 2024-02-05T08:08:39.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:39.125 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:39.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:39.227 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:39.271 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:39.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:39.372 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:39.419 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:39.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:39.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:39.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:39.546 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:39.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:39.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:39.644 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:08:39.648 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:39.711 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:39.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:39.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:39.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:39.856 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:39.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:39.953 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:39 smithi134 ceph-mon[39278]: 3.1 scrub starts 2024-02-05T08:08:39.954 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:39 smithi134 ceph-mon[39278]: 3.1 scrub ok 2024-02-05T08:08:39.954 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:39 smithi134 ceph-mon[39278]: 6.5 scrub starts 2024-02-05T08:08:39.954 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:39 smithi134 ceph-mon[39278]: 6.5 scrub ok 2024-02-05T08:08:39.954 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:39 smithi134 ceph-mon[30086]: 3.1 scrub starts 2024-02-05T08:08:39.954 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:39 smithi134 ceph-mon[30086]: 3.1 scrub ok 2024-02-05T08:08:39.954 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:39 smithi134 ceph-mon[30086]: 6.5 scrub starts 2024-02-05T08:08:39.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:39 smithi134 ceph-mon[30086]: 6.5 scrub ok 2024-02-05T08:08:39.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:40.001 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:40.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:39 smithi196 ceph-mon[33018]: 3.1 scrub starts 2024-02-05T08:08:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:39 smithi196 ceph-mon[33018]: 3.1 scrub ok 2024-02-05T08:08:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:39 smithi196 ceph-mon[33018]: 6.5 scrub starts 2024-02-05T08:08:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:39 smithi196 ceph-mon[33018]: 6.5 scrub ok 2024-02-05T08:08:40.101 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:40.143 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:40.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:40.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:40.245 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:40.271 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:40.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:40.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:40.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:40.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:40.417 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:40.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:40.518 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:40.560 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:40.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:40.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:40.700 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:40.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:40.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:40.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:40.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:40.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:40.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:40.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:40.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:40.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:40 smithi134 ceph-mon[30086]: 7.e scrub ok 2024-02-05T08:08:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:40 smithi134 ceph-mon[30086]: 3.2 deep-scrub starts 2024-02-05T08:08:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:40 smithi134 ceph-mon[30086]: 3.2 deep-scrub ok 2024-02-05T08:08:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:40 smithi134 ceph-mon[30086]: pgmap v149: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 989 op/s 2024-02-05T08:08:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:40 smithi134 ceph-mon[30086]: 5.d deep-scrub starts 2024-02-05T08:08:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:40 smithi134 ceph-mon[30086]: 5.d deep-scrub ok 2024-02-05T08:08:41.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:40 smithi134 ceph-mon[39278]: 7.e scrub ok 2024-02-05T08:08:41.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:40 smithi134 ceph-mon[39278]: 3.2 deep-scrub starts 2024-02-05T08:08:41.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:40 smithi134 ceph-mon[39278]: 3.2 deep-scrub ok 2024-02-05T08:08:41.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:40 smithi134 ceph-mon[39278]: pgmap v149: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 989 op/s 2024-02-05T08:08:41.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:40 smithi134 ceph-mon[39278]: 5.d deep-scrub starts 2024-02-05T08:08:41.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:40 smithi134 ceph-mon[39278]: 5.d deep-scrub ok 2024-02-05T08:08:41.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:40 smithi196 ceph-mon[33018]: 7.e scrub ok 2024-02-05T08:08:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:40 smithi196 ceph-mon[33018]: 3.2 deep-scrub starts 2024-02-05T08:08:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:40 smithi196 ceph-mon[33018]: 3.2 deep-scrub ok 2024-02-05T08:08:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:40 smithi196 ceph-mon[33018]: pgmap v149: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 989 op/s 2024-02-05T08:08:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:40 smithi196 ceph-mon[33018]: 5.d deep-scrub starts 2024-02-05T08:08:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:40 smithi196 ceph-mon[33018]: 5.d deep-scrub ok 2024-02-05T08:08:41.085 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:41.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:41.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:41.213 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:41.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:41.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:41.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:41.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:41.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:41.446 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:41.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:41.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:41.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:41.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:41.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:41.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:41.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:41.729 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:41.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:41.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:41.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:41.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:41.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:41.872 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[30086]: 4.10 scrub starts 2024-02-05T08:08:41.872 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[30086]: 4.10 scrub ok 2024-02-05T08:08:41.872 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[30086]: 7.0 deep-scrub starts 2024-02-05T08:08:41.872 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[30086]: 4.1c scrub starts 2024-02-05T08:08:41.872 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[30086]: 4.1c scrub ok 2024-02-05T08:08:41.872 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[30086]: 3.0 scrub starts 2024-02-05T08:08:41.872 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[30086]: 3.0 scrub ok 2024-02-05T08:08:41.872 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[39278]: 4.10 scrub starts 2024-02-05T08:08:41.873 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[39278]: 4.10 scrub ok 2024-02-05T08:08:41.873 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[39278]: 7.0 deep-scrub starts 2024-02-05T08:08:41.873 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[39278]: 4.1c scrub starts 2024-02-05T08:08:41.873 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[39278]: 4.1c scrub ok 2024-02-05T08:08:41.873 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[39278]: 3.0 scrub starts 2024-02-05T08:08:41.873 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:41 smithi134 ceph-mon[39278]: 3.0 scrub ok 2024-02-05T08:08:41.918 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T08:08:41.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:42.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:42.045 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:42.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:41 smithi196 ceph-mon[33018]: 4.10 scrub starts 2024-02-05T08:08:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:41 smithi196 ceph-mon[33018]: 4.10 scrub ok 2024-02-05T08:08:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:41 smithi196 ceph-mon[33018]: 7.0 deep-scrub starts 2024-02-05T08:08:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:41 smithi196 ceph-mon[33018]: 4.1c scrub starts 2024-02-05T08:08:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:41 smithi196 ceph-mon[33018]: 4.1c scrub ok 2024-02-05T08:08:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:41 smithi196 ceph-mon[33018]: 3.0 scrub starts 2024-02-05T08:08:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:41 smithi196 ceph-mon[33018]: 3.0 scrub ok 2024-02-05T08:08:42.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:42.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:42.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:42.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:42.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:42.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:42.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:42.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:42.403 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:42.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:42.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:42.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:42.598 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:42.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:42.676 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:08:42.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:42.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:42.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:42.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:42.725 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:08:42.769 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:08:42.813 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:08:42.826 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:42.860 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:08:42.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:42.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:42.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:42.902 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:08:42.947 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:08:42.948 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[30086]: 6.a scrub starts 2024-02-05T08:08:42.948 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[30086]: 6.a scrub ok 2024-02-05T08:08:42.948 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[30086]: 7.5 scrub starts 2024-02-05T08:08:42.948 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[30086]: pgmap v150: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 990 op/s 2024-02-05T08:08:42.948 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:08:42.949 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:08:42.949 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1027251630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:08:42.949 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[39278]: 6.a scrub starts 2024-02-05T08:08:42.949 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[39278]: 6.a scrub ok 2024-02-05T08:08:42.949 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[39278]: 7.5 scrub starts 2024-02-05T08:08:42.949 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[39278]: pgmap v150: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 990 op/s 2024-02-05T08:08:42.949 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:08:42.950 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:08:42.950 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:42 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1027251630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:08:42.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:42.994 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:08:42.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:42.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:43.004 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:08:43.004 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:08:43.004 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:08:43.004 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:08:43.005 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:08:43.005 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:08:43.005 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:08:43.005 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:08:43.012 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:43.018 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:42 smithi196 ceph-mon[33018]: 6.a scrub starts 2024-02-05T08:08:43.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:42 smithi196 ceph-mon[33018]: 6.a scrub ok 2024-02-05T08:08:43.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:42 smithi196 ceph-mon[33018]: 7.5 scrub starts 2024-02-05T08:08:43.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:42 smithi196 ceph-mon[33018]: pgmap v150: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 990 op/s 2024-02-05T08:08:43.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:42 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:08:43.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:42 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:08:43.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:42 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1027251630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:08:43.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:43.154 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:43.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:43.255 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:43.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:43.300 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:43.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:43.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:08:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:08:43.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:43.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:43.447 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:43.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:43.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:43.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:43.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:43.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:43.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:43.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:43.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:43.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:43.770 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.0 to 1 (10000) 2024-02-05T08:08:43.813 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:43.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:43.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:43.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:43.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:43.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:44.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[30086]: 7.3 scrub starts 2024-02-05T08:08:44.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[30086]: 7.5 scrub ok 2024-02-05T08:08:44.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[30086]: 5.1e scrub starts 2024-02-05T08:08:44.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[30086]: 5.1e scrub ok 2024-02-05T08:08:44.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:44.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[30086]: 6.3 scrub starts 2024-02-05T08:08:44.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[30086]: 6.3 scrub ok 2024-02-05T08:08:44.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1492551504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:08:44.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:08:44.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[39278]: 7.3 scrub starts 2024-02-05T08:08:44.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[39278]: 7.5 scrub ok 2024-02-05T08:08:44.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[39278]: 5.1e scrub starts 2024-02-05T08:08:44.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[39278]: 5.1e scrub ok 2024-02-05T08:08:44.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:44.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[39278]: 6.3 scrub starts 2024-02-05T08:08:44.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[39278]: 6.3 scrub ok 2024-02-05T08:08:44.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1492551504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:08:44.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:43 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:08:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:43 smithi196 ceph-mon[33018]: 7.3 scrub starts 2024-02-05T08:08:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:43 smithi196 ceph-mon[33018]: 7.5 scrub ok 2024-02-05T08:08:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:43 smithi196 ceph-mon[33018]: 5.1e scrub starts 2024-02-05T08:08:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:43 smithi196 ceph-mon[33018]: 5.1e scrub ok 2024-02-05T08:08:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:43 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:43 smithi196 ceph-mon[33018]: 6.3 scrub starts 2024-02-05T08:08:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:43 smithi196 ceph-mon[33018]: 6.3 scrub ok 2024-02-05T08:08:44.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:43 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1492551504' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:08:44.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:43 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:08:44.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:44.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:44.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:44.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:44.203 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:44.241 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:44.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:44.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:44.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:44.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:44.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:44.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:44.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:44.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:44.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:44.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:44.625 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:44.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:44.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:44.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:44.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:44.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:44.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:44.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:44.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:44.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:45.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:45.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[30086]: 6.d scrub starts 2024-02-05T08:08:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[30086]: 6.d scrub ok 2024-02-05T08:08:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[30086]: pgmap v151: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 948 op/s 2024-02-05T08:08:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:08:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:08:45.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:08:45.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[30086]: osdmap e130: 8 total, 8 up, 7 in 2024-02-05T08:08:45.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[30086]: 4.12 scrub starts 2024-02-05T08:08:45.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[30086]: 4.12 scrub ok 2024-02-05T08:08:45.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:08:45.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:08:45.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[39278]: 6.d scrub starts 2024-02-05T08:08:45.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[39278]: 6.d scrub ok 2024-02-05T08:08:45.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[39278]: pgmap v151: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 948 op/s 2024-02-05T08:08:45.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:08:45.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:08:45.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:08:45.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[39278]: osdmap e130: 8 total, 8 up, 7 in 2024-02-05T08:08:45.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[39278]: 4.12 scrub starts 2024-02-05T08:08:45.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[39278]: 4.12 scrub ok 2024-02-05T08:08:45.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:08:45.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:44 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:08:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:44 smithi196 ceph-mon[33018]: 6.d scrub starts 2024-02-05T08:08:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:44 smithi196 ceph-mon[33018]: 6.d scrub ok 2024-02-05T08:08:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:44 smithi196 ceph-mon[33018]: pgmap v151: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 948 op/s 2024-02-05T08:08:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:44 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:08:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:44 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:08:45.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:44 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:08:45.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:44 smithi196 ceph-mon[33018]: osdmap e130: 8 total, 8 up, 7 in 2024-02-05T08:08:45.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:44 smithi196 ceph-mon[33018]: 4.12 scrub starts 2024-02-05T08:08:45.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:44 smithi196 ceph-mon[33018]: 4.12 scrub ok 2024-02-05T08:08:45.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:44 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:08:45.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:44 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:08:45.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:45.156 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:45.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:45.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:45.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:45.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:45.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:45.433 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:45.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:45.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:45.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:45.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:45.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:45.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:45.701 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:45.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:45.802 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:45.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:45.841 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:45.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:45.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:45.967 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:45.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[30086]: 7.3 scrub ok 2024-02-05T08:08:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[30086]: 3.12 scrub starts 2024-02-05T08:08:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[30086]: 3.12 scrub ok 2024-02-05T08:08:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[30086]: 2.1 scrub starts 2024-02-05T08:08:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[30086]: 2.1 scrub ok 2024-02-05T08:08:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[30086]: 3.4 scrub starts 2024-02-05T08:08:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[30086]: 3.4 scrub ok 2024-02-05T08:08:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[30086]: 7.1 scrub starts 2024-02-05T08:08:46.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[39278]: 7.3 scrub ok 2024-02-05T08:08:46.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[39278]: 3.12 scrub starts 2024-02-05T08:08:46.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[39278]: 3.12 scrub ok 2024-02-05T08:08:46.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[39278]: 2.1 scrub starts 2024-02-05T08:08:46.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[39278]: 2.1 scrub ok 2024-02-05T08:08:46.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[39278]: 3.4 scrub starts 2024-02-05T08:08:46.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[39278]: 3.4 scrub ok 2024-02-05T08:08:46.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:45 smithi134 ceph-mon[39278]: 7.1 scrub starts 2024-02-05T08:08:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:45 smithi196 ceph-mon[33018]: 7.3 scrub ok 2024-02-05T08:08:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:45 smithi196 ceph-mon[33018]: 3.12 scrub starts 2024-02-05T08:08:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:45 smithi196 ceph-mon[33018]: 3.12 scrub ok 2024-02-05T08:08:46.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:45 smithi196 ceph-mon[33018]: 2.1 scrub starts 2024-02-05T08:08:46.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:45 smithi196 ceph-mon[33018]: 2.1 scrub ok 2024-02-05T08:08:46.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:45 smithi196 ceph-mon[33018]: 3.4 scrub starts 2024-02-05T08:08:46.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:45 smithi196 ceph-mon[33018]: 3.4 scrub ok 2024-02-05T08:08:46.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:45 smithi196 ceph-mon[33018]: 7.1 scrub starts 2024-02-05T08:08:46.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:46.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:46.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:46.198 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:46.249 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:46.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:46.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:46.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:46.397 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:46.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:46.498 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:46.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:46.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:46.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:46.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:46.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:46.668 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:46.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:46.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:46.796 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:46.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:46.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:46.924 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:46.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:47.025 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:47.062 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:47.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:47.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:46 smithi196 ceph-mon[33018]: 7.a scrub starts 2024-02-05T08:08:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:46 smithi196 ceph-mon[33018]: 3.16 scrub starts 2024-02-05T08:08:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:46 smithi196 ceph-mon[33018]: 3.16 scrub ok 2024-02-05T08:08:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:46 smithi196 ceph-mon[33018]: pgmap v153: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-02-05T08:08:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:46 smithi196 ceph-mon[33018]: 4.16 scrub starts 2024-02-05T08:08:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:46 smithi196 ceph-mon[33018]: 4.16 scrub ok 2024-02-05T08:08:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:46 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:08:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:46 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:08:47.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:47.200 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:47.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:47.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:47.247 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[30086]: 7.a scrub starts 2024-02-05T08:08:47.247 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[30086]: 3.16 scrub starts 2024-02-05T08:08:47.247 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[30086]: 3.16 scrub ok 2024-02-05T08:08:47.247 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[30086]: pgmap v153: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-02-05T08:08:47.247 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[30086]: 4.16 scrub starts 2024-02-05T08:08:47.247 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[30086]: 4.16 scrub ok 2024-02-05T08:08:47.247 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:08:47.247 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:08:47.248 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[39278]: 7.a scrub starts 2024-02-05T08:08:47.248 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[39278]: 3.16 scrub starts 2024-02-05T08:08:47.248 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[39278]: 3.16 scrub ok 2024-02-05T08:08:47.248 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[39278]: pgmap v153: 148 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 145 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-02-05T08:08:47.248 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[39278]: 4.16 scrub starts 2024-02-05T08:08:47.248 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[39278]: 4.16 scrub ok 2024-02-05T08:08:47.248 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:08:47.248 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:46 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:08:47.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:47.350 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:47.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:47.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:47.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:47.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:47.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:47.594 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:47.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:47.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:47.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:47.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:47.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:47.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:47.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:47.835 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:47.875 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:47.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:47.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:48.018 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:48.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:48.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:48.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:47 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:08:48.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:47 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:08:48.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:47 smithi134 ceph-mon[30086]: 7.1 scrub ok 2024-02-05T08:08:48.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:47 smithi134 ceph-mon[30086]: osdmap e131: 8 total, 8 up, 7 in 2024-02-05T08:08:48.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:47 smithi134 ceph-mon[30086]: 5.18 deep-scrub starts 2024-02-05T08:08:48.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:47 smithi134 ceph-mon[30086]: 5.18 deep-scrub ok 2024-02-05T08:08:48.022 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:08:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:08:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:08:48.022 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:47 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:08:48.022 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:47 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:08:48.023 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:47 smithi134 ceph-mon[39278]: 7.1 scrub ok 2024-02-05T08:08:48.023 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:47 smithi134 ceph-mon[39278]: osdmap e131: 8 total, 8 up, 7 in 2024-02-05T08:08:48.023 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:47 smithi134 ceph-mon[39278]: 5.18 deep-scrub starts 2024-02-05T08:08:48.023 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:47 smithi134 ceph-mon[39278]: 5.18 deep-scrub ok 2024-02-05T08:08:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:47 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:08:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:47 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:08:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:47 smithi196 ceph-mon[33018]: 7.1 scrub ok 2024-02-05T08:08:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:47 smithi196 ceph-mon[33018]: osdmap e131: 8 total, 8 up, 7 in 2024-02-05T08:08:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:47 smithi196 ceph-mon[33018]: 5.18 deep-scrub starts 2024-02-05T08:08:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:47 smithi196 ceph-mon[33018]: 5.18 deep-scrub ok 2024-02-05T08:08:48.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 1 1 2024-02-05T08:08:48.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:48.199 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:48.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:48.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:48.247 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:08:48.302 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:48.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:48.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:48.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:48.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:48.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:48.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:48.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:48.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:48.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:48.655 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:48.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:48.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:48.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:48.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:48.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:48.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:48.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:48.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:48.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:48.978 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:48 smithi134 ceph-mon[30086]: pgmap v155: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 660 op/s 2024-02-05T08:08:48.978 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:48 smithi134 ceph-mon[30086]: 7.a scrub ok 2024-02-05T08:08:48.978 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:48 smithi134 ceph-mon[30086]: 6.1c deep-scrub starts 2024-02-05T08:08:48.978 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:48 smithi134 ceph-mon[30086]: 6.1c deep-scrub ok 2024-02-05T08:08:48.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:48 smithi134 ceph-mon[39278]: pgmap v155: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 660 op/s 2024-02-05T08:08:48.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:48 smithi134 ceph-mon[39278]: 7.a scrub ok 2024-02-05T08:08:48.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:48 smithi134 ceph-mon[39278]: 6.1c deep-scrub starts 2024-02-05T08:08:48.979 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:48 smithi134 ceph-mon[39278]: 6.1c deep-scrub ok 2024-02-05T08:08:49.008 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:49.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:49.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:49.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:48 smithi196 ceph-mon[33018]: pgmap v155: 148 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 146 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 660 op/s 2024-02-05T08:08:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:48 smithi196 ceph-mon[33018]: 7.a scrub ok 2024-02-05T08:08:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:48 smithi196 ceph-mon[33018]: 6.1c deep-scrub starts 2024-02-05T08:08:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:48 smithi196 ceph-mon[33018]: 6.1c deep-scrub ok 2024-02-05T08:08:49.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:49.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:49.202 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:49.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:49.303 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:49.345 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:49.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:49.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:49.446 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:49.484 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:49.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:49.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:49.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:49.630 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:49.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:49.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:49.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:49.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:49.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:49.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:49.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:49.927 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:49.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:49 smithi134 ceph-mon[30086]: 7.f scrub starts 2024-02-05T08:08:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:49 smithi134 ceph-mon[30086]: 1.0 scrub starts 2024-02-05T08:08:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:49 smithi134 ceph-mon[30086]: 1.0 scrub ok 2024-02-05T08:08:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:49 smithi134 ceph-mon[30086]: 6.e deep-scrub starts 2024-02-05T08:08:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:49 smithi134 ceph-mon[30086]: 6.e deep-scrub ok 2024-02-05T08:08:50.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:49 smithi134 ceph-mon[39278]: 7.f scrub starts 2024-02-05T08:08:50.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:49 smithi134 ceph-mon[39278]: 1.0 scrub starts 2024-02-05T08:08:50.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:49 smithi134 ceph-mon[39278]: 1.0 scrub ok 2024-02-05T08:08:50.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:49 smithi134 ceph-mon[39278]: 6.e deep-scrub starts 2024-02-05T08:08:50.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:49 smithi134 ceph-mon[39278]: 6.e deep-scrub ok 2024-02-05T08:08:50.028 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:50.053 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:50.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:50.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:50.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:50.195 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:50.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:50.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:49 smithi196 ceph-mon[33018]: 7.f scrub starts 2024-02-05T08:08:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:49 smithi196 ceph-mon[33018]: 1.0 scrub starts 2024-02-05T08:08:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:49 smithi196 ceph-mon[33018]: 1.0 scrub ok 2024-02-05T08:08:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:49 smithi196 ceph-mon[33018]: 6.e deep-scrub starts 2024-02-05T08:08:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:49 smithi196 ceph-mon[33018]: 6.e deep-scrub ok 2024-02-05T08:08:50.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:50.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:50.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:50.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:50.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:50.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:50.550 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:50.576 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:50.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:50.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:50.596 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:50.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:50.702 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:50.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:50.804 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:50.831 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:50.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:50.866 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:08:50.933 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:50.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:50.972 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:50.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:51.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:51.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:51.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:51.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:51.229 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[30086]: 7.c scrub starts 2024-02-05T08:08:51.229 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[30086]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-02-05T08:08:51.230 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[30086]: 5.19 scrub starts 2024-02-05T08:08:51.230 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[30086]: 5.19 scrub ok 2024-02-05T08:08:51.230 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[30086]: 7.2 scrub starts 2024-02-05T08:08:51.230 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/610130362' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:08:51.230 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[30086]: 7.f scrub ok 2024-02-05T08:08:51.230 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:08:51.230 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[39278]: 7.c scrub starts 2024-02-05T08:08:51.230 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[39278]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-02-05T08:08:51.230 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[39278]: 5.19 scrub starts 2024-02-05T08:08:51.230 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[39278]: 5.19 scrub ok 2024-02-05T08:08:51.230 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[39278]: 7.2 scrub starts 2024-02-05T08:08:51.230 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/610130362' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:08:51.231 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[39278]: 7.f scrub ok 2024-02-05T08:08:51.231 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:50 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:08:51.237 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:51.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:51.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:50 smithi196 ceph-mon[33018]: 7.c scrub starts 2024-02-05T08:08:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:50 smithi196 ceph-mon[33018]: pgmap v156: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-02-05T08:08:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:50 smithi196 ceph-mon[33018]: 5.19 scrub starts 2024-02-05T08:08:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:50 smithi196 ceph-mon[33018]: 5.19 scrub ok 2024-02-05T08:08:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:50 smithi196 ceph-mon[33018]: 7.2 scrub starts 2024-02-05T08:08:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:50 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/610130362' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:08:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:50 smithi196 ceph-mon[33018]: 7.f scrub ok 2024-02-05T08:08:51.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:50 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:08:51.339 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:51.374 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:51.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:51.476 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:51.518 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:51.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:51.619 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:51.644 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:51.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:51.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:51.746 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:51.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:51.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:51.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:51.872 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:51.913 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:51.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:52.014 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:52.050 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:52.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:52.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:52.178 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:52.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:51 smithi134 ceph-mon[30086]: 7.9 scrub starts 2024-02-05T08:08:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:51 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:08:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:51 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:08:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:51 smithi134 ceph-mon[30086]: osdmap e132: 8 total, 8 up, 7 in 2024-02-05T08:08:52.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:51 smithi134 ceph-mon[39278]: 7.9 scrub starts 2024-02-05T08:08:52.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:51 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:08:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:51 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:08:52.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:51 smithi134 ceph-mon[39278]: osdmap e132: 8 total, 8 up, 7 in 2024-02-05T08:08:52.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:51 smithi196 ceph-mon[33018]: 7.9 scrub starts 2024-02-05T08:08:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:51 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:08:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:51 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:08:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:51 smithi196 ceph-mon[33018]: osdmap e132: 8 total, 8 up, 7 in 2024-02-05T08:08:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:52.315 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:52.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:52.416 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:52.456 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:52.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:52.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:52.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:52.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:52.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:52.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:52.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:52.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:52.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:52.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:52.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:52.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:52.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:53.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:53.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:53.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:52 smithi134 ceph-mon[30086]: pgmap v158: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 656 B/s rd, 0 B/s wr, 479 op/s 2024-02-05T08:08:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:52 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:52 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:08:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:52 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:08:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:52 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:08:53.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:52 smithi134 ceph-mon[39278]: pgmap v158: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 656 B/s rd, 0 B/s wr, 479 op/s 2024-02-05T08:08:53.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:52 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:53.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:52 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:08:53.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:52 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:08:53.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:52 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:08:53.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:53.131 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:53.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:53.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:53.233 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:53.270 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:53.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:53.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:53.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:08:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:08:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:52 smithi196 ceph-mon[33018]: pgmap v158: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 656 B/s rd, 0 B/s wr, 479 op/s 2024-02-05T08:08:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:52 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:08:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:52 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:08:53.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:52 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:08:53.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:52 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:08:53.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:53.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:53.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:53.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:53.499 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:53.536 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:53.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:53.540 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:08:53.638 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:53.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:53.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:53.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:53.834 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:53.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:53.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:53.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:53.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:53.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:54.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:54.115 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:54.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:54.193 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:53 smithi134 ceph-mon[30086]: osdmap e133: 8 total, 8 up, 7 in 2024-02-05T08:08:54.193 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:53 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:54.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:53 smithi134 ceph-mon[39278]: osdmap e133: 8 total, 8 up, 7 in 2024-02-05T08:08:54.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:53 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:54.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:54.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:54.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:54.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:53 smithi196 ceph-mon[33018]: osdmap e133: 8 total, 8 up, 7 in 2024-02-05T08:08:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:53 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:08:54.343 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:54.386 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:54.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:54.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:54.488 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:54.527 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:54.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:54.629 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:54.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:54.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:54.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:54.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:54.807 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:54.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:54.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:54.953 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:54.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:55.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:54 smithi134 ceph-mon[30086]: pgmap v160: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T08:08:55.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:54 smithi134 ceph-mon[39278]: pgmap v160: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T08:08:55.033 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:08:55.033 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:08:55.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:55.078 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:55.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:55.180 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:55.220 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:55.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:54 smithi196 ceph-mon[33018]: pgmap v160: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T08:08:55.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:55.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:55.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:55.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:55.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:55.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:55.486 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:55.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:55.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:55.613 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:55.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:55.715 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:55.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:55.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:55.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:55.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:55.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:55.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:55.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:55.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:55.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:56.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:56.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:08:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:56.129 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:56.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:56.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:56.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:56.279 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:56.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:56.380 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:56.423 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:56.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:56.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:56.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:56.550 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:56.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:56.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:56.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:56.699 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:56.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:56.801 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:56.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:56.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:56.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:56.986 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:56.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:56.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:57.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:57.126 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:57.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:57.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:57.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:57.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:57.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:57.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:57.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:56 smithi134 ceph-mon[30086]: pgmap v161: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 881 op/s 2024-02-05T08:08:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:56 smithi134 ceph-mon[39278]: pgmap v161: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 881 op/s 2024-02-05T08:08:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:56 smithi196 ceph-mon[33018]: pgmap v161: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 881 op/s 2024-02-05T08:08:57.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:57.380 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:57.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:57.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:57.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:08:57.507 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T08:08:57.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:57.508 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:57.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:08:57.609 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:57.660 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:57.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:57.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:57.762 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:08:57.810 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:57.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:57.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:08:57.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:57.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:57.937 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:57.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:57.983 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.1 to 1 (10000) 2024-02-05T08:08:58.001 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:57 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1901086241' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:08:58.001 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:08:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:08:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:08:58.002 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:57 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1901086241' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:08:58.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:58.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:58.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:58.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:58.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:58.198 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:58.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:58.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:58.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:57 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1901086241' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:08:58.338 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:58.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:58.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:58.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:58.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:58.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:58.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:08:58.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:58.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:58.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:08:58.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:08:58.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:58.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:08:58.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:08:58.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:08:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:58.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:08:58.952 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:59.002 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:59.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:59.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:59.159 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:59.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:59.260 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:59.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:58 smithi134 ceph-mon[30086]: pgmap v162: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 601 op/s 2024-02-05T08:08:59.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:58 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1901086241' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:08:59.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:08:58 smithi134 ceph-mon[30086]: osdmap e134: 8 total, 8 up, 7 in 2024-02-05T08:08:59.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:58 smithi134 ceph-mon[39278]: pgmap v162: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 601 op/s 2024-02-05T08:08:59.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:58 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1901086241' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:08:59.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:08:58 smithi134 ceph-mon[39278]: osdmap e134: 8 total, 8 up, 7 in 2024-02-05T08:08:59.303 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:59.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:59.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:58 smithi196 ceph-mon[33018]: pgmap v162: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 601 op/s 2024-02-05T08:08:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:58 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1901086241' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:08:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:08:58 smithi196 ceph-mon[33018]: osdmap e134: 8 total, 8 up, 7 in 2024-02-05T08:08:59.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:59.460 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:59.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:59.561 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:08:59.610 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:59.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:59.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:08:59.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:08:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:59.754 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:59.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:08:59.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:08:59.860 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 2 1 2024-02-05T08:08:59.894 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:08:59.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:08:59.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:08:59.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:00.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:00.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:00.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:00.124 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:00.196 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:00.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:00.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:00.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:00.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:00.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:00.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:00.471 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:00.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:00.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:00.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:00.574 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:09:00.574 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:09:00.574 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:09:00.614 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:09:00.620 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:00.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:00.654 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:09:00.681 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:00.700 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:09:00.723 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:00.744 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:09:00.761 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:00.786 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:00.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:00.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:00.791 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:09:00.795 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:09:00.795 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:09:00.795 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:09:00.795 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:09:00.795 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:09:00.796 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:09:00.796 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:09:00.796 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:09:00.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:00.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:00.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:01.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:01.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:00 smithi134 ceph-mon[30086]: pgmap v164: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T08:09:01.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:00 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/41555783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:00 smithi134 ceph-mon[39278]: pgmap v164: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T08:09:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:00 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/41555783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:01.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:01.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:01.090 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:01.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:01.195 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:01.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:01.297 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:00 smithi196 ceph-mon[33018]: pgmap v164: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T08:09:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:00 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/41555783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:01.341 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:01.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:01.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:01.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:01.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:01.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:01.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:01.570 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:01.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:01.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:01.715 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:01.755 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:01.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:01.857 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:01.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:01.902 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:01.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:02.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:02.043 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:02.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:02.145 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:02.199 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:02.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:02.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:02.342 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:02.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:02.443 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:02.481 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:02.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:02.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:02.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:02.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:02.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:02.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:02.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:02.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:02.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:02.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:02.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:02.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:02.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:02.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:02.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:02.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:03.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:03.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:03.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:03.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:03.225 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:03.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:03.269 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:03.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:03.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:02 smithi134 ceph-mon[30086]: pgmap v165: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 590 B/s rd, 0 B/s wr, 483 op/s 2024-02-05T08:09:03.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:02 smithi134 ceph-mon[39278]: pgmap v165: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 590 B/s rd, 0 B/s wr, 483 op/s 2024-02-05T08:09:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:02 smithi196 ceph-mon[33018]: pgmap v165: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 590 B/s rd, 0 B/s wr, 483 op/s 2024-02-05T08:09:03.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:09:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:09:03.370 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:03.408 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:03.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:03.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:03.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:03.562 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:03.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:03.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:03.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:03.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:03.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:03.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:03.818 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:03.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:03.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:03.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:03.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:03.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:04.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:04.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:04.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:04.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:04.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:04.197 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:04.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:04.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:03 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:04 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:04.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:03 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:04.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:04.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:04.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:04.465 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:04.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:04.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:04.527 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:09:04.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:04.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:04.594 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:04.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:04.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:04.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:04.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:04.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:04.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:04.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:04.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:04.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:04.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:04.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:04.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:05.029 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:09:05.078 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:05.138 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:05.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:05.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:05 smithi134 ceph-mon[30086]: pgmap v166: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 844 op/s 2024-02-05T08:09:05.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:05 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3374408175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:05.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:05 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:05.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:05 smithi134 ceph-mon[39278]: pgmap v166: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 844 op/s 2024-02-05T08:09:05.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:05 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3374408175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:05.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:05 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:05.279 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:05.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:05 smithi196 ceph-mon[33018]: pgmap v166: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 844 op/s 2024-02-05T08:09:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:05 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3374408175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:05 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:05.380 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:05.425 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:05.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:05.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:05.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:05.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:05.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:05.679 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:05.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:05.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:05.807 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:05.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:05.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:05.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:05.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:05.935 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:05.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:06.036 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:06.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:06.072 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:06.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:06.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:06.199 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:06.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:06.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:06.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:06 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:09:06.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:06 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:09:06.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:06 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:09:06.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:06 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:09:06.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:06 smithi134 ceph-mon[30086]: osdmap e135: 8 total, 8 up, 7 in 2024-02-05T08:09:06.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:06 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:09:06.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:06 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:09:06.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:06 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:09:06.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:06 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:09:06.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:06 smithi134 ceph-mon[39278]: osdmap e135: 8 total, 8 up, 7 in 2024-02-05T08:09:06.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:06 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:09:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:06 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:09:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:06 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:09:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:06 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:09:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:06 smithi196 ceph-mon[33018]: osdmap e135: 8 total, 8 up, 7 in 2024-02-05T08:09:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:06.336 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:06.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:06.437 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:06.482 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:06.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:06.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:06.585 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:06.628 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:06.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:06.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:06.770 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:06.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:06.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:06.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:06.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:06.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:07.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:07.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:07.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:07.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:07.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:07.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:07.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:07.182 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:07 smithi134 ceph-mon[30086]: pgmap v168: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 955 op/s 2024-02-05T08:09:07.182 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:09:07.183 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:07 smithi134 ceph-mon[39278]: pgmap v168: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 955 op/s 2024-02-05T08:09:07.183 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:09:07.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:07.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:07.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:07.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:07 smithi196 ceph-mon[33018]: pgmap v168: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 955 op/s 2024-02-05T08:09:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:09:07.381 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:07.424 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:07.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:07.429 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:09:07.429 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:09:07.429 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:09:07.430 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:09:07.430 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.430 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:19159 7.3 7:c55d0d37:::benchmark_data_smithi134_83016_object19158:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:09:07.430 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.638521+0000", 2024-02-05T08:09:07.430 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 128.75369419099999, 2024-02-05T08:09:07.430 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13998516499999999, 2024-02-05T08:09:07.430 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.430 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.430 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:07.430 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:07.430 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:07.430 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 19159 2024-02-05T08:09:07.430 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.638521+0000", 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.638521+0000", 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.431 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.638522+0000", 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5300000000000002e-07 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.638543+0000", 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0217999999999999e-05 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.638551+0000", 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0730000000000005e-06 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.432 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.638557+0000", 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0390000000000001e-06 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.638772+0000", 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021574699999999999 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.638876+0000", 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000103553 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.433 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.638996+0000", 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000119846 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.639913+0000", 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00091749099999999999 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.776516+0000", 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13660303300000001 2024-02-05T08:09:07.434 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.778413+0000", 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001896327 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.778470+0000", 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7216999999999998e-05 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.778507+0000", 2024-02-05T08:09:07.435 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6668000000000002e-05 2024-02-05T08:09:07.436 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.436 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.436 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.436 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:19166 7.a 7:5e2b7dbe:::benchmark_data_smithi134_83016_object19165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:09:07.437 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.642046+0000", 2024-02-05T08:09:07.437 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 128.750169527, 2024-02-05T08:09:07.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14206841100000001, 2024-02-05T08:09:07.437 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.437 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.437 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:07.437 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:07.437 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 19166 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642046+0000", 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642046+0000", 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642047+0000", 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2720000000000001e-06 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642068+0000", 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0588999999999999e-05 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642078+0000", 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0611000000000001e-05 2024-02-05T08:09:07.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642086+0000", 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2010000000000002e-06 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.777725+0000", 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13563899800000001 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.777866+0000", 2024-02-05T08:09:07.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014176600000000001 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.777994+0000", 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012737100000000001 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.779259+0000", 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0012653289999999999 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.783603+0000", 2024-02-05T08:09:07.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0043439739999999996 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.784027+0000", 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00042381299999999999 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.784079+0000", 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2469000000000002e-05 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.442 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.784114+0000", 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5018000000000003e-05 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:19171 7.a 7:5efae3ab:::benchmark_data_smithi134_83016_object19170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.642948+0000", 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 128.749267976, 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14479762299999999, 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:07.443 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:07.444 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:07.444 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 19171 2024-02-05T08:09:07.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.444 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.444 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.444 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642948+0000", 2024-02-05T08:09:07.444 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.444 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.444 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642948+0000", 2024-02-05T08:09:07.444 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642949+0000", 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1400000000000001e-06 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642971+0000", 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2257000000000001e-05 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642983+0000", 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.221e-05 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.642991+0000", 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3070000000000003e-06 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.779840+0000", 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13684860700000001 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.446 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.447 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.779948+0000", 2024-02-05T08:09:07.447 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107884 2024-02-05T08:09:07.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.447 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-02-05T08:09:07.447 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.780090+0000", 2024-02-05T08:09:07.447 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014155600000000001 2024-02-05T08:09:07.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.447 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.447 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.783429+0000", 2024-02-05T08:09:07.447 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033398429999999999 2024-02-05T08:09:07.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.448 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.448 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.448 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.783696+0000", 2024-02-05T08:09:07.448 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00026629600000000001 2024-02-05T08:09:07.448 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.787659+0000", 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0039635240000000004 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.787709+0000", 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0096999999999998e-05 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.787745+0000", 2024-02-05T08:09:07.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5902000000000003e-05 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20662 7.c 7:38ee9845:::benchmark_data_smithi134_83016_object20661:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.662054+0000", 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 127.730161055, 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15580438599999999, 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:07.450 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20662 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.662054+0000", 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.662054+0000", 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.452 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.452 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.662056+0000", 2024-02-05T08:09:07.452 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9800000000000001e-06 2024-02-05T08:09:07.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.452 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.452 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.452 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.662078+0000", 2024-02-05T08:09:07.452 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1447999999999999e-05 2024-02-05T08:09:07.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.452 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.453 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.453 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.662094+0000", 2024-02-05T08:09:07.453 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5798e-05 2024-02-05T08:09:07.453 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.453 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.453 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.453 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.662102+0000", 2024-02-05T08:09:07.453 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4249999999999998e-06 2024-02-05T08:09:07.453 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.453 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.453 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.453 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.664388+0000", 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002285881 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.664564+0000", 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017629700000000001 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,7", 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.664655+0000", 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0607999999999997e-05 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.454 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.455 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666164+0000", 2024-02-05T08:09:07.455 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001509465 2024-02-05T08:09:07.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.455 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:09:07.455 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666692+0000", 2024-02-05T08:09:07.455 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052759699999999998 2024-02-05T08:09:07.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.455 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.455 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817770+0000", 2024-02-05T08:09:07.455 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.151077878 2024-02-05T08:09:07.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817822+0000", 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1786000000000002e-05 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817859+0000", 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7222999999999999e-05 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20677 7.3 7:c2631682:::benchmark_data_smithi134_83016_object20676:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.668436+0000", 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 127.723779966, 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.148324966, 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20677 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.457 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.458 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668436+0000", 2024-02-05T08:09:07.458 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.458 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.458 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668436+0000", 2024-02-05T08:09:07.458 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.458 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.458 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668437+0000", 2024-02-05T08:09:07.458 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8729999999999999e-06 2024-02-05T08:09:07.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668462+0000", 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4161000000000001e-05 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668474+0000", 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2503999999999999e-05 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668483+0000", 2024-02-05T08:09:07.459 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8219999999999996e-06 2024-02-05T08:09:07.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.460 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.460 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668595+0000", 2024-02-05T08:09:07.460 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000112166 2024-02-05T08:09:07.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.461 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.462 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668726+0000", 2024-02-05T08:09:07.462 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013133900000000001 2024-02-05T08:09:07.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.462 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.462 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-02-05T08:09:07.462 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669029+0000", 2024-02-05T08:09:07.462 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030240399999999999 2024-02-05T08:09:07.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.463 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.463 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671163+0000", 2024-02-05T08:09:07.464 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002133912 2024-02-05T08:09:07.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.464 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:09:07.464 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671175+0000", 2024-02-05T08:09:07.464 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2464e-05 2024-02-05T08:09:07.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.464 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.464 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816655+0000", 2024-02-05T08:09:07.464 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.145479634 2024-02-05T08:09:07.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.465 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.465 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816719+0000", 2024-02-05T08:09:07.465 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4251000000000005e-05 2024-02-05T08:09:07.465 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.465 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.465 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.465 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816761+0000", 2024-02-05T08:09:07.465 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1436000000000001e-05 2024-02-05T08:09:07.465 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.465 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.465 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.465 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20679 7.3 7:c41cf0a1:::benchmark_data_smithi134_83016_object20678:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.669896+0000", 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 127.72231993699999, 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14711723600000001, 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20679 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669896+0000", 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669896+0000", 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669897+0000", 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.64e-07 2024-02-05T08:09:07.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.468 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.468 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.468 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669917+0000", 2024-02-05T08:09:07.468 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0774000000000001e-05 2024-02-05T08:09:07.468 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.468 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.468 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.468 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669926+0000", 2024-02-05T08:09:07.468 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0620000000000007e-06 2024-02-05T08:09:07.469 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.469 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.469 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.469 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669933+0000", 2024-02-05T08:09:07.469 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.674e-06 2024-02-05T08:09:07.469 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.469 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.469 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.469 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670103+0000", 2024-02-05T08:09:07.469 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016991099999999999 2024-02-05T08:09:07.469 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.470 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.470 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.470 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670258+0000", 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015542900000000001 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670394+0000", 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013585 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671337+0000", 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00094267899999999998 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.471 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671621+0000", 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00028454899999999999 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816928+0000", 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14530680200000001 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816978+0000", 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9808999999999997e-05 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.472 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817013+0000", 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4733000000000001e-05 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20684 7.2 7:47ab7ab0:::benchmark_data_smithi134_83016_object20683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.672242+0000", 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 127.719973157, 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.145381759, 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.473 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20684 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672242+0000", 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672242+0000", 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672244+0000", 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0779999999999998e-06 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672264+0000", 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9582999999999999e-05 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672276+0000", 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1959e-05 2024-02-05T08:09:07.475 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672283+0000", 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4429999999999997e-06 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672593+0000", 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030986099999999998 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672690+0000", 2024-02-05T08:09:07.476 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6805999999999994e-05 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672819+0000", 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012925 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673873+0000", 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001053448 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816996+0000", 2024-02-05T08:09:07.477 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.143123422 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817535+0000", 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00053839100000000004 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817587+0000", 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2167999999999999e-05 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.478 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817624+0000", 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.735e-05 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20687 7.c 7:32cda045:::benchmark_data_smithi134_83016_object20686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.672775+0000", 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 127.71944042600001, 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14526989600000001, 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:07.479 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:07.480 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20687 2024-02-05T08:09:07.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.480 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.480 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672775+0000", 2024-02-05T08:09:07.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.480 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672775+0000", 2024-02-05T08:09:07.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672776+0000", 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.09e-07 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672796+0000", 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0568e-05 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672806+0000", 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6679999999999996e-06 2024-02-05T08:09:07.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672814+0000", 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6389999999999992e-06 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672978+0000", 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016463799999999999 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673143+0000", 2024-02-05T08:09:07.482 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016464999999999999 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,7", 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673280+0000", 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013700499999999999 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817907+0000", 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.144627167 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817948+0000", 2024-02-05T08:09:07.483 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0405999999999999e-05 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817961+0000", 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3043e-05 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818005+0000", 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4446000000000001e-05 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818045+0000", 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9857000000000001e-05 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:20662 7.c e104/97 ondisk, result = 0)", 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.673606+0000", 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 127.718609985, 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.144264645, 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.485 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673606+0000", 2024-02-05T08:09:07.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673606+0000", 2024-02-05T08:09:07.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673606+0000", 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9500000000000001e-07 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673611+0000", 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2420000000000002e-06 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673620+0000", 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8700000000000004e-06 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673627+0000", 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7710000000000001e-06 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817756+0000", 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14412877900000001 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817763+0000", 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4769999999999997e-06 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.488 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817870+0000", 2024-02-05T08:09:07.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000106711 2024-02-05T08:09:07.489 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.489 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.489 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.489 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:20677 7.3 e104/95 ondisk, result = 0)", 2024-02-05T08:09:07.489 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.673682+0000", 2024-02-05T08:09:07.489 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 127.71853305499999, 2024-02-05T08:09:07.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.143087835, 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673682+0000", 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673682+0000", 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673683+0000", 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.75e-07 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673686+0000", 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3129999999999999e-06 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673695+0000", 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3310000000000004e-06 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673701+0000", 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4840000000000001e-06 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816639+0000", 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14293777399999999 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816648+0000", 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2350000000000005e-06 2024-02-05T08:09:07.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816770+0000", 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000122223 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:20679 7.3 e104/95 ondisk, result = 0)", 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.673718+0000", 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 127.718497391, 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14330227800000001, 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.493 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673718+0000", 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673718+0000", 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673718+0000", 2024-02-05T08:09:07.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3999999999999997e-07 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673721+0000", 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5909999999999998e-06 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673727+0000", 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7690000000000002e-06 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673732+0000", 2024-02-05T08:09:07.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0459999999999996e-06 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816915+0000", 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14318307599999999 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816923+0000", 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9449999999999993e-06 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817020+0000", 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7510999999999995e-05 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:20685 7.d e104/97 7:b47cd13c:::benchmark_data_smithi134_83016_object20684:head v 104'1336, mlcod=104'1334)", 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.673829+0000", 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 127.71838607399999, 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.144020222, 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673829+0000", 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673829+0000", 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673830+0000", 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2200000000000004e-07 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673859+0000", 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8892999999999999e-05 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673868+0000", 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6950000000000006e-06 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673874+0000", 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3720000000000001e-06 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673937+0000", 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2497999999999997e-05 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673954+0000", 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7348999999999999e-05 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817811+0000", 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.143857551 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.500 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817850+0000", 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8241999999999997e-05 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:20691 7.0 e104/95 7:09ebe114:::benchmark_data_smithi134_83016_object20690:head v 104'1323, mlcod=104'1321)", 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.676085+0000", 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 127.716130832, 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14682956799999999, 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.676085+0000", 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.676085+0000", 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.676090+0000", 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4210000000000003e-06 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.676139+0000", 2024-02-05T08:09:07.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9361999999999999e-05 2024-02-05T08:09:07.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816236+0000", 2024-02-05T08:09:07.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.140096938 2024-02-05T08:09:07.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816249+0000", 2024-02-05T08:09:07.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2262000000000001e-05 2024-02-05T08:09:07.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816356+0000", 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107379 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816473+0000", 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000116458 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.822878+0000", 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0064056199999999999 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.822914+0000", 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6127999999999997e-05 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21738 7.a 7:5f5817b2:::benchmark_data_smithi134_83016_object21737:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.326765+0000", 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 127.06545054199999, 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15386414800000001, 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:07.506 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21738 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326765+0000", 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326765+0000", 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326765+0000", 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3599999999999999e-07 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326784+0000", 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8717000000000001e-05 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326791+0000", 2024-02-05T08:09:07.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4679999999999997e-06 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326796+0000", 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1340000000000003e-06 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327302+0000", 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00050595499999999999 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327451+0000", 2024-02-05T08:09:07.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014938700000000001 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327673+0000", 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00022182699999999999 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.459480+0000", 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.131807534 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.460675+0000", 2024-02-05T08:09:07.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011940710000000001 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480519+0000", 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.019844164000000001 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480582+0000", 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3275000000000007e-05 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480629+0000", 2024-02-05T08:09:07.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7179999999999999e-05 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21745 7.a 7:5329033d:::benchmark_data_smithi134_83016_object21744:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.329111+0000", 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 127.063104763, 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.155629409, 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:07.512 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21745 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329111+0000", 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329111+0000", 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329112+0000", 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6600000000000005e-07 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329129+0000", 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7503000000000002e-05 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329138+0000", 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2750000000000001e-06 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329144+0000", 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5450000000000003e-06 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.457754+0000", 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12861018399999999 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.458217+0000", 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00046327199999999998 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-02-05T08:09:07.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.458372+0000", 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015474 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.461668+0000", 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032962960000000002 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.463096+0000", 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0014271310000000001 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.484644+0000", 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.02154876 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.484700+0000", 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6144999999999999e-05 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.484740+0000", 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9691999999999999e-05 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.517 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:99785 7.d e126/97 7:b0ae1387:::benchmark_data_smithi134_83016_object99784:head v 126'6269, mlcod=126'6265)", 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:08.940511+0000", 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 58.451704045, 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14054608499999999, 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.940511+0000", 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.940511+0000", 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.940512+0000", 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6000000000000003e-07 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.940539+0000", 2024-02-05T08:09:07.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6602999999999999e-05 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.940548+0000", 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5100000000000004e-06 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.940555+0000", 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7349999999999999e-06 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.942016+0000", 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0014609499999999999 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.942035+0000", 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9151e-05 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.081015+0000", 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13898007200000001 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.081058+0000", 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2404000000000003e-05 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:99786 7.d e126/97 7:b2cade7f:::benchmark_data_smithi134_83016_object99785:head v 126'6270, mlcod=126'6265)", 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:08.943610+0000", 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 58.448605411999999, 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13999288500000001, 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.943610+0000", 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.943610+0000", 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.943611+0000", 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6700000000000003e-07 2024-02-05T08:09:07.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.943660+0000", 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9011999999999998e-05 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.943676+0000", 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5832e-05 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.943683+0000", 2024-02-05T08:09:07.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1380000000000001e-06 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.949461+0000", 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0057780430000000001 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.949490+0000", 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9386999999999998e-05 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.083400+0000", 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13390982000000001 2024-02-05T08:09:07.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.083603+0000", 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000202886 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106466 7.3 7:c61c9291:::benchmark_data_smithi134_83016_object106465:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.277873+0000", 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 51.114342770999997, 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.145889092, 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.526 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106466 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.277873+0000", 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.277873+0000", 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.277874+0000", 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6160000000000001e-06 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.277896+0000", 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1886e-05 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.277908+0000", 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1545999999999999e-05 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.277916+0000", 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4270000000000008e-06 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.403729+0000", 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12581266499999999 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.403853+0000", 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000124268 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.403998+0000", 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014531599999999999 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.406666+0000", 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0026674060000000002 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.416244+0000", 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0095785660000000002 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.423660+0000", 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0074155059999999997 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.423719+0000", 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8931999999999998e-05 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.423762+0000", 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2957999999999998e-05 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:131219 7.0 e129/95 7:0a5fbf5e:::benchmark_data_smithi134_83016_object21285:head v 129'8503, mlcod=129'8484)", 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:42.118561+0000", 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 25.273654095000001, 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.150483543, 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.532 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.118561+0000", 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.118561+0000", 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.118562+0000", 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5600000000000005e-07 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.118567+0000", 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.724e-06 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.118580+0000", 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2952999999999999e-05 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.118587+0000", 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0339999999999999e-06 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.191391+0000", 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.072804167000000003 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.191411+0000", 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0060000000000001e-05 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.535 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.268975+0000", 2024-02-05T08:09:07.536 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.077564007000000004 2024-02-05T08:09:07.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.269045+0000", 2024-02-05T08:09:07.536 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9942000000000002e-05 2024-02-05T08:09:07.536 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.536 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.536 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.536 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:131230 7.0 e129/95 7:08d1f14e:::benchmark_data_smithi134_83016_object21296:head v 129'8504, mlcod=129'8484)", 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:42.121067+0000", 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 25.271148352000001, 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.153810419, 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.121067+0000", 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.121067+0000", 2024-02-05T08:09:07.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.121068+0000", 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8999999999999996e-07 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.121075+0000", 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4460000000000004e-06 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:07.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.121090+0000", 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.517e-05 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.121097+0000", 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4540000000000004e-06 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.194450+0000", 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.073352602000000003 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:07.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.194471+0000", 2024-02-05T08:09:07.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0981000000000001e-05 2024-02-05T08:09:07.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:09:07.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.274841+0000", 2024-02-05T08:09:07.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.080370769999999994 2024-02-05T08:09:07.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:07.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:07.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:09:07.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:42.274878+0000", 2024-02-05T08:09:07.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6306000000000001e-05 2024-02-05T08:09:07.540 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.540 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.541 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.541 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:07.541 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:07.541 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:09:07.541 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:07.584 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:07.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:07.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:07.686 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:07.727 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:07.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:07.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:07.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:07.829 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:07.898 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:07.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:08.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:08.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:08.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:09:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:09:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:09:08.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:08.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:08.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:08.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:08.169 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:08.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:08.270 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:08.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:08.311 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:08.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:08.395 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T08:09:08.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:08.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:08.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:08.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:08.571 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:08.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:08.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:08.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:08.695 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:09:08.715 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:08.750 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:08.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:08.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:08.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:08.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:08.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:08.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:09.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:09.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:09.085 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:09 smithi196 ceph-mon[33018]: pgmap v169: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 532 op/s 2024-02-05T08:09:09.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:09.136 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:09.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:09.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:09.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:09.274 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:09 smithi134 ceph-mon[30086]: pgmap v169: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 532 op/s 2024-02-05T08:09:09.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:09 smithi134 ceph-mon[39278]: pgmap v169: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 532 op/s 2024-02-05T08:09:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:09.278 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:09.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:09.352 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:09:09.379 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:09.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:09.462 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:09.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:09.563 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:09.626 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:09.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:09.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:09.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:09.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:09.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:09.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:09.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:09.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:10.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:10.030 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:10.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:10.131 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:10.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:10.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:10.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:10.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:10.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:10.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:10.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:10.386 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:10.401 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:10.461 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:10.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:10.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:10.563 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:10.604 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.2 to 1 (10000) 2024-02-05T08:09:10.617 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:10.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:10.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:10.719 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:10.772 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:10.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:10.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:10 smithi196 ceph-mon[33018]: pgmap v170: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 894 op/s 2024-02-05T08:09:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:10 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3375938224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:09:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:10 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:09:10.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:10.901 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:10.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:10.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:11.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:10 smithi134 ceph-mon[30086]: pgmap v170: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 894 op/s 2024-02-05T08:09:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:10 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3375938224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:09:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:10 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:09:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:10 smithi134 ceph-mon[39278]: pgmap v170: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 894 op/s 2024-02-05T08:09:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:10 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3375938224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:09:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:10 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:09:11.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:11.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:11.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:11.110 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:09:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:09:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.14802.0:155938 7.4 e136/95 7:25955430:::benchmark_data_smithi134_83016_object46004:head v 136'9869, mlcod=136'9867)", 2024-02-05T08:09:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:11.108811+0000", 2024-02-05T08:09:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0013591269999999999, 2024-02-05T08:09:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013882079999999999, 2024-02-05T08:09:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T08:09:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:11.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.108811+0000", 2024-02-05T08:09:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.108811+0000", 2024-02-05T08:09:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.108812+0000", 2024-02-05T08:09:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7899999999999998e-07 2024-02-05T08:09:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.108817+0000", 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3879999999999999e-06 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.108829+0000", 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1416999999999999e-05 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.108835+0000", 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2020000000000001e-06 2024-02-05T08:09:11.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.108887+0000", 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2332e-05 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.108908+0000", 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1100000000000001e-05 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.114 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:155940 7.1 7:87a82119:::benchmark_data_smithi134_83016_object46006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:11.109356+0000", 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.00081424299999999998, 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00091144299999999995, 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 155940 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:11.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.109356+0000", 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.109356+0000", 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.109358+0000", 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3680000000000001e-06 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:11.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.109364+0000", 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0109999999999997e-06 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.109379+0000", 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5328999999999998e-05 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.109388+0000", 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0350000000000007e-06 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:11.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.109440+0000", 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2234000000000002e-05 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.109564+0000", 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012368899999999999 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:11.109682+0000", 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011845699999999999 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:11.118 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:11.119 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:11.119 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-02-05T08:09:11.119 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 2 2024-02-05T08:09:11.119 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:09:11.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:11.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:11.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:11.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:11.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:11.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:11.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:11.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:11.332 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:09:11.381 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:09:11.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:11.422 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:09:11.425 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:11.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:11.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:11.471 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:09:11.509 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:09:11.527 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:11.550 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:09:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:11.569 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:11.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:11.592 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:09:11.637 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:09:11.645 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:09:11.645 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:09:11.645 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:09:11.645 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:09:11.645 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:09:11.645 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:09:11.645 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:09:11.645 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:09:11.671 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:11.716 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:11.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:11.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:11 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:09:11.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:11 smithi196 ceph-mon[33018]: osdmap e136: 8 total, 8 up, 7 in 2024-02-05T08:09:11.784 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:11 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1093820301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:11.817 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:11.819 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T08:09:11.856 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:11.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:11.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:11.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:11.996 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:11.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:11.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:12.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:11 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:09:12.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:11 smithi134 ceph-mon[30086]: osdmap e136: 8 total, 8 up, 7 in 2024-02-05T08:09:12.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:11 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1093820301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:12.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:11 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:09:12.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:11 smithi134 ceph-mon[39278]: osdmap e136: 8 total, 8 up, 7 in 2024-02-05T08:09:12.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:11 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1093820301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:12.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:12.112 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:09:12.137 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:12.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:12.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:12.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:12.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:12.366 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:12.408 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:12.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:12.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:12.509 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:12.559 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:12.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:12.660 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:12.669 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 3 1 2024-02-05T08:09:12.686 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:12.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:12.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:12.788 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:12 smithi196 ceph-mon[33018]: pgmap v172: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 585 op/s 2024-02-05T08:09:12.835 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:12.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:12.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:12.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:12.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:12.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:13.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:12 smithi134 ceph-mon[30086]: pgmap v172: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 585 op/s 2024-02-05T08:09:13.101 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:12 smithi134 ceph-mon[39278]: pgmap v172: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 585 op/s 2024-02-05T08:09:13.102 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:13.156 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:13.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:13.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:13.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:13.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:13.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:13.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:09:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:09:13.387 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:13.411 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:13.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:13.513 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:13.555 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:13.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:13.651 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:13.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:13.716 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:13.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:13.817 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:13.859 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:13.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:13.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:13.898 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:13 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:13.899 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:13 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:13.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:13 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:13.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:14.008 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:14.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:14.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:14.085 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:09:14.085 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:09:14.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:14.155 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:14.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:14.257 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:14.292 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:14.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:14.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:14.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:14.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:14.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:14.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:14.566 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:14.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:14.667 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:14.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:14.705 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:14.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:14.739 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:14 smithi196 ceph-mon[33018]: pgmap v173: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T08:09:14.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T08:09:14.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:14.844 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:14.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:14.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:14.916 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:14 smithi134 ceph-mon[30086]: pgmap v173: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T08:09:14.917 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:14 smithi134 ceph-mon[39278]: pgmap v173: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T08:09:14.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:14.990 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:14.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:15.066 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:09:15.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:15.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:15.118 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:15.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:15.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:09:15.219 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:15.268 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:15.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:15.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:15.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:15.395 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:15.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:15.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:15.552 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:15.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:15.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:15.653 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:15.688 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:15.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:15.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:15.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:15.816 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:15.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:15.918 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:15.947 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:15.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:16.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:16.089 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:16.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:16.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:16.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:16.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:16.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:16.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:16.341 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:16.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:16.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:16.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:16.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:16.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:16.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:16.594 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:16.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:16.681 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:09:16.696 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:16.737 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:16.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:16.748 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:16 smithi134 ceph-mon[30086]: pgmap v174: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 975 op/s 2024-02-05T08:09:16.748 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:16 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:09:16.748 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:16 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:09:16.748 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:16 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1057775695' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:09:16.748 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:16 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:09:16.748 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:16 smithi134 ceph-mon[39278]: pgmap v174: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 975 op/s 2024-02-05T08:09:16.749 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:16 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:09:16.749 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:16 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:09:16.749 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:16 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1057775695' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:09:16.749 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:16 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:09:16.839 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:16.877 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:16.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:16.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:17.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:16 smithi196 ceph-mon[33018]: pgmap v174: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 975 op/s 2024-02-05T08:09:17.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:16 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:09:17.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:16 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:09:17.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:16 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1057775695' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:09:17.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:16 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:09:17.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:17.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:17.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:17.134 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:17.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:17.236 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:17.273 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:17.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:17.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:17.299 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:09:17.299 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:09:17.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:17.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:17.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:17.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:17.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:17.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:17.545 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:17.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:17.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:17.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:17.674 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:17.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:17.775 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:17.817 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:17.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:17.918 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:17.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:17.945 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:17.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:17.981 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:09:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:09:17.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:09:17.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:09:17.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[30086]: osdmap e137: 8 total, 8 up, 7 in 2024-02-05T08:09:17.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[30086]: 4.f scrub starts 2024-02-05T08:09:17.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[30086]: 4.f scrub ok 2024-02-05T08:09:17.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[30086]: 5.e scrub starts 2024-02-05T08:09:17.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[30086]: 5.e scrub ok 2024-02-05T08:09:17.982 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:09:17.982 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:09:17.983 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[39278]: osdmap e137: 8 total, 8 up, 7 in 2024-02-05T08:09:17.983 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[39278]: 4.f scrub starts 2024-02-05T08:09:17.983 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[39278]: 4.f scrub ok 2024-02-05T08:09:17.983 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[39278]: 5.e scrub starts 2024-02-05T08:09:17.983 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:17 smithi134 ceph-mon[39278]: 5.e scrub ok 2024-02-05T08:09:18.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:17 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:09:18.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:17 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:09:18.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:17 smithi196 ceph-mon[33018]: osdmap e137: 8 total, 8 up, 7 in 2024-02-05T08:09:18.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:17 smithi196 ceph-mon[33018]: 4.f scrub starts 2024-02-05T08:09:18.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:17 smithi196 ceph-mon[33018]: 4.f scrub ok 2024-02-05T08:09:18.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:17 smithi196 ceph-mon[33018]: 5.e scrub starts 2024-02-05T08:09:18.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:17 smithi196 ceph-mon[33018]: 5.e scrub ok 2024-02-05T08:09:18.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T08:09:18.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:18.093 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:18.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:18.128 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:18.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:18.243 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:18.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:18.331 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:09:18.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:18.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:18.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:18.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:18.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:18.518 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:18.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:18.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:18.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:18.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:18.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:18.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:18.751 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:18.775 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:18.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:18.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:18.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:18 smithi196 ceph-mon[33018]: 7.0 scrub starts 2024-02-05T08:09:18.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:18 smithi196 ceph-mon[33018]: pgmap v176: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 741 op/s 2024-02-05T08:09:18.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:18 smithi196 ceph-mon[33018]: 6.17 scrub starts 2024-02-05T08:09:18.792 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:18 smithi196 ceph-mon[33018]: 6.17 scrub ok 2024-02-05T08:09:18.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:18.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:18.904 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:18.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:18.937 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:18 smithi134 ceph-mon[30086]: 7.0 scrub starts 2024-02-05T08:09:18.937 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:18 smithi134 ceph-mon[30086]: pgmap v176: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 741 op/s 2024-02-05T08:09:18.937 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:18 smithi134 ceph-mon[30086]: 6.17 scrub starts 2024-02-05T08:09:18.937 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:18 smithi134 ceph-mon[30086]: 6.17 scrub ok 2024-02-05T08:09:18.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:18 smithi134 ceph-mon[39278]: 7.0 scrub starts 2024-02-05T08:09:18.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:18 smithi134 ceph-mon[39278]: pgmap v176: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 741 op/s 2024-02-05T08:09:18.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:18 smithi134 ceph-mon[39278]: 6.17 scrub starts 2024-02-05T08:09:18.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:18 smithi134 ceph-mon[39278]: 6.17 scrub ok 2024-02-05T08:09:19.005 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:19.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:19.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:19.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:19.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:19.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:19.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:19.260 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:19.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:19.304 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:19.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:19.405 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:19.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:19.449 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:19.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:19.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:19.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:19.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:19.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:19.693 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.3 to 1 (10000) 2024-02-05T08:09:19.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:19.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:19.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:19.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:19.825 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:19.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:19.866 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:19.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:19.967 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:20.009 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:20.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:20.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:19 smithi134 ceph-mon[30086]: 7.9 scrub starts 2024-02-05T08:09:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:19 smithi134 ceph-mon[30086]: 7.0 scrub ok 2024-02-05T08:09:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:19 smithi134 ceph-mon[30086]: 7.c scrub starts 2024-02-05T08:09:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:19 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3167110491' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T08:09:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:19 smithi134 ceph-mon[30086]: 3.13 scrub starts 2024-02-05T08:09:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:19 smithi134 ceph-mon[30086]: 3.13 scrub ok 2024-02-05T08:09:20.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:19 smithi134 ceph-mon[39278]: 7.9 scrub starts 2024-02-05T08:09:20.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:19 smithi134 ceph-mon[39278]: 7.0 scrub ok 2024-02-05T08:09:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:19 smithi134 ceph-mon[39278]: 7.c scrub starts 2024-02-05T08:09:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:19 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3167110491' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T08:09:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:19 smithi134 ceph-mon[39278]: 3.13 scrub starts 2024-02-05T08:09:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:19 smithi134 ceph-mon[39278]: 3.13 scrub ok 2024-02-05T08:09:20.047 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:19 smithi196 ceph-mon[33018]: 7.9 scrub starts 2024-02-05T08:09:20.047 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:19 smithi196 ceph-mon[33018]: 7.0 scrub ok 2024-02-05T08:09:20.047 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:19 smithi196 ceph-mon[33018]: 7.c scrub starts 2024-02-05T08:09:20.047 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:19 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3167110491' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T08:09:20.047 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:19 smithi196 ceph-mon[33018]: 3.13 scrub starts 2024-02-05T08:09:20.047 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:19 smithi196 ceph-mon[33018]: 3.13 scrub ok 2024-02-05T08:09:20.092 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:09:20.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:20.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:20.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:20.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:20.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:20.278 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:20.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:20.329 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:09:20.329 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:09:20.379 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:20.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:20.437 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:20.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:20.479 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 4 1 2024-02-05T08:09:20.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:20.565 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:20.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:20.666 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:20.692 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:20.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:20.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:20.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:20.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:20.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:20.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:20.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:20.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[30086]: 7.9 scrub ok 2024-02-05T08:09:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[30086]: pgmap v177: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-02-05T08:09:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3167110491' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T08:09:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[30086]: 7.c scrub ok 2024-02-05T08:09:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[30086]: osdmap e138: 8 total, 8 up, 7 in 2024-02-05T08:09:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:09:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:09:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[30086]: 5.9 scrub starts 2024-02-05T08:09:21.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[30086]: 5.9 scrub ok 2024-02-05T08:09:21.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[30086]: 5.1 scrub starts 2024-02-05T08:09:21.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[30086]: 5.1 scrub ok 2024-02-05T08:09:21.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[39278]: 7.9 scrub ok 2024-02-05T08:09:21.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[39278]: pgmap v177: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-02-05T08:09:21.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3167110491' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T08:09:21.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[39278]: 7.c scrub ok 2024-02-05T08:09:21.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[39278]: osdmap e138: 8 total, 8 up, 7 in 2024-02-05T08:09:21.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:09:21.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:09:21.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[39278]: 5.9 scrub starts 2024-02-05T08:09:21.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[39278]: 5.9 scrub ok 2024-02-05T08:09:21.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[39278]: 5.1 scrub starts 2024-02-05T08:09:21.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:20 smithi134 ceph-mon[39278]: 5.1 scrub ok 2024-02-05T08:09:21.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:20 smithi196 ceph-mon[33018]: 7.9 scrub ok 2024-02-05T08:09:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:20 smithi196 ceph-mon[33018]: pgmap v177: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-02-05T08:09:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:20 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3167110491' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T08:09:21.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:20 smithi196 ceph-mon[33018]: 7.c scrub ok 2024-02-05T08:09:21.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:20 smithi196 ceph-mon[33018]: osdmap e138: 8 total, 8 up, 7 in 2024-02-05T08:09:21.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:20 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:09:21.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:20 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:09:21.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:20 smithi196 ceph-mon[33018]: 5.9 scrub starts 2024-02-05T08:09:21.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:20 smithi196 ceph-mon[33018]: 5.9 scrub ok 2024-02-05T08:09:21.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:20 smithi196 ceph-mon[33018]: 5.1 scrub starts 2024-02-05T08:09:21.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:20 smithi196 ceph-mon[33018]: 5.1 scrub ok 2024-02-05T08:09:21.071 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:21.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:21.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T08:09:21.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:21.216 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:21.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:21.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:21.346 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:21.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:21.399 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:09:21.447 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:21.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:21.490 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:21.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:21.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:21.633 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:21.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:21.734 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:21.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:21.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:21 smithi196 ceph-mon[33018]: 4.6 scrub starts 2024-02-05T08:09:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:21 smithi196 ceph-mon[33018]: 4.6 scrub ok 2024-02-05T08:09:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:21 smithi196 ceph-mon[33018]: 7.2 scrub starts 2024-02-05T08:09:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:21 smithi196 ceph-mon[33018]: osdmap e139: 8 total, 8 up, 7 in 2024-02-05T08:09:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:21 smithi196 ceph-mon[33018]: 4.d scrub starts 2024-02-05T08:09:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:21 smithi196 ceph-mon[33018]: 4.d scrub ok 2024-02-05T08:09:21.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:21.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:21.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:21.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:21.932 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:21 smithi134 ceph-mon[30086]: 4.6 scrub starts 2024-02-05T08:09:21.932 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:21 smithi134 ceph-mon[30086]: 4.6 scrub ok 2024-02-05T08:09:21.932 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:21 smithi134 ceph-mon[30086]: 7.2 scrub starts 2024-02-05T08:09:21.932 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:21 smithi134 ceph-mon[30086]: osdmap e139: 8 total, 8 up, 7 in 2024-02-05T08:09:21.932 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:21 smithi134 ceph-mon[30086]: 4.d scrub starts 2024-02-05T08:09:21.932 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:21 smithi134 ceph-mon[30086]: 4.d scrub ok 2024-02-05T08:09:21.932 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:21 smithi134 ceph-mon[39278]: 4.6 scrub starts 2024-02-05T08:09:21.932 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:21 smithi134 ceph-mon[39278]: 4.6 scrub ok 2024-02-05T08:09:21.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:21 smithi134 ceph-mon[39278]: 7.2 scrub starts 2024-02-05T08:09:21.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:21 smithi134 ceph-mon[39278]: osdmap e139: 8 total, 8 up, 7 in 2024-02-05T08:09:21.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:21 smithi134 ceph-mon[39278]: 4.d scrub starts 2024-02-05T08:09:21.933 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:21 smithi134 ceph-mon[39278]: 4.d scrub ok 2024-02-05T08:09:21.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:22.014 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:22.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:22.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:22.116 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:22.143 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:22.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:22.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:22.245 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:22.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:22.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:22.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:22.414 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:22.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:22.516 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:22.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:22.543 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:22.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:22.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:22.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:22.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:22.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:22.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:22.795 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:22.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:22 smithi196 ceph-mon[33018]: 6.1f scrub starts 2024-02-05T08:09:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:22 smithi196 ceph-mon[33018]: 6.1f scrub ok 2024-02-05T08:09:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:22 smithi196 ceph-mon[33018]: pgmap v180: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 752 op/s 2024-02-05T08:09:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:22 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:09:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:22 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:09:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:22 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/206517924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:09:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:22 smithi196 ceph-mon[33018]: 3.9 scrub starts 2024-02-05T08:09:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:22 smithi196 ceph-mon[33018]: 3.9 scrub ok 2024-02-05T08:09:22.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:22.944 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:22.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:23.005 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:09:23.005 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:09:23.005 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:09:23.005 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:09:23.005 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:09:23.005 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:09:23.006 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:09:23.006 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:09:23.006 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:09:23.006 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:09:23.006 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:09:23.006 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:09:23.006 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:09:23.006 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:09:23.006 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:09:23.006 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:09:23.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[30086]: 6.1f scrub starts 2024-02-05T08:09:23.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[30086]: 6.1f scrub ok 2024-02-05T08:09:23.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[30086]: pgmap v180: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 752 op/s 2024-02-05T08:09:23.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:09:23.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:09:23.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/206517924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:23.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:09:23.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[30086]: 3.9 scrub starts 2024-02-05T08:09:23.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[30086]: 3.9 scrub ok 2024-02-05T08:09:23.007 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[39278]: 6.1f scrub starts 2024-02-05T08:09:23.008 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[39278]: 6.1f scrub ok 2024-02-05T08:09:23.008 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[39278]: pgmap v180: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 3.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 752 op/s 2024-02-05T08:09:23.008 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:09:23.008 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:09:23.008 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/206517924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:23.008 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:09:23.008 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[39278]: 3.9 scrub starts 2024-02-05T08:09:23.008 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:22 smithi134 ceph-mon[39278]: 3.9 scrub ok 2024-02-05T08:09:23.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:23.077 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:23.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:23.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:23.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:23.235 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:23.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:23.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:23.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:09:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:09:23.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:23.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:23.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:23.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:23.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:23.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:23.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:23.599 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:23.648 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:23.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:23.749 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:23.756 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:23 smithi196 ceph-mon[33018]: 3.b scrub starts 2024-02-05T08:09:23.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:23 smithi196 ceph-mon[33018]: 3.b scrub ok 2024-02-05T08:09:23.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:23 smithi196 ceph-mon[33018]: 7.2 scrub ok 2024-02-05T08:09:23.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:09:23.757 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:23 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:23.777 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:23.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:23.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:23.786 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:09:23.786 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:09:23.786 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T08:09:23.786 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T08:09:23.786 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:09:23.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:23.916 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:23.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:24.018 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:24.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:23 smithi134 ceph-mon[30086]: 3.b scrub starts 2024-02-05T08:09:24.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:23 smithi134 ceph-mon[30086]: 3.b scrub ok 2024-02-05T08:09:24.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:23 smithi134 ceph-mon[30086]: 7.2 scrub ok 2024-02-05T08:09:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:09:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:23 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:23 smithi134 ceph-mon[39278]: 3.b scrub starts 2024-02-05T08:09:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:23 smithi134 ceph-mon[39278]: 3.b scrub ok 2024-02-05T08:09:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:23 smithi134 ceph-mon[39278]: 7.2 scrub ok 2024-02-05T08:09:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:09:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:23 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:24.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:24.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:24.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:24.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:24.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:24.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:24.233 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:24.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:24.343 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:24.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:24.445 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:24.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:24.494 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:24.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:24.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T08:09:24.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:24.631 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:24.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:24.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:24.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:24.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:24.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:24 smithi196 ceph-mon[33018]: 5.8 scrub starts 2024-02-05T08:09:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:24 smithi196 ceph-mon[33018]: 5.8 scrub ok 2024-02-05T08:09:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:24 smithi196 ceph-mon[33018]: pgmap v181: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T08:09:24.860 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:24.904 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:24.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:24.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:25.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:25.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:24 smithi134 ceph-mon[30086]: 5.8 scrub starts 2024-02-05T08:09:25.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:24 smithi134 ceph-mon[30086]: 5.8 scrub ok 2024-02-05T08:09:25.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:24 smithi134 ceph-mon[30086]: pgmap v181: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T08:09:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:24 smithi134 ceph-mon[39278]: 5.8 scrub starts 2024-02-05T08:09:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:24 smithi134 ceph-mon[39278]: 5.8 scrub ok 2024-02-05T08:09:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:24 smithi134 ceph-mon[39278]: pgmap v181: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T08:09:25.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:25.062 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:25.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:25.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:25.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:25.214 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:25.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:25.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:25.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:25.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:25.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:25.351 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:09:25.443 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:25.482 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:25.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:25.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:25.584 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:25.650 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:25.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:25.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:25.752 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:25.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:25.798 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:25.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:25.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:25.926 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:25.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:26.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:25 smithi134 ceph-mon[30086]: 4.1e scrub starts 2024-02-05T08:09:26.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:25 smithi134 ceph-mon[30086]: 4.1e scrub ok 2024-02-05T08:09:26.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:25 smithi134 ceph-mon[30086]: 3.14 scrub starts 2024-02-05T08:09:26.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:25 smithi134 ceph-mon[30086]: 3.14 scrub ok 2024-02-05T08:09:26.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:25 smithi134 ceph-mon[39278]: 4.1e scrub starts 2024-02-05T08:09:26.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:25 smithi134 ceph-mon[39278]: 4.1e scrub ok 2024-02-05T08:09:26.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:25 smithi134 ceph-mon[39278]: 3.14 scrub starts 2024-02-05T08:09:26.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:25 smithi134 ceph-mon[39278]: 3.14 scrub ok 2024-02-05T08:09:26.028 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:25 smithi196 ceph-mon[33018]: 4.1e scrub starts 2024-02-05T08:09:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:25 smithi196 ceph-mon[33018]: 4.1e scrub ok 2024-02-05T08:09:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:25 smithi196 ceph-mon[33018]: 3.14 scrub starts 2024-02-05T08:09:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:25 smithi196 ceph-mon[33018]: 3.14 scrub ok 2024-02-05T08:09:26.072 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:26.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:26.076 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:26.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:26.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:26.230 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:26.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:26.332 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:26.359 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:26.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:26.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:26.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:26.515 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:26.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:26.618 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:26.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:26.662 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:26.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:26.712 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:09:26.763 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:26.813 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:26.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:26.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:26.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:26.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:26.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:26.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:26.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: 7.4 scrub starts 2024-02-05T08:09:26.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: 6.b scrub starts 2024-02-05T08:09:26.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: 6.1d scrub starts 2024-02-05T08:09:26.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: 6.1d scrub ok 2024-02-05T08:09:26.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: 6.b scrub ok 2024-02-05T08:09:26.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: 4.1f scrub starts 2024-02-05T08:09:26.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: 4.1f scrub ok 2024-02-05T08:09:26.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:09:26.956 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:09:26.956 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: pgmap v182: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 986 op/s 2024-02-05T08:09:26.956 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: 6.1e scrub starts 2024-02-05T08:09:26.956 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: 6.1e scrub ok 2024-02-05T08:09:26.956 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: 2.0 scrub starts 2024-02-05T08:09:26.956 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[30086]: 2.0 scrub ok 2024-02-05T08:09:26.956 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: 7.4 scrub starts 2024-02-05T08:09:26.956 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: 6.b scrub starts 2024-02-05T08:09:26.956 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: 6.1d scrub starts 2024-02-05T08:09:26.956 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: 6.1d scrub ok 2024-02-05T08:09:26.956 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: 6.b scrub ok 2024-02-05T08:09:26.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: 4.1f scrub starts 2024-02-05T08:09:26.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: 4.1f scrub ok 2024-02-05T08:09:26.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:09:26.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:09:26.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: pgmap v182: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 986 op/s 2024-02-05T08:09:26.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: 6.1e scrub starts 2024-02-05T08:09:26.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: 6.1e scrub ok 2024-02-05T08:09:26.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: 2.0 scrub starts 2024-02-05T08:09:26.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:26 smithi134 ceph-mon[39278]: 2.0 scrub ok 2024-02-05T08:09:27.043 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:27.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: 7.4 scrub starts 2024-02-05T08:09:27.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: 6.b scrub starts 2024-02-05T08:09:27.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: 6.1d scrub starts 2024-02-05T08:09:27.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: 6.1d scrub ok 2024-02-05T08:09:27.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: 6.b scrub ok 2024-02-05T08:09:27.050 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: 4.1f scrub starts 2024-02-05T08:09:27.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: 4.1f scrub ok 2024-02-05T08:09:27.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:09:27.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:09:27.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: pgmap v182: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 986 op/s 2024-02-05T08:09:27.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: 6.1e scrub starts 2024-02-05T08:09:27.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: 6.1e scrub ok 2024-02-05T08:09:27.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: 2.0 scrub starts 2024-02-05T08:09:27.051 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:26 smithi196 ceph-mon[33018]: 2.0 scrub ok 2024-02-05T08:09:27.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:27.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:27.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:27.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:27.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:27.236 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:27.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:27.337 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:27.370 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:09:27.371 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:09:27.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:09:27.371 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:09:27.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.371 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:19167 7.0 7:0be58fc5:::benchmark_data_smithi134_83016_object19166:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:09:27.371 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:06:58.641992+0000", 2024-02-05T08:09:27.371 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 148.726276087, 2024-02-05T08:09:27.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.136069783, 2024-02-05T08:09:27.371 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.371 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.371 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.371 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.371 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 19167 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.641992+0000", 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.641992+0000", 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.641993+0000", 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.429e-06 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.642043+0000", 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9499000000000002e-05 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.642055+0000", 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2564e-05 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.642064+0000", 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3280000000000006e-06 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.642125+0000", 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1540999999999999e-05 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.642228+0000", 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000102768 2024-02-05T08:09:27.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.642332+0000", 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104149 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.643173+0000", 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084053700000000003 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.644974+0000", 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0018014839999999999 2024-02-05T08:09:27.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.777969+0000", 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13299466200000001 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.778020+0000", 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1471999999999998e-05 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.778062+0000", 2024-02-05T08:09:27.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1350000000000002e-05 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:20691 7.0 7:09ebe114:::benchmark_data_smithi134_83016_object20690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:06:59.675626+0000", 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 147.69264155600001, 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14747305399999999, 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.377 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 20691 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.675626+0000", 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.675626+0000", 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.675628+0000", 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3260000000000001e-06 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.675692+0000", 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3769000000000006e-05 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.675703+0000", 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1771e-05 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.675709+0000", 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9220000000000002e-06 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.675755+0000", 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5629e-05 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.675840+0000", 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4983999999999998e-05 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.675927+0000", 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7590999999999995e-05 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.676703+0000", 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00077519700000000004 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.676927+0000", 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022389 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.823027+0000", 2024-02-05T08:09:27.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14610082899999999 2024-02-05T08:09:27.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.823077+0000", 2024-02-05T08:09:27.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9670999999999999e-05 2024-02-05T08:09:27.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.823099+0000", 2024-02-05T08:09:27.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2475000000000001e-05 2024-02-05T08:09:27.383 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.383 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.383 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.383 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:21739 7.0 7:08521992:::benchmark_data_smithi134_83016_object21738:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:09:27.383 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:07:00.327315+0000", 2024-02-05T08:09:27.383 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 147.040953062, 2024-02-05T08:09:27.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13073753199999999, 2024-02-05T08:09:27.383 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.383 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.383 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.383 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.384 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.384 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 21739 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.327315+0000", 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.327315+0000", 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.327316+0000", 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4099999999999997e-07 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.327352+0000", 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6359999999999997e-05 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.327364+0000", 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1341e-05 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.327369+0000", 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7699999999999998e-06 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.327412+0000", 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2450999999999997e-05 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.327489+0000", 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6794999999999999e-05 2024-02-05T08:09:27.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.327594+0000", 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010545800000000001 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.328417+0000", 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082340099999999997 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.329642+0000", 2024-02-05T08:09:27.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012245310000000001 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.457979+0000", 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12833660199999999 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.458024+0000", 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5871999999999997e-05 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:07:00.458052+0000", 2024-02-05T08:09:27.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8010000000000001e-05 2024-02-05T08:09:27.390 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.390 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.390 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.390 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:129677 7.0 7:0db9f5dd:::benchmark_data_smithi134_83016_object19743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-05T08:09:27.390 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:40.361634+0000", 2024-02-05T08:09:27.390 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 47.006633557999997, 2024-02-05T08:09:27.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13172866599999999, 2024-02-05T08:09:27.390 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.390 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.390 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 129677 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.361634+0000", 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.361634+0000", 2024-02-05T08:09:27.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.361635+0000", 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1399999999999996e-07 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.361640+0000", 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9089999999999999e-06 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.361651+0000", 2024-02-05T08:09:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0997e-05 2024-02-05T08:09:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.361658+0000", 2024-02-05T08:09:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9999999999999999e-06 2024-02-05T08:09:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.448384+0000", 2024-02-05T08:09:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.086726300000000006 2024-02-05T08:09:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.448501+0000", 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116381 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.448619+0000", 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117992 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.453563+0000", 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049440730000000002 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.492500+0000", 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.038937307999999997 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.493292+0000", 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00079222600000000004 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.493334+0000", 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1699999999999997e-05 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:40.493363+0000", 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8966e-05 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:131219 7.0 7:0a5fbf5e:::benchmark_data_smithi134_83016_object21285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:42.092019+0000", 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 45.276249225999997, 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.21407338000000001, 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.396 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 131219 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.092019+0000", 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.092019+0000", 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.092020+0000", 2024-02-05T08:09:27.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.638e-06 2024-02-05T08:09:27.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.092026+0000", 2024-02-05T08:09:27.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3959999999999997e-06 2024-02-05T08:09:27.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.092039+0000", 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3655999999999999e-05 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.092048+0000", 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8470000000000002e-06 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.118291+0000", 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.026242999999999999 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.118379+0000", 2024-02-05T08:09:27.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8046000000000004e-05 2024-02-05T08:09:27.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.118453+0000", 2024-02-05T08:09:27.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3709999999999997e-05 2024-02-05T08:09:27.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.122773+0000", 2024-02-05T08:09:27.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043195760000000003 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.171921+0000", 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.049148385000000003 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.305963+0000", 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13404182000000001 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.306029+0000", 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6192999999999995e-05 2024-02-05T08:09:27.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.306092+0000", 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3113000000000004e-05 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:131230 7.0 7:08d1f14e:::benchmark_data_smithi134_83016_object21296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:42.105972+0000", 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 45.262296235000001, 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.20050992200000001, 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.402 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 131230 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.105972+0000", 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.105972+0000", 2024-02-05T08:09:27.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.105973+0000", 2024-02-05T08:09:27.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.683e-06 2024-02-05T08:09:27.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.105978+0000", 2024-02-05T08:09:27.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0409999999999997e-06 2024-02-05T08:09:27.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.105991+0000", 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2169e-05 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.105998+0000", 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7069999999999992e-06 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.120706+0000", 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014707886999999999 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.120825+0000", 2024-02-05T08:09:27.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011919200000000001 2024-02-05T08:09:27.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.120931+0000", 2024-02-05T08:09:27.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010571800000000001 2024-02-05T08:09:27.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.122902+0000", 2024-02-05T08:09:27.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019710299999999999 2024-02-05T08:09:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.174034+0000", 2024-02-05T08:09:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.051131395000000003 2024-02-05T08:09:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.306374+0000", 2024-02-05T08:09:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13234056699999999 2024-02-05T08:09:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.306430+0000", 2024-02-05T08:09:27.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5535000000000003e-05 2024-02-05T08:09:27.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.306482+0000", 2024-02-05T08:09:27.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1997999999999999e-05 2024-02-05T08:09:27.408 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.408 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.408 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.408 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:131260 7.0 7:00672691:::benchmark_data_smithi134_83016_object21326:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:42.146218+0000", 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 45.222050447000001, 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16043051899999999, 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 131260 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.146218+0000", 2024-02-05T08:09:27.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.146218+0000", 2024-02-05T08:09:27.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.146219+0000", 2024-02-05T08:09:27.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.265e-06 2024-02-05T08:09:27.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.146223+0000", 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5650000000000003e-06 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.146234+0000", 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0998e-05 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.146242+0000", 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8159999999999994e-06 2024-02-05T08:09:27.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.146595+0000", 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000352968 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.146677+0000", 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2262000000000006e-05 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.146759+0000", 2024-02-05T08:09:27.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1893999999999999e-05 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.149205+0000", 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002445876 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.174063+0000", 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.024857707999999999 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.306534+0000", 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13247156199999999 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.306595+0000", 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0455999999999999e-05 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.306648+0000", 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3149000000000002e-05 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:131479 7.0 7:0775a329:::benchmark_data_smithi134_83016_object21545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:42.437642+0000", 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 44.930625898999999, 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.146340637, 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 131479 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.437642+0000", 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.437642+0000", 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.437643+0000", 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.395e-06 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.437649+0000", 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.485e-06 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.437663+0000", 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4062e-05 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.437673+0000", 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5529999999999998e-06 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.455672+0000", 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017999864000000001 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.455756+0000", 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3992000000000005e-05 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.455834+0000", 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7922000000000004e-05 2024-02-05T08:09:27.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.459305+0000", 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034701889999999998 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.557811+0000", 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.09850631 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.583897+0000", 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.026085916000000001 2024-02-05T08:09:27.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.583934+0000", 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7283000000000002e-05 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.583983+0000", 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8665999999999998e-05 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:131489 7.0 7:0a3bca1e:::benchmark_data_smithi134_83016_object21555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:42.445271+0000", 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 44.922997090000003, 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13880785600000001, 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 131489 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.445271+0000", 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.445271+0000", 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.445272+0000", 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.299e-06 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.445281+0000", 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3159999999999997e-06 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.445297+0000", 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6430999999999999e-05 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.445305+0000", 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6690000000000006e-06 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.457156+0000", 2024-02-05T08:09:27.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011851482999999999 2024-02-05T08:09:27.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.457280+0000", 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000124337 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.457419+0000", 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000138441 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.459444+0000", 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020255329999999999 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:27.428 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:27.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.557838+0000", 2024-02-05T08:09:27.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.098393464999999999 2024-02-05T08:09:27.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.584019+0000", 2024-02-05T08:09:27.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.026181171999999999 2024-02-05T08:09:27.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.584049+0000", 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0423999999999999e-05 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.584079+0000", 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9286e-05 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:131548 7.0 7:0d75c80b:::benchmark_data_smithi134_83016_object21614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:42.533665+0000", 2024-02-05T08:09:27.429 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 44.834602730999997, 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13361277299999999, 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 131548 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533665+0000", 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533665+0000", 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533666+0000", 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0750000000000001e-06 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533671+0000", 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6820000000000002e-06 2024-02-05T08:09:27.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533683+0000", 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2142e-05 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533692+0000", 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4670000000000004e-06 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533738+0000", 2024-02-05T08:09:27.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6774999999999999e-05 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533831+0000", 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2844999999999996e-05 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533911+0000", 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0154999999999994e-05 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.537546+0000", 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036343450000000002 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.612663+0000", 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.075117580000000003 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.667158+0000", 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.054494854000000002 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.667225+0000", 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6459999999999997e-05 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.667278+0000", 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3393000000000002e-05 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:131549 7.0 7:062037b3:::benchmark_data_smithi134_83016_object21615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:42.533950+0000", 2024-02-05T08:09:27.435 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 44.834318283999998, 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13344335900000001, 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 131549 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533950+0000", 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533950+0000", 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533951+0000", 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3300000000000001e-07 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533958+0000", 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4080000000000001e-06 2024-02-05T08:09:27.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533969+0000", 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1177e-05 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.533977+0000", 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2220000000000002e-06 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.535928+0000", 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00195061 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.536013+0000", 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5315999999999994e-05 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.536088+0000", 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4318000000000002e-05 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.537686+0000", 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015982399999999999 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.612698+0000", 2024-02-05T08:09:27.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.075012624 2024-02-05T08:09:27.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.667320+0000", 2024-02-05T08:09:27.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.054621952000000001 2024-02-05T08:09:27.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.667361+0000", 2024-02-05T08:09:27.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1063000000000003e-05 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:42.667393+0000", 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1596e-05 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:146056 7.0 7:0528c05c:::benchmark_data_smithi134_83016_object36122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:00.708650+0000", 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 26.659617880999999, 2024-02-05T08:09:27.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15959896300000001, 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 146056 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.708650+0000", 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.708650+0000", 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.708652+0000", 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.556e-06 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.708660+0000", 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3240000000000003e-06 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.708673+0000", 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3444999999999999e-05 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.708682+0000", 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0900000000000005e-06 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.741604+0000", 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.032922541 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.741701+0000", 2024-02-05T08:09:27.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.645e-05 2024-02-05T08:09:27.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.741808+0000", 2024-02-05T08:09:27.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107749 2024-02-05T08:09:27.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.743552+0000", 2024-02-05T08:09:27.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017439020000000001 2024-02-05T08:09:27.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.828389+0000", 2024-02-05T08:09:27.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.084836461000000002 2024-02-05T08:09:27.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.868172+0000", 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039783157999999999 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.868212+0000", 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0136999999999999e-05 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:00.868249+0000", 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7150000000000002e-05 2024-02-05T08:09:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:146329 7.0 7:09d9dc48:::benchmark_data_smithi134_83016_object36395:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:01.047433+0000", 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 26.320834877999999, 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13077583100000001, 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.450 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 146329 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.047433+0000", 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.047433+0000", 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.047434+0000", 2024-02-05T08:09:27.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7499999999999998e-07 2024-02-05T08:09:27.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.047439+0000", 2024-02-05T08:09:27.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4190000000000004e-06 2024-02-05T08:09:27.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.047452+0000", 2024-02-05T08:09:27.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3655999999999999e-05 2024-02-05T08:09:27.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.047463+0000", 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0353999999999999e-05 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.084776+0000", 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037313239999999998 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.084862+0000", 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6452000000000005e-05 2024-02-05T08:09:27.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.084940+0000", 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7628999999999998e-05 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.086529+0000", 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015891670000000001 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.152519+0000", 2024-02-05T08:09:27.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065990336999999996 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.178144+0000", 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.025624470999999999 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.178180+0000", 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6609000000000001e-05 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.178209+0000", 2024-02-05T08:09:27.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8521999999999999e-05 2024-02-05T08:09:27.456 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.456 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.456 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.456 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:147104 7.0 7:070cf0b4:::benchmark_data_smithi134_83016_object37170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-02-05T08:09:27.456 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:01.909737+0000", 2024-02-05T08:09:27.456 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.458531169, 2024-02-05T08:09:27.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14311674699999999, 2024-02-05T08:09:27.456 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.456 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.456 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 147104 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.909737+0000", 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.909737+0000", 2024-02-05T08:09:27.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.909738+0000", 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.066e-06 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.909742+0000", 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0169999999999998e-06 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.909753+0000", 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1539000000000001e-05 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.909761+0000", 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7719999999999994e-06 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.937678+0000", 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.027916782000000001 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.937803+0000", 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012534800000000001 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.937942+0000", 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013912399999999999 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.941887+0000", 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039443489999999998 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.025387+0000", 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.083500397000000004 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.052736+0000", 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.027349091999999998 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.052799+0000", 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3112000000000002e-05 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.052854+0000", 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4148999999999999e-05 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:147109 7.0 7:03a18faa:::benchmark_data_smithi134_83016_object37175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:01.920181+0000", 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.44808681, 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13278246199999999, 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.462 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 147109 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.920181+0000", 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.920181+0000", 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.920182+0000", 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.071e-06 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.920186+0000", 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1030000000000004e-06 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.920197+0000", 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1134e-05 2024-02-05T08:09:27.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.920204+0000", 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2519999999999996e-06 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.939242+0000", 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019038513 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.939315+0000", 2024-02-05T08:09:27.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2849000000000007e-05 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.939409+0000", 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3628999999999994e-05 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.941890+0000", 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024811579999999998 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.028004+0000", 2024-02-05T08:09:27.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.086113822000000007 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.052894+0000", 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.024889960999999999 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.052927+0000", 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3751999999999998e-05 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.052964+0000", 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6217999999999998e-05 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:147110 7.0 7:042c2798:::benchmark_data_smithi134_83016_object37176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:01.920220+0000", 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.448048335999999, 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13290571900000001, 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.468 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 147110 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.920220+0000", 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.920220+0000", 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.920220+0000", 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1600000000000001e-07 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.920223+0000", 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5040000000000001e-06 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.920229+0000", 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8059999999999997e-06 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.920235+0000", 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9089999999999997e-06 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.940630+0000", 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020394981 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.940709+0000", 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8283999999999998e-05 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.940786+0000", 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7113000000000005e-05 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.941994+0000", 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001207931 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.031953+0000", 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.089959771999999993 2024-02-05T08:09:27.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.053021+0000", 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021067191999999998 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.053076+0000", 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5714999999999998e-05 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.053125+0000", 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8996000000000003e-05 2024-02-05T08:09:27.473 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:147162 7.0 7:00cb83f3:::benchmark_data_smithi134_83016_object37228:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:01.983251+0000", 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.385016948000001, 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.138733472, 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.474 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 147162 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.983251+0000", 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.983251+0000", 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.983253+0000", 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4839999999999999e-06 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.983258+0000", 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0730000000000004e-06 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.983272+0000", 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4443e-05 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.983279+0000", 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2490000000000004e-06 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.983387+0000", 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107756 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.983501+0000", 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011421600000000001 2024-02-05T08:09:27.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.983606+0000", 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104546 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:01.986217+0000", 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026113690000000001 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.038737+0000", 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.052520318000000003 2024-02-05T08:09:27.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.121866+0000", 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.083128446999999994 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.121927+0000", 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1208000000000001e-05 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:02.121985+0000", 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7362999999999999e-05 2024-02-05T08:09:27.479 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:148743 7.0 7:0bbb042f:::benchmark_data_smithi134_83016_object38809:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:03.734739+0000", 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.633528908999999, 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13517489999999999, 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 148743 2024-02-05T08:09:27.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.734739+0000", 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.734739+0000", 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.734740+0000", 2024-02-05T08:09:27.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1829999999999999e-06 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.734745+0000", 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9699999999999998e-06 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.734757+0000", 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1780000000000001e-05 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.734766+0000", 2024-02-05T08:09:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3999999999999998e-06 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.735114+0000", 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00034715100000000002 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.735205+0000", 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1027999999999997e-05 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.735283+0000", 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8798e-05 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.737463+0000", 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021798569999999999 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.741579+0000", 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041153170000000003 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.869794+0000", 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.128215308 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.869856+0000", 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1675999999999997e-05 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:03.869914+0000", 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8431999999999999e-05 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.485 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:149235 7.0 7:0f6b4ad8:::benchmark_data_smithi134_83016_object39301:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:04.309820+0000", 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.058448180999999, 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.140838515, 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 149235 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.486 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.309820+0000", 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.309820+0000", 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.309821+0000", 2024-02-05T08:09:27.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3239999999999999e-06 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.309827+0000", 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.592e-06 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.309839+0000", 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2089000000000001e-05 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.309847+0000", 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4959999999999996e-06 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.348614+0000", 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.038766950000000001 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.348726+0000", 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000111387 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.348842+0000", 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116055 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.353116+0000", 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042743210000000002 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.417754+0000", 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.064638340000000002 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.450574+0000", 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.032819598999999998 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.450610+0000", 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6436999999999997e-05 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.450658+0000", 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7924999999999999e-05 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.491 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:149244 7.0 7:0803a78c:::benchmark_data_smithi134_83016_object39310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:04.319349+0000", 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 23.048918613000001, 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13139463000000001, 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 149244 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.492 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.319349+0000", 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.319349+0000", 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.319352+0000", 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1289999999999999e-06 2024-02-05T08:09:27.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.319357+0000", 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6060000000000002e-06 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.319376+0000", 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9108999999999999e-05 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.319388+0000", 2024-02-05T08:09:27.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1601e-05 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.350666+0000", 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.031278621999999999 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.350755+0000", 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8300999999999994e-05 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.350831+0000", 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6695999999999999e-05 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.353124+0000", 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022922609999999999 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.418796+0000", 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065672382000000001 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.450683+0000", 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.031886558000000002 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.450714+0000", 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1838000000000002e-05 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:04.450744+0000", 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9527e-05 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.498 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.498 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:09:27.498 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:09:27.498 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:09:27.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:27.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:27.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:27.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:27.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:27.755 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:27.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:27.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:27.857 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:27.894 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:27.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:27.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:27.996 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:27 smithi134 ceph-mon[30086]: 7.4 scrub ok 2024-02-05T08:09:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:27 smithi134 ceph-mon[30086]: 3.1a scrub starts 2024-02-05T08:09:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:27 smithi134 ceph-mon[30086]: 3.1a scrub ok 2024-02-05T08:09:28.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:09:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:09:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:09:28.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:27 smithi134 ceph-mon[39278]: 7.4 scrub ok 2024-02-05T08:09:28.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:27 smithi134 ceph-mon[39278]: 3.1a scrub starts 2024-02-05T08:09:28.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:27 smithi134 ceph-mon[39278]: 3.1a scrub ok 2024-02-05T08:09:28.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:28.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:28.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:28.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:27 smithi196 ceph-mon[33018]: 7.4 scrub ok 2024-02-05T08:09:28.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:27 smithi196 ceph-mon[33018]: 3.1a scrub starts 2024-02-05T08:09:28.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:27 smithi196 ceph-mon[33018]: 3.1a scrub ok 2024-02-05T08:09:28.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:28.169 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:28.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:28.271 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:28.322 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:28.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:28.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T08:09:28.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:28.469 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:28.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:28.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:28.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:28.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:28.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:28.735 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:28.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:28.775 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:09:28.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:28.875 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:09:28.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:28.882 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:28.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:28.984 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:29.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:28 smithi196 ceph-mon[33018]: pgmap v183: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 638 op/s 2024-02-05T08:09:29.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:28 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/743114248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:29.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:29.026 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:29.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:29.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:28 smithi134 ceph-mon[30086]: pgmap v183: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 638 op/s 2024-02-05T08:09:29.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:28 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/743114248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:29.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:28 smithi134 ceph-mon[39278]: pgmap v183: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 638 op/s 2024-02-05T08:09:29.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:28 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/743114248' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:29.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:29.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:29.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:29.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:29.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:29.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:29.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:29.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:29.445 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:29.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:29.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:29.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:29.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:29.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:29.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:29.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:29.718 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:29.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:29.819 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:29.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:29.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:29.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:29.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:29.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:29.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:29.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:30.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:29 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:09:30.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:29 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:09:30.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:29 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/743114248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:09:30.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:29 smithi134 ceph-mon[30086]: osdmap e140: 8 total, 8 up, 7 in 2024-02-05T08:09:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:29 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:09:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:29 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:09:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:29 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/743114248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:09:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:29 smithi134 ceph-mon[39278]: osdmap e140: 8 total, 8 up, 7 in 2024-02-05T08:09:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:29 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:09:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:29 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:09:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:29 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/743114248' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:09:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:29 smithi196 ceph-mon[33018]: osdmap e140: 8 total, 8 up, 7 in 2024-02-05T08:09:30.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:30.149 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:30.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:30.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:30.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:30.278 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:30.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:30.378 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:30.423 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:30.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:30.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:30.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:30.575 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:30.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:30.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:30.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:30.718 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:30.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:30.804 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:30.819 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:30.880 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:30.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:30.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[30086]: 6.11 deep-scrub starts 2024-02-05T08:09:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[30086]: 6.11 deep-scrub ok 2024-02-05T08:09:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[30086]: 5.b scrub starts 2024-02-05T08:09:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[30086]: pgmap v185: 148 pgs: 148 active+clean; 3.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1k op/s 2024-02-05T08:09:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[30086]: 5.b scrub ok 2024-02-05T08:09:31.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[30086]: 4.1b scrub starts 2024-02-05T08:09:31.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[30086]: 4.1b scrub ok 2024-02-05T08:09:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[39278]: 6.11 deep-scrub starts 2024-02-05T08:09:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[39278]: 6.11 deep-scrub ok 2024-02-05T08:09:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[39278]: 5.b scrub starts 2024-02-05T08:09:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[39278]: pgmap v185: 148 pgs: 148 active+clean; 3.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1k op/s 2024-02-05T08:09:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[39278]: 5.b scrub ok 2024-02-05T08:09:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[39278]: 4.1b scrub starts 2024-02-05T08:09:31.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:30 smithi134 ceph-mon[39278]: 4.1b scrub ok 2024-02-05T08:09:31.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:31.030 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:31.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:30 smithi196 ceph-mon[33018]: 6.11 deep-scrub starts 2024-02-05T08:09:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:30 smithi196 ceph-mon[33018]: 6.11 deep-scrub ok 2024-02-05T08:09:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:30 smithi196 ceph-mon[33018]: 5.b scrub starts 2024-02-05T08:09:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:30 smithi196 ceph-mon[33018]: pgmap v185: 148 pgs: 148 active+clean; 3.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1k op/s 2024-02-05T08:09:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:30 smithi196 ceph-mon[33018]: 5.b scrub ok 2024-02-05T08:09:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:30 smithi196 ceph-mon[33018]: 4.1b scrub starts 2024-02-05T08:09:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:30 smithi196 ceph-mon[33018]: 4.1b scrub ok 2024-02-05T08:09:31.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:31.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:31.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:31.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:31.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:31.286 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:31.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:31.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:31.441 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:31.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:31.542 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:31.616 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:31.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:31.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:31.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:31.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:31.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:31.786 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.4 to 1 (10000) 2024-02-05T08:09:31.848 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:31.874 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:31.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:31.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:32.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:32.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:31 smithi196 ceph-mon[33018]: 4.9 scrub starts 2024-02-05T08:09:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:31 smithi196 ceph-mon[33018]: 4.9 scrub ok 2024-02-05T08:09:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:31 smithi196 ceph-mon[33018]: 3.d scrub starts 2024-02-05T08:09:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:31 smithi196 ceph-mon[33018]: 3.d scrub ok 2024-02-05T08:09:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:31 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3070082802' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:09:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:31 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:09:32.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:32.140 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:32.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:32.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:31 smithi134 ceph-mon[30086]: 4.9 scrub starts 2024-02-05T08:09:32.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:31 smithi134 ceph-mon[30086]: 4.9 scrub ok 2024-02-05T08:09:32.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:31 smithi134 ceph-mon[30086]: 3.d scrub starts 2024-02-05T08:09:32.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:31 smithi134 ceph-mon[30086]: 3.d scrub ok 2024-02-05T08:09:32.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:31 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3070082802' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:09:32.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:31 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:09:32.162 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:31 smithi134 ceph-mon[39278]: 4.9 scrub starts 2024-02-05T08:09:32.162 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:31 smithi134 ceph-mon[39278]: 4.9 scrub ok 2024-02-05T08:09:32.163 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:31 smithi134 ceph-mon[39278]: 3.d scrub starts 2024-02-05T08:09:32.163 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:31 smithi134 ceph-mon[39278]: 3.d scrub ok 2024-02-05T08:09:32.163 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:31 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3070082802' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:09:32.163 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:31 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:09:32.242 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:32.283 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:32.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:32.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:32.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:32.430 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:32.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:32.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:32.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:32.554 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:09:32.560 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:32.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:32.648 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:09:32.649 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:09:32.649 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T08:09:32.649 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:09:32.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:32.720 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:32.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:32.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:32.846 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:32.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:32.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:32.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:33.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:33.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:33.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:33.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 5 1 2024-02-05T08:09:33.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:32 smithi196 ceph-mon[33018]: pgmap v186: 148 pgs: 148 active+clean; 3.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 886 op/s 2024-02-05T08:09:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:32 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:09:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:32 smithi196 ceph-mon[33018]: osdmap e141: 8 total, 8 up, 7 in 2024-02-05T08:09:33.105 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:33.129 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:32 smithi134 ceph-mon[30086]: pgmap v186: 148 pgs: 148 active+clean; 3.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 886 op/s 2024-02-05T08:09:33.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:32 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:09:33.130 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:32 smithi134 ceph-mon[30086]: osdmap e141: 8 total, 8 up, 7 in 2024-02-05T08:09:33.130 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:32 smithi134 ceph-mon[39278]: pgmap v186: 148 pgs: 148 active+clean; 3.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 886 op/s 2024-02-05T08:09:33.130 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:32 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:09:33.130 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:32 smithi134 ceph-mon[39278]: osdmap e141: 8 total, 8 up, 7 in 2024-02-05T08:09:33.159 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:33.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:33.261 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:33.308 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T08:09:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:33.327 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:33.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:33.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:33.472 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:33.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:33.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:09:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:09:33.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:33.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:33.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:33.700 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:33.707 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:33.754 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:09:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:33.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:33.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:33.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:33.942 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:33 smithi134 ceph-mon[39278]: 6.c scrub starts 2024-02-05T08:09:33.942 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:33 smithi134 ceph-mon[39278]: 6.c scrub ok 2024-02-05T08:09:33.943 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:33 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:33.943 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:33 smithi134 ceph-mon[39278]: 7.b scrub starts 2024-02-05T08:09:33.943 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:33 smithi134 ceph-mon[39278]: 6.15 scrub starts 2024-02-05T08:09:33.943 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:33 smithi134 ceph-mon[39278]: 6.15 scrub ok 2024-02-05T08:09:33.943 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:33 smithi134 ceph-mon[30086]: 6.c scrub starts 2024-02-05T08:09:33.943 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:33 smithi134 ceph-mon[30086]: 6.c scrub ok 2024-02-05T08:09:33.943 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:33 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:33.943 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:33 smithi134 ceph-mon[30086]: 7.b scrub starts 2024-02-05T08:09:33.943 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:33 smithi134 ceph-mon[30086]: 6.15 scrub starts 2024-02-05T08:09:33.944 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:33 smithi134 ceph-mon[30086]: 6.15 scrub ok 2024-02-05T08:09:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:33.947 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:33.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:34.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:33 smithi196 ceph-mon[33018]: 6.c scrub starts 2024-02-05T08:09:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:33 smithi196 ceph-mon[33018]: 6.c scrub ok 2024-02-05T08:09:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:33 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:33 smithi196 ceph-mon[33018]: 7.b scrub starts 2024-02-05T08:09:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:33 smithi196 ceph-mon[33018]: 6.15 scrub starts 2024-02-05T08:09:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:33 smithi196 ceph-mon[33018]: 6.15 scrub ok 2024-02-05T08:09:34.078 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:34.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:34.180 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:34.234 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:34.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:34.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:34.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:34.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:34.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:34.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:34.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:34.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:34.504 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:34.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:34.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:34.631 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:34.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:34.733 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:34.758 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:34.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:34.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:34.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:34.916 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:34.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:35.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:35.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:35.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:34 smithi196 ceph-mon[33018]: 4.0 scrub starts 2024-02-05T08:09:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:34 smithi196 ceph-mon[33018]: 4.0 scrub ok 2024-02-05T08:09:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:34 smithi196 ceph-mon[33018]: pgmap v188: 148 pgs: 148 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T08:09:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:34 smithi196 ceph-mon[33018]: 7.b scrub ok 2024-02-05T08:09:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:34 smithi196 ceph-mon[33018]: 7.b scrub starts 2024-02-05T08:09:35.064 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:34 smithi134 ceph-mon[30086]: 4.0 scrub starts 2024-02-05T08:09:35.065 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:34 smithi134 ceph-mon[30086]: 4.0 scrub ok 2024-02-05T08:09:35.065 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:34 smithi134 ceph-mon[30086]: pgmap v188: 148 pgs: 148 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T08:09:35.065 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:34 smithi134 ceph-mon[30086]: 7.b scrub ok 2024-02-05T08:09:35.065 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:34 smithi134 ceph-mon[30086]: 7.b scrub starts 2024-02-05T08:09:35.065 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:34 smithi134 ceph-mon[39278]: 4.0 scrub starts 2024-02-05T08:09:35.065 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:34 smithi134 ceph-mon[39278]: 4.0 scrub ok 2024-02-05T08:09:35.065 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:34 smithi134 ceph-mon[39278]: pgmap v188: 148 pgs: 148 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T08:09:35.065 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:34 smithi134 ceph-mon[39278]: 7.b scrub ok 2024-02-05T08:09:35.065 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:34 smithi134 ceph-mon[39278]: 7.b scrub starts 2024-02-05T08:09:35.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:35.168 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:35.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:35.270 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:35.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:35.316 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:35.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:35.320 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:09:35.331 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:09:35.373 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:09:35.413 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:09:35.417 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:35.455 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:09:35.459 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:35.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:35.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:35.493 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:09:35.529 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:09:35.562 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:35.572 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:09:35.584 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:09:35.584 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:09:35.584 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:09:35.584 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:09:35.584 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:09:35.584 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:09:35.584 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:09:35.585 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:09:35.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:35.599 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:35.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:35.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:35.736 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:35.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:35.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:35.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:35.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:35.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:35.992 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:35.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:09:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:09:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[30086]: 7.7 deep-scrub starts 2024-02-05T08:09:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[30086]: 4.4 scrub starts 2024-02-05T08:09:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[30086]: 4.4 scrub ok 2024-02-05T08:09:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[30086]: 7.b scrub ok 2024-02-05T08:09:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3054672869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:09:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:09:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[39278]: 7.7 deep-scrub starts 2024-02-05T08:09:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[39278]: 4.4 scrub starts 2024-02-05T08:09:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[39278]: 4.4 scrub ok 2024-02-05T08:09:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[39278]: 7.b scrub ok 2024-02-05T08:09:36.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:35 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3054672869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:35 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:09:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:35 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:09:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:35 smithi196 ceph-mon[33018]: 7.7 deep-scrub starts 2024-02-05T08:09:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:35 smithi196 ceph-mon[33018]: 4.4 scrub starts 2024-02-05T08:09:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:35 smithi196 ceph-mon[33018]: 4.4 scrub ok 2024-02-05T08:09:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:35 smithi196 ceph-mon[33018]: 7.b scrub ok 2024-02-05T08:09:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:35 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3054672869' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:36.094 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:36.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:36.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:36.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:36.222 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:36.262 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:36.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:36.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:36.400 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:36.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:36.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:36.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:36.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:36.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:36.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:36.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:36.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:36.755 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:36.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:36.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:36.926 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:36.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:37.028 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:37.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:37.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:37.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:36 smithi196 ceph-mon[33018]: pgmap v189: 148 pgs: 148 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s 2024-02-05T08:09:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:36 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:36 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:36 smithi196 ceph-mon[33018]: 5.10 deep-scrub starts 2024-02-05T08:09:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:36 smithi196 ceph-mon[33018]: 5.10 deep-scrub ok 2024-02-05T08:09:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:36 smithi196 ceph-mon[33018]: 5.11 scrub starts 2024-02-05T08:09:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:36 smithi196 ceph-mon[33018]: 5.11 scrub ok 2024-02-05T08:09:37.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[30086]: pgmap v189: 148 pgs: 148 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s 2024-02-05T08:09:37.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:37.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:37.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[30086]: 5.10 deep-scrub starts 2024-02-05T08:09:37.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[30086]: 5.10 deep-scrub ok 2024-02-05T08:09:37.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[30086]: 5.11 scrub starts 2024-02-05T08:09:37.090 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[30086]: 5.11 scrub ok 2024-02-05T08:09:37.090 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[39278]: pgmap v189: 148 pgs: 148 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s 2024-02-05T08:09:37.090 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:37.091 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:37.091 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[39278]: 5.10 deep-scrub starts 2024-02-05T08:09:37.091 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[39278]: 5.10 deep-scrub ok 2024-02-05T08:09:37.091 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[39278]: 5.11 scrub starts 2024-02-05T08:09:37.091 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:36 smithi134 ceph-mon[39278]: 5.11 scrub ok 2024-02-05T08:09:37.157 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:37.186 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:37.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:37.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:37.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:37.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:37.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:37.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:37.442 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:37.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:37.543 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:37.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:37.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:37.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:37.714 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:37.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:37.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:37.763 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:09:37.763 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:09:37.816 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:37.856 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:37.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:37.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:37.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:37.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:37.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:38.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:09:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:09:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[30086]: 3.7 scrub starts 2024-02-05T08:09:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[30086]: 3.7 scrub ok 2024-02-05T08:09:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[30086]: 4.15 scrub starts 2024-02-05T08:09:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[30086]: 4.15 scrub ok 2024-02-05T08:09:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:09:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[30086]: 7.6 scrub starts 2024-02-05T08:09:38.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[39278]: 3.7 scrub starts 2024-02-05T08:09:38.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[39278]: 3.7 scrub ok 2024-02-05T08:09:38.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[39278]: 4.15 scrub starts 2024-02-05T08:09:38.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[39278]: 4.15 scrub ok 2024-02-05T08:09:38.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:38.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:09:38.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:37 smithi134 ceph-mon[39278]: 7.6 scrub starts 2024-02-05T08:09:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:37 smithi196 ceph-mon[33018]: 3.7 scrub starts 2024-02-05T08:09:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:37 smithi196 ceph-mon[33018]: 3.7 scrub ok 2024-02-05T08:09:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:37 smithi196 ceph-mon[33018]: 4.15 scrub starts 2024-02-05T08:09:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:37 smithi196 ceph-mon[33018]: 4.15 scrub ok 2024-02-05T08:09:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:37 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:37 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:09:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:37 smithi196 ceph-mon[33018]: 7.6 scrub starts 2024-02-05T08:09:38.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:38.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:38.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:38.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:38.226 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:38.249 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:38.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:38.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:38.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:38.394 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:38.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:38.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:38.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:38.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T08:09:38.524 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:38.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:38.626 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:38.701 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:38.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:38.792 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:09:38.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:38.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:38.841 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:38.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:38.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:38.978 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:38.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:38.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:39.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:39.108 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:39.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:39.128 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:38 smithi196 ceph-mon[33018]: 6.1b scrub starts 2024-02-05T08:09:39.128 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:38 smithi196 ceph-mon[33018]: 6.1b scrub ok 2024-02-05T08:09:39.128 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:38 smithi196 ceph-mon[33018]: pgmap v190: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 647 op/s 2024-02-05T08:09:39.128 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:38 smithi196 ceph-mon[33018]: 7.6 scrub ok 2024-02-05T08:09:39.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:39.235 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:39.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:39.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:39.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:38 smithi134 ceph-mon[30086]: 6.1b scrub starts 2024-02-05T08:09:39.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:38 smithi134 ceph-mon[30086]: 6.1b scrub ok 2024-02-05T08:09:39.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:38 smithi134 ceph-mon[30086]: pgmap v190: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 647 op/s 2024-02-05T08:09:39.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:38 smithi134 ceph-mon[30086]: 7.6 scrub ok 2024-02-05T08:09:39.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:38 smithi134 ceph-mon[39278]: 6.1b scrub starts 2024-02-05T08:09:39.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:38 smithi134 ceph-mon[39278]: 6.1b scrub ok 2024-02-05T08:09:39.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:38 smithi134 ceph-mon[39278]: pgmap v190: 148 pgs: 148 active+clean; 2.6 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 647 op/s 2024-02-05T08:09:39.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:38 smithi134 ceph-mon[39278]: 7.6 scrub ok 2024-02-05T08:09:39.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:39.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:39.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:39.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:39.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:39.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:39.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:39.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:39.631 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:39.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:39.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:39.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:39.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:39.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:39.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:39.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:39.905 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:39.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:40.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:40.048 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:40.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:39 smithi196 ceph-mon[33018]: 3.e scrub starts 2024-02-05T08:09:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:39 smithi196 ceph-mon[33018]: 3.e scrub ok 2024-02-05T08:09:40.149 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:40.188 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:40.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:40.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:39 smithi134 ceph-mon[30086]: 3.e scrub starts 2024-02-05T08:09:40.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:39 smithi134 ceph-mon[30086]: 3.e scrub ok 2024-02-05T08:09:40.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:39 smithi134 ceph-mon[39278]: 3.e scrub starts 2024-02-05T08:09:40.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:39 smithi134 ceph-mon[39278]: 3.e scrub ok 2024-02-05T08:09:40.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:40.315 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:40.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:40.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:40.417 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:40.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:40.458 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:40.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:40.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:40.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:40.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:40.686 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:40.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:40.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:40.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:40.765 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:09:40.765 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:09:40.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:40.861 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:40.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:40.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:40.883 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:40.925 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:09:40.963 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:40.988 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:40.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:40 smithi196 ceph-mon[33018]: 4.11 deep-scrub starts 2024-02-05T08:09:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:40 smithi196 ceph-mon[33018]: 4.11 deep-scrub ok 2024-02-05T08:09:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:40 smithi196 ceph-mon[33018]: 3.1d scrub starts 2024-02-05T08:09:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:40 smithi196 ceph-mon[33018]: 3.1d scrub ok 2024-02-05T08:09:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:40 smithi196 ceph-mon[33018]: pgmap v191: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 901 op/s 2024-02-05T08:09:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:40 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1357783714' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:09:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:40 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:09:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:40 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:41.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:40 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:41.090 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:41.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:41.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:41.236 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:41.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[30086]: 4.11 deep-scrub starts 2024-02-05T08:09:41.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[30086]: 4.11 deep-scrub ok 2024-02-05T08:09:41.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[30086]: 3.1d scrub starts 2024-02-05T08:09:41.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[30086]: 3.1d scrub ok 2024-02-05T08:09:41.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[30086]: pgmap v191: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 901 op/s 2024-02-05T08:09:41.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1357783714' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:09:41.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:09:41.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:41.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:41.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[39278]: 4.11 deep-scrub starts 2024-02-05T08:09:41.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[39278]: 4.11 deep-scrub ok 2024-02-05T08:09:41.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[39278]: 3.1d scrub starts 2024-02-05T08:09:41.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[39278]: 3.1d scrub ok 2024-02-05T08:09:41.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[39278]: pgmap v191: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 901 op/s 2024-02-05T08:09:41.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1357783714' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:09:41.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:09:41.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:41.280 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:40 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:41.299 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:41.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:41.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:41.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:41.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:41.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T08:09:41.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:41.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:41.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:41.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:41.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:41.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:41.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:41.828 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:09:41.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:41.899 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:41.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:41.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:42.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:42.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:42.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:42.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:42.163 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:41 smithi134 ceph-mon[30086]: 3.5 scrub starts 2024-02-05T08:09:42.163 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:41 smithi134 ceph-mon[30086]: 3.5 scrub ok 2024-02-05T08:09:42.163 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:41 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:09:42.163 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:41 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:09:42.164 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:41 smithi134 ceph-mon[30086]: osdmap e142: 8 total, 8 up, 7 in 2024-02-05T08:09:42.164 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:41 smithi134 ceph-mon[39278]: 3.5 scrub starts 2024-02-05T08:09:42.164 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:41 smithi134 ceph-mon[39278]: 3.5 scrub ok 2024-02-05T08:09:42.164 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:41 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:09:42.164 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:41 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:09:42.164 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:41 smithi134 ceph-mon[39278]: osdmap e142: 8 total, 8 up, 7 in 2024-02-05T08:09:42.169 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:42.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:42.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:42.257 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:41 smithi196 ceph-mon[33018]: 3.5 scrub starts 2024-02-05T08:09:42.257 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:41 smithi196 ceph-mon[33018]: 3.5 scrub ok 2024-02-05T08:09:42.257 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:41 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:09:42.257 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:41 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:09:42.257 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:41 smithi196 ceph-mon[33018]: osdmap e142: 8 total, 8 up, 7 in 2024-02-05T08:09:42.271 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:42.314 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:42.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:42.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:42.415 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:42.442 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:42.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:42.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:42.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:42.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:42.672 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:42.707 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:42.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:42.809 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:42.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:42.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:42.935 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:42.941 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:09:42.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:42.974 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:42.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:42 smithi196 ceph-mon[33018]: pgmap v193: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 919 op/s 2024-02-05T08:09:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:42 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:09:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:42 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:09:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:42 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:43.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:43.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:43.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:43.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:43.166 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:42 smithi134 ceph-mon[30086]: pgmap v193: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 919 op/s 2024-02-05T08:09:43.166 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:42 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:09:43.166 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:42 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:09:43.166 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:42 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:43.167 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:42 smithi134 ceph-mon[39278]: pgmap v193: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 919 op/s 2024-02-05T08:09:43.167 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:42 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:09:43.167 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:42 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:09:43.167 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:42 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:43.203 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:43.258 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:43.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:43.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:09:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:09:43.359 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:43.389 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:43.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:43.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:43.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:43.516 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:43.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:43.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:43.618 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:43.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:43.663 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:43.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:43.764 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:43.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:43.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:43.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:43.883 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:09:43.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:43.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:43.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:43.929 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:09:43.929 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:09:43.946 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:44.015 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:43 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:44.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:43 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:44.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:44.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:44.045 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:44.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:44.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:44.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:44.207 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:44.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:44.263 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:43 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:44.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:44.375 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:44.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:44.478 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:44.532 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:44.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:44.634 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:44.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:44.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:44.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T08:09:44.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:44.796 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:44.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:44.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:44.834 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:44.898 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:44.948 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:44.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:44.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:44.980 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.5 to 1 (10000) 2024-02-05T08:09:45.008 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:09:45.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:45.075 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:45.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:45.176 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:45.200 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:45.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:45.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:44 smithi134 ceph-mon[30086]: pgmap v194: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 797 op/s 2024-02-05T08:09:45.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:44 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2001009865' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:09:45.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:44 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:09:45.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:44 smithi134 ceph-mon[39278]: pgmap v194: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 797 op/s 2024-02-05T08:09:45.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:44 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2001009865' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:09:45.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:44 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:09:45.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:44 smithi196 ceph-mon[33018]: pgmap v194: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 797 op/s 2024-02-05T08:09:45.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:44 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2001009865' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:09:45.295 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:44 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:09:45.301 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:45.342 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:45.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:45.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:45.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:45.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:45.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:45.571 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:45.607 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:45.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:45.708 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:45.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:45.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:45.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:45.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:45.868 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:45.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:45.969 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:45.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:45.999 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:45.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:46.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:46.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:46.125 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:46.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:46.226 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:46.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:46.261 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:46.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:46.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:45 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:09:46.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:45 smithi134 ceph-mon[30086]: osdmap e143: 8 total, 8 up, 7 in 2024-02-05T08:09:46.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:45 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:09:46.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:45 smithi134 ceph-mon[39278]: osdmap e143: 8 total, 8 up, 7 in 2024-02-05T08:09:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:45 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:09:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:45 smithi196 ceph-mon[33018]: osdmap e143: 8 total, 8 up, 7 in 2024-02-05T08:09:46.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:46.401 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:46.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:46.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:46.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:46.531 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:46.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:46.634 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:46.660 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:46.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:46.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:46.786 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:46.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:46.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:46.929 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:46.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:47.022 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:09:47.022 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:09:47.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:47.057 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:47.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:47.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:47.159 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:47.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:47.182 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:47.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:47.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:46 smithi134 ceph-mon[30086]: pgmap v196: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 897 op/s 2024-02-05T08:09:47.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:46 smithi134 ceph-mon[39278]: pgmap v196: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 897 op/s 2024-02-05T08:09:47.284 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:46 smithi196 ceph-mon[33018]: pgmap v196: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 897 op/s 2024-02-05T08:09:47.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:47.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:47.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:47.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:47.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:47.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:47.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:47.565 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:47.589 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:47.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:47.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:47.691 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:47.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:47.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:47.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T08:09:47.817 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:47.841 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:47.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:47.943 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:47.984 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:47.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:47.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:47.998 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:09:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:09:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:09:48.060 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:09:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:47 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:09:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:47 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:09:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:47 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:09:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:47 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:09:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:47 smithi196 ceph-mon[33018]: osdmap e144: 8 total, 8 up, 7 in 2024-02-05T08:09:48.086 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:48.125 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:48.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:48.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:48.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:48.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:48.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:48.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:48.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:48 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:09:48.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:48 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:09:48.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:48 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:09:48.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:48 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:09:48.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:48 smithi134 ceph-mon[30086]: osdmap e144: 8 total, 8 up, 7 in 2024-02-05T08:09:48.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:47 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:09:48.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:48 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:09:48.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:48 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:09:48.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:48 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:09:48.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:48 smithi134 ceph-mon[39278]: osdmap e144: 8 total, 8 up, 7 in 2024-02-05T08:09:48.356 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:48.382 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:48.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:48.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:48.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:48.511 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:48.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:48.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:48.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:48.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:48.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:48.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:48.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:48.797 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:48.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:48.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:48.922 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:48.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:48.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:49.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:49.048 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:49.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:49.081 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:49 smithi134 ceph-mon[30086]: pgmap v198: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 686 op/s 2024-02-05T08:09:49.082 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:49 smithi134 ceph-mon[39278]: pgmap v198: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 686 op/s 2024-02-05T08:09:49.132 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 7 1 2024-02-05T08:09:49.150 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:49.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:49.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:49.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:49.276 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:49.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:49.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:49.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:49.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:49 smithi196 ceph-mon[33018]: pgmap v198: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.2 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 686 op/s 2024-02-05T08:09:49.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:49.430 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:49.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:49.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:49.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:49.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:49.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:49.658 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:49.704 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:49.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:49.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:49.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:49.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:49.832 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:49.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:49.933 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:49.966 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:49.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:49.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:49.980 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:09:49.980 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:09:50.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:50.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:50.094 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:50.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:50.164 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:50.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:50.238 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:50.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:50.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:50.340 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:50.384 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:50.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:50.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:50.486 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:50.510 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:50.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:50.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:50.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:50.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:50.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:50.739 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:50.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T08:09:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:50.770 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:50.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:50.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:50.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:50.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:50.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:50.999 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:51.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:51.057 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:51.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:51.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:51.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:51.217 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:51.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:51.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:51 smithi134 ceph-mon[30086]: pgmap v199: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.0 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 927 op/s 2024-02-05T08:09:51.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:51 smithi134 ceph-mon[39278]: pgmap v199: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.0 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 927 op/s 2024-02-05T08:09:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:51 smithi196 ceph-mon[33018]: pgmap v199: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.0 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 927 op/s 2024-02-05T08:09:51.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:51.345 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:51.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:51.447 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:51.473 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:51.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:51.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:51.600 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:51.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:51.702 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:51.707 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:09:51.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:51.762 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:51.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:51.762 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:09:51.813 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:09:51.854 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:09:51.863 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:51.895 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:09:51.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:51.919 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:51.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:51.934 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:09:51.978 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:09:52.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:52.027 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:09:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:52.047 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:52.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:52.047 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:09:52.048 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:09:52.048 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:09:52.048 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:09:52.048 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:09:52.048 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:09:52.048 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:09:52.048 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:09:52.122 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:52 smithi134 ceph-mon[30086]: 7.7 deep-scrub ok 2024-02-05T08:09:52.122 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:52 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1409689006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:52.122 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:52 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:52.122 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:52 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:09:52.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:52 smithi134 ceph-mon[39278]: 7.7 deep-scrub ok 2024-02-05T08:09:52.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:52 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1409689006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:52.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:52 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:52.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:52 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:09:52.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:52.193 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:52.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:52.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:52.295 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:52 smithi196 ceph-mon[33018]: 7.7 deep-scrub ok 2024-02-05T08:09:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:52 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1409689006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:09:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:52 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:09:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:52 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:09:52.332 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:52.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:52.434 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:52.482 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:52.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:52.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:52.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:52.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:52.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:52.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:52.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:52.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:52.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:52.839 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:52.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:52.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:52.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:52.978 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:53.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:53.018 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:53.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:53.045 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:09:53.051 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:53 smithi134 ceph-mon[30086]: pgmap v200: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.0 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 445 op/s 2024-02-05T08:09:53.052 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:53 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3039462941' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:53.052 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:53 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:53.052 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:53 smithi134 ceph-mon[39278]: pgmap v200: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.0 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 445 op/s 2024-02-05T08:09:53.052 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:53 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3039462941' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:53.052 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:53 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:53.119 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:53.160 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:53.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:53.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:53.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:53.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:53.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:53 smithi196 ceph-mon[33018]: pgmap v200: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 2.0 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 445 op/s 2024-02-05T08:09:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:53 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3039462941' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:53 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:09:53.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:09:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:09:53.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:53.415 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:53.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:53.517 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:53.563 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:53.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:53.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:53.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:53.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:53.794 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:53.834 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:53.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:53.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:53.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:53.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:53.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:54.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:54.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:54.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:54.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:54.096 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:54 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:09:54.096 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:54 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:09:54.096 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:54 smithi134 ceph-mon[30086]: osdmap e145: 8 total, 8 up, 7 in 2024-02-05T08:09:54.096 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:54 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:54.096 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:54 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:09:54.096 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:54 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:09:54.097 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:54 smithi134 ceph-mon[39278]: osdmap e145: 8 total, 8 up, 7 in 2024-02-05T08:09:54.097 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:54 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:54.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:54.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:54.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:54.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:54 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:09:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:54 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:09:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:54 smithi196 ceph-mon[33018]: osdmap e145: 8 total, 8 up, 7 in 2024-02-05T08:09:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:54 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:09:54.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:54.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:54.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:54.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:54.348 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:54.447 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:54.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:54.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:54.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:54.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:54.612 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:54.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:54.714 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:54.765 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:54.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:54.867 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:54.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:54.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:54.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:55.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:55.034 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:55.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:55.135 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:55.176 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:55.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:55.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:55.279 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:55.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:55 smithi134 ceph-mon[30086]: pgmap v202: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 939 op/s 2024-02-05T08:09:55.280 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:55 smithi134 ceph-mon[39278]: pgmap v202: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 939 op/s 2024-02-05T08:09:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:55 smithi196 ceph-mon[33018]: pgmap v202: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 939 op/s 2024-02-05T08:09:55.319 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:55.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:55.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:55.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:55.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:55.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:55.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:55.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:55.605 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:55.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:55.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:55.707 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:55.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:55.751 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:55.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:55.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:55.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:55.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:55.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:09:55.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:56.031 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:56.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:56.056 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.7 to 1 (10000) 2024-02-05T08:09:56.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:09:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:56.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:56.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:09:56.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:56.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:56 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3911642283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:09:56.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:56 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:09:56.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:56 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3911642283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:09:56.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:56 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:09:56.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:56.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:56.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:56 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3911642283' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:09:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:56 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:09:56.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:56.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:56.414 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:56.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:56.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:56.565 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:56.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:56.666 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:56.706 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:56.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:56.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:56.850 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:56.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:56.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:56.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:56.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:57.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:57.108 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-05T08:09:57.108 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:09:57.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:57.161 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:57.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:57.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:57.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:57.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:57.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:57.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:57 smithi134 ceph-mon[30086]: pgmap v203: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 873 op/s 2024-02-05T08:09:57.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:57 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:09:57.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:57 smithi134 ceph-mon[30086]: osdmap e146: 8 total, 8 up, 7 in 2024-02-05T08:09:57.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:57 smithi134 ceph-mon[39278]: pgmap v203: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 873 op/s 2024-02-05T08:09:57.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:57 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:09:57.354 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:57 smithi134 ceph-mon[39278]: osdmap e146: 8 total, 8 up, 7 in 2024-02-05T08:09:57.371 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:09:57.390 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:57.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:57.415 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:57.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:57.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:57.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:57.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:57.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:57 smithi196 ceph-mon[33018]: pgmap v203: 148 pgs: 148 active+clean; 1.7 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 873 op/s 2024-02-05T08:09:57.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:57 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:09:57.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:57 smithi196 ceph-mon[33018]: osdmap e146: 8 total, 8 up, 7 in 2024-02-05T08:09:57.644 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:57.653 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:57.742 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:09:57.743 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:09:57.743 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.743 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:196110 7.f 7:f3d6f847:::benchmark_data_smithi134_83016_object86176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-02-05T08:09:57.743 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:09:57.721250+0000", 2024-02-05T08:09:57.743 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.015166664999999999, 2024-02-05T08:09:57.743 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.015200534, 2024-02-05T08:09:57.743 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:57.743 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:09:57.743 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:57.743 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:57.743 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:57.743 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 196110 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.721250+0000", 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.721250+0000", 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.744 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:57.745 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.721252+0000", 2024-02-05T08:09:57.745 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5379999999999999e-06 2024-02-05T08:09:57.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.745 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.745 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:57.745 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.721256+0000", 2024-02-05T08:09:57.745 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5199999999999999e-06 2024-02-05T08:09:57.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.745 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.745 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:57.745 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.721273+0000", 2024-02-05T08:09:57.745 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6382e-05 2024-02-05T08:09:57.745 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.721281+0000", 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1100000000000003e-06 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.721368+0000", 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6928999999999994e-05 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.721491+0000", 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000123363 2024-02-05T08:09:57.746 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.721604+0000", 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000113266 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:196115 7.e e146/95 7:741a5845:::benchmark_data_smithi134_83016_object86181:head v 146'12398, mlcod=146'12396)", 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:09:57.729713+0000", 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0067036819999999999, 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0068201429999999999, 2024-02-05T08:09:57.747 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.729713+0000", 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.729713+0000", 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.748 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.729714+0000", 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.212e-06 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.729722+0000", 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8669999999999994e-06 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.729741+0000", 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8759000000000001e-05 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.749 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.729754+0000", 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3009e-05 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731975+0000", 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00222135 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731996+0000", 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0231e-05 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.750 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:196118 7.6 7:6bf1d0d5:::benchmark_data_smithi134_83016_object86184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:09:57.729793+0000", 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0066233949999999998, 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0068063139999999999, 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 196118 2024-02-05T08:09:57.751 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.729793+0000", 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.729793+0000", 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:57.752 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.729795+0000", 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6199999999999999e-06 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.729799+0000", 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.194e-06 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.729810+0000", 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0487e-05 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:57.753 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.729817+0000", 2024-02-05T08:09:57.754 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.469e-06 2024-02-05T08:09:57.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.755 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:57.755 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.732351+0000", 2024-02-05T08:09:57.755 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002534232 2024-02-05T08:09:57.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.756 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:57.756 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.732454+0000", 2024-02-05T08:09:57.756 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010270199999999999 2024-02-05T08:09:57.756 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.756 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.756 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:09:57.756 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.732556+0000", 2024-02-05T08:09:57.756 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000101656 2024-02-05T08:09:57.756 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.757 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:57.757 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.757 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.757 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.758 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:196119 7.7 7:e8b14719:::benchmark_data_smithi134_83016_object86185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-02-05T08:09:57.758 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:09:57.731100+0000", 2024-02-05T08:09:57.758 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0053163189999999999, 2024-02-05T08:09:57.758 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0055812759999999996, 2024-02-05T08:09:57.758 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:57.758 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:09:57.758 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:57.758 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:57.758 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:57.758 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 196119 2024-02-05T08:09:57.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.758 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:57.758 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.759 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:57.759 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731100+0000", 2024-02-05T08:09:57.759 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.759 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:57.759 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731100+0000", 2024-02-05T08:09:57.759 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.759 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:57.759 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731102+0000", 2024-02-05T08:09:57.759 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.787e-06 2024-02-05T08:09:57.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.760 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.760 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:57.760 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731108+0000", 2024-02-05T08:09:57.760 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4319999999999998e-06 2024-02-05T08:09:57.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.760 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.760 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:57.760 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731122+0000", 2024-02-05T08:09:57.760 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4834000000000001e-05 2024-02-05T08:09:57.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.760 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.760 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:57.761 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731131+0000", 2024-02-05T08:09:57.761 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6340000000000007e-06 2024-02-05T08:09:57.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.761 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.761 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:57.761 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731239+0000", 2024-02-05T08:09:57.761 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107775 2024-02-05T08:09:57.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731379+0000", 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000139992 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731501+0000", 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012248599999999999 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.762 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:196122 7.7 7.b91f8027 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:09:57.731150+0000", 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.005266265, 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0056058460000000003, 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 196122 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:57.763 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731150+0000", 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731150+0000", 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731151+0000", 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4099999999999998e-07 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.764 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731154+0000", 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7389999999999999e-06 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731161+0000", 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8719999999999996e-06 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731166+0000", 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6720000000000001e-06 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:196123 7.6 7.8310d446 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:09:57.731173+0000", 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0052437550000000001, 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0056422570000000003, 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 196123 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.766 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731173+0000", 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731173+0000", 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731173+0000", 2024-02-05T08:09:57.767 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9999999999999999e-07 2024-02-05T08:09:57.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.768 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.768 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:57.768 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731175+0000", 2024-02-05T08:09:57.768 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.097e-06 2024-02-05T08:09:57.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.768 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.768 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:57.768 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731186+0000", 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0311e-05 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731190+0000", 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4220000000000002e-06 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:196121 7.5 e146/95 7:a863164d:::benchmark_data_smithi134_83016_object86187:head v 146'12272, mlcod=146'12270)", 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:09:57.731370+0000", 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0050467419999999999, 2024-02-05T08:09:57.769 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0055033590000000002, 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731370+0000", 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731370+0000", 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731371+0000", 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6199999999999997e-07 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731377+0000", 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9020000000000004e-06 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731387+0000", 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0757000000000001e-05 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731394+0000", 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8340000000000001e-06 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731482+0000", 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7253e-05 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.731502+0000", 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0285e-05 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.772 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:196118 7.6 e146/95)", 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:09:57.736099+0000", 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.000317702, 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00083859099999999997, 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.736099+0000", 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.736099+0000", 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.736100+0000", 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5600000000000004e-07 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.736110+0000", 2024-02-05T08:09:57.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9830000000000008e-06 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.736127+0000", 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6909999999999999e-05 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:57.736135+0000", 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1419999999999993e-06 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:09:57.775 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:09:57.776 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 8 2024-02-05T08:09:57.776 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:09:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:57.777 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:57.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:57.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:57.898 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:09:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:09:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:09:57.904 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:57.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:57.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:58.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:58.045 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:58.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:58.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:58.189 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:58.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:58.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:58.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:58.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:58.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:09:58.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:58.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:09:58.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:09:58.573 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:58.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:09:58.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:09:58.700 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:09:58.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:58.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:09:58.802 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:58.841 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:58.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:58.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:58.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:58.982 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:58.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:58.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:59.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:59.127 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:59.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:59.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:09:59.300 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:59.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:59.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:09:59.304 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:09:59 smithi134 ceph-mon[30086]: pgmap v205: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 609 op/s 2024-02-05T08:09:59.305 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:09:59 smithi134 ceph-mon[39278]: pgmap v205: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 609 op/s 2024-02-05T08:09:59.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:09:59.445 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:59.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:59.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:09:59.547 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:59.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:09:59 smithi196 ceph-mon[33018]: pgmap v205: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 609 op/s 2024-02-05T08:09:59.587 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:59.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:59.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:59.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T08:09:59.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:09:59.752 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:59.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:09:59.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:09:59.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:09:59.897 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:09:59.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:09:59.961 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:09:59.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:00.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:00.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:00.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:00.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:00.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:00.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:00.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:00.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:00.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:00.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:00.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:00.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:00.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:00.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:00.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:00.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:00.663 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:00.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:00.713 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:00.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:00.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:00.817 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:00.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:00.918 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:00.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:00 smithi134 ceph-mon[30086]: pgmap v206: 148 pgs: 148 active+clean; 1.4 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-02-05T08:10:00.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:00 smithi134 ceph-mon[30086]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-05T08:10:00.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:00 smithi134 ceph-mon[30086]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T08:10:00.953 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:00 smithi134 ceph-mon[39278]: pgmap v206: 148 pgs: 148 active+clean; 1.4 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-02-05T08:10:00.953 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:00 smithi134 ceph-mon[39278]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-05T08:10:00.953 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:00 smithi134 ceph-mon[39278]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T08:10:00.968 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:00.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:00.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:01.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:00 smithi196 ceph-mon[33018]: pgmap v206: 148 pgs: 148 active+clean; 1.4 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-02-05T08:10:01.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:00 smithi196 ceph-mon[33018]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-05T08:10:01.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:00 smithi196 ceph-mon[33018]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T08:10:01.070 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:01.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:01.113 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:01.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:01.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:01.265 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:01.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:01.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:01.367 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:01.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:01.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:01.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:01.482 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:01.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:01.535 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:01.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:01.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:01.637 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:01.663 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:01.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:01.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:01.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:01.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:01.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:01.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:01.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:01.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:01.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:02.018 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:10:02.018 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:10:02.018 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:10:02.018 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:02.027 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:10:02.043 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:02.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:02.066 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:10:02.105 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:10:02.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:02.148 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:10:02.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:02.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:02.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:02.186 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:10:02.190 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:10:02.190 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:10:02.190 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:10:02.190 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:10:02.190 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:10:02.190 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:10:02.191 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:10:02.191 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:10:02.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:02.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:02.310 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:02.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:02.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:02.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:02.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:02.541 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:02.580 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:02.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:02.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:02.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:02.708 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:02.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:02.810 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:02.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:02.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:02.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:02.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:02.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:02 smithi134 ceph-mon[30086]: pgmap v207: 148 pgs: 148 active+clean; 1.4 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 598 B/s rd, 0 B/s wr, 575 op/s 2024-02-05T08:10:02.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:02 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1073840779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:10:02.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:02 smithi134 ceph-mon[39278]: pgmap v207: 148 pgs: 148 active+clean; 1.4 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 598 B/s rd, 0 B/s wr, 575 op/s 2024-02-05T08:10:02.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:02 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1073840779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:10:02.985 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:02.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:03.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:02 smithi196 ceph-mon[33018]: pgmap v207: 148 pgs: 148 active+clean; 1.4 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 598 B/s rd, 0 B/s wr, 575 op/s 2024-02-05T08:10:03.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:02 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1073840779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:10:03.086 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:03.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:03.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:03.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:03.229 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:03.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:03.275 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:03.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:03.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:03.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:03.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:03.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:03.505 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:03.548 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:03.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:03.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:03.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:10:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:10:03.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:03.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:03.680 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:03.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:03.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:03.821 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:03.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:03.922 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:03.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:03.964 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:03.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:04.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:03 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:04.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:03 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:03 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:04.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:04.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:04.092 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:04.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:04.193 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:04.235 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:04.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:04.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:04.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:04.363 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:04.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:04.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:04.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:04.500 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:04.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:04.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:04.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:04.628 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:04.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:04.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:04.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:04.771 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:04.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:04.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:04.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:04.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:04.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:05.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:05.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:05.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:04 smithi134 ceph-mon[30086]: pgmap v208: 148 pgs: 148 active+clean; 1.1 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 984 op/s 2024-02-05T08:10:05.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:04 smithi134 ceph-mon[39278]: pgmap v208: 148 pgs: 148 active+clean; 1.1 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 984 op/s 2024-02-05T08:10:05.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:05.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:05.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:04 smithi196 ceph-mon[33018]: pgmap v208: 148 pgs: 148 active+clean; 1.1 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 984 op/s 2024-02-05T08:10:05.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:05.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:05.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:05.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:05.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:05.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:05.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:05.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:05.373 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:10:05.380 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:10:05.400 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:05.443 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:05.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:05.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:05.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:05.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:05.673 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:05.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:05.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:05.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:05.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:05.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:05.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:05.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:05.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:05.996 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:05.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:06.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:05 smithi134 ceph-mon[30086]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:10:06.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:05 smithi134 ceph-mon[39278]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:10:06.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:05 smithi196 ceph-mon[33018]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:10:06.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:06.135 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:06.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:06.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:06.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:06.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:06.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:06.364 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:06.415 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:06.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:06.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:06.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:06.543 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:06.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:06.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:06.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:06.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:06.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:06.773 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:06.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:06 smithi134 ceph-mon[30086]: pgmap v209: 148 pgs: 148 active+clean; 1.1 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 984 op/s 2024-02-05T08:10:06.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:06 smithi134 ceph-mon[39278]: pgmap v209: 148 pgs: 148 active+clean; 1.1 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 984 op/s 2024-02-05T08:10:06.799 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:06.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:06.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:06.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:06.940 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:06.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:07.041 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:06 smithi196 ceph-mon[33018]: pgmap v209: 148 pgs: 148 active+clean; 1.1 GiB data, 4.4 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 984 op/s 2024-02-05T08:10:07.078 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:07.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:07.179 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:07.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:07.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:07.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:07.307 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:07.344 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:07.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:07.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:07.419 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-02-05T08:10:03.592902+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1228151156,"num_objects":18930,"num_object_clones":0,"num_object_copies":56790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18930,"num_whiteouts":0,"num_read":1247,"num_read_kb":1093,"num_write":201747,"num_write_kb":7038046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":2363363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76021,"ondisk_log_size":76021,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":4580732,"kb_used_data":3707140,"kb_used_omap":10,"kb_used_meta":873397,"kb_avail":651635332,"statfs":{"total":671965249536,"available":667274579968,"internally_reserved":0,"allocated":3796111360,"data_stored":3793977640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11155,"internal_metadata":894358637},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,4,2,6,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-644677632,"num_objects":-9837,"num_object_clones":0,"num_object_copies":-29511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9837,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9837,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002567"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049294+0000","last_change":"2024-02-05T08:07:44.926305+0000","last_active":"2024-02-05T08:09:56.049294+0000","last_peered":"2024-02-05T08:09:56.049294+0000","last_clean":"2024-02-05T08:09:56.049294+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:09:56.049294+0000","last_undegraded":"2024-02-05T08:09:56.049294+0000","last_fullsized":"2024-02-05T08:09:56.049294+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:44.926230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:44.926230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.386321+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060182+0000","last_change":"2024-02-05T08:08:50.003502+0000","last_active":"2024-02-05T08:09:56.060182+0000","last_peered":"2024-02-05T08:09:56.060182+0000","last_clean":"2024-02-05T08:09:56.060182+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:09:56.060182+0000","last_undegraded":"2024-02-05T08:09:56.060182+0000","last_fullsized":"2024-02-05T08:09:56.060182+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:50.003400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:50.003400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.277515+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.058933+0000","last_change":"2024-02-05T08:07:20.353726+0000","last_active":"2024-02-05T08:09:56.058933+0000","last_peered":"2024-02-05T08:09:56.058933+0000","last_clean":"2024-02-05T08:09:56.058933+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:09:56.058933+0000","last_undegraded":"2024-02-05T08:09:56.058933+0000","last_fullsized":"2024-02-05T08:09:56.058933+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:20.353423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:20.353423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.344254+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"90'13","reported_seq":218,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049130+0000","last_change":"2024-02-05T08:08:31.425402+0000","last_active":"2024-02-05T08:09:56.049130+0000","last_peered":"2024-02-05T08:09:56.049130+0000","last_clean":"2024-02-05T08:09:56.049130+0000","last_became_active":"2024-02-05T08:06:39.560434+0000","last_became_peered":"2024-02-05T08:06:39.560434+0000","last_unstale":"2024-02-05T08:09:56.049130+0000","last_undegraded":"2024-02-05T08:09:56.049130+0000","last_fullsized":"2024-02-05T08:09:56.049130+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:08:31.425318+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:08:31.425318+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.386384+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":188,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.076429+0000","last_change":"2024-02-05T08:09:37.106387+0000","last_active":"2024-02-05T08:09:56.076429+0000","last_peered":"2024-02-05T08:09:56.076429+0000","last_clean":"2024-02-05T08:09:56.076429+0000","last_became_active":"2024-02-05T08:06:39.558790+0000","last_became_peered":"2024-02-05T08:06:39.558790+0000","last_unstale":"2024-02-05T08:09:56.076429+0000","last_undegraded":"2024-02-05T08:09:56.076429+0000","last_fullsized":"2024-02-05T08:09:56.076429+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:37.106147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:37.106147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:05.992123+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":184,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.050147+0000","last_change":"2024-02-05T08:08:47.432025+0000","last_active":"2024-02-05T08:09:56.050147+0000","last_peered":"2024-02-05T08:09:56.050147+0000","last_clean":"2024-02-05T08:09:56.050147+0000","last_became_active":"2024-02-05T08:06:39.561731+0000","last_became_peered":"2024-02-05T08:06:39.561731+0000","last_unstale":"2024-02-05T08:09:56.050147+0000","last_undegraded":"2024-02-05T08:09:56.050147+0000","last_fullsized":"2024-02-05T08:09:56.050147+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:47.431940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:47.431940+0000","last_clean_scrub_stamp":"2024-02-05T08:08:47.431940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:59.761856+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.076947+0000","last_change":"2024-02-05T08:07:27.703599+0000","last_active":"2024-02-05T08:09:56.076947+0000","last_peered":"2024-02-05T08:09:56.076947+0000","last_clean":"2024-02-05T08:09:56.076947+0000","last_became_active":"2024-02-05T08:06:39.558101+0000","last_became_peered":"2024-02-05T08:06:39.558101+0000","last_unstale":"2024-02-05T08:09:56.076947+0000","last_undegraded":"2024-02-05T08:09:56.076947+0000","last_fullsized":"2024-02-05T08:09:56.076947+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:27.703488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:27.703488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:56.133429+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"118'22","reported_seq":214,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.076680+0000","last_change":"2024-02-05T08:07:45.819143+0000","last_active":"2024-02-05T08:09:56.076680+0000","last_peered":"2024-02-05T08:09:56.076680+0000","last_clean":"2024-02-05T08:09:56.076680+0000","last_became_active":"2024-02-05T08:04:43.560660+0000","last_became_peered":"2024-02-05T08:04:43.560660+0000","last_unstale":"2024-02-05T08:09:56.076680+0000","last_undegraded":"2024-02-05T08:09:56.076680+0000","last_fullsized":"2024-02-05T08:09:56.076680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:07:45.819004+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:56.133576+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.058108+0000","last_change":"2024-02-05T08:07:38.563648+0000","last_active":"2024-02-05T08:09:56.058108+0000","last_peered":"2024-02-05T08:09:56.058108+0000","last_clean":"2024-02-05T08:09:56.058108+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:09:56.058108+0000","last_undegraded":"2024-02-05T08:09:56.058108+0000","last_fullsized":"2024-02-05T08:09:56.058108+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:38.563533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:38.563533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.344329+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":178,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060098+0000","last_change":"2024-02-05T08:08:35.065367+0000","last_active":"2024-02-05T08:09:56.060098+0000","last_peered":"2024-02-05T08:09:56.060098+0000","last_clean":"2024-02-05T08:09:56.060098+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:09:56.060098+0000","last_undegraded":"2024-02-05T08:09:56.060098+0000","last_fullsized":"2024-02-05T08:09:56.060098+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:35.065258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:37.352358+0000","last_clean_scrub_stamp":"2024-02-05T08:08:35.065258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.490716+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":200,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060190+0000","last_change":"2024-02-05T08:09:39.410126+0000","last_active":"2024-02-05T08:09:56.060190+0000","last_peered":"2024-02-05T08:09:56.060190+0000","last_clean":"2024-02-05T08:09:56.060190+0000","last_became_active":"2024-02-05T08:06:39.559461+0000","last_became_peered":"2024-02-05T08:06:39.559461+0000","last_unstale":"2024-02-05T08:09:56.060190+0000","last_undegraded":"2024-02-05T08:09:56.060190+0000","last_fullsized":"2024-02-05T08:09:56.060190+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:39.410054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.410054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.039707+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":214,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049440+0000","last_change":"2024-02-05T08:08:38.437994+0000","last_active":"2024-02-05T08:09:56.049440+0000","last_peered":"2024-02-05T08:09:56.049440+0000","last_clean":"2024-02-05T08:09:56.049440+0000","last_became_active":"2024-02-05T08:07:22.653892+0000","last_became_peered":"2024-02-05T08:07:22.653892+0000","last_unstale":"2024-02-05T08:09:56.049440+0000","last_undegraded":"2024-02-05T08:09:56.049440+0000","last_fullsized":"2024-02-05T08:09:56.049440+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:08:38.437850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:38.437850+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:07.072663+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.059744+0000","last_change":"2024-02-05T08:07:47.742914+0000","last_active":"2024-02-05T08:09:56.059744+0000","last_peered":"2024-02-05T08:09:56.059744+0000","last_clean":"2024-02-05T08:09:56.059744+0000","last_became_active":"2024-02-05T08:04:47.587515+0000","last_became_peered":"2024-02-05T08:04:47.587515+0000","last_unstale":"2024-02-05T08:09:56.059744+0000","last_undegraded":"2024-02-05T08:09:56.059744+0000","last_fullsized":"2024-02-05T08:09:56.059744+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:47.742836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:47.742836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.490542+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":179,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.069598+0000","last_change":"2024-02-05T08:07:36.546190+0000","last_active":"2024-02-05T08:09:56.069598+0000","last_peered":"2024-02-05T08:09:56.069598+0000","last_clean":"2024-02-05T08:09:56.069598+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:09:56.069598+0000","last_undegraded":"2024-02-05T08:09:56.069598+0000","last_fullsized":"2024-02-05T08:09:56.069598+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:36.546090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:36.546090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:59.070041+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.059995+0000","last_change":"2024-02-05T08:07:39.835029+0000","last_active":"2024-02-05T08:09:56.059995+0000","last_peered":"2024-02-05T08:09:56.059995+0000","last_clean":"2024-02-05T08:09:56.059995+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:09:56.059995+0000","last_undegraded":"2024-02-05T08:09:56.059995+0000","last_fullsized":"2024-02-05T08:09:56.059995+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:39.834957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:07:39.834957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.490630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.050438+0000","last_change":"2024-02-05T08:09:30.448604+0000","last_active":"2024-02-05T08:09:56.050438+0000","last_peered":"2024-02-05T08:09:56.050438+0000","last_clean":"2024-02-05T08:09:56.050438+0000","last_became_active":"2024-02-05T08:04:43.557559+0000","last_became_peered":"2024-02-05T08:04:43.557559+0000","last_unstale":"2024-02-05T08:09:56.050438+0000","last_undegraded":"2024-02-05T08:09:56.050438+0000","last_fullsized":"2024-02-05T08:09:56.050438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:09:30.448493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:30.448493+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:40.766789+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":188,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049836+0000","last_change":"2024-02-05T08:09:26.466381+0000","last_active":"2024-02-05T08:09:56.049836+0000","last_peered":"2024-02-05T08:09:56.049836+0000","last_clean":"2024-02-05T08:09:56.049836+0000","last_became_active":"2024-02-05T08:06:39.547906+0000","last_became_peered":"2024-02-05T08:06:39.547906+0000","last_unstale":"2024-02-05T08:09:56.049836+0000","last_undegraded":"2024-02-05T08:09:56.049836+0000","last_fullsized":"2024-02-05T08:09:56.049836+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:26.466302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:09:26.466302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:54.690678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":179,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.058733+0000","last_change":"2024-02-05T08:07:48.943001+0000","last_active":"2024-02-05T08:09:56.058733+0000","last_peered":"2024-02-05T08:09:56.058733+0000","last_clean":"2024-02-05T08:09:56.058733+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:09:56.058733+0000","last_undegraded":"2024-02-05T08:09:56.058733+0000","last_fullsized":"2024-02-05T08:09:56.058733+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:07:48.942873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.277405+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.057945+0000","last_change":"2024-02-05T08:08:28.601390+0000","last_active":"2024-02-05T08:09:56.057945+0000","last_peered":"2024-02-05T08:09:56.057945+0000","last_clean":"2024-02-05T08:09:56.057945+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:09:56.057945+0000","last_undegraded":"2024-02-05T08:09:56.057945+0000","last_fullsized":"2024-02-05T08:09:56.057945+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:28.601275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:28.601275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.344568+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":216,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.070585+0000","last_change":"2024-02-05T08:08:40.732291+0000","last_active":"2024-02-05T08:09:56.070585+0000","last_peered":"2024-02-05T08:09:56.070585+0000","last_clean":"2024-02-05T08:09:56.070585+0000","last_became_active":"2024-02-05T08:04:43.557496+0000","last_became_peered":"2024-02-05T08:04:43.557496+0000","last_unstale":"2024-02-05T08:09:56.070585+0000","last_undegraded":"2024-02-05T08:09:56.070585+0000","last_fullsized":"2024-02-05T08:09:56.070585+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:08:40.729049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:40.729049+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":180,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.073167+0000","last_change":"2024-02-05T08:09:21.354430+0000","last_active":"2024-02-05T08:09:56.073167+0000","last_peered":"2024-02-05T08:09:56.073167+0000","last_clean":"2024-02-05T08:09:56.073167+0000","last_became_active":"2024-02-05T08:06:39.560948+0000","last_became_peered":"2024-02-05T08:06:39.560948+0000","last_unstale":"2024-02-05T08:09:56.073167+0000","last_undegraded":"2024-02-05T08:09:56.073167+0000","last_fullsized":"2024-02-05T08:09:56.073167+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:21.354158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.354158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:26.901104+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":178,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049768+0000","last_change":"2024-02-05T08:07:55.895894+0000","last_active":"2024-02-05T08:09:56.049768+0000","last_peered":"2024-02-05T08:09:56.049768+0000","last_clean":"2024-02-05T08:09:56.049768+0000","last_became_active":"2024-02-05T08:04:45.563948+0000","last_became_peered":"2024-02-05T08:04:45.563948+0000","last_unstale":"2024-02-05T08:09:56.049768+0000","last_undegraded":"2024-02-05T08:09:56.049768+0000","last_fullsized":"2024-02-05T08:09:56.049768+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:55.895795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:07:55.895795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.386349+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049478+0000","last_change":"2024-02-05T08:09:27.431319+0000","last_active":"2024-02-05T08:09:56.049478+0000","last_peered":"2024-02-05T08:09:56.049478+0000","last_clean":"2024-02-05T08:09:56.049478+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:09:56.049478+0000","last_undegraded":"2024-02-05T08:09:56.049478+0000","last_fullsized":"2024-02-05T08:09:56.049478+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:27.431221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:27.431221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:44.170149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"90'17","reported_seq":210,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.068626+0000","last_change":"2024-02-05T08:07:24.701607+0000","last_active":"2024-02-05T08:09:56.068626+0000","last_peered":"2024-02-05T08:09:56.068626+0000","last_clean":"2024-02-05T08:09:56.068626+0000","last_became_active":"2024-02-05T08:04:43.557596+0000","last_became_peered":"2024-02-05T08:04:43.557596+0000","last_unstale":"2024-02-05T08:09:56.068626+0000","last_undegraded":"2024-02-05T08:09:56.068626+0000","last_fullsized":"2024-02-05T08:09:56.068626+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:07:24.701465+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:07:24.701465+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:56.133463+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":185,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.071206+0000","last_change":"2024-02-05T08:08:48.447437+0000","last_active":"2024-02-05T08:09:56.071206+0000","last_peered":"2024-02-05T08:09:56.071206+0000","last_clean":"2024-02-05T08:09:56.071206+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:09:56.071206+0000","last_undegraded":"2024-02-05T08:09:56.071206+0000","last_fullsized":"2024-02-05T08:09:56.071206+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:08:48.447317+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.965137+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":193,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.650639+0000","last_change":"2024-02-05T08:08:44.441278+0000","last_active":"2024-02-05T08:09:56.650639+0000","last_peered":"2024-02-05T08:09:56.650639+0000","last_clean":"2024-02-05T08:09:56.650639+0000","last_became_active":"2024-02-05T08:06:39.552760+0000","last_became_peered":"2024-02-05T08:06:39.552760+0000","last_unstale":"2024-02-05T08:09:56.650639+0000","last_undegraded":"2024-02-05T08:09:56.650639+0000","last_fullsized":"2024-02-05T08:09:56.650639+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:08:44.441144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.441144+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:02.682475+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":188,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060099+0000","last_change":"2024-02-05T08:07:35.997681+0000","last_active":"2024-02-05T08:09:56.060099+0000","last_peered":"2024-02-05T08:09:56.060099+0000","last_clean":"2024-02-05T08:09:56.060099+0000","last_became_active":"2024-02-05T08:04:41.532339+0000","last_became_peered":"2024-02-05T08:04:41.532339+0000","last_unstale":"2024-02-05T08:09:56.060099+0000","last_undegraded":"2024-02-05T08:09:56.060099+0000","last_fullsized":"2024-02-05T08:09:56.060099+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:07:35.997535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:35.997535+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.277314+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":197,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.058940+0000","last_change":"2024-02-05T08:09:24.000902+0000","last_active":"2024-02-05T08:09:56.058940+0000","last_peered":"2024-02-05T08:09:56.058940+0000","last_clean":"2024-02-05T08:09:56.058940+0000","last_became_active":"2024-02-05T08:04:43.605952+0000","last_became_peered":"2024-02-05T08:04:43.605952+0000","last_unstale":"2024-02-05T08:09:56.058940+0000","last_undegraded":"2024-02-05T08:09:56.058940+0000","last_fullsized":"2024-02-05T08:09:56.058940+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:09:24.000675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:24.000675+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:53.229695+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":183,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.059054+0000","last_change":"2024-02-05T08:09:25.266619+0000","last_active":"2024-02-05T08:09:56.059054+0000","last_peered":"2024-02-05T08:09:56.059054+0000","last_clean":"2024-02-05T08:09:56.059054+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:09:56.059054+0000","last_undegraded":"2024-02-05T08:09:56.059054+0000","last_fullsized":"2024-02-05T08:09:56.059054+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.266508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.266508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:35.256058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":178,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.067658+0000","last_change":"2024-02-05T08:08:42.676027+0000","last_active":"2024-02-05T08:09:56.067658+0000","last_peered":"2024-02-05T08:09:56.067658+0000","last_clean":"2024-02-05T08:09:56.067658+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:09:56.067658+0000","last_undegraded":"2024-02-05T08:09:56.067658+0000","last_fullsized":"2024-02-05T08:09:56.067658+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:42.675782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:42.675782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.073526+0000","last_change":"2024-02-05T08:07:50.777386+0000","last_active":"2024-02-05T08:09:56.073526+0000","last_peered":"2024-02-05T08:09:56.073526+0000","last_clean":"2024-02-05T08:09:56.073526+0000","last_became_active":"2024-02-05T08:04:41.534448+0000","last_became_peered":"2024-02-05T08:04:41.534448+0000","last_unstale":"2024-02-05T08:09:56.073526+0000","last_undegraded":"2024-02-05T08:09:56.073526+0000","last_fullsized":"2024-02-05T08:09:56.073526+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:50.777273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:07:50.777273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:56.133600+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":175,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060512+0000","last_change":"2024-02-05T08:09:25.380533+0000","last_active":"2024-02-05T08:09:56.060512+0000","last_peered":"2024-02-05T08:09:56.060512+0000","last_clean":"2024-02-05T08:09:56.060512+0000","last_became_active":"2024-02-05T08:06:39.558747+0000","last_became_peered":"2024-02-05T08:06:39.558747+0000","last_unstale":"2024-02-05T08:09:56.060512+0000","last_undegraded":"2024-02-05T08:09:56.060512+0000","last_fullsized":"2024-02-05T08:09:56.060512+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:09:25.380390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.380390+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:41.147157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":175,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.069495+0000","last_change":"2024-02-05T08:08:31.483194+0000","last_active":"2024-02-05T08:09:56.069495+0000","last_peered":"2024-02-05T08:09:56.069495+0000","last_clean":"2024-02-05T08:09:56.069495+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:09:56.069495+0000","last_undegraded":"2024-02-05T08:09:56.069495+0000","last_fullsized":"2024-02-05T08:09:56.069495+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:31.483087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:31.483087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:59.070094+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":125,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049287+0000","last_change":"2024-02-05T08:09:36.434985+0000","last_active":"2024-02-05T08:09:56.049287+0000","last_peered":"2024-02-05T08:09:56.049287+0000","last_clean":"2024-02-05T08:09:56.049287+0000","last_became_active":"2024-02-05T08:06:39.551173+0000","last_became_peered":"2024-02-05T08:06:39.551173+0000","last_unstale":"2024-02-05T08:09:56.049287+0000","last_undegraded":"2024-02-05T08:09:56.049287+0000","last_fullsized":"2024-02-05T08:09:56.049287+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:36.434873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.434873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:05.817457+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.059151+0000","last_change":"2024-02-05T08:08:30.632883+0000","last_active":"2024-02-05T08:09:56.059151+0000","last_peered":"2024-02-05T08:09:56.059151+0000","last_clean":"2024-02-05T08:09:56.059151+0000","last_became_active":"2024-02-05T08:04:41.536019+0000","last_became_peered":"2024-02-05T08:04:41.536019+0000","last_unstale":"2024-02-05T08:09:56.059151+0000","last_undegraded":"2024-02-05T08:09:56.059151+0000","last_fullsized":"2024-02-05T08:09:56.059151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:30.632785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:30.632785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.344514+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":200,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.073605+0000","last_change":"2024-02-05T08:08:40.082709+0000","last_active":"2024-02-05T08:09:56.073605+0000","last_peered":"2024-02-05T08:09:56.073605+0000","last_clean":"2024-02-05T08:09:56.073605+0000","last_became_active":"2024-02-05T08:06:39.560932+0000","last_became_peered":"2024-02-05T08:06:39.560932+0000","last_unstale":"2024-02-05T08:09:56.073605+0000","last_undegraded":"2024-02-05T08:09:56.073605+0000","last_fullsized":"2024-02-05T08:09:56.073605+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-02-05T08:08:40.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:40.082577+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:00.576851+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":180,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.076710+0000","last_change":"2024-02-05T08:07:28.666539+0000","last_active":"2024-02-05T08:09:56.076710+0000","last_peered":"2024-02-05T08:09:56.076710+0000","last_clean":"2024-02-05T08:09:56.076710+0000","last_became_active":"2024-02-05T08:06:39.559212+0000","last_became_peered":"2024-02-05T08:06:39.559212+0000","last_unstale":"2024-02-05T08:09:56.076710+0000","last_undegraded":"2024-02-05T08:09:56.076710+0000","last_fullsized":"2024-02-05T08:09:56.076710+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:28.666433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:28.666433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:56.133368+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":193,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.069675+0000","last_change":"2024-02-05T08:09:36.353456+0000","last_active":"2024-02-05T08:09:56.069675+0000","last_peered":"2024-02-05T08:09:56.069675+0000","last_clean":"2024-02-05T08:09:56.069675+0000","last_became_active":"2024-02-05T08:06:39.555597+0000","last_became_peered":"2024-02-05T08:06:39.555597+0000","last_unstale":"2024-02-05T08:09:56.069675+0000","last_undegraded":"2024-02-05T08:09:56.069675+0000","last_fullsized":"2024-02-05T08:09:56.069675+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.353351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:38.202761+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060353+0000","last_change":"2024-02-05T08:08:45.166139+0000","last_active":"2024-02-05T08:09:56.060353+0000","last_peered":"2024-02-05T08:09:56.060353+0000","last_clean":"2024-02-05T08:09:56.060353+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:09:56.060353+0000","last_undegraded":"2024-02-05T08:09:56.060353+0000","last_fullsized":"2024-02-05T08:09:56.060353+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:45.166006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:45.166006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"141'16","reported_seq":232,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.076582+0000","last_change":"2024-02-05T08:09:39.113079+0000","last_active":"2024-02-05T08:09:56.076582+0000","last_peered":"2024-02-05T08:09:56.076582+0000","last_clean":"2024-02-05T08:09:56.076582+0000","last_became_active":"2024-02-05T08:06:39.556873+0000","last_became_peered":"2024-02-05T08:06:39.556873+0000","last_unstale":"2024-02-05T08:09:56.076582+0000","last_undegraded":"2024-02-05T08:09:56.076582+0000","last_fullsized":"2024-02-05T08:09:56.076582+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'16","last_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.112936+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:45.211059+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.056821+0000","last_change":"2024-02-05T08:07:45.514975+0000","last_active":"2024-02-05T08:09:56.056821+0000","last_peered":"2024-02-05T08:09:56.056821+0000","last_clean":"2024-02-05T08:09:56.056821+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:09:56.056821+0000","last_undegraded":"2024-02-05T08:09:56.056821+0000","last_fullsized":"2024-02-05T08:09:56.056821+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:45.514863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:45.514863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.344436+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":178,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.076344+0000","last_change":"2024-02-05T08:07:25.644883+0000","last_active":"2024-02-05T08:09:56.076344+0000","last_peered":"2024-02-05T08:09:56.076344+0000","last_clean":"2024-02-05T08:09:56.076344+0000","last_became_active":"2024-02-05T08:04:45.568484+0000","last_became_peered":"2024-02-05T08:04:45.568484+0000","last_unstale":"2024-02-05T08:09:56.076344+0000","last_undegraded":"2024-02-05T08:09:56.076344+0000","last_fullsized":"2024-02-05T08:09:56.076344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:25.644770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:25.644770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:56.133398+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.069849+0000","last_change":"2024-02-05T08:08:32.528082+0000","last_active":"2024-02-05T08:09:56.069849+0000","last_peered":"2024-02-05T08:09:56.069849+0000","last_clean":"2024-02-05T08:09:56.069849+0000","last_became_active":"2024-02-05T08:04:41.539698+0000","last_became_peered":"2024-02-05T08:04:41.539698+0000","last_unstale":"2024-02-05T08:09:56.069849+0000","last_undegraded":"2024-02-05T08:09:56.069849+0000","last_fullsized":"2024-02-05T08:09:56.069849+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:32.527978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:32.527978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:59.070078+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":223,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.061581+0000","last_change":"2024-02-05T08:08:43.941119+0000","last_active":"2024-02-05T08:09:56.061581+0000","last_peered":"2024-02-05T08:09:56.061581+0000","last_clean":"2024-02-05T08:09:56.061581+0000","last_became_active":"2024-02-05T08:06:39.560259+0000","last_became_peered":"2024-02-05T08:06:39.560259+0000","last_unstale":"2024-02-05T08:09:56.061581+0000","last_undegraded":"2024-02-05T08:09:56.061581+0000","last_fullsized":"2024-02-05T08:09:56.061581+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:08:43.940980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:43.940980+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.277483+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.076253+0000","last_change":"2024-02-05T08:09:29.262843+0000","last_active":"2024-02-05T08:09:56.076253+0000","last_peered":"2024-02-05T08:09:56.076253+0000","last_clean":"2024-02-05T08:09:56.076253+0000","last_became_active":"2024-02-05T08:04:47.585496+0000","last_became_peered":"2024-02-05T08:04:47.585496+0000","last_unstale":"2024-02-05T08:09:56.076253+0000","last_undegraded":"2024-02-05T08:09:56.076253+0000","last_fullsized":"2024-02-05T08:09:56.076253+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:09:29.262582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:47.044333+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":179,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.061349+0000","last_change":"2024-02-05T08:07:32.955213+0000","last_active":"2024-02-05T08:09:56.061349+0000","last_peered":"2024-02-05T08:09:56.061349+0000","last_clean":"2024-02-05T08:09:56.061349+0000","last_became_active":"2024-02-05T08:04:45.560815+0000","last_became_peered":"2024-02-05T08:04:45.560815+0000","last_unstale":"2024-02-05T08:09:56.061349+0000","last_undegraded":"2024-02-05T08:09:56.061349+0000","last_fullsized":"2024-02-05T08:09:56.061349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:32.955088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:32.955088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.277280+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":200,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049551+0000","last_change":"2024-02-05T08:09:25.486212+0000","last_active":"2024-02-05T08:09:56.049551+0000","last_peered":"2024-02-05T08:09:56.049551+0000","last_clean":"2024-02-05T08:09:56.049551+0000","last_became_active":"2024-02-05T08:06:39.545576+0000","last_became_peered":"2024-02-05T08:06:39.545576+0000","last_unstale":"2024-02-05T08:09:56.049551+0000","last_undegraded":"2024-02-05T08:09:56.049551+0000","last_fullsized":"2024-02-05T08:09:56.049551+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.486141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.966886+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.486141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:39.358720+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":219,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.050487+0000","last_change":"2024-02-05T08:08:26.541304+0000","last_active":"2024-02-05T08:09:56.050487+0000","last_peered":"2024-02-05T08:09:56.050487+0000","last_clean":"2024-02-05T08:09:56.050487+0000","last_became_active":"2024-02-05T08:06:39.558257+0000","last_became_peered":"2024-02-05T08:06:39.558257+0000","last_unstale":"2024-02-05T08:09:56.050487+0000","last_undegraded":"2024-02-05T08:09:56.050487+0000","last_fullsized":"2024-02-05T08:09:56.050487+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:08:26.541177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:26.541177+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.386366+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.058621+0000","last_change":"2024-02-05T08:08:36.679472+0000","last_active":"2024-02-05T08:09:56.058621+0000","last_peered":"2024-02-05T08:09:56.058621+0000","last_clean":"2024-02-05T08:09:56.058621+0000","last_became_active":"2024-02-05T08:04:47.584235+0000","last_became_peered":"2024-02-05T08:04:47.584235+0000","last_unstale":"2024-02-05T08:09:56.058621+0000","last_undegraded":"2024-02-05T08:09:56.058621+0000","last_fullsized":"2024-02-05T08:09:56.058621+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:36.679368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:36.679368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":251,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.650634+0000","last_change":"2024-02-05T08:07:52.659017+0000","last_active":"2024-02-05T08:09:56.650634+0000","last_peered":"2024-02-05T08:09:56.650634+0000","last_clean":"2024-02-05T08:09:56.650634+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:09:56.650634+0000","last_undegraded":"2024-02-05T08:09:56.650634+0000","last_fullsized":"2024-02-05T08:09:56.650634+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:07:52.658883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:52.658883+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:05.390478+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.069645+0000","last_change":"2024-02-05T08:09:19.417311+0000","last_active":"2024-02-05T08:09:56.069645+0000","last_peered":"2024-02-05T08:09:56.069645+0000","last_clean":"2024-02-05T08:09:56.069645+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:09:56.069645+0000","last_undegraded":"2024-02-05T08:09:56.069645+0000","last_fullsized":"2024-02-05T08:09:56.069645+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:19.417231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.417231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:25.958103+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"129'15","reported_seq":202,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.068029+0000","last_change":"2024-02-05T08:08:37.128945+0000","last_active":"2024-02-05T08:09:56.068029+0000","last_peered":"2024-02-05T08:09:56.068029+0000","last_clean":"2024-02-05T08:09:56.068029+0000","last_became_active":"2024-02-05T08:04:43.605981+0000","last_became_peered":"2024-02-05T08:04:43.605981+0000","last_unstale":"2024-02-05T08:09:56.068029+0000","last_undegraded":"2024-02-05T08:09:56.068029+0000","last_fullsized":"2024-02-05T08:09:56.068029+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'15","last_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:08:37.128817+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:56.133683+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049994+0000","last_change":"2024-02-05T08:09:18.397753+0000","last_active":"2024-02-05T08:09:56.049994+0000","last_peered":"2024-02-05T08:09:56.049994+0000","last_clean":"2024-02-05T08:09:56.049994+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:09:56.049994+0000","last_undegraded":"2024-02-05T08:09:56.049994+0000","last_fullsized":"2024-02-05T08:09:56.049994+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:18.397684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:18.397684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:42.614612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":251,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.654119+0000","last_change":"2024-02-05T08:07:29.660531+0000","last_active":"2024-02-05T08:09:56.654119+0000","last_peered":"2024-02-05T08:09:56.654119+0000","last_clean":"2024-02-05T08:09:56.654119+0000","last_became_active":"2024-02-05T08:04:45.565240+0000","last_became_peered":"2024-02-05T08:04:45.565240+0000","last_unstale":"2024-02-05T08:09:56.654119+0000","last_undegraded":"2024-02-05T08:09:56.654119+0000","last_fullsized":"2024-02-05T08:09:56.654119+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:07:29.660395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:29.660395+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:02.050569+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.057908+0000","last_change":"2024-02-05T08:08:44.680464+0000","last_active":"2024-02-05T08:09:56.057908+0000","last_peered":"2024-02-05T08:09:56.057908+0000","last_clean":"2024-02-05T08:09:56.057908+0000","last_became_active":"2024-02-05T08:04:41.540252+0000","last_became_peered":"2024-02-05T08:04:41.540252+0000","last_unstale":"2024-02-05T08:09:56.057908+0000","last_undegraded":"2024-02-05T08:09:56.057908+0000","last_fullsized":"2024-02-05T08:09:56.057908+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:44.680353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.680353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":206,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060410+0000","last_change":"2024-02-05T08:09:36.437558+0000","last_active":"2024-02-05T08:09:56.060410+0000","last_peered":"2024-02-05T08:09:56.060410+0000","last_clean":"2024-02-05T08:09:56.060410+0000","last_became_active":"2024-02-05T08:04:43.606414+0000","last_became_peered":"2024-02-05T08:04:43.606414+0000","last_unstale":"2024-02-05T08:09:56.060410+0000","last_undegraded":"2024-02-05T08:09:56.060410+0000","last_fullsized":"2024-02-05T08:09:56.060410+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:09:36.437435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.437435+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:44.544063+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":183,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.070911+0000","last_change":"2024-02-05T08:08:41.738958+0000","last_active":"2024-02-05T08:09:56.070911+0000","last_peered":"2024-02-05T08:09:56.070911+0000","last_clean":"2024-02-05T08:09:56.070911+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:09:56.070911+0000","last_undegraded":"2024-02-05T08:09:56.070911+0000","last_fullsized":"2024-02-05T08:09:56.070911+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:41.738836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:41.738836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":178,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.073025+0000","last_change":"2024-02-05T08:07:16.750620+0000","last_active":"2024-02-05T08:09:56.073025+0000","last_peered":"2024-02-05T08:09:56.073025+0000","last_clean":"2024-02-05T08:09:56.073025+0000","last_became_active":"2024-02-05T08:04:45.567622+0000","last_became_peered":"2024-02-05T08:04:45.567622+0000","last_unstale":"2024-02-05T08:09:56.073025+0000","last_undegraded":"2024-02-05T08:09:56.073025+0000","last_fullsized":"2024-02-05T08:09:56.073025+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:16.750514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:16.750514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:56.133306+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":193,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.069634+0000","last_change":"2024-02-05T08:08:35.480440+0000","last_active":"2024-02-05T08:09:56.069634+0000","last_peered":"2024-02-05T08:09:56.069634+0000","last_clean":"2024-02-05T08:09:56.069634+0000","last_became_active":"2024-02-05T08:06:39.557018+0000","last_became_peered":"2024-02-05T08:06:39.557018+0000","last_unstale":"2024-02-05T08:09:56.069634+0000","last_undegraded":"2024-02-05T08:09:56.069634+0000","last_fullsized":"2024-02-05T08:09:56.069634+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:35.480367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:35.480367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:02.684273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":198,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.067463+0000","last_change":"2024-02-05T08:08:45.712868+0000","last_active":"2024-02-05T08:09:56.067463+0000","last_peered":"2024-02-05T08:09:56.067463+0000","last_clean":"2024-02-05T08:09:56.067463+0000","last_became_active":"2024-02-05T08:04:43.608489+0000","last_became_peered":"2024-02-05T08:04:43.608489+0000","last_unstale":"2024-02-05T08:09:56.067463+0000","last_undegraded":"2024-02-05T08:09:56.067463+0000","last_fullsized":"2024-02-05T08:09:56.067463+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:08:45.712396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:45.712396+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":189,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.069825+0000","last_change":"2024-02-05T08:09:33.391323+0000","last_active":"2024-02-05T08:09:56.069825+0000","last_peered":"2024-02-05T08:09:56.069825+0000","last_clean":"2024-02-05T08:09:56.069825+0000","last_became_active":"2024-02-05T08:06:39.548837+0000","last_became_peered":"2024-02-05T08:06:39.548837+0000","last_unstale":"2024-02-05T08:09:56.069825+0000","last_undegraded":"2024-02-05T08:09:56.069825+0000","last_fullsized":"2024-02-05T08:09:56.069825+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:33.391219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:33.391219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:38.950366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":178,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060374+0000","last_change":"2024-02-05T08:07:44.778884+0000","last_active":"2024-02-05T08:09:56.060374+0000","last_peered":"2024-02-05T08:09:56.060374+0000","last_clean":"2024-02-05T08:09:56.060374+0000","last_became_active":"2024-02-05T08:04:45.560241+0000","last_became_peered":"2024-02-05T08:04:45.560241+0000","last_unstale":"2024-02-05T08:09:56.060374+0000","last_undegraded":"2024-02-05T08:09:56.060374+0000","last_fullsized":"2024-02-05T08:09:56.060374+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:44.778772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:44.778772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.490688+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":117,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.058420+0000","last_change":"2024-02-05T08:07:36.550407+0000","last_active":"2024-02-05T08:09:56.058420+0000","last_peered":"2024-02-05T08:09:56.058420+0000","last_clean":"2024-02-05T08:09:56.058420+0000","last_became_active":"2024-02-05T08:06:39.554673+0000","last_became_peered":"2024-02-05T08:06:39.554673+0000","last_unstale":"2024-02-05T08:09:56.058420+0000","last_undegraded":"2024-02-05T08:09:56.058420+0000","last_fullsized":"2024-02-05T08:09:56.058420+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:36.550290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:36.550290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.344411+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"116'9","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.073259+0000","last_change":"2024-02-05T08:07:42.729822+0000","last_active":"2024-02-05T08:09:56.073259+0000","last_peered":"2024-02-05T08:09:56.073259+0000","last_clean":"2024-02-05T08:09:56.073259+0000","last_became_active":"2024-02-05T08:04:43.606624+0000","last_became_peered":"2024-02-05T08:04:43.606624+0000","last_unstale":"2024-02-05T08:09:56.073259+0000","last_undegraded":"2024-02-05T08:09:56.073259+0000","last_fullsized":"2024-02-05T08:09:56.073259+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:07:42.729538+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:56.133492+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"146'12450","reported_seq":12695,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:02.820912+0000","last_change":"2024-02-05T08:09:35.129348+0000","last_active":"2024-02-05T08:10:02.820912+0000","last_peered":"2024-02-05T08:10:02.820912+0000","last_clean":"2024-02-05T08:10:02.820912+0000","last_became_active":"2024-02-05T08:06:39.556800+0000","last_became_peered":"2024-02-05T08:06:39.556800+0000","last_unstale":"2024-02-05T08:10:02.820912+0000","last_undegraded":"2024-02-05T08:10:02.820912+0000","last_fullsized":"2024-02-05T08:10:02.820912+0000","mapping_epoch":97,"log_start":"130'8200","ondisk_log_start":"130'8200","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'11079","last_scrub_stamp":"2024-02-05T08:09:35.129197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:35.129197+0000","objects_scrubbed":2569,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:53.986660+0000","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77332480,"num_objects":1180,"num_object_clones":0,"num_object_copies":3540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12450,"num_write_kb":436160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":188,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060099+0000","last_change":"2024-02-05T08:08:41.139144+0000","last_active":"2024-02-05T08:09:56.060099+0000","last_peered":"2024-02-05T08:09:56.060099+0000","last_clean":"2024-02-05T08:09:56.060099+0000","last_became_active":"2024-02-05T08:06:39.551759+0000","last_became_peered":"2024-02-05T08:06:39.551759+0000","last_unstale":"2024-02-05T08:09:56.060099+0000","last_undegraded":"2024-02-05T08:09:56.060099+0000","last_fullsized":"2024-02-05T08:09:56.060099+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:41.139067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:41.139067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:06.245836+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":199,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.070009+0000","last_change":"2024-02-05T08:07:55.497839+0000","last_active":"2024-02-05T08:09:56.070009+0000","last_peered":"2024-02-05T08:09:56.070009+0000","last_clean":"2024-02-05T08:09:56.070009+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:09:56.070009+0000","last_undegraded":"2024-02-05T08:09:56.070009+0000","last_fullsized":"2024-02-05T08:09:56.070009+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:07:55.497748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:55.497748+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:59.070061+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":229,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.059797+0000","last_change":"2024-02-05T08:08:33.007515+0000","last_active":"2024-02-05T08:09:56.059797+0000","last_peered":"2024-02-05T08:09:56.059797+0000","last_clean":"2024-02-05T08:09:56.059797+0000","last_became_active":"2024-02-05T08:06:39.546607+0000","last_became_peered":"2024-02-05T08:06:39.546607+0000","last_unstale":"2024-02-05T08:09:56.059797+0000","last_undegraded":"2024-02-05T08:09:56.059797+0000","last_fullsized":"2024-02-05T08:09:56.059797+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:08:33.007404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:33.007404+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.490739+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":270,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.650691+0000","last_change":"2024-02-05T08:09:20.414916+0000","last_active":"2024-02-05T08:09:56.650691+0000","last_peered":"2024-02-05T08:09:56.650691+0000","last_clean":"2024-02-05T08:09:56.650691+0000","last_became_active":"2024-02-05T08:06:39.556562+0000","last_became_peered":"2024-02-05T08:06:39.556562+0000","last_unstale":"2024-02-05T08:09:56.650691+0000","last_undegraded":"2024-02-05T08:09:56.650691+0000","last_fullsized":"2024-02-05T08:09:56.650691+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:09:20.414837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:20.414837+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:36.384339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"146'12238","reported_seq":12409,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:02.807227+0000","last_change":"2024-02-05T08:08:47.609361+0000","last_active":"2024-02-05T08:10:02.807227+0000","last_peered":"2024-02-05T08:10:02.807227+0000","last_clean":"2024-02-05T08:10:02.807227+0000","last_became_active":"2024-02-05T08:06:37.621544+0000","last_became_peered":"2024-02-05T08:06:37.621544+0000","last_unstale":"2024-02-05T08:10:02.807227+0000","last_undegraded":"2024-02-05T08:10:02.807227+0000","last_fullsized":"2024-02-05T08:10:02.807227+0000","mapping_epoch":95,"log_start":"129'8000","ondisk_log_start":"129'8000","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8264","last_scrub_stamp":"2024-02-05T08:08:47.608801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:47.608801+0000","objects_scrubbed":5157,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.881697+0000","scrub_duration":2457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73662464,"num_objects":1124,"num_object_clones":0,"num_object_copies":3372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12238,"num_write_kb":427584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.076519+0000","last_change":"2024-02-05T08:09:25.266619+0000","last_active":"2024-02-05T08:09:56.076519+0000","last_peered":"2024-02-05T08:09:56.076519+0000","last_clean":"2024-02-05T08:09:56.076519+0000","last_became_active":"2024-02-05T08:04:47.586044+0000","last_became_peered":"2024-02-05T08:04:47.586044+0000","last_unstale":"2024-02-05T08:09:56.076519+0000","last_undegraded":"2024-02-05T08:09:56.076519+0000","last_fullsized":"2024-02-05T08:09:56.076519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.266513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.266513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:48.723648+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.069866+0000","last_change":"2024-02-05T08:09:39.244280+0000","last_active":"2024-02-05T08:09:56.069866+0000","last_peered":"2024-02-05T08:09:56.069866+0000","last_clean":"2024-02-05T08:09:56.069866+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:09:56.069866+0000","last_undegraded":"2024-02-05T08:09:56.069866+0000","last_fullsized":"2024-02-05T08:09:56.069866+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:39.244179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.244179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:55.149887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":205,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049769+0000","last_change":"2024-02-05T08:09:31.469380+0000","last_active":"2024-02-05T08:09:56.049769+0000","last_peered":"2024-02-05T08:09:56.049769+0000","last_clean":"2024-02-05T08:09:56.049769+0000","last_became_active":"2024-02-05T08:04:43.556580+0000","last_became_peered":"2024-02-05T08:04:43.556580+0000","last_unstale":"2024-02-05T08:09:56.049769+0000","last_undegraded":"2024-02-05T08:09:56.049769+0000","last_fullsized":"2024-02-05T08:09:56.049769+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:09:31.469295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:31.469295+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.626653+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.072502+0000","last_change":"2024-02-05T08:09:23.533364+0000","last_active":"2024-02-05T08:09:56.072502+0000","last_peered":"2024-02-05T08:09:56.072502+0000","last_clean":"2024-02-05T08:09:56.072502+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:09:56.072502+0000","last_undegraded":"2024-02-05T08:09:56.072502+0000","last_fullsized":"2024-02-05T08:09:56.072502+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:23.533259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_clean_scrub_stamp":"2024-02-05T08:09:23.533259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:35.020206+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"146'12424","reported_seq":12590,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:03.033703+0000","last_change":"2024-02-05T08:09:19.227093+0000","last_active":"2024-02-05T08:10:03.033703+0000","last_peered":"2024-02-05T08:10:03.033703+0000","last_clean":"2024-02-05T08:10:03.033703+0000","last_became_active":"2024-02-05T08:06:37.619775+0000","last_became_peered":"2024-02-05T08:06:37.619775+0000","last_unstale":"2024-02-05T08:10:03.033703+0000","last_undegraded":"2024-02-05T08:10:03.033703+0000","last_fullsized":"2024-02-05T08:10:03.033703+0000","mapping_epoch":95,"log_start":"128'7600","ondisk_log_start":"128'7600","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'10065","last_scrub_stamp":"2024-02-05T08:09:19.226651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.226651+0000","objects_scrubbed":3510,"log_size":4824,"log_dups_size":0,"ondisk_log_size":4824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:24.312423+0000","scrub_duration":1408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73007104,"num_objects":1114,"num_object_clones":0,"num_object_copies":3342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12424,"num_write_kb":433216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":175,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.069405+0000","last_change":"2024-02-05T08:07:40.688555+0000","last_active":"2024-02-05T08:09:56.069405+0000","last_peered":"2024-02-05T08:09:56.069405+0000","last_clean":"2024-02-05T08:09:56.069405+0000","last_became_active":"2024-02-05T08:04:47.587082+0000","last_became_peered":"2024-02-05T08:04:47.587082+0000","last_unstale":"2024-02-05T08:09:56.069405+0000","last_undegraded":"2024-02-05T08:09:56.069405+0000","last_fullsized":"2024-02-05T08:09:56.069405+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:40.688451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:40.688451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:59.070022+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.070896+0000","last_change":"2024-02-05T08:09:29.579918+0000","last_active":"2024-02-05T08:09:56.070896+0000","last_peered":"2024-02-05T08:09:56.070896+0000","last_clean":"2024-02-05T08:09:56.070896+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:09:56.070896+0000","last_undegraded":"2024-02-05T08:09:56.070896+0000","last_fullsized":"2024-02-05T08:09:56.070896+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:29.578521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:29.578521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:44.673091+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":201,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.070044+0000","last_change":"2024-02-05T08:09:31.480648+0000","last_active":"2024-02-05T08:09:56.070044+0000","last_peered":"2024-02-05T08:09:56.070044+0000","last_clean":"2024-02-05T08:09:56.070044+0000","last_became_active":"2024-02-05T08:06:39.554318+0000","last_became_peered":"2024-02-05T08:06:39.554318+0000","last_unstale":"2024-02-05T08:09:56.070044+0000","last_undegraded":"2024-02-05T08:09:56.070044+0000","last_fullsized":"2024-02-05T08:09:56.070044+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:31.480558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:31.480558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:42.719453+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":200,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.071526+0000","last_change":"2024-02-05T08:09:34.530885+0000","last_active":"2024-02-05T08:09:56.071526+0000","last_peered":"2024-02-05T08:09:56.071526+0000","last_clean":"2024-02-05T08:09:56.071526+0000","last_became_active":"2024-02-05T08:06:39.552819+0000","last_became_peered":"2024-02-05T08:06:39.552819+0000","last_unstale":"2024-02-05T08:09:56.071526+0000","last_undegraded":"2024-02-05T08:09:56.071526+0000","last_fullsized":"2024-02-05T08:09:56.071526+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:09:34.530767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:34.530767+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:35.663461+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"146'12584","reported_seq":12769,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:02.704967+0000","last_change":"2024-02-05T08:08:30.453801+0000","last_active":"2024-02-05T08:10:02.704967+0000","last_peered":"2024-02-05T08:10:02.704967+0000","last_clean":"2024-02-05T08:10:02.704967+0000","last_became_active":"2024-02-05T08:06:37.623334+0000","last_became_peered":"2024-02-05T08:06:37.623334+0000","last_unstale":"2024-02-05T08:10:02.704967+0000","last_undegraded":"2024-02-05T08:10:02.704967+0000","last_fullsized":"2024-02-05T08:10:02.704967+0000","mapping_epoch":95,"log_start":"126'7200","ondisk_log_start":"126'7200","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'7516","last_scrub_stamp":"2024-02-05T08:08:30.453634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:30.453634+0000","objects_scrubbed":6304,"log_size":5384,"log_dups_size":0,"ondisk_log_size":5384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77201408,"num_objects":1178,"num_object_clones":0,"num_object_copies":3534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12584,"num_write_kb":440384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.058330+0000","last_change":"2024-02-05T08:07:55.608340+0000","last_active":"2024-02-05T08:09:56.058330+0000","last_peered":"2024-02-05T08:09:56.058330+0000","last_clean":"2024-02-05T08:09:56.058330+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:09:56.058330+0000","last_undegraded":"2024-02-05T08:09:56.058330+0000","last_fullsized":"2024-02-05T08:09:56.058330+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:55.608083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:55.608083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.344461+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.071301+0000","last_change":"2024-02-05T08:09:21.583142+0000","last_active":"2024-02-05T08:09:56.071301+0000","last_peered":"2024-02-05T08:09:56.071301+0000","last_clean":"2024-02-05T08:09:56.071301+0000","last_became_active":"2024-02-05T08:04:45.567341+0000","last_became_peered":"2024-02-05T08:04:45.567341+0000","last_unstale":"2024-02-05T08:09:56.071301+0000","last_undegraded":"2024-02-05T08:09:56.071301+0000","last_fullsized":"2024-02-05T08:09:56.071301+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:21.582910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.582910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:23.887947+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060030+0000","last_change":"2024-02-05T08:07:45.721633+0000","last_active":"2024-02-05T08:09:56.060030+0000","last_peered":"2024-02-05T08:09:56.060030+0000","last_clean":"2024-02-05T08:09:56.060030+0000","last_became_active":"2024-02-05T08:06:39.558993+0000","last_became_peered":"2024-02-05T08:06:39.558993+0000","last_unstale":"2024-02-05T08:09:56.060030+0000","last_undegraded":"2024-02-05T08:09:56.060030+0000","last_fullsized":"2024-02-05T08:09:56.060030+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:45.721527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:45.721527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.490658+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":198,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.058221+0000","last_change":"2024-02-05T08:07:50.544800+0000","last_active":"2024-02-05T08:09:56.058221+0000","last_peered":"2024-02-05T08:09:56.058221+0000","last_clean":"2024-02-05T08:09:56.058221+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:09:56.058221+0000","last_undegraded":"2024-02-05T08:09:56.058221+0000","last_fullsized":"2024-02-05T08:09:56.058221+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:07:50.544673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:07:50.544673+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.344386+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"146'14596","reported_seq":14802,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:03.013600+0000","last_change":"2024-02-05T08:09:51.200716+0000","last_active":"2024-02-05T08:10:03.013600+0000","last_peered":"2024-02-05T08:10:03.013600+0000","last_clean":"2024-02-05T08:10:03.013600+0000","last_became_active":"2024-02-05T08:06:39.561430+0000","last_became_peered":"2024-02-05T08:06:39.561430+0000","last_unstale":"2024-02-05T08:10:03.013600+0000","last_undegraded":"2024-02-05T08:10:03.013600+0000","last_fullsized":"2024-02-05T08:10:03.013600+0000","mapping_epoch":97,"log_start":"136'9800","ondisk_log_start":"136'9800","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'13917","last_scrub_stamp":"2024-02-05T08:09:51.200561+0000","last_deep_scrub":"144'13917","last_deep_scrub_stamp":"2024-02-05T08:09:51.200561+0000","last_clean_scrub_stamp":"2024-02-05T08:09:51.200561+0000","objects_scrubbed":2109,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":17,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:18.777091+0000","scrub_duration":16228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71892992,"num_objects":1097,"num_object_clones":0,"num_object_copies":3291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12487,"num_write_kb":434688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":166,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.077159+0000","last_change":"2024-02-05T08:07:19.682609+0000","last_active":"2024-02-05T08:09:56.077159+0000","last_peered":"2024-02-05T08:09:56.077159+0000","last_clean":"2024-02-05T08:09:56.077159+0000","last_became_active":"2024-02-05T08:04:47.588395+0000","last_became_peered":"2024-02-05T08:04:47.588395+0000","last_unstale":"2024-02-05T08:09:56.077159+0000","last_undegraded":"2024-02-05T08:09:56.077159+0000","last_fullsized":"2024-02-05T08:09:56.077159+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:19.682361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:19.682361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:56.133338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":208,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.059946+0000","last_change":"2024-02-05T08:07:40.810244+0000","last_active":"2024-02-05T08:09:56.059946+0000","last_peered":"2024-02-05T08:09:56.059946+0000","last_clean":"2024-02-05T08:09:56.059946+0000","last_became_active":"2024-02-05T08:06:39.560631+0000","last_became_peered":"2024-02-05T08:06:39.560631+0000","last_unstale":"2024-02-05T08:09:56.059946+0000","last_undegraded":"2024-02-05T08:09:56.059946+0000","last_fullsized":"2024-02-05T08:09:56.059946+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:07:40.810076+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:07:40.810076+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.490571+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.050365+0000","last_change":"2024-02-05T08:08:34.373002+0000","last_active":"2024-02-05T08:09:56.050365+0000","last_peered":"2024-02-05T08:09:56.050365+0000","last_clean":"2024-02-05T08:09:56.050365+0000","last_became_active":"2024-02-05T08:06:39.563051+0000","last_became_peered":"2024-02-05T08:06:39.563051+0000","last_unstale":"2024-02-05T08:09:56.050365+0000","last_undegraded":"2024-02-05T08:09:56.050365+0000","last_fullsized":"2024-02-05T08:09:56.050365+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:34.372891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:08:34.372891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.386404+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"85'38","reported_seq":314,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.655588+0000","last_change":"2024-02-05T08:09:35.108648+0000","last_active":"2024-02-05T08:09:56.655588+0000","last_peered":"2024-02-05T08:09:56.655588+0000","last_clean":"2024-02-05T08:09:56.655588+0000","last_became_active":"2024-02-05T08:04:43.554536+0000","last_became_peered":"2024-02-05T08:04:43.554536+0000","last_unstale":"2024-02-05T08:09:56.655588+0000","last_undegraded":"2024-02-05T08:09:56.655588+0000","last_fullsized":"2024-02-05T08:09:56.655588+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:09:35.108489+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:09:35.108489+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:50.058243+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":170,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.059222+0000","last_change":"2024-02-05T08:07:19.436649+0000","last_active":"2024-02-05T08:09:56.059222+0000","last_peered":"2024-02-05T08:09:56.059222+0000","last_clean":"2024-02-05T08:09:56.059222+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:09:56.059222+0000","last_undegraded":"2024-02-05T08:09:56.059222+0000","last_fullsized":"2024-02-05T08:09:56.059222+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:19.436541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:19.436541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.344279+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"146'12758","reported_seq":12914,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:02.989960+0000","last_change":"2024-02-05T08:09:26.200715+0000","last_active":"2024-02-05T08:10:02.989960+0000","last_peered":"2024-02-05T08:10:02.989960+0000","last_clean":"2024-02-05T08:10:02.989960+0000","last_became_active":"2024-02-05T08:06:37.618966+0000","last_became_peered":"2024-02-05T08:06:37.618966+0000","last_unstale":"2024-02-05T08:10:02.989960+0000","last_undegraded":"2024-02-05T08:10:02.989960+0000","last_fullsized":"2024-02-05T08:10:02.989960+0000","mapping_epoch":95,"log_start":"128'7900","ondisk_log_start":"128'7900","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10764","last_scrub_stamp":"2024-02-05T08:09:26.200471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:26.200471+0000","objects_scrubbed":3228,"log_size":4858,"log_dups_size":0,"ondisk_log_size":4858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:26.980375+0000","scrub_duration":1266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78381056,"num_objects":1196,"num_object_clones":0,"num_object_copies":3588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12758,"num_write_kb":446528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.070111+0000","last_change":"2024-02-05T08:08:39.475407+0000","last_active":"2024-02-05T08:09:56.070111+0000","last_peered":"2024-02-05T08:09:56.070111+0000","last_clean":"2024-02-05T08:09:56.070111+0000","last_became_active":"2024-02-05T08:06:39.555147+0000","last_became_peered":"2024-02-05T08:06:39.555147+0000","last_unstale":"2024-02-05T08:09:56.070111+0000","last_undegraded":"2024-02-05T08:09:56.070111+0000","last_fullsized":"2024-02-05T08:09:56.070111+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:39.475316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.475316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:02.082601+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":204,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.055235+0000","last_change":"2024-02-05T08:08:41.006239+0000","last_active":"2024-02-05T08:09:56.055235+0000","last_peered":"2024-02-05T08:09:56.055235+0000","last_clean":"2024-02-05T08:09:56.055235+0000","last_became_active":"2024-02-05T08:06:39.560285+0000","last_became_peered":"2024-02-05T08:06:39.560285+0000","last_unstale":"2024-02-05T08:09:56.055235+0000","last_undegraded":"2024-02-05T08:09:56.055235+0000","last_fullsized":"2024-02-05T08:09:56.055235+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:08:41.005907+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:08:41.005907+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:03.040823+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":204,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.070420+0000","last_change":"2024-02-05T08:08:44.750119+0000","last_active":"2024-02-05T08:09:56.070420+0000","last_peered":"2024-02-05T08:09:56.070420+0000","last_clean":"2024-02-05T08:09:56.070420+0000","last_became_active":"2024-02-05T08:04:40.529073+0000","last_became_peered":"2024-02-05T08:04:40.529073+0000","last_unstale":"2024-02-05T08:09:56.070420+0000","last_undegraded":"2024-02-05T08:09:56.070420+0000","last_fullsized":"2024-02-05T08:09:56.070420+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:44.750015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.750015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:08.380412+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.070075+0000","last_change":"2024-02-05T08:08:37.676223+0000","last_active":"2024-02-05T08:09:56.070075+0000","last_peered":"2024-02-05T08:09:56.070075+0000","last_clean":"2024-02-05T08:09:56.070075+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:09:56.070075+0000","last_undegraded":"2024-02-05T08:09:56.070075+0000","last_fullsized":"2024-02-05T08:09:56.070075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:08:37.676082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:37.676082+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:05.050980+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.071227+0000","last_change":"2024-02-05T08:09:25.547689+0000","last_active":"2024-02-05T08:09:56.071227+0000","last_peered":"2024-02-05T08:09:56.071227+0000","last_clean":"2024-02-05T08:09:56.071227+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:09:56.071227+0000","last_undegraded":"2024-02-05T08:09:56.071227+0000","last_fullsized":"2024-02-05T08:09:56.071227+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.547581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.547581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:55.227516+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"146'12590","reported_seq":12736,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:03.034268+0000","last_change":"2024-02-05T08:09:37.898638+0000","last_active":"2024-02-05T08:10:03.034268+0000","last_peered":"2024-02-05T08:10:03.034268+0000","last_clean":"2024-02-05T08:10:03.034268+0000","last_became_active":"2024-02-05T08:06:37.622047+0000","last_became_peered":"2024-02-05T08:06:37.622047+0000","last_unstale":"2024-02-05T08:10:03.034268+0000","last_undegraded":"2024-02-05T08:10:03.034268+0000","last_fullsized":"2024-02-05T08:10:03.034268+0000","mapping_epoch":95,"log_start":"128'7800","ondisk_log_start":"128'7800","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'11226","last_scrub_stamp":"2024-02-05T08:09:37.898513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:37.898513+0000","objects_scrubbed":2513,"log_size":4790,"log_dups_size":0,"ondisk_log_size":4790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:04.919652+0000","scrub_duration":880,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74448896,"num_objects":1136,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12590,"num_write_kb":439232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060379+0000","last_change":"2024-02-05T08:08:34.051757+0000","last_active":"2024-02-05T08:09:56.060379+0000","last_peered":"2024-02-05T08:09:56.060379+0000","last_clean":"2024-02-05T08:09:56.060379+0000","last_became_active":"2024-02-05T08:04:47.586358+0000","last_became_peered":"2024-02-05T08:04:47.586358+0000","last_unstale":"2024-02-05T08:09:56.060379+0000","last_undegraded":"2024-02-05T08:09:56.060379+0000","last_fullsized":"2024-02-05T08:09:56.060379+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:34.051658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:34.051658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.490754+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":459,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.069521+0000","last_change":"2024-02-05T08:08:49.423194+0000","last_active":"2024-02-05T08:09:56.069521+0000","last_peered":"2024-02-05T08:09:56.069521+0000","last_clean":"2024-02-05T08:09:56.069521+0000","last_became_active":"2024-02-05T08:06:42.566771+0000","last_became_peered":"2024-02-05T08:06:42.566771+0000","last_unstale":"2024-02-05T08:09:56.069521+0000","last_undegraded":"2024-02-05T08:09:56.069521+0000","last_fullsized":"2024-02-05T08:09:56.069521+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:08:49.423065+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:08:49.423065+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.639264+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":192,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.083065+0000","last_change":"2024-02-05T08:08:39.073558+0000","last_active":"2024-02-05T08:09:56.083065+0000","last_peered":"2024-02-05T08:09:56.083065+0000","last_clean":"2024-02-05T08:09:56.083065+0000","last_became_active":"2024-02-05T08:06:39.553704+0000","last_became_peered":"2024-02-05T08:06:39.553704+0000","last_unstale":"2024-02-05T08:09:56.083065+0000","last_undegraded":"2024-02-05T08:09:56.083065+0000","last_fullsized":"2024-02-05T08:09:56.083065+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.071354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:56.133654+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":219,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.650679+0000","last_change":"2024-02-05T08:07:51.975957+0000","last_active":"2024-02-05T08:09:56.650679+0000","last_peered":"2024-02-05T08:09:56.650679+0000","last_clean":"2024-02-05T08:09:56.650679+0000","last_became_active":"2024-02-05T08:06:39.563373+0000","last_became_peered":"2024-02-05T08:06:39.563373+0000","last_unstale":"2024-02-05T08:09:56.650679+0000","last_undegraded":"2024-02-05T08:09:56.650679+0000","last_fullsized":"2024-02-05T08:09:56.650679+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-02-05T08:07:51.975838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:07:51.975838+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":179,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.071269+0000","last_change":"2024-02-05T08:08:38.472753+0000","last_active":"2024-02-05T08:09:56.071269+0000","last_peered":"2024-02-05T08:09:56.071269+0000","last_clean":"2024-02-05T08:09:56.071269+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:09:56.071269+0000","last_undegraded":"2024-02-05T08:09:56.071269+0000","last_fullsized":"2024-02-05T08:09:56.071269+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:38.472647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:38.472647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:59.070109+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"146'12568","reported_seq":12745,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:02.438756+0000","last_change":"2024-02-05T08:08:42.646048+0000","last_active":"2024-02-05T08:10:02.438756+0000","last_peered":"2024-02-05T08:10:02.438756+0000","last_clean":"2024-02-05T08:10:02.438756+0000","last_became_active":"2024-02-05T08:06:37.623983+0000","last_became_peered":"2024-02-05T08:06:37.623983+0000","last_unstale":"2024-02-05T08:10:02.438756+0000","last_undegraded":"2024-02-05T08:10:02.438756+0000","last_fullsized":"2024-02-05T08:10:02.438756+0000","mapping_epoch":95,"log_start":"128'7500","ondisk_log_start":"128'7500","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'8153","last_scrub_stamp":"2024-02-05T08:08:42.645901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:42.645901+0000","objects_scrubbed":5574,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:08.439891+0000","scrub_duration":1122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74055680,"num_objects":1130,"num_object_clones":0,"num_object_copies":3390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12568,"num_write_kb":438336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":175,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.059741+0000","last_change":"2024-02-05T08:07:43.938444+0000","last_active":"2024-02-05T08:09:56.059741+0000","last_peered":"2024-02-05T08:09:56.059741+0000","last_clean":"2024-02-05T08:09:56.059741+0000","last_became_active":"2024-02-05T08:04:47.586774+0000","last_became_peered":"2024-02-05T08:04:47.586774+0000","last_unstale":"2024-02-05T08:09:56.059741+0000","last_undegraded":"2024-02-05T08:09:56.059741+0000","last_fullsized":"2024-02-05T08:09:56.059741+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:43.938278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:43.938278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.277342+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.057950+0000","last_change":"2024-02-05T08:08:38.705016+0000","last_active":"2024-02-05T08:09:56.057950+0000","last_peered":"2024-02-05T08:09:56.057950+0000","last_clean":"2024-02-05T08:09:56.057950+0000","last_became_active":"2024-02-05T08:04:41.537872+0000","last_became_peered":"2024-02-05T08:04:41.537872+0000","last_unstale":"2024-02-05T08:09:56.057950+0000","last_undegraded":"2024-02-05T08:09:56.057950+0000","last_fullsized":"2024-02-05T08:09:56.057950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:38.704911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:38.704911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.112238+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":522,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:01.173602+0000","last_change":"2024-02-05T08:09:26.492000+0000","last_active":"2024-02-05T08:10:01.173602+0000","last_peered":"2024-02-05T08:10:01.173602+0000","last_clean":"2024-02-05T08:10:01.173602+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:10:01.173602+0000","last_undegraded":"2024-02-05T08:10:01.173602+0000","last_fullsized":"2024-02-05T08:10:01.173602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:09:26.491884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:09:26.491884+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:39.146672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":315,"num_read_kb":310,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":200,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.058712+0000","last_change":"2024-02-05T08:09:19.836523+0000","last_active":"2024-02-05T08:09:56.058712+0000","last_peered":"2024-02-05T08:09:56.058712+0000","last_clean":"2024-02-05T08:09:56.058712+0000","last_became_active":"2024-02-05T08:04:43.549687+0000","last_became_peered":"2024-02-05T08:04:43.549687+0000","last_unstale":"2024-02-05T08:09:56.058712+0000","last_undegraded":"2024-02-05T08:09:56.058712+0000","last_fullsized":"2024-02-05T08:09:56.058712+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:09:19.836401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.836401+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:38.806378+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":178,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060598+0000","last_change":"2024-02-05T08:08:29.956921+0000","last_active":"2024-02-05T08:09:56.060598+0000","last_peered":"2024-02-05T08:09:56.060598+0000","last_clean":"2024-02-05T08:09:56.060598+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:09:56.060598+0000","last_undegraded":"2024-02-05T08:09:56.060598+0000","last_fullsized":"2024-02-05T08:09:56.060598+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:29.956845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:29.956845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.490770+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"146'15938","reported_seq":16239,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:02.519118+0000","last_change":"2024-02-05T08:09:18.016743+0000","last_active":"2024-02-05T08:10:02.519118+0000","last_peered":"2024-02-05T08:10:02.519118+0000","last_clean":"2024-02-05T08:10:02.519118+0000","last_became_active":"2024-02-05T08:06:37.624241+0000","last_became_peered":"2024-02-05T08:06:37.624241+0000","last_unstale":"2024-02-05T08:10:02.519118+0000","last_undegraded":"2024-02-05T08:10:02.519118+0000","last_fullsized":"2024-02-05T08:10:02.519118+0000","mapping_epoch":95,"log_start":"132'10402","ondisk_log_start":"132'10402","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'13560","last_scrub_stamp":"2024-02-05T08:09:18.016504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:18.016504+0000","objects_scrubbed":3593,"log_size":5536,"log_dups_size":0,"ondisk_log_size":5536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:30.849106+0000","scrub_duration":742,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78446592,"num_objects":1197,"num_object_clones":0,"num_object_copies":3591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12505,"num_write_kb":438464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060253+0000","last_change":"2024-02-05T08:07:41.002345+0000","last_active":"2024-02-05T08:09:56.060253+0000","last_peered":"2024-02-05T08:09:56.060253+0000","last_clean":"2024-02-05T08:09:56.060253+0000","last_became_active":"2024-02-05T08:06:39.559847+0000","last_became_peered":"2024-02-05T08:06:39.559847+0000","last_unstale":"2024-02-05T08:09:56.060253+0000","last_undegraded":"2024-02-05T08:09:56.060253+0000","last_fullsized":"2024-02-05T08:09:56.060253+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:41.002067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:41.002067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.277216+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.062112+0000","last_change":"2024-02-05T08:08:44.964475+0000","last_active":"2024-02-05T08:09:56.062112+0000","last_peered":"2024-02-05T08:09:56.062112+0000","last_clean":"2024-02-05T08:09:56.062112+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:09:56.062112+0000","last_undegraded":"2024-02-05T08:09:56.062112+0000","last_fullsized":"2024-02-05T08:09:56.062112+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:44.964357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.964357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.850260+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":218,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.058026+0000","last_change":"2024-02-05T08:07:33.417917+0000","last_active":"2024-02-05T08:09:56.058026+0000","last_peered":"2024-02-05T08:09:56.058026+0000","last_clean":"2024-02-05T08:09:56.058026+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:09:56.058026+0000","last_undegraded":"2024-02-05T08:09:56.058026+0000","last_fullsized":"2024-02-05T08:09:56.058026+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:07:33.417781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:07:33.417781+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.344226+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":117,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.058121+0000","last_change":"2024-02-05T08:08:31.623904+0000","last_active":"2024-02-05T08:09:56.058121+0000","last_peered":"2024-02-05T08:09:56.058121+0000","last_clean":"2024-02-05T08:09:56.058121+0000","last_became_active":"2024-02-05T08:06:39.558100+0000","last_became_peered":"2024-02-05T08:06:39.558100+0000","last_unstale":"2024-02-05T08:09:56.058121+0000","last_undegraded":"2024-02-05T08:09:56.058121+0000","last_fullsized":"2024-02-05T08:09:56.058121+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:31.623748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:08:31.623748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.344541+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"146'12551","reported_seq":12730,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:02.481336+0000","last_change":"2024-02-05T08:08:46.770155+0000","last_active":"2024-02-05T08:10:02.481336+0000","last_peered":"2024-02-05T08:10:02.481336+0000","last_clean":"2024-02-05T08:10:02.481336+0000","last_became_active":"2024-02-05T08:06:39.561154+0000","last_became_peered":"2024-02-05T08:06:39.561154+0000","last_unstale":"2024-02-05T08:10:02.481336+0000","last_undegraded":"2024-02-05T08:10:02.481336+0000","last_fullsized":"2024-02-05T08:10:02.481336+0000","mapping_epoch":97,"log_start":"129'8000","ondisk_log_start":"129'8000","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8468","last_scrub_stamp":"2024-02-05T08:08:46.770050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:46.770050+0000","objects_scrubbed":5286,"log_size":4551,"log_dups_size":0,"ondisk_log_size":4551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77135872,"num_objects":1177,"num_object_clones":0,"num_object_copies":3531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12551,"num_write_kb":439296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.056924+0000","last_change":"2024-02-05T08:07:34.484047+0000","last_active":"2024-02-05T08:09:56.056924+0000","last_peered":"2024-02-05T08:09:56.056924+0000","last_clean":"2024-02-05T08:09:56.056924+0000","last_became_active":"2024-02-05T08:04:47.583866+0000","last_became_peered":"2024-02-05T08:04:47.583866+0000","last_unstale":"2024-02-05T08:09:56.056924+0000","last_undegraded":"2024-02-05T08:09:56.056924+0000","last_fullsized":"2024-02-05T08:09:56.056924+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:34.483937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:34.483937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.344304+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060290+0000","last_change":"2024-02-05T08:09:40.402414+0000","last_active":"2024-02-05T08:09:56.060290+0000","last_peered":"2024-02-05T08:09:56.060290+0000","last_clean":"2024-02-05T08:09:56.060290+0000","last_became_active":"2024-02-05T08:04:41.537818+0000","last_became_peered":"2024-02-05T08:04:41.537818+0000","last_unstale":"2024-02-05T08:09:56.060290+0000","last_undegraded":"2024-02-05T08:09:56.060290+0000","last_fullsized":"2024-02-05T08:09:56.060290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:40.402311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:40.402311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.704969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":198,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.061436+0000","last_change":"2024-02-05T08:07:47.955278+0000","last_active":"2024-02-05T08:09:56.061436+0000","last_peered":"2024-02-05T08:09:56.061436+0000","last_clean":"2024-02-05T08:09:56.061436+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:09:56.061436+0000","last_undegraded":"2024-02-05T08:09:56.061436+0000","last_fullsized":"2024-02-05T08:09:56.061436+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:07:47.955013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:07:47.955013+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.277252+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":261,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.650659+0000","last_change":"2024-02-05T08:07:48.538033+0000","last_active":"2024-02-05T08:09:56.650659+0000","last_peered":"2024-02-05T08:09:56.650659+0000","last_clean":"2024-02-05T08:09:56.650659+0000","last_became_active":"2024-02-05T08:06:39.554860+0000","last_became_peered":"2024-02-05T08:06:39.554860+0000","last_unstale":"2024-02-05T08:09:56.650659+0000","last_undegraded":"2024-02-05T08:09:56.650659+0000","last_fullsized":"2024-02-05T08:09:56.650659+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:07:48.537913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:48.537913+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:04.258596+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"146'12622","reported_seq":12794,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:02.816197+0000","last_change":"2024-02-05T08:09:22.744155+0000","last_active":"2024-02-05T08:10:02.816197+0000","last_peered":"2024-02-05T08:10:02.816197+0000","last_clean":"2024-02-05T08:10:02.816197+0000","last_became_active":"2024-02-05T08:06:37.621499+0000","last_became_peered":"2024-02-05T08:06:37.621499+0000","last_unstale":"2024-02-05T08:10:02.816197+0000","last_undegraded":"2024-02-05T08:10:02.816197+0000","last_fullsized":"2024-02-05T08:10:02.816197+0000","mapping_epoch":95,"log_start":"130'8400","ondisk_log_start":"130'8400","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10467","last_scrub_stamp":"2024-02-05T08:09:22.743920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.743920+0000","objects_scrubbed":3450,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:42.513741+0000","scrub_duration":2390,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80347136,"num_objects":1226,"num_object_clones":0,"num_object_copies":3678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12622,"num_write_kb":443136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":181,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.069928+0000","last_change":"2024-02-05T08:08:43.475720+0000","last_active":"2024-02-05T08:09:56.069928+0000","last_peered":"2024-02-05T08:09:56.069928+0000","last_clean":"2024-02-05T08:09:56.069928+0000","last_became_active":"2024-02-05T08:06:39.554165+0000","last_became_peered":"2024-02-05T08:06:39.554165+0000","last_unstale":"2024-02-05T08:09:56.069928+0000","last_undegraded":"2024-02-05T08:09:56.069928+0000","last_fullsized":"2024-02-05T08:09:56.069928+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:43.475636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:43.475636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:07.795888+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.058797+0000","last_change":"2024-02-05T08:07:25.394409+0000","last_active":"2024-02-05T08:09:56.058797+0000","last_peered":"2024-02-05T08:09:56.058797+0000","last_clean":"2024-02-05T08:09:56.058797+0000","last_became_active":"2024-02-05T08:04:41.534433+0000","last_became_peered":"2024-02-05T08:04:41.534433+0000","last_unstale":"2024-02-05T08:09:56.058797+0000","last_undegraded":"2024-02-05T08:09:56.058797+0000","last_fullsized":"2024-02-05T08:09:56.058797+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:07:25.394122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:25.394122+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.344358+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":217,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049515+0000","last_change":"2024-02-05T08:07:34.844340+0000","last_active":"2024-02-05T08:09:56.049515+0000","last_peered":"2024-02-05T08:09:56.049515+0000","last_clean":"2024-02-05T08:09:56.049515+0000","last_became_active":"2024-02-05T08:06:39.566434+0000","last_became_peered":"2024-02-05T08:06:39.566434+0000","last_unstale":"2024-02-05T08:09:56.049515+0000","last_undegraded":"2024-02-05T08:09:56.049515+0000","last_fullsized":"2024-02-05T08:09:56.049515+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-02-05T08:07:34.844214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:07:34.844214+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.386256+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":251,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.654112+0000","last_change":"2024-02-05T08:07:47.765640+0000","last_active":"2024-02-05T08:09:56.654112+0000","last_peered":"2024-02-05T08:09:56.654112+0000","last_clean":"2024-02-05T08:09:56.654112+0000","last_became_active":"2024-02-05T08:04:45.562597+0000","last_became_peered":"2024-02-05T08:04:45.562597+0000","last_unstale":"2024-02-05T08:09:56.654112+0000","last_undegraded":"2024-02-05T08:09:56.654112+0000","last_fullsized":"2024-02-05T08:09:56.654112+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:07:47.765523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:47.765523+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:02.050676+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"146'12627","reported_seq":12804,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:02.737657+0000","last_change":"2024-02-05T08:08:44.289281+0000","last_active":"2024-02-05T08:10:02.737657+0000","last_peered":"2024-02-05T08:10:02.737657+0000","last_clean":"2024-02-05T08:10:02.737657+0000","last_became_active":"2024-02-05T08:06:37.621439+0000","last_became_peered":"2024-02-05T08:06:37.621439+0000","last_unstale":"2024-02-05T08:10:02.737657+0000","last_undegraded":"2024-02-05T08:10:02.737657+0000","last_fullsized":"2024-02-05T08:10:02.737657+0000","mapping_epoch":95,"log_start":"130'8400","ondisk_log_start":"130'8400","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8323","last_scrub_stamp":"2024-02-05T08:08:44.289025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.289025+0000","objects_scrubbed":5548,"log_size":4227,"log_dups_size":0,"ondisk_log_size":4227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.077945+0000","scrub_duration":2139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78315520,"num_objects":1195,"num_object_clones":0,"num_object_copies":3585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12627,"num_write_kb":442304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049337+0000","last_change":"2024-02-05T08:07:40.845208+0000","last_active":"2024-02-05T08:09:56.049337+0000","last_peered":"2024-02-05T08:09:56.049337+0000","last_clean":"2024-02-05T08:09:56.049337+0000","last_became_active":"2024-02-05T08:04:47.582776+0000","last_became_peered":"2024-02-05T08:04:47.582776+0000","last_unstale":"2024-02-05T08:09:56.049337+0000","last_undegraded":"2024-02-05T08:09:56.049337+0000","last_fullsized":"2024-02-05T08:09:56.049337+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:40.845130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:40.845130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.386232+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":200,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.072821+0000","last_change":"2024-02-05T08:09:36.111657+0000","last_active":"2024-02-05T08:09:56.072821+0000","last_peered":"2024-02-05T08:09:56.072821+0000","last_clean":"2024-02-05T08:09:56.072821+0000","last_became_active":"2024-02-05T08:04:41.540818+0000","last_became_peered":"2024-02-05T08:04:41.540818+0000","last_unstale":"2024-02-05T08:09:56.072821+0000","last_undegraded":"2024-02-05T08:09:56.072821+0000","last_fullsized":"2024-02-05T08:09:56.072821+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:09:36.111537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.111537+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.119806+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":222,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.073840+0000","last_change":"2024-02-05T08:09:33.222821+0000","last_active":"2024-02-05T08:09:56.073840+0000","last_peered":"2024-02-05T08:09:56.073840+0000","last_clean":"2024-02-05T08:09:56.073840+0000","last_became_active":"2024-02-05T08:04:43.607546+0000","last_became_peered":"2024-02-05T08:04:43.607546+0000","last_unstale":"2024-02-05T08:09:56.073840+0000","last_undegraded":"2024-02-05T08:09:56.073840+0000","last_fullsized":"2024-02-05T08:09:56.073840+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-05T08:09:33.222532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:33.222532+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:53.393561+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":178,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049293+0000","last_change":"2024-02-05T08:09:20.449787+0000","last_active":"2024-02-05T08:09:56.049293+0000","last_peered":"2024-02-05T08:09:56.049293+0000","last_clean":"2024-02-05T08:09:56.049293+0000","last_became_active":"2024-02-05T08:04:45.561639+0000","last_became_peered":"2024-02-05T08:04:45.561639+0000","last_unstale":"2024-02-05T08:09:56.049293+0000","last_undegraded":"2024-02-05T08:09:56.049293+0000","last_fullsized":"2024-02-05T08:09:56.049293+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:20.449711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:20.449711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:22.864643+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"146'12522","reported_seq":12727,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:02.808663+0000","last_change":"2024-02-05T08:09:19.688660+0000","last_active":"2024-02-05T08:10:02.808663+0000","last_peered":"2024-02-05T08:10:02.808663+0000","last_clean":"2024-02-05T08:10:02.808663+0000","last_became_active":"2024-02-05T08:06:39.556908+0000","last_became_peered":"2024-02-05T08:06:39.556908+0000","last_unstale":"2024-02-05T08:10:02.808663+0000","last_undegraded":"2024-02-05T08:10:02.808663+0000","last_fullsized":"2024-02-05T08:10:02.808663+0000","mapping_epoch":97,"log_start":"130'8300","ondisk_log_start":"130'8300","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'10128","last_scrub_stamp":"2024-02-05T08:09:19.688502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.688502+0000","objects_scrubbed":3625,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:37.670381+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77594624,"num_objects":1184,"num_object_clones":0,"num_object_copies":3552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12522,"num_write_kb":438592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060188+0000","last_change":"2024-02-05T08:08:43.142178+0000","last_active":"2024-02-05T08:09:56.060188+0000","last_peered":"2024-02-05T08:09:56.060188+0000","last_clean":"2024-02-05T08:09:56.060188+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:09:56.060188+0000","last_undegraded":"2024-02-05T08:09:56.060188+0000","last_fullsized":"2024-02-05T08:09:56.060188+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:43.142051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:43.142051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:56.132856+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":251,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.650731+0000","last_change":"2024-02-05T08:09:17.391826+0000","last_active":"2024-02-05T08:09:56.650731+0000","last_peered":"2024-02-05T08:09:56.650731+0000","last_clean":"2024-02-05T08:09:56.650731+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:09:56.650731+0000","last_undegraded":"2024-02-05T08:09:56.650731+0000","last_fullsized":"2024-02-05T08:09:56.650731+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:09:17.391673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:17.391673+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:31.892417+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.073761+0000","last_change":"2024-02-05T08:07:36.643035+0000","last_active":"2024-02-05T08:09:56.073761+0000","last_peered":"2024-02-05T08:09:56.073761+0000","last_clean":"2024-02-05T08:09:56.073761+0000","last_became_active":"2024-02-05T08:04:41.541327+0000","last_became_peered":"2024-02-05T08:04:41.541327+0000","last_unstale":"2024-02-05T08:09:56.073761+0000","last_undegraded":"2024-02-05T08:09:56.073761+0000","last_fullsized":"2024-02-05T08:09:56.073761+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:36.642926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:36.642926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:56.133547+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":216,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.061983+0000","last_change":"2024-02-05T08:09:17.191152+0000","last_active":"2024-02-05T08:09:56.061983+0000","last_peered":"2024-02-05T08:09:56.061983+0000","last_clean":"2024-02-05T08:09:56.061983+0000","last_became_active":"2024-02-05T08:04:43.555028+0000","last_became_peered":"2024-02-05T08:04:43.555028+0000","last_unstale":"2024-02-05T08:09:56.061983+0000","last_undegraded":"2024-02-05T08:09:56.061983+0000","last_fullsized":"2024-02-05T08:09:56.061983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:09:17.190290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:17.190290+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:23.351754+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"146'12625","reported_seq":12793,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:02.811294+0000","last_change":"2024-02-05T08:07:48.231620+0000","last_active":"2024-02-05T08:10:02.811294+0000","last_peered":"2024-02-05T08:10:02.811294+0000","last_clean":"2024-02-05T08:10:02.811294+0000","last_became_active":"2024-02-05T08:06:39.555144+0000","last_became_peered":"2024-02-05T08:06:39.555144+0000","last_unstale":"2024-02-05T08:10:02.811294+0000","last_undegraded":"2024-02-05T08:10:02.811294+0000","last_fullsized":"2024-02-05T08:10:02.811294+0000","mapping_epoch":97,"log_start":"130'8400","ondisk_log_start":"130'8400","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4972","last_scrub_stamp":"2024-02-05T08:07:48.231433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:07:48.231433+0000","objects_scrubbed":4921,"log_size":4225,"log_dups_size":0,"ondisk_log_size":4225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:04.426185+0000","scrub_duration":2285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80609304,"num_objects":1231,"num_object_clones":0,"num_object_copies":3693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1231,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12625,"num_write_kb":443329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":180,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.067938+0000","last_change":"2024-02-05T08:09:32.213156+0000","last_active":"2024-02-05T08:09:56.067938+0000","last_peered":"2024-02-05T08:09:56.067938+0000","last_clean":"2024-02-05T08:09:56.067938+0000","last_became_active":"2024-02-05T08:06:39.558047+0000","last_became_peered":"2024-02-05T08:06:39.558047+0000","last_unstale":"2024-02-05T08:09:56.067938+0000","last_undegraded":"2024-02-05T08:09:56.067938+0000","last_fullsized":"2024-02-05T08:09:56.067938+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:32.213044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:09:32.213044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:54.206176+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":184,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.060525+0000","last_change":"2024-02-05T08:07:46.752867+0000","last_active":"2024-02-05T08:09:56.060525+0000","last_peered":"2024-02-05T08:09:56.060525+0000","last_clean":"2024-02-05T08:09:56.060525+0000","last_became_active":"2024-02-05T08:06:39.551933+0000","last_became_peered":"2024-02-05T08:06:39.551933+0000","last_unstale":"2024-02-05T08:09:56.060525+0000","last_undegraded":"2024-02-05T08:09:56.060525+0000","last_fullsized":"2024-02-05T08:09:56.060525+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:46.752781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:46.752781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.490600+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049907+0000","last_change":"2024-02-05T08:09:22.454321+0000","last_active":"2024-02-05T08:09:56.049907+0000","last_peered":"2024-02-05T08:09:56.049907+0000","last_clean":"2024-02-05T08:09:56.049907+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:09:56.049907+0000","last_undegraded":"2024-02-05T08:09:56.049907+0000","last_fullsized":"2024-02-05T08:09:56.049907+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:22.454247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.454247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:32.361376+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":219,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049744+0000","last_change":"2024-02-05T08:08:32.390911+0000","last_active":"2024-02-05T08:09:56.049744+0000","last_peered":"2024-02-05T08:09:56.049744+0000","last_clean":"2024-02-05T08:09:56.049744+0000","last_became_active":"2024-02-05T08:06:39.556433+0000","last_became_peered":"2024-02-05T08:06:39.556433+0000","last_unstale":"2024-02-05T08:09:56.049744+0000","last_undegraded":"2024-02-05T08:09:56.049744+0000","last_fullsized":"2024-02-05T08:09:56.049744+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:08:32.390785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:32.390785+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.386422+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"146'12689","reported_seq":12869,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:02.680360+0000","last_change":"2024-02-05T08:08:39.042845+0000","last_active":"2024-02-05T08:10:02.680360+0000","last_peered":"2024-02-05T08:10:02.680360+0000","last_clean":"2024-02-05T08:10:02.680360+0000","last_became_active":"2024-02-05T08:06:37.624003+0000","last_became_peered":"2024-02-05T08:06:37.624003+0000","last_unstale":"2024-02-05T08:10:02.680360+0000","last_undegraded":"2024-02-05T08:10:02.680360+0000","last_fullsized":"2024-02-05T08:10:02.680360+0000","mapping_epoch":95,"log_start":"127'7300","ondisk_log_start":"127'7300","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'8070","last_scrub_stamp":"2024-02-05T08:08:39.042536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.042536+0000","objects_scrubbed":5875,"log_size":5389,"log_dups_size":0,"ondisk_log_size":5389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:02.957363+0000","scrub_duration":2379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76873728,"num_objects":1173,"num_object_clones":0,"num_object_copies":3519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12689,"num_write_kb":443584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":184,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.070333+0000","last_change":"2024-02-05T08:08:45.814521+0000","last_active":"2024-02-05T08:09:56.070333+0000","last_peered":"2024-02-05T08:09:56.070333+0000","last_clean":"2024-02-05T08:09:56.070333+0000","last_became_active":"2024-02-05T08:04:47.585962+0000","last_became_peered":"2024-02-05T08:04:47.585962+0000","last_unstale":"2024-02-05T08:09:56.070333+0000","last_undegraded":"2024-02-05T08:09:56.070333+0000","last_fullsized":"2024-02-05T08:09:56.070333+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:08:45.814384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:45.814384+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:08.553734+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":179,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.062342+0000","last_change":"2024-02-05T08:08:29.100467+0000","last_active":"2024-02-05T08:09:56.062342+0000","last_peered":"2024-02-05T08:09:56.062342+0000","last_clean":"2024-02-05T08:09:56.062342+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:09:56.062342+0000","last_undegraded":"2024-02-05T08:09:56.062342+0000","last_fullsized":"2024-02-05T08:09:56.062342+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:29.100364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:29.100364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:58.277432+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":117,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049390+0000","last_change":"2024-02-05T08:07:53.915557+0000","last_active":"2024-02-05T08:09:56.049390+0000","last_peered":"2024-02-05T08:09:56.049390+0000","last_clean":"2024-02-05T08:09:56.049390+0000","last_became_active":"2024-02-05T08:06:39.559090+0000","last_became_peered":"2024-02-05T08:06:39.559090+0000","last_unstale":"2024-02-05T08:09:56.049390+0000","last_undegraded":"2024-02-05T08:09:56.049390+0000","last_fullsized":"2024-02-05T08:09:56.049390+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:07:53.915449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:57.386304+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":210,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049758+0000","last_change":"2024-02-05T08:09:21.434983+0000","last_active":"2024-02-05T08:09:56.049758+0000","last_peered":"2024-02-05T08:09:56.049758+0000","last_clean":"2024-02-05T08:09:56.049758+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:09:56.049758+0000","last_undegraded":"2024-02-05T08:09:56.049758+0000","last_fullsized":"2024-02-05T08:09:56.049758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-02-05T08:09:21.434891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.434891+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:48.312307+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"146'12892","reported_seq":13067,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:10:02.991903+0000","last_change":"2024-02-05T08:08:50.574791+0000","last_active":"2024-02-05T08:10:02.991903+0000","last_peered":"2024-02-05T08:10:02.991903+0000","last_clean":"2024-02-05T08:10:02.991903+0000","last_became_active":"2024-02-05T08:06:37.622936+0000","last_became_peered":"2024-02-05T08:06:37.622936+0000","last_unstale":"2024-02-05T08:10:02.991903+0000","last_undegraded":"2024-02-05T08:10:02.991903+0000","last_fullsized":"2024-02-05T08:10:02.991903+0000","mapping_epoch":95,"log_start":"129'8100","ondisk_log_start":"129'8100","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8798","last_scrub_stamp":"2024-02-05T08:08:50.574449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:50.574449+0000","objects_scrubbed":5317,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:13.984937+0000","scrub_duration":1873,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78249984,"num_objects":1194,"num_object_clones":0,"num_object_copies":3582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12892,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.050019+0000","last_change":"2024-02-05T08:08:49.450529+0000","last_active":"2024-02-05T08:09:56.050019+0000","last_peered":"2024-02-05T08:09:56.050019+0000","last_clean":"2024-02-05T08:09:56.050019+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:09:56.050019+0000","last_undegraded":"2024-02-05T08:09:56.050019+0000","last_fullsized":"2024-02-05T08:09:56.050019+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:49.450400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:49.450400+0000","last_clean_scrub_stamp":"2024-02-05T08:08:49.450400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:59.625254+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":261,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.651280+0000","last_change":"2024-02-05T08:08:39.742015+0000","last_active":"2024-02-05T08:09:56.651280+0000","last_peered":"2024-02-05T08:09:56.651280+0000","last_clean":"2024-02-05T08:09:56.651280+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:09:56.651280+0000","last_undegraded":"2024-02-05T08:09:56.651280+0000","last_fullsized":"2024-02-05T08:09:56.651280+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.741871+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:07.770788+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.076252+0000","last_change":"2024-02-05T08:09:22.380135+0000","last_active":"2024-02-05T08:09:56.076252+0000","last_peered":"2024-02-05T08:09:56.076252+0000","last_clean":"2024-02-05T08:09:56.076252+0000","last_became_active":"2024-02-05T08:04:41.533838+0000","last_became_peered":"2024-02-05T08:04:41.533838+0000","last_unstale":"2024-02-05T08:09:56.076252+0000","last_undegraded":"2024-02-05T08:09:56.076252+0000","last_fullsized":"2024-02-05T08:09:56.076252+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:22.380016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.380016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:34.894648+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":215,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-02-05T08:09:56.049681+0000","last_change":"2024-02-05T08:08:35.372047+0000","last_active":"2024-02-05T08:09:56.049681+0000","last_peered":"2024-02-05T08:09:56.049681+0000","last_clean":"2024-02-05T08:09:56.049681+0000","last_became_active":"2024-02-05T08:06:39.555561+0000","last_became_peered":"2024-02-05T08:06:39.555561+0000","last_unstale":"2024-02-05T08:09:56.049681+0000","last_undegraded":"2024-02-05T08:09:56.049681+0000","last_fullsized":"2024-02-05T08:09:56.049681+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:08:35.371853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:35.371853+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:09:59.486787+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1227554840,"num_objects":18732,"num_object_clones":0,"num_object_copies":56196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":201132,"num_write_kb":7035585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3692179456,"data_stored":3692167240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75372,"ondisk_log_size":75372,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":315,"num_read_kb":310,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":1474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":433,"ondisk_log_size":433,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037829,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":499408,"kb_used_data":387720,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93245744,"statfs":{"total":95995035648,"available":95483641856,"internally_reserved":0,"allocated":397025280,"data_stored":396732310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":3.9300000000000002,"5min":1.468,"15min":0.85999999999999999},"min":{"1min":0.23000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":60.127000000000002,"5min":60.127000000000002,"15min":60.127000000000002},"last":2.7709999999999999},{"interface":"front","average":{"1min":4.2430000000000003,"5min":1.508,"15min":0.85199999999999998},"min":{"1min":0.251,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":58.283000000000001,"5min":58.283000000000001,"15min":58.283000000000001},"last":0.311}]},{"osd":1,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":2.3999999999999999,"5min":1.331,"15min":0.81499999999999995},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.774000000000001,"5min":20.774000000000001,"15min":20.774000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.702,"5min":1.0249999999999999,"15min":0.70899999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":20.715,"5min":20.715,"15min":20.715},"last":1.419}]},{"osd":2,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.78400000000000003,"15min":0.64900000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6779999999999999,"5min":7.6779999999999999,"15min":7.6779999999999999},"last":3.1269999999999998},{"interface":"front","average":{"1min":1.8109999999999999,"5min":1.1000000000000001,"15min":0.72599999999999998},"min":{"1min":0.26300000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":8.9100000000000001,"5min":8.9100000000000001,"15min":8.9100000000000001},"last":0.253}]},{"osd":3,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":5.4989999999999997,"5min":1.859,"15min":0.999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":58.515000000000001,"5min":58.515000000000001,"15min":58.515000000000001},"last":3.0510000000000002},{"interface":"front","average":{"1min":5.4779999999999998,"5min":1.71,"15min":0.93000000000000005},"min":{"1min":0.307,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":58.225999999999999,"5min":58.225999999999999,"15min":58.225999999999999},"last":2.6930000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.40000000000000002,"15min":0.42499999999999999},"min":{"1min":0.23899999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.49299999999999999,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.41799999999999998,"15min":0.42799999999999999},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.53000000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.28999999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.437,"15min":0.44},"min":{"1min":0.29999999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.90200000000000002,"5min":1.2,"15min":1.2},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46999999999999997,"15min":0.48399999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.81999999999999995,"5min":0.998,"15min":0.998},"last":0.27000000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.87,"5min":1.256,"15min":1.256},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.33700000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.56799999999999995,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.32800000000000001}]}]},{"osd":6,"up_from":0,"seq":219043332171,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":764076,"kb_used_data":616292,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92981076,"statfs":{"total":95995035648,"available":95212621824,"internally_reserved":0,"allocated":631083008,"data_stored":630761098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.95,"5min":1.8720000000000001,"15min":0.82599999999999996},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.14699999999999999},"max":{"1min":6.4779999999999998,"5min":86.105000000000004,"15min":86.105000000000004},"last":1.7190000000000001},{"interface":"front","average":{"1min":2.867,"5min":2.258,"15min":0.96099999999999997},"min":{"1min":0.245,"5min":0.22900000000000001,"15min":0.16},"max":{"1min":10.068,"5min":85.665000000000006,"15min":85.665000000000006},"last":2.5950000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:09:32 2024","interfaces":[{"interface":"back","average":{"1min":1.8380000000000001,"5min":1.972,"15min":0.90600000000000003},"min":{"1min":0.19700000000000001,"5min":0.19500000000000001,"15min":0.17799999999999999},"max":{"1min":6.3079999999999998,"5min":49.030999999999999,"15min":49.030999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":2.2959999999999998,"5min":2.1269999999999998,"15min":0.97799999999999998},"min":{"1min":0.34300000000000003,"5min":0.23100000000000001,"15min":0.21099999999999999},"max":{"1min":13.707000000000001,"5min":48.984000000000002,"15min":48.984000000000002},"last":6.5590000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:09:59 2024","interfaces":[{"interface":"back","average":{"1min":2.3730000000000002,"5min":2.2349999999999999,"15min":1.04},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":12.468999999999999,"5min":49.067999999999998,"15min":49.067999999999998},"last":1.8540000000000001},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.3130000000000002,"15min":1.0780000000000001},"min":{"1min":0.38,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.3940000000000001,"5min":48.734999999999999,"15min":48.734999999999999},"last":1.5029999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:09:24 2024","interfaces":[{"interface":"back","average":{"1min":1.905,"5min":1.071,"15min":0.748},"min":{"1min":0.29699999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":9.4269999999999996,"5min":9.4269999999999996,"15min":9.4269999999999996},"last":0.37},{"interface":"front","average":{"1min":1.901,"5min":1.0449999999999999,"15min":0.73299999999999998},"min":{"1min":0.30499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.3890000000000002,"5min":6.3890000000000002,"15min":6.3890000000000002},"last":0.47199999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.163,"15min":0.751},"min":{"1min":0.35899999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":7.7000000000000002,"5min":9.4559999999999995,"15min":9.4559999999999995},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.159,"15min":0.75700000000000001},"min":{"1min":0.40400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":13.776,"5min":13.776,"15min":13.776},"last":2.9660000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:09:13 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":0.98499999999999999,"15min":0.72299999999999998},"min":{"1min":0.38800000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.4279999999999999,"5min":6.4279999999999999,"15min":6.4279999999999999},"last":0.497},{"interface":"front","average":{"1min":1.8009999999999999,"5min":1.046,"15min":0.77200000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":0.41899999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:09:32 2024","interfaces":[{"interface":"back","average":{"1min":2.395,"5min":1.2809999999999999,"15min":0.84099999999999997},"min":{"1min":0.434,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":13.749000000000001,"5min":13.749000000000001,"15min":13.749000000000001},"last":3.0009999999999999},{"interface":"front","average":{"1min":2.246,"5min":1.1970000000000001,"15min":0.80500000000000005},"min":{"1min":0.56499999999999995,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.45000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741676,"kb_used_data":616036,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93003476,"statfs":{"total":95995035648,"available":95235559424,"internally_reserved":0,"allocated":630820864,"data_stored":630499780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:09:12 2024","interfaces":[{"interface":"back","average":{"1min":6.7000000000000002,"5min":2.1429999999999998,"15min":0.93500000000000005},"min":{"1min":0.19,"5min":0.19,"15min":0.152},"max":{"1min":66.772999999999996,"5min":66.772999999999996,"15min":66.772999999999996},"last":0.53900000000000003},{"interface":"front","average":{"1min":5.4530000000000003,"5min":1.722,"15min":0.79200000000000004},"min":{"1min":0.30599999999999999,"5min":0.249,"15min":0.17199999999999999},"max":{"1min":46.198999999999998,"5min":46.198999999999998,"15min":46.198999999999998},"last":0.33700000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:09:36 2024","interfaces":[{"interface":"back","average":{"1min":5.1539999999999999,"5min":1.9490000000000001,"15min":0.89800000000000002},"min":{"1min":0.56100000000000005,"5min":0.252,"15min":0.218},"max":{"1min":46.170999999999999,"5min":46.170999999999999,"15min":46.170999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":4.7699999999999996,"5min":2.0960000000000001,"15min":0.95799999999999996},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":46.543999999999997,"5min":46.543999999999997,"15min":46.543999999999997},"last":3.7160000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:10:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7869999999999999,"5min":2.419,"15min":1.113},"min":{"1min":0.42099999999999999,"5min":0.28399999999999997,"15min":0.21099999999999999},"max":{"1min":9.6829999999999998,"5min":67.888000000000005,"15min":67.888000000000005},"last":7.7050000000000001},{"interface":"front","average":{"1min":2.7349999999999999,"5min":2.7229999999999999,"15min":1.22},"min":{"1min":0.41099999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":8.8979999999999997,"5min":66.891999999999996,"15min":66.891999999999996},"last":3.673}]},{"osd":4,"last update":"Mon Feb 5 08:09:22 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":0.96899999999999997,"15min":0.69899999999999995},"min":{"1min":0.38900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.5140000000000002,"5min":13.295,"15min":13.295},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.161,"5min":1.0269999999999999,"15min":0.746},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.5449999999999999,"5min":12.989000000000001,"15min":12.989000000000001},"last":0.56499999999999995}]},{"osd":5,"last update":"Mon Feb 5 08:09:50 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.141,"15min":0.78500000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.0190000000000001,"5min":13.073,"15min":13.073},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.175,"15min":0.80800000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":9.5289999999999999,"5min":12.94,"15min":12.94},"last":0.60599999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.0980000000000001,"15min":0.76700000000000002},"min":{"1min":0.42499999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":4.4290000000000003,"5min":12.967000000000001,"15min":12.967000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.79,"5min":1.1719999999999999,"15min":0.80500000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":5.4560000000000004,"5min":13.108000000000001,"15min":13.108000000000001},"last":0.48899999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.0469999999999999,"15min":0.72499999999999998},"min":{"1min":0.432,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":5.6150000000000002,"5min":13.263,"15min":13.263},"last":0.39000000000000001},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.244,"15min":0.82099999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":9.6300000000000008,"5min":13.012,"15min":13.012},"last":0.66800000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741456,"kb_used_data":611720,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93003696,"statfs":{"total":95995035648,"available":95235784704,"internally_reserved":0,"allocated":626401280,"data_stored":626107026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":2.254,"5min":1.667,"15min":0.80400000000000005},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":12.355,"5min":16.155999999999999,"15min":16.155999999999999},"last":8.0350000000000001},{"interface":"front","average":{"1min":2.1920000000000002,"5min":1.8240000000000001,"15min":0.86299999999999999},"min":{"1min":0.26500000000000001,"5min":0.23300000000000001,"15min":0.20699999999999999},"max":{"1min":14.243,"5min":15.973000000000001,"15min":15.973000000000001},"last":3.9740000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1419999999999999,"5min":1.4179999999999999,"15min":0.71199999999999997},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":7.0519999999999996,"5min":9.1340000000000003,"15min":9.1340000000000003},"last":4.9180000000000001},{"interface":"front","average":{"1min":1.9630000000000001,"5min":1.4259999999999999,"15min":0.72899999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":7.7919999999999998,"5min":9.2769999999999992,"15min":9.2769999999999992},"last":7.9400000000000004}]},{"osd":3,"last update":"Mon Feb 5 08:09:59 2024","interfaces":[{"interface":"back","average":{"1min":5.7229999999999999,"5min":2.984,"15min":1.3069999999999999},"min":{"1min":0.314,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":68.108000000000004,"5min":68.108000000000004,"15min":68.108000000000004},"last":7.9800000000000004},{"interface":"front","average":{"1min":5.7880000000000003,"5min":2.6589999999999998,"15min":1.1919999999999999},"min":{"1min":0.33500000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":67.945999999999998,"5min":67.945999999999998,"15min":67.945999999999998},"last":7.992}]},{"osd":4,"last update":"Mon Feb 5 08:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1739999999999999,"15min":0.79200000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":6.0650000000000004,"5min":15.923999999999999,"15min":15.923999999999999},"last":3.863},{"interface":"front","average":{"1min":1.4390000000000001,"5min":0.94699999999999995,"15min":0.70099999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":6.008,"5min":6.008,"15min":6.008},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:09:50 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.1659999999999999,"15min":0.78100000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.6669999999999998,"5min":15.835000000000001,"15min":15.835000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.036,"15min":0.72699999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":5.4429999999999996,"5min":9.1809999999999992,"15min":9.1809999999999992},"last":0.63500000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:09:15 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.2470000000000001,"15min":0.84099999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":6.0999999999999996,"5min":15.859,"15min":15.859},"last":3.9449999999999998},{"interface":"front","average":{"1min":1.506,"5min":1,"15min":0.76100000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":6.117,"5min":6.117,"15min":6.117},"last":0.57599999999999996}]},{"osd":7,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.2549999999999999,"15min":0.83299999999999996},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.4870000000000001,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.3,"15min":0.88100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.0540000000000003,"5min":15.951000000000001,"15min":15.951000000000001},"last":3.9140000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117082,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":972776,"kb_used_data":788832,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92772376,"statfs":{"total":95995035648,"available":94998913024,"internally_reserved":0,"allocated":807763968,"data_stored":807445865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":188283337},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.1549999999999998,"5min":2.1379999999999999,"15min":0.98499999999999999},"min":{"1min":0.42099999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":6.4169999999999998,"5min":63.43,"15min":63.43},"last":0.39900000000000002},{"interface":"front","average":{"1min":2.5430000000000001,"5min":2.1829999999999998,"15min":1.028},"min":{"1min":0.34300000000000003,"5min":0.186,"15min":0.186},"max":{"1min":8.2650000000000006,"5min":63.048999999999999,"15min":63.048999999999999},"last":0.378}]},{"osd":1,"last update":"Mon Feb 5 08:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":1.496,"15min":0.81999999999999995},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":6.085,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":2.266},{"interface":"front","average":{"1min":3.7719999999999998,"5min":2.0539999999999998,"15min":0.995},"min":{"1min":0.34300000000000003,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":8.1760000000000002,"5min":12.087999999999999,"15min":12.087999999999999},"last":2.278}]},{"osd":2,"last update":"Mon Feb 5 08:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.3919999999999999,"5min":2.3559999999999999,"15min":1.1160000000000001},"min":{"1min":0.29999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":7.2800000000000002,"5min":62.902999999999999,"15min":62.902999999999999},"last":2.012},{"interface":"front","average":{"1min":2.105,"5min":2.198,"15min":1.054},"min":{"1min":0.54100000000000004,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.3730000000000002,"5min":63.468000000000004,"15min":63.468000000000004},"last":0.45400000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:09:20 2024","interfaces":[{"interface":"back","average":{"1min":1.99,"5min":1.2210000000000001,"15min":0.78100000000000003},"min":{"1min":0.30099999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":7.7229999999999999,"5min":7.7229999999999999,"15min":7.7229999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":2.0790000000000002,"5min":1.323,"15min":0.83699999999999997},"min":{"1min":0.32700000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":8.7059999999999995,"5min":8.7059999999999995,"15min":8.7059999999999995},"last":0.34300000000000003}]},{"osd":5,"last update":"Mon Feb 5 08:09:46 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.276,"15min":0.81799999999999995},"min":{"1min":0.33300000000000002,"5min":0.255,"15min":0.255},"max":{"1min":3.9950000000000001,"5min":7.6680000000000001,"15min":7.6680000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.9139999999999999,"5min":1.4159999999999999,"15min":0.872},"min":{"1min":0.42399999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.8860000000000001,"5min":8.734,"15min":8.734},"last":0.439}]},{"osd":6,"last update":"Mon Feb 5 08:09:10 2024","interfaces":[{"interface":"back","average":{"1min":2.2490000000000001,"5min":1.304,"15min":0.83299999999999996},"min":{"1min":0.38400000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":8.9429999999999996,"5min":8.9429999999999996,"15min":8.9429999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.663,"5min":1.1040000000000001,"15min":0.754},"min":{"1min":0.38,"5min":0.308,"15min":0.308},"max":{"1min":5.9509999999999996,"5min":7.8639999999999999,"15min":7.8639999999999999},"last":2.0449999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:09:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.2889999999999999,"15min":0.82499999999999996},"min":{"1min":0.35699999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":7.7000000000000002,"5min":7.7000000000000002,"15min":7.7000000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":2.294,"5min":1.458,"15min":0.90700000000000003},"min":{"1min":0.41399999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":8.9900000000000002,"5min":8.9900000000000002,"15min":8.9900000000000002},"last":0.55400000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953557,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":488024,"kb_used_data":380496,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93257128,"statfs":{"total":95995035648,"available":95495299072,"internally_reserved":0,"allocated":389627904,"data_stored":389325001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:13 2024","interfaces":[{"interface":"back","average":{"1min":2.9889999999999999,"5min":2.137,"15min":1.083},"min":{"1min":0.24199999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":27.966000000000001,"5min":73.078999999999994,"15min":73.078999999999994},"last":0.51400000000000001},{"interface":"front","average":{"1min":2.1110000000000002,"5min":1.9450000000000001,"15min":1.0149999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":26.097999999999999,"5min":73.412000000000006,"15min":73.412000000000006},"last":0.31900000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:09:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":0.80800000000000005,"15min":0.63400000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.0890000000000004,"5min":4.0890000000000004,"15min":4.0890000000000004},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.66700000000000004,"15min":0.58699999999999997},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.956,"5min":2.956,"15min":2.956},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.736,"15min":0.94199999999999995},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.8130000000000002,"5min":96.849999999999994,"15min":96.849999999999994},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.8700000000000001,"15min":1.0089999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.032,"5min":96.143000000000001,"15min":96.143000000000001},"last":0.36599999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:09:13 2024","interfaces":[{"interface":"back","average":{"1min":3.2610000000000001,"5min":2.093,"15min":1.0649999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":26.077000000000002,"5min":73.602999999999994,"15min":73.602999999999994},"last":0.47899999999999998},{"interface":"front","average":{"1min":2.4820000000000002,"5min":2.0169999999999999,"15min":1.04},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":27.425000000000001,"5min":74.817999999999998,"15min":74.817999999999998},"last":0.85999999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:09:50 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.40999999999999998,"15min":0.38},"min":{"1min":0.28100000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63900000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.25},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.44500000000000001,"15min":0.40999999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.439}]},{"osd":6,"last update":"Mon Feb 5 08:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.45500000000000002,"15min":0.433},"min":{"1min":0.30499999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.61299999999999999,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.46200000000000002,"15min":0.43099999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.60499999999999998,"5min":0.747,"15min":0.747},"last":0.35199999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:09:33 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.29299999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.77100000000000002,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.5,"15min":0.502},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68600000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.39700000000000002}]}]},{"osd":5,"up_from":42,"seq":180388626511,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":373316,"kb_used_data":306044,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93371836,"statfs":{"total":95995035648,"available":95612760064,"internally_reserved":0,"allocated":313389056,"data_stored":313106560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":2.2519999999999998,"15min":1.1200000000000001},"min":{"1min":0.23200000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6150000000000002,"5min":143.06299999999999,"15min":143.06299999999999},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.974,"5min":2.2829999999999999,"15min":1.119},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.3570000000000002,"5min":143.09700000000001,"15min":143.09700000000001},"last":4.4699999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.69699999999999995,"15min":0.61399999999999999},"min":{"1min":0.23200000000000001,"5min":0.215,"15min":0.215},"max":{"1min":2.286,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":2.2290000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.755,"15min":0.61499999999999999},"min":{"1min":0.25800000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.2170000000000001,"5min":3.544,"15min":3.544},"last":7.0949999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.90400000000000003,"15min":0.68400000000000005},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.673,"5min":5.0049999999999999,"15min":5.0049999999999999},"last":10.733000000000001},{"interface":"front","average":{"1min":1.272,"5min":1.01,"15min":0.71599999999999997},"min":{"1min":0.28799999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.657,"5min":5.4089999999999998,"15min":5.4089999999999998},"last":0.30199999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":2.375,"15min":1.167},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":7.8410000000000002,"5min":143.03399999999999,"15min":143.03399999999999},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.712,"5min":2.202,"15min":1.1299999999999999},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":9.1839999999999993,"5min":143.46799999999999,"15min":143.46799999999999},"last":3.2090000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.436,"15min":0.40899999999999997},"min":{"1min":0.25800000000000001,"5min":0.253,"15min":0.253},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45000000000000001,"15min":0.42799999999999999},"min":{"1min":0.30399999999999999,"5min":0.26900000000000002,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.32900000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:09:08 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.45600000000000002,"15min":0.44600000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.68799999999999994,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.374},{"interface":"front","average":{"1min":0.501,"5min":0.495,"15min":0.48699999999999999},"min":{"1min":0.34999999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65500000000000003,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.40400000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.47299999999999998},"min":{"1min":0.315,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.76600000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.34499999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.41799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":602935296,"data_stored":602931224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":616960000,"data_stored":616955928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":616562688,"data_stored":616562688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":770379776,"data_stored":770375704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":388628480,"data_stored":388628480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":309198848,"data_stored":309198848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":387514368,"data_stored":387514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:10:07.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:10:07.446 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:07.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:07.526 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:07.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:07.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:07.688 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:07.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:07.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:10:07.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:07.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:07.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:07.918 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:07.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:07.974 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:07.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:08.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:10:08.028 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:10:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:10:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:10:08.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:10:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:10:08.076 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:08.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:08.127 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:08.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:08.228 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:08.274 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:08.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:08.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:08.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:08.401 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:08.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:08.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:08.556 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:08.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:08.657 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:08.706 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:08.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:08.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:08.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:08.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:08.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:08.844 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:08 smithi134 ceph-mon[30086]: pgmap v210: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 932 op/s 2024-02-05T08:10:08.844 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:08 smithi134 ceph-mon[39278]: pgmap v210: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 932 op/s 2024-02-05T08:10:08.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:08.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:08.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:09.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:09.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:08 smithi196 ceph-mon[33018]: pgmap v210: 148 pgs: 148 active+clean; 1.1 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 932 op/s 2024-02-05T08:10:09.089 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:09.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:09.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:09.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:09.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:09.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:09.318 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:09.363 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:09.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:09.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:09.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:09.507 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:09.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:09.608 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:09.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:09.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:09.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:09.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:09.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:09.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:09.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:09.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:09.876 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:10:09.876 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:10:09.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:09.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:09.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:10.026 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:10.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:10.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:10.179 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:10.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:10.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:10.281 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:10.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:10.328 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:10.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:10.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:10.471 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:10.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:10.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:10.561 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:10.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:10.604 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T08:10:10.614 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:10.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:10.642 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:10.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:10.707 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:10:10.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:10.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:10.801 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:10.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:10.856 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:10 smithi134 ceph-mon[30086]: pgmap v211: 148 pgs: 148 active+clean; 861 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.16k op/s 2024-02-05T08:10:10.856 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:10 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1504877260' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:10:10.856 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:10 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:10:10.856 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:10 smithi134 ceph-mon[39278]: pgmap v211: 148 pgs: 148 active+clean; 861 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.16k op/s 2024-02-05T08:10:10.856 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:10 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1504877260' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:10:10.856 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:10 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:10:10.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:10.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:10.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:10.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:11.031 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:11.057 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:11.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:10 smithi196 ceph-mon[33018]: pgmap v211: 148 pgs: 148 active+clean; 861 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.16k op/s 2024-02-05T08:10:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:10 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1504877260' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:10:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:10 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:10:11.159 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:11.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:11.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:11.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:11.286 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:11.312 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:11.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:11.414 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:11.441 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:11.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:11.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:11.542 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:11.586 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:11.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:11.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:11.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:11.737 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:11.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:11.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:11.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:11.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:11.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:11.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:11.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:12.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:11 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:10:12.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:11 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:10:12.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:11 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:10:12.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:11 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:10:12.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:11 smithi134 ceph-mon[30086]: osdmap e147: 8 total, 8 up, 7 in 2024-02-05T08:10:12.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:11 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:10:12.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:11 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:10:12.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:11 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:10:12.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:11 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:10:12.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:11 smithi134 ceph-mon[39278]: osdmap e147: 8 total, 8 up, 7 in 2024-02-05T08:10:12.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:11 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:10:12.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:11 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:10:12.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:11 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:10:12.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:11 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:10:12.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:11 smithi196 ceph-mon[33018]: osdmap e147: 8 total, 8 up, 7 in 2024-02-05T08:10:12.094 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:12.118 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:12.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:12.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:12.220 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:12.256 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:12.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:12.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:12.383 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:12.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:12.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:12.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:12.511 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:12.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:12.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:12.613 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:12.649 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:12.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:12.751 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:12.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:12.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:12.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:12.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:12.916 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:12.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:12.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:13.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:13.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:12 smithi134 ceph-mon[39278]: pgmap v213: 148 pgs: 148 active+clean; 861 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 990 op/s 2024-02-05T08:10:13.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:12 smithi134 ceph-mon[30086]: pgmap v213: 148 pgs: 148 active+clean; 861 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 990 op/s 2024-02-05T08:10:13.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:13.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:13.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:12 smithi196 ceph-mon[33018]: pgmap v213: 148 pgs: 148 active+clean; 861 MiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 990 op/s 2024-02-05T08:10:13.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:13.184 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:13.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:13.286 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:13.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:13.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:13.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:13.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:13.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:13.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:13.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:13.541 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:13.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:10:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:10:13.582 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:13.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:13.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:13.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:13.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:13.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:13.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:13.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:13.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:13.882 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:13 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:13.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:13.999 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:13.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:14.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:14.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:13 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:14.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:13 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:14.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:14.131 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:14.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:14.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:14.233 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:14.296 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:14.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:14.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:14.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:14.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:14.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:14.438 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:10:14.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:14.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:14.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:14.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:14.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:14.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:14.697 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:14.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:14.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:14 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:10:14.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:14 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:10:14.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:14 smithi134 ceph-mon[30086]: pgmap v214: 148 pgs: 148 active+clean; 550 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 994 op/s 2024-02-05T08:10:14.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:14 smithi134 ceph-mon[30086]: osdmap e148: 8 total, 8 up, 7 in 2024-02-05T08:10:14.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:14 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:10:14.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:14 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:10:14.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:14 smithi134 ceph-mon[39278]: pgmap v214: 148 pgs: 148 active+clean; 550 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 994 op/s 2024-02-05T08:10:14.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:14 smithi134 ceph-mon[39278]: osdmap e148: 8 total, 8 up, 7 in 2024-02-05T08:10:14.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:14.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:14.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:14.928 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:14.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:14 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:10:14.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:14 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:10:14.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:14 smithi196 ceph-mon[33018]: pgmap v214: 148 pgs: 148 active+clean; 550 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 994 op/s 2024-02-05T08:10:14.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:14 smithi196 ceph-mon[33018]: osdmap e148: 8 total, 8 up, 7 in 2024-02-05T08:10:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:14.985 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:14.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:15.086 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:15.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:15.142 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:15.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:15.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:15.295 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:15.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:15.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:15.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:15.437 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:15.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:15.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:15.565 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:15.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:15.667 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:15.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:15.709 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:15.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:15.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:15.862 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:15.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:15.963 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:15.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:15 smithi134 ceph-mon[30086]: 5.15 scrub starts 2024-02-05T08:10:15.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:15 smithi134 ceph-mon[30086]: 5.15 scrub ok 2024-02-05T08:10:15.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:15 smithi134 ceph-mon[30086]: 6.2 scrub starts 2024-02-05T08:10:15.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:15 smithi134 ceph-mon[30086]: 6.2 scrub ok 2024-02-05T08:10:15.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:15 smithi134 ceph-mon[39278]: 5.15 scrub starts 2024-02-05T08:10:15.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:15 smithi134 ceph-mon[39278]: 5.15 scrub ok 2024-02-05T08:10:15.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:15 smithi134 ceph-mon[39278]: 6.2 scrub starts 2024-02-05T08:10:15.973 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:15 smithi134 ceph-mon[39278]: 6.2 scrub ok 2024-02-05T08:10:15.983 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:15.988 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:15.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:16.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:15 smithi196 ceph-mon[33018]: 5.15 scrub starts 2024-02-05T08:10:16.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:15 smithi196 ceph-mon[33018]: 5.15 scrub ok 2024-02-05T08:10:16.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:15 smithi196 ceph-mon[33018]: 6.2 scrub starts 2024-02-05T08:10:16.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:15 smithi196 ceph-mon[33018]: 6.2 scrub ok 2024-02-05T08:10:16.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:16.141 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:16.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:16.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:16.245 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:16.291 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:16.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:16.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:16.393 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:16.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:16.446 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:16.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:16.473 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:10:16.481 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:10:16.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:16.576 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:16.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:16.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:16.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:16.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:16.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:16.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:16 smithi134 ceph-mon[30086]: pgmap v216: 148 pgs: 148 active+clean; 550 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-05T08:10:16.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:16 smithi134 ceph-mon[30086]: from='client.24766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:10:16.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:16 smithi134 ceph-mon[30086]: 4.1 scrub starts 2024-02-05T08:10:16.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:16 smithi134 ceph-mon[30086]: 4.1 scrub ok 2024-02-05T08:10:16.829 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:16 smithi134 ceph-mon[39278]: pgmap v216: 148 pgs: 148 active+clean; 550 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-05T08:10:16.829 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:16 smithi134 ceph-mon[39278]: from='client.24766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:10:16.829 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:16 smithi134 ceph-mon[39278]: 4.1 scrub starts 2024-02-05T08:10:16.829 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:16 smithi134 ceph-mon[39278]: 4.1 scrub ok 2024-02-05T08:10:16.847 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:16.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:16.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:16.949 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:16.986 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:16.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:17.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:16 smithi196 ceph-mon[33018]: pgmap v216: 148 pgs: 148 active+clean; 550 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-05T08:10:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:16 smithi196 ceph-mon[33018]: from='client.24766 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:10:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:16 smithi196 ceph-mon[33018]: 4.1 scrub starts 2024-02-05T08:10:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:16 smithi196 ceph-mon[33018]: 4.1 scrub ok 2024-02-05T08:10:17.087 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:17.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:17.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:17.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:17.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:17.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:17.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:17.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:17.384 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-02-05T08:10:15.596672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":577050996,"num_objects":8995,"num_object_clones":0,"num_object_copies":26985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8995,"num_whiteouts":0,"num_read":1257,"num_read_kb":1103,"num_write":211682,"num_write_kb":7038046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":2363363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76156,"ondisk_log_size":76156,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2720088,"kb_used_data":1806104,"kb_used_omap":10,"kb_used_meta":913653,"kb_avail":653495976,"statfs":{"total":671965249536,"available":669179879424,"internally_reserved":0,"allocated":1849450496,"data_stored":1847287824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11155,"internal_metadata":935580781},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-591134720,"num_objects":-9020,"num_object_clones":0,"num_object_copies":-27060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9020,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9020,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002549"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":176,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.693882+0000","last_change":"2024-02-05T08:07:44.926305+0000","last_active":"2024-02-05T08:10:10.693882+0000","last_peered":"2024-02-05T08:10:10.693882+0000","last_clean":"2024-02-05T08:10:10.693882+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:10:10.693882+0000","last_undegraded":"2024-02-05T08:10:10.693882+0000","last_fullsized":"2024-02-05T08:10:10.693882+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:44.926230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:44.926230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536037+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":189,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695008+0000","last_change":"2024-02-05T08:08:50.003502+0000","last_active":"2024-02-05T08:10:10.695008+0000","last_peered":"2024-02-05T08:10:10.695008+0000","last_clean":"2024-02-05T08:10:10.695008+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:10:10.695008+0000","last_undegraded":"2024-02-05T08:10:10.695008+0000","last_fullsized":"2024-02-05T08:10:10.695008+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:50.003400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:50.003400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.438043+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":188,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.717402+0000","last_change":"2024-02-05T08:07:20.353726+0000","last_active":"2024-02-05T08:10:10.717402+0000","last_peered":"2024-02-05T08:10:10.717402+0000","last_clean":"2024-02-05T08:10:10.717402+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:10:10.717402+0000","last_undegraded":"2024-02-05T08:10:10.717402+0000","last_fullsized":"2024-02-05T08:10:10.717402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:20.353423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:20.353423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"90'13","reported_seq":220,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.693619+0000","last_change":"2024-02-05T08:08:31.425402+0000","last_active":"2024-02-05T08:10:10.693619+0000","last_peered":"2024-02-05T08:10:10.693619+0000","last_clean":"2024-02-05T08:10:10.693619+0000","last_became_active":"2024-02-05T08:06:39.560434+0000","last_became_peered":"2024-02-05T08:06:39.560434+0000","last_unstale":"2024-02-05T08:10:10.693619+0000","last_undegraded":"2024-02-05T08:10:10.693619+0000","last_fullsized":"2024-02-05T08:10:10.693619+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:08:31.425318+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:08:31.425318+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536088+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":190,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.705875+0000","last_change":"2024-02-05T08:09:37.106387+0000","last_active":"2024-02-05T08:10:10.705875+0000","last_peered":"2024-02-05T08:10:10.705875+0000","last_clean":"2024-02-05T08:10:10.705875+0000","last_became_active":"2024-02-05T08:06:39.558790+0000","last_became_peered":"2024-02-05T08:06:39.558790+0000","last_unstale":"2024-02-05T08:10:10.705875+0000","last_undegraded":"2024-02-05T08:10:10.705875+0000","last_fullsized":"2024-02-05T08:10:10.705875+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:37.106147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:37.106147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:05.992123+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":186,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.693797+0000","last_change":"2024-02-05T08:08:47.432025+0000","last_active":"2024-02-05T08:10:10.693797+0000","last_peered":"2024-02-05T08:10:10.693797+0000","last_clean":"2024-02-05T08:10:10.693797+0000","last_became_active":"2024-02-05T08:06:39.561731+0000","last_became_peered":"2024-02-05T08:06:39.561731+0000","last_unstale":"2024-02-05T08:10:10.693797+0000","last_undegraded":"2024-02-05T08:10:10.693797+0000","last_fullsized":"2024-02-05T08:10:10.693797+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:47.431940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:47.431940+0000","last_clean_scrub_stamp":"2024-02-05T08:08:47.431940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536187+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":194,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.706821+0000","last_change":"2024-02-05T08:07:27.703599+0000","last_active":"2024-02-05T08:10:10.706821+0000","last_peered":"2024-02-05T08:10:10.706821+0000","last_clean":"2024-02-05T08:10:10.706821+0000","last_became_active":"2024-02-05T08:06:39.558101+0000","last_became_peered":"2024-02-05T08:06:39.558101+0000","last_unstale":"2024-02-05T08:10:10.706821+0000","last_undegraded":"2024-02-05T08:10:10.706821+0000","last_fullsized":"2024-02-05T08:10:10.706821+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:27.703488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:27.703488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962728+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"118'22","reported_seq":216,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.713289+0000","last_change":"2024-02-05T08:07:45.819143+0000","last_active":"2024-02-05T08:10:10.713289+0000","last_peered":"2024-02-05T08:10:10.713289+0000","last_clean":"2024-02-05T08:10:10.713289+0000","last_became_active":"2024-02-05T08:04:43.560660+0000","last_became_peered":"2024-02-05T08:04:43.560660+0000","last_unstale":"2024-02-05T08:10:10.713289+0000","last_undegraded":"2024-02-05T08:10:10.713289+0000","last_fullsized":"2024-02-05T08:10:10.713289+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:07:45.819004+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962876+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":176,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.697213+0000","last_change":"2024-02-05T08:07:38.563648+0000","last_active":"2024-02-05T08:10:10.697213+0000","last_peered":"2024-02-05T08:10:10.697213+0000","last_clean":"2024-02-05T08:10:10.697213+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:10:10.697213+0000","last_undegraded":"2024-02-05T08:10:10.697213+0000","last_fullsized":"2024-02-05T08:10:10.697213+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:38.563533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:38.563533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":180,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699113+0000","last_change":"2024-02-05T08:08:35.065367+0000","last_active":"2024-02-05T08:10:10.699113+0000","last_peered":"2024-02-05T08:10:10.699113+0000","last_clean":"2024-02-05T08:10:10.699113+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:10:10.699113+0000","last_undegraded":"2024-02-05T08:10:10.699113+0000","last_fullsized":"2024-02-05T08:10:10.699113+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:35.065258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:37.352358+0000","last_clean_scrub_stamp":"2024-02-05T08:08:35.065258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494794+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":202,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699374+0000","last_change":"2024-02-05T08:09:39.410126+0000","last_active":"2024-02-05T08:10:10.699374+0000","last_peered":"2024-02-05T08:10:10.699374+0000","last_clean":"2024-02-05T08:10:10.699374+0000","last_became_active":"2024-02-05T08:06:39.559461+0000","last_became_peered":"2024-02-05T08:06:39.559461+0000","last_unstale":"2024-02-05T08:10:10.699374+0000","last_undegraded":"2024-02-05T08:10:10.699374+0000","last_fullsized":"2024-02-05T08:10:10.699374+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:39.410054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.410054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.039707+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":216,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694021+0000","last_change":"2024-02-05T08:08:38.437994+0000","last_active":"2024-02-05T08:10:10.694021+0000","last_peered":"2024-02-05T08:10:10.694021+0000","last_clean":"2024-02-05T08:10:10.694021+0000","last_became_active":"2024-02-05T08:07:22.653892+0000","last_became_peered":"2024-02-05T08:07:22.653892+0000","last_unstale":"2024-02-05T08:10:10.694021+0000","last_undegraded":"2024-02-05T08:10:10.694021+0000","last_fullsized":"2024-02-05T08:10:10.694021+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:08:38.437850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:38.437850+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536223+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":176,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.698683+0000","last_change":"2024-02-05T08:07:47.742914+0000","last_active":"2024-02-05T08:10:10.698683+0000","last_peered":"2024-02-05T08:10:10.698683+0000","last_clean":"2024-02-05T08:10:10.698683+0000","last_became_active":"2024-02-05T08:04:47.587515+0000","last_became_peered":"2024-02-05T08:04:47.587515+0000","last_unstale":"2024-02-05T08:10:10.698683+0000","last_undegraded":"2024-02-05T08:10:10.698683+0000","last_fullsized":"2024-02-05T08:10:10.698683+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:47.742836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:47.742836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494675+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":181,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704257+0000","last_change":"2024-02-05T08:07:36.546190+0000","last_active":"2024-02-05T08:10:10.704257+0000","last_peered":"2024-02-05T08:10:10.704257+0000","last_clean":"2024-02-05T08:10:10.704257+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:10:10.704257+0000","last_undegraded":"2024-02-05T08:10:10.704257+0000","last_fullsized":"2024-02-05T08:10:10.704257+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:36.546090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:36.546090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.194521+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":188,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699072+0000","last_change":"2024-02-05T08:07:39.835029+0000","last_active":"2024-02-05T08:10:10.699072+0000","last_peered":"2024-02-05T08:10:10.699072+0000","last_clean":"2024-02-05T08:10:10.699072+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:10:10.699072+0000","last_undegraded":"2024-02-05T08:10:10.699072+0000","last_fullsized":"2024-02-05T08:10:10.699072+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:39.834957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:07:39.834957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":194,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694262+0000","last_change":"2024-02-05T08:09:30.448604+0000","last_active":"2024-02-05T08:10:10.694262+0000","last_peered":"2024-02-05T08:10:10.694262+0000","last_clean":"2024-02-05T08:10:10.694262+0000","last_became_active":"2024-02-05T08:04:43.557559+0000","last_became_peered":"2024-02-05T08:04:43.557559+0000","last_unstale":"2024-02-05T08:10:10.694262+0000","last_undegraded":"2024-02-05T08:10:10.694262+0000","last_fullsized":"2024-02-05T08:10:10.694262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:09:30.448493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:30.448493+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:40.766789+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":190,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695094+0000","last_change":"2024-02-05T08:09:26.466381+0000","last_active":"2024-02-05T08:10:10.695094+0000","last_peered":"2024-02-05T08:10:10.695094+0000","last_clean":"2024-02-05T08:10:10.695094+0000","last_became_active":"2024-02-05T08:06:39.547906+0000","last_became_peered":"2024-02-05T08:06:39.547906+0000","last_unstale":"2024-02-05T08:10:10.695094+0000","last_undegraded":"2024-02-05T08:10:10.695094+0000","last_fullsized":"2024-02-05T08:10:10.695094+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:26.466302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:09:26.466302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:54.690678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":181,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695788+0000","last_change":"2024-02-05T08:07:48.943001+0000","last_active":"2024-02-05T08:10:10.695788+0000","last_peered":"2024-02-05T08:10:10.695788+0000","last_clean":"2024-02-05T08:10:10.695788+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:10:10.695788+0000","last_undegraded":"2024-02-05T08:10:10.695788+0000","last_fullsized":"2024-02-05T08:10:10.695788+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:07:48.942873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.437957+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":188,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.698270+0000","last_change":"2024-02-05T08:08:28.601390+0000","last_active":"2024-02-05T08:10:10.698270+0000","last_peered":"2024-02-05T08:10:10.698270+0000","last_clean":"2024-02-05T08:10:10.698270+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:10:10.698270+0000","last_undegraded":"2024-02-05T08:10:10.698270+0000","last_fullsized":"2024-02-05T08:10:10.698270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:28.601275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:28.601275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":218,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.714401+0000","last_change":"2024-02-05T08:08:40.732291+0000","last_active":"2024-02-05T08:10:10.714401+0000","last_peered":"2024-02-05T08:10:10.714401+0000","last_clean":"2024-02-05T08:10:10.714401+0000","last_became_active":"2024-02-05T08:04:43.557496+0000","last_became_peered":"2024-02-05T08:04:43.557496+0000","last_unstale":"2024-02-05T08:10:10.714401+0000","last_undegraded":"2024-02-05T08:10:10.714401+0000","last_fullsized":"2024-02-05T08:10:10.714401+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:08:40.729049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:40.729049+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.489559+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":182,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.706853+0000","last_change":"2024-02-05T08:09:21.354430+0000","last_active":"2024-02-05T08:10:10.706853+0000","last_peered":"2024-02-05T08:10:10.706853+0000","last_clean":"2024-02-05T08:10:10.706853+0000","last_became_active":"2024-02-05T08:06:39.560948+0000","last_became_peered":"2024-02-05T08:06:39.560948+0000","last_unstale":"2024-02-05T08:10:10.706853+0000","last_undegraded":"2024-02-05T08:10:10.706853+0000","last_fullsized":"2024-02-05T08:10:10.706853+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:21.354158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.354158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:26.901104+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":180,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694349+0000","last_change":"2024-02-05T08:07:55.895894+0000","last_active":"2024-02-05T08:10:10.694349+0000","last_peered":"2024-02-05T08:10:10.694349+0000","last_clean":"2024-02-05T08:10:10.694349+0000","last_became_active":"2024-02-05T08:04:45.563948+0000","last_became_peered":"2024-02-05T08:04:45.563948+0000","last_unstale":"2024-02-05T08:10:10.694349+0000","last_undegraded":"2024-02-05T08:10:10.694349+0000","last_fullsized":"2024-02-05T08:10:10.694349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:55.895795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:07:55.895795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536053+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":188,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694048+0000","last_change":"2024-02-05T08:09:27.431319+0000","last_active":"2024-02-05T08:10:10.694048+0000","last_peered":"2024-02-05T08:10:10.694048+0000","last_clean":"2024-02-05T08:10:10.694048+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:10:10.694048+0000","last_undegraded":"2024-02-05T08:10:10.694048+0000","last_fullsized":"2024-02-05T08:10:10.694048+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:27.431221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:27.431221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:44.170149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"90'17","reported_seq":212,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.707434+0000","last_change":"2024-02-05T08:07:24.701607+0000","last_active":"2024-02-05T08:10:10.707434+0000","last_peered":"2024-02-05T08:10:10.707434+0000","last_clean":"2024-02-05T08:10:10.707434+0000","last_became_active":"2024-02-05T08:04:43.557596+0000","last_became_peered":"2024-02-05T08:04:43.557596+0000","last_unstale":"2024-02-05T08:10:10.707434+0000","last_undegraded":"2024-02-05T08:10:10.707434+0000","last_fullsized":"2024-02-05T08:10:10.707434+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:07:24.701465+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:07:24.701465+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962758+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":187,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704775+0000","last_change":"2024-02-05T08:08:48.447437+0000","last_active":"2024-02-05T08:10:10.704775+0000","last_peered":"2024-02-05T08:10:10.704775+0000","last_clean":"2024-02-05T08:10:10.704775+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:10:10.704775+0000","last_undegraded":"2024-02-05T08:10:10.704775+0000","last_fullsized":"2024-02-05T08:10:10.704775+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:08:48.447317+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.965137+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":197,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695443+0000","last_change":"2024-02-05T08:08:44.441278+0000","last_active":"2024-02-05T08:10:10.695443+0000","last_peered":"2024-02-05T08:10:10.695443+0000","last_clean":"2024-02-05T08:10:10.695443+0000","last_became_active":"2024-02-05T08:06:39.552760+0000","last_became_peered":"2024-02-05T08:06:39.552760+0000","last_unstale":"2024-02-05T08:10:10.695443+0000","last_undegraded":"2024-02-05T08:10:10.695443+0000","last_fullsized":"2024-02-05T08:10:10.695443+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:08:44.441144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.441144+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536205+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":190,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694918+0000","last_change":"2024-02-05T08:07:35.997681+0000","last_active":"2024-02-05T08:10:10.694918+0000","last_peered":"2024-02-05T08:10:10.694918+0000","last_clean":"2024-02-05T08:10:10.694918+0000","last_became_active":"2024-02-05T08:04:41.532339+0000","last_became_peered":"2024-02-05T08:04:41.532339+0000","last_unstale":"2024-02-05T08:10:10.694918+0000","last_undegraded":"2024-02-05T08:10:10.694918+0000","last_fullsized":"2024-02-05T08:10:10.694918+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:07:35.997535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:35.997535+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.435441+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":199,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.707531+0000","last_change":"2024-02-05T08:09:24.000902+0000","last_active":"2024-02-05T08:10:10.707531+0000","last_peered":"2024-02-05T08:10:10.707531+0000","last_clean":"2024-02-05T08:10:10.707531+0000","last_became_active":"2024-02-05T08:04:43.605952+0000","last_became_peered":"2024-02-05T08:04:43.605952+0000","last_unstale":"2024-02-05T08:10:10.707531+0000","last_undegraded":"2024-02-05T08:10:10.707531+0000","last_fullsized":"2024-02-05T08:10:10.707531+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:09:24.000675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:24.000675+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:53.229695+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":185,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695983+0000","last_change":"2024-02-05T08:09:25.266619+0000","last_active":"2024-02-05T08:10:10.695983+0000","last_peered":"2024-02-05T08:10:10.695983+0000","last_clean":"2024-02-05T08:10:10.695983+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:10:10.695983+0000","last_undegraded":"2024-02-05T08:10:10.695983+0000","last_fullsized":"2024-02-05T08:10:10.695983+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.266508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.266508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:35.256058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":180,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.707876+0000","last_change":"2024-02-05T08:08:42.676027+0000","last_active":"2024-02-05T08:10:10.707876+0000","last_peered":"2024-02-05T08:10:10.707876+0000","last_clean":"2024-02-05T08:10:10.707876+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:10:10.707876+0000","last_undegraded":"2024-02-05T08:10:10.707876+0000","last_fullsized":"2024-02-05T08:10:10.707876+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:42.675782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:42.675782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":188,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.708696+0000","last_change":"2024-02-05T08:07:50.777386+0000","last_active":"2024-02-05T08:10:10.708696+0000","last_peered":"2024-02-05T08:10:10.708696+0000","last_clean":"2024-02-05T08:10:10.708696+0000","last_became_active":"2024-02-05T08:04:41.534448+0000","last_became_peered":"2024-02-05T08:04:41.534448+0000","last_unstale":"2024-02-05T08:10:10.708696+0000","last_undegraded":"2024-02-05T08:10:10.708696+0000","last_fullsized":"2024-02-05T08:10:10.708696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:50.777273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:07:50.777273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962902+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":177,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699531+0000","last_change":"2024-02-05T08:09:25.380533+0000","last_active":"2024-02-05T08:10:10.699531+0000","last_peered":"2024-02-05T08:10:10.699531+0000","last_clean":"2024-02-05T08:10:10.699531+0000","last_became_active":"2024-02-05T08:06:39.558747+0000","last_became_peered":"2024-02-05T08:06:39.558747+0000","last_unstale":"2024-02-05T08:10:10.699531+0000","last_undegraded":"2024-02-05T08:10:10.699531+0000","last_fullsized":"2024-02-05T08:10:10.699531+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:09:25.380390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.380390+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:41.147157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":177,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704131+0000","last_change":"2024-02-05T08:08:31.483194+0000","last_active":"2024-02-05T08:10:10.704131+0000","last_peered":"2024-02-05T08:10:10.704131+0000","last_clean":"2024-02-05T08:10:10.704131+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:10:10.704131+0000","last_undegraded":"2024-02-05T08:10:10.704131+0000","last_fullsized":"2024-02-05T08:10:10.704131+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:31.483087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:31.483087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.194572+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":127,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694745+0000","last_change":"2024-02-05T08:09:36.434985+0000","last_active":"2024-02-05T08:10:10.694745+0000","last_peered":"2024-02-05T08:10:10.694745+0000","last_clean":"2024-02-05T08:10:10.694745+0000","last_became_active":"2024-02-05T08:06:39.551173+0000","last_became_peered":"2024-02-05T08:06:39.551173+0000","last_unstale":"2024-02-05T08:10:10.694745+0000","last_undegraded":"2024-02-05T08:10:10.694745+0000","last_fullsized":"2024-02-05T08:10:10.694745+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:36.434873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.434873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:05.817457+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":188,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.717573+0000","last_change":"2024-02-05T08:08:30.632883+0000","last_active":"2024-02-05T08:10:10.717573+0000","last_peered":"2024-02-05T08:10:10.717573+0000","last_clean":"2024-02-05T08:10:10.717573+0000","last_became_active":"2024-02-05T08:04:41.536019+0000","last_became_peered":"2024-02-05T08:04:41.536019+0000","last_unstale":"2024-02-05T08:10:10.717573+0000","last_undegraded":"2024-02-05T08:10:10.717573+0000","last_fullsized":"2024-02-05T08:10:10.717573+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:30.632785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:30.632785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":202,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.708786+0000","last_change":"2024-02-05T08:08:40.082709+0000","last_active":"2024-02-05T08:10:10.708786+0000","last_peered":"2024-02-05T08:10:10.708786+0000","last_clean":"2024-02-05T08:10:10.708786+0000","last_became_active":"2024-02-05T08:06:39.560932+0000","last_became_peered":"2024-02-05T08:06:39.560932+0000","last_unstale":"2024-02-05T08:10:10.708786+0000","last_undegraded":"2024-02-05T08:10:10.708786+0000","last_fullsized":"2024-02-05T08:10:10.708786+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-02-05T08:08:40.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:40.082577+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.963014+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":182,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.706140+0000","last_change":"2024-02-05T08:07:28.666539+0000","last_active":"2024-02-05T08:10:10.706140+0000","last_peered":"2024-02-05T08:10:10.706140+0000","last_clean":"2024-02-05T08:10:10.706140+0000","last_became_active":"2024-02-05T08:06:39.559212+0000","last_became_peered":"2024-02-05T08:06:39.559212+0000","last_unstale":"2024-02-05T08:10:10.706140+0000","last_undegraded":"2024-02-05T08:10:10.706140+0000","last_fullsized":"2024-02-05T08:10:10.706140+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:28.666433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:28.666433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962673+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":195,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704272+0000","last_change":"2024-02-05T08:09:36.353456+0000","last_active":"2024-02-05T08:10:10.704272+0000","last_peered":"2024-02-05T08:10:10.704272+0000","last_clean":"2024-02-05T08:10:10.704272+0000","last_became_active":"2024-02-05T08:06:39.555597+0000","last_became_peered":"2024-02-05T08:06:39.555597+0000","last_unstale":"2024-02-05T08:10:10.704272+0000","last_undegraded":"2024-02-05T08:10:10.704272+0000","last_fullsized":"2024-02-05T08:10:10.704272+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.353351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:38.202761+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":196,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699523+0000","last_change":"2024-02-05T08:08:45.166139+0000","last_active":"2024-02-05T08:10:10.699523+0000","last_peered":"2024-02-05T08:10:10.699523+0000","last_clean":"2024-02-05T08:10:10.699523+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:10:10.699523+0000","last_undegraded":"2024-02-05T08:10:10.699523+0000","last_fullsized":"2024-02-05T08:10:10.699523+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:45.166006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:45.166006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"141'16","reported_seq":234,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.713198+0000","last_change":"2024-02-05T08:09:39.113079+0000","last_active":"2024-02-05T08:10:10.713198+0000","last_peered":"2024-02-05T08:10:10.713198+0000","last_clean":"2024-02-05T08:10:10.713198+0000","last_became_active":"2024-02-05T08:06:39.556873+0000","last_became_peered":"2024-02-05T08:06:39.556873+0000","last_unstale":"2024-02-05T08:10:10.713198+0000","last_undegraded":"2024-02-05T08:10:10.713198+0000","last_fullsized":"2024-02-05T08:10:10.713198+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'16","last_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.112936+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:45.211059+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":176,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.696913+0000","last_change":"2024-02-05T08:07:45.514975+0000","last_active":"2024-02-05T08:10:10.696913+0000","last_peered":"2024-02-05T08:10:10.696913+0000","last_clean":"2024-02-05T08:10:10.696913+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:10:10.696913+0000","last_undegraded":"2024-02-05T08:10:10.696913+0000","last_fullsized":"2024-02-05T08:10:10.696913+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:45.514863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:45.514863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":180,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.705788+0000","last_change":"2024-02-05T08:07:25.644883+0000","last_active":"2024-02-05T08:10:10.705788+0000","last_peered":"2024-02-05T08:10:10.705788+0000","last_clean":"2024-02-05T08:10:10.705788+0000","last_became_active":"2024-02-05T08:04:45.568484+0000","last_became_peered":"2024-02-05T08:04:45.568484+0000","last_unstale":"2024-02-05T08:10:10.705788+0000","last_undegraded":"2024-02-05T08:10:10.705788+0000","last_fullsized":"2024-02-05T08:10:10.705788+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:25.644770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:25.644770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962700+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":189,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704782+0000","last_change":"2024-02-05T08:08:32.528082+0000","last_active":"2024-02-05T08:10:10.704782+0000","last_peered":"2024-02-05T08:10:10.704782+0000","last_clean":"2024-02-05T08:10:10.704782+0000","last_became_active":"2024-02-05T08:04:41.539698+0000","last_became_peered":"2024-02-05T08:04:41.539698+0000","last_unstale":"2024-02-05T08:10:10.704782+0000","last_undegraded":"2024-02-05T08:10:10.704782+0000","last_fullsized":"2024-02-05T08:10:10.704782+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:32.527978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:32.527978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.194556+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":225,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695584+0000","last_change":"2024-02-05T08:08:43.941119+0000","last_active":"2024-02-05T08:10:10.695584+0000","last_peered":"2024-02-05T08:10:10.695584+0000","last_clean":"2024-02-05T08:10:10.695584+0000","last_became_active":"2024-02-05T08:06:39.560259+0000","last_became_peered":"2024-02-05T08:06:39.560259+0000","last_unstale":"2024-02-05T08:10:10.695584+0000","last_undegraded":"2024-02-05T08:10:10.695584+0000","last_fullsized":"2024-02-05T08:10:10.695584+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:08:43.940980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:43.940980+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.438017+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":176,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.712996+0000","last_change":"2024-02-05T08:09:29.262843+0000","last_active":"2024-02-05T08:10:10.712996+0000","last_peered":"2024-02-05T08:10:10.712996+0000","last_clean":"2024-02-05T08:10:10.712996+0000","last_became_active":"2024-02-05T08:04:47.585496+0000","last_became_peered":"2024-02-05T08:04:47.585496+0000","last_unstale":"2024-02-05T08:10:10.712996+0000","last_undegraded":"2024-02-05T08:10:10.712996+0000","last_fullsized":"2024-02-05T08:10:10.712996+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:09:29.262582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:47.044333+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":181,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695342+0000","last_change":"2024-02-05T08:07:32.955213+0000","last_active":"2024-02-05T08:10:10.695342+0000","last_peered":"2024-02-05T08:10:10.695342+0000","last_clean":"2024-02-05T08:10:10.695342+0000","last_became_active":"2024-02-05T08:04:45.560815+0000","last_became_peered":"2024-02-05T08:04:45.560815+0000","last_unstale":"2024-02-05T08:10:10.695342+0000","last_undegraded":"2024-02-05T08:10:10.695342+0000","last_fullsized":"2024-02-05T08:10:10.695342+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:32.955088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:32.955088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.435410+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":202,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694124+0000","last_change":"2024-02-05T08:09:25.486212+0000","last_active":"2024-02-05T08:10:10.694124+0000","last_peered":"2024-02-05T08:10:10.694124+0000","last_clean":"2024-02-05T08:10:10.694124+0000","last_became_active":"2024-02-05T08:06:39.545576+0000","last_became_peered":"2024-02-05T08:06:39.545576+0000","last_unstale":"2024-02-05T08:10:10.694124+0000","last_undegraded":"2024-02-05T08:10:10.694124+0000","last_fullsized":"2024-02-05T08:10:10.694124+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.486141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.966886+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.486141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:39.358720+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":221,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694386+0000","last_change":"2024-02-05T08:08:26.541304+0000","last_active":"2024-02-05T08:10:10.694386+0000","last_peered":"2024-02-05T08:10:10.694386+0000","last_clean":"2024-02-05T08:10:10.694386+0000","last_became_active":"2024-02-05T08:06:39.558257+0000","last_became_peered":"2024-02-05T08:06:39.558257+0000","last_unstale":"2024-02-05T08:10:10.694386+0000","last_undegraded":"2024-02-05T08:10:10.694386+0000","last_fullsized":"2024-02-05T08:10:10.694386+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:08:26.541177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:26.541177+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":176,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.707165+0000","last_change":"2024-02-05T08:08:36.679472+0000","last_active":"2024-02-05T08:10:10.707165+0000","last_peered":"2024-02-05T08:10:10.707165+0000","last_clean":"2024-02-05T08:10:10.707165+0000","last_became_active":"2024-02-05T08:04:47.584235+0000","last_became_peered":"2024-02-05T08:04:47.584235+0000","last_unstale":"2024-02-05T08:10:10.707165+0000","last_undegraded":"2024-02-05T08:10:10.707165+0000","last_fullsized":"2024-02-05T08:10:10.707165+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:36.679368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:36.679368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":255,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699477+0000","last_change":"2024-02-05T08:07:52.659017+0000","last_active":"2024-02-05T08:10:10.699477+0000","last_peered":"2024-02-05T08:10:10.699477+0000","last_clean":"2024-02-05T08:10:10.699477+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:10:10.699477+0000","last_undegraded":"2024-02-05T08:10:10.699477+0000","last_fullsized":"2024-02-05T08:10:10.699477+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:07:52.658883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:52.658883+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494648+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":197,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704500+0000","last_change":"2024-02-05T08:09:19.417311+0000","last_active":"2024-02-05T08:10:10.704500+0000","last_peered":"2024-02-05T08:10:10.704500+0000","last_clean":"2024-02-05T08:10:10.704500+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:10:10.704500+0000","last_undegraded":"2024-02-05T08:10:10.704500+0000","last_fullsized":"2024-02-05T08:10:10.704500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:19.417231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.417231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:25.958103+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"129'15","reported_seq":204,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.705683+0000","last_change":"2024-02-05T08:08:37.128945+0000","last_active":"2024-02-05T08:10:10.705683+0000","last_peered":"2024-02-05T08:10:10.705683+0000","last_clean":"2024-02-05T08:10:10.705683+0000","last_became_active":"2024-02-05T08:04:43.605981+0000","last_became_peered":"2024-02-05T08:04:43.605981+0000","last_unstale":"2024-02-05T08:10:10.705683+0000","last_undegraded":"2024-02-05T08:10:10.705683+0000","last_fullsized":"2024-02-05T08:10:10.705683+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'15","last_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:08:37.128817+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962985+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":176,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695156+0000","last_change":"2024-02-05T08:09:18.397753+0000","last_active":"2024-02-05T08:10:10.695156+0000","last_peered":"2024-02-05T08:10:10.695156+0000","last_clean":"2024-02-05T08:10:10.695156+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:10:10.695156+0000","last_undegraded":"2024-02-05T08:10:10.695156+0000","last_fullsized":"2024-02-05T08:10:10.695156+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:18.397684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:18.397684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:42.614612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":255,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.705978+0000","last_change":"2024-02-05T08:07:29.660531+0000","last_active":"2024-02-05T08:10:10.705978+0000","last_peered":"2024-02-05T08:10:10.705978+0000","last_clean":"2024-02-05T08:10:10.705978+0000","last_became_active":"2024-02-05T08:04:45.565240+0000","last_became_peered":"2024-02-05T08:04:45.565240+0000","last_unstale":"2024-02-05T08:10:10.705978+0000","last_undegraded":"2024-02-05T08:10:10.705978+0000","last_fullsized":"2024-02-05T08:10:10.705978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:07:29.660395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:29.660395+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962818+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":188,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.697864+0000","last_change":"2024-02-05T08:08:44.680464+0000","last_active":"2024-02-05T08:10:10.697864+0000","last_peered":"2024-02-05T08:10:10.697864+0000","last_clean":"2024-02-05T08:10:10.697864+0000","last_became_active":"2024-02-05T08:04:41.540252+0000","last_became_peered":"2024-02-05T08:04:41.540252+0000","last_unstale":"2024-02-05T08:10:10.697864+0000","last_undegraded":"2024-02-05T08:10:10.697864+0000","last_fullsized":"2024-02-05T08:10:10.697864+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:44.680353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.680353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":208,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699563+0000","last_change":"2024-02-05T08:09:36.437558+0000","last_active":"2024-02-05T08:10:10.699563+0000","last_peered":"2024-02-05T08:10:10.699563+0000","last_clean":"2024-02-05T08:10:10.699563+0000","last_became_active":"2024-02-05T08:04:43.606414+0000","last_became_peered":"2024-02-05T08:04:43.606414+0000","last_unstale":"2024-02-05T08:10:10.699563+0000","last_undegraded":"2024-02-05T08:10:10.699563+0000","last_fullsized":"2024-02-05T08:10:10.699563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:09:36.437435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.437435+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:44.544063+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":185,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.714709+0000","last_change":"2024-02-05T08:08:41.738958+0000","last_active":"2024-02-05T08:10:10.714709+0000","last_peered":"2024-02-05T08:10:10.714709+0000","last_clean":"2024-02-05T08:10:10.714709+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:10:10.714709+0000","last_undegraded":"2024-02-05T08:10:10.714709+0000","last_fullsized":"2024-02-05T08:10:10.714709+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:41.738836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:41.738836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.489584+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":180,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.706721+0000","last_change":"2024-02-05T08:07:16.750620+0000","last_active":"2024-02-05T08:10:10.706721+0000","last_peered":"2024-02-05T08:10:10.706721+0000","last_clean":"2024-02-05T08:10:10.706721+0000","last_became_active":"2024-02-05T08:04:45.567622+0000","last_became_peered":"2024-02-05T08:04:45.567622+0000","last_unstale":"2024-02-05T08:10:10.706721+0000","last_undegraded":"2024-02-05T08:10:10.706721+0000","last_fullsized":"2024-02-05T08:10:10.706721+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:16.750514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:16.750514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962616+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":195,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704241+0000","last_change":"2024-02-05T08:08:35.480440+0000","last_active":"2024-02-05T08:10:10.704241+0000","last_peered":"2024-02-05T08:10:10.704241+0000","last_clean":"2024-02-05T08:10:10.704241+0000","last_became_active":"2024-02-05T08:06:39.557018+0000","last_became_peered":"2024-02-05T08:06:39.557018+0000","last_unstale":"2024-02-05T08:10:10.704241+0000","last_undegraded":"2024-02-05T08:10:10.704241+0000","last_fullsized":"2024-02-05T08:10:10.704241+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:35.480367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:35.480367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.194635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":200,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.707723+0000","last_change":"2024-02-05T08:08:45.712868+0000","last_active":"2024-02-05T08:10:10.707723+0000","last_peered":"2024-02-05T08:10:10.707723+0000","last_clean":"2024-02-05T08:10:10.707723+0000","last_became_active":"2024-02-05T08:04:43.608489+0000","last_became_peered":"2024-02-05T08:04:43.608489+0000","last_unstale":"2024-02-05T08:10:10.707723+0000","last_undegraded":"2024-02-05T08:10:10.707723+0000","last_fullsized":"2024-02-05T08:10:10.707723+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:08:45.712396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:45.712396+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":191,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704694+0000","last_change":"2024-02-05T08:09:33.391323+0000","last_active":"2024-02-05T08:10:10.704694+0000","last_peered":"2024-02-05T08:10:10.704694+0000","last_clean":"2024-02-05T08:10:10.704694+0000","last_became_active":"2024-02-05T08:06:39.548837+0000","last_became_peered":"2024-02-05T08:06:39.548837+0000","last_unstale":"2024-02-05T08:10:10.704694+0000","last_undegraded":"2024-02-05T08:10:10.704694+0000","last_fullsized":"2024-02-05T08:10:10.704694+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:33.391219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:33.391219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:38.950366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":180,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699569+0000","last_change":"2024-02-05T08:07:44.778884+0000","last_active":"2024-02-05T08:10:10.699569+0000","last_peered":"2024-02-05T08:10:10.699569+0000","last_clean":"2024-02-05T08:10:10.699569+0000","last_became_active":"2024-02-05T08:04:45.560241+0000","last_became_peered":"2024-02-05T08:04:45.560241+0000","last_unstale":"2024-02-05T08:10:10.699569+0000","last_undegraded":"2024-02-05T08:10:10.699569+0000","last_fullsized":"2024-02-05T08:10:10.699569+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:44.778772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:44.778772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494778+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":119,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.697410+0000","last_change":"2024-02-05T08:07:36.550407+0000","last_active":"2024-02-05T08:10:10.697410+0000","last_peered":"2024-02-05T08:10:10.697410+0000","last_clean":"2024-02-05T08:10:10.697410+0000","last_became_active":"2024-02-05T08:06:39.554673+0000","last_became_peered":"2024-02-05T08:06:39.554673+0000","last_unstale":"2024-02-05T08:10:10.697410+0000","last_undegraded":"2024-02-05T08:10:10.697410+0000","last_fullsized":"2024-02-05T08:10:10.697410+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:36.550290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:36.550290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"116'9","reported_seq":196,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.706933+0000","last_change":"2024-02-05T08:07:42.729822+0000","last_active":"2024-02-05T08:10:10.706933+0000","last_peered":"2024-02-05T08:10:10.706933+0000","last_clean":"2024-02-05T08:10:10.706933+0000","last_became_active":"2024-02-05T08:04:43.606624+0000","last_became_peered":"2024-02-05T08:04:43.606624+0000","last_unstale":"2024-02-05T08:10:10.706933+0000","last_undegraded":"2024-02-05T08:10:10.706933+0000","last_fullsized":"2024-02-05T08:10:10.706933+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:07:42.729538+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962788+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"147'13047","reported_seq":13294,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:12.827456+0000","last_change":"2024-02-05T08:09:35.129348+0000","last_active":"2024-02-05T08:10:12.827456+0000","last_peered":"2024-02-05T08:10:12.827456+0000","last_clean":"2024-02-05T08:10:12.827456+0000","last_became_active":"2024-02-05T08:06:39.556800+0000","last_became_peered":"2024-02-05T08:06:39.556800+0000","last_unstale":"2024-02-05T08:10:12.827456+0000","last_undegraded":"2024-02-05T08:10:12.827456+0000","last_fullsized":"2024-02-05T08:10:12.827456+0000","mapping_epoch":97,"log_start":"133'8800","ondisk_log_start":"133'8800","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'11079","last_scrub_stamp":"2024-02-05T08:09:35.129197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:35.129197+0000","objects_scrubbed":2569,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:53.986660+0000","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38207488,"num_objects":583,"num_object_clones":0,"num_object_copies":1749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13047,"num_write_kb":436160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":190,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699289+0000","last_change":"2024-02-05T08:08:41.139144+0000","last_active":"2024-02-05T08:10:10.699289+0000","last_peered":"2024-02-05T08:10:10.699289+0000","last_clean":"2024-02-05T08:10:10.699289+0000","last_became_active":"2024-02-05T08:06:39.551759+0000","last_became_peered":"2024-02-05T08:06:39.551759+0000","last_unstale":"2024-02-05T08:10:10.699289+0000","last_undegraded":"2024-02-05T08:10:10.699289+0000","last_fullsized":"2024-02-05T08:10:10.699289+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:41.139067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:41.139067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494887+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":201,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.705694+0000","last_change":"2024-02-05T08:07:55.497839+0000","last_active":"2024-02-05T08:10:10.705694+0000","last_peered":"2024-02-05T08:10:10.705694+0000","last_clean":"2024-02-05T08:10:10.705694+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:10:10.705694+0000","last_undegraded":"2024-02-05T08:10:10.705694+0000","last_fullsized":"2024-02-05T08:10:10.705694+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:07:55.497748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:55.497748+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.194540+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":231,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.698729+0000","last_change":"2024-02-05T08:08:33.007515+0000","last_active":"2024-02-05T08:10:10.698729+0000","last_peered":"2024-02-05T08:10:10.698729+0000","last_clean":"2024-02-05T08:10:10.698729+0000","last_became_active":"2024-02-05T08:06:39.546607+0000","last_became_peered":"2024-02-05T08:06:39.546607+0000","last_unstale":"2024-02-05T08:10:10.698729+0000","last_undegraded":"2024-02-05T08:10:10.698729+0000","last_fullsized":"2024-02-05T08:10:10.698729+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:08:33.007404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:33.007404+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494812+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":274,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704402+0000","last_change":"2024-02-05T08:09:20.414916+0000","last_active":"2024-02-05T08:10:10.704402+0000","last_peered":"2024-02-05T08:10:10.704402+0000","last_clean":"2024-02-05T08:10:10.704402+0000","last_became_active":"2024-02-05T08:06:39.556562+0000","last_became_peered":"2024-02-05T08:06:39.556562+0000","last_unstale":"2024-02-05T08:10:10.704402+0000","last_undegraded":"2024-02-05T08:10:10.704402+0000","last_fullsized":"2024-02-05T08:10:10.704402+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:09:20.414837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:20.414837+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:36.384339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"147'12823","reported_seq":12996,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:12.827019+0000","last_change":"2024-02-05T08:08:47.609361+0000","last_active":"2024-02-05T08:10:12.827019+0000","last_peered":"2024-02-05T08:10:12.827019+0000","last_clean":"2024-02-05T08:10:12.827019+0000","last_became_active":"2024-02-05T08:06:37.621544+0000","last_became_peered":"2024-02-05T08:06:37.621544+0000","last_unstale":"2024-02-05T08:10:12.827019+0000","last_undegraded":"2024-02-05T08:10:12.827019+0000","last_fullsized":"2024-02-05T08:10:12.827019+0000","mapping_epoch":95,"log_start":"133'8600","ondisk_log_start":"133'8600","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8264","last_scrub_stamp":"2024-02-05T08:08:47.608801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:47.608801+0000","objects_scrubbed":5157,"log_size":4223,"log_dups_size":0,"ondisk_log_size":4223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.881697+0000","scrub_duration":2457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35323904,"num_objects":539,"num_object_clones":0,"num_object_copies":1617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12823,"num_write_kb":427584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":176,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.705965+0000","last_change":"2024-02-05T08:09:25.266619+0000","last_active":"2024-02-05T08:10:10.705965+0000","last_peered":"2024-02-05T08:10:10.705965+0000","last_clean":"2024-02-05T08:10:10.705965+0000","last_became_active":"2024-02-05T08:04:47.586044+0000","last_became_peered":"2024-02-05T08:04:47.586044+0000","last_unstale":"2024-02-05T08:10:10.705965+0000","last_undegraded":"2024-02-05T08:10:10.705965+0000","last_fullsized":"2024-02-05T08:10:10.705965+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.266513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.266513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:48.723648+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":197,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704899+0000","last_change":"2024-02-05T08:09:39.244280+0000","last_active":"2024-02-05T08:10:10.704899+0000","last_peered":"2024-02-05T08:10:10.704899+0000","last_clean":"2024-02-05T08:10:10.704899+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:10:10.704899+0000","last_undegraded":"2024-02-05T08:10:10.704899+0000","last_fullsized":"2024-02-05T08:10:10.704899+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:39.244179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.244179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:55.149887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":207,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694908+0000","last_change":"2024-02-05T08:09:31.469380+0000","last_active":"2024-02-05T08:10:10.694908+0000","last_peered":"2024-02-05T08:10:10.694908+0000","last_clean":"2024-02-05T08:10:10.694908+0000","last_became_active":"2024-02-05T08:04:43.556580+0000","last_became_peered":"2024-02-05T08:04:43.556580+0000","last_unstale":"2024-02-05T08:10:10.694908+0000","last_undegraded":"2024-02-05T08:10:10.694908+0000","last_fullsized":"2024-02-05T08:10:10.694908+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:09:31.469295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:31.469295+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.626653+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":189,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.713831+0000","last_change":"2024-02-05T08:09:23.533364+0000","last_active":"2024-02-05T08:10:10.713831+0000","last_peered":"2024-02-05T08:10:10.713831+0000","last_clean":"2024-02-05T08:10:10.713831+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:10:10.713831+0000","last_undegraded":"2024-02-05T08:10:10.713831+0000","last_fullsized":"2024-02-05T08:10:10.713831+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:23.533259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_clean_scrub_stamp":"2024-02-05T08:09:23.533259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:35.020206+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"147'13021","reported_seq":13189,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:13.038237+0000","last_change":"2024-02-05T08:09:19.227093+0000","last_active":"2024-02-05T08:10:13.038237+0000","last_peered":"2024-02-05T08:10:13.038237+0000","last_clean":"2024-02-05T08:10:13.038237+0000","last_became_active":"2024-02-05T08:06:37.619775+0000","last_became_peered":"2024-02-05T08:06:37.619775+0000","last_unstale":"2024-02-05T08:10:13.038237+0000","last_undegraded":"2024-02-05T08:10:13.038237+0000","last_fullsized":"2024-02-05T08:10:13.038237+0000","mapping_epoch":95,"log_start":"130'8200","ondisk_log_start":"130'8200","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'10065","last_scrub_stamp":"2024-02-05T08:09:19.226651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.226651+0000","objects_scrubbed":3510,"log_size":4821,"log_dups_size":0,"ondisk_log_size":4821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:24.312423+0000","scrub_duration":1408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33882112,"num_objects":517,"num_object_clones":0,"num_object_copies":1551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13021,"num_write_kb":433216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":177,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704028+0000","last_change":"2024-02-05T08:07:40.688555+0000","last_active":"2024-02-05T08:10:10.704028+0000","last_peered":"2024-02-05T08:10:10.704028+0000","last_clean":"2024-02-05T08:10:10.704028+0000","last_became_active":"2024-02-05T08:04:47.587082+0000","last_became_peered":"2024-02-05T08:04:47.587082+0000","last_unstale":"2024-02-05T08:10:10.704028+0000","last_undegraded":"2024-02-05T08:10:10.704028+0000","last_fullsized":"2024-02-05T08:10:10.704028+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:40.688451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:40.688451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.194502+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":189,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.714476+0000","last_change":"2024-02-05T08:09:29.579918+0000","last_active":"2024-02-05T08:10:10.714476+0000","last_peered":"2024-02-05T08:10:10.714476+0000","last_clean":"2024-02-05T08:10:10.714476+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:10:10.714476+0000","last_undegraded":"2024-02-05T08:10:10.714476+0000","last_fullsized":"2024-02-05T08:10:10.714476+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:29.578521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:29.578521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:44.673091+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":203,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704975+0000","last_change":"2024-02-05T08:09:31.480648+0000","last_active":"2024-02-05T08:10:10.704975+0000","last_peered":"2024-02-05T08:10:10.704975+0000","last_clean":"2024-02-05T08:10:10.704975+0000","last_became_active":"2024-02-05T08:06:39.554318+0000","last_became_peered":"2024-02-05T08:06:39.554318+0000","last_unstale":"2024-02-05T08:10:10.704975+0000","last_undegraded":"2024-02-05T08:10:10.704975+0000","last_fullsized":"2024-02-05T08:10:10.704975+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:31.480558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:31.480558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:42.719453+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":202,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.715557+0000","last_change":"2024-02-05T08:09:34.530885+0000","last_active":"2024-02-05T08:10:10.715557+0000","last_peered":"2024-02-05T08:10:10.715557+0000","last_clean":"2024-02-05T08:10:10.715557+0000","last_became_active":"2024-02-05T08:06:39.552819+0000","last_became_peered":"2024-02-05T08:06:39.552819+0000","last_unstale":"2024-02-05T08:10:10.715557+0000","last_undegraded":"2024-02-05T08:10:10.715557+0000","last_fullsized":"2024-02-05T08:10:10.715557+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:09:34.530767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:34.530767+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:35.663461+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"147'13253","reported_seq":13440,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:12.664674+0000","last_change":"2024-02-05T08:08:30.453801+0000","last_active":"2024-02-05T08:10:12.664674+0000","last_peered":"2024-02-05T08:10:12.664674+0000","last_clean":"2024-02-05T08:10:12.664674+0000","last_became_active":"2024-02-05T08:06:37.623334+0000","last_became_peered":"2024-02-05T08:06:37.623334+0000","last_unstale":"2024-02-05T08:10:12.664674+0000","last_undegraded":"2024-02-05T08:10:12.664674+0000","last_fullsized":"2024-02-05T08:10:12.664674+0000","mapping_epoch":95,"log_start":"128'7800","ondisk_log_start":"128'7800","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'7516","last_scrub_stamp":"2024-02-05T08:08:30.453634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:30.453634+0000","objects_scrubbed":6304,"log_size":5453,"log_dups_size":0,"ondisk_log_size":5453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.489527+0000","scrub_duration":1924,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33357824,"num_objects":509,"num_object_clones":0,"num_object_copies":1527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13253,"num_write_kb":440384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":176,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.703938+0000","last_change":"2024-02-05T08:07:55.608340+0000","last_active":"2024-02-05T08:10:10.703938+0000","last_peered":"2024-02-05T08:10:10.703938+0000","last_clean":"2024-02-05T08:10:10.703938+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:10:10.703938+0000","last_undegraded":"2024-02-05T08:10:10.703938+0000","last_fullsized":"2024-02-05T08:10:10.703938+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:55.608083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:55.608083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":189,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.715275+0000","last_change":"2024-02-05T08:09:21.583142+0000","last_active":"2024-02-05T08:10:10.715275+0000","last_peered":"2024-02-05T08:10:10.715275+0000","last_clean":"2024-02-05T08:10:10.715275+0000","last_became_active":"2024-02-05T08:04:45.567341+0000","last_became_peered":"2024-02-05T08:04:45.567341+0000","last_unstale":"2024-02-05T08:10:10.715275+0000","last_undegraded":"2024-02-05T08:10:10.715275+0000","last_fullsized":"2024-02-05T08:10:10.715275+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:21.582910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.582910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:23.887947+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":194,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699144+0000","last_change":"2024-02-05T08:07:45.721633+0000","last_active":"2024-02-05T08:10:10.699144+0000","last_peered":"2024-02-05T08:10:10.699144+0000","last_clean":"2024-02-05T08:10:10.699144+0000","last_became_active":"2024-02-05T08:06:39.558993+0000","last_became_peered":"2024-02-05T08:06:39.558993+0000","last_unstale":"2024-02-05T08:10:10.699144+0000","last_undegraded":"2024-02-05T08:10:10.699144+0000","last_fullsized":"2024-02-05T08:10:10.699144+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:45.721527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:45.721527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494749+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":200,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.698484+0000","last_change":"2024-02-05T08:07:50.544800+0000","last_active":"2024-02-05T08:10:10.698484+0000","last_peered":"2024-02-05T08:10:10.698484+0000","last_clean":"2024-02-05T08:10:10.698484+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:10:10.698484+0000","last_undegraded":"2024-02-05T08:10:10.698484+0000","last_fullsized":"2024-02-05T08:10:10.698484+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:07:50.544673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:07:50.544673+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"147'15186","reported_seq":15394,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:12.995709+0000","last_change":"2024-02-05T08:09:51.200716+0000","last_active":"2024-02-05T08:10:12.995709+0000","last_peered":"2024-02-05T08:10:12.995709+0000","last_clean":"2024-02-05T08:10:12.995709+0000","last_became_active":"2024-02-05T08:06:39.561430+0000","last_became_peered":"2024-02-05T08:06:39.561430+0000","last_unstale":"2024-02-05T08:10:12.995709+0000","last_undegraded":"2024-02-05T08:10:12.995709+0000","last_fullsized":"2024-02-05T08:10:12.995709+0000","mapping_epoch":97,"log_start":"139'10400","ondisk_log_start":"139'10400","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'13917","last_scrub_stamp":"2024-02-05T08:09:51.200561+0000","last_deep_scrub":"144'13917","last_deep_scrub_stamp":"2024-02-05T08:09:51.200561+0000","last_clean_scrub_stamp":"2024-02-05T08:09:51.200561+0000","objects_scrubbed":2109,"log_size":4786,"log_dups_size":0,"ondisk_log_size":4786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":17,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:18.777091+0000","scrub_duration":16228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33226752,"num_objects":507,"num_object_clones":0,"num_object_copies":1521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13077,"num_write_kb":434688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":168,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.709257+0000","last_change":"2024-02-05T08:07:19.682609+0000","last_active":"2024-02-05T08:10:10.709257+0000","last_peered":"2024-02-05T08:10:10.709257+0000","last_clean":"2024-02-05T08:10:10.709257+0000","last_became_active":"2024-02-05T08:04:47.588395+0000","last_became_peered":"2024-02-05T08:04:47.588395+0000","last_unstale":"2024-02-05T08:10:10.709257+0000","last_undegraded":"2024-02-05T08:10:10.709257+0000","last_fullsized":"2024-02-05T08:10:10.709257+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:19.682361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:19.682361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":210,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699060+0000","last_change":"2024-02-05T08:07:40.810244+0000","last_active":"2024-02-05T08:10:10.699060+0000","last_peered":"2024-02-05T08:10:10.699060+0000","last_clean":"2024-02-05T08:10:10.699060+0000","last_became_active":"2024-02-05T08:06:39.560631+0000","last_became_peered":"2024-02-05T08:06:39.560631+0000","last_unstale":"2024-02-05T08:10:10.699060+0000","last_undegraded":"2024-02-05T08:10:10.699060+0000","last_fullsized":"2024-02-05T08:10:10.699060+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:07:40.810076+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:07:40.810076+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494693+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":194,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.693933+0000","last_change":"2024-02-05T08:08:34.373002+0000","last_active":"2024-02-05T08:10:10.693933+0000","last_peered":"2024-02-05T08:10:10.693933+0000","last_clean":"2024-02-05T08:10:10.693933+0000","last_became_active":"2024-02-05T08:06:39.563051+0000","last_became_peered":"2024-02-05T08:06:39.563051+0000","last_unstale":"2024-02-05T08:10:10.693933+0000","last_undegraded":"2024-02-05T08:10:10.693933+0000","last_fullsized":"2024-02-05T08:10:10.693933+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:34.372891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:08:34.372891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536103+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"85'38","reported_seq":318,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699898+0000","last_change":"2024-02-05T08:09:35.108648+0000","last_active":"2024-02-05T08:10:10.699898+0000","last_peered":"2024-02-05T08:10:10.699898+0000","last_clean":"2024-02-05T08:10:10.699898+0000","last_became_active":"2024-02-05T08:04:43.554536+0000","last_became_peered":"2024-02-05T08:04:43.554536+0000","last_unstale":"2024-02-05T08:10:10.699898+0000","last_undegraded":"2024-02-05T08:10:10.699898+0000","last_fullsized":"2024-02-05T08:10:10.699898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:09:35.108489+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:09:35.108489+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:50.058243+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":172,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.703599+0000","last_change":"2024-02-05T08:07:19.436649+0000","last_active":"2024-02-05T08:10:10.703599+0000","last_peered":"2024-02-05T08:10:10.703599+0000","last_clean":"2024-02-05T08:10:10.703599+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:10:10.703599+0000","last_undegraded":"2024-02-05T08:10:10.703599+0000","last_fullsized":"2024-02-05T08:10:10.703599+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:19.436541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:19.436541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"147'13410","reported_seq":13568,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:13.043502+0000","last_change":"2024-02-05T08:09:26.200715+0000","last_active":"2024-02-05T08:10:13.043502+0000","last_peered":"2024-02-05T08:10:13.043502+0000","last_clean":"2024-02-05T08:10:13.043502+0000","last_became_active":"2024-02-05T08:06:37.618966+0000","last_became_peered":"2024-02-05T08:06:37.618966+0000","last_unstale":"2024-02-05T08:10:13.043502+0000","last_undegraded":"2024-02-05T08:10:13.043502+0000","last_fullsized":"2024-02-05T08:10:13.043502+0000","mapping_epoch":95,"log_start":"131'8600","ondisk_log_start":"131'8600","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10764","last_scrub_stamp":"2024-02-05T08:09:26.200471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:26.200471+0000","objects_scrubbed":3228,"log_size":4810,"log_dups_size":0,"ondisk_log_size":4810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:26.980375+0000","scrub_duration":1266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35651584,"num_objects":544,"num_object_clones":0,"num_object_copies":1632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13410,"num_write_kb":446528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":183,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.705052+0000","last_change":"2024-02-05T08:08:39.475407+0000","last_active":"2024-02-05T08:10:10.705052+0000","last_peered":"2024-02-05T08:10:10.705052+0000","last_clean":"2024-02-05T08:10:10.705052+0000","last_became_active":"2024-02-05T08:06:39.555147+0000","last_became_peered":"2024-02-05T08:06:39.555147+0000","last_unstale":"2024-02-05T08:10:10.705052+0000","last_undegraded":"2024-02-05T08:10:10.705052+0000","last_fullsized":"2024-02-05T08:10:10.705052+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:39.475316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.475316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.194619+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":206,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.697319+0000","last_change":"2024-02-05T08:08:41.006239+0000","last_active":"2024-02-05T08:10:10.697319+0000","last_peered":"2024-02-05T08:10:10.697319+0000","last_clean":"2024-02-05T08:10:10.697319+0000","last_became_active":"2024-02-05T08:06:39.560285+0000","last_became_peered":"2024-02-05T08:06:39.560285+0000","last_unstale":"2024-02-05T08:10:10.697319+0000","last_undegraded":"2024-02-05T08:10:10.697319+0000","last_fullsized":"2024-02-05T08:10:10.697319+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:08:41.005907+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:08:41.005907+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.438097+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":206,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.714084+0000","last_change":"2024-02-05T08:08:44.750119+0000","last_active":"2024-02-05T08:10:10.714084+0000","last_peered":"2024-02-05T08:10:10.714084+0000","last_clean":"2024-02-05T08:10:10.714084+0000","last_became_active":"2024-02-05T08:04:40.529073+0000","last_became_peered":"2024-02-05T08:04:40.529073+0000","last_unstale":"2024-02-05T08:10:10.714084+0000","last_undegraded":"2024-02-05T08:10:10.714084+0000","last_fullsized":"2024-02-05T08:10:10.714084+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:44.750015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.750015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.490966+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":189,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.713898+0000","last_change":"2024-02-05T08:08:37.676223+0000","last_active":"2024-02-05T08:10:10.713898+0000","last_peered":"2024-02-05T08:10:10.713898+0000","last_clean":"2024-02-05T08:10:10.713898+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:10:10.713898+0000","last_undegraded":"2024-02-05T08:10:10.713898+0000","last_fullsized":"2024-02-05T08:10:10.713898+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:08:37.676082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:37.676082+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.490911+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":189,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.714906+0000","last_change":"2024-02-05T08:09:25.547689+0000","last_active":"2024-02-05T08:10:10.714906+0000","last_peered":"2024-02-05T08:10:10.714906+0000","last_clean":"2024-02-05T08:10:10.714906+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:10:10.714906+0000","last_undegraded":"2024-02-05T08:10:10.714906+0000","last_fullsized":"2024-02-05T08:10:10.714906+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.547581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.547581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:55.227516+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"147'13204","reported_seq":13351,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:13.033270+0000","last_change":"2024-02-05T08:09:37.898638+0000","last_active":"2024-02-05T08:10:13.033270+0000","last_peered":"2024-02-05T08:10:13.033270+0000","last_clean":"2024-02-05T08:10:13.033270+0000","last_became_active":"2024-02-05T08:06:37.622047+0000","last_became_peered":"2024-02-05T08:06:37.622047+0000","last_unstale":"2024-02-05T08:10:13.033270+0000","last_undegraded":"2024-02-05T08:10:13.033270+0000","last_fullsized":"2024-02-05T08:10:13.033270+0000","mapping_epoch":95,"log_start":"131'8400","ondisk_log_start":"131'8400","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'11226","last_scrub_stamp":"2024-02-05T08:09:37.898513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:37.898513+0000","objects_scrubbed":2513,"log_size":4804,"log_dups_size":0,"ondisk_log_size":4804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:04.919652+0000","scrub_duration":880,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34209792,"num_objects":522,"num_object_clones":0,"num_object_copies":1566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13204,"num_write_kb":439232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":176,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699401+0000","last_change":"2024-02-05T08:08:34.051757+0000","last_active":"2024-02-05T08:10:10.699401+0000","last_peered":"2024-02-05T08:10:10.699401+0000","last_clean":"2024-02-05T08:10:10.699401+0000","last_became_active":"2024-02-05T08:04:47.586358+0000","last_became_peered":"2024-02-05T08:04:47.586358+0000","last_unstale":"2024-02-05T08:10:10.699401+0000","last_undegraded":"2024-02-05T08:10:10.699401+0000","last_fullsized":"2024-02-05T08:10:10.699401+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:34.051658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:34.051658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494827+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":461,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704142+0000","last_change":"2024-02-05T08:08:49.423194+0000","last_active":"2024-02-05T08:10:10.704142+0000","last_peered":"2024-02-05T08:10:10.704142+0000","last_clean":"2024-02-05T08:10:10.704142+0000","last_became_active":"2024-02-05T08:06:42.566771+0000","last_became_peered":"2024-02-05T08:06:42.566771+0000","last_unstale":"2024-02-05T08:10:10.704142+0000","last_undegraded":"2024-02-05T08:10:10.704142+0000","last_fullsized":"2024-02-05T08:10:10.704142+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:08:49.423065+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:08:49.423065+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.194604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":194,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.708681+0000","last_change":"2024-02-05T08:08:39.073558+0000","last_active":"2024-02-05T08:10:10.708681+0000","last_peered":"2024-02-05T08:10:10.708681+0000","last_clean":"2024-02-05T08:10:10.708681+0000","last_became_active":"2024-02-05T08:06:39.553704+0000","last_became_peered":"2024-02-05T08:06:39.553704+0000","last_unstale":"2024-02-05T08:10:10.708681+0000","last_undegraded":"2024-02-05T08:10:10.708681+0000","last_fullsized":"2024-02-05T08:10:10.708681+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.071354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962956+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":223,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694545+0000","last_change":"2024-02-05T08:07:51.975957+0000","last_active":"2024-02-05T08:10:10.694545+0000","last_peered":"2024-02-05T08:10:10.694545+0000","last_clean":"2024-02-05T08:10:10.694545+0000","last_became_active":"2024-02-05T08:06:39.563373+0000","last_became_peered":"2024-02-05T08:06:39.563373+0000","last_unstale":"2024-02-05T08:10:10.694545+0000","last_undegraded":"2024-02-05T08:10:10.694545+0000","last_fullsized":"2024-02-05T08:10:10.694545+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-02-05T08:07:51.975838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:07:51.975838+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536005+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":181,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704779+0000","last_change":"2024-02-05T08:08:38.472753+0000","last_active":"2024-02-05T08:10:10.704779+0000","last_peered":"2024-02-05T08:10:10.704779+0000","last_clean":"2024-02-05T08:10:10.704779+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:10:10.704779+0000","last_undegraded":"2024-02-05T08:10:10.704779+0000","last_fullsized":"2024-02-05T08:10:10.704779+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:38.472647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:38.472647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.194587+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"147'13160","reported_seq":13339,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:12.488190+0000","last_change":"2024-02-05T08:08:42.646048+0000","last_active":"2024-02-05T08:10:12.488190+0000","last_peered":"2024-02-05T08:10:12.488190+0000","last_clean":"2024-02-05T08:10:12.488190+0000","last_became_active":"2024-02-05T08:06:37.623983+0000","last_became_peered":"2024-02-05T08:06:37.623983+0000","last_unstale":"2024-02-05T08:10:12.488190+0000","last_undegraded":"2024-02-05T08:10:12.488190+0000","last_fullsized":"2024-02-05T08:10:12.488190+0000","mapping_epoch":95,"log_start":"129'8100","ondisk_log_start":"129'8100","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'8153","last_scrub_stamp":"2024-02-05T08:08:42.645901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:42.645901+0000","objects_scrubbed":5574,"log_size":5060,"log_dups_size":0,"ondisk_log_size":5060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.194669+0000","scrub_duration":1122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":538,"num_object_clones":0,"num_object_copies":1614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13160,"num_write_kb":438336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":177,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.698410+0000","last_change":"2024-02-05T08:07:43.938444+0000","last_active":"2024-02-05T08:10:10.698410+0000","last_peered":"2024-02-05T08:10:10.698410+0000","last_clean":"2024-02-05T08:10:10.698410+0000","last_became_active":"2024-02-05T08:04:47.586774+0000","last_became_peered":"2024-02-05T08:04:47.586774+0000","last_unstale":"2024-02-05T08:10:10.698410+0000","last_undegraded":"2024-02-05T08:10:10.698410+0000","last_fullsized":"2024-02-05T08:10:10.698410+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:43.938278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:43.938278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.435475+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":188,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.703627+0000","last_change":"2024-02-05T08:08:38.705016+0000","last_active":"2024-02-05T08:10:10.703627+0000","last_peered":"2024-02-05T08:10:10.703627+0000","last_clean":"2024-02-05T08:10:10.703627+0000","last_became_active":"2024-02-05T08:04:41.537872+0000","last_became_peered":"2024-02-05T08:04:41.537872+0000","last_unstale":"2024-02-05T08:10:10.703627+0000","last_undegraded":"2024-02-05T08:10:10.703627+0000","last_fullsized":"2024-02-05T08:10:10.703627+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:38.704911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:38.704911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":534,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:11.183621+0000","last_change":"2024-02-05T08:09:26.492000+0000","last_active":"2024-02-05T08:10:11.183621+0000","last_peered":"2024-02-05T08:10:11.183621+0000","last_clean":"2024-02-05T08:10:11.183621+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:10:11.183621+0000","last_undegraded":"2024-02-05T08:10:11.183621+0000","last_fullsized":"2024-02-05T08:10:11.183621+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:09:26.491884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:09:26.491884+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:39.146672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":325,"num_read_kb":320,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":202,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.707279+0000","last_change":"2024-02-05T08:09:19.836523+0000","last_active":"2024-02-05T08:10:10.707279+0000","last_peered":"2024-02-05T08:10:10.707279+0000","last_clean":"2024-02-05T08:10:10.707279+0000","last_became_active":"2024-02-05T08:04:43.549687+0000","last_became_peered":"2024-02-05T08:04:43.549687+0000","last_unstale":"2024-02-05T08:10:10.707279+0000","last_undegraded":"2024-02-05T08:10:10.707279+0000","last_fullsized":"2024-02-05T08:10:10.707279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:09:19.836401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.836401+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:38.806378+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":180,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699621+0000","last_change":"2024-02-05T08:08:29.956921+0000","last_active":"2024-02-05T08:10:10.699621+0000","last_peered":"2024-02-05T08:10:10.699621+0000","last_clean":"2024-02-05T08:10:10.699621+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:10:10.699621+0000","last_undegraded":"2024-02-05T08:10:10.699621+0000","last_fullsized":"2024-02-05T08:10:10.699621+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:29.956845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:29.956845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494842+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"147'16522","reported_seq":16827,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:12.538925+0000","last_change":"2024-02-05T08:09:18.016743+0000","last_active":"2024-02-05T08:10:12.538925+0000","last_peered":"2024-02-05T08:10:12.538925+0000","last_clean":"2024-02-05T08:10:12.538925+0000","last_became_active":"2024-02-05T08:06:37.624241+0000","last_became_peered":"2024-02-05T08:06:37.624241+0000","last_unstale":"2024-02-05T08:10:12.538925+0000","last_undegraded":"2024-02-05T08:10:12.538925+0000","last_fullsized":"2024-02-05T08:10:12.538925+0000","mapping_epoch":95,"log_start":"133'11002","ondisk_log_start":"133'11002","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'13560","last_scrub_stamp":"2024-02-05T08:09:18.016504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:18.016504+0000","objects_scrubbed":3593,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:30.849106+0000","scrub_duration":742,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40173568,"num_objects":613,"num_object_clones":0,"num_object_copies":1839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13089,"num_write_kb":438464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":183,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695524+0000","last_change":"2024-02-05T08:07:41.002345+0000","last_active":"2024-02-05T08:10:10.695524+0000","last_peered":"2024-02-05T08:10:10.695524+0000","last_clean":"2024-02-05T08:10:10.695524+0000","last_became_active":"2024-02-05T08:06:39.559847+0000","last_became_peered":"2024-02-05T08:06:39.559847+0000","last_unstale":"2024-02-05T08:10:10.695524+0000","last_undegraded":"2024-02-05T08:10:10.695524+0000","last_fullsized":"2024-02-05T08:10:10.695524+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:41.002067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:41.002067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.435337+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":197,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699687+0000","last_change":"2024-02-05T08:08:44.964475+0000","last_active":"2024-02-05T08:10:10.699687+0000","last_peered":"2024-02-05T08:10:10.699687+0000","last_clean":"2024-02-05T08:10:10.699687+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:10:10.699687+0000","last_undegraded":"2024-02-05T08:10:10.699687+0000","last_fullsized":"2024-02-05T08:10:10.699687+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:44.964357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.964357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.438068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":220,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.698355+0000","last_change":"2024-02-05T08:07:33.417917+0000","last_active":"2024-02-05T08:10:10.698355+0000","last_peered":"2024-02-05T08:10:10.698355+0000","last_clean":"2024-02-05T08:10:10.698355+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:10:10.698355+0000","last_undegraded":"2024-02-05T08:10:10.698355+0000","last_fullsized":"2024-02-05T08:10:10.698355+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:07:33.417781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:07:33.417781+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":119,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.697784+0000","last_change":"2024-02-05T08:08:31.623904+0000","last_active":"2024-02-05T08:10:10.697784+0000","last_peered":"2024-02-05T08:10:10.697784+0000","last_clean":"2024-02-05T08:10:10.697784+0000","last_became_active":"2024-02-05T08:06:39.558100+0000","last_became_peered":"2024-02-05T08:06:39.558100+0000","last_unstale":"2024-02-05T08:10:10.697784+0000","last_undegraded":"2024-02-05T08:10:10.697784+0000","last_fullsized":"2024-02-05T08:10:10.697784+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:31.623748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:08:31.623748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"147'13198","reported_seq":13378,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:12.514190+0000","last_change":"2024-02-05T08:08:46.770155+0000","last_active":"2024-02-05T08:10:12.514190+0000","last_peered":"2024-02-05T08:10:12.514190+0000","last_clean":"2024-02-05T08:10:12.514190+0000","last_became_active":"2024-02-05T08:06:39.561154+0000","last_became_peered":"2024-02-05T08:06:39.561154+0000","last_unstale":"2024-02-05T08:10:12.514190+0000","last_undegraded":"2024-02-05T08:10:12.514190+0000","last_fullsized":"2024-02-05T08:10:12.514190+0000","mapping_epoch":97,"log_start":"131'8600","ondisk_log_start":"131'8600","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8468","last_scrub_stamp":"2024-02-05T08:08:46.770050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:46.770050+0000","objects_scrubbed":5286,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536138+0000","scrub_duration":1335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34734080,"num_objects":530,"num_object_clones":0,"num_object_copies":1590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13198,"num_write_kb":439296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":176,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.697000+0000","last_change":"2024-02-05T08:07:34.484047+0000","last_active":"2024-02-05T08:10:10.697000+0000","last_peered":"2024-02-05T08:10:10.697000+0000","last_clean":"2024-02-05T08:10:10.697000+0000","last_became_active":"2024-02-05T08:04:47.583866+0000","last_became_peered":"2024-02-05T08:04:47.583866+0000","last_unstale":"2024-02-05T08:10:10.697000+0000","last_undegraded":"2024-02-05T08:10:10.697000+0000","last_fullsized":"2024-02-05T08:10:10.697000+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:34.483937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:34.483937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":196,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699456+0000","last_change":"2024-02-05T08:09:40.402414+0000","last_active":"2024-02-05T08:10:10.699456+0000","last_peered":"2024-02-05T08:10:10.699456+0000","last_clean":"2024-02-05T08:10:10.699456+0000","last_became_active":"2024-02-05T08:04:41.537818+0000","last_became_peered":"2024-02-05T08:04:41.537818+0000","last_unstale":"2024-02-05T08:10:10.699456+0000","last_undegraded":"2024-02-05T08:10:10.699456+0000","last_fullsized":"2024-02-05T08:10:10.699456+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:40.402311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:40.402311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.704969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":200,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695428+0000","last_change":"2024-02-05T08:07:47.955278+0000","last_active":"2024-02-05T08:10:10.695428+0000","last_peered":"2024-02-05T08:10:10.695428+0000","last_clean":"2024-02-05T08:10:10.695428+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:10:10.695428+0000","last_undegraded":"2024-02-05T08:10:10.695428+0000","last_fullsized":"2024-02-05T08:10:10.695428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:07:47.955013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:07:47.955013+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.435373+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":265,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.698621+0000","last_change":"2024-02-05T08:07:48.538033+0000","last_active":"2024-02-05T08:10:10.698621+0000","last_peered":"2024-02-05T08:10:10.698621+0000","last_clean":"2024-02-05T08:10:10.698621+0000","last_became_active":"2024-02-05T08:06:39.554860+0000","last_became_peered":"2024-02-05T08:06:39.554860+0000","last_unstale":"2024-02-05T08:10:10.698621+0000","last_undegraded":"2024-02-05T08:10:10.698621+0000","last_fullsized":"2024-02-05T08:10:10.698621+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:07:48.537913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:48.537913+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"147'13265","reported_seq":13439,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:12.810321+0000","last_change":"2024-02-05T08:09:22.744155+0000","last_active":"2024-02-05T08:10:12.810321+0000","last_peered":"2024-02-05T08:10:12.810321+0000","last_clean":"2024-02-05T08:10:12.810321+0000","last_became_active":"2024-02-05T08:06:37.621499+0000","last_became_peered":"2024-02-05T08:06:37.621499+0000","last_unstale":"2024-02-05T08:10:12.810321+0000","last_undegraded":"2024-02-05T08:10:12.810321+0000","last_fullsized":"2024-02-05T08:10:12.810321+0000","mapping_epoch":95,"log_start":"133'9000","ondisk_log_start":"133'9000","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10467","last_scrub_stamp":"2024-02-05T08:09:22.743920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.743920+0000","objects_scrubbed":3450,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:42.513741+0000","scrub_duration":2390,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38207488,"num_objects":583,"num_object_clones":0,"num_object_copies":1749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13265,"num_write_kb":443136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":183,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.704641+0000","last_change":"2024-02-05T08:08:43.475720+0000","last_active":"2024-02-05T08:10:10.704641+0000","last_peered":"2024-02-05T08:10:10.704641+0000","last_clean":"2024-02-05T08:10:10.704641+0000","last_became_active":"2024-02-05T08:06:39.554165+0000","last_became_peered":"2024-02-05T08:06:39.554165+0000","last_unstale":"2024-02-05T08:10:10.704641+0000","last_undegraded":"2024-02-05T08:10:10.704641+0000","last_fullsized":"2024-02-05T08:10:10.704641+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:43.475636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:43.475636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.194651+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":189,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.707378+0000","last_change":"2024-02-05T08:07:25.394409+0000","last_active":"2024-02-05T08:10:10.707378+0000","last_peered":"2024-02-05T08:10:10.707378+0000","last_clean":"2024-02-05T08:10:10.707378+0000","last_became_active":"2024-02-05T08:04:41.534433+0000","last_became_peered":"2024-02-05T08:04:41.534433+0000","last_unstale":"2024-02-05T08:10:10.707378+0000","last_undegraded":"2024-02-05T08:10:10.707378+0000","last_fullsized":"2024-02-05T08:10:10.707378+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:07:25.394122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:25.394122+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":219,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694789+0000","last_change":"2024-02-05T08:07:34.844340+0000","last_active":"2024-02-05T08:10:10.694789+0000","last_peered":"2024-02-05T08:10:10.694789+0000","last_clean":"2024-02-05T08:10:10.694789+0000","last_became_active":"2024-02-05T08:06:39.566434+0000","last_became_peered":"2024-02-05T08:06:39.566434+0000","last_unstale":"2024-02-05T08:10:10.694789+0000","last_undegraded":"2024-02-05T08:10:10.694789+0000","last_fullsized":"2024-02-05T08:10:10.694789+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-02-05T08:07:34.844214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:07:34.844214+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.535985+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":255,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.709232+0000","last_change":"2024-02-05T08:07:47.765640+0000","last_active":"2024-02-05T08:10:10.709232+0000","last_peered":"2024-02-05T08:10:10.709232+0000","last_clean":"2024-02-05T08:10:10.709232+0000","last_became_active":"2024-02-05T08:04:45.562597+0000","last_became_peered":"2024-02-05T08:04:45.562597+0000","last_unstale":"2024-02-05T08:10:10.709232+0000","last_undegraded":"2024-02-05T08:10:10.709232+0000","last_fullsized":"2024-02-05T08:10:10.709232+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:07:47.765523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:47.765523+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962931+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"147'13259","reported_seq":13438,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:12.815042+0000","last_change":"2024-02-05T08:08:44.289281+0000","last_active":"2024-02-05T08:10:12.815042+0000","last_peered":"2024-02-05T08:10:12.815042+0000","last_clean":"2024-02-05T08:10:12.815042+0000","last_became_active":"2024-02-05T08:06:37.621439+0000","last_became_peered":"2024-02-05T08:06:37.621439+0000","last_unstale":"2024-02-05T08:10:12.815042+0000","last_undegraded":"2024-02-05T08:10:12.815042+0000","last_fullsized":"2024-02-05T08:10:12.815042+0000","mapping_epoch":95,"log_start":"133'9000","ondisk_log_start":"133'9000","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8323","last_scrub_stamp":"2024-02-05T08:08:44.289025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.289025+0000","objects_scrubbed":5548,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.077945+0000","scrub_duration":2139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36896768,"num_objects":563,"num_object_clones":0,"num_object_copies":1689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13259,"num_write_kb":442304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":176,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.693918+0000","last_change":"2024-02-05T08:07:40.845208+0000","last_active":"2024-02-05T08:10:10.693918+0000","last_peered":"2024-02-05T08:10:10.693918+0000","last_clean":"2024-02-05T08:10:10.693918+0000","last_became_active":"2024-02-05T08:04:47.582776+0000","last_became_peered":"2024-02-05T08:04:47.582776+0000","last_unstale":"2024-02-05T08:10:10.693918+0000","last_undegraded":"2024-02-05T08:10:10.693918+0000","last_fullsized":"2024-02-05T08:10:10.693918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:40.845130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:40.845130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.535964+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":202,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.706452+0000","last_change":"2024-02-05T08:09:36.111657+0000","last_active":"2024-02-05T08:10:10.706452+0000","last_peered":"2024-02-05T08:10:10.706452+0000","last_clean":"2024-02-05T08:10:10.706452+0000","last_became_active":"2024-02-05T08:04:41.540818+0000","last_became_peered":"2024-02-05T08:04:41.540818+0000","last_unstale":"2024-02-05T08:10:10.706452+0000","last_undegraded":"2024-02-05T08:10:10.706452+0000","last_fullsized":"2024-02-05T08:10:10.706452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:09:36.111537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.111537+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.119806+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":224,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.709057+0000","last_change":"2024-02-05T08:09:33.222821+0000","last_active":"2024-02-05T08:10:10.709057+0000","last_peered":"2024-02-05T08:10:10.709057+0000","last_clean":"2024-02-05T08:10:10.709057+0000","last_became_active":"2024-02-05T08:04:43.607546+0000","last_became_peered":"2024-02-05T08:04:43.607546+0000","last_unstale":"2024-02-05T08:10:10.709057+0000","last_undegraded":"2024-02-05T08:10:10.709057+0000","last_fullsized":"2024-02-05T08:10:10.709057+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-05T08:09:33.222532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:33.222532+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:53.393561+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":180,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694571+0000","last_change":"2024-02-05T08:09:20.449787+0000","last_active":"2024-02-05T08:10:10.694571+0000","last_peered":"2024-02-05T08:10:10.694571+0000","last_clean":"2024-02-05T08:10:10.694571+0000","last_became_active":"2024-02-05T08:04:45.561639+0000","last_became_peered":"2024-02-05T08:04:45.561639+0000","last_unstale":"2024-02-05T08:10:10.694571+0000","last_undegraded":"2024-02-05T08:10:10.694571+0000","last_fullsized":"2024-02-05T08:10:10.694571+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:20.449711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:20.449711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:22.864643+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"147'13142","reported_seq":13349,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:12.856443+0000","last_change":"2024-02-05T08:09:19.688660+0000","last_active":"2024-02-05T08:10:12.856443+0000","last_peered":"2024-02-05T08:10:12.856443+0000","last_clean":"2024-02-05T08:10:12.856443+0000","last_became_active":"2024-02-05T08:06:39.556908+0000","last_became_peered":"2024-02-05T08:06:39.556908+0000","last_unstale":"2024-02-05T08:10:12.856443+0000","last_undegraded":"2024-02-05T08:10:12.856443+0000","last_fullsized":"2024-02-05T08:10:12.856443+0000","mapping_epoch":97,"log_start":"134'8900","ondisk_log_start":"134'8900","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'10128","last_scrub_stamp":"2024-02-05T08:09:19.688502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.688502+0000","objects_scrubbed":3625,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:37.670381+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36962304,"num_objects":564,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13142,"num_write_kb":438592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":184,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699320+0000","last_change":"2024-02-05T08:08:43.142178+0000","last_active":"2024-02-05T08:10:10.699320+0000","last_peered":"2024-02-05T08:10:10.699320+0000","last_clean":"2024-02-05T08:10:10.699320+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:10:10.699320+0000","last_undegraded":"2024-02-05T08:10:10.699320+0000","last_fullsized":"2024-02-05T08:10:10.699320+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:43.142051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:43.142051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494872+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":255,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695213+0000","last_change":"2024-02-05T08:09:17.391826+0000","last_active":"2024-02-05T08:10:10.695213+0000","last_peered":"2024-02-05T08:10:10.695213+0000","last_clean":"2024-02-05T08:10:10.695213+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:10:10.695213+0000","last_undegraded":"2024-02-05T08:10:10.695213+0000","last_fullsized":"2024-02-05T08:10:10.695213+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:09:17.391673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:17.391673+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:31.892417+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":188,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.708946+0000","last_change":"2024-02-05T08:07:36.643035+0000","last_active":"2024-02-05T08:10:10.708946+0000","last_peered":"2024-02-05T08:10:10.708946+0000","last_clean":"2024-02-05T08:10:10.708946+0000","last_became_active":"2024-02-05T08:04:41.541327+0000","last_became_peered":"2024-02-05T08:04:41.541327+0000","last_unstale":"2024-02-05T08:10:10.708946+0000","last_undegraded":"2024-02-05T08:10:10.708946+0000","last_fullsized":"2024-02-05T08:10:10.708946+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:36.642926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:36.642926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:12.962846+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":218,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695398+0000","last_change":"2024-02-05T08:09:17.191152+0000","last_active":"2024-02-05T08:10:10.695398+0000","last_peered":"2024-02-05T08:10:10.695398+0000","last_clean":"2024-02-05T08:10:10.695398+0000","last_became_active":"2024-02-05T08:04:43.555028+0000","last_became_peered":"2024-02-05T08:04:43.555028+0000","last_unstale":"2024-02-05T08:10:10.695398+0000","last_undegraded":"2024-02-05T08:10:10.695398+0000","last_fullsized":"2024-02-05T08:10:10.695398+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:09:17.190290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:17.190290+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:23.351754+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"147'13273","reported_seq":13445,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:12.806509+0000","last_change":"2024-02-05T08:07:48.231620+0000","last_active":"2024-02-05T08:10:12.806509+0000","last_peered":"2024-02-05T08:10:12.806509+0000","last_clean":"2024-02-05T08:10:12.806509+0000","last_became_active":"2024-02-05T08:06:39.555144+0000","last_became_peered":"2024-02-05T08:06:39.555144+0000","last_unstale":"2024-02-05T08:10:12.806509+0000","last_undegraded":"2024-02-05T08:10:12.806509+0000","last_fullsized":"2024-02-05T08:10:12.806509+0000","mapping_epoch":97,"log_start":"134'9100","ondisk_log_start":"134'9100","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4972","last_scrub_stamp":"2024-02-05T08:07:48.231433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:07:48.231433+0000","objects_scrubbed":4921,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.437921+0000","scrub_duration":2285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38141976,"num_objects":583,"num_object_clones":0,"num_object_copies":1749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":583,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13273,"num_write_kb":443329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":182,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.705592+0000","last_change":"2024-02-05T08:09:32.213156+0000","last_active":"2024-02-05T08:10:10.705592+0000","last_peered":"2024-02-05T08:10:10.705592+0000","last_clean":"2024-02-05T08:10:10.705592+0000","last_became_active":"2024-02-05T08:06:39.558047+0000","last_became_peered":"2024-02-05T08:06:39.558047+0000","last_unstale":"2024-02-05T08:10:10.705592+0000","last_undegraded":"2024-02-05T08:10:10.705592+0000","last_fullsized":"2024-02-05T08:10:10.705592+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:32.213044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:09:32.213044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:54.206176+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":186,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.699535+0000","last_change":"2024-02-05T08:07:46.752867+0000","last_active":"2024-02-05T08:10:10.699535+0000","last_peered":"2024-02-05T08:10:10.699535+0000","last_clean":"2024-02-05T08:10:10.699535+0000","last_became_active":"2024-02-05T08:06:39.551933+0000","last_became_peered":"2024-02-05T08:06:39.551933+0000","last_unstale":"2024-02-05T08:10:10.699535+0000","last_undegraded":"2024-02-05T08:10:10.699535+0000","last_fullsized":"2024-02-05T08:10:10.699535+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:46.752781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:46.752781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.494713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":196,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694650+0000","last_change":"2024-02-05T08:09:22.454321+0000","last_active":"2024-02-05T08:10:10.694650+0000","last_peered":"2024-02-05T08:10:10.694650+0000","last_clean":"2024-02-05T08:10:10.694650+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:10:10.694650+0000","last_undegraded":"2024-02-05T08:10:10.694650+0000","last_fullsized":"2024-02-05T08:10:10.694650+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:22.454247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.454247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:32.361376+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":221,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695009+0000","last_change":"2024-02-05T08:08:32.390911+0000","last_active":"2024-02-05T08:10:10.695009+0000","last_peered":"2024-02-05T08:10:10.695009+0000","last_clean":"2024-02-05T08:10:10.695009+0000","last_became_active":"2024-02-05T08:06:39.556433+0000","last_became_peered":"2024-02-05T08:06:39.556433+0000","last_unstale":"2024-02-05T08:10:10.695009+0000","last_undegraded":"2024-02-05T08:10:10.695009+0000","last_fullsized":"2024-02-05T08:10:10.695009+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:08:32.390785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:32.390785+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536121+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"147'13295","reported_seq":13477,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:12.714877+0000","last_change":"2024-02-05T08:08:39.042845+0000","last_active":"2024-02-05T08:10:12.714877+0000","last_peered":"2024-02-05T08:10:12.714877+0000","last_clean":"2024-02-05T08:10:12.714877+0000","last_became_active":"2024-02-05T08:06:37.624003+0000","last_became_peered":"2024-02-05T08:06:37.624003+0000","last_unstale":"2024-02-05T08:10:12.714877+0000","last_undegraded":"2024-02-05T08:10:12.714877+0000","last_fullsized":"2024-02-05T08:10:12.714877+0000","mapping_epoch":95,"log_start":"128'7900","ondisk_log_start":"128'7900","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'8070","last_scrub_stamp":"2024-02-05T08:08:39.042536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.042536+0000","objects_scrubbed":5875,"log_size":5395,"log_dups_size":0,"ondisk_log_size":5395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.490877+0000","scrub_duration":2379,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37158912,"num_objects":567,"num_object_clones":0,"num_object_copies":1701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13295,"num_write_kb":443584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":186,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.714080+0000","last_change":"2024-02-05T08:08:45.814521+0000","last_active":"2024-02-05T08:10:10.714080+0000","last_peered":"2024-02-05T08:10:10.714080+0000","last_clean":"2024-02-05T08:10:10.714080+0000","last_became_active":"2024-02-05T08:04:47.585962+0000","last_became_peered":"2024-02-05T08:04:47.585962+0000","last_unstale":"2024-02-05T08:10:10.714080+0000","last_undegraded":"2024-02-05T08:10:10.714080+0000","last_fullsized":"2024-02-05T08:10:10.714080+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:08:45.814384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:45.814384+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.490994+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":181,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.700007+0000","last_change":"2024-02-05T08:08:29.100467+0000","last_active":"2024-02-05T08:10:10.700007+0000","last_peered":"2024-02-05T08:10:10.700007+0000","last_clean":"2024-02-05T08:10:10.700007+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:10:10.700007+0000","last_undegraded":"2024-02-05T08:10:10.700007+0000","last_fullsized":"2024-02-05T08:10:10.700007+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:29.100364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:29.100364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:15.437986+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":119,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.693964+0000","last_change":"2024-02-05T08:07:53.915557+0000","last_active":"2024-02-05T08:10:10.693964+0000","last_peered":"2024-02-05T08:10:10.693964+0000","last_clean":"2024-02-05T08:10:10.693964+0000","last_became_active":"2024-02-05T08:06:39.559090+0000","last_became_peered":"2024-02-05T08:06:39.559090+0000","last_unstale":"2024-02-05T08:10:10.693964+0000","last_undegraded":"2024-02-05T08:10:10.693964+0000","last_fullsized":"2024-02-05T08:10:10.693964+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:07:53.915449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536021+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":212,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694370+0000","last_change":"2024-02-05T08:09:21.434983+0000","last_active":"2024-02-05T08:10:10.694370+0000","last_peered":"2024-02-05T08:10:10.694370+0000","last_clean":"2024-02-05T08:10:10.694370+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:10:10.694370+0000","last_undegraded":"2024-02-05T08:10:10.694370+0000","last_fullsized":"2024-02-05T08:10:10.694370+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-02-05T08:09:21.434891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.434891+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:48.312307+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"147'13551","reported_seq":13728,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:12.995795+0000","last_change":"2024-02-05T08:08:50.574791+0000","last_active":"2024-02-05T08:10:12.995795+0000","last_peered":"2024-02-05T08:10:12.995795+0000","last_clean":"2024-02-05T08:10:12.995795+0000","last_became_active":"2024-02-05T08:06:37.622936+0000","last_became_peered":"2024-02-05T08:06:37.622936+0000","last_unstale":"2024-02-05T08:10:12.995795+0000","last_undegraded":"2024-02-05T08:10:12.995795+0000","last_fullsized":"2024-02-05T08:10:12.995795+0000","mapping_epoch":95,"log_start":"131'8700","ondisk_log_start":"131'8700","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8798","last_scrub_stamp":"2024-02-05T08:08:50.574449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:50.574449+0000","objects_scrubbed":5317,"log_size":4851,"log_dups_size":0,"ondisk_log_size":4851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:13.984937+0000","scrub_duration":1873,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35061760,"num_objects":535,"num_object_clones":0,"num_object_copies":1605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13551,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":176,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.695259+0000","last_change":"2024-02-05T08:08:49.450529+0000","last_active":"2024-02-05T08:10:10.695259+0000","last_peered":"2024-02-05T08:10:10.695259+0000","last_clean":"2024-02-05T08:10:10.695259+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:10:10.695259+0000","last_undegraded":"2024-02-05T08:10:10.695259+0000","last_fullsized":"2024-02-05T08:10:10.695259+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:49.450400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:49.450400+0000","last_clean_scrub_stamp":"2024-02-05T08:08:49.450400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536171+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":265,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.713990+0000","last_change":"2024-02-05T08:08:39.742015+0000","last_active":"2024-02-05T08:10:10.713990+0000","last_peered":"2024-02-05T08:10:10.713990+0000","last_clean":"2024-02-05T08:10:10.713990+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:10:10.713990+0000","last_undegraded":"2024-02-05T08:10:10.713990+0000","last_fullsized":"2024-02-05T08:10:10.713990+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.741871+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.490940+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":188,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.705701+0000","last_change":"2024-02-05T08:09:22.380135+0000","last_active":"2024-02-05T08:10:10.705701+0000","last_peered":"2024-02-05T08:10:10.705701+0000","last_clean":"2024-02-05T08:10:10.705701+0000","last_became_active":"2024-02-05T08:04:41.533838+0000","last_became_peered":"2024-02-05T08:04:41.533838+0000","last_unstale":"2024-02-05T08:10:10.705701+0000","last_undegraded":"2024-02-05T08:10:10.705701+0000","last_fullsized":"2024-02-05T08:10:10.705701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:22.380016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.380016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:34.894648+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":217,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-02-05T08:10:10.694247+0000","last_change":"2024-02-05T08:08:35.372047+0000","last_active":"2024-02-05T08:10:10.694247+0000","last_peered":"2024-02-05T08:10:10.694247+0000","last_clean":"2024-02-05T08:10:10.694247+0000","last_became_active":"2024-02-05T08:06:39.555561+0000","last_became_peered":"2024-02-05T08:06:39.555561+0000","last_unstale":"2024-02-05T08:10:10.694247+0000","last_undegraded":"2024-02-05T08:10:10.694247+0000","last_fullsized":"2024-02-05T08:10:10.694247+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:08:35.371853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:35.371853+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:14.536155+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":576454680,"num_objects":8797,"num_object_clones":0,"num_object_copies":26391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8797,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211067,"num_write_kb":7035585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1743007744,"data_stored":1742995528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75507,"ondisk_log_size":75507,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":325,"num_read_kb":320,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":1474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":433,"ondisk_log_size":433,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037831,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":293020,"kb_used_data":185492,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93452132,"statfs":{"total":95995035648,"available":95694983168,"internally_reserved":0,"allocated":189943808,"data_stored":189646702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":3.9300000000000002,"5min":1.468,"15min":0.85999999999999999},"min":{"1min":0.23000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":60.127000000000002,"5min":60.127000000000002,"15min":60.127000000000002},"last":0.314},{"interface":"front","average":{"1min":4.2430000000000003,"5min":1.508,"15min":0.85199999999999998},"min":{"1min":0.251,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":58.283000000000001,"5min":58.283000000000001,"15min":58.283000000000001},"last":0.38600000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":2.3999999999999999,"5min":1.331,"15min":0.81499999999999995},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.774000000000001,"5min":20.774000000000001,"15min":20.774000000000001},"last":2.48},{"interface":"front","average":{"1min":1.702,"5min":1.0249999999999999,"15min":0.70899999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":20.715,"5min":20.715,"15min":20.715},"last":1.903}]},{"osd":2,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.78400000000000003,"15min":0.64900000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6779999999999999,"5min":7.6779999999999999,"15min":7.6779999999999999},"last":3.0579999999999998},{"interface":"front","average":{"1min":1.8109999999999999,"5min":1.1000000000000001,"15min":0.72599999999999998},"min":{"1min":0.26300000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":8.9100000000000001,"5min":8.9100000000000001,"15min":8.9100000000000001},"last":2.3599999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":5.4989999999999997,"5min":1.859,"15min":0.999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":58.515000000000001,"5min":58.515000000000001,"15min":58.515000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":5.4779999999999998,"5min":1.71,"15min":0.93000000000000005},"min":{"1min":0.307,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":58.225999999999999,"5min":58.225999999999999,"15min":58.225999999999999},"last":0.68799999999999994}]},{"osd":4,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.40000000000000002,"15min":0.42499999999999999},"min":{"1min":0.23899999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.49299999999999999,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.37},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.41799999999999998,"15min":0.42799999999999999},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.53000000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40000000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.437,"15min":0.44},"min":{"1min":0.29999999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.90200000000000002,"5min":1.2,"15min":1.2},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46999999999999997,"15min":0.48399999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.81999999999999995,"5min":0.998,"15min":0.998},"last":0.442}]},{"osd":6,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.87,"5min":1.256,"15min":1.256},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.33700000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.56799999999999995,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46400000000000002}]}]},{"osd":6,"up_from":0,"seq":219043332173,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":445948,"kb_used_data":298032,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93299204,"statfs":{"total":95995035648,"available":95538384896,"internally_reserved":0,"allocated":305184768,"data_stored":304858722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:10:03 2024","interfaces":[{"interface":"back","average":{"1min":2.7360000000000002,"5min":2.3450000000000002,"15min":0.98899999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.14699999999999999},"max":{"1min":7.7300000000000004,"5min":86.105000000000004,"15min":86.105000000000004},"last":2.5499999999999998},{"interface":"front","average":{"1min":2.5880000000000001,"5min":2.6930000000000001,"15min":1.1140000000000001},"min":{"1min":0.28999999999999998,"5min":0.22900000000000001,"15min":0.16},"max":{"1min":13.800000000000001,"5min":85.665000000000006,"15min":85.665000000000006},"last":8.4510000000000005}]},{"osd":2,"last update":"Mon Feb 5 08:09:32 2024","interfaces":[{"interface":"back","average":{"1min":1.8380000000000001,"5min":1.972,"15min":0.90600000000000003},"min":{"1min":0.19700000000000001,"5min":0.19500000000000001,"15min":0.17799999999999999},"max":{"1min":6.3079999999999998,"5min":49.030999999999999,"15min":49.030999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":2.2959999999999998,"5min":2.1269999999999998,"15min":0.97799999999999998},"min":{"1min":0.34300000000000003,"5min":0.23100000000000001,"15min":0.21099999999999999},"max":{"1min":13.707000000000001,"5min":48.984000000000002,"15min":48.984000000000002},"last":2.6259999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:09:59 2024","interfaces":[{"interface":"back","average":{"1min":2.3730000000000002,"5min":2.2349999999999999,"15min":1.04},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":12.468999999999999,"5min":49.067999999999998,"15min":49.067999999999998},"last":1.4930000000000001},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.3130000000000002,"15min":1.0780000000000001},"min":{"1min":0.38,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.3940000000000001,"5min":48.734999999999999,"15min":48.734999999999999},"last":1.5109999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:09:24 2024","interfaces":[{"interface":"back","average":{"1min":1.905,"5min":1.071,"15min":0.748},"min":{"1min":0.29699999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":9.4269999999999996,"5min":9.4269999999999996,"15min":9.4269999999999996},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.901,"5min":1.0449999999999999,"15min":0.73299999999999998},"min":{"1min":0.30499999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.3890000000000002,"5min":6.3890000000000002,"15min":6.3890000000000002},"last":1.702}]},{"osd":5,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.163,"15min":0.751},"min":{"1min":0.35899999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":7.7000000000000002,"5min":9.4559999999999995,"15min":9.4559999999999995},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.159,"15min":0.75700000000000001},"min":{"1min":0.40400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":13.776,"5min":13.776,"15min":13.776},"last":1.536}]},{"osd":6,"last update":"Mon Feb 5 08:09:13 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":0.98499999999999999,"15min":0.72299999999999998},"min":{"1min":0.38800000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.4279999999999999,"5min":6.4279999999999999,"15min":6.4279999999999999},"last":2.5139999999999998},{"interface":"front","average":{"1min":1.8009999999999999,"5min":1.046,"15min":0.77200000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:09:32 2024","interfaces":[{"interface":"back","average":{"1min":2.395,"5min":1.2809999999999999,"15min":0.84099999999999997},"min":{"1min":0.434,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":13.749000000000001,"5min":13.749000000000001,"15min":13.749000000000001},"last":2.5979999999999999},{"interface":"front","average":{"1min":2.246,"5min":1.1970000000000001,"15min":0.80500000000000005},"min":{"1min":0.56499999999999995,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":2.4860000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738475,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":419768,"kb_used_data":294128,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93325384,"statfs":{"total":95995035648,"available":95565193216,"internally_reserved":0,"allocated":301187072,"data_stored":300861852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:09:12 2024","interfaces":[{"interface":"back","average":{"1min":6.7000000000000002,"5min":2.1429999999999998,"15min":0.93500000000000005},"min":{"1min":0.19,"5min":0.19,"15min":0.152},"max":{"1min":66.772999999999996,"5min":66.772999999999996,"15min":66.772999999999996},"last":5.2869999999999999},{"interface":"front","average":{"1min":5.4530000000000003,"5min":1.722,"15min":0.79200000000000004},"min":{"1min":0.30599999999999999,"5min":0.249,"15min":0.17199999999999999},"max":{"1min":46.198999999999998,"5min":46.198999999999998,"15min":46.198999999999998},"last":0.51100000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:09:36 2024","interfaces":[{"interface":"back","average":{"1min":5.1539999999999999,"5min":1.9490000000000001,"15min":0.89800000000000002},"min":{"1min":0.56100000000000005,"5min":0.252,"15min":0.218},"max":{"1min":46.170999999999999,"5min":46.170999999999999,"15min":46.170999999999999},"last":0.216},{"interface":"front","average":{"1min":4.7699999999999996,"5min":2.0960000000000001,"15min":0.95799999999999996},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":46.543999999999997,"5min":46.543999999999997,"15min":46.543999999999997},"last":1.1579999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:10:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7869999999999999,"5min":2.419,"15min":1.113},"min":{"1min":0.42099999999999999,"5min":0.28399999999999997,"15min":0.21099999999999999},"max":{"1min":9.6829999999999998,"5min":67.888000000000005,"15min":67.888000000000005},"last":5.3550000000000004},{"interface":"front","average":{"1min":2.7349999999999999,"5min":2.7229999999999999,"15min":1.22},"min":{"1min":0.41099999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":8.8979999999999997,"5min":66.891999999999996,"15min":66.891999999999996},"last":5.2389999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:09:22 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":0.96899999999999997,"15min":0.69899999999999995},"min":{"1min":0.38900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.5140000000000002,"5min":13.295,"15min":13.295},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.161,"5min":1.0269999999999999,"15min":0.746},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.5449999999999999,"5min":12.989000000000001,"15min":12.989000000000001},"last":0.496}]},{"osd":5,"last update":"Mon Feb 5 08:09:50 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.141,"15min":0.78500000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.0190000000000001,"5min":13.073,"15min":13.073},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.175,"15min":0.80800000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":9.5289999999999999,"5min":12.94,"15min":12.94},"last":0.42599999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:09:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5129999999999999,"5min":1.0980000000000001,"15min":0.76700000000000002},"min":{"1min":0.42499999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":4.4290000000000003,"5min":12.967000000000001,"15min":12.967000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.79,"5min":1.1719999999999999,"15min":0.80500000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":5.4560000000000004,"5min":13.108000000000001,"15min":13.108000000000001},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Feb 5 08:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.0469999999999999,"15min":0.72499999999999998},"min":{"1min":0.432,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":5.6150000000000002,"5min":13.263,"15min":13.263},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.244,"15min":0.82099999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":9.6300000000000008,"5min":13.012,"15min":13.012},"last":0.52500000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":422880,"kb_used_data":293076,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93322272,"statfs":{"total":95995035648,"available":95562006528,"internally_reserved":0,"allocated":300109824,"data_stored":299811434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":2.254,"5min":1.667,"15min":0.80400000000000005},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":12.355,"5min":16.155999999999999,"15min":16.155999999999999},"last":2.7269999999999999},{"interface":"front","average":{"1min":2.1920000000000002,"5min":1.8240000000000001,"15min":0.86299999999999999},"min":{"1min":0.26500000000000001,"5min":0.23300000000000001,"15min":0.20699999999999999},"max":{"1min":14.243,"5min":15.973000000000001,"15min":15.973000000000001},"last":3.1459999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1419999999999999,"5min":1.4179999999999999,"15min":0.71199999999999997},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":7.0519999999999996,"5min":9.1340000000000003,"15min":9.1340000000000003},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.9630000000000001,"5min":1.4259999999999999,"15min":0.72899999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":7.7919999999999998,"5min":9.2769999999999992,"15min":9.2769999999999992},"last":3.1850000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:09:59 2024","interfaces":[{"interface":"back","average":{"1min":5.7229999999999999,"5min":2.984,"15min":1.3069999999999999},"min":{"1min":0.314,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":68.108000000000004,"5min":68.108000000000004,"15min":68.108000000000004},"last":3.238},{"interface":"front","average":{"1min":5.7880000000000003,"5min":2.6589999999999998,"15min":1.1919999999999999},"min":{"1min":0.33500000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":67.945999999999998,"5min":67.945999999999998,"15min":67.945999999999998},"last":1.6910000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:09:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1739999999999999,"15min":0.79200000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":6.0650000000000004,"5min":15.923999999999999,"15min":15.923999999999999},"last":3.214},{"interface":"front","average":{"1min":1.4390000000000001,"5min":0.94699999999999995,"15min":0.70099999999999996},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":6.008,"5min":6.008,"15min":6.008},"last":1.6399999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:09:50 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.1659999999999999,"15min":0.78100000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.6669999999999998,"5min":15.835000000000001,"15min":15.835000000000001},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.036,"15min":0.72699999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":5.4429999999999996,"5min":9.1809999999999992,"15min":9.1809999999999992},"last":0.317}]},{"osd":6,"last update":"Mon Feb 5 08:09:15 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.2470000000000001,"15min":0.84099999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":6.0999999999999996,"5min":15.859,"15min":15.859},"last":3.2829999999999999},{"interface":"front","average":{"1min":1.506,"5min":1,"15min":0.76100000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":6.117,"5min":6.117,"15min":6.117},"last":1.6639999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.2549999999999999,"15min":0.83299999999999996},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.4870000000000001,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.3,"15min":0.88100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.0540000000000003,"5min":15.951000000000001,"15min":15.951000000000001},"last":3.2599999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117084,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":578356,"kb_used_data":390316,"kb_used_omap":1,"kb_used_meta":188030,"kb_avail":93166796,"statfs":{"total":95995035648,"available":95402799104,"internally_reserved":0,"allocated":399683584,"data_stored":399361345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":192543177},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.1549999999999998,"5min":2.1379999999999999,"15min":0.98499999999999999},"min":{"1min":0.42099999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":6.4169999999999998,"5min":63.43,"15min":63.43},"last":6.2199999999999998},{"interface":"front","average":{"1min":2.5430000000000001,"5min":2.1829999999999998,"15min":1.028},"min":{"1min":0.34300000000000003,"5min":0.186,"15min":0.186},"max":{"1min":8.2650000000000006,"5min":63.048999999999999,"15min":63.048999999999999},"last":0.34699999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":1.496,"15min":0.81999999999999995},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":6.085,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":2.4289999999999998},{"interface":"front","average":{"1min":3.7719999999999998,"5min":2.0539999999999998,"15min":0.995},"min":{"1min":0.34300000000000003,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":8.1760000000000002,"5min":12.087999999999999,"15min":12.087999999999999},"last":0.53800000000000003}]},{"osd":2,"last update":"Mon Feb 5 08:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.3919999999999999,"5min":2.3559999999999999,"15min":1.1160000000000001},"min":{"1min":0.29999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":7.2800000000000002,"5min":62.902999999999999,"15min":62.902999999999999},"last":1.472},{"interface":"front","average":{"1min":2.105,"5min":2.198,"15min":1.054},"min":{"1min":0.54100000000000004,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.3730000000000002,"5min":63.468000000000004,"15min":63.468000000000004},"last":1.2490000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:09:20 2024","interfaces":[{"interface":"back","average":{"1min":1.99,"5min":1.2210000000000001,"15min":0.78100000000000003},"min":{"1min":0.30099999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":7.7229999999999999,"5min":7.7229999999999999,"15min":7.7229999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":2.0790000000000002,"5min":1.323,"15min":0.83699999999999997},"min":{"1min":0.32700000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":8.7059999999999995,"5min":8.7059999999999995,"15min":8.7059999999999995},"last":0.48399999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:09:46 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.276,"15min":0.81799999999999995},"min":{"1min":0.33300000000000002,"5min":0.255,"15min":0.255},"max":{"1min":3.9950000000000001,"5min":7.6680000000000001,"15min":7.6680000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.9139999999999999,"5min":1.4159999999999999,"15min":0.872},"min":{"1min":0.42399999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.8860000000000001,"5min":8.734,"15min":8.734},"last":0.30299999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:10:11 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.4339999999999999,"15min":0.876},"min":{"1min":0.372,"5min":0.372,"15min":0.36499999999999999},"max":{"1min":4.5,"5min":8.9429999999999996,"15min":8.9429999999999996},"last":0.372},{"interface":"front","average":{"1min":2.2869999999999999,"5min":1.4450000000000001,"15min":0.86699999999999999},"min":{"1min":0.34300000000000003,"5min":0.308,"15min":0.308},"max":{"1min":6.9400000000000004,"5min":7.8639999999999999,"15min":7.8639999999999999},"last":0.57499999999999996}]},{"osd":7,"last update":"Mon Feb 5 08:09:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.2889999999999999,"15min":0.82499999999999996},"min":{"1min":0.35699999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":7.7000000000000002,"5min":7.7000000000000002,"15min":7.7000000000000002},"last":0.628},{"interface":"front","average":{"1min":2.294,"5min":1.458,"15min":0.90700000000000003},"min":{"1min":0.41399999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":8.9900000000000002,"5min":8.9900000000000002,"15min":8.9900000000000002},"last":0.39700000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953559,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":282084,"kb_used_data":178716,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93463068,"statfs":{"total":95995035648,"available":95706181632,"internally_reserved":0,"allocated":183005184,"data_stored":182698145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:13 2024","interfaces":[{"interface":"back","average":{"1min":2.9889999999999999,"5min":2.137,"15min":1.083},"min":{"1min":0.24199999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":27.966000000000001,"5min":73.078999999999994,"15min":73.078999999999994},"last":1.1559999999999999},{"interface":"front","average":{"1min":2.1110000000000002,"5min":1.9450000000000001,"15min":1.0149999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":26.097999999999999,"5min":73.412000000000006,"15min":73.412000000000006},"last":0.26400000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:09:13 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":0.80800000000000005,"15min":0.63400000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.0890000000000004,"5min":4.0890000000000004,"15min":4.0890000000000004},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.66700000000000004,"15min":0.58699999999999997},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.956,"5min":2.956,"15min":2.956},"last":0.38700000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:09:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.736,"15min":0.94199999999999995},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.8130000000000002,"5min":96.849999999999994,"15min":96.849999999999994},"last":1.202},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.8700000000000001,"15min":1.0089999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.032,"5min":96.143000000000001,"15min":96.143000000000001},"last":0.374}]},{"osd":3,"last update":"Mon Feb 5 08:09:13 2024","interfaces":[{"interface":"back","average":{"1min":3.2610000000000001,"5min":2.093,"15min":1.0649999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":26.077000000000002,"5min":73.602999999999994,"15min":73.602999999999994},"last":4.4580000000000002},{"interface":"front","average":{"1min":2.4820000000000002,"5min":2.0169999999999999,"15min":1.04},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":27.425000000000001,"5min":74.817999999999998,"15min":74.817999999999998},"last":1.5569999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:09:50 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.40999999999999998,"15min":0.38},"min":{"1min":0.28100000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63900000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.44500000000000001,"15min":0.40999999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.40999999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:09:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.45500000000000002,"15min":0.433},"min":{"1min":0.30499999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.61299999999999999,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.311},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.46200000000000002,"15min":0.43099999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.60499999999999998,"5min":0.747,"15min":0.747},"last":0.36199999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:09:33 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.29299999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.77100000000000002,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.5,"15min":0.502},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68600000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.39900000000000002}]}]},{"osd":5,"up_from":42,"seq":180388626513,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":278032,"kb_used_data":166344,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93467120,"statfs":{"total":95995035648,"available":95710330880,"internally_reserved":0,"allocated":170336256,"data_stored":170049624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":2.2519999999999998,"15min":1.1200000000000001},"min":{"1min":0.23200000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6150000000000002,"5min":143.06299999999999,"15min":143.06299999999999},"last":2.3919999999999999},{"interface":"front","average":{"1min":1.974,"5min":2.2829999999999999,"15min":1.119},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.3570000000000002,"5min":143.09700000000001,"15min":143.09700000000001},"last":0.23699999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.69699999999999995,"15min":0.61399999999999999},"min":{"1min":0.23200000000000001,"5min":0.215,"15min":0.215},"max":{"1min":2.286,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.755,"15min":0.61499999999999999},"min":{"1min":0.25800000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.2170000000000001,"5min":3.544,"15min":3.544},"last":0.497}]},{"osd":2,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.90400000000000003,"15min":0.68400000000000005},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.673,"5min":5.0049999999999999,"15min":5.0049999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":1.272,"5min":1.01,"15min":0.71599999999999997},"min":{"1min":0.28799999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.657,"5min":5.4089999999999998,"15min":5.4089999999999998},"last":0.32600000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":2.375,"15min":1.167},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":7.8410000000000002,"5min":143.03399999999999,"15min":143.03399999999999},"last":1.9690000000000001},{"interface":"front","average":{"1min":1.712,"5min":2.202,"15min":1.1299999999999999},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":9.1839999999999993,"5min":143.46799999999999,"15min":143.46799999999999},"last":1.571}]},{"osd":4,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.436,"15min":0.40899999999999997},"min":{"1min":0.25800000000000001,"5min":0.253,"15min":0.253},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.308},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45000000000000001,"15min":0.42799999999999999},"min":{"1min":0.30399999999999999,"5min":0.26900000000000002,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.42099999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:10:09 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.45300000000000001,"15min":0.44500000000000001},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.40000000000000002,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":0.71099999999999997,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.53600000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.47299999999999998},"min":{"1min":0.315,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.76600000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.34499999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.55500000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":278204416,"data_stored":278200344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292163584,"data_stored":292159512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":294256640,"data_stored":294256640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":362549248,"data_stored":362545176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":181207040,"data_stored":181207040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":153747456,"data_stored":153747456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":180879360,"data_stored":180879360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:10:17.391 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-05T08:10:17.392 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:10:17.392 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-02-05T08:10:17.392 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-02-05T08:10:17.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:10:17.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:17.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:17.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:17.494 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:17.547 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:17.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:17.582 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:10:17.582 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:10:17.582 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:10:17.582 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:10:17.582 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:10:17.582 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:10:17.583 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:10:17.583 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:10:17.583 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:10:17.583 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:10:17.583 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:10:17.583 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:10:17.583 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:10:17.583 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:10:17.583 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:10:17.583 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:10:17.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:17.672 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:17.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:17.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:17.774 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:17.799 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:17.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:17.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:17.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:17.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:17.949 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:17.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:18.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:10:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:10:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[30086]: 5.17 scrub starts 2024-02-05T08:10:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[30086]: 5.17 scrub ok 2024-02-05T08:10:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[30086]: 4.3 scrub starts 2024-02-05T08:10:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[30086]: 4.3 scrub ok 2024-02-05T08:10:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[30086]: 6.19 scrub starts 2024-02-05T08:10:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[30086]: 6.19 scrub ok 2024-02-05T08:10:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/846512330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:10:18.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[39278]: 5.17 scrub starts 2024-02-05T08:10:18.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[39278]: 5.17 scrub ok 2024-02-05T08:10:18.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[39278]: 4.3 scrub starts 2024-02-05T08:10:18.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[39278]: 4.3 scrub ok 2024-02-05T08:10:18.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[39278]: 6.19 scrub starts 2024-02-05T08:10:18.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[39278]: 6.19 scrub ok 2024-02-05T08:10:18.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:17 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/846512330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:10:18.049 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:17 smithi196 ceph-mon[33018]: 5.17 scrub starts 2024-02-05T08:10:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:17 smithi196 ceph-mon[33018]: 5.17 scrub ok 2024-02-05T08:10:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:17 smithi196 ceph-mon[33018]: 4.3 scrub starts 2024-02-05T08:10:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:17 smithi196 ceph-mon[33018]: 4.3 scrub ok 2024-02-05T08:10:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:17 smithi196 ceph-mon[33018]: 6.19 scrub starts 2024-02-05T08:10:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:17 smithi196 ceph-mon[33018]: 6.19 scrub ok 2024-02-05T08:10:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:17 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/846512330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:10:18.093 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:18.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:18.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:18.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:18.219 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:18.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:18.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:18.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:18.365 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:18.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:18.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:18.492 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:18.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:18.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:18.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:18.621 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:18.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:18.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:18.723 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:18.755 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:18.765 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:18.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:18.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:18.913 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:18.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:19.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:19.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[30086]: 6.6 scrub starts 2024-02-05T08:10:19.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[30086]: 6.6 scrub ok 2024-02-05T08:10:19.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:10:19.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:10:19.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[30086]: pgmap v217: 148 pgs: 148 active+clean; 497 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 728 op/s 2024-02-05T08:10:19.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[30086]: 7.8 scrub starts 2024-02-05T08:10:19.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[30086]: osdmap e149: 8 total, 8 up, 7 in 2024-02-05T08:10:19.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[30086]: 7.8 scrub ok 2024-02-05T08:10:19.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[39278]: 6.6 scrub starts 2024-02-05T08:10:19.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[39278]: 6.6 scrub ok 2024-02-05T08:10:19.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:10:19.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:10:19.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[39278]: pgmap v217: 148 pgs: 148 active+clean; 497 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 728 op/s 2024-02-05T08:10:19.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[39278]: 7.8 scrub starts 2024-02-05T08:10:19.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[39278]: osdmap e149: 8 total, 8 up, 7 in 2024-02-05T08:10:19.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:18 smithi134 ceph-mon[39278]: 7.8 scrub ok 2024-02-05T08:10:19.051 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:19.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:19.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:18 smithi196 ceph-mon[33018]: 6.6 scrub starts 2024-02-05T08:10:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:18 smithi196 ceph-mon[33018]: 6.6 scrub ok 2024-02-05T08:10:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:18 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:10:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:18 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:10:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:18 smithi196 ceph-mon[33018]: pgmap v217: 148 pgs: 148 active+clean; 497 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 728 op/s 2024-02-05T08:10:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:18 smithi196 ceph-mon[33018]: 7.8 scrub starts 2024-02-05T08:10:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:18 smithi196 ceph-mon[33018]: osdmap e149: 8 total, 8 up, 7 in 2024-02-05T08:10:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:18 smithi196 ceph-mon[33018]: 7.8 scrub ok 2024-02-05T08:10:19.153 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:19.199 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:19.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:19.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:19.326 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:19.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:19.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:19.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:19.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:19.469 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:19.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:19.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:19.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:19.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:19.773 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:19.798 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:19.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:19.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:19.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:19.943 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:19.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:19.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:20.020 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[30086]: 6.13 scrub starts 2024-02-05T08:10:20.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[30086]: 6.13 scrub ok 2024-02-05T08:10:20.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[30086]: 3.1f scrub starts 2024-02-05T08:10:20.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[30086]: 3.1f scrub ok 2024-02-05T08:10:20.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[30086]: 2.2 scrub starts 2024-02-05T08:10:20.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[30086]: 2.2 scrub ok 2024-02-05T08:10:20.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[30086]: 6.1 scrub starts 2024-02-05T08:10:20.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[30086]: 6.1 scrub ok 2024-02-05T08:10:20.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[30086]: 4.5 scrub starts 2024-02-05T08:10:20.021 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[30086]: 4.5 scrub ok 2024-02-05T08:10:20.021 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[39278]: 6.13 scrub starts 2024-02-05T08:10:20.022 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[39278]: 6.13 scrub ok 2024-02-05T08:10:20.022 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[39278]: 3.1f scrub starts 2024-02-05T08:10:20.022 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[39278]: 3.1f scrub ok 2024-02-05T08:10:20.022 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[39278]: 2.2 scrub starts 2024-02-05T08:10:20.022 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[39278]: 2.2 scrub ok 2024-02-05T08:10:20.022 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[39278]: 6.1 scrub starts 2024-02-05T08:10:20.022 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[39278]: 6.1 scrub ok 2024-02-05T08:10:20.022 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[39278]: 4.5 scrub starts 2024-02-05T08:10:20.022 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:19 smithi134 ceph-mon[39278]: 4.5 scrub ok 2024-02-05T08:10:20.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:19 smithi196 ceph-mon[33018]: 6.13 scrub starts 2024-02-05T08:10:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:19 smithi196 ceph-mon[33018]: 6.13 scrub ok 2024-02-05T08:10:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:19 smithi196 ceph-mon[33018]: 3.1f scrub starts 2024-02-05T08:10:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:19 smithi196 ceph-mon[33018]: 3.1f scrub ok 2024-02-05T08:10:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:19 smithi196 ceph-mon[33018]: 2.2 scrub starts 2024-02-05T08:10:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:19 smithi196 ceph-mon[33018]: 2.2 scrub ok 2024-02-05T08:10:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:19 smithi196 ceph-mon[33018]: 6.1 scrub starts 2024-02-05T08:10:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:19 smithi196 ceph-mon[33018]: 6.1 scrub ok 2024-02-05T08:10:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:19 smithi196 ceph-mon[33018]: 4.5 scrub starts 2024-02-05T08:10:20.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:19 smithi196 ceph-mon[33018]: 4.5 scrub ok 2024-02-05T08:10:20.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:20.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:20.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:20.211 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:20.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:20.313 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:20.342 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:10:20.353 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:20.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:20.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:20.456 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:20.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:20.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:20.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:20.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:20.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:20.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:20.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:20.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:20.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:20.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:20.878 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:20.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:20.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:21.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:21.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:21.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:21.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:20 smithi134 ceph-mon[30086]: 5.5 scrub starts 2024-02-05T08:10:21.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:20 smithi134 ceph-mon[30086]: 5.5 scrub ok 2024-02-05T08:10:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:20 smithi134 ceph-mon[30086]: pgmap v219: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 259 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-02-05T08:10:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:20 smithi134 ceph-mon[30086]: 4.1c scrub starts 2024-02-05T08:10:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:20 smithi134 ceph-mon[30086]: 4.1c scrub ok 2024-02-05T08:10:21.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:20 smithi134 ceph-mon[39278]: 5.5 scrub starts 2024-02-05T08:10:21.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:20 smithi134 ceph-mon[39278]: 5.5 scrub ok 2024-02-05T08:10:21.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:20 smithi134 ceph-mon[39278]: pgmap v219: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 259 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-02-05T08:10:21.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:20 smithi134 ceph-mon[39278]: 4.1c scrub starts 2024-02-05T08:10:21.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:20 smithi134 ceph-mon[39278]: 4.1c scrub ok 2024-02-05T08:10:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:20 smithi196 ceph-mon[33018]: 5.5 scrub starts 2024-02-05T08:10:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:20 smithi196 ceph-mon[33018]: 5.5 scrub ok 2024-02-05T08:10:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:20 smithi196 ceph-mon[33018]: pgmap v219: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 259 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-02-05T08:10:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:20 smithi196 ceph-mon[33018]: 4.1c scrub starts 2024-02-05T08:10:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:20 smithi196 ceph-mon[33018]: 4.1c scrub ok 2024-02-05T08:10:21.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:21.134 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:21.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:21.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:21.263 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:21.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:21.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:21.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:21.394 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:21.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:21.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:21.538 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:21.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:21.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:21.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:21.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:21.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:21.767 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:21.816 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:21.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:21.918 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:21.961 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:21.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[30086]: 5.f scrub starts 2024-02-05T08:10:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[30086]: 5.f scrub ok 2024-02-05T08:10:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:10:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:10:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:10:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:10:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[30086]: 6.8 scrub starts 2024-02-05T08:10:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[30086]: 6.8 scrub ok 2024-02-05T08:10:22.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[39278]: 5.f scrub starts 2024-02-05T08:10:22.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[39278]: 5.f scrub ok 2024-02-05T08:10:22.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:10:22.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:10:22.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:10:22.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:10:22.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[39278]: 6.8 scrub starts 2024-02-05T08:10:22.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:21 smithi134 ceph-mon[39278]: 6.8 scrub ok 2024-02-05T08:10:22.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:21 smithi196 ceph-mon[33018]: 5.f scrub starts 2024-02-05T08:10:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:21 smithi196 ceph-mon[33018]: 5.f scrub ok 2024-02-05T08:10:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:21 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:10:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:21 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:10:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:21 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:10:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:21 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:10:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:21 smithi196 ceph-mon[33018]: 6.8 scrub starts 2024-02-05T08:10:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:21 smithi196 ceph-mon[33018]: 6.8 scrub ok 2024-02-05T08:10:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:22.104 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:22.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:22.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:22.247 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:22.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:22.261 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Removed 109931 objects 2024-02-05T08:10:22.261 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Clean up completed and total clean up time :121.811 2024-02-05T08:10:22.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:22.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:22.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:22.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:22.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:22.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:22.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:22.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:22.627 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:22.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:22.728 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:22.751 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:22.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:22.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:22.838 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:22 smithi134 ceph-mon[30086]: 4.2 scrub starts 2024-02-05T08:10:22.838 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:22 smithi134 ceph-mon[30086]: 4.2 scrub ok 2024-02-05T08:10:22.838 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:22 smithi134 ceph-mon[30086]: pgmap v220: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 259 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 583 op/s 2024-02-05T08:10:22.838 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:22 smithi134 ceph-mon[30086]: osdmap e150: 8 total, 8 up, 7 in 2024-02-05T08:10:22.838 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:10:22.838 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:22 smithi134 ceph-mon[39278]: 4.2 scrub starts 2024-02-05T08:10:22.839 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:22 smithi134 ceph-mon[39278]: 4.2 scrub ok 2024-02-05T08:10:22.839 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:22 smithi134 ceph-mon[39278]: pgmap v220: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 259 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 583 op/s 2024-02-05T08:10:22.839 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:22 smithi134 ceph-mon[39278]: osdmap e150: 8 total, 8 up, 7 in 2024-02-05T08:10:22.839 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:10:22.853 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:22.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:22.878 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:22.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:22.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:23.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:23.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:23.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:23.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:22 smithi196 ceph-mon[33018]: 4.2 scrub starts 2024-02-05T08:10:23.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:22 smithi196 ceph-mon[33018]: 4.2 scrub ok 2024-02-05T08:10:23.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:22 smithi196 ceph-mon[33018]: pgmap v220: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 259 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 583 op/s 2024-02-05T08:10:23.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:22 smithi196 ceph-mon[33018]: osdmap e150: 8 total, 8 up, 7 in 2024-02-05T08:10:23.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:10:23.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:23.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:23.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:23.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:23.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:23.257 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:23.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:23.334 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:10:23.334 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:10:23.334 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:10:23.334 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:10:23.334 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.334 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:19156 7.f 7:f07da8a0:::benchmark_data_smithi134_83016_object19155:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:23.334 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.637666+0000", 2024-02-05T08:10:23.334 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 204.442661544, 2024-02-05T08:10:23.334 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14367015799999999, 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 19156 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.637666+0000", 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.335 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.336 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.336 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.637666+0000", 2024-02-05T08:10:23.336 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.336 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.336 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.336 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.336 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.637667+0000", 2024-02-05T08:10:23.336 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.048e-06 2024-02-05T08:10:23.336 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.336 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.336 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.336 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.637687+0000", 2024-02-05T08:10:23.337 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9792999999999998e-05 2024-02-05T08:10:23.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.337 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.337 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.337 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.637695+0000", 2024-02-05T08:10:23.337 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1300000000000001e-06 2024-02-05T08:10:23.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.337 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.337 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.337 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.637700+0000", 2024-02-05T08:10:23.337 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.322e-06 2024-02-05T08:10:23.337 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.338 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.338 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.338 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.640284+0000", 2024-02-05T08:10:23.338 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002583784 2024-02-05T08:10:23.338 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.338 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.338 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.338 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.640390+0000", 2024-02-05T08:10:23.338 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000106376 2024-02-05T08:10:23.338 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.338 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.338 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:10:23.339 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.640510+0000", 2024-02-05T08:10:23.339 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011983499999999999 2024-02-05T08:10:23.339 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.339 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.339 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.339 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.641696+0000", 2024-02-05T08:10:23.339 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011860659999999999 2024-02-05T08:10:23.339 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.339 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.339 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.339 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644278+0000", 2024-02-05T08:10:23.339 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0025819150000000002 2024-02-05T08:10:23.340 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.340 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.340 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.340 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.781111+0000", 2024-02-05T08:10:23.340 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13683325499999999 2024-02-05T08:10:23.340 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.340 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.340 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.340 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.781296+0000", 2024-02-05T08:10:23.340 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018474999999999999 2024-02-05T08:10:23.340 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.340 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.340 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.341 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.781336+0000", 2024-02-05T08:10:23.341 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9884e-05 2024-02-05T08:10:23.341 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.341 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.341 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.341 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.341 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.341 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20666 7.7 7:e8712a05:::benchmark_data_smithi134_83016_object20665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:23.341 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.663704+0000", 2024-02-05T08:10:23.341 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 203.41662334, 2024-02-05T08:10:23.341 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15447154699999999, 2024-02-05T08:10:23.341 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.341 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.342 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.342 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.342 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.342 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20666 2024-02-05T08:10:23.342 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.342 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.342 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.342 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.342 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.663704+0000", 2024-02-05T08:10:23.342 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.342 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.342 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.342 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.663704+0000", 2024-02-05T08:10:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.663705+0000", 2024-02-05T08:10:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.54e-07 2024-02-05T08:10:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.663727+0000", 2024-02-05T08:10:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2719000000000001e-05 2024-02-05T08:10:23.343 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.344 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.344 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.344 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.663737+0000", 2024-02-05T08:10:23.344 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.312e-06 2024-02-05T08:10:23.344 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.344 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.344 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.344 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.663744+0000", 2024-02-05T08:10:23.344 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5610000000000001e-06 2024-02-05T08:10:23.344 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.344 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.345 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.345 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.665902+0000", 2024-02-05T08:10:23.345 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00215796 2024-02-05T08:10:23.345 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.345 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.345 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.345 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666007+0000", 2024-02-05T08:10:23.345 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010508 2024-02-05T08:10:23.345 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.345 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.345 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:10:23.345 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666127+0000", 2024-02-05T08:10:23.345 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011996400000000001 2024-02-05T08:10:23.346 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.346 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.346 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.346 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669365+0000", 2024-02-05T08:10:23.346 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032372899999999999 2024-02-05T08:10:23.346 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.346 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.346 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.346 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670222+0000", 2024-02-05T08:10:23.346 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00085750099999999999 2024-02-05T08:10:23.346 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.346 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.346 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.347 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818094+0000", 2024-02-05T08:10:23.347 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14787227999999999 2024-02-05T08:10:23.347 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.347 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.347 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.347 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818142+0000", 2024-02-05T08:10:23.347 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7299999999999998e-05 2024-02-05T08:10:23.347 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.347 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.347 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.347 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818175+0000", 2024-02-05T08:10:23.348 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3825999999999998e-05 2024-02-05T08:10:23.348 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.348 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.349 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.349 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.349 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.349 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20673 7.f 7:f0d5ee1c:::benchmark_data_smithi134_83016_object20672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:23.349 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.666508+0000", 2024-02-05T08:10:23.349 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 203.41381969400001, 2024-02-05T08:10:23.349 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.149946568, 2024-02-05T08:10:23.349 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.349 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.349 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.349 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.349 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20673 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666508+0000", 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666508+0000", 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.350 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666509+0000", 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5600000000000004e-07 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666532+0000", 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3519e-05 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666543+0000", 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0713000000000001e-05 2024-02-05T08:10:23.351 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.352 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.352 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.352 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.666550+0000", 2024-02-05T08:10:23.352 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0779999999999998e-06 2024-02-05T08:10:23.352 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.352 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.352 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.352 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667364+0000", 2024-02-05T08:10:23.352 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000814506 2024-02-05T08:10:23.352 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.352 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.352 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.353 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667478+0000", 2024-02-05T08:10:23.353 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011346899999999999 2024-02-05T08:10:23.353 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.353 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.353 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:10:23.353 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667603+0000", 2024-02-05T08:10:23.353 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012499200000000001 2024-02-05T08:10:23.353 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.353 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.353 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.353 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669370+0000", 2024-02-05T08:10:23.353 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017667609999999999 2024-02-05T08:10:23.353 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816310+0000", 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14694006400000001 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816351+0000", 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1686e-05 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816413+0000", 2024-02-05T08:10:23.354 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2018000000000002e-05 2024-02-05T08:10:23.355 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.355 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.355 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.355 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816454+0000", 2024-02-05T08:10:23.355 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0806000000000002e-05 2024-02-05T08:10:23.355 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.355 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.355 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.355 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.355 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.355 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20676 7.6 7:66971f6f:::benchmark_data_smithi134_83016_object20675:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:23.355 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.667124+0000", 2024-02-05T08:10:23.355 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 203.41320347199999, 2024-02-05T08:10:23.356 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14974601900000001, 2024-02-05T08:10:23.356 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.356 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.356 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.356 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.356 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.356 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20676 2024-02-05T08:10:23.356 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.356 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.356 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.356 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.356 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667124+0000", 2024-02-05T08:10:23.356 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667124+0000", 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667125+0000", 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8199999999999999e-07 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.357 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667145+0000", 2024-02-05T08:10:23.358 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0829999999999999e-05 2024-02-05T08:10:23.358 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.358 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.358 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.358 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667155+0000", 2024-02-05T08:10:23.358 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6989999999999998e-06 2024-02-05T08:10:23.358 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.358 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.358 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.358 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.667162+0000", 2024-02-05T08:10:23.358 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0380000000000002e-06 2024-02-05T08:10:23.358 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.358 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668327+0000", 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011653309999999999 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668440+0000", 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011294900000000001 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.668571+0000", 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000130801 2024-02-05T08:10:23.359 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.360 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.360 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.360 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672190+0000", 2024-02-05T08:10:23.360 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036185119999999999 2024-02-05T08:10:23.360 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.360 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.360 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.360 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816650+0000", 2024-02-05T08:10:23.360 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14446054799999999 2024-02-05T08:10:23.360 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.360 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.360 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.361 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816783+0000", 2024-02-05T08:10:23.361 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013320000000000001 2024-02-05T08:10:23.361 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.361 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.361 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.361 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816835+0000", 2024-02-05T08:10:23.361 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1891000000000002e-05 2024-02-05T08:10:23.361 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.361 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.361 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.361 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816870+0000", 2024-02-05T08:10:23.361 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4437999999999998e-05 2024-02-05T08:10:23.361 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20678 7.6 7:6bafa7b0:::benchmark_data_smithi134_83016_object20677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.669255+0000", 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 203.41107246499999, 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.147953636, 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.362 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20678 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669255+0000", 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669255+0000", 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.363 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669256+0000", 2024-02-05T08:10:23.364 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.212e-06 2024-02-05T08:10:23.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.364 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.364 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.364 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669276+0000", 2024-02-05T08:10:23.364 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9670999999999999e-05 2024-02-05T08:10:23.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.364 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.364 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.364 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669286+0000", 2024-02-05T08:10:23.364 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0563e-05 2024-02-05T08:10:23.364 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.364 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669294+0000", 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6029999999999999e-06 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669393+0000", 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9204999999999996e-05 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.669513+0000", 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012022 2024-02-05T08:10:23.365 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671056+0000", 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015427990000000001 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816671+0000", 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14561491500000001 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816705+0000", 2024-02-05T08:10:23.366 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4248000000000001e-05 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817079+0000", 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00037375199999999998 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817168+0000", 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8786e-05 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817208+0000", 2024-02-05T08:10:23.367 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0661999999999998e-05 2024-02-05T08:10:23.368 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.368 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.368 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.368 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.368 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.368 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20681 7.7 7:ea75bb27:::benchmark_data_smithi134_83016_object20680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:23.368 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.670167+0000", 2024-02-05T08:10:23.368 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 203.410160491, 2024-02-05T08:10:23.368 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.148127287, 2024-02-05T08:10:23.368 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.368 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.368 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.368 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.369 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.369 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20681 2024-02-05T08:10:23.369 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.369 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.369 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.369 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.369 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670167+0000", 2024-02-05T08:10:23.369 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.369 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.369 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.369 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.369 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670167+0000", 2024-02-05T08:10:23.369 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670168+0000", 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7599999999999996e-07 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670195+0000", 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7202000000000001e-05 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.370 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670204+0000", 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9910000000000008e-06 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670529+0000", 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00032530200000000002 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670654+0000", 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012481600000000001 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.371 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670757+0000", 2024-02-05T08:10:23.372 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010313 2024-02-05T08:10:23.372 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.372 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.372 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:10:23.372 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670878+0000", 2024-02-05T08:10:23.372 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000120541 2024-02-05T08:10:23.372 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.372 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.372 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.372 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818057+0000", 2024-02-05T08:10:23.372 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14717940500000001 2024-02-05T08:10:23.372 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.372 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818068+0000", 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0699e-05 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818216+0000", 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000148088 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818257+0000", 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1600000000000002e-05 2024-02-05T08:10:23.373 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818294+0000", 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6637000000000002e-05 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20682 7.7 7:ec8770ce:::benchmark_data_smithi134_83016_object20681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.670551+0000", 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 203.40977659500001, 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.153080151, 2024-02-05T08:10:23.374 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20682 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670551+0000", 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.375 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670551+0000", 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670551+0000", 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.99e-07 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670579+0000", 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7994999999999999e-05 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.376 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670589+0000", 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4080000000000004e-06 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670596+0000", 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8020000000000003e-06 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816434+0000", 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14583796900000001 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.377 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816554+0000", 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000120381 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816699+0000", 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014480800000000001 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.820595+0000", 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0038963589999999998 2024-02-05T08:10:23.378 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.379 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.379 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.379 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.822835+0000", 2024-02-05T08:10:23.379 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0022401370000000001 2024-02-05T08:10:23.379 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.379 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.379 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.379 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.823546+0000", 2024-02-05T08:10:23.379 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071045400000000001 2024-02-05T08:10:23.379 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.379 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.379 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.379 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.823595+0000", 2024-02-05T08:10:23.380 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9200000000000003e-05 2024-02-05T08:10:23.380 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.380 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.380 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.380 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.823631+0000", 2024-02-05T08:10:23.380 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5939000000000003e-05 2024-02-05T08:10:23.380 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.380 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.380 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.380 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.380 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.380 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:20673 7.f e104/95 ondisk, result = 0)", 2024-02-05T08:10:23.380 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.670931+0000", 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 203.409396168, 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.145386667, 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670931+0000", 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670931+0000", 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.381 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.382 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.382 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.382 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670932+0000", 2024-02-05T08:10:23.382 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7300000000000004e-07 2024-02-05T08:10:23.382 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.382 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.382 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.382 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670936+0000", 2024-02-05T08:10:23.382 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3939999999999998e-06 2024-02-05T08:10:23.382 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670947+0000", 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0305999999999999e-05 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.670953+0000", 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6329999999999999e-06 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816293+0000", 2024-02-05T08:10:23.383 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.145339473 2024-02-05T08:10:23.384 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.384 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.384 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.384 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816303+0000", 2024-02-05T08:10:23.384 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.012e-05 2024-02-05T08:10:23.384 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.384 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.384 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.384 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816318+0000", 2024-02-05T08:10:23.384 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4768e-05 2024-02-05T08:10:23.384 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.384 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.385 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.385 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.385 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.385 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:20676 7.6 e104/95 ondisk, result = 0)", 2024-02-05T08:10:23.385 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.671365+0000", 2024-02-05T08:10:23.385 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 203.40896268399999, 2024-02-05T08:10:23.385 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.145297906, 2024-02-05T08:10:23.385 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.385 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:10:23.385 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.385 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.385 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.386 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671365+0000", 2024-02-05T08:10:23.386 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.386 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.386 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.386 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.386 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671365+0000", 2024-02-05T08:10:23.386 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.386 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.386 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.386 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.386 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671365+0000", 2024-02-05T08:10:23.386 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1399999999999997e-07 2024-02-05T08:10:23.386 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.387 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.387 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.387 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671368+0000", 2024-02-05T08:10:23.387 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.309e-06 2024-02-05T08:10:23.387 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.387 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.387 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.387 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671377+0000", 2024-02-05T08:10:23.387 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4069999999999993e-06 2024-02-05T08:10:23.387 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.387 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.387 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.387 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671382+0000", 2024-02-05T08:10:23.388 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6400000000000002e-06 2024-02-05T08:10:23.388 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.388 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.388 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.388 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816635+0000", 2024-02-05T08:10:23.388 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14525242199999999 2024-02-05T08:10:23.388 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.388 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.388 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.388 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816644+0000", 2024-02-05T08:10:23.388 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7809999999999994e-06 2024-02-05T08:10:23.388 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.389 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.389 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.389 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816662+0000", 2024-02-05T08:10:23.389 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8833000000000001e-05 2024-02-05T08:10:23.389 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.389 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.389 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.389 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.389 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.389 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:20673 7.f e104/95 ondisk, result = 0)", 2024-02-05T08:10:23.389 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.671934+0000", 2024-02-05T08:10:23.389 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 203.40839346300001, 2024-02-05T08:10:23.389 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.144526025, 2024-02-05T08:10:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:10:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671934+0000", 2024-02-05T08:10:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671934+0000", 2024-02-05T08:10:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.390 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671934+0000", 2024-02-05T08:10:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4899999999999995e-07 2024-02-05T08:10:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671938+0000", 2024-02-05T08:10:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6330000000000002e-06 2024-02-05T08:10:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.391 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671950+0000", 2024-02-05T08:10:23.392 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2138e-05 2024-02-05T08:10:23.392 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.392 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.392 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.392 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.671957+0000", 2024-02-05T08:10:23.392 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5780000000000004e-06 2024-02-05T08:10:23.392 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.392 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.392 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.392 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816340+0000", 2024-02-05T08:10:23.392 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14438348600000001 2024-02-05T08:10:23.392 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.392 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.393 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.393 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816347+0000", 2024-02-05T08:10:23.393 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4640000000000003e-06 2024-02-05T08:10:23.393 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.393 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.393 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.393 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816460+0000", 2024-02-05T08:10:23.393 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000113077 2024-02-05T08:10:23.393 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.393 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.393 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.393 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.394 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.394 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:20678 7.6 e104/95 ondisk, result = 0)", 2024-02-05T08:10:23.394 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.672005+0000", 2024-02-05T08:10:23.394 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 203.40832180500001, 2024-02-05T08:10:23.394 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14470676399999999, 2024-02-05T08:10:23.394 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.394 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:10:23.394 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.394 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.394 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.394 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672005+0000", 2024-02-05T08:10:23.394 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.394 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672005+0000", 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672006+0000", 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7399999999999998e-07 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672009+0000", 2024-02-05T08:10:23.395 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1580000000000001e-06 2024-02-05T08:10:23.396 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.396 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.396 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.396 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672017+0000", 2024-02-05T08:10:23.396 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0749999999999998e-06 2024-02-05T08:10:23.396 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.396 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.396 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.396 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672022+0000", 2024-02-05T08:10:23.396 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3240000000000002e-06 2024-02-05T08:10:23.396 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.396 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.396 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.397 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816694+0000", 2024-02-05T08:10:23.397 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.144671833 2024-02-05T08:10:23.397 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.397 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.397 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.397 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816700+0000", 2024-02-05T08:10:23.397 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1319999999999999e-06 2024-02-05T08:10:23.397 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.397 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.397 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.397 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816712+0000", 2024-02-05T08:10:23.397 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1768e-05 2024-02-05T08:10:23.397 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.398 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.398 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.398 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.398 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.398 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:20676 7.6 e104/95 ondisk, result = 0)", 2024-02-05T08:10:23.398 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.672729+0000", 2024-02-05T08:10:23.398 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 203.40759788700001, 2024-02-05T08:10:23.398 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14414776600000001, 2024-02-05T08:10:23.398 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.398 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:10:23.398 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.398 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.398 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.399 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672729+0000", 2024-02-05T08:10:23.399 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.399 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.399 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.399 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.399 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672729+0000", 2024-02-05T08:10:23.399 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.399 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.399 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.399 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.399 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672730+0000", 2024-02-05T08:10:23.400 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3499999999999996e-07 2024-02-05T08:10:23.400 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.400 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.400 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.400 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672734+0000", 2024-02-05T08:10:23.400 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.963e-06 2024-02-05T08:10:23.400 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.400 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.400 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.400 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816648+0000", 2024-02-05T08:10:23.400 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14391443000000001 2024-02-05T08:10:23.400 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.400 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.401 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.401 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816658+0000", 2024-02-05T08:10:23.401 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.251e-06 2024-02-05T08:10:23.401 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.401 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.401 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.401 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816771+0000", 2024-02-05T08:10:23.401 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011333200000000001 2024-02-05T08:10:23.401 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.401 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.401 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.401 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816778+0000", 2024-02-05T08:10:23.401 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0940000000000002e-06 2024-02-05T08:10:23.402 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.402 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.402 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.402 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816877+0000", 2024-02-05T08:10:23.402 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9061000000000006e-05 2024-02-05T08:10:23.402 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.402 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.402 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.402 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.402 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.402 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:20686 7.5 e104/95 7:ae49ae8c:::benchmark_data_smithi134_83016_object20685:head v 104'1244, mlcod=104'1242)", 2024-02-05T08:10:23.402 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.673008+0000", 2024-02-05T08:10:23.402 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 203.40731934799999, 2024-02-05T08:10:23.403 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14735541999999999, 2024-02-05T08:10:23.403 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.403 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.403 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.403 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.403 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.403 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673008+0000", 2024-02-05T08:10:23.403 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.403 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.403 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.403 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.403 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673008+0000", 2024-02-05T08:10:23.403 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673009+0000", 2024-02-05T08:10:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8899999999999999e-07 2024-02-05T08:10:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.673168+0000", 2024-02-05T08:10:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015927700000000001 2024-02-05T08:10:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.404 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816728+0000", 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14355969099999999 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.816734+0000", 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8020000000000003e-06 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817049+0000", 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031493299999999999 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.405 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.817070+0000", 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1016999999999999e-05 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.820256+0000", 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031859319999999998 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.820363+0000", 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107079 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.406 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:99777 7.7 7:e173e941:::benchmark_data_smithi134_83016_object99776:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:08.934031+0000", 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 134.14629614500001, 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.147308359, 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 99777 2024-02-05T08:10:23.407 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.934031+0000", 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.934031+0000", 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.408 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.934032+0000", 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.189e-06 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.934054+0000", 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1971e-05 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.934066+0000", 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2087e-05 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.409 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.410 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.934133+0000", 2024-02-05T08:10:23.410 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6847000000000006e-05 2024-02-05T08:10:23.410 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.410 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.410 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.410 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.934253+0000", 2024-02-05T08:10:23.410 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000119967 2024-02-05T08:10:23.410 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.410 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.410 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.410 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.934364+0000", 2024-02-05T08:10:23.410 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011077399999999999 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.934495+0000", 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000131006 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.938736+0000", 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042408680000000001 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.411 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947096+0000", 2024-02-05T08:10:23.412 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0083597610000000003 2024-02-05T08:10:23.412 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.412 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.412 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.412 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.080506+0000", 2024-02-05T08:10:23.412 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.133410467 2024-02-05T08:10:23.412 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.412 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.412 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.412 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.081299+0000", 2024-02-05T08:10:23.412 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00079315400000000002 2024-02-05T08:10:23.412 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.412 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.081339+0000", 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0268000000000001e-05 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:99782 7.7 7:e2f85fa1:::benchmark_data_smithi134_83016_object99781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:08.939288+0000", 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 134.14103932200001, 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.1428256, 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.413 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 99782 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939288+0000", 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939288+0000", 2024-02-05T08:10:23.414 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939289+0000", 2024-02-05T08:10:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4589999999999999e-06 2024-02-05T08:10:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939352+0000", 2024-02-05T08:10:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2334000000000003e-05 2024-02-05T08:10:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.415 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.416 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939364+0000", 2024-02-05T08:10:23.416 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2267e-05 2024-02-05T08:10:23.416 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.416 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.416 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.416 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939372+0000", 2024-02-05T08:10:23.416 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.661e-06 2024-02-05T08:10:23.416 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.416 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.416 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.416 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939422+0000", 2024-02-05T08:10:23.416 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0458999999999999e-05 2024-02-05T08:10:23.416 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939545+0000", 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012283500000000001 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939814+0000", 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00026911699999999998 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947136+0000", 2024-02-05T08:10:23.417 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0073221240000000002 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.081349+0000", 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13421325000000001 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.081757+0000", 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00040726100000000002 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.082078+0000", 2024-02-05T08:10:23.418 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00032154699999999999 2024-02-05T08:10:23.419 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.419 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.419 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.419 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.082114+0000", 2024-02-05T08:10:23.419 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5286000000000001e-05 2024-02-05T08:10:23.419 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.419 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.419 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.419 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.419 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.419 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:99789 7.6 7:6a372906:::benchmark_data_smithi134_83016_object99788:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:10:23.419 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:08.947868+0000", 2024-02-05T08:10:23.419 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 134.13245904600001, 2024-02-05T08:10:23.420 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.146593579, 2024-02-05T08:10:23.420 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.420 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.420 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.420 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.420 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.420 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 99789 2024-02-05T08:10:23.420 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.420 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.420 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.420 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.420 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947868+0000", 2024-02-05T08:10:23.420 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947868+0000", 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947869+0000", 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.268e-06 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.421 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947899+0000", 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9490999999999999e-05 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947911+0000", 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2364000000000001e-05 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947920+0000", 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2600000000000005e-06 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.422 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947999+0000", 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9554000000000004e-05 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.948143+0000", 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014341399999999999 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.077622+0000", 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12947962700000001 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.423 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.082325+0000", 2024-02-05T08:10:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0047027019999999996 2024-02-05T08:10:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.082516+0000", 2024-02-05T08:10:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019123299999999999 2024-02-05T08:10:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.094074+0000", 2024-02-05T08:10:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.011557705999999999 2024-02-05T08:10:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.094420+0000", 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00034639499999999999 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.094462+0000", 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1564999999999999e-05 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.425 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106441 7.7 7:e75e735e:::benchmark_data_smithi134_83016_object106440:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:10:23.426 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.244918+0000", 2024-02-05T08:10:23.426 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 126.835409423, 2024-02-05T08:10:23.426 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15496602500000001, 2024-02-05T08:10:23.426 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.426 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.426 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.426 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.426 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.426 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106441 2024-02-05T08:10:23.426 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.426 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.426 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.426 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.427 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.244918+0000", 2024-02-05T08:10:23.427 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.427 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.427 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.244918+0000", 2024-02-05T08:10:23.427 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.427 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.427 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.244919+0000", 2024-02-05T08:10:23.427 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9800000000000003e-07 2024-02-05T08:10:23.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.428 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.428 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.428 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.244939+0000", 2024-02-05T08:10:23.428 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0426999999999999e-05 2024-02-05T08:10:23.428 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.428 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.428 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.428 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.244948+0000", 2024-02-05T08:10:23.428 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6000000000000007e-06 2024-02-05T08:10:23.428 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.429 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.429 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.429 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.244954+0000", 2024-02-05T08:10:23.429 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5529999999999998e-06 2024-02-05T08:10:23.429 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.429 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.429 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.429 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.251670+0000", 2024-02-05T08:10:23.429 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0067162419999999999 2024-02-05T08:10:23.429 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.430 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.430 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.430 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.251790+0000", 2024-02-05T08:10:23.430 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000119448 2024-02-05T08:10:23.430 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.430 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.430 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:10:23.430 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.251936+0000", 2024-02-05T08:10:23.430 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000145866 2024-02-05T08:10:23.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.431 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.431 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.258959+0000", 2024-02-05T08:10:23.431 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0070230129999999998 2024-02-05T08:10:23.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.431 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.431 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.268296+0000", 2024-02-05T08:10:23.431 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0093372379999999994 2024-02-05T08:10:23.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.432 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.432 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.278343+0000", 2024-02-05T08:10:23.432 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010046878 2024-02-05T08:10:23.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.432 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.432 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.399838+0000", 2024-02-05T08:10:23.432 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.121495119 2024-02-05T08:10:23.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.432 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.432 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.399884+0000", 2024-02-05T08:10:23.433 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5942999999999997e-05 2024-02-05T08:10:23.433 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.433 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.433 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.433 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.433 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.433 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106444 7.7 7:e2b925a5:::benchmark_data_smithi134_83016_object106443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:10:23.433 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.251941+0000", 2024-02-05T08:10:23.433 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 126.828386736, 2024-02-05T08:10:23.433 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14808711399999999, 2024-02-05T08:10:23.433 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.433 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.433 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106444 2024-02-05T08:10:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.251941+0000", 2024-02-05T08:10:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.434 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.251941+0000", 2024-02-05T08:10:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.251942+0000", 2024-02-05T08:10:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4890000000000001e-06 2024-02-05T08:10:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.251966+0000", 2024-02-05T08:10:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4026999999999999e-05 2024-02-05T08:10:23.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.251976+0000", 2024-02-05T08:10:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4800000000000007e-06 2024-02-05T08:10:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.251983+0000", 2024-02-05T08:10:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8019999999999992e-06 2024-02-05T08:10:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.436 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.254627+0000", 2024-02-05T08:10:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0026436229999999999 2024-02-05T08:10:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.254760+0000", 2024-02-05T08:10:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013303100000000001 2024-02-05T08:10:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:10:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.254906+0000", 2024-02-05T08:10:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014566899999999999 2024-02-05T08:10:23.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.258963+0000", 2024-02-05T08:10:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.004057331 2024-02-05T08:10:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.269218+0000", 2024-02-05T08:10:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010254694 2024-02-05T08:10:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.438 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.399937+0000", 2024-02-05T08:10:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.130718959 2024-02-05T08:10:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.399993+0000", 2024-02-05T08:10:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6056e-05 2024-02-05T08:10:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.400028+0000", 2024-02-05T08:10:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4953000000000002e-05 2024-02-05T08:10:23.439 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106446 7.f 7:f6742bdf:::benchmark_data_smithi134_83016_object106445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:10:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.254780+0000", 2024-02-05T08:10:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 126.825546813, 2024-02-05T08:10:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14448743, 2024-02-05T08:10:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.440 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106446 2024-02-05T08:10:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.254780+0000", 2024-02-05T08:10:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.254780+0000", 2024-02-05T08:10:23.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.442 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.254782+0000", 2024-02-05T08:10:23.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.39e-06 2024-02-05T08:10:23.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.442 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.254816+0000", 2024-02-05T08:10:23.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4103000000000002e-05 2024-02-05T08:10:23.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.443 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.254827+0000", 2024-02-05T08:10:23.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1185e-05 2024-02-05T08:10:23.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.443 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.443 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.254836+0000", 2024-02-05T08:10:23.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4449999999999996e-06 2024-02-05T08:10:23.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.443 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.443 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.257584+0000", 2024-02-05T08:10:23.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027481139999999999 2024-02-05T08:10:23.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.444 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.444 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.257711+0000", 2024-02-05T08:10:23.444 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012692899999999999 2024-02-05T08:10:23.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.444 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:10:23.444 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.257844+0000", 2024-02-05T08:10:23.444 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013386 2024-02-05T08:10:23.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.444 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.265106+0000", 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.007261216 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.266174+0000", 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010680469999999999 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.399171+0000", 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13299685 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.446 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.446 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.399232+0000", 2024-02-05T08:10:23.446 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1360999999999997e-05 2024-02-05T08:10:23.446 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.446 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.446 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.399268+0000", 2024-02-05T08:10:23.446 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5930000000000003e-05 2024-02-05T08:10:23.446 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.446 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.446 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.446 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106465 7.f 7:f97ecec6:::benchmark_data_smithi134_83016_object106464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.273146+0000", 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 126.807180927, 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14578748699999999, 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106465 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.447 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:23.448 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.273146+0000", 2024-02-05T08:10:23.448 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.448 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.448 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.448 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:23.448 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.273146+0000", 2024-02-05T08:10:23.448 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:23.448 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.448 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.448 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.273147+0000", 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8599999999999998e-07 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.273167+0000", 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9639999999999998e-05 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.273175+0000", 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0239999999999997e-06 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.450 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:23.450 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.273189+0000", 2024-02-05T08:10:23.450 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4674999999999999e-05 2024-02-05T08:10:23.450 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.450 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.450 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:23.450 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.273251+0000", 2024-02-05T08:10:23.450 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1348999999999998e-05 2024-02-05T08:10:23.450 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.450 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.450 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:23.450 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.273377+0000", 2024-02-05T08:10:23.450 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012581300000000001 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.273572+0000", 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019583400000000001 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.276297+0000", 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027243229999999998 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:23.451 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.400038+0000", 2024-02-05T08:10:23.452 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12374173099999999 2024-02-05T08:10:23.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.452 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.452 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:23.452 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.418805+0000", 2024-02-05T08:10:23.452 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.018766409000000001 2024-02-05T08:10:23.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.452 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.452 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:23.452 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.418896+0000", 2024-02-05T08:10:23.452 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1170999999999998e-05 2024-02-05T08:10:23.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:23.452 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:23.453 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:23.453 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.418934+0000", 2024-02-05T08:10:23.453 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7832e-05 2024-02-05T08:10:23.453 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.453 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.453 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.453 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:23.453 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:23.453 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:10:23.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:23.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:23.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:23.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:10:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:10:23.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:23.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:23.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:23.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:23.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:23.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:23.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:23.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:23.783 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:10:23.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:23.865 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:23.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:23.966 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:23.991 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:23.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:24.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:23 smithi134 ceph-mon[30086]: 7.e scrub starts 2024-02-05T08:10:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:23 smithi134 ceph-mon[30086]: 7.e scrub ok 2024-02-05T08:10:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:23 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:23 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3673893565' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:10:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:23 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:10:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:23 smithi134 ceph-mon[39278]: 7.e scrub starts 2024-02-05T08:10:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:23 smithi134 ceph-mon[39278]: 7.e scrub ok 2024-02-05T08:10:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:23 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:23 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3673893565' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:10:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:23 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:10:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:23 smithi196 ceph-mon[33018]: 7.e scrub starts 2024-02-05T08:10:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:23 smithi196 ceph-mon[33018]: 7.e scrub ok 2024-02-05T08:10:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:23 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:23 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3673893565' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:10:24.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:23 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:10:24.092 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:24.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:24.116 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:24.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:24.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:24.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:24.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:24.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:24.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:24.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:24.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T08:10:24.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:24.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:24.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:24.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:24.603 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:24.628 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:24.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:24.634 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:24.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:24.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:24.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:24.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:24.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:24.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:24.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:24.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:25.012 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:25.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:25.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[39278]: pgmap v222: 148 pgs: 148 active+clean; 582 KiB data, 1002 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-02-05T08:10:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[39278]: 3.1c scrub starts 2024-02-05T08:10:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[39278]: 3.1c scrub ok 2024-02-05T08:10:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:10:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:10:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:10:25.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[39278]: osdmap e151: 8 total, 8 up, 7 in 2024-02-05T08:10:25.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[39278]: 5.1a scrub starts 2024-02-05T08:10:25.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[39278]: 5.1a scrub ok 2024-02-05T08:10:25.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[30086]: pgmap v222: 148 pgs: 148 active+clean; 582 KiB data, 1002 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-02-05T08:10:25.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[30086]: 3.1c scrub starts 2024-02-05T08:10:25.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[30086]: 3.1c scrub ok 2024-02-05T08:10:25.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:10:25.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:10:25.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:10:25.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[30086]: osdmap e151: 8 total, 8 up, 7 in 2024-02-05T08:10:25.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[30086]: 5.1a scrub starts 2024-02-05T08:10:25.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:24 smithi134 ceph-mon[30086]: 5.1a scrub ok 2024-02-05T08:10:25.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:24 smithi196 ceph-mon[33018]: pgmap v222: 148 pgs: 148 active+clean; 582 KiB data, 1002 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s 2024-02-05T08:10:25.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:24 smithi196 ceph-mon[33018]: 3.1c scrub starts 2024-02-05T08:10:25.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:24 smithi196 ceph-mon[33018]: 3.1c scrub ok 2024-02-05T08:10:25.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:24 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:10:25.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:24 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:10:25.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:24 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:10:25.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:24 smithi196 ceph-mon[33018]: osdmap e151: 8 total, 8 up, 7 in 2024-02-05T08:10:25.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:24 smithi196 ceph-mon[33018]: 5.1a scrub starts 2024-02-05T08:10:25.041 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:24 smithi196 ceph-mon[33018]: 5.1a scrub ok 2024-02-05T08:10:25.112 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:25.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:25.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:25.142 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:25.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:25.153 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:25.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:25.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:25.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:25.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:25.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:25.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:25.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:25.497 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:25.522 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:25.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:25.623 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:25.647 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:25.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:25.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:25.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:25.773 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:25.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:25.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:25.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:25.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:26.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:26.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:26.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:25 smithi196 ceph-mon[33018]: 4.7 scrub starts 2024-02-05T08:10:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:25 smithi196 ceph-mon[33018]: 4.7 scrub ok 2024-02-05T08:10:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:25 smithi196 ceph-mon[33018]: 3.a scrub starts 2024-02-05T08:10:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:25 smithi196 ceph-mon[33018]: 3.a scrub ok 2024-02-05T08:10:26.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:26.155 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:26.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:26.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:26.257 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:26.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:25 smithi134 ceph-mon[30086]: 4.7 scrub starts 2024-02-05T08:10:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:25 smithi134 ceph-mon[30086]: 4.7 scrub ok 2024-02-05T08:10:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:25 smithi134 ceph-mon[30086]: 3.a scrub starts 2024-02-05T08:10:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:25 smithi134 ceph-mon[30086]: 3.a scrub ok 2024-02-05T08:10:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:25 smithi134 ceph-mon[39278]: 4.7 scrub starts 2024-02-05T08:10:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:25 smithi134 ceph-mon[39278]: 4.7 scrub ok 2024-02-05T08:10:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:25 smithi134 ceph-mon[39278]: 3.a scrub starts 2024-02-05T08:10:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:25 smithi134 ceph-mon[39278]: 3.a scrub ok 2024-02-05T08:10:26.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:26.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:26.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:26.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:26.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:26.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:26.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:26.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:26.543 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:26.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:26.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:26.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:26.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:26.671 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:26.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:26.772 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:26.797 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:26.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:26.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:26.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:26 smithi134 ceph-mon[30086]: 5.12 scrub starts 2024-02-05T08:10:26.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:26 smithi134 ceph-mon[30086]: 5.12 scrub ok 2024-02-05T08:10:26.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:26 smithi134 ceph-mon[30086]: pgmap v224: 148 pgs: 148 active+clean; 582 KiB data, 1002 MiB used, 625 GiB / 626 GiB avail; 651 B/s rd, 0 B/s wr, 942 op/s 2024-02-05T08:10:26.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:26 smithi134 ceph-mon[30086]: 6.1a scrub starts 2024-02-05T08:10:26.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:26 smithi134 ceph-mon[30086]: 6.1a scrub ok 2024-02-05T08:10:26.876 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:26 smithi134 ceph-mon[39278]: 5.12 scrub starts 2024-02-05T08:10:26.876 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:26 smithi134 ceph-mon[39278]: 5.12 scrub ok 2024-02-05T08:10:26.876 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:26 smithi134 ceph-mon[39278]: pgmap v224: 148 pgs: 148 active+clean; 582 KiB data, 1002 MiB used, 625 GiB / 626 GiB avail; 651 B/s rd, 0 B/s wr, 942 op/s 2024-02-05T08:10:26.876 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:26 smithi134 ceph-mon[39278]: 6.1a scrub starts 2024-02-05T08:10:26.876 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:26 smithi134 ceph-mon[39278]: 6.1a scrub ok 2024-02-05T08:10:26.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:26.926 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:26.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:27.028 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:27.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:27.053 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:27.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:26 smithi196 ceph-mon[33018]: 5.12 scrub starts 2024-02-05T08:10:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:26 smithi196 ceph-mon[33018]: 5.12 scrub ok 2024-02-05T08:10:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:26 smithi196 ceph-mon[33018]: pgmap v224: 148 pgs: 148 active+clean; 582 KiB data, 1002 MiB used, 625 GiB / 626 GiB avail; 651 B/s rd, 0 B/s wr, 942 op/s 2024-02-05T08:10:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:26 smithi196 ceph-mon[33018]: 6.1a scrub starts 2024-02-05T08:10:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:26 smithi196 ceph-mon[33018]: 6.1a scrub ok 2024-02-05T08:10:27.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:10:27.154 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:27.181 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:27.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:27.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:27.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:27.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:27.310 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:27.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:27.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:27.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:27.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:27.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:27.540 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:27.566 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:27.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:27.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:27.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:27.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:27.797 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:27.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:27.823 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:27.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:27.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:27.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:27.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:28.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:10:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:10:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:10:28.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:28.081 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:28.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:28.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:28.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:28.210 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:28.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:28.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:28.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:28.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:28.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:28.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:28.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:28.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:28.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:28.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:28.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:28.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:28.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:28.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:28.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:28.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:28.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:28.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:28.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:28.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:28.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:29.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:28 smithi134 ceph-mon[30086]: 5.13 scrub starts 2024-02-05T08:10:29.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:28 smithi134 ceph-mon[30086]: 5.13 scrub ok 2024-02-05T08:10:29.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:28 smithi134 ceph-mon[30086]: pgmap v225: 148 pgs: 148 active+clean; 582 KiB data, 993 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 517 op/s 2024-02-05T08:10:29.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:28 smithi134 ceph-mon[30086]: 3.f scrub starts 2024-02-05T08:10:29.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:28 smithi134 ceph-mon[30086]: 3.f scrub ok 2024-02-05T08:10:29.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:28 smithi134 ceph-mon[39278]: 5.13 scrub starts 2024-02-05T08:10:29.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:28 smithi134 ceph-mon[39278]: 5.13 scrub ok 2024-02-05T08:10:29.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:28 smithi134 ceph-mon[39278]: pgmap v225: 148 pgs: 148 active+clean; 582 KiB data, 993 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 517 op/s 2024-02-05T08:10:29.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:28 smithi134 ceph-mon[39278]: 3.f scrub starts 2024-02-05T08:10:29.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:28 smithi134 ceph-mon[39278]: 3.f scrub ok 2024-02-05T08:10:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:28 smithi196 ceph-mon[33018]: 5.13 scrub starts 2024-02-05T08:10:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:28 smithi196 ceph-mon[33018]: 5.13 scrub ok 2024-02-05T08:10:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:28 smithi196 ceph-mon[33018]: pgmap v225: 148 pgs: 148 active+clean; 582 KiB data, 993 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 517 op/s 2024-02-05T08:10:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:28 smithi196 ceph-mon[33018]: 3.f scrub starts 2024-02-05T08:10:29.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:28 smithi196 ceph-mon[33018]: 3.f scrub ok 2024-02-05T08:10:29.085 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:29.114 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:29.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:29.216 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:29.241 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:29.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:29.342 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:29.366 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:29.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:29.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:29.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:29.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:29.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:29.594 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:29.618 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:29.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:29.720 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:29.748 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:29.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:29.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:29 smithi134 ceph-mon[30086]: 3.1e deep-scrub starts 2024-02-05T08:10:29.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:29 smithi134 ceph-mon[30086]: 3.1e deep-scrub ok 2024-02-05T08:10:29.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:29 smithi134 ceph-mon[30086]: 6.0 scrub starts 2024-02-05T08:10:29.828 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:29 smithi134 ceph-mon[30086]: 6.0 scrub ok 2024-02-05T08:10:29.828 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:29 smithi134 ceph-mon[39278]: 3.1e deep-scrub starts 2024-02-05T08:10:29.828 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:29 smithi134 ceph-mon[39278]: 3.1e deep-scrub ok 2024-02-05T08:10:29.828 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:29 smithi134 ceph-mon[39278]: 6.0 scrub starts 2024-02-05T08:10:29.828 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:29 smithi134 ceph-mon[39278]: 6.0 scrub ok 2024-02-05T08:10:29.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:29.874 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:29.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:29.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:30.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:30.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:30.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:29 smithi196 ceph-mon[33018]: 3.1e deep-scrub starts 2024-02-05T08:10:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:29 smithi196 ceph-mon[33018]: 3.1e deep-scrub ok 2024-02-05T08:10:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:29 smithi196 ceph-mon[33018]: 6.0 scrub starts 2024-02-05T08:10:30.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:29 smithi196 ceph-mon[33018]: 6.0 scrub ok 2024-02-05T08:10:30.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:30.133 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:30.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:30.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:30.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:30.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:30.275 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:10:30.275 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:10:30.275 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-02-05T08:10:30.276 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:10:30.341 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:30.366 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:30.390 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:30.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:30.470 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:10:30.489 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:10:30.491 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:30.508 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:10:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:30.521 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:30.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:30.528 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:10:30.548 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:10:30.570 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:10:30.591 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:10:30.611 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:10:30.612 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:10:30.612 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:10:30.613 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:10:30.613 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:10:30.613 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:10:30.613 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:10:30.613 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:10:30.613 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:10:30.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:30.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:30.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:30.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:30.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:30.777 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:30.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:30.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:30.906 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:30.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:30.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:31.007 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:31.032 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:31.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:30 smithi196 ceph-mon[33018]: 6.18 scrub starts 2024-02-05T08:10:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:30 smithi196 ceph-mon[33018]: 6.18 scrub ok 2024-02-05T08:10:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:30 smithi196 ceph-mon[33018]: pgmap v226: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 517 op/s 2024-02-05T08:10:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:30 smithi196 ceph-mon[33018]: 5.1c scrub starts 2024-02-05T08:10:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:30 smithi196 ceph-mon[33018]: 5.1c scrub ok 2024-02-05T08:10:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:30 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2578837609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:10:31.070 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:30 smithi134 ceph-mon[30086]: 6.18 scrub starts 2024-02-05T08:10:31.070 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:30 smithi134 ceph-mon[30086]: 6.18 scrub ok 2024-02-05T08:10:31.071 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:30 smithi134 ceph-mon[30086]: pgmap v226: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 517 op/s 2024-02-05T08:10:31.071 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:30 smithi134 ceph-mon[30086]: 5.1c scrub starts 2024-02-05T08:10:31.071 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:30 smithi134 ceph-mon[30086]: 5.1c scrub ok 2024-02-05T08:10:31.071 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:30 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2578837609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:10:31.071 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:30 smithi134 ceph-mon[39278]: 6.18 scrub starts 2024-02-05T08:10:31.071 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:30 smithi134 ceph-mon[39278]: 6.18 scrub ok 2024-02-05T08:10:31.071 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:30 smithi134 ceph-mon[39278]: pgmap v226: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 517 op/s 2024-02-05T08:10:31.071 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:30 smithi134 ceph-mon[39278]: 5.1c scrub starts 2024-02-05T08:10:31.071 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:30 smithi134 ceph-mon[39278]: 5.1c scrub ok 2024-02-05T08:10:31.071 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:30 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2578837609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:10:31.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:31.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:31.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:31.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:31.288 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:31.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:31.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:31.395 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:10:31.399 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:10:31.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:31.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:31.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:31.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:31.552 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:31.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:31.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:31.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:31.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:31.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:31.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:31.808 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:31.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:31.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:31.933 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:31.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:31.961 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[30086]: 3.6 scrub starts 2024-02-05T08:10:31.961 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[30086]: 3.6 scrub ok 2024-02-05T08:10:31.961 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[30086]: 3.c scrub starts 2024-02-05T08:10:31.961 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[30086]: 3.c scrub ok 2024-02-05T08:10:31.962 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[30086]: 4.1d scrub starts 2024-02-05T08:10:31.962 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[30086]: 4.1d scrub ok 2024-02-05T08:10:31.962 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[30086]: 3.15 scrub starts 2024-02-05T08:10:31.962 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[30086]: 3.15 scrub ok 2024-02-05T08:10:31.962 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[30086]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:10:31.962 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[39278]: 3.6 scrub starts 2024-02-05T08:10:31.962 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[39278]: 3.6 scrub ok 2024-02-05T08:10:31.962 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[39278]: 3.c scrub starts 2024-02-05T08:10:31.962 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[39278]: 3.c scrub ok 2024-02-05T08:10:31.962 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[39278]: 4.1d scrub starts 2024-02-05T08:10:31.962 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[39278]: 4.1d scrub ok 2024-02-05T08:10:31.962 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[39278]: 3.15 scrub starts 2024-02-05T08:10:31.963 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[39278]: 3.15 scrub ok 2024-02-05T08:10:31.963 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:31 smithi134 ceph-mon[39278]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:10:32.035 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:32.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:32.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:32.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:31 smithi196 ceph-mon[33018]: 3.6 scrub starts 2024-02-05T08:10:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:31 smithi196 ceph-mon[33018]: 3.6 scrub ok 2024-02-05T08:10:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:31 smithi196 ceph-mon[33018]: 3.c scrub starts 2024-02-05T08:10:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:31 smithi196 ceph-mon[33018]: 3.c scrub ok 2024-02-05T08:10:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:31 smithi196 ceph-mon[33018]: 4.1d scrub starts 2024-02-05T08:10:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:31 smithi196 ceph-mon[33018]: 4.1d scrub ok 2024-02-05T08:10:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:31 smithi196 ceph-mon[33018]: 3.15 scrub starts 2024-02-05T08:10:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:31 smithi196 ceph-mon[33018]: 3.15 scrub ok 2024-02-05T08:10:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:31 smithi196 ceph-mon[33018]: from='client.15012 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:10:32.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:32.189 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:32.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:32.291 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:32.314 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:32.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:32.369 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-02-05T08:10:29.600408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1272,"num_read_kb":1118,"num_write":220479,"num_write_kb":7038046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":2363363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76153,"ondisk_log_size":76153,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1000096,"kb_used_data":7196,"kb_used_omap":10,"kb_used_meta":992821,"kb_avail":655215968,"statfs":{"total":671965249536,"available":670941151232,"internally_reserved":0,"allocated":7368704,"data_stored":5102090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11155,"internal_metadata":1016648813},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-270598168,"num_objects":-4130,"num_object_clones":0,"num_object_copies":-12390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4130,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4130,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002100"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:26.432384+0000","last_change":"2024-02-05T08:10:26.432413+0000","last_active":"2024-02-05T08:10:26.432384+0000","last_peered":"2024-02-05T08:10:26.432384+0000","last_clean":"2024-02-05T08:10:26.432384+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:10:26.432384+0000","last_undegraded":"2024-02-05T08:10:26.432384+0000","last_fullsized":"2024-02-05T08:10:26.432384+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:26.432294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:26.432294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.412548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778189+0000","last_change":"2024-02-05T08:08:50.003502+0000","last_active":"2024-02-05T08:10:23.778189+0000","last_peered":"2024-02-05T08:10:23.778189+0000","last_clean":"2024-02-05T08:10:23.778189+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:10:23.778189+0000","last_undegraded":"2024-02-05T08:10:23.778189+0000","last_fullsized":"2024-02-05T08:10:23.778189+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:50.003400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:50.003400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":204,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851452+0000","last_change":"2024-02-05T08:10:18.254802+0000","last_active":"2024-02-05T08:10:23.851452+0000","last_peered":"2024-02-05T08:10:23.851452+0000","last_clean":"2024-02-05T08:10:23.851452+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:10:23.851452+0000","last_undegraded":"2024-02-05T08:10:23.851452+0000","last_fullsized":"2024-02-05T08:10:23.851452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.254568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.254568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:46.157955+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"90'13","reported_seq":228,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850480+0000","last_change":"2024-02-05T08:08:31.425402+0000","last_active":"2024-02-05T08:10:23.850480+0000","last_peered":"2024-02-05T08:10:23.850480+0000","last_clean":"2024-02-05T08:10:23.850480+0000","last_became_active":"2024-02-05T08:06:39.560434+0000","last_became_peered":"2024-02-05T08:06:39.560434+0000","last_unstale":"2024-02-05T08:10:23.850480+0000","last_undegraded":"2024-02-05T08:10:23.850480+0000","last_fullsized":"2024-02-05T08:10:23.850480+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:08:31.425318+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:08:31.425318+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":198,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779285+0000","last_change":"2024-02-05T08:09:37.106387+0000","last_active":"2024-02-05T08:10:23.779285+0000","last_peered":"2024-02-05T08:10:23.779285+0000","last_clean":"2024-02-05T08:10:23.779285+0000","last_became_active":"2024-02-05T08:06:39.558790+0000","last_became_peered":"2024-02-05T08:06:39.558790+0000","last_unstale":"2024-02-05T08:10:23.779285+0000","last_undegraded":"2024-02-05T08:10:23.779285+0000","last_fullsized":"2024-02-05T08:10:23.779285+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:37.106147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:37.106147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:05.992123+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850755+0000","last_change":"2024-02-05T08:08:47.432025+0000","last_active":"2024-02-05T08:10:23.850755+0000","last_peered":"2024-02-05T08:10:23.850755+0000","last_clean":"2024-02-05T08:10:23.850755+0000","last_became_active":"2024-02-05T08:06:39.561731+0000","last_became_peered":"2024-02-05T08:06:39.561731+0000","last_unstale":"2024-02-05T08:10:23.850755+0000","last_undegraded":"2024-02-05T08:10:23.850755+0000","last_fullsized":"2024-02-05T08:10:23.850755+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:47.431940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:47.431940+0000","last_clean_scrub_stamp":"2024-02-05T08:08:47.431940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":202,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.781233+0000","last_change":"2024-02-05T08:07:27.703599+0000","last_active":"2024-02-05T08:10:23.781233+0000","last_peered":"2024-02-05T08:10:23.781233+0000","last_clean":"2024-02-05T08:10:23.781233+0000","last_became_active":"2024-02-05T08:06:39.558101+0000","last_became_peered":"2024-02-05T08:06:39.558101+0000","last_unstale":"2024-02-05T08:10:23.781233+0000","last_undegraded":"2024-02-05T08:10:23.781233+0000","last_fullsized":"2024-02-05T08:10:23.781233+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:27.703488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:27.703488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"118'22","reported_seq":224,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.780166+0000","last_change":"2024-02-05T08:07:45.819143+0000","last_active":"2024-02-05T08:10:23.780166+0000","last_peered":"2024-02-05T08:10:23.780166+0000","last_clean":"2024-02-05T08:10:23.780166+0000","last_became_active":"2024-02-05T08:04:43.560660+0000","last_became_peered":"2024-02-05T08:04:43.560660+0000","last_unstale":"2024-02-05T08:10:23.780166+0000","last_undegraded":"2024-02-05T08:10:23.780166+0000","last_fullsized":"2024-02-05T08:10:23.780166+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:07:45.819004+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.849665+0000","last_change":"2024-02-05T08:07:38.563648+0000","last_active":"2024-02-05T08:10:23.849665+0000","last_peered":"2024-02-05T08:10:23.849665+0000","last_clean":"2024-02-05T08:10:23.849665+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:10:23.849665+0000","last_undegraded":"2024-02-05T08:10:23.849665+0000","last_fullsized":"2024-02-05T08:10:23.849665+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:38.563533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:38.563533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851634+0000","last_change":"2024-02-05T08:08:35.065367+0000","last_active":"2024-02-05T08:10:23.851634+0000","last_peered":"2024-02-05T08:10:23.851634+0000","last_clean":"2024-02-05T08:10:23.851634+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:10:23.851634+0000","last_undegraded":"2024-02-05T08:10:23.851634+0000","last_fullsized":"2024-02-05T08:10:23.851634+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:35.065258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:37.352358+0000","last_clean_scrub_stamp":"2024-02-05T08:08:35.065258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":210,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850848+0000","last_change":"2024-02-05T08:09:39.410126+0000","last_active":"2024-02-05T08:10:23.850848+0000","last_peered":"2024-02-05T08:10:23.850848+0000","last_clean":"2024-02-05T08:10:23.850848+0000","last_became_active":"2024-02-05T08:06:39.559461+0000","last_became_peered":"2024-02-05T08:06:39.559461+0000","last_unstale":"2024-02-05T08:10:23.850848+0000","last_undegraded":"2024-02-05T08:10:23.850848+0000","last_fullsized":"2024-02-05T08:10:23.850848+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:39.410054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.410054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.039707+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":224,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850952+0000","last_change":"2024-02-05T08:08:38.437994+0000","last_active":"2024-02-05T08:10:23.850952+0000","last_peered":"2024-02-05T08:10:23.850952+0000","last_clean":"2024-02-05T08:10:23.850952+0000","last_became_active":"2024-02-05T08:07:22.653892+0000","last_became_peered":"2024-02-05T08:07:22.653892+0000","last_unstale":"2024-02-05T08:10:23.850952+0000","last_undegraded":"2024-02-05T08:10:23.850952+0000","last_fullsized":"2024-02-05T08:10:23.850952+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:08:38.437850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:38.437850+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850013+0000","last_change":"2024-02-05T08:10:16.560606+0000","last_active":"2024-02-05T08:10:23.850013+0000","last_peered":"2024-02-05T08:10:23.850013+0000","last_clean":"2024-02-05T08:10:23.850013+0000","last_became_active":"2024-02-05T08:04:47.587515+0000","last_became_peered":"2024-02-05T08:04:47.587515+0000","last_unstale":"2024-02-05T08:10:23.850013+0000","last_undegraded":"2024-02-05T08:10:23.850013+0000","last_fullsized":"2024-02-05T08:10:23.850013+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:16.560531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:16.560531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:46.471717+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:24.239420+0000","last_change":"2024-02-05T08:10:24.239442+0000","last_active":"2024-02-05T08:10:24.239420+0000","last_peered":"2024-02-05T08:10:24.239420+0000","last_clean":"2024-02-05T08:10:24.239420+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:10:24.239420+0000","last_undegraded":"2024-02-05T08:10:24.239420+0000","last_fullsized":"2024-02-05T08:10:24.239420+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:24.239334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:24.239334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:53.342841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":204,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850508+0000","last_change":"2024-02-05T08:10:23.663012+0000","last_active":"2024-02-05T08:10:23.850508+0000","last_peered":"2024-02-05T08:10:23.850508+0000","last_clean":"2024-02-05T08:10:23.850508+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:10:23.850508+0000","last_undegraded":"2024-02-05T08:10:23.850508+0000","last_fullsized":"2024-02-05T08:10:23.850508+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:23.662919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:10:23.662919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:40.039778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":202,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850585+0000","last_change":"2024-02-05T08:09:30.448604+0000","last_active":"2024-02-05T08:10:23.850585+0000","last_peered":"2024-02-05T08:10:23.850585+0000","last_clean":"2024-02-05T08:10:23.850585+0000","last_became_active":"2024-02-05T08:04:43.557559+0000","last_became_peered":"2024-02-05T08:04:43.557559+0000","last_unstale":"2024-02-05T08:10:23.850585+0000","last_undegraded":"2024-02-05T08:10:23.850585+0000","last_fullsized":"2024-02-05T08:10:23.850585+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:09:30.448493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:30.448493+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:40.766789+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":198,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851792+0000","last_change":"2024-02-05T08:09:26.466381+0000","last_active":"2024-02-05T08:10:23.851792+0000","last_peered":"2024-02-05T08:10:23.851792+0000","last_clean":"2024-02-05T08:10:23.851792+0000","last_became_active":"2024-02-05T08:06:39.547906+0000","last_became_peered":"2024-02-05T08:06:39.547906+0000","last_unstale":"2024-02-05T08:10:23.851792+0000","last_undegraded":"2024-02-05T08:10:23.851792+0000","last_fullsized":"2024-02-05T08:10:23.851792+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:26.466302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:09:26.466302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:54.690678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":189,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778538+0000","last_change":"2024-02-05T08:07:48.943001+0000","last_active":"2024-02-05T08:10:23.778538+0000","last_peered":"2024-02-05T08:10:23.778538+0000","last_clean":"2024-02-05T08:10:23.778538+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:10:23.778538+0000","last_undegraded":"2024-02-05T08:10:23.778538+0000","last_fullsized":"2024-02-05T08:10:23.778538+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:07:48.942873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850562+0000","last_change":"2024-02-05T08:08:28.601390+0000","last_active":"2024-02-05T08:10:23.850562+0000","last_peered":"2024-02-05T08:10:23.850562+0000","last_clean":"2024-02-05T08:10:23.850562+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:10:23.850562+0000","last_undegraded":"2024-02-05T08:10:23.850562+0000","last_fullsized":"2024-02-05T08:10:23.850562+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:28.601275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:28.601275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":234,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850417+0000","last_change":"2024-02-05T08:10:19.645020+0000","last_active":"2024-02-05T08:10:23.850417+0000","last_peered":"2024-02-05T08:10:23.850417+0000","last_clean":"2024-02-05T08:10:23.850417+0000","last_became_active":"2024-02-05T08:04:43.557496+0000","last_became_peered":"2024-02-05T08:04:43.557496+0000","last_unstale":"2024-02-05T08:10:23.850417+0000","last_undegraded":"2024-02-05T08:10:23.850417+0000","last_fullsized":"2024-02-05T08:10:23.850417+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:10:19.644900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:19.644900+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:31.573739+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":190,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.781031+0000","last_change":"2024-02-05T08:09:21.354430+0000","last_active":"2024-02-05T08:10:23.781031+0000","last_peered":"2024-02-05T08:10:23.781031+0000","last_clean":"2024-02-05T08:10:23.781031+0000","last_became_active":"2024-02-05T08:06:39.560948+0000","last_became_peered":"2024-02-05T08:06:39.560948+0000","last_unstale":"2024-02-05T08:10:23.781031+0000","last_undegraded":"2024-02-05T08:10:23.781031+0000","last_fullsized":"2024-02-05T08:10:23.781031+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:21.354158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.354158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:26.901104+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851167+0000","last_change":"2024-02-05T08:07:55.895894+0000","last_active":"2024-02-05T08:10:23.851167+0000","last_peered":"2024-02-05T08:10:23.851167+0000","last_clean":"2024-02-05T08:10:23.851167+0000","last_became_active":"2024-02-05T08:04:45.563948+0000","last_became_peered":"2024-02-05T08:04:45.563948+0000","last_unstale":"2024-02-05T08:10:23.851167+0000","last_undegraded":"2024-02-05T08:10:23.851167+0000","last_fullsized":"2024-02-05T08:10:23.851167+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:55.895795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:07:55.895795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851043+0000","last_change":"2024-02-05T08:09:27.431319+0000","last_active":"2024-02-05T08:10:23.851043+0000","last_peered":"2024-02-05T08:10:23.851043+0000","last_clean":"2024-02-05T08:10:23.851043+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:10:23.851043+0000","last_undegraded":"2024-02-05T08:10:23.851043+0000","last_fullsized":"2024-02-05T08:10:23.851043+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:27.431221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:27.431221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:44.170149+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"90'17","reported_seq":220,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.780096+0000","last_change":"2024-02-05T08:07:24.701607+0000","last_active":"2024-02-05T08:10:23.780096+0000","last_peered":"2024-02-05T08:10:23.780096+0000","last_clean":"2024-02-05T08:10:23.780096+0000","last_became_active":"2024-02-05T08:04:43.557596+0000","last_became_peered":"2024-02-05T08:04:43.557596+0000","last_unstale":"2024-02-05T08:10:23.780096+0000","last_undegraded":"2024-02-05T08:10:23.780096+0000","last_fullsized":"2024-02-05T08:10:23.780096+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:07:24.701465+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:07:24.701465+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":195,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779092+0000","last_change":"2024-02-05T08:08:48.447437+0000","last_active":"2024-02-05T08:10:23.779092+0000","last_peered":"2024-02-05T08:10:23.779092+0000","last_clean":"2024-02-05T08:10:23.779092+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:10:23.779092+0000","last_undegraded":"2024-02-05T08:10:23.779092+0000","last_fullsized":"2024-02-05T08:10:23.779092+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:08:48.447317+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":208,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851937+0000","last_change":"2024-02-05T08:08:44.441278+0000","last_active":"2024-02-05T08:10:23.851937+0000","last_peered":"2024-02-05T08:10:23.851937+0000","last_clean":"2024-02-05T08:10:23.851937+0000","last_became_active":"2024-02-05T08:06:39.552760+0000","last_became_peered":"2024-02-05T08:06:39.552760+0000","last_unstale":"2024-02-05T08:10:23.851937+0000","last_undegraded":"2024-02-05T08:10:23.851937+0000","last_fullsized":"2024-02-05T08:10:23.851937+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:08:44.441144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.441144+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":198,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778075+0000","last_change":"2024-02-05T08:07:35.997681+0000","last_active":"2024-02-05T08:10:23.778075+0000","last_peered":"2024-02-05T08:10:23.778075+0000","last_clean":"2024-02-05T08:10:23.778075+0000","last_became_active":"2024-02-05T08:04:41.532339+0000","last_became_peered":"2024-02-05T08:04:41.532339+0000","last_unstale":"2024-02-05T08:10:23.778075+0000","last_undegraded":"2024-02-05T08:10:23.778075+0000","last_fullsized":"2024-02-05T08:10:23.778075+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:07:35.997535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:35.997535+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":207,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850803+0000","last_change":"2024-02-05T08:09:24.000902+0000","last_active":"2024-02-05T08:10:23.850803+0000","last_peered":"2024-02-05T08:10:23.850803+0000","last_clean":"2024-02-05T08:10:23.850803+0000","last_became_active":"2024-02-05T08:04:43.605952+0000","last_became_peered":"2024-02-05T08:04:43.605952+0000","last_unstale":"2024-02-05T08:10:23.850803+0000","last_undegraded":"2024-02-05T08:10:23.850803+0000","last_fullsized":"2024-02-05T08:10:23.850803+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:09:24.000675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:24.000675+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:53.229695+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":193,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779345+0000","last_change":"2024-02-05T08:09:25.266619+0000","last_active":"2024-02-05T08:10:23.779345+0000","last_peered":"2024-02-05T08:10:23.779345+0000","last_clean":"2024-02-05T08:10:23.779345+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:10:23.779345+0000","last_undegraded":"2024-02-05T08:10:23.779345+0000","last_fullsized":"2024-02-05T08:10:23.779345+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.266508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.266508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:35.256058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851146+0000","last_change":"2024-02-05T08:08:42.676027+0000","last_active":"2024-02-05T08:10:23.851146+0000","last_peered":"2024-02-05T08:10:23.851146+0000","last_clean":"2024-02-05T08:10:23.851146+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:10:23.851146+0000","last_undegraded":"2024-02-05T08:10:23.851146+0000","last_fullsized":"2024-02-05T08:10:23.851146+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:42.675782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:42.675782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778752+0000","last_change":"2024-02-05T08:07:50.777386+0000","last_active":"2024-02-05T08:10:23.778752+0000","last_peered":"2024-02-05T08:10:23.778752+0000","last_clean":"2024-02-05T08:10:23.778752+0000","last_became_active":"2024-02-05T08:04:41.534448+0000","last_became_peered":"2024-02-05T08:04:41.534448+0000","last_unstale":"2024-02-05T08:10:23.778752+0000","last_undegraded":"2024-02-05T08:10:23.778752+0000","last_fullsized":"2024-02-05T08:10:23.778752+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:50.777273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:07:50.777273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851181+0000","last_change":"2024-02-05T08:09:25.380533+0000","last_active":"2024-02-05T08:10:23.851181+0000","last_peered":"2024-02-05T08:10:23.851181+0000","last_clean":"2024-02-05T08:10:23.851181+0000","last_became_active":"2024-02-05T08:06:39.558747+0000","last_became_peered":"2024-02-05T08:06:39.558747+0000","last_unstale":"2024-02-05T08:10:23.851181+0000","last_undegraded":"2024-02-05T08:10:23.851181+0000","last_fullsized":"2024-02-05T08:10:23.851181+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:09:25.380390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.380390+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:41.147157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778622+0000","last_change":"2024-02-05T08:08:31.483194+0000","last_active":"2024-02-05T08:10:23.778622+0000","last_peered":"2024-02-05T08:10:23.778622+0000","last_clean":"2024-02-05T08:10:23.778622+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:10:23.778622+0000","last_undegraded":"2024-02-05T08:10:23.778622+0000","last_fullsized":"2024-02-05T08:10:23.778622+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:31.483087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:31.483087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":135,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850293+0000","last_change":"2024-02-05T08:09:36.434985+0000","last_active":"2024-02-05T08:10:23.850293+0000","last_peered":"2024-02-05T08:10:23.850293+0000","last_clean":"2024-02-05T08:10:23.850293+0000","last_became_active":"2024-02-05T08:06:39.551173+0000","last_became_peered":"2024-02-05T08:06:39.551173+0000","last_unstale":"2024-02-05T08:10:23.850293+0000","last_undegraded":"2024-02-05T08:10:23.850293+0000","last_fullsized":"2024-02-05T08:10:23.850293+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:36.434873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.434873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:05.817457+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851541+0000","last_change":"2024-02-05T08:08:30.632883+0000","last_active":"2024-02-05T08:10:23.851541+0000","last_peered":"2024-02-05T08:10:23.851541+0000","last_clean":"2024-02-05T08:10:23.851541+0000","last_became_active":"2024-02-05T08:04:41.536019+0000","last_became_peered":"2024-02-05T08:04:41.536019+0000","last_unstale":"2024-02-05T08:10:23.851541+0000","last_undegraded":"2024-02-05T08:10:23.851541+0000","last_fullsized":"2024-02-05T08:10:23.851541+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:30.632785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:30.632785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":210,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778843+0000","last_change":"2024-02-05T08:08:40.082709+0000","last_active":"2024-02-05T08:10:23.778843+0000","last_peered":"2024-02-05T08:10:23.778843+0000","last_clean":"2024-02-05T08:10:23.778843+0000","last_became_active":"2024-02-05T08:06:39.560932+0000","last_became_peered":"2024-02-05T08:06:39.560932+0000","last_unstale":"2024-02-05T08:10:23.778843+0000","last_undegraded":"2024-02-05T08:10:23.778843+0000","last_fullsized":"2024-02-05T08:10:23.778843+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-02-05T08:08:40.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:40.082577+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":198,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779613+0000","last_change":"2024-02-05T08:10:18.022780+0000","last_active":"2024-02-05T08:10:23.779613+0000","last_peered":"2024-02-05T08:10:23.779613+0000","last_clean":"2024-02-05T08:10:23.779613+0000","last_became_active":"2024-02-05T08:06:39.559212+0000","last_became_peered":"2024-02-05T08:06:39.559212+0000","last_unstale":"2024-02-05T08:10:23.779613+0000","last_undegraded":"2024-02-05T08:10:23.779613+0000","last_fullsized":"2024-02-05T08:10:23.779613+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.022670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.022670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:22.583988+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778796+0000","last_change":"2024-02-05T08:09:36.353456+0000","last_active":"2024-02-05T08:10:23.778796+0000","last_peered":"2024-02-05T08:10:23.778796+0000","last_clean":"2024-02-05T08:10:23.778796+0000","last_became_active":"2024-02-05T08:06:39.555597+0000","last_became_peered":"2024-02-05T08:06:39.555597+0000","last_unstale":"2024-02-05T08:10:23.778796+0000","last_undegraded":"2024-02-05T08:10:23.778796+0000","last_fullsized":"2024-02-05T08:10:23.778796+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.353351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:38.202761+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":204,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850843+0000","last_change":"2024-02-05T08:08:45.166139+0000","last_active":"2024-02-05T08:10:23.850843+0000","last_peered":"2024-02-05T08:10:23.850843+0000","last_clean":"2024-02-05T08:10:23.850843+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:10:23.850843+0000","last_undegraded":"2024-02-05T08:10:23.850843+0000","last_fullsized":"2024-02-05T08:10:23.850843+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:45.166006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:45.166006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"141'16","reported_seq":242,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.780111+0000","last_change":"2024-02-05T08:09:39.113079+0000","last_active":"2024-02-05T08:10:23.780111+0000","last_peered":"2024-02-05T08:10:23.780111+0000","last_clean":"2024-02-05T08:10:23.780111+0000","last_became_active":"2024-02-05T08:06:39.556873+0000","last_became_peered":"2024-02-05T08:06:39.556873+0000","last_unstale":"2024-02-05T08:10:23.780111+0000","last_undegraded":"2024-02-05T08:10:23.780111+0000","last_fullsized":"2024-02-05T08:10:23.780111+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'16","last_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.112936+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:45.211059+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.849485+0000","last_change":"2024-02-05T08:07:45.514975+0000","last_active":"2024-02-05T08:10:23.849485+0000","last_peered":"2024-02-05T08:10:23.849485+0000","last_clean":"2024-02-05T08:10:23.849485+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:10:23.849485+0000","last_undegraded":"2024-02-05T08:10:23.849485+0000","last_fullsized":"2024-02-05T08:10:23.849485+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:45.514863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:45.514863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:26.987823+0000","last_change":"2024-02-05T08:10:26.987843+0000","last_active":"2024-02-05T08:10:26.987823+0000","last_peered":"2024-02-05T08:10:26.987823+0000","last_clean":"2024-02-05T08:10:26.987823+0000","last_became_active":"2024-02-05T08:04:45.568484+0000","last_became_peered":"2024-02-05T08:04:45.568484+0000","last_unstale":"2024-02-05T08:10:26.987823+0000","last_undegraded":"2024-02-05T08:10:26.987823+0000","last_fullsized":"2024-02-05T08:10:26.987823+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:26.987763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:26.987763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:37.413701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779160+0000","last_change":"2024-02-05T08:08:32.528082+0000","last_active":"2024-02-05T08:10:23.779160+0000","last_peered":"2024-02-05T08:10:23.779160+0000","last_clean":"2024-02-05T08:10:23.779160+0000","last_became_active":"2024-02-05T08:04:41.539698+0000","last_became_peered":"2024-02-05T08:04:41.539698+0000","last_unstale":"2024-02-05T08:10:23.779160+0000","last_undegraded":"2024-02-05T08:10:23.779160+0000","last_fullsized":"2024-02-05T08:10:23.779160+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:32.527978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:32.527978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":233,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779011+0000","last_change":"2024-02-05T08:08:43.941119+0000","last_active":"2024-02-05T08:10:23.779011+0000","last_peered":"2024-02-05T08:10:23.779011+0000","last_clean":"2024-02-05T08:10:23.779011+0000","last_became_active":"2024-02-05T08:06:39.560259+0000","last_became_peered":"2024-02-05T08:06:39.560259+0000","last_unstale":"2024-02-05T08:10:23.779011+0000","last_undegraded":"2024-02-05T08:10:23.779011+0000","last_fullsized":"2024-02-05T08:10:23.779011+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:08:43.940980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:43.940980+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779891+0000","last_change":"2024-02-05T08:09:29.262843+0000","last_active":"2024-02-05T08:10:23.779891+0000","last_peered":"2024-02-05T08:10:23.779891+0000","last_clean":"2024-02-05T08:10:23.779891+0000","last_became_active":"2024-02-05T08:04:47.585496+0000","last_became_peered":"2024-02-05T08:04:47.585496+0000","last_unstale":"2024-02-05T08:10:23.779891+0000","last_undegraded":"2024-02-05T08:10:23.779891+0000","last_fullsized":"2024-02-05T08:10:23.779891+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:09:29.262582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:47.044333+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:25.493071+0000","last_change":"2024-02-05T08:10:25.493102+0000","last_active":"2024-02-05T08:10:25.493071+0000","last_peered":"2024-02-05T08:10:25.493071+0000","last_clean":"2024-02-05T08:10:25.493071+0000","last_became_active":"2024-02-05T08:04:45.560815+0000","last_became_peered":"2024-02-05T08:04:45.560815+0000","last_unstale":"2024-02-05T08:10:25.493071+0000","last_undegraded":"2024-02-05T08:10:25.493071+0000","last_fullsized":"2024-02-05T08:10:25.493071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:25.493013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:25.493013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:33.003654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":210,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851302+0000","last_change":"2024-02-05T08:09:25.486212+0000","last_active":"2024-02-05T08:10:23.851302+0000","last_peered":"2024-02-05T08:10:23.851302+0000","last_clean":"2024-02-05T08:10:23.851302+0000","last_became_active":"2024-02-05T08:06:39.545576+0000","last_became_peered":"2024-02-05T08:06:39.545576+0000","last_unstale":"2024-02-05T08:10:23.851302+0000","last_undegraded":"2024-02-05T08:10:23.851302+0000","last_fullsized":"2024-02-05T08:10:23.851302+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.486141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.966886+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.486141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:39.358720+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":229,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850743+0000","last_change":"2024-02-05T08:08:26.541304+0000","last_active":"2024-02-05T08:10:23.850743+0000","last_peered":"2024-02-05T08:10:23.850743+0000","last_clean":"2024-02-05T08:10:23.850743+0000","last_became_active":"2024-02-05T08:06:39.558257+0000","last_became_peered":"2024-02-05T08:06:39.558257+0000","last_unstale":"2024-02-05T08:10:23.850743+0000","last_undegraded":"2024-02-05T08:10:23.850743+0000","last_fullsized":"2024-02-05T08:10:23.850743+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:08:26.541177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:26.541177+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850437+0000","last_change":"2024-02-05T08:08:36.679472+0000","last_active":"2024-02-05T08:10:23.850437+0000","last_peered":"2024-02-05T08:10:23.850437+0000","last_clean":"2024-02-05T08:10:23.850437+0000","last_became_active":"2024-02-05T08:04:47.584235+0000","last_became_peered":"2024-02-05T08:04:47.584235+0000","last_unstale":"2024-02-05T08:10:23.850437+0000","last_undegraded":"2024-02-05T08:10:23.850437+0000","last_fullsized":"2024-02-05T08:10:23.850437+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:36.679368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:36.679368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":274,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850984+0000","last_change":"2024-02-05T08:10:14.583618+0000","last_active":"2024-02-05T08:10:23.850984+0000","last_peered":"2024-02-05T08:10:23.850984+0000","last_clean":"2024-02-05T08:10:23.850984+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:10:23.850984+0000","last_undegraded":"2024-02-05T08:10:23.850984+0000","last_fullsized":"2024-02-05T08:10:23.850984+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:10:14.583489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:14.583489+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:28.694742+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":205,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778878+0000","last_change":"2024-02-05T08:09:19.417311+0000","last_active":"2024-02-05T08:10:23.778878+0000","last_peered":"2024-02-05T08:10:23.778878+0000","last_clean":"2024-02-05T08:10:23.778878+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:10:23.778878+0000","last_undegraded":"2024-02-05T08:10:23.778878+0000","last_fullsized":"2024-02-05T08:10:23.778878+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:19.417231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.417231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:25.958103+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"129'15","reported_seq":212,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779367+0000","last_change":"2024-02-05T08:08:37.128945+0000","last_active":"2024-02-05T08:10:23.779367+0000","last_peered":"2024-02-05T08:10:23.779367+0000","last_clean":"2024-02-05T08:10:23.779367+0000","last_became_active":"2024-02-05T08:04:43.605981+0000","last_became_peered":"2024-02-05T08:04:43.605981+0000","last_unstale":"2024-02-05T08:10:23.779367+0000","last_undegraded":"2024-02-05T08:10:23.779367+0000","last_fullsized":"2024-02-05T08:10:23.779367+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'15","last_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:08:37.128817+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851809+0000","last_change":"2024-02-05T08:09:18.397753+0000","last_active":"2024-02-05T08:10:23.851809+0000","last_peered":"2024-02-05T08:10:23.851809+0000","last_clean":"2024-02-05T08:10:23.851809+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:10:23.851809+0000","last_undegraded":"2024-02-05T08:10:23.851809+0000","last_fullsized":"2024-02-05T08:10:23.851809+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:18.397684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:18.397684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:42.614612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":266,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779809+0000","last_change":"2024-02-05T08:07:29.660531+0000","last_active":"2024-02-05T08:10:23.779809+0000","last_peered":"2024-02-05T08:10:23.779809+0000","last_clean":"2024-02-05T08:10:23.779809+0000","last_became_active":"2024-02-05T08:04:45.565240+0000","last_became_peered":"2024-02-05T08:04:45.565240+0000","last_unstale":"2024-02-05T08:10:23.779809+0000","last_undegraded":"2024-02-05T08:10:23.779809+0000","last_fullsized":"2024-02-05T08:10:23.779809+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:07:29.660395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:29.660395+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851072+0000","last_change":"2024-02-05T08:08:44.680464+0000","last_active":"2024-02-05T08:10:23.851072+0000","last_peered":"2024-02-05T08:10:23.851072+0000","last_clean":"2024-02-05T08:10:23.851072+0000","last_became_active":"2024-02-05T08:04:41.540252+0000","last_became_peered":"2024-02-05T08:04:41.540252+0000","last_unstale":"2024-02-05T08:10:23.851072+0000","last_undegraded":"2024-02-05T08:10:23.851072+0000","last_fullsized":"2024-02-05T08:10:23.851072+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:44.680353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.680353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":216,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851104+0000","last_change":"2024-02-05T08:09:36.437558+0000","last_active":"2024-02-05T08:10:23.851104+0000","last_peered":"2024-02-05T08:10:23.851104+0000","last_clean":"2024-02-05T08:10:23.851104+0000","last_became_active":"2024-02-05T08:04:43.606414+0000","last_became_peered":"2024-02-05T08:04:43.606414+0000","last_unstale":"2024-02-05T08:10:23.851104+0000","last_undegraded":"2024-02-05T08:10:23.851104+0000","last_fullsized":"2024-02-05T08:10:23.851104+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:09:36.437435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.437435+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:44.544063+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":201,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850728+0000","last_change":"2024-02-05T08:10:20.671244+0000","last_active":"2024-02-05T08:10:23.850728+0000","last_peered":"2024-02-05T08:10:23.850728+0000","last_clean":"2024-02-05T08:10:23.850728+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:10:23.850728+0000","last_undegraded":"2024-02-05T08:10:23.850728+0000","last_fullsized":"2024-02-05T08:10:23.850728+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:20.671130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:20.671130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:39.009177+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.780882+0000","last_change":"2024-02-05T08:10:15.994743+0000","last_active":"2024-02-05T08:10:23.780882+0000","last_peered":"2024-02-05T08:10:23.780882+0000","last_clean":"2024-02-05T08:10:23.780882+0000","last_became_active":"2024-02-05T08:04:45.567622+0000","last_became_peered":"2024-02-05T08:04:45.567622+0000","last_unstale":"2024-02-05T08:10:23.780882+0000","last_undegraded":"2024-02-05T08:10:23.780882+0000","last_fullsized":"2024-02-05T08:10:23.780882+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:15.994621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:15.994621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:33.060659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":203,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778690+0000","last_change":"2024-02-05T08:08:35.480440+0000","last_active":"2024-02-05T08:10:23.778690+0000","last_peered":"2024-02-05T08:10:23.778690+0000","last_clean":"2024-02-05T08:10:23.778690+0000","last_became_active":"2024-02-05T08:06:39.557018+0000","last_became_peered":"2024-02-05T08:06:39.557018+0000","last_unstale":"2024-02-05T08:10:23.778690+0000","last_undegraded":"2024-02-05T08:10:23.778690+0000","last_fullsized":"2024-02-05T08:10:23.778690+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:35.480367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:35.480367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":208,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850992+0000","last_change":"2024-02-05T08:08:45.712868+0000","last_active":"2024-02-05T08:10:23.850992+0000","last_peered":"2024-02-05T08:10:23.850992+0000","last_clean":"2024-02-05T08:10:23.850992+0000","last_became_active":"2024-02-05T08:04:43.608489+0000","last_became_peered":"2024-02-05T08:04:43.608489+0000","last_unstale":"2024-02-05T08:10:23.850992+0000","last_undegraded":"2024-02-05T08:10:23.850992+0000","last_fullsized":"2024-02-05T08:10:23.850992+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:08:45.712396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:45.712396+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":199,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779080+0000","last_change":"2024-02-05T08:09:33.391323+0000","last_active":"2024-02-05T08:10:23.779080+0000","last_peered":"2024-02-05T08:10:23.779080+0000","last_clean":"2024-02-05T08:10:23.779080+0000","last_became_active":"2024-02-05T08:06:39.548837+0000","last_became_peered":"2024-02-05T08:06:39.548837+0000","last_unstale":"2024-02-05T08:10:23.779080+0000","last_undegraded":"2024-02-05T08:10:23.779080+0000","last_fullsized":"2024-02-05T08:10:23.779080+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:33.391219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:33.391219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:38.950366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851053+0000","last_change":"2024-02-05T08:07:44.778884+0000","last_active":"2024-02-05T08:10:23.851053+0000","last_peered":"2024-02-05T08:10:23.851053+0000","last_clean":"2024-02-05T08:10:23.851053+0000","last_became_active":"2024-02-05T08:04:45.560241+0000","last_became_peered":"2024-02-05T08:04:45.560241+0000","last_unstale":"2024-02-05T08:10:23.851053+0000","last_undegraded":"2024-02-05T08:10:23.851053+0000","last_fullsized":"2024-02-05T08:10:23.851053+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:44.778772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:44.778772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":127,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.849820+0000","last_change":"2024-02-05T08:07:36.550407+0000","last_active":"2024-02-05T08:10:23.849820+0000","last_peered":"2024-02-05T08:10:23.849820+0000","last_clean":"2024-02-05T08:10:23.849820+0000","last_became_active":"2024-02-05T08:06:39.554673+0000","last_became_peered":"2024-02-05T08:06:39.554673+0000","last_unstale":"2024-02-05T08:10:23.849820+0000","last_undegraded":"2024-02-05T08:10:23.849820+0000","last_fullsized":"2024-02-05T08:10:23.849820+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:36.550290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:36.550290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"116'9","reported_seq":204,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.781122+0000","last_change":"2024-02-05T08:07:42.729822+0000","last_active":"2024-02-05T08:10:23.781122+0000","last_peered":"2024-02-05T08:10:23.781122+0000","last_clean":"2024-02-05T08:10:23.781122+0000","last_became_active":"2024-02-05T08:04:43.606624+0000","last_became_peered":"2024-02-05T08:04:43.606624+0000","last_unstale":"2024-02-05T08:10:23.781122+0000","last_undegraded":"2024-02-05T08:10:23.781122+0000","last_fullsized":"2024-02-05T08:10:23.781122+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:07:42.729538+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"150'13630","reported_seq":13885,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778845+0000","last_change":"2024-02-05T08:09:35.129348+0000","last_active":"2024-02-05T08:10:23.778845+0000","last_peered":"2024-02-05T08:10:23.778845+0000","last_clean":"2024-02-05T08:10:23.778845+0000","last_became_active":"2024-02-05T08:06:39.556800+0000","last_became_peered":"2024-02-05T08:06:39.556800+0000","last_unstale":"2024-02-05T08:10:23.778845+0000","last_undegraded":"2024-02-05T08:10:23.778845+0000","last_fullsized":"2024-02-05T08:10:23.778845+0000","mapping_epoch":97,"log_start":"135'9400","ondisk_log_start":"135'9400","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'11079","last_scrub_stamp":"2024-02-05T08:09:35.129197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:35.129197+0000","objects_scrubbed":2569,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:53.986660+0000","scrub_duration":999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13630,"num_write_kb":436160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":198,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851308+0000","last_change":"2024-02-05T08:08:41.139144+0000","last_active":"2024-02-05T08:10:23.851308+0000","last_peered":"2024-02-05T08:10:23.851308+0000","last_clean":"2024-02-05T08:10:23.851308+0000","last_became_active":"2024-02-05T08:06:39.551759+0000","last_became_peered":"2024-02-05T08:06:39.551759+0000","last_unstale":"2024-02-05T08:10:23.851308+0000","last_undegraded":"2024-02-05T08:10:23.851308+0000","last_fullsized":"2024-02-05T08:10:23.851308+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:41.139067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:41.139067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":209,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779286+0000","last_change":"2024-02-05T08:07:55.497839+0000","last_active":"2024-02-05T08:10:23.779286+0000","last_peered":"2024-02-05T08:10:23.779286+0000","last_clean":"2024-02-05T08:10:23.779286+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:10:23.779286+0000","last_undegraded":"2024-02-05T08:10:23.779286+0000","last_fullsized":"2024-02-05T08:10:23.779286+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:07:55.497748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:55.497748+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":239,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850080+0000","last_change":"2024-02-05T08:08:33.007515+0000","last_active":"2024-02-05T08:10:23.850080+0000","last_peered":"2024-02-05T08:10:23.850080+0000","last_clean":"2024-02-05T08:10:23.850080+0000","last_became_active":"2024-02-05T08:06:39.546607+0000","last_became_peered":"2024-02-05T08:06:39.546607+0000","last_unstale":"2024-02-05T08:10:23.850080+0000","last_undegraded":"2024-02-05T08:10:23.850080+0000","last_fullsized":"2024-02-05T08:10:23.850080+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:08:33.007404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:33.007404+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":285,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778905+0000","last_change":"2024-02-05T08:09:20.414916+0000","last_active":"2024-02-05T08:10:23.778905+0000","last_peered":"2024-02-05T08:10:23.778905+0000","last_clean":"2024-02-05T08:10:23.778905+0000","last_became_active":"2024-02-05T08:06:39.556562+0000","last_became_peered":"2024-02-05T08:06:39.556562+0000","last_unstale":"2024-02-05T08:10:23.778905+0000","last_undegraded":"2024-02-05T08:10:23.778905+0000","last_fullsized":"2024-02-05T08:10:23.778905+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:09:20.414837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:20.414837+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:36.384339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"150'13362","reported_seq":13544,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779750+0000","last_change":"2024-02-05T08:08:47.609361+0000","last_active":"2024-02-05T08:10:23.779750+0000","last_peered":"2024-02-05T08:10:23.779750+0000","last_clean":"2024-02-05T08:10:23.779750+0000","last_became_active":"2024-02-05T08:06:37.621544+0000","last_became_peered":"2024-02-05T08:06:37.621544+0000","last_unstale":"2024-02-05T08:10:23.779750+0000","last_undegraded":"2024-02-05T08:10:23.779750+0000","last_fullsized":"2024-02-05T08:10:23.779750+0000","mapping_epoch":95,"log_start":"134'9100","ondisk_log_start":"134'9100","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8264","last_scrub_stamp":"2024-02-05T08:08:47.608801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:47.608801+0000","objects_scrubbed":5157,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13362,"num_write_kb":427584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779366+0000","last_change":"2024-02-05T08:09:25.266619+0000","last_active":"2024-02-05T08:10:23.779366+0000","last_peered":"2024-02-05T08:10:23.779366+0000","last_clean":"2024-02-05T08:10:23.779366+0000","last_became_active":"2024-02-05T08:04:47.586044+0000","last_became_peered":"2024-02-05T08:04:47.586044+0000","last_unstale":"2024-02-05T08:10:23.779366+0000","last_undegraded":"2024-02-05T08:10:23.779366+0000","last_fullsized":"2024-02-05T08:10:23.779366+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.266513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.266513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:48.723648+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":205,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779427+0000","last_change":"2024-02-05T08:09:39.244280+0000","last_active":"2024-02-05T08:10:23.779427+0000","last_peered":"2024-02-05T08:10:23.779427+0000","last_clean":"2024-02-05T08:10:23.779427+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:10:23.779427+0000","last_undegraded":"2024-02-05T08:10:23.779427+0000","last_fullsized":"2024-02-05T08:10:23.779427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:39.244179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.244179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:55.149887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":215,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850878+0000","last_change":"2024-02-05T08:09:31.469380+0000","last_active":"2024-02-05T08:10:23.850878+0000","last_peered":"2024-02-05T08:10:23.850878+0000","last_clean":"2024-02-05T08:10:23.850878+0000","last_became_active":"2024-02-05T08:04:43.556580+0000","last_became_peered":"2024-02-05T08:04:43.556580+0000","last_unstale":"2024-02-05T08:10:23.850878+0000","last_undegraded":"2024-02-05T08:10:23.850878+0000","last_fullsized":"2024-02-05T08:10:23.850878+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:09:31.469295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:31.469295+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.626653+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.849661+0000","last_change":"2024-02-05T08:09:23.533364+0000","last_active":"2024-02-05T08:10:23.849661+0000","last_peered":"2024-02-05T08:10:23.849661+0000","last_clean":"2024-02-05T08:10:23.849661+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:10:23.849661+0000","last_undegraded":"2024-02-05T08:10:23.849661+0000","last_fullsized":"2024-02-05T08:10:23.849661+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:23.533259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_clean_scrub_stamp":"2024-02-05T08:09:23.533259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:35.020206+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"150'13538","reported_seq":13714,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.849845+0000","last_change":"2024-02-05T08:09:19.227093+0000","last_active":"2024-02-05T08:10:23.849845+0000","last_peered":"2024-02-05T08:10:23.849845+0000","last_clean":"2024-02-05T08:10:23.849845+0000","last_became_active":"2024-02-05T08:06:37.619775+0000","last_became_peered":"2024-02-05T08:06:37.619775+0000","last_unstale":"2024-02-05T08:10:23.849845+0000","last_undegraded":"2024-02-05T08:10:23.849845+0000","last_fullsized":"2024-02-05T08:10:23.849845+0000","mapping_epoch":95,"log_start":"133'8700","ondisk_log_start":"133'8700","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'10065","last_scrub_stamp":"2024-02-05T08:09:19.226651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.226651+0000","objects_scrubbed":3510,"log_size":4838,"log_dups_size":0,"ondisk_log_size":4838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:24.312423+0000","scrub_duration":1408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13538,"num_write_kb":433216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":193,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778495+0000","last_change":"2024-02-05T08:10:21.221033+0000","last_active":"2024-02-05T08:10:23.778495+0000","last_peered":"2024-02-05T08:10:23.778495+0000","last_clean":"2024-02-05T08:10:23.778495+0000","last_became_active":"2024-02-05T08:04:47.587082+0000","last_became_peered":"2024-02-05T08:04:47.587082+0000","last_unstale":"2024-02-05T08:10:23.778495+0000","last_undegraded":"2024-02-05T08:10:23.778495+0000","last_fullsized":"2024-02-05T08:10:23.778495+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:21.220941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:21.220941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:37.669661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850311+0000","last_change":"2024-02-05T08:09:29.579918+0000","last_active":"2024-02-05T08:10:23.850311+0000","last_peered":"2024-02-05T08:10:23.850311+0000","last_clean":"2024-02-05T08:10:23.850311+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:10:23.850311+0000","last_undegraded":"2024-02-05T08:10:23.850311+0000","last_fullsized":"2024-02-05T08:10:23.850311+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:29.578521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:29.578521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:44.673091+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":211,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779272+0000","last_change":"2024-02-05T08:09:31.480648+0000","last_active":"2024-02-05T08:10:23.779272+0000","last_peered":"2024-02-05T08:10:23.779272+0000","last_clean":"2024-02-05T08:10:23.779272+0000","last_became_active":"2024-02-05T08:06:39.554318+0000","last_became_peered":"2024-02-05T08:06:39.554318+0000","last_unstale":"2024-02-05T08:10:23.779272+0000","last_undegraded":"2024-02-05T08:10:23.779272+0000","last_fullsized":"2024-02-05T08:10:23.779272+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:31.480558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:31.480558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:42.719453+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":210,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850487+0000","last_change":"2024-02-05T08:09:34.530885+0000","last_active":"2024-02-05T08:10:23.850487+0000","last_peered":"2024-02-05T08:10:23.850487+0000","last_clean":"2024-02-05T08:10:23.850487+0000","last_became_active":"2024-02-05T08:06:39.552819+0000","last_became_peered":"2024-02-05T08:06:39.552819+0000","last_unstale":"2024-02-05T08:10:23.850487+0000","last_undegraded":"2024-02-05T08:10:23.850487+0000","last_fullsized":"2024-02-05T08:10:23.850487+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:09:34.530767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:34.530767+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:35.663461+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"150'13762","reported_seq":13968,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.849462+0000","last_change":"2024-02-05T08:10:17.795809+0000","last_active":"2024-02-05T08:10:23.849462+0000","last_peered":"2024-02-05T08:10:23.849462+0000","last_clean":"2024-02-05T08:10:23.849462+0000","last_became_active":"2024-02-05T08:06:37.623334+0000","last_became_peered":"2024-02-05T08:06:37.623334+0000","last_unstale":"2024-02-05T08:10:23.849462+0000","last_undegraded":"2024-02-05T08:10:23.849462+0000","last_fullsized":"2024-02-05T08:10:23.849462+0000","mapping_epoch":95,"log_start":"130'8400","ondisk_log_start":"130'8400","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'13521","last_scrub_stamp":"2024-02-05T08:10:17.795604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:10:17.795604+0000","objects_scrubbed":245,"log_size":5362,"log_dups_size":0,"ondisk_log_size":5362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:47.794005+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13762,"num_write_kb":440384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.849837+0000","last_change":"2024-02-05T08:07:55.608340+0000","last_active":"2024-02-05T08:10:23.849837+0000","last_peered":"2024-02-05T08:10:23.849837+0000","last_clean":"2024-02-05T08:10:23.849837+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:10:23.849837+0000","last_undegraded":"2024-02-05T08:10:23.849837+0000","last_fullsized":"2024-02-05T08:10:23.849837+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:55.608083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:55.608083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850240+0000","last_change":"2024-02-05T08:09:21.583142+0000","last_active":"2024-02-05T08:10:23.850240+0000","last_peered":"2024-02-05T08:10:23.850240+0000","last_clean":"2024-02-05T08:10:23.850240+0000","last_became_active":"2024-02-05T08:04:45.567341+0000","last_became_peered":"2024-02-05T08:04:45.567341+0000","last_unstale":"2024-02-05T08:10:23.850240+0000","last_undegraded":"2024-02-05T08:10:23.850240+0000","last_fullsized":"2024-02-05T08:10:23.850240+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:21.582910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.582910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:23.887947+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":202,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850606+0000","last_change":"2024-02-05T08:07:45.721633+0000","last_active":"2024-02-05T08:10:23.850606+0000","last_peered":"2024-02-05T08:10:23.850606+0000","last_clean":"2024-02-05T08:10:23.850606+0000","last_became_active":"2024-02-05T08:06:39.558993+0000","last_became_peered":"2024-02-05T08:06:39.558993+0000","last_unstale":"2024-02-05T08:10:23.850606+0000","last_undegraded":"2024-02-05T08:10:23.850606+0000","last_fullsized":"2024-02-05T08:10:23.850606+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:45.721527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:45.721527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":208,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850802+0000","last_change":"2024-02-05T08:07:50.544800+0000","last_active":"2024-02-05T08:10:23.850802+0000","last_peered":"2024-02-05T08:10:23.850802+0000","last_clean":"2024-02-05T08:10:23.850802+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:10:23.850802+0000","last_undegraded":"2024-02-05T08:10:23.850802+0000","last_fullsized":"2024-02-05T08:10:23.850802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:07:50.544673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:07:50.544673+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"150'15693","reported_seq":15909,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851328+0000","last_change":"2024-02-05T08:09:51.200716+0000","last_active":"2024-02-05T08:10:23.851328+0000","last_peered":"2024-02-05T08:10:23.851328+0000","last_clean":"2024-02-05T08:10:23.851328+0000","last_became_active":"2024-02-05T08:06:39.561430+0000","last_became_peered":"2024-02-05T08:06:39.561430+0000","last_unstale":"2024-02-05T08:10:23.851328+0000","last_undegraded":"2024-02-05T08:10:23.851328+0000","last_fullsized":"2024-02-05T08:10:23.851328+0000","mapping_epoch":97,"log_start":"141'10900","ondisk_log_start":"141'10900","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'13917","last_scrub_stamp":"2024-02-05T08:09:51.200561+0000","last_deep_scrub":"144'13917","last_deep_scrub_stamp":"2024-02-05T08:09:51.200561+0000","last_clean_scrub_stamp":"2024-02-05T08:09:51.200561+0000","objects_scrubbed":2109,"log_size":4793,"log_dups_size":0,"ondisk_log_size":4793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":17,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:18.777091+0000","scrub_duration":16228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13584,"num_write_kb":434688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.780585+0000","last_change":"2024-02-05T08:10:17.009112+0000","last_active":"2024-02-05T08:10:23.780585+0000","last_peered":"2024-02-05T08:10:23.780585+0000","last_clean":"2024-02-05T08:10:23.780585+0000","last_became_active":"2024-02-05T08:04:47.588395+0000","last_became_peered":"2024-02-05T08:04:47.588395+0000","last_unstale":"2024-02-05T08:10:23.780585+0000","last_undegraded":"2024-02-05T08:10:23.780585+0000","last_fullsized":"2024-02-05T08:10:23.780585+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:17.008882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:17.008882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:17.659938+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":226,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851191+0000","last_change":"2024-02-05T08:10:18.580307+0000","last_active":"2024-02-05T08:10:23.851191+0000","last_peered":"2024-02-05T08:10:23.851191+0000","last_clean":"2024-02-05T08:10:23.851191+0000","last_became_active":"2024-02-05T08:06:39.560631+0000","last_became_peered":"2024-02-05T08:06:39.560631+0000","last_unstale":"2024-02-05T08:10:23.851191+0000","last_undegraded":"2024-02-05T08:10:23.851191+0000","last_fullsized":"2024-02-05T08:10:23.851191+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:10:18.580209+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.580209+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:45.524064+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":202,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850370+0000","last_change":"2024-02-05T08:08:34.373002+0000","last_active":"2024-02-05T08:10:23.850370+0000","last_peered":"2024-02-05T08:10:23.850370+0000","last_clean":"2024-02-05T08:10:23.850370+0000","last_became_active":"2024-02-05T08:06:39.563051+0000","last_became_peered":"2024-02-05T08:06:39.563051+0000","last_unstale":"2024-02-05T08:10:23.850370+0000","last_undegraded":"2024-02-05T08:10:23.850370+0000","last_fullsized":"2024-02-05T08:10:23.850370+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:34.372891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:08:34.372891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"85'38","reported_seq":329,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779627+0000","last_change":"2024-02-05T08:09:35.108648+0000","last_active":"2024-02-05T08:10:23.779627+0000","last_peered":"2024-02-05T08:10:23.779627+0000","last_clean":"2024-02-05T08:10:23.779627+0000","last_became_active":"2024-02-05T08:04:43.554536+0000","last_became_peered":"2024-02-05T08:04:43.554536+0000","last_unstale":"2024-02-05T08:10:23.779627+0000","last_undegraded":"2024-02-05T08:10:23.779627+0000","last_fullsized":"2024-02-05T08:10:23.779627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:09:35.108489+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:09:35.108489+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:50.058243+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851279+0000","last_change":"2024-02-05T08:10:19.265168+0000","last_active":"2024-02-05T08:10:23.851279+0000","last_peered":"2024-02-05T08:10:23.851279+0000","last_clean":"2024-02-05T08:10:23.851279+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:10:23.851279+0000","last_undegraded":"2024-02-05T08:10:23.851279+0000","last_fullsized":"2024-02-05T08:10:23.851279+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:19.265057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:19.265057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:45.883831+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"150'13954","reported_seq":14120,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850635+0000","last_change":"2024-02-05T08:09:26.200715+0000","last_active":"2024-02-05T08:10:23.850635+0000","last_peered":"2024-02-05T08:10:23.850635+0000","last_clean":"2024-02-05T08:10:23.850635+0000","last_became_active":"2024-02-05T08:06:37.618966+0000","last_became_peered":"2024-02-05T08:06:37.618966+0000","last_unstale":"2024-02-05T08:10:23.850635+0000","last_undegraded":"2024-02-05T08:10:23.850635+0000","last_fullsized":"2024-02-05T08:10:23.850635+0000","mapping_epoch":95,"log_start":"134'9100","ondisk_log_start":"134'9100","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10764","last_scrub_stamp":"2024-02-05T08:09:26.200471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:26.200471+0000","objects_scrubbed":3228,"log_size":4854,"log_dups_size":0,"ondisk_log_size":4854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:26.980375+0000","scrub_duration":1266,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13954,"num_write_kb":446528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":191,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779388+0000","last_change":"2024-02-05T08:08:39.475407+0000","last_active":"2024-02-05T08:10:23.779388+0000","last_peered":"2024-02-05T08:10:23.779388+0000","last_clean":"2024-02-05T08:10:23.779388+0000","last_became_active":"2024-02-05T08:06:39.555147+0000","last_became_peered":"2024-02-05T08:06:39.555147+0000","last_unstale":"2024-02-05T08:10:23.779388+0000","last_undegraded":"2024-02-05T08:10:23.779388+0000","last_fullsized":"2024-02-05T08:10:23.779388+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:39.475316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.475316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":214,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778125+0000","last_change":"2024-02-05T08:08:41.006239+0000","last_active":"2024-02-05T08:10:23.778125+0000","last_peered":"2024-02-05T08:10:23.778125+0000","last_clean":"2024-02-05T08:10:23.778125+0000","last_became_active":"2024-02-05T08:06:39.560285+0000","last_became_peered":"2024-02-05T08:06:39.560285+0000","last_unstale":"2024-02-05T08:10:23.778125+0000","last_undegraded":"2024-02-05T08:10:23.778125+0000","last_fullsized":"2024-02-05T08:10:23.778125+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:08:41.005907+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:08:41.005907+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":214,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.849640+0000","last_change":"2024-02-05T08:08:44.750119+0000","last_active":"2024-02-05T08:10:23.849640+0000","last_peered":"2024-02-05T08:10:23.849640+0000","last_clean":"2024-02-05T08:10:23.849640+0000","last_became_active":"2024-02-05T08:04:40.529073+0000","last_became_peered":"2024-02-05T08:04:40.529073+0000","last_unstale":"2024-02-05T08:10:23.849640+0000","last_undegraded":"2024-02-05T08:10:23.849640+0000","last_fullsized":"2024-02-05T08:10:23.849640+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:44.750015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.750015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":205,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:24.681809+0000","last_change":"2024-02-05T08:10:24.681839+0000","last_active":"2024-02-05T08:10:24.681809+0000","last_peered":"2024-02-05T08:10:24.681809+0000","last_clean":"2024-02-05T08:10:24.681809+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:10:24.681809+0000","last_undegraded":"2024-02-05T08:10:24.681809+0000","last_fullsized":"2024-02-05T08:10:24.681809+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:10:24.681753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:24.681753+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:25.001232+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851109+0000","last_change":"2024-02-05T08:09:25.547689+0000","last_active":"2024-02-05T08:10:23.851109+0000","last_peered":"2024-02-05T08:10:23.851109+0000","last_clean":"2024-02-05T08:10:23.851109+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:10:23.851109+0000","last_undegraded":"2024-02-05T08:10:23.851109+0000","last_fullsized":"2024-02-05T08:10:23.851109+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.547581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.547581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:55.227516+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"150'13726","reported_seq":13882,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850897+0000","last_change":"2024-02-05T08:09:37.898638+0000","last_active":"2024-02-05T08:10:23.850897+0000","last_peered":"2024-02-05T08:10:23.850897+0000","last_clean":"2024-02-05T08:10:23.850897+0000","last_became_active":"2024-02-05T08:06:37.622047+0000","last_became_peered":"2024-02-05T08:06:37.622047+0000","last_unstale":"2024-02-05T08:10:23.850897+0000","last_undegraded":"2024-02-05T08:10:23.850897+0000","last_fullsized":"2024-02-05T08:10:23.850897+0000","mapping_epoch":95,"log_start":"133'8900","ondisk_log_start":"133'8900","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'11226","last_scrub_stamp":"2024-02-05T08:09:37.898513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:37.898513+0000","objects_scrubbed":2513,"log_size":4826,"log_dups_size":0,"ondisk_log_size":4826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:04.919652+0000","scrub_duration":880,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13726,"num_write_kb":439232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850886+0000","last_change":"2024-02-05T08:08:34.051757+0000","last_active":"2024-02-05T08:10:23.850886+0000","last_peered":"2024-02-05T08:10:23.850886+0000","last_clean":"2024-02-05T08:10:23.850886+0000","last_became_active":"2024-02-05T08:04:47.586358+0000","last_became_peered":"2024-02-05T08:04:47.586358+0000","last_unstale":"2024-02-05T08:10:23.850886+0000","last_undegraded":"2024-02-05T08:10:23.850886+0000","last_fullsized":"2024-02-05T08:10:23.850886+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:34.051658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:34.051658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":469,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778694+0000","last_change":"2024-02-05T08:08:49.423194+0000","last_active":"2024-02-05T08:10:23.778694+0000","last_peered":"2024-02-05T08:10:23.778694+0000","last_clean":"2024-02-05T08:10:23.778694+0000","last_became_active":"2024-02-05T08:06:42.566771+0000","last_became_peered":"2024-02-05T08:06:42.566771+0000","last_unstale":"2024-02-05T08:10:23.778694+0000","last_undegraded":"2024-02-05T08:10:23.778694+0000","last_fullsized":"2024-02-05T08:10:23.778694+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:08:49.423065+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:08:49.423065+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":202,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779600+0000","last_change":"2024-02-05T08:08:39.073558+0000","last_active":"2024-02-05T08:10:23.779600+0000","last_peered":"2024-02-05T08:10:23.779600+0000","last_clean":"2024-02-05T08:10:23.779600+0000","last_became_active":"2024-02-05T08:06:39.553704+0000","last_became_peered":"2024-02-05T08:06:39.553704+0000","last_unstale":"2024-02-05T08:10:23.779600+0000","last_undegraded":"2024-02-05T08:10:23.779600+0000","last_fullsized":"2024-02-05T08:10:23.779600+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.071354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":242,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851786+0000","last_change":"2024-02-05T08:10:19.453291+0000","last_active":"2024-02-05T08:10:23.851786+0000","last_peered":"2024-02-05T08:10:23.851786+0000","last_clean":"2024-02-05T08:10:23.851786+0000","last_became_active":"2024-02-05T08:06:39.563373+0000","last_became_peered":"2024-02-05T08:06:39.563373+0000","last_unstale":"2024-02-05T08:10:23.851786+0000","last_undegraded":"2024-02-05T08:10:23.851786+0000","last_fullsized":"2024-02-05T08:10:23.851786+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-02-05T08:10:19.453162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:19.453162+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:39.266463+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":189,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779154+0000","last_change":"2024-02-05T08:08:38.472753+0000","last_active":"2024-02-05T08:10:23.779154+0000","last_peered":"2024-02-05T08:10:23.779154+0000","last_clean":"2024-02-05T08:10:23.779154+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:10:23.779154+0000","last_undegraded":"2024-02-05T08:10:23.779154+0000","last_fullsized":"2024-02-05T08:10:23.779154+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:38.472647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:38.472647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"150'13698","reported_seq":13885,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779032+0000","last_change":"2024-02-05T08:08:42.646048+0000","last_active":"2024-02-05T08:10:23.779032+0000","last_peered":"2024-02-05T08:10:23.779032+0000","last_clean":"2024-02-05T08:10:23.779032+0000","last_became_active":"2024-02-05T08:06:37.623983+0000","last_became_peered":"2024-02-05T08:06:37.623983+0000","last_unstale":"2024-02-05T08:10:23.779032+0000","last_undegraded":"2024-02-05T08:10:23.779032+0000","last_fullsized":"2024-02-05T08:10:23.779032+0000","mapping_epoch":95,"log_start":"132'8600","ondisk_log_start":"132'8600","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'8153","last_scrub_stamp":"2024-02-05T08:08:42.645901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:42.645901+0000","objects_scrubbed":5574,"log_size":5098,"log_dups_size":0,"ondisk_log_size":5098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13698,"num_write_kb":438336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":185,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779317+0000","last_change":"2024-02-05T08:07:43.938444+0000","last_active":"2024-02-05T08:10:23.779317+0000","last_peered":"2024-02-05T08:10:23.779317+0000","last_clean":"2024-02-05T08:10:23.779317+0000","last_became_active":"2024-02-05T08:04:47.586774+0000","last_became_peered":"2024-02-05T08:04:47.586774+0000","last_unstale":"2024-02-05T08:10:23.779317+0000","last_undegraded":"2024-02-05T08:10:23.779317+0000","last_fullsized":"2024-02-05T08:10:23.779317+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:43.938278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:43.938278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.849629+0000","last_change":"2024-02-05T08:08:38.705016+0000","last_active":"2024-02-05T08:10:23.849629+0000","last_peered":"2024-02-05T08:10:23.849629+0000","last_clean":"2024-02-05T08:10:23.849629+0000","last_became_active":"2024-02-05T08:04:41.537872+0000","last_became_peered":"2024-02-05T08:04:41.537872+0000","last_unstale":"2024-02-05T08:10:23.849629+0000","last_undegraded":"2024-02-05T08:10:23.849629+0000","last_fullsized":"2024-02-05T08:10:23.849629+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:38.704911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:38.704911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":557,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:26.201044+0000","last_change":"2024-02-05T08:09:26.492000+0000","last_active":"2024-02-05T08:10:26.201044+0000","last_peered":"2024-02-05T08:10:26.201044+0000","last_clean":"2024-02-05T08:10:26.201044+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:10:26.201044+0000","last_undegraded":"2024-02-05T08:10:26.201044+0000","last_fullsized":"2024-02-05T08:10:26.201044+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:09:26.491884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:09:26.491884+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:39.146672+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":340,"num_read_kb":335,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":210,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850540+0000","last_change":"2024-02-05T08:09:19.836523+0000","last_active":"2024-02-05T08:10:23.850540+0000","last_peered":"2024-02-05T08:10:23.850540+0000","last_clean":"2024-02-05T08:10:23.850540+0000","last_became_active":"2024-02-05T08:04:43.549687+0000","last_became_peered":"2024-02-05T08:04:43.549687+0000","last_unstale":"2024-02-05T08:10:23.850540+0000","last_undegraded":"2024-02-05T08:10:23.850540+0000","last_fullsized":"2024-02-05T08:10:23.850540+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:09:19.836401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.836401+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:38.806378+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851382+0000","last_change":"2024-02-05T08:08:29.956921+0000","last_active":"2024-02-05T08:10:23.851382+0000","last_peered":"2024-02-05T08:10:23.851382+0000","last_clean":"2024-02-05T08:10:23.851382+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:10:23.851382+0000","last_undegraded":"2024-02-05T08:10:23.851382+0000","last_fullsized":"2024-02-05T08:10:23.851382+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:29.956845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:29.956845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"150'17135","reported_seq":17448,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850159+0000","last_change":"2024-02-05T08:09:18.016743+0000","last_active":"2024-02-05T08:10:23.850159+0000","last_peered":"2024-02-05T08:10:23.850159+0000","last_clean":"2024-02-05T08:10:23.850159+0000","last_became_active":"2024-02-05T08:06:37.624241+0000","last_became_peered":"2024-02-05T08:06:37.624241+0000","last_unstale":"2024-02-05T08:10:23.850159+0000","last_undegraded":"2024-02-05T08:10:23.850159+0000","last_fullsized":"2024-02-05T08:10:23.850159+0000","mapping_epoch":95,"log_start":"134'11602","ondisk_log_start":"134'11602","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'13560","last_scrub_stamp":"2024-02-05T08:09:18.016504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:18.016504+0000","objects_scrubbed":3593,"log_size":5533,"log_dups_size":0,"ondisk_log_size":5533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:30.849106+0000","scrub_duration":742,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13702,"num_write_kb":438464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":199,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778271+0000","last_change":"2024-02-05T08:10:18.609876+0000","last_active":"2024-02-05T08:10:23.778271+0000","last_peered":"2024-02-05T08:10:23.778271+0000","last_clean":"2024-02-05T08:10:23.778271+0000","last_became_active":"2024-02-05T08:06:39.559847+0000","last_became_peered":"2024-02-05T08:06:39.559847+0000","last_unstale":"2024-02-05T08:10:23.778271+0000","last_undegraded":"2024-02-05T08:10:23.778271+0000","last_fullsized":"2024-02-05T08:10:23.778271+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.609767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.609767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:43.852340+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":205,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779501+0000","last_change":"2024-02-05T08:08:44.964475+0000","last_active":"2024-02-05T08:10:23.779501+0000","last_peered":"2024-02-05T08:10:23.779501+0000","last_clean":"2024-02-05T08:10:23.779501+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:10:23.779501+0000","last_undegraded":"2024-02-05T08:10:23.779501+0000","last_fullsized":"2024-02-05T08:10:23.779501+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:44.964357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.964357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":236,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850654+0000","last_change":"2024-02-05T08:10:16.309686+0000","last_active":"2024-02-05T08:10:23.850654+0000","last_peered":"2024-02-05T08:10:23.850654+0000","last_clean":"2024-02-05T08:10:23.850654+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:10:23.850654+0000","last_undegraded":"2024-02-05T08:10:23.850654+0000","last_fullsized":"2024-02-05T08:10:23.850654+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:10:16.309551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:16.309551+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:43.442839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":127,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851207+0000","last_change":"2024-02-05T08:08:31.623904+0000","last_active":"2024-02-05T08:10:23.851207+0000","last_peered":"2024-02-05T08:10:23.851207+0000","last_clean":"2024-02-05T08:10:23.851207+0000","last_became_active":"2024-02-05T08:06:39.558100+0000","last_became_peered":"2024-02-05T08:06:39.558100+0000","last_unstale":"2024-02-05T08:10:23.851207+0000","last_undegraded":"2024-02-05T08:10:23.851207+0000","last_fullsized":"2024-02-05T08:10:23.851207+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:31.623748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:08:31.623748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"150'13728","reported_seq":13917,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850122+0000","last_change":"2024-02-05T08:08:46.770155+0000","last_active":"2024-02-05T08:10:23.850122+0000","last_peered":"2024-02-05T08:10:23.850122+0000","last_clean":"2024-02-05T08:10:23.850122+0000","last_became_active":"2024-02-05T08:06:39.561154+0000","last_became_peered":"2024-02-05T08:06:39.561154+0000","last_unstale":"2024-02-05T08:10:23.850122+0000","last_undegraded":"2024-02-05T08:10:23.850122+0000","last_fullsized":"2024-02-05T08:10:23.850122+0000","mapping_epoch":97,"log_start":"134'9100","ondisk_log_start":"134'9100","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8468","last_scrub_stamp":"2024-02-05T08:08:46.770050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:46.770050+0000","objects_scrubbed":5286,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1335,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13728,"num_write_kb":439296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.849552+0000","last_change":"2024-02-05T08:07:34.484047+0000","last_active":"2024-02-05T08:10:23.849552+0000","last_peered":"2024-02-05T08:10:23.849552+0000","last_clean":"2024-02-05T08:10:23.849552+0000","last_became_active":"2024-02-05T08:04:47.583866+0000","last_became_peered":"2024-02-05T08:04:47.583866+0000","last_unstale":"2024-02-05T08:10:23.849552+0000","last_undegraded":"2024-02-05T08:10:23.849552+0000","last_fullsized":"2024-02-05T08:10:23.849552+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:34.483937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:34.483937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":204,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851261+0000","last_change":"2024-02-05T08:09:40.402414+0000","last_active":"2024-02-05T08:10:23.851261+0000","last_peered":"2024-02-05T08:10:23.851261+0000","last_clean":"2024-02-05T08:10:23.851261+0000","last_became_active":"2024-02-05T08:04:41.537818+0000","last_became_peered":"2024-02-05T08:04:41.537818+0000","last_unstale":"2024-02-05T08:10:23.851261+0000","last_undegraded":"2024-02-05T08:10:23.851261+0000","last_fullsized":"2024-02-05T08:10:23.851261+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:40.402311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:40.402311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.704969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":216,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.778866+0000","last_change":"2024-02-05T08:10:21.554686+0000","last_active":"2024-02-05T08:10:23.778866+0000","last_peered":"2024-02-05T08:10:23.778866+0000","last_clean":"2024-02-05T08:10:23.778866+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:10:23.778866+0000","last_undegraded":"2024-02-05T08:10:23.778866+0000","last_fullsized":"2024-02-05T08:10:23.778866+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:10:21.554300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:21.554300+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:39.632000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":276,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850941+0000","last_change":"2024-02-05T08:07:48.538033+0000","last_active":"2024-02-05T08:10:23.850941+0000","last_peered":"2024-02-05T08:10:23.850941+0000","last_clean":"2024-02-05T08:10:23.850941+0000","last_became_active":"2024-02-05T08:06:39.554860+0000","last_became_peered":"2024-02-05T08:06:39.554860+0000","last_unstale":"2024-02-05T08:10:23.850941+0000","last_undegraded":"2024-02-05T08:10:23.850941+0000","last_fullsized":"2024-02-05T08:10:23.850941+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:07:48.537913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:48.537913+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"150'13848","reported_seq":14030,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779448+0000","last_change":"2024-02-05T08:09:22.744155+0000","last_active":"2024-02-05T08:10:23.779448+0000","last_peered":"2024-02-05T08:10:23.779448+0000","last_clean":"2024-02-05T08:10:23.779448+0000","last_became_active":"2024-02-05T08:06:37.621499+0000","last_became_peered":"2024-02-05T08:06:37.621499+0000","last_unstale":"2024-02-05T08:10:23.779448+0000","last_undegraded":"2024-02-05T08:10:23.779448+0000","last_fullsized":"2024-02-05T08:10:23.779448+0000","mapping_epoch":95,"log_start":"135'9600","ondisk_log_start":"135'9600","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10467","last_scrub_stamp":"2024-02-05T08:09:22.743920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.743920+0000","objects_scrubbed":3450,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:42.513741+0000","scrub_duration":2390,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13848,"num_write_kb":443136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":191,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779037+0000","last_change":"2024-02-05T08:08:43.475720+0000","last_active":"2024-02-05T08:10:23.779037+0000","last_peered":"2024-02-05T08:10:23.779037+0000","last_clean":"2024-02-05T08:10:23.779037+0000","last_became_active":"2024-02-05T08:06:39.554165+0000","last_became_peered":"2024-02-05T08:06:39.554165+0000","last_unstale":"2024-02-05T08:10:23.779037+0000","last_undegraded":"2024-02-05T08:10:23.779037+0000","last_fullsized":"2024-02-05T08:10:23.779037+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:43.475636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:43.475636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":197,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850645+0000","last_change":"2024-02-05T08:07:25.394409+0000","last_active":"2024-02-05T08:10:23.850645+0000","last_peered":"2024-02-05T08:10:23.850645+0000","last_clean":"2024-02-05T08:10:23.850645+0000","last_became_active":"2024-02-05T08:04:41.534433+0000","last_became_peered":"2024-02-05T08:04:41.534433+0000","last_unstale":"2024-02-05T08:10:23.850645+0000","last_undegraded":"2024-02-05T08:10:23.850645+0000","last_fullsized":"2024-02-05T08:10:23.850645+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:07:25.394122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:25.394122+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":235,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850656+0000","last_change":"2024-02-05T08:10:16.458012+0000","last_active":"2024-02-05T08:10:23.850656+0000","last_peered":"2024-02-05T08:10:23.850656+0000","last_clean":"2024-02-05T08:10:23.850656+0000","last_became_active":"2024-02-05T08:06:39.566434+0000","last_became_peered":"2024-02-05T08:06:39.566434+0000","last_unstale":"2024-02-05T08:10:23.850656+0000","last_undegraded":"2024-02-05T08:10:23.850656+0000","last_fullsized":"2024-02-05T08:10:23.850656+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-02-05T08:10:16.457895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:16.457895+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:40.130069+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":266,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779147+0000","last_change":"2024-02-05T08:07:47.765640+0000","last_active":"2024-02-05T08:10:23.779147+0000","last_peered":"2024-02-05T08:10:23.779147+0000","last_clean":"2024-02-05T08:10:23.779147+0000","last_became_active":"2024-02-05T08:04:45.562597+0000","last_became_peered":"2024-02-05T08:04:45.562597+0000","last_unstale":"2024-02-05T08:10:23.779147+0000","last_undegraded":"2024-02-05T08:10:23.779147+0000","last_fullsized":"2024-02-05T08:10:23.779147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:07:47.765523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:47.765523+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"150'13822","reported_seq":14009,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779453+0000","last_change":"2024-02-05T08:08:44.289281+0000","last_active":"2024-02-05T08:10:23.779453+0000","last_peered":"2024-02-05T08:10:23.779453+0000","last_clean":"2024-02-05T08:10:23.779453+0000","last_became_active":"2024-02-05T08:06:37.621439+0000","last_became_peered":"2024-02-05T08:06:37.621439+0000","last_unstale":"2024-02-05T08:10:23.779453+0000","last_undegraded":"2024-02-05T08:10:23.779453+0000","last_fullsized":"2024-02-05T08:10:23.779453+0000","mapping_epoch":95,"log_start":"135'9600","ondisk_log_start":"135'9600","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8323","last_scrub_stamp":"2024-02-05T08:08:44.289025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.289025+0000","objects_scrubbed":5548,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13822,"num_write_kb":442304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850497+0000","last_change":"2024-02-05T08:10:15.488463+0000","last_active":"2024-02-05T08:10:23.850497+0000","last_peered":"2024-02-05T08:10:23.850497+0000","last_clean":"2024-02-05T08:10:23.850497+0000","last_became_active":"2024-02-05T08:04:47.582776+0000","last_became_peered":"2024-02-05T08:04:47.582776+0000","last_unstale":"2024-02-05T08:10:23.850497+0000","last_undegraded":"2024-02-05T08:10:23.850497+0000","last_fullsized":"2024-02-05T08:10:23.850497+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:15.488380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:15.488380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:31.489301+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":210,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.780660+0000","last_change":"2024-02-05T08:09:36.111657+0000","last_active":"2024-02-05T08:10:23.780660+0000","last_peered":"2024-02-05T08:10:23.780660+0000","last_clean":"2024-02-05T08:10:23.780660+0000","last_became_active":"2024-02-05T08:04:41.540818+0000","last_became_peered":"2024-02-05T08:04:41.540818+0000","last_unstale":"2024-02-05T08:10:23.780660+0000","last_undegraded":"2024-02-05T08:10:23.780660+0000","last_fullsized":"2024-02-05T08:10:23.780660+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:09:36.111537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.111537+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.119806+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":232,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779017+0000","last_change":"2024-02-05T08:09:33.222821+0000","last_active":"2024-02-05T08:10:23.779017+0000","last_peered":"2024-02-05T08:10:23.779017+0000","last_clean":"2024-02-05T08:10:23.779017+0000","last_became_active":"2024-02-05T08:04:43.607546+0000","last_became_peered":"2024-02-05T08:04:43.607546+0000","last_unstale":"2024-02-05T08:10:23.779017+0000","last_undegraded":"2024-02-05T08:10:23.779017+0000","last_fullsized":"2024-02-05T08:10:23.779017+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-05T08:09:33.222532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:33.222532+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:53.393561+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":188,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850387+0000","last_change":"2024-02-05T08:09:20.449787+0000","last_active":"2024-02-05T08:10:23.850387+0000","last_peered":"2024-02-05T08:10:23.850387+0000","last_clean":"2024-02-05T08:10:23.850387+0000","last_became_active":"2024-02-05T08:04:45.561639+0000","last_became_peered":"2024-02-05T08:04:45.561639+0000","last_unstale":"2024-02-05T08:10:23.850387+0000","last_undegraded":"2024-02-05T08:10:23.850387+0000","last_fullsized":"2024-02-05T08:10:23.850387+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:20.449711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:20.449711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"150'13706","reported_seq":13921,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779099+0000","last_change":"2024-02-05T08:09:19.688660+0000","last_active":"2024-02-05T08:10:23.779099+0000","last_peered":"2024-02-05T08:10:23.779099+0000","last_clean":"2024-02-05T08:10:23.779099+0000","last_became_active":"2024-02-05T08:06:39.556908+0000","last_became_peered":"2024-02-05T08:06:39.556908+0000","last_unstale":"2024-02-05T08:10:23.779099+0000","last_undegraded":"2024-02-05T08:10:23.779099+0000","last_fullsized":"2024-02-05T08:10:23.779099+0000","mapping_epoch":97,"log_start":"135'9500","ondisk_log_start":"135'9500","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'10128","last_scrub_stamp":"2024-02-05T08:09:19.688502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.688502+0000","objects_scrubbed":3625,"log_size":4206,"log_dups_size":0,"ondisk_log_size":4206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:37.670381+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13706,"num_write_kb":438592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":192,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850684+0000","last_change":"2024-02-05T08:08:43.142178+0000","last_active":"2024-02-05T08:10:23.850684+0000","last_peered":"2024-02-05T08:10:23.850684+0000","last_clean":"2024-02-05T08:10:23.850684+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:10:23.850684+0000","last_undegraded":"2024-02-05T08:10:23.850684+0000","last_fullsized":"2024-02-05T08:10:23.850684+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:43.142051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:43.142051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":266,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851898+0000","last_change":"2024-02-05T08:09:17.391826+0000","last_active":"2024-02-05T08:10:23.851898+0000","last_peered":"2024-02-05T08:10:23.851898+0000","last_clean":"2024-02-05T08:10:23.851898+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:10:23.851898+0000","last_undegraded":"2024-02-05T08:10:23.851898+0000","last_fullsized":"2024-02-05T08:10:23.851898+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:09:17.391673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:17.391673+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:31.892417+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779474+0000","last_change":"2024-02-05T08:07:36.643035+0000","last_active":"2024-02-05T08:10:23.779474+0000","last_peered":"2024-02-05T08:10:23.779474+0000","last_clean":"2024-02-05T08:10:23.779474+0000","last_became_active":"2024-02-05T08:04:41.541327+0000","last_became_peered":"2024-02-05T08:04:41.541327+0000","last_unstale":"2024-02-05T08:10:23.779474+0000","last_undegraded":"2024-02-05T08:10:23.779474+0000","last_fullsized":"2024-02-05T08:10:23.779474+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:36.642926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:36.642926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":226,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779686+0000","last_change":"2024-02-05T08:09:17.191152+0000","last_active":"2024-02-05T08:10:23.779686+0000","last_peered":"2024-02-05T08:10:23.779686+0000","last_clean":"2024-02-05T08:10:23.779686+0000","last_became_active":"2024-02-05T08:04:43.555028+0000","last_became_peered":"2024-02-05T08:04:43.555028+0000","last_unstale":"2024-02-05T08:10:23.779686+0000","last_undegraded":"2024-02-05T08:10:23.779686+0000","last_fullsized":"2024-02-05T08:10:23.779686+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:09:17.190290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:17.190290+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"150'13856","reported_seq":14036,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779252+0000","last_change":"2024-02-05T08:07:48.231620+0000","last_active":"2024-02-05T08:10:23.779252+0000","last_peered":"2024-02-05T08:10:23.779252+0000","last_clean":"2024-02-05T08:10:23.779252+0000","last_became_active":"2024-02-05T08:06:39.555144+0000","last_became_peered":"2024-02-05T08:06:39.555144+0000","last_unstale":"2024-02-05T08:10:23.779252+0000","last_undegraded":"2024-02-05T08:10:23.779252+0000","last_fullsized":"2024-02-05T08:10:23.779252+0000","mapping_epoch":97,"log_start":"135'9600","ondisk_log_start":"135'9600","created":95,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'4972","last_scrub_stamp":"2024-02-05T08:07:48.231433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:07:48.231433+0000","objects_scrubbed":4921,"log_size":4256,"log_dups_size":0,"ondisk_log_size":4256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2285,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13856,"num_write_kb":443329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":190,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779277+0000","last_change":"2024-02-05T08:09:32.213156+0000","last_active":"2024-02-05T08:10:23.779277+0000","last_peered":"2024-02-05T08:10:23.779277+0000","last_clean":"2024-02-05T08:10:23.779277+0000","last_became_active":"2024-02-05T08:06:39.558047+0000","last_became_peered":"2024-02-05T08:06:39.558047+0000","last_unstale":"2024-02-05T08:10:23.779277+0000","last_undegraded":"2024-02-05T08:10:23.779277+0000","last_fullsized":"2024-02-05T08:10:23.779277+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:32.213044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:09:32.213044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:54.206176+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":202,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851284+0000","last_change":"2024-02-05T08:10:20.643993+0000","last_active":"2024-02-05T08:10:23.851284+0000","last_peered":"2024-02-05T08:10:23.851284+0000","last_clean":"2024-02-05T08:10:23.851284+0000","last_became_active":"2024-02-05T08:06:39.551933+0000","last_became_peered":"2024-02-05T08:06:39.551933+0000","last_unstale":"2024-02-05T08:10:23.851284+0000","last_undegraded":"2024-02-05T08:10:23.851284+0000","last_fullsized":"2024-02-05T08:10:23.851284+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:20.643918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:20.643918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:27.975950+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":204,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850479+0000","last_change":"2024-02-05T08:09:22.454321+0000","last_active":"2024-02-05T08:10:23.850479+0000","last_peered":"2024-02-05T08:10:23.850479+0000","last_clean":"2024-02-05T08:10:23.850479+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:10:23.850479+0000","last_undegraded":"2024-02-05T08:10:23.850479+0000","last_fullsized":"2024-02-05T08:10:23.850479+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:22.454247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.454247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:32.361376+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":229,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851741+0000","last_change":"2024-02-05T08:08:32.390911+0000","last_active":"2024-02-05T08:10:23.851741+0000","last_peered":"2024-02-05T08:10:23.851741+0000","last_clean":"2024-02-05T08:10:23.851741+0000","last_became_active":"2024-02-05T08:06:39.556433+0000","last_became_peered":"2024-02-05T08:06:39.556433+0000","last_unstale":"2024-02-05T08:10:23.851741+0000","last_undegraded":"2024-02-05T08:10:23.851741+0000","last_fullsized":"2024-02-05T08:10:23.851741+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:08:32.390785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:32.390785+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"150'13862","reported_seq":14060,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851252+0000","last_change":"2024-02-05T08:10:22.671762+0000","last_active":"2024-02-05T08:10:23.851252+0000","last_peered":"2024-02-05T08:10:23.851252+0000","last_clean":"2024-02-05T08:10:23.851252+0000","last_became_active":"2024-02-05T08:06:37.624003+0000","last_became_peered":"2024-02-05T08:06:37.624003+0000","last_unstale":"2024-02-05T08:10:23.851252+0000","last_undegraded":"2024-02-05T08:10:23.851252+0000","last_fullsized":"2024-02-05T08:10:23.851252+0000","mapping_epoch":95,"log_start":"131'8500","ondisk_log_start":"131'8500","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'13862","last_scrub_stamp":"2024-02-05T08:10:22.671661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:10:22.671661+0000","objects_scrubbed":0,"log_size":5362,"log_dups_size":0,"ondisk_log_size":5362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:50.686105+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13862,"num_write_kb":443584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":194,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850859+0000","last_change":"2024-02-05T08:08:45.814521+0000","last_active":"2024-02-05T08:10:23.850859+0000","last_peered":"2024-02-05T08:10:23.850859+0000","last_clean":"2024-02-05T08:10:23.850859+0000","last_became_active":"2024-02-05T08:04:47.585962+0000","last_became_peered":"2024-02-05T08:04:47.585962+0000","last_unstale":"2024-02-05T08:10:23.850859+0000","last_undegraded":"2024-02-05T08:10:23.850859+0000","last_fullsized":"2024-02-05T08:10:23.850859+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:08:45.814384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:45.814384+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":189,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779723+0000","last_change":"2024-02-05T08:08:29.100467+0000","last_active":"2024-02-05T08:10:23.779723+0000","last_peered":"2024-02-05T08:10:23.779723+0000","last_clean":"2024-02-05T08:10:23.779723+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:10:23.779723+0000","last_undegraded":"2024-02-05T08:10:23.779723+0000","last_fullsized":"2024-02-05T08:10:23.779723+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:29.100364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:29.100364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":135,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:25.446784+0000","last_change":"2024-02-05T08:10:25.446805+0000","last_active":"2024-02-05T08:10:25.446784+0000","last_peered":"2024-02-05T08:10:25.446784+0000","last_clean":"2024-02-05T08:10:25.446784+0000","last_became_active":"2024-02-05T08:06:39.559090+0000","last_became_peered":"2024-02-05T08:06:39.559090+0000","last_unstale":"2024-02-05T08:10:25.446784+0000","last_undegraded":"2024-02-05T08:10:25.446784+0000","last_fullsized":"2024-02-05T08:10:25.446784+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:25.446724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:10:25.446724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:50.729956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":220,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851504+0000","last_change":"2024-02-05T08:09:21.434983+0000","last_active":"2024-02-05T08:10:23.851504+0000","last_peered":"2024-02-05T08:10:23.851504+0000","last_clean":"2024-02-05T08:10:23.851504+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:10:23.851504+0000","last_undegraded":"2024-02-05T08:10:23.851504+0000","last_fullsized":"2024-02-05T08:10:23.851504+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-02-05T08:09:21.434891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.434891+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:48.312307+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"150'14086","reported_seq":14271,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851427+0000","last_change":"2024-02-05T08:08:50.574791+0000","last_active":"2024-02-05T08:10:23.851427+0000","last_peered":"2024-02-05T08:10:23.851427+0000","last_clean":"2024-02-05T08:10:23.851427+0000","last_became_active":"2024-02-05T08:06:37.622936+0000","last_became_peered":"2024-02-05T08:06:37.622936+0000","last_unstale":"2024-02-05T08:10:23.851427+0000","last_undegraded":"2024-02-05T08:10:23.851427+0000","last_fullsized":"2024-02-05T08:10:23.851427+0000","mapping_epoch":95,"log_start":"134'9300","ondisk_log_start":"134'9300","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'8798","last_scrub_stamp":"2024-02-05T08:08:50.574449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.607533+0000","last_clean_scrub_stamp":"2024-02-05T08:08:50.574449+0000","objects_scrubbed":5317,"log_size":4786,"log_dups_size":0,"ondisk_log_size":4786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1873,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14086,"num_write_kb":450752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":184,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851866+0000","last_change":"2024-02-05T08:08:49.450529+0000","last_active":"2024-02-05T08:10:23.851866+0000","last_peered":"2024-02-05T08:10:23.851866+0000","last_clean":"2024-02-05T08:10:23.851866+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:10:23.851866+0000","last_undegraded":"2024-02-05T08:10:23.851866+0000","last_fullsized":"2024-02-05T08:10:23.851866+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:49.450400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:49.450400+0000","last_clean_scrub_stamp":"2024-02-05T08:08:49.450400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":276,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.850924+0000","last_change":"2024-02-05T08:08:39.742015+0000","last_active":"2024-02-05T08:10:23.850924+0000","last_peered":"2024-02-05T08:10:23.850924+0000","last_clean":"2024-02-05T08:10:23.850924+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:10:23.850924+0000","last_undegraded":"2024-02-05T08:10:23.850924+0000","last_fullsized":"2024-02-05T08:10:23.850924+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.741871+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":196,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.779140+0000","last_change":"2024-02-05T08:09:22.380135+0000","last_active":"2024-02-05T08:10:23.779140+0000","last_peered":"2024-02-05T08:10:23.779140+0000","last_clean":"2024-02-05T08:10:23.779140+0000","last_became_active":"2024-02-05T08:04:41.533838+0000","last_became_peered":"2024-02-05T08:04:41.533838+0000","last_unstale":"2024-02-05T08:10:23.779140+0000","last_undegraded":"2024-02-05T08:10:23.779140+0000","last_fullsized":"2024-02-05T08:10:23.779140+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:22.380016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.380016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:34.894648+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":225,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-02-05T08:10:23.851071+0000","last_change":"2024-02-05T08:08:35.372047+0000","last_active":"2024-02-05T08:10:23.851071+0000","last_peered":"2024-02-05T08:10:23.851071+0000","last_clean":"2024-02-05T08:10:23.851071+0000","last_became_active":"2024-02-05T08:06:39.555561+0000","last_became_peered":"2024-02-05T08:06:39.555561+0000","last_unstale":"2024-02-05T08:10:23.851071+0000","last_undegraded":"2024-02-05T08:10:23.851071+0000","last_fullsized":"2024-02-05T08:10:23.851071+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:08:35.371853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:35.371853+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":219864,"num_write_kb":7035585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75504,"ondisk_log_size":75504,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":340,"num_read_kb":335,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":1474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":433,"ondisk_log_size":433,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037834,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104716,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93640436,"statfs":{"total":95995035648,"available":95887806464,"internally_reserved":0,"allocated":1380352,"data_stored":1066652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":3.9300000000000002,"5min":1.468,"15min":0.85999999999999999},"min":{"1min":0.23000000000000001,"5min":0.217,"15min":0.217},"max":{"1min":60.127000000000002,"5min":60.127000000000002,"15min":60.127000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":4.2430000000000003,"5min":1.508,"15min":0.85199999999999998},"min":{"1min":0.251,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":58.283000000000001,"5min":58.283000000000001,"15min":58.283000000000001},"last":0.53900000000000003}]},{"osd":1,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":2.3999999999999999,"5min":1.331,"15min":0.81499999999999995},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":20.774000000000001,"5min":20.774000000000001,"15min":20.774000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.702,"5min":1.0249999999999999,"15min":0.70899999999999996},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":20.715,"5min":20.715,"15min":20.715},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.78400000000000003,"15min":0.64900000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.6779999999999999,"5min":7.6779999999999999,"15min":7.6779999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.8109999999999999,"5min":1.1000000000000001,"15min":0.72599999999999998},"min":{"1min":0.26300000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":8.9100000000000001,"5min":8.9100000000000001,"15min":8.9100000000000001},"last":0.57899999999999996}]},{"osd":3,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":5.4989999999999997,"5min":1.859,"15min":0.999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":58.515000000000001,"5min":58.515000000000001,"15min":58.515000000000001},"last":0.5},{"interface":"front","average":{"1min":5.4779999999999998,"5min":1.71,"15min":0.93000000000000005},"min":{"1min":0.307,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":58.225999999999999,"5min":58.225999999999999,"15min":58.225999999999999},"last":0.59099999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.40000000000000002,"15min":0.42499999999999999},"min":{"1min":0.23899999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.49299999999999999,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.41799999999999998,"15min":0.42799999999999999},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.53000000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.44900000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.437,"15min":0.44},"min":{"1min":0.29999999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.90200000000000002,"5min":1.2,"15min":1.2},"last":0.433},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46999999999999997,"15min":0.48399999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.81999999999999995,"5min":0.998,"15min":0.998},"last":0.52500000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.48899999999999999,"15min":0.5},"min":{"1min":0.33700000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.87,"5min":1.256,"15min":1.256},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.33700000000000002,"5min":0.308,"15min":0.308},"max":{"1min":0.56799999999999995,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.54800000000000004}]}]},{"osd":6,"up_from":0,"seq":219043332176,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":185252,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93559900,"statfs":{"total":95995035648,"available":95805337600,"internally_reserved":0,"allocated":1404928,"data_stored":1066360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:10:03 2024","interfaces":[{"interface":"back","average":{"1min":2.7360000000000002,"5min":2.3450000000000002,"15min":0.98899999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.14699999999999999},"max":{"1min":7.7300000000000004,"5min":86.105000000000004,"15min":86.105000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":2.5880000000000001,"5min":2.6930000000000001,"15min":1.1140000000000001},"min":{"1min":0.28999999999999998,"5min":0.22900000000000001,"15min":0.16},"max":{"1min":13.800000000000001,"5min":85.665000000000006,"15min":85.665000000000006},"last":0.47299999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:09:32 2024","interfaces":[{"interface":"back","average":{"1min":1.8380000000000001,"5min":1.972,"15min":0.90600000000000003},"min":{"1min":0.19700000000000001,"5min":0.19500000000000001,"15min":0.17799999999999999},"max":{"1min":6.3079999999999998,"5min":49.030999999999999,"15min":49.030999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":2.2959999999999998,"5min":2.1269999999999998,"15min":0.97799999999999998},"min":{"1min":0.34300000000000003,"5min":0.23100000000000001,"15min":0.21099999999999999},"max":{"1min":13.707000000000001,"5min":48.984000000000002,"15min":48.984000000000002},"last":0.52700000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:09:59 2024","interfaces":[{"interface":"back","average":{"1min":2.3730000000000002,"5min":2.2349999999999999,"15min":1.04},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":12.468999999999999,"5min":49.067999999999998,"15min":49.067999999999998},"last":0.496},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.3130000000000002,"15min":1.0780000000000001},"min":{"1min":0.38,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.3940000000000001,"5min":48.734999999999999,"15min":48.734999999999999},"last":0.65400000000000003}]},{"osd":4,"last update":"Mon Feb 5 08:10:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":1.093,"15min":0.755},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.8129999999999999,"5min":9.4269999999999996,"15min":9.4269999999999996},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.244,"5min":1.1819999999999999,"15min":0.77800000000000002},"min":{"1min":0.30399999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.9300000000000002,"5min":6.3890000000000002,"15min":6.3890000000000002},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.163,"15min":0.751},"min":{"1min":0.35899999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":7.7000000000000002,"5min":9.4559999999999995,"15min":9.4559999999999995},"last":0.752},{"interface":"front","average":{"1min":1.6759999999999999,"5min":1.159,"15min":0.75700000000000001},"min":{"1min":0.40400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":13.776,"5min":13.776,"15min":13.776},"last":0.60999999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:10:18 2024","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.2,"15min":0.79500000000000004},"min":{"1min":0.36799999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.369,"5min":1.1930000000000001,"15min":0.82099999999999995},"min":{"1min":0.41899999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":7.7409999999999997,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":0.73399999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:09:32 2024","interfaces":[{"interface":"back","average":{"1min":2.395,"5min":1.2809999999999999,"15min":0.84099999999999997},"min":{"1min":0.434,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":13.749000000000001,"5min":13.749000000000001,"15min":13.749000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":2.246,"5min":1.1970000000000001,"15min":0.80500000000000005},"min":{"1min":0.56499999999999995,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":6.6029999999999998,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.68999999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163048,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93582104,"statfs":{"total":95995035648,"available":95828074496,"internally_reserved":0,"allocated":1404928,"data_stored":1067186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:10:15 2024","interfaces":[{"interface":"back","average":{"1min":2.9159999999999999,"5min":2.649,"15min":1.1080000000000001},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.152},"max":{"1min":10.186,"5min":66.772999999999996,"15min":66.772999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":2.2599999999999998,"5min":2.0920000000000001,"15min":0.92100000000000004},"min":{"1min":0.33700000000000002,"5min":0.249,"15min":0.17199999999999999},"max":{"1min":6.2060000000000004,"5min":46.198999999999998,"15min":46.198999999999998},"last":0.40300000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:09:36 2024","interfaces":[{"interface":"back","average":{"1min":5.1539999999999999,"5min":1.9490000000000001,"15min":0.89800000000000002},"min":{"1min":0.56100000000000005,"5min":0.252,"15min":0.218},"max":{"1min":46.170999999999999,"5min":46.170999999999999,"15min":46.170999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":4.7699999999999996,"5min":2.0960000000000001,"15min":0.95799999999999996},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":46.543999999999997,"5min":46.543999999999997,"15min":46.543999999999997},"last":0.51200000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:10:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7869999999999999,"5min":2.419,"15min":1.113},"min":{"1min":0.42099999999999999,"5min":0.28399999999999997,"15min":0.21099999999999999},"max":{"1min":9.6829999999999998,"5min":67.888000000000005,"15min":67.888000000000005},"last":0.47799999999999998},{"interface":"front","average":{"1min":2.7349999999999999,"5min":2.7229999999999999,"15min":1.22},"min":{"1min":0.41099999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":8.8979999999999997,"5min":66.891999999999996,"15min":66.891999999999996},"last":0.46000000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:10:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.1060000000000001,"15min":0.746},"min":{"1min":0.34599999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":3.5089999999999999,"5min":13.295,"15min":13.295},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.647,"5min":1.2290000000000001,"15min":0.81499999999999995},"min":{"1min":0.35399999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":9.5739999999999998,"5min":12.989000000000001,"15min":12.989000000000001},"last":0.72799999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:09:50 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.141,"15min":0.78500000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.0190000000000001,"5min":13.073,"15min":13.073},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.175,"15min":0.80800000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":9.5289999999999999,"5min":12.94,"15min":12.94},"last":0.70299999999999996}]},{"osd":6,"last update":"Mon Feb 5 08:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.292,"15min":0.83199999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":9.6039999999999992,"5min":12.967000000000001,"15min":12.967000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.472,"5min":1.3420000000000001,"15min":0.86199999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":6.335,"5min":13.108000000000001,"15min":13.108000000000001},"last":0.65900000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:09:36 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.0469999999999999,"15min":0.72499999999999998},"min":{"1min":0.432,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":5.6150000000000002,"5min":13.263,"15min":13.263},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.244,"15min":0.82099999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":9.6300000000000008,"5min":13.012,"15min":13.012},"last":0.58799999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411428,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130512,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93614640,"statfs":{"total":95995035648,"available":95861391360,"internally_reserved":0,"allocated":790528,"data_stored":475544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":2.254,"5min":1.667,"15min":0.80400000000000005},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":12.355,"5min":16.155999999999999,"15min":16.155999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":2.1920000000000002,"5min":1.8240000000000001,"15min":0.86299999999999999},"min":{"1min":0.26500000000000001,"5min":0.23300000000000001,"15min":0.20699999999999999},"max":{"1min":14.243,"5min":15.973000000000001,"15min":15.973000000000001},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":2.1419999999999999,"5min":1.4179999999999999,"15min":0.71199999999999997},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":7.0519999999999996,"5min":9.1340000000000003,"15min":9.1340000000000003},"last":0.441},{"interface":"front","average":{"1min":1.9630000000000001,"5min":1.4259999999999999,"15min":0.72899999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":7.7919999999999998,"5min":9.2769999999999992,"15min":9.2769999999999992},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:09:59 2024","interfaces":[{"interface":"back","average":{"1min":5.7229999999999999,"5min":2.984,"15min":1.3069999999999999},"min":{"1min":0.314,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":68.108000000000004,"5min":68.108000000000004,"15min":68.108000000000004},"last":0.46100000000000002},{"interface":"front","average":{"1min":5.7880000000000003,"5min":2.6589999999999998,"15min":1.1919999999999999},"min":{"1min":0.33500000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":67.945999999999998,"5min":67.945999999999998,"15min":67.945999999999998},"last":0.38800000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:10:23 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.3089999999999999,"15min":0.83699999999999997},"min":{"1min":0.34699999999999998,"5min":0.315,"15min":0.315},"max":{"1min":3.863,"5min":15.923999999999999,"15min":15.923999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.0529999999999999,"15min":0.73899999999999999},"min":{"1min":0.27200000000000002,"5min":0.252,"15min":0.252},"max":{"1min":3.5880000000000001,"5min":6.008,"15min":6.008},"last":0.60699999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:09:50 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.1659999999999999,"15min":0.78100000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.6669999999999998,"5min":15.835000000000001,"15min":15.835000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.036,"15min":0.72699999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":5.4429999999999996,"5min":9.1809999999999992,"15min":9.1809999999999992},"last":0.51000000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:10:20 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.359,"15min":0.878},"min":{"1min":0.32500000000000001,"5min":0.308,"15min":0.308},"max":{"1min":3.9449999999999998,"5min":15.859,"15min":15.859},"last":0.499},{"interface":"front","average":{"1min":1.284,"5min":1.129,"15min":0.80400000000000005},"min":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.6389999999999998,"5min":6.117,"15min":6.117},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.2549999999999999,"15min":0.83299999999999996},"min":{"1min":0.314,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":5.4870000000000001,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.3,"15min":0.88100000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.0540000000000003,"5min":15.951000000000001,"15min":15.951000000000001},"last":0.629}]}]},{"osd":3,"up_from":27,"seq":115964117087,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":208288,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93536864,"statfs":{"total":95995035648,"available":95781748736,"internally_reserved":0,"allocated":811008,"data_stored":476247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":212466121},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.1549999999999998,"5min":2.1379999999999999,"15min":0.98499999999999999},"min":{"1min":0.42099999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":6.4169999999999998,"5min":63.43,"15min":63.43},"last":0.27200000000000002},{"interface":"front","average":{"1min":2.5430000000000001,"5min":2.1829999999999998,"15min":1.028},"min":{"1min":0.34300000000000003,"5min":0.186,"15min":0.186},"max":{"1min":8.2650000000000006,"5min":63.048999999999999,"15min":63.048999999999999},"last":0.312}]},{"osd":1,"last update":"Mon Feb 5 08:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":1.496,"15min":0.81999999999999995},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":6.085,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":0.371},{"interface":"front","average":{"1min":3.7719999999999998,"5min":2.0539999999999998,"15min":0.995},"min":{"1min":0.34300000000000003,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":8.1760000000000002,"5min":12.087999999999999,"15min":12.087999999999999},"last":0.438}]},{"osd":2,"last update":"Mon Feb 5 08:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.3919999999999999,"5min":2.3559999999999999,"15min":1.1160000000000001},"min":{"1min":0.29999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":7.2800000000000002,"5min":62.902999999999999,"15min":62.902999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":2.105,"5min":2.198,"15min":1.054},"min":{"1min":0.54100000000000004,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.3730000000000002,"5min":63.468000000000004,"15min":63.468000000000004},"last":0.33400000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:10:21 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.302,"15min":0.80800000000000005},"min":{"1min":0.40200000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.0760000000000001,"5min":7.7229999999999999,"15min":7.7229999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.399,"15min":0.86099999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.8039999999999998,"5min":8.7059999999999995,"15min":8.7059999999999995},"last":0.45300000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:09:46 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.276,"15min":0.81799999999999995},"min":{"1min":0.33300000000000002,"5min":0.255,"15min":0.255},"max":{"1min":3.9950000000000001,"5min":7.6680000000000001,"15min":7.6680000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.9139999999999999,"5min":1.4159999999999999,"15min":0.872},"min":{"1min":0.42399999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.8860000000000001,"5min":8.734,"15min":8.734},"last":0.47299999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:10:11 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.4339999999999999,"15min":0.876},"min":{"1min":0.372,"5min":0.372,"15min":0.36499999999999999},"max":{"1min":4.5,"5min":8.9429999999999996,"15min":8.9429999999999996},"last":0.40000000000000002},{"interface":"front","average":{"1min":2.2869999999999999,"5min":1.4450000000000001,"15min":0.86699999999999999},"min":{"1min":0.34300000000000003,"5min":0.308,"15min":0.308},"max":{"1min":6.9400000000000004,"5min":7.8639999999999999,"15min":7.8639999999999999},"last":0.53600000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:09:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":1.2889999999999999,"15min":0.82499999999999996},"min":{"1min":0.35699999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":7.7000000000000002,"5min":7.7000000000000002,"15min":7.7000000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":2.294,"5min":1.458,"15min":0.90700000000000003},"min":{"1min":0.41399999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":8.9900000000000002,"5min":8.9900000000000002,"15min":8.9900000000000002},"last":0.49099999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953562,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99988,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93645164,"statfs":{"total":95995035648,"available":95892647936,"internally_reserved":0,"allocated":798720,"data_stored":475087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:10:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":2.2400000000000002,"15min":1.1140000000000001},"min":{"1min":0.27900000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":5.4409999999999998,"5min":73.078999999999994,"15min":73.078999999999994},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.25,"5min":2.0880000000000001,"15min":1.0609999999999999},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":4.7709999999999999,"5min":73.412000000000006,"15min":73.412000000000006},"last":0.45500000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:10:14 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":0.97699999999999998,"15min":0.68799999999999994},"min":{"1min":0.252,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":8.3409999999999993,"5min":8.3409999999999993,"15min":8.3409999999999993},"last":0.437},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.74099999999999999,"15min":0.61199999999999999},"min":{"1min":0.25900000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":4.3120000000000003,"5min":4.3120000000000003,"15min":4.3120000000000003},"last":0.55900000000000005}]},{"osd":2,"last update":"Mon Feb 5 08:10:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.8740000000000001,"15min":0.98699999999999999},"min":{"1min":0.26500000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.9640000000000004,"5min":96.849999999999994,"15min":96.849999999999994},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.9490000000000001,"15min":1.0329999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.6309999999999998,"5min":96.143000000000001,"15min":96.143000000000001},"last":0.41099999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:10:14 2024","interfaces":[{"interface":"back","average":{"1min":2.8220000000000001,"5min":2.5489999999999999,"15min":1.2170000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":28.472999999999999,"5min":73.602999999999994,"15min":73.602999999999994},"last":0.49399999999999999},{"interface":"front","average":{"1min":3.2749999999999999,"5min":2.5680000000000001,"15min":1.222},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":28.861000000000001,"5min":74.817999999999998,"15min":74.817999999999998},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:09:50 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.40999999999999998,"15min":0.38},"min":{"1min":0.28100000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.63900000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.44500000000000001,"15min":0.40999999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.312}]},{"osd":6,"last update":"Mon Feb 5 08:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46100000000000002,"15min":0.435},"min":{"1min":0.311,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.46899999999999997,"15min":0.433},"min":{"1min":0.34100000000000003,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:09:33 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.29299999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.77100000000000002,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.5,"15min":0.502},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.68600000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.54800000000000004}]}]},{"osd":5,"up_from":42,"seq":180388626516,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108292,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636860,"statfs":{"total":95995035648,"available":95884144640,"internally_reserved":0,"allocated":778240,"data_stored":475014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.794,"5min":2.2519999999999998,"15min":1.1200000000000001},"min":{"1min":0.23200000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":7.6150000000000002,"5min":143.06299999999999,"15min":143.06299999999999},"last":0.63},{"interface":"front","average":{"1min":1.974,"5min":2.2829999999999999,"15min":1.119},"min":{"1min":0.218,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.3570000000000002,"5min":143.09700000000001,"15min":143.09700000000001},"last":0.755}]},{"osd":1,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.69699999999999995,"15min":0.61399999999999999},"min":{"1min":0.23200000000000001,"5min":0.215,"15min":0.215},"max":{"1min":2.286,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.755,"15min":0.61499999999999999},"min":{"1min":0.25800000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.2170000000000001,"5min":3.544,"15min":3.544},"last":0.59599999999999997}]},{"osd":2,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.90400000000000003,"15min":0.68400000000000005},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.673,"5min":5.0049999999999999,"15min":5.0049999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.272,"5min":1.01,"15min":0.71599999999999997},"min":{"1min":0.28799999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.657,"5min":5.4089999999999998,"15min":5.4089999999999998},"last":0.77300000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":2.375,"15min":1.167},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":7.8410000000000002,"5min":143.03399999999999,"15min":143.03399999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.712,"5min":2.202,"15min":1.1299999999999999},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":9.1839999999999993,"5min":143.46799999999999,"15min":143.46799999999999},"last":0.68400000000000005}]},{"osd":4,"last update":"Mon Feb 5 08:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.436,"15min":0.40899999999999997},"min":{"1min":0.25800000000000001,"5min":0.253,"15min":0.253},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.45000000000000001,"15min":0.42799999999999999},"min":{"1min":0.30399999999999999,"5min":0.26900000000000002,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.66500000000000004}]},{"osd":6,"last update":"Mon Feb 5 08:10:09 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.45300000000000001,"15min":0.44500000000000001},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.40000000000000002,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":0.71099999999999997,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.74299999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:09:38 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.47299999999999998},"min":{"1min":0.315,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.76600000000000001,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.34499999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:10:32.376 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-02-05T08:10:32.376 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-02-05T08:10:32.376 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-02-05T08:10:32.416 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:32.441 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:32.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:32.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:32.543 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:32.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:32.572 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:32.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:32.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:32.702 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:32.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:32.804 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:32.830 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:32.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:32.932 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:32.956 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:32.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:32.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:33.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:32 smithi196 ceph-mon[33018]: 5.d scrub starts 2024-02-05T08:10:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:32 smithi196 ceph-mon[33018]: 5.d scrub ok 2024-02-05T08:10:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:32 smithi196 ceph-mon[33018]: pgmap v227: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 421 op/s 2024-02-05T08:10:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:32 smithi196 ceph-mon[33018]: 5.16 deep-scrub starts 2024-02-05T08:10:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:32 smithi196 ceph-mon[33018]: 5.16 deep-scrub ok 2024-02-05T08:10:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:32 smithi196 ceph-mon[33018]: 4.13 scrub starts 2024-02-05T08:10:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:32 smithi196 ceph-mon[33018]: 4.13 scrub ok 2024-02-05T08:10:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:33.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:33.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:33.087 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[30086]: 5.d scrub starts 2024-02-05T08:10:33.087 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[30086]: 5.d scrub ok 2024-02-05T08:10:33.087 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[30086]: pgmap v227: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 421 op/s 2024-02-05T08:10:33.088 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[30086]: 5.16 deep-scrub starts 2024-02-05T08:10:33.088 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[30086]: 5.16 deep-scrub ok 2024-02-05T08:10:33.088 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[30086]: 4.13 scrub starts 2024-02-05T08:10:33.088 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[30086]: 4.13 scrub ok 2024-02-05T08:10:33.088 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[39278]: 5.d scrub starts 2024-02-05T08:10:33.088 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[39278]: 5.d scrub ok 2024-02-05T08:10:33.088 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[39278]: pgmap v227: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 421 op/s 2024-02-05T08:10:33.088 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[39278]: 5.16 deep-scrub starts 2024-02-05T08:10:33.088 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[39278]: 5.16 deep-scrub ok 2024-02-05T08:10:33.088 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[39278]: 4.13 scrub starts 2024-02-05T08:10:33.088 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:32 smithi134 ceph-mon[39278]: 4.13 scrub ok 2024-02-05T08:10:33.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:33.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:33.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:33.316 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:33.341 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:33.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:33.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T08:10:33.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:33.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:33.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:33.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:33.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:10:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:10:33.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:33.598 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:33.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:33.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:33.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:33.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:33.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:33.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:33.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:33.853 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:33.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:33.894 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:33.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:33.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:33.980 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:33.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:33 smithi196 ceph-mon[33018]: 3.19 scrub starts 2024-02-05T08:10:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:33 smithi196 ceph-mon[33018]: 3.19 scrub ok 2024-02-05T08:10:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:33 smithi196 ceph-mon[33018]: 4.17 scrub starts 2024-02-05T08:10:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:33 smithi196 ceph-mon[33018]: 4.17 scrub ok 2024-02-05T08:10:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:33 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:34.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:34.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:34.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:34.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:34.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:34.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:33 smithi134 ceph-mon[30086]: 3.19 scrub starts 2024-02-05T08:10:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:33 smithi134 ceph-mon[30086]: 3.19 scrub ok 2024-02-05T08:10:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:33 smithi134 ceph-mon[30086]: 4.17 scrub starts 2024-02-05T08:10:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:33 smithi134 ceph-mon[30086]: 4.17 scrub ok 2024-02-05T08:10:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:33 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:33 smithi134 ceph-mon[39278]: 3.19 scrub starts 2024-02-05T08:10:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:33 smithi134 ceph-mon[39278]: 3.19 scrub ok 2024-02-05T08:10:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:33 smithi134 ceph-mon[39278]: 4.17 scrub starts 2024-02-05T08:10:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:33 smithi134 ceph-mon[39278]: 4.17 scrub ok 2024-02-05T08:10:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:33 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:34.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:34.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:34.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:34.443 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:34.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:34.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:34.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:34.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:34.594 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:34.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:34.625 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:34.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:34.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:34.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:34.754 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:34.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:34.817 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:10:34.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:34.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:34.881 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:34.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:34.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:35.011 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:35.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:35.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:34 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:10:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:34 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:10:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:34 smithi196 ceph-mon[33018]: pgmap v228: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:10:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:34 smithi196 ceph-mon[33018]: osdmap e152: 8 total, 8 up, 7 in 2024-02-05T08:10:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:34 smithi196 ceph-mon[33018]: 5.14 scrub starts 2024-02-05T08:10:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:34 smithi196 ceph-mon[33018]: 5.14 scrub ok 2024-02-05T08:10:35.100 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:34 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:10:35.100 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:34 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:10:35.101 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:34 smithi134 ceph-mon[30086]: pgmap v228: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:10:35.101 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:34 smithi134 ceph-mon[30086]: osdmap e152: 8 total, 8 up, 7 in 2024-02-05T08:10:35.101 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:34 smithi134 ceph-mon[30086]: 5.14 scrub starts 2024-02-05T08:10:35.101 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:34 smithi134 ceph-mon[30086]: 5.14 scrub ok 2024-02-05T08:10:35.101 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:34 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:10:35.101 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:34 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:10:35.101 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:34 smithi134 ceph-mon[39278]: pgmap v228: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:10:35.101 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:34 smithi134 ceph-mon[39278]: osdmap e152: 8 total, 8 up, 7 in 2024-02-05T08:10:35.101 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:34 smithi134 ceph-mon[39278]: 5.14 scrub starts 2024-02-05T08:10:35.101 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:34 smithi134 ceph-mon[39278]: 5.14 scrub ok 2024-02-05T08:10:35.113 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:35.136 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:35.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:35.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:35.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:35.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:35.263 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:35.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:35.364 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:35.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:35.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:35.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:35.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:35.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:35.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:35.620 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:35.648 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:35.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:35.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:35.749 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:35.775 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:35.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:35.820 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:10:35.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:35.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:35.901 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:35.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:36.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:36.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:35 smithi134 ceph-mon[30086]: 4.b scrub starts 2024-02-05T08:10:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:35 smithi134 ceph-mon[30086]: 4.b scrub ok 2024-02-05T08:10:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:35 smithi134 ceph-mon[30086]: 2.1 scrub starts 2024-02-05T08:10:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:35 smithi134 ceph-mon[30086]: 2.1 scrub ok 2024-02-05T08:10:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:35 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1757190401' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:10:36.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:35 smithi134 ceph-mon[39278]: 4.b scrub starts 2024-02-05T08:10:36.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:35 smithi134 ceph-mon[39278]: 4.b scrub ok 2024-02-05T08:10:36.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:35 smithi134 ceph-mon[39278]: 2.1 scrub starts 2024-02-05T08:10:36.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:35 smithi134 ceph-mon[39278]: 2.1 scrub ok 2024-02-05T08:10:36.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:35 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1757190401' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:10:36.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:36.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:36.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:36.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:36.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:36.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:36.261 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:36.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:36.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:36.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:36.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:35 smithi196 ceph-mon[33018]: 4.b scrub starts 2024-02-05T08:10:36.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:35 smithi196 ceph-mon[33018]: 4.b scrub ok 2024-02-05T08:10:36.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:35 smithi196 ceph-mon[33018]: 2.1 scrub starts 2024-02-05T08:10:36.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:35 smithi196 ceph-mon[33018]: 2.1 scrub ok 2024-02-05T08:10:36.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:35 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1757190401' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:10:36.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:36.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:36.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:36.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:36.550 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:36.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:36.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:36.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:36.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:36.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:36.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:36.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:36.806 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:36.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:36.907 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:36.935 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:36.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:36.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[30086]: 6.4 scrub starts 2024-02-05T08:10:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[30086]: 6.4 scrub ok 2024-02-05T08:10:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[30086]: pgmap v230: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:10:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[30086]: 5.1b deep-scrub starts 2024-02-05T08:10:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[30086]: 5.1b deep-scrub ok 2024-02-05T08:10:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:10:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1757190401' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:10:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[30086]: osdmap e153: 8 total, 8 up, 7 in 2024-02-05T08:10:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[30086]: 3.8 scrub starts 2024-02-05T08:10:37.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[39278]: 6.4 scrub starts 2024-02-05T08:10:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[39278]: 6.4 scrub ok 2024-02-05T08:10:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[39278]: pgmap v230: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:10:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[39278]: 5.1b deep-scrub starts 2024-02-05T08:10:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[39278]: 5.1b deep-scrub ok 2024-02-05T08:10:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:10:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1757190401' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:10:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[39278]: osdmap e153: 8 total, 8 up, 7 in 2024-02-05T08:10:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:36 smithi134 ceph-mon[39278]: 3.8 scrub starts 2024-02-05T08:10:37.037 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:37.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:37.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:37.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:37.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:37.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:37.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:37.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:37.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:37.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:36 smithi196 ceph-mon[33018]: 6.4 scrub starts 2024-02-05T08:10:37.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:36 smithi196 ceph-mon[33018]: 6.4 scrub ok 2024-02-05T08:10:37.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:36 smithi196 ceph-mon[33018]: pgmap v230: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:10:37.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:36 smithi196 ceph-mon[33018]: 5.1b deep-scrub starts 2024-02-05T08:10:37.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:36 smithi196 ceph-mon[33018]: 5.1b deep-scrub ok 2024-02-05T08:10:37.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:36 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:10:37.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1757190401' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:10:37.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:36 smithi196 ceph-mon[33018]: osdmap e153: 8 total, 8 up, 7 in 2024-02-05T08:10:37.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:36 smithi196 ceph-mon[33018]: 3.8 scrub starts 2024-02-05T08:10:37.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:37.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:37.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:37.377 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:10:37.377 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:10:37.377 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-05T08:10:37.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T08:10:37.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:37.447 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:37.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:37.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:37.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:37.575 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:37.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:37.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:37.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:37.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:37.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:37.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:37.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:37.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:37.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:37.964 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:37.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:37.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:37 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:10:38.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:37 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:10:38.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:10:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:10:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:10:38.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:38.090 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:38.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:38.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:38.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:38.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:38.220 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:38.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:38.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:37 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:10:38.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:38.347 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:38.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:38.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:38.374 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:38.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:38.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:38.478 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:38.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:38.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:38.603 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:38.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:38.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:38.705 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:38.726 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:10:38.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:38.729 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:38.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:38.830 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:38.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:38.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:38.880 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:38 smithi134 ceph-mon[30086]: pgmap v232: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:10:38.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:38 smithi134 ceph-mon[39278]: pgmap v232: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:10:38.960 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:38.988 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:38.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:39.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:39.118 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:39.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:39.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:39.220 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:39.245 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:39.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:39.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:38 smithi196 ceph-mon[33018]: pgmap v232: 148 pgs: 148 active+clean; 582 KiB data, 977 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:10:39.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:39.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:39.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:39.478 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:39.502 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:39.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:39.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:39.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:39.630 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:39.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:39.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:39.699 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:10:39.699 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:10:39.732 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:39.756 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:39.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:39.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:39.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:39.887 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:39.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:39.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:40.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:40.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:40.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:40.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:40.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:40.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:40.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:40.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:40.271 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:40.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:40.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:40.373 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:40.401 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:40.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:40.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:40.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:40.531 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:40.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:40.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:40.660 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:40.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:40.750 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T08:10:40.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:40.819 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:40.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:40.920 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:40.944 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:40.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:41.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:40 smithi134 ceph-mon[30086]: pgmap v233: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 981 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:10:41.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:40 smithi134 ceph-mon[39278]: pgmap v233: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 981 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:10:41.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:41.059 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:41.074 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:41.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:41.176 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:41.200 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:41.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:41.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:41.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:40 smithi196 ceph-mon[33018]: pgmap v233: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 981 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:10:41.330 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:41.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:41.431 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:41.455 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:41.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:41.511 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:41.534 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:41.557 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:41.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:41.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:41.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:41.687 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:41.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:41.712 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:41.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:41.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:41.842 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:41.842 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'unique_pool_0' removed 2024-02-05T08:10:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:41.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:41.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:41.972 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:41.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:42.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:41 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3205046701' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:10:42.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:41 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:10:42.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:41 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3205046701' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:10:42.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:41 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:10:42.074 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:42.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:42.100 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:42.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:42.201 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:42.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:42.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:42.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:41 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3205046701' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:10:42.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:41 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:10:42.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:42.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:42.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:42.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:42.478 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:42.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:42.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:42.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:42.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:42.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:42.706 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:42.730 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:42.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:42.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:42.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:42.858 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:42.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:42.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:42 smithi134 ceph-mon[30086]: pgmap v234: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 981 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:10:42.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:42 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:10:42.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:42 smithi134 ceph-mon[30086]: osdmap e154: 8 total, 8 up, 7 in 2024-02-05T08:10:42.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:42 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:10:42.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:42 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:10:42.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:42 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:10:42.866 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:42 smithi134 ceph-mon[39278]: pgmap v234: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 981 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:10:42.866 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:42 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:10:42.866 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:42 smithi134 ceph-mon[39278]: osdmap e154: 8 total, 8 up, 7 in 2024-02-05T08:10:42.866 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:42 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:10:42.867 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:42 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:10:42.867 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:42 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:10:42.960 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:42.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:42.985 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:42.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:43.086 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:43.110 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:43.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:43.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:43.169 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:42 smithi196 ceph-mon[33018]: pgmap v234: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 981 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:10:43.169 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:42 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:10:43.169 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:42 smithi196 ceph-mon[33018]: osdmap e154: 8 total, 8 up, 7 in 2024-02-05T08:10:43.169 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:42 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:10:43.169 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:42 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:10:43.169 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:42 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:10:43.212 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:43.235 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:43.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:43.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:43.337 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:43.362 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:43.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:43.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:43.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:43.487 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:43.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:43.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:10:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:10:43.589 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:43.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:43.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:43.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:43.715 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:43.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:43.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:43.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:43.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:43.855 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T08:10:43.856 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T08:10:43.856 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-02-05T08:10:43.856 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-02-05T08:10:43.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:43.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:43.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:43.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:43.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:43.990 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:43.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:44.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:44.096 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:43 smithi134 ceph-mon[39278]: osdmap e155: 8 total, 8 up, 7 in 2024-02-05T08:10:44.097 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:43 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:44.097 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:43 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:44.097 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:43 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:44.097 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:43 smithi134 ceph-mon[30086]: osdmap e155: 8 total, 8 up, 7 in 2024-02-05T08:10:44.097 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:43 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:44.097 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:43 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:44.097 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:43 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:44.119 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:44.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:44.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:44.245 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:44.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:44.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:43 smithi196 ceph-mon[33018]: osdmap e155: 8 total, 8 up, 7 in 2024-02-05T08:10:44.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:43 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:44.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:43 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:44.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:43 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:44.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:44.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:44.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:44.473 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:44.497 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:44.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:44.599 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:44.628 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:44.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:44.647 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:10:44.647 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:10:44.647 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 155, 2024-02-05T08:10:44.647 INFO:teuthology.orchestra.run.smithi134.stdout: "fsid": "43e13aa4-c3fc-11ee-95b6-87774f69a715", 2024-02-05T08:10:44.647 INFO:teuthology.orchestra.run.smithi134.stdout: "created": "2024-02-05T07:59:09.321262+0000", 2024-02-05T08:10:44.647 INFO:teuthology.orchestra.run.smithi134.stdout: "modified": "2024-02-05T08:10:42.829396+0000", 2024-02-05T08:10:44.648 INFO:teuthology.orchestra.run.smithi134.stdout: "last_up_change": "2024-02-05T08:04:26.849889+0000", 2024-02-05T08:10:44.648 INFO:teuthology.orchestra.run.smithi134.stdout: "last_in_change": "2024-02-05T08:06:38.610101+0000", 2024-02-05T08:10:44.648 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T08:10:44.648 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_num": 5801984, 2024-02-05T08:10:44.648 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_set": [ 2024-02-05T08:10:44.648 INFO:teuthology.orchestra.run.smithi134.stdout: "noscrub", 2024-02-05T08:10:44.648 INFO:teuthology.orchestra.run.smithi134.stdout: "pglog_hardlimit", 2024-02-05T08:10:44.648 INFO:teuthology.orchestra.run.smithi134.stdout: "purged_snapdirs", 2024-02-05T08:10:44.648 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_deletes", 2024-02-05T08:10:44.648 INFO:teuthology.orchestra.run.smithi134.stdout: "sortbitwise" 2024-02-05T08:10:44.648 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:10:44.648 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_version": 18, 2024-02-05T08:10:44.648 INFO:teuthology.orchestra.run.smithi134.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_snapshot": "", 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_max": 7, 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "max_osd": 8, 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "require_min_compat_client": "luminous", 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "min_compat_client": "luminous", 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "require_osd_release": "squid", 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "allow_crimson": false, 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "pools": [ 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 1, 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": ".mgr", 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:02:21.034012+0000", 2024-02-05T08:10:44.649 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:10:44.650 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:10:44.650 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:10:44.650 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:10:44.650 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:10:44.650 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:10:44.650 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:10:44.650 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:10:44.650 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:10:44.650 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:10:44.650 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:10:44.650 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 1, 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 1, 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 1, 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 1, 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 1, 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.651 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "22", 2024-02-05T08:10:44.652 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:10:44.652 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:10:44.652 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:10:44.652 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:10:44.652 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:10:44.652 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:10:44.652 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:10:44.652 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:10:44.652 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:10:44.652 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:10:44.653 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "options": { 2024-02-05T08:10:44.654 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_max": 32, 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_min": 1 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr": {} 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 7.8899998664855957, 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 7.8899998664855957, 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.37999999523162842, 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 3, 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 3, 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:10:44.655 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:10:44.656 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:10:44.656 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.656 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 2, 2024-02-05T08:10:44.656 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "datapool", 2024-02-05T08:10:44.656 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:39.400746+0000", 2024-02-05T08:10:44.656 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 8193, 2024-02-05T08:10:44.656 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T08:10:44.656 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:10:44.656 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:10:44.656 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:10:44.656 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:10:44.657 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:10:44.657 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:10:44.657 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:10:44.657 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:10:44.657 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:10:44.657 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:10:44.657 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 3, 2024-02-05T08:10:44.657 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 3, 2024-02-05T08:10:44.657 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 3, 2024-02-05T08:10:44.657 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 3, 2024-02-05T08:10:44.657 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 3, 2024-02-05T08:10:44.657 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:10:44.657 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:10:44.658 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:10:44.658 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:10:44.658 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:10:44.658 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:10:44.658 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:10:44.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.658 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "70", 2024-02-05T08:10:44.658 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:10:44.658 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:10:44.658 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:10:44.658 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:10:44.658 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:10:44.658 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 2, 2024-02-05T08:10:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 70, 2024-02-05T08:10:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:10:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:10:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:10:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:10:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:10:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:10:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:10:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:10:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:10:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:10:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:10:44.659 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:10:44.660 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd": {} 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.6700000762939453, 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.6700000762939453, 2024-02-05T08:10:44.661 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.75, 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 2, 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 2, 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 3, 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": ".rgw.root", 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:40.005769+0000", 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:10:44.662 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:10:44.663 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:10:44.664 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:10:44.664 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:10:44.664 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:10:44.664 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:10:44.664 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:10:44.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.664 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "68", 2024-02-05T08:10:44.665 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:10:44.665 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:10:44.665 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:10:44.665 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:10:44.665 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:10:44.665 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:10:44.665 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:10:44.665 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:10:44.666 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:10:44.667 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:10:44.668 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:10:44.668 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:10:44.668 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:10:44.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.668 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:10:44.668 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:10:44.668 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.7400000095367432, 2024-02-05T08:10:44.668 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.7400000095367432, 2024-02-05T08:10:44.668 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:10:44.668 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.5299999713897705, 2024-02-05T08:10:44.668 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.5299999713897705, 2024-02-05T08:10:44.668 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:10:44.668 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 4, 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.log", 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:41.851891+0000", 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:10:44.669 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:10:44.670 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "70", 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:10:44.671 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:10:44.672 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:10:44.672 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:10:44.672 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:10:44.672 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:10:44.672 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:10:44.672 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:10:44.672 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:10:44.672 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:10:44.672 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:10:44.672 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:10:44.672 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:10:44.672 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:10:44.672 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:10:44.673 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.4900000095367432, 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.4900000095367432, 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 2.190000057220459, 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 2.190000057220459, 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 5, 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.control", 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:43.793804+0000", 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:10:44.675 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:10:44.676 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "72", 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:10:44.677 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:10:44.678 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.679 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:10:44.680 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:10:44.680 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.4900000095367432, 2024-02-05T08:10:44.680 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.4900000095367432, 2024-02-05T08:10:44.680 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:10:44.680 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.309999942779541, 2024-02-05T08:10:44.680 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.309999942779541, 2024-02-05T08:10:44.680 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:10:44.680 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:10:44.680 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:10:44.680 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.680 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 6, 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.meta", 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:45.853999+0000", 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:10:44.681 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "74", 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:10:44.682 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:10:44.683 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:10:44.683 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:10:44.683 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:10:44.683 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:10:44.683 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:10:44.683 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:10:44.683 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:10:44.683 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:10:44.683 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:10:44.683 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:10:44.684 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:10:44.684 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:10:44.684 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:10:44.684 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:10:44.684 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:10:44.684 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:10:44.684 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:10:44.684 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:10:44.684 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:10:44.685 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: "options": { 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_bias": 4 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.7400000095367432, 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.7400000095367432, 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.5299999713897705, 2024-02-05T08:10:44.686 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.5299999713897705, 2024-02-05T08:10:44.687 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:10:44.687 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:10:44.687 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:10:44.687 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.687 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.687 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:10:44.687 INFO:teuthology.orchestra.run.smithi134.stdout: "osds": [ 2024-02-05T08:10:44.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.687 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 0, 2024-02-05T08:10:44.688 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "d90a5ea0-9a19-452e-9564-ca7b7a7c45e4", 2024-02-05T08:10:44.688 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:10:44.688 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:10:44.688 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:10:44.688 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:10:44.688 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:10:44.688 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:10:44.688 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 8, 2024-02-05T08:10:44.688 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 97, 2024-02-05T08:10:44.688 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:10:44.688 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:10:44.688 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:10:44.689 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.689 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.689 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6802", 2024-02-05T08:10:44.689 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:10:44.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.689 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.689 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6803", 2024-02-05T08:10:44.689 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:10:44.689 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.689 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6804", 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6805", 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6808", 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6809", 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:10:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6806", 2024-02-05T08:10:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:10:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6807", 2024-02-05T08:10:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:10:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.692 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6803/2540271081", 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6805/2540271081", 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6809/2540271081", 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6807/2540271081", 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 1, 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4", 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:10:44.693 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 13, 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 97, 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6810", 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:10:44.694 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6811", 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6812", 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:10:44.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6813", 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6816", 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6817", 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6814", 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.697 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.698 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.698 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6815", 2024-02-05T08:10:44.698 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:10:44.698 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.698 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.698 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.698 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6811/2032802960", 2024-02-05T08:10:44.698 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6813/2032802960", 2024-02-05T08:10:44.698 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6817/2032802960", 2024-02-05T08:10:44.698 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6815/2032802960", 2024-02-05T08:10:44.698 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:10:44.698 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:10:44.698 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:10:44.699 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.699 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 2, 2024-02-05T08:10:44.699 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708", 2024-02-05T08:10:44.699 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:10:44.699 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:10:44.699 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:10:44.699 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:10:44.699 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:10:44.699 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:10:44.699 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 18, 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 97, 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6818", 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6819", 2024-02-05T08:10:44.700 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6820", 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6821", 2024-02-05T08:10:44.701 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6824", 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6825", 2024-02-05T08:10:44.702 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6822", 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6823", 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:10:44.703 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.704 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.704 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6819/153827987", 2024-02-05T08:10:44.704 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6821/153827987", 2024-02-05T08:10:44.704 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6825/153827987", 2024-02-05T08:10:44.704 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6823/153827987", 2024-02-05T08:10:44.704 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:10:44.704 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:10:44.704 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:10:44.704 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.704 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.704 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.704 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 3, 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "6774e330-3202-4671-894e-172cc2f6189d", 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 27, 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 97, 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.705 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6826", 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6827", 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.706 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6828", 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6829", 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.707 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6832", 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6833", 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.708 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6830", 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6831", 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6827/1527507056", 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6829/1527507056", 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6833/1527507056", 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6831/1527507056", 2024-02-05T08:10:44.709 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 4, 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693", 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:10:44.710 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 32, 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 111, 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6800", 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6801", 2024-02-05T08:10:44.711 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6802", 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6803", 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:10:44.712 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6806", 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6807", 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:10:44.713 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6804", 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6805", 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.714 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6801/3005156808", 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6803/3005156808", 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3005156808", 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3005156808", 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 5, 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955", 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:10:44.715 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:10:44.716 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:10:44.716 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:10:44.716 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:10:44.716 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:10:44.716 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 42, 2024-02-05T08:10:44.716 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 97, 2024-02-05T08:10:44.716 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:10:44.716 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:10:44.716 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:10:44.716 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.716 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.716 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.717 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6808", 2024-02-05T08:10:44.717 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:10:44.717 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.717 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.718 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.718 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6809", 2024-02-05T08:10:44.718 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:10:44.718 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.718 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.718 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.718 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:10:44.718 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.718 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.718 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.718 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6810", 2024-02-05T08:10:44.718 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:10:44.718 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6811", 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6814", 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:10:44.719 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6815", 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6812", 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:10:44.720 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6813", 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6809/1603224792", 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6811/1603224792", 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6815/1603224792", 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6813/1603224792", 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:10:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 6, 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a", 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 0, 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 0, 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 51, 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 95, 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:10:44.722 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6816", 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6817", 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.723 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6818", 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6819", 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.724 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6822", 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6823", 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.725 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6820", 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6821", 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6817/3727413143", 2024-02-05T08:10:44.726 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6819/3727413143", 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6823/3727413143", 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6821/3727413143", 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 7, 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057", 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:10:44.727 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 60, 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 100, 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6824", 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.728 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6825", 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6826", 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.729 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6827", 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6830", 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.730 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6831", 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6828", 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.731 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:10:44.732 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6829", 2024-02-05T08:10:44.732 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:10:44.732 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.732 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.732 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.732 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6825/4206571010", 2024-02-05T08:10:44.732 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6827/4206571010", 2024-02-05T08:10:44.732 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6831/4206571010", 2024-02-05T08:10:44.732 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6829/4206571010", 2024-02-05T08:10:44.732 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:10:44.732 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:10:44.732 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:10:44.732 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_xinfo": [ 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 0, 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:10:17.014114+0000", 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.733 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 1, 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:09:46.884893+0000", 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 2, 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:10:44.734 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:09:46.205412+0000", 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 3, 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:10:10.160642+0000", 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.735 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 4, 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:10:42.181358+0000", 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 5, 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:10:44.736 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:10:13.092082+0000", 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 6, 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 65536, 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:10:33.231007+0000", 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.737 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 7, 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:10:21.154217+0000", 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap": [], 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap_items": [ 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.738 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.7", 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.a", 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:10:44.739 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.1a", 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 3, 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.1f", 2024-02-05T08:10:44.740 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap_primaries": [], 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_temp": [], 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_temp": [], 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: "blocklist": { 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/788812699": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/1046500326": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:10:44.741 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/464208089": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/301941854": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/3607168099": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/673876301": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/4225724311": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/1525062017": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/3607168099": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/673876301": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3200884805": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/2952444615": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3325835814": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3161126612": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/69097091": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/1525062017": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:10:44.742 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/1645475987": "2024-02-06T08:05:21.465844+0000" 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: "range_blocklist": {}, 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profiles": { 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: "default": { 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: "crush-failure-domain": "osd", 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: "k": "2", 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: "m": "1", 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: "plugin": "jerasure", 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: "technique": "reed_sol_van" 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps_queue": [], 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: "new_removed_snaps": [], 2024-02-05T08:10:44.743 INFO:teuthology.orchestra.run.smithi134.stdout: "new_purged_snaps": [], 2024-02-05T08:10:44.744 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_node_flags": {}, 2024-02-05T08:10:44.744 INFO:teuthology.orchestra.run.smithi134.stdout: "device_class_flags": {}, 2024-02-05T08:10:44.744 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode": { 2024-02-05T08:10:44.744 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode_enabled": false, 2024-02-05T08:10:44.744 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_bucket_count": 0, 2024-02-05T08:10:44.744 INFO:teuthology.orchestra.run.smithi134.stdout: "degraded_stretch_mode": 0, 2024-02-05T08:10:44.744 INFO:teuthology.orchestra.run.smithi134.stdout: "recovering_stretch_mode": 0, 2024-02-05T08:10:44.744 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode_bucket": 0 2024-02-05T08:10:44.744 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:44.744 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:10:44.744 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:44.768 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:44.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:44.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:44.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:44.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:44.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:44.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:45.023 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:45.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:45.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:45.125 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:45.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:45.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:45.232 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[30086]: pgmap v237: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:10:45.232 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:45.232 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:45.232 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:10:45.232 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:10:45.232 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:45.232 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4043549893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:10:45.233 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[39278]: pgmap v237: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:10:45.233 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:45.233 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:45.233 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:10:45.233 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:10:45.233 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:45.233 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:44 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4043549893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:10:45.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:45.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:45.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:44 smithi196 ceph-mon[33018]: pgmap v237: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:10:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:44 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:44 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:44 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:10:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:44 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:10:45.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:44 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:10:45.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:44 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4043549893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:10:45.379 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:45.405 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:45.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:45.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:45.529 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:45.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:45.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:45.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:45.658 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:45.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:45.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:45.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:45.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:45.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:45.807 INFO:tasks.thrashosds.thrasher:j is {'epoch': 155, 'fsid': '43e13aa4-c3fc-11ee-95b6-87774f69a715', 'created': '2024-02-05T07:59:09.321262+0000', 'modified': '2024-02-05T08:10:42.829396+0000', 'last_up_change': '2024-02-05T08:04:26.849889+0000', 'last_in_change': '2024-02-05T08:06:38.610101+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T08:02:21.034012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T08:04:39.400746+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T08:04:40.005769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T08:04:41.851891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T08:04:43.793804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T08:04:45.853999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'd90a5ea0-9a19-452e-9564-ca7b7a7c45e4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6802', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 2540271081}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6804', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 2540271081}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6808', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2540271081}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6806', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 2540271081}]}, 'public_addr': '172.21.15.134:6803/2540271081', 'cluster_addr': '172.21.15.134:6805/2540271081', 'heartbeat_back_addr': '172.21.15.134:6809/2540271081', 'heartbeat_front_addr': '172.21.15.134:6807/2540271081', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6810', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2032802960}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6812', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 2032802960}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6816', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6817', 'nonce': 2032802960}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6814', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 2032802960}]}, 'public_addr': '172.21.15.134:6811/2032802960', 'cluster_addr': '172.21.15.134:6813/2032802960', 'heartbeat_back_addr': '172.21.15.134:6817/2032802960', 'heartbeat_front_addr': '172.21.15.134:6815/2032802960', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '078e578b-855b-49ff-83bd-0548a9c5a708', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6818', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6819', 'nonce': 153827987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6820', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6821', 'nonce': 153827987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 153827987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6822', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6823', 'nonce': 153827987}]}, 'public_addr': '172.21.15.134:6819/153827987', 'cluster_addr': '172.21.15.134:6821/153827987', 'heartbeat_back_addr': '172.21.15.134:6825/153827987', 'heartbeat_front_addr': '172.21.15.134:6823/153827987', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6774e330-3202-4671-894e-172cc2f6189d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 1527507056}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6828', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6829', 'nonce': 1527507056}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6832', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6833', 'nonce': 1527507056}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6830', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6831', 'nonce': 1527507056}]}, 'public_addr': '172.21.15.134:6827/1527507056', 'cluster_addr': '172.21.15.134:6829/1527507056', 'heartbeat_back_addr': '172.21.15.134:6833/1527507056', 'heartbeat_front_addr': '172.21.15.134:6831/1527507056', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '52f70cfa-ab0e-4751-a5f8-1045c61c1693', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3005156808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3005156808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3005156808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3005156808}]}, 'public_addr': '172.21.15.196:6801/3005156808', 'cluster_addr': '172.21.15.196:6803/3005156808', 'heartbeat_back_addr': '172.21.15.196:6807/3005156808', 'heartbeat_front_addr': '172.21.15.196:6805/3005156808', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b4b21769-d40f-43fd-b7e9-2477b74d3955', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 1603224792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 1603224792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 1603224792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 1603224792}]}, 'public_addr': '172.21.15.196:6809/1603224792', 'cluster_addr': '172.21.15.196:6811/1603224792', 'heartbeat_back_addr': '172.21.15.196:6815/1603224792', 'heartbeat_front_addr': '172.21.15.196:6813/1603224792', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'af1cced4-0015-47f5-b105-27641bfe4f4a', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 3727413143}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 3727413143}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 3727413143}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 3727413143}]}, 'public_addr': '172.21.15.196:6817/3727413143', 'cluster_addr': '172.21.15.196:6819/3727413143', 'heartbeat_back_addr': '172.21.15.196:6823/3727413143', 'heartbeat_front_addr': '172.21.15.196:6821/3727413143', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c54beeae-9f7c-4723-a369-13bb531b4057', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 100, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 4206571010}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 4206571010}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 4206571010}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 4206571010}]}, 'public_addr': '172.21.15.196:6825/4206571010', 'cluster_addr': '172.21.15.196:6827/4206571010', 'heartbeat_back_addr': '172.21.15.196:6831/4206571010', 'heartbeat_front_addr': '172.21.15.196:6829/4206571010', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:10:17.014114+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:09:46.884893+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:09:46.205412+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:10:10.160642+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:10:42.181358+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:10:13.092082+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T08:10:33.231007+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:10:21.154217+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1a', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.134:0/788812699': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/1046500326': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/464208089': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/301941854': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:6800/3607168099': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6800/673876301': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/4225724311': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6800/1525062017': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:6801/3607168099': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6801/673876301': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/3200884805': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/2952444615': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:0/3325835814': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/3161126612': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:0/69097091': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:6801/1525062017': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/1645475987': '2024-02-06T08:05:21.465844+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T08:10:45.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:10:45.889 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:45.913 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:45.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:46.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:46.043 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:46.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:46.145 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:46.168 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:46.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:46.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:46.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:46.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:46.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:46.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:46.400 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:46.424 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:46.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:46.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:46.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:46.549 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:46.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:46.651 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:46.679 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:46.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:46.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:46.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:46.808 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:46.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:46.910 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:46.937 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:46.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:46.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:47.014 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:10:47.014 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:10:47.014 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:10:47.014 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:10:47.014 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:10:47.014 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:10:47.015 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:10:47.015 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:10:47.015 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:10:47.015 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:10:47.015 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:10:47.016 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:10:47.016 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:10:47.016 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:10:47.016 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:10:47.016 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:10:47.017 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:46 smithi134 ceph-mon[30086]: pgmap v238: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T08:10:47.017 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:46 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4057318302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:10:47.017 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:46 smithi134 ceph-mon[39278]: pgmap v238: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T08:10:47.017 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:46 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4057318302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:10:47.039 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:47.063 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:47.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:47.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:47.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:47.189 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:47.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:47.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:46 smithi196 ceph-mon[33018]: pgmap v238: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T08:10:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:46 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4057318302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:10:47.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:47.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:47.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:47.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:47.446 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:47.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:47.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:47.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:47.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:47.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:47.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:47.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:47.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:47.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:47.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:47.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:47.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:47.963 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:10:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:10:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:10:47.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:47.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:47.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:48.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:48.093 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:48.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:48.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:48.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:48.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:48.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:48.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:48.349 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:48.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:48.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:48.451 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:48.475 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:48.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:48.515 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:48.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:48.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:48.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:48.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:48.706 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:48.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:48.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:48.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:48.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:48.857 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:48.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:48.949 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:10:48.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:48.985 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:48.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:48.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:49.068 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:10:49.068 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:10:49.068 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:10:49.069 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:10:49.069 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.069 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-02-05T08:10:49.069 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:02:22.701226+0000", 2024-02-05T08:10:49.069 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 506.364992895, 2024-02-05T08:10:49.069 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.608413223, 2024-02-05T08:10:49.069 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.069 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.069 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.069 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14158", 2024-02-05T08:10:49.069 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3200884805", 2024-02-05T08:10:49.069 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 1 2024-02-05T08:10:49.069 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:22.701226+0000", 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:22.701226+0000", 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.070 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:22.701227+0000", 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.372e-06 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:22.701233+0000", 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9800000000000003e-06 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:22.701243+0000", 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0317e-05 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.071 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:22.701255+0000", 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1413000000000001e-05 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:22.701320+0000", 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5656999999999998e-05 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:22.701328+0000", 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0989999999999999e-06 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.072 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:23.308533+0000", 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.60720420900000005 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:23.308659+0000", 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000125902 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,2", 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:23.308771+0000", 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000112364 2024-02-05T08:10:49.073 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:23.309287+0000", 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00051588800000000005 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:23.309520+0000", 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023310099999999999 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:23.309560+0000", 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9718999999999997e-05 2024-02-05T08:10:49.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.075 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.075 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.075 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:23.309595+0000", 2024-02-05T08:10:49.075 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5788999999999999e-05 2024-02-05T08:10:49.075 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.075 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.075 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.075 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:02:23.309639+0000", 2024-02-05T08:10:49.075 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3412999999999999e-05 2024-02-05T08:10:49.075 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.075 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.075 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.075 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:19160 7.d 7:b2dfbc8c:::benchmark_data_smithi134_83016_object19159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.639350+0000", 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 230.42686847100001, 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13890740700000001, 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 19160 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.076 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.639350+0000", 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.639350+0000", 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.639352+0000", 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.61e-06 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.077 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.639374+0000", 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2574000000000001e-05 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.639390+0000", 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5764e-05 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.639399+0000", 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1290000000000002e-06 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.078 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.640619+0000", 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0012195610000000001 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.640745+0000", 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000126227 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.640881+0000", 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000136186 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.079 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.641954+0000", 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010725050000000001 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.645546+0000", 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0035924619999999998 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.778151+0000", 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13260445300000001 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.080 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.778215+0000", 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4003000000000004e-05 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.778257+0000", 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2932999999999997e-05 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:19173 7.d 7:ba0725d6:::benchmark_data_smithi134_83016_object19172:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:49.081 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.644221+0000", 2024-02-05T08:10:49.082 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 230.42199754000001, 2024-02-05T08:10:49.082 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.138435907, 2024-02-05T08:10:49.082 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.082 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.082 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.082 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.082 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.082 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 19173 2024-02-05T08:10:49.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.082 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.082 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.082 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644221+0000", 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644221+0000", 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644222+0000", 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7700000000000004e-07 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.083 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644245+0000", 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.357e-05 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644254+0000", 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4139999999999994e-06 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644260+0000", 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0310000000000004e-06 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644383+0000", 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012328099999999999 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644519+0000", 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013595999999999999 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644692+0000", 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017349800000000001 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.085 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.646541+0000", 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0018485159999999999 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.646576+0000", 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4830999999999999e-05 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.782558+0000", 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13598219 2024-02-05T08:10:49.086 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.782612+0000", 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4089999999999999e-05 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.782657+0000", 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4948999999999999e-05 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.087 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:19175 7.d 7:bcc36355:::benchmark_data_smithi134_83016_object19174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.644811+0000", 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 230.42140747400001, 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13802329999999999, 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 19175 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.088 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644811+0000", 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644811+0000", 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644812+0000", 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.54e-07 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.089 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644834+0000", 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1787e-05 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644844+0000", 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.03e-05 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.644854+0000", 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.012e-05 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.090 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.645576+0000", 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00072181000000000003 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.645683+0000", 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010684 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.645815+0000", 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000131965 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.647210+0000", 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001395166 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.647854+0000", 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00064441100000000001 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.782743+0000", 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.134889128 2024-02-05T08:10:49.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.782800+0000", 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6396e-05 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.782834+0000", 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4622999999999997e-05 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.093 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:20685 7.d 7:b47cd13c:::benchmark_data_smithi134_83016_object20684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:49.094 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:59.672470+0000", 2024-02-05T08:10:49.094 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 229.39374859599999, 2024-02-05T08:10:49.094 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15066810999999999, 2024-02-05T08:10:49.094 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.094 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.094 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.094 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.094 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.094 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 20685 2024-02-05T08:10:49.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.094 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:49.096 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.096 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.096 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672470+0000", 2024-02-05T08:10:49.096 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.096 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.096 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672470+0000", 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672471+0000", 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3570000000000001e-06 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672493+0000", 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2019e-05 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.097 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672505+0000", 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2148e-05 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672514+0000", 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2609999999999993e-06 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672593+0000", 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9610000000000005e-05 2024-02-05T08:10:49.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672682+0000", 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8912999999999994e-05 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.672770+0000", 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7639000000000006e-05 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.674472+0000", 2024-02-05T08:10:49.099 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017022980000000001 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.818008+0000", 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.143536106 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.823031+0000", 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0050227079999999999 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.100 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.823089+0000", 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7762e-05 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:59.823138+0000", 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9289000000000002e-05 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21727 7.d 7:b77a35a3:::benchmark_data_smithi134_83016_object21726:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.322233+0000", 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 228.74398593800001, 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13558008399999999, 2024-02-05T08:10:49.101 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.102 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.102 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.102 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.102 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.102 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21727 2024-02-05T08:10:49.102 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.102 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.102 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.102 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.102 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322233+0000", 2024-02-05T08:10:49.102 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.102 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.102 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322233+0000", 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322240+0000", 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9000000000000006e-06 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322288+0000", 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.791e-05 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.103 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322330+0000", 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2088000000000001e-05 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322359+0000", 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8992e-05 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322775+0000", 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00041601000000000002 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.104 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322907+0000", 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013129400000000001 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.323074+0000", 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016710999999999999 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325314+0000", 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0022401510000000001 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327844+0000", 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0025299329999999998 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.457409+0000", 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.129564982 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.457775+0000", 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00036568000000000001 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.106 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.457813+0000", 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8034000000000002e-05 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21730 7.d 7:b1aeee65:::benchmark_data_smithi134_83016_object21729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.322494+0000", 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 228.74372450800001, 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13628263800000001, 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.107 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21730 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322494+0000", 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322494+0000", 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322500+0000", 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9750000000000004e-06 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322565+0000", 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5425999999999993e-05 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322602+0000", 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6860999999999998e-05 2024-02-05T08:10:49.109 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.110 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.110 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.110 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.322655+0000", 2024-02-05T08:10:49.110 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2701000000000002e-05 2024-02-05T08:10:49.110 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.110 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.110 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.110 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.323956+0000", 2024-02-05T08:10:49.110 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0013013199999999999 2024-02-05T08:10:49.110 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.110 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.110 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.110 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.324052+0000", 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5600000000000006e-05 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.324189+0000", 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013720299999999999 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325319+0000", 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011298160000000001 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329077+0000", 2024-02-05T08:10:49.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0037580780000000002 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.458661+0000", 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12958445699999999 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.458742+0000", 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0995000000000006e-05 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.112 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.458777+0000", 2024-02-05T08:10:49.113 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4205999999999997e-05 2024-02-05T08:10:49.113 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.113 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.113 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.113 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.113 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.113 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21736 7.d 7:bc7a8f2d:::benchmark_data_smithi134_83016_object21735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:10:49.113 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.325876+0000", 2024-02-05T08:10:49.113 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 228.74034221599999, 2024-02-05T08:10:49.113 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13300265, 2024-02-05T08:10:49.113 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.113 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.113 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.114 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.114 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.114 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21736 2024-02-05T08:10:49.114 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.114 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.114 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.114 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.114 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325876+0000", 2024-02-05T08:10:49.114 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.114 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.114 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325876+0000", 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325878+0000", 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.48e-06 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325900+0000", 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2688e-05 2024-02-05T08:10:49.115 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325914+0000", 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3236e-05 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.325922+0000", 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3270000000000001e-06 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326007+0000", 2024-02-05T08:10:49.116 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4846999999999996e-05 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326162+0000", 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000154792 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.326510+0000", 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00034801800000000001 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327861+0000", 2024-02-05T08:10:49.117 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0013510740000000001 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.458433+0000", 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13057202400000001 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.458811+0000", 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000378243 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.118 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.458852+0000", 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0766000000000002e-05 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.458879+0000", 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7155e-05 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:99785 7.d 7:b0ae1387:::benchmark_data_smithi134_83016_object99784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:08.939677+0000", 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 160.12654188299999, 2024-02-05T08:10:49.119 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14258880099999999, 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 99785 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939677+0000", 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.120 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939677+0000", 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939678+0000", 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2589999999999999e-06 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939699+0000", 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.086e-05 2024-02-05T08:10:49.121 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939711+0000", 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2058e-05 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939718+0000", 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6020000000000002e-06 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939879+0000", 2024-02-05T08:10:49.122 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000160815 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.940016+0000", 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013711800000000001 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.940148+0000", 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013143900000000001 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.123 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947997+0000", 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0078493450000000006 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.948460+0000", 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00046273199999999999 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.082162+0000", 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13370251699999999 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.124 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.082228+0000", 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5143999999999998e-05 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.082265+0000", 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7911999999999999e-05 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:99786 7.d 7:b2cade7f:::benchmark_data_smithi134_83016_object99785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:08.939740+0000", 2024-02-05T08:10:49.125 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 160.12647867699999, 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.144116357, 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 99786 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939740+0000", 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.126 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939740+0000", 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939740+0000", 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4499999999999997e-07 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939758+0000", 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8117999999999998e-05 2024-02-05T08:10:49.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.128 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.128 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.128 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939765+0000", 2024-02-05T08:10:49.128 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9990000000000002e-06 2024-02-05T08:10:49.128 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.128 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.128 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.128 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939771+0000", 2024-02-05T08:10:49.128 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6570000000000002e-06 2024-02-05T08:10:49.128 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.128 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.128 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.128 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.942573+0000", 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00280241 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.942688+0000", 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114951 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.942820+0000", 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000131791 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.129 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.948465+0000", 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0056451699999999997 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.083475+0000", 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13500931599999999 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.083749+0000", 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00027406700000000002 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.130 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.083815+0000", 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5845000000000004e-05 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.083856+0000", 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1588000000000002e-05 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:99790 7.2 e126/95 7:44e11273:::benchmark_data_smithi134_83016_object99789:head v 126'6255, mlcod=126'6253)", 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:08.947413+0000", 2024-02-05T08:10:49.131 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 160.118805651, 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.129610585, 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947413+0000", 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947413+0000", 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.133 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.133 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.133 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947414+0000", 2024-02-05T08:10:49.133 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9699999999999994e-07 2024-02-05T08:10:49.133 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.133 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.133 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.133 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947439+0000", 2024-02-05T08:10:49.133 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5537e-05 2024-02-05T08:10:49.133 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.133 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.133 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.135 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947449+0000", 2024-02-05T08:10:49.135 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9809999999999998e-06 2024-02-05T08:10:49.135 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.135 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.135 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.135 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947455+0000", 2024-02-05T08:10:49.135 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4729999999999999e-06 2024-02-05T08:10:49.135 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.135 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.135 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947538+0000", 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3269000000000005e-05 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947558+0000", 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0347e-05 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.950615+0000", 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003056203 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.136 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.077023+0000", 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.126408778 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:99788 7.9 e126/95 7:94fa4113:::benchmark_data_smithi134_83016_object99787:head v 126'6146, mlcod=126'6143)", 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:08.947736+0000", 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 160.11848270799999, 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12928541199999999, 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.137 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947736+0000", 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947736+0000", 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.138 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947737+0000", 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8899999999999999e-07 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947770+0000", 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3334000000000003e-05 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947781+0000", 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0906e-05 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.139 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947788+0000", 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9550000000000003e-06 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947839+0000", 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0883000000000001e-05 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947859+0000", 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0208e-05 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.140 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.950611+0000", 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002751856 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.077021+0000", 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12641058099999999 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:99792 7.1 e126/97 7:8fc2463e:::benchmark_data_smithi134_83016_object99791:head v 126'6258, mlcod=126'6256)", 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:08.948336+0000", 2024-02-05T08:10:49.141 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 160.117882845, 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12895827900000001, 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.948336+0000", 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.948336+0000", 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.142 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.948336+0000", 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4900000000000005e-07 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.948505+0000", 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016831699999999999 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.948515+0000", 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0614999999999999e-05 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.143 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.948523+0000", 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2049999999999996e-06 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.948587+0000", 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4791999999999999e-05 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.948607+0000", 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9480999999999999e-05 2024-02-05T08:10:49.144 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.077247+0000", 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.128639793 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.077294+0000", 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7327000000000003e-05 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.145 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:99796 7.d 7:b4052ca7:::benchmark_data_smithi134_83016_object99795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:08.949219+0000", 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 160.11699915599999, 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13502135700000001, 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 99796 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.949219+0000", 2024-02-05T08:10:49.146 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.949219+0000", 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.949221+0000", 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.364e-06 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.147 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.949241+0000", 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0744999999999999e-05 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.949253+0000", 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1984000000000001e-05 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.949261+0000", 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7789999999999996e-06 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.148 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.950036+0000", 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00077464400000000001 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.950175+0000", 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000138858 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.950327+0000", 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000152394 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.149 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.078760+0000", 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12843323100000001 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.083623+0000", 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0048624050000000002 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.084143+0000", 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052018099999999999 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.150 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.084207+0000", 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3848000000000003e-05 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.084241+0000", 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3924000000000002e-05 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106469 7.d 7:bc03ac98:::benchmark_data_smithi134_83016_object106468:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.279219+0000", 2024-02-05T08:10:49.151 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 152.78699978899999, 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12577576500000001, 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106469 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.279219+0000", 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.152 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.279219+0000", 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.279220+0000", 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.61e-06 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.279241+0000", 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0758e-05 2024-02-05T08:10:49.153 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.279254+0000", 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3012e-05 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.279262+0000", 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8830000000000006e-06 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.279331+0000", 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9132000000000002e-05 2024-02-05T08:10:49.154 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.279476+0000", 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000144599 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.279739+0000", 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00026306499999999999 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.282093+0000", 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023542820000000001 2024-02-05T08:10:49.155 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.403635+0000", 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12154190500000001 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.404900+0000", 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0012649580000000001 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.404958+0000", 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8248000000000002e-05 2024-02-05T08:10:49.156 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.404995+0000", 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6313000000000003e-05 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106686 7.d 7:b6c5ae00:::benchmark_data_smithi134_83016_object106685:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.637152+0000", 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 152.42906633800001, 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.124828418, 2024-02-05T08:10:49.157 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106686 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.637152+0000", 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.158 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.637152+0000", 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.637154+0000", 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9570000000000001e-06 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.637189+0000", 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4560000000000001e-05 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.159 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.637205+0000", 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6184e-05 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.637215+0000", 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9299999999999998e-06 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.637281+0000", 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6568000000000004e-05 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.160 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.637427+0000", 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014577200000000001 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.637578+0000", 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000150576 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.643207+0000", 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0056291300000000004 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.161 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.644421+0000", 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0012140009999999999 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.761701+0000", 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.117280074 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.761933+0000", 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000231926 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.162 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.163 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.163 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.761981+0000", 2024-02-05T08:10:49.163 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.774e-05 2024-02-05T08:10:49.163 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.163 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.163 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.163 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.163 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.163 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:147162 7.0 e134/95 7:00cb83f3:::benchmark_data_smithi134_83016_object37228:head v 134'12188, mlcod=134'12167)", 2024-02-05T08:10:49.164 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:09:01.984755+0000", 2024-02-05T08:10:49.164 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 107.08146320500001, 2024-02-05T08:10:49.164 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12335964200000001, 2024-02-05T08:10:49.164 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.164 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.164 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.164 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:01.984755+0000", 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:01.984755+0000", 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:01.984756+0000", 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2999999999999999e-07 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.165 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:01.984762+0000", 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7740000000000001e-06 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:01.984781+0000", 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8745000000000001e-05 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:01.984790+0000", 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9250000000000001e-06 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.166 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:02.047199+0000", 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.062409887999999997 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:02.047216+0000", 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7003999999999998e-05 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:02.108074+0000", 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.060857535999999997 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.167 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:02.108115+0000", 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0939999999999998e-05 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:166244 7.d 7:bf7d0b65:::benchmark_data_smithi134_83016_object56310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:09:22.632441+0000", 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 86.433777933000002, 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.126325942, 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.168 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 166244 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.632441+0000", 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.632441+0000", 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.169 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.632442+0000", 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2839999999999999e-06 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.632447+0000", 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9250000000000003e-06 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.632460+0000", 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2771e-05 2024-02-05T08:10:49.170 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.632468+0000", 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1040000000000007e-06 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.632518+0000", 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9839000000000002e-05 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.632623+0000", 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010564 2024-02-05T08:10:49.171 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.632735+0000", 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111759 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.640058+0000", 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0073234909999999997 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.741255+0000", 2024-02-05T08:10:49.172 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.101196651 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.758678+0000", 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.017422975 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.758727+0000", 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8480999999999999e-05 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.758767+0000", 2024-02-05T08:10:49.173 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0021999999999997e-05 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:166248 7.d 7:bfe5a98c:::benchmark_data_smithi134_83016_object56314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:09:22.635037+0000", 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 86.431181230000007, 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.123863316, 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.174 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 166248 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.635037+0000", 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.635037+0000", 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.175 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.635038+0000", 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0699999999999999e-06 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.635041+0000", 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8839999999999998e-06 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.635049+0000", 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0460000000000006e-06 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.176 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.635055+0000", 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9270000000000002e-06 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.635450+0000", 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00039511499999999997 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.635554+0000", 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000103217 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.177 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.635661+0000", 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107909 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.642348+0000", 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0066864719999999997 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.741291+0000", 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.098942890000000006 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.178 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.758821+0000", 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.017530173999999999 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.758866+0000", 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5207999999999998e-05 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.758901+0000", 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4403999999999998e-05 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.179 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.180 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.180 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.180 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:166250 7.d 7:bb5a8a3e:::benchmark_data_smithi134_83016_object56316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T08:10:49.180 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:09:22.636781+0000", 2024-02-05T08:10:49.180 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 86.429437285999995, 2024-02-05T08:10:49.180 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.122352883, 2024-02-05T08:10:49.180 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:10:49.180 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:10:49.180 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:10:49.180 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:10:49.180 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:10:49.180 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 166250 2024-02-05T08:10:49.180 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.636781+0000", 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.636781+0000", 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.636782+0000", 2024-02-05T08:10:49.181 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6199999999999998e-07 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.636784+0000", 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4590000000000001e-06 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.636790+0000", 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0909999999999998e-06 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.636795+0000", 2024-02-05T08:10:49.182 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1830000000000002e-06 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.640086+0000", 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003290332 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.640200+0000", 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011455600000000001 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.640306+0000", 2024-02-05T08:10:49.183 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010552600000000001 2024-02-05T08:10:49.184 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.184 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.184 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:10:49.184 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.739857+0000", 2024-02-05T08:10:49.184 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.099550924999999998 2024-02-05T08:10:49.184 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.184 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.184 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.184 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.754119+0000", 2024-02-05T08:10:49.184 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.014262323 2024-02-05T08:10:49.184 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.184 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.184 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.759071+0000", 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0049523350000000004 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.759116+0000", 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4384000000000001e-05 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:09:22.759134+0000", 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8306999999999999e-05 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.185 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:10:49.186 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:10:49.186 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:10:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:49.187 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:49.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:48 smithi134 ceph-mon[30086]: pgmap v239: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:10:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:48 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4255905781' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:10:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:48 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:10:49.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:48 smithi134 ceph-mon[39278]: pgmap v239: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:10:49.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:48 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4255905781' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:10:49.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:48 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:10:49.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:49.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:48 smithi196 ceph-mon[33018]: pgmap v239: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:10:49.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:48 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4255905781' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:10:49.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:48 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:10:49.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:49.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:49.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:49.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:49.443 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:49.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:49.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:49.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:49.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:49.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:49.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:49.701 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:49.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:49.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:10:49.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:49.830 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:49.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:49.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:49.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:49.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:49.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:49 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:10:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:49 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:10:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:49 smithi134 ceph-mon[30086]: osdmap e156: 8 total, 8 up, 7 in 2024-02-05T08:10:50.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:49 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:10:50.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:49 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:10:50.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:49 smithi134 ceph-mon[39278]: osdmap e156: 8 total, 8 up, 7 in 2024-02-05T08:10:50.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:50.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:50.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:50.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:50.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:50.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:49 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:10:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:49 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:10:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:49 smithi196 ceph-mon[33018]: osdmap e156: 8 total, 8 up, 7 in 2024-02-05T08:10:50.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:50.345 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:50.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:50.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:50.447 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:50.474 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:50.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:50.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:50.603 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:50.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:50.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:50.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:50.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:50.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:50.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:50.834 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:50.862 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:50.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:50.883 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:50.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:50.988 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:50.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:51.089 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:51.113 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:51.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:51.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:51.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:51.238 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:51.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:51.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:50 smithi134 ceph-mon[30086]: pgmap v241: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:10:51.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:50 smithi134 ceph-mon[39278]: pgmap v241: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:10:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:50 smithi196 ceph-mon[33018]: pgmap v241: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:10:51.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:51.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:51.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:51.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T08:10:51.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:51.498 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:51.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:51.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:51.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:51.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:51.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:51.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:51.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:51.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:51.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:51.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:51.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:51.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:51.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:10:51.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:52.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:52.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:52.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:52.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:52.144 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:52.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:52.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:52.247 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:52.272 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:52.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:52.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:52.397 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:52.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:52.498 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:52.522 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:52.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:52.624 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:52.652 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:52.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:52.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:52.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:52.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:52.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:52.883 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:52.908 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:52.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:52.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:53.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:53.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:53.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:53.088 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:53.090 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:53.139 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:53.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:53.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:53.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:53.202 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:52 smithi134 ceph-mon[30086]: pgmap v242: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 584 B/s rd, 0 op/s 2024-02-05T08:10:53.202 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:52 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:10:53.202 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:52 smithi134 ceph-mon[39278]: pgmap v242: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 584 B/s rd, 0 op/s 2024-02-05T08:10:53.202 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:52 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:10:53.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:53.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:53.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:53.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:52 smithi196 ceph-mon[33018]: pgmap v242: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 584 B/s rd, 0 op/s 2024-02-05T08:10:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:52 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:10:53.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:10:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:10:53.395 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:53.419 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:53.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:53.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:53.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:53.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:53.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:53.599 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:53.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:53.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:53.674 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:53.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:53.775 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:53.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:53.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:53.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:53.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:53.930 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:53.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:53.982 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'unique_pool_1' created 2024-02-05T08:10:54.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:54.060 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:54.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:54.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:54.098 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:53 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:54.098 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:53 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/200114253' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T08:10:54.098 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:53 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T08:10:54.099 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:53 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:54.099 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:53 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/200114253' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T08:10:54.099 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:53 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T08:10:54.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:54.189 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:54.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:54.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:54.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:53 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:10:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:53 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/200114253' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T08:10:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:53 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T08:10:54.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:54.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:54.420 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:54.446 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:54.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:54.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:54.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:54.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:54.664 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:10:54.667 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:10:54.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:54.700 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:54.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:54.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:54.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:54.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:54.915 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-02-05T08:10:54.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:54.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:54.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:54.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:55.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:55.081 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:55.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:55.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:55.205 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:55.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:55.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:54 smithi134 ceph-mon[30086]: pgmap v243: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:10:55.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:54 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-05T08:10:55.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:54 smithi134 ceph-mon[30086]: osdmap e157: 8 total, 8 up, 7 in 2024-02-05T08:10:55.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:54 smithi134 ceph-mon[39278]: pgmap v243: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:10:55.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:54 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-05T08:10:55.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:54 smithi134 ceph-mon[39278]: osdmap e157: 8 total, 8 up, 7 in 2024-02-05T08:10:55.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:55.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:54 smithi196 ceph-mon[33018]: pgmap v243: 132 pgs: 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:10:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:54 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-05T08:10:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:54 smithi196 ceph-mon[33018]: osdmap e157: 8 total, 8 up, 7 in 2024-02-05T08:10:55.336 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:55.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:55.438 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:55.510 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:55.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:55.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:55.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:55.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:55.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:55.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:55.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:55.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:55.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:55.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:55.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:55.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:55.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:56.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:56.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:56.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:56.126 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:56.131 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:55 smithi134 ceph-mon[30086]: from='client.24926 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:10:56.132 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:55 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:10:56.132 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:55 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:10:56.132 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:55 smithi134 ceph-mon[30086]: osdmap e158: 8 total, 8 up, 7 in 2024-02-05T08:10:56.132 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:55 smithi134 ceph-mon[39278]: from='client.24926 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:10:56.132 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:55 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:10:56.132 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:55 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:10:56.132 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:55 smithi134 ceph-mon[39278]: osdmap e158: 8 total, 8 up, 7 in 2024-02-05T08:10:56.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:56.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:56.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:56.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:56.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:56.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:56.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:55 smithi196 ceph-mon[33018]: from='client.24926 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:10:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:55 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:10:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:55 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:10:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:55 smithi196 ceph-mon[33018]: osdmap e158: 8 total, 8 up, 7 in 2024-02-05T08:10:56.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:56.408 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:56.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:56.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:56.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:56.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:56.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:56.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:56.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:56.768 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:56.793 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:56.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:56.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:56.918 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:56.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:56.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:57.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:57.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:56 smithi134 ceph-mon[30086]: pgmap v246: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 768 B/s rd, 0 op/s 2024-02-05T08:10:57.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:56 smithi134 ceph-mon[30086]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:10:57.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:56 smithi134 ceph-mon[30086]: osdmap e159: 8 total, 8 up, 7 in 2024-02-05T08:10:57.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:56 smithi134 ceph-mon[39278]: pgmap v246: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 768 B/s rd, 0 op/s 2024-02-05T08:10:57.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:56 smithi134 ceph-mon[39278]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:10:57.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:56 smithi134 ceph-mon[39278]: osdmap e159: 8 total, 8 up, 7 in 2024-02-05T08:10:57.048 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:57.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:57.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:57.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:57.167 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-02-05T08:10:53.973579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1296,"num_read_kb":1142,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":2363363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":649,"ondisk_log_size":649,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1022768,"kb_used_data":6720,"kb_used_omap":10,"kb_used_meta":1015029,"kb_avail":655193296,"statfs":{"total":671965249536,"available":670917935104,"internally_reserved":0,"allocated":6881280,"data_stored":4746203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11155,"internal_metadata":1039389805},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.369530"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":202,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.945846+0000","last_change":"2024-02-05T08:10:26.432413+0000","last_active":"2024-02-05T08:10:48.945846+0000","last_peered":"2024-02-05T08:10:48.945846+0000","last_clean":"2024-02-05T08:10:48.945846+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:10:48.945846+0000","last_undegraded":"2024-02-05T08:10:48.945846+0000","last_fullsized":"2024-02-05T08:10:48.945846+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:26.432294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:26.432294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.412548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":207,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946339+0000","last_change":"2024-02-05T08:08:50.003502+0000","last_active":"2024-02-05T08:10:48.946339+0000","last_peered":"2024-02-05T08:10:48.946339+0000","last_clean":"2024-02-05T08:10:48.946339+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:10:48.946339+0000","last_undegraded":"2024-02-05T08:10:48.946339+0000","last_fullsized":"2024-02-05T08:10:48.946339+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:50.003400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:50.003400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.486932+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":214,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177394+0000","last_change":"2024-02-05T08:10:18.254802+0000","last_active":"2024-02-05T08:10:50.177394+0000","last_peered":"2024-02-05T08:10:50.177394+0000","last_clean":"2024-02-05T08:10:50.177394+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:10:50.177394+0000","last_undegraded":"2024-02-05T08:10:50.177394+0000","last_fullsized":"2024-02-05T08:10:50.177394+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.254568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.254568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:46.157955+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"90'13","reported_seq":238,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.945549+0000","last_change":"2024-02-05T08:08:31.425402+0000","last_active":"2024-02-05T08:10:48.945549+0000","last_peered":"2024-02-05T08:10:48.945549+0000","last_clean":"2024-02-05T08:10:48.945549+0000","last_became_active":"2024-02-05T08:06:39.560434+0000","last_became_peered":"2024-02-05T08:06:39.560434+0000","last_unstale":"2024-02-05T08:10:48.945549+0000","last_undegraded":"2024-02-05T08:10:48.945549+0000","last_fullsized":"2024-02-05T08:10:48.945549+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:08:31.425318+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:08:31.425318+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.443421+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":208,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943310+0000","last_change":"2024-02-05T08:09:37.106387+0000","last_active":"2024-02-05T08:10:48.943310+0000","last_peered":"2024-02-05T08:10:48.943310+0000","last_clean":"2024-02-05T08:10:48.943310+0000","last_became_active":"2024-02-05T08:06:39.558790+0000","last_became_peered":"2024-02-05T08:06:39.558790+0000","last_unstale":"2024-02-05T08:10:48.943310+0000","last_undegraded":"2024-02-05T08:10:48.943310+0000","last_fullsized":"2024-02-05T08:10:48.943310+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:37.106147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:37.106147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:05.992123+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":204,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.945761+0000","last_change":"2024-02-05T08:08:47.432025+0000","last_active":"2024-02-05T08:10:48.945761+0000","last_peered":"2024-02-05T08:10:48.945761+0000","last_clean":"2024-02-05T08:10:48.945761+0000","last_became_active":"2024-02-05T08:06:39.561731+0000","last_became_peered":"2024-02-05T08:06:39.561731+0000","last_unstale":"2024-02-05T08:10:48.945761+0000","last_undegraded":"2024-02-05T08:10:48.945761+0000","last_fullsized":"2024-02-05T08:10:48.945761+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:47.431940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:47.431940+0000","last_clean_scrub_stamp":"2024-02-05T08:08:47.431940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.443527+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":221,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943842+0000","last_change":"2024-02-05T08:10:28.007794+0000","last_active":"2024-02-05T08:10:48.943842+0000","last_peered":"2024-02-05T08:10:48.943842+0000","last_clean":"2024-02-05T08:10:48.943842+0000","last_became_active":"2024-02-05T08:06:39.558101+0000","last_became_peered":"2024-02-05T08:06:39.558101+0000","last_unstale":"2024-02-05T08:10:48.943842+0000","last_undegraded":"2024-02-05T08:10:48.943842+0000","last_fullsized":"2024-02-05T08:10:48.943842+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:28.007713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:28.007713+0000","last_clean_scrub_stamp":"2024-02-05T08:10:28.007713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:39.654852+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"118'22","reported_seq":234,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943337+0000","last_change":"2024-02-05T08:07:45.819143+0000","last_active":"2024-02-05T08:10:48.943337+0000","last_peered":"2024-02-05T08:10:48.943337+0000","last_clean":"2024-02-05T08:10:48.943337+0000","last_became_active":"2024-02-05T08:04:43.560660+0000","last_became_peered":"2024-02-05T08:04:43.560660+0000","last_unstale":"2024-02-05T08:10:48.943337+0000","last_undegraded":"2024-02-05T08:10:48.943337+0000","last_fullsized":"2024-02-05T08:10:48.943337+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:07:45.819004+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.941068+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":202,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177483+0000","last_change":"2024-02-05T08:10:29.223232+0000","last_active":"2024-02-05T08:10:50.177483+0000","last_peered":"2024-02-05T08:10:50.177483+0000","last_clean":"2024-02-05T08:10:50.177483+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:10:50.177483+0000","last_undegraded":"2024-02-05T08:10:50.177483+0000","last_fullsized":"2024-02-05T08:10:50.177483+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:29.223145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:29.223145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:55.416210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":206,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034166+0000","last_change":"2024-02-05T08:10:35.612852+0000","last_active":"2024-02-05T08:10:49.034166+0000","last_peered":"2024-02-05T08:10:49.034166+0000","last_clean":"2024-02-05T08:10:49.034166+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:10:49.034166+0000","last_undegraded":"2024-02-05T08:10:49.034166+0000","last_fullsized":"2024-02-05T08:10:49.034166+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:10:35.612720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:52.008786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":220,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034305+0000","last_change":"2024-02-05T08:09:39.410126+0000","last_active":"2024-02-05T08:10:49.034305+0000","last_peered":"2024-02-05T08:10:49.034305+0000","last_clean":"2024-02-05T08:10:49.034305+0000","last_became_active":"2024-02-05T08:06:39.559461+0000","last_became_peered":"2024-02-05T08:06:39.559461+0000","last_unstale":"2024-02-05T08:10:49.034305+0000","last_undegraded":"2024-02-05T08:10:49.034305+0000","last_fullsized":"2024-02-05T08:10:49.034305+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:39.410054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.410054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.039707+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":234,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946033+0000","last_change":"2024-02-05T08:08:38.437994+0000","last_active":"2024-02-05T08:10:48.946033+0000","last_peered":"2024-02-05T08:10:48.946033+0000","last_clean":"2024-02-05T08:10:48.946033+0000","last_became_active":"2024-02-05T08:07:22.653892+0000","last_became_peered":"2024-02-05T08:07:22.653892+0000","last_unstale":"2024-02-05T08:10:48.946033+0000","last_undegraded":"2024-02-05T08:10:48.946033+0000","last_fullsized":"2024-02-05T08:10:48.946033+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:08:38.437850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:38.437850+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.443569+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":202,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.033706+0000","last_change":"2024-02-05T08:10:16.560606+0000","last_active":"2024-02-05T08:10:49.033706+0000","last_peered":"2024-02-05T08:10:49.033706+0000","last_clean":"2024-02-05T08:10:49.033706+0000","last_became_active":"2024-02-05T08:04:47.587515+0000","last_became_peered":"2024-02-05T08:04:47.587515+0000","last_unstale":"2024-02-05T08:10:49.033706+0000","last_undegraded":"2024-02-05T08:10:49.033706+0000","last_fullsized":"2024-02-05T08:10:49.033706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:16.560531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:16.560531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:46.471717+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":207,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.458052+0000","last_change":"2024-02-05T08:10:24.239442+0000","last_active":"2024-02-05T08:10:50.458052+0000","last_peered":"2024-02-05T08:10:50.458052+0000","last_clean":"2024-02-05T08:10:50.458052+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:10:50.458052+0000","last_undegraded":"2024-02-05T08:10:50.458052+0000","last_fullsized":"2024-02-05T08:10:50.458052+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:24.239334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:24.239334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:53.342841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":214,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034007+0000","last_change":"2024-02-05T08:10:23.663012+0000","last_active":"2024-02-05T08:10:49.034007+0000","last_peered":"2024-02-05T08:10:49.034007+0000","last_clean":"2024-02-05T08:10:49.034007+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:10:49.034007+0000","last_undegraded":"2024-02-05T08:10:49.034007+0000","last_fullsized":"2024-02-05T08:10:49.034007+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:23.662919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:10:23.662919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:40.039778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":212,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946332+0000","last_change":"2024-02-05T08:09:30.448604+0000","last_active":"2024-02-05T08:10:48.946332+0000","last_peered":"2024-02-05T08:10:48.946332+0000","last_clean":"2024-02-05T08:10:48.946332+0000","last_became_active":"2024-02-05T08:04:43.557559+0000","last_became_peered":"2024-02-05T08:04:43.557559+0000","last_unstale":"2024-02-05T08:10:48.946332+0000","last_undegraded":"2024-02-05T08:10:48.946332+0000","last_fullsized":"2024-02-05T08:10:48.946332+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:09:30.448493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:30.448493+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.443663+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":209,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946915+0000","last_change":"2024-02-05T08:09:26.466381+0000","last_active":"2024-02-05T08:10:48.946915+0000","last_peered":"2024-02-05T08:10:48.946915+0000","last_clean":"2024-02-05T08:10:48.946915+0000","last_became_active":"2024-02-05T08:06:39.547906+0000","last_became_peered":"2024-02-05T08:06:39.547906+0000","last_unstale":"2024-02-05T08:10:48.946915+0000","last_undegraded":"2024-02-05T08:10:48.946915+0000","last_fullsized":"2024-02-05T08:10:48.946915+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:26.466302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:09:26.466302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:54.690678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":199,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946571+0000","last_change":"2024-02-05T08:07:48.943001+0000","last_active":"2024-02-05T08:10:48.946571+0000","last_peered":"2024-02-05T08:10:48.946571+0000","last_clean":"2024-02-05T08:10:48.946571+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:10:48.946571+0000","last_undegraded":"2024-02-05T08:10:48.946571+0000","last_fullsized":"2024-02-05T08:10:48.946571+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:07:48.942873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.486832+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":206,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177097+0000","last_change":"2024-02-05T08:08:28.601390+0000","last_active":"2024-02-05T08:10:50.177097+0000","last_peered":"2024-02-05T08:10:50.177097+0000","last_clean":"2024-02-05T08:10:50.177097+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:10:50.177097+0000","last_undegraded":"2024-02-05T08:10:50.177097+0000","last_fullsized":"2024-02-05T08:10:50.177097+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:28.601275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:28.601275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.286381+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":244,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.360482+0000","last_change":"2024-02-05T08:10:19.645020+0000","last_active":"2024-02-05T08:10:50.360482+0000","last_peered":"2024-02-05T08:10:50.360482+0000","last_clean":"2024-02-05T08:10:50.360482+0000","last_became_active":"2024-02-05T08:04:43.557496+0000","last_became_peered":"2024-02-05T08:04:43.557496+0000","last_unstale":"2024-02-05T08:10:50.360482+0000","last_undegraded":"2024-02-05T08:10:50.360482+0000","last_fullsized":"2024-02-05T08:10:50.360482+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:10:19.644900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:19.644900+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:31.573739+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":200,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.944033+0000","last_change":"2024-02-05T08:09:21.354430+0000","last_active":"2024-02-05T08:10:48.944033+0000","last_peered":"2024-02-05T08:10:48.944033+0000","last_clean":"2024-02-05T08:10:48.944033+0000","last_became_active":"2024-02-05T08:06:39.560948+0000","last_became_peered":"2024-02-05T08:06:39.560948+0000","last_unstale":"2024-02-05T08:10:48.944033+0000","last_undegraded":"2024-02-05T08:10:48.944033+0000","last_fullsized":"2024-02-05T08:10:48.944033+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:21.354158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.354158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.941223+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":206,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946461+0000","last_change":"2024-02-05T08:10:30.419053+0000","last_active":"2024-02-05T08:10:48.946461+0000","last_peered":"2024-02-05T08:10:48.946461+0000","last_clean":"2024-02-05T08:10:48.946461+0000","last_became_active":"2024-02-05T08:04:45.563948+0000","last_became_peered":"2024-02-05T08:04:45.563948+0000","last_unstale":"2024-02-05T08:10:48.946461+0000","last_undegraded":"2024-02-05T08:10:48.946461+0000","last_fullsized":"2024-02-05T08:10:48.946461+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:30.418931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.418931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:51.770800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":206,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946126+0000","last_change":"2024-02-05T08:09:27.431319+0000","last_active":"2024-02-05T08:10:48.946126+0000","last_peered":"2024-02-05T08:10:48.946126+0000","last_clean":"2024-02-05T08:10:48.946126+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:10:48.946126+0000","last_undegraded":"2024-02-05T08:10:48.946126+0000","last_fullsized":"2024-02-05T08:10:48.946126+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:27.431221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:27.431221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:50.510892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"90'17","reported_seq":238,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943583+0000","last_change":"2024-02-05T08:10:30.977423+0000","last_active":"2024-02-05T08:10:48.943583+0000","last_peered":"2024-02-05T08:10:48.943583+0000","last_clean":"2024-02-05T08:10:48.943583+0000","last_became_active":"2024-02-05T08:04:43.557596+0000","last_became_peered":"2024-02-05T08:04:43.557596+0000","last_unstale":"2024-02-05T08:10:48.943583+0000","last_undegraded":"2024-02-05T08:10:48.943583+0000","last_fullsized":"2024-02-05T08:10:48.943583+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:10:30.977334+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.977334+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:45.611556+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":205,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.458605+0000","last_change":"2024-02-05T08:08:48.447437+0000","last_active":"2024-02-05T08:10:50.458605+0000","last_peered":"2024-02-05T08:10:50.458605+0000","last_clean":"2024-02-05T08:10:50.458605+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:10:50.458605+0000","last_undegraded":"2024-02-05T08:10:50.458605+0000","last_fullsized":"2024-02-05T08:10:50.458605+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:08:48.447317+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":224,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:51.662052+0000","last_change":"2024-02-05T08:08:44.441278+0000","last_active":"2024-02-05T08:10:51.662052+0000","last_peered":"2024-02-05T08:10:51.662052+0000","last_clean":"2024-02-05T08:10:51.662052+0000","last_became_active":"2024-02-05T08:06:39.552760+0000","last_became_peered":"2024-02-05T08:06:39.552760+0000","last_unstale":"2024-02-05T08:10:51.662052+0000","last_undegraded":"2024-02-05T08:10:51.662052+0000","last_fullsized":"2024-02-05T08:10:51.662052+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:08:44.441144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.441144+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":216,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946254+0000","last_change":"2024-02-05T08:10:32.403469+0000","last_active":"2024-02-05T08:10:48.946254+0000","last_peered":"2024-02-05T08:10:48.946254+0000","last_clean":"2024-02-05T08:10:48.946254+0000","last_became_active":"2024-02-05T08:04:41.532339+0000","last_became_peered":"2024-02-05T08:04:41.532339+0000","last_unstale":"2024-02-05T08:10:48.946254+0000","last_undegraded":"2024-02-05T08:10:48.946254+0000","last_fullsized":"2024-02-05T08:10:48.946254+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:10:32.403342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:32.403342+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:58.710603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":218,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.178023+0000","last_change":"2024-02-05T08:09:24.000902+0000","last_active":"2024-02-05T08:10:50.178023+0000","last_peered":"2024-02-05T08:10:50.178023+0000","last_clean":"2024-02-05T08:10:50.178023+0000","last_became_active":"2024-02-05T08:04:43.605952+0000","last_became_peered":"2024-02-05T08:04:43.605952+0000","last_unstale":"2024-02-05T08:10:50.178023+0000","last_undegraded":"2024-02-05T08:10:50.178023+0000","last_fullsized":"2024-02-05T08:10:50.178023+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:09:24.000675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:24.000675+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:53.229695+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":203,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946659+0000","last_change":"2024-02-05T08:09:25.266619+0000","last_active":"2024-02-05T08:10:48.946659+0000","last_peered":"2024-02-05T08:10:48.946659+0000","last_clean":"2024-02-05T08:10:48.946659+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:10:48.946659+0000","last_undegraded":"2024-02-05T08:10:48.946659+0000","last_fullsized":"2024-02-05T08:10:48.946659+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.266508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.266508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.487058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":199,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.178165+0000","last_change":"2024-02-05T08:08:42.676027+0000","last_active":"2024-02-05T08:10:50.178165+0000","last_peered":"2024-02-05T08:10:50.178165+0000","last_clean":"2024-02-05T08:10:50.178165+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:10:50.178165+0000","last_undegraded":"2024-02-05T08:10:50.178165+0000","last_fullsized":"2024-02-05T08:10:50.178165+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:42.675782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:42.675782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:56.293782+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":206,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.942897+0000","last_change":"2024-02-05T08:07:50.777386+0000","last_active":"2024-02-05T08:10:48.942897+0000","last_peered":"2024-02-05T08:10:48.942897+0000","last_clean":"2024-02-05T08:10:48.942897+0000","last_became_active":"2024-02-05T08:04:41.534448+0000","last_became_peered":"2024-02-05T08:04:41.534448+0000","last_unstale":"2024-02-05T08:10:48.942897+0000","last_undegraded":"2024-02-05T08:10:48.942897+0000","last_fullsized":"2024-02-05T08:10:48.942897+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:50.777273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:07:50.777273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.941087+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":196,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034685+0000","last_change":"2024-02-05T08:09:25.380533+0000","last_active":"2024-02-05T08:10:49.034685+0000","last_peered":"2024-02-05T08:10:49.034685+0000","last_clean":"2024-02-05T08:10:49.034685+0000","last_became_active":"2024-02-05T08:06:39.558747+0000","last_became_peered":"2024-02-05T08:06:39.558747+0000","last_unstale":"2024-02-05T08:10:49.034685+0000","last_undegraded":"2024-02-05T08:10:49.034685+0000","last_fullsized":"2024-02-05T08:10:49.034685+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:09:25.380390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.380390+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:54.610020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":195,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.458386+0000","last_change":"2024-02-05T08:08:31.483194+0000","last_active":"2024-02-05T08:10:50.458386+0000","last_peered":"2024-02-05T08:10:50.458386+0000","last_clean":"2024-02-05T08:10:50.458386+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:10:50.458386+0000","last_undegraded":"2024-02-05T08:10:50.458386+0000","last_fullsized":"2024-02-05T08:10:50.458386+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:31.483087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:31.483087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":145,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.945682+0000","last_change":"2024-02-05T08:09:36.434985+0000","last_active":"2024-02-05T08:10:48.945682+0000","last_peered":"2024-02-05T08:10:48.945682+0000","last_clean":"2024-02-05T08:10:48.945682+0000","last_became_active":"2024-02-05T08:06:39.551173+0000","last_became_peered":"2024-02-05T08:06:39.551173+0000","last_unstale":"2024-02-05T08:10:48.945682+0000","last_undegraded":"2024-02-05T08:10:48.945682+0000","last_fullsized":"2024-02-05T08:10:48.945682+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:36.434873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.434873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:05.817457+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":206,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177167+0000","last_change":"2024-02-05T08:08:30.632883+0000","last_active":"2024-02-05T08:10:50.177167+0000","last_peered":"2024-02-05T08:10:50.177167+0000","last_clean":"2024-02-05T08:10:50.177167+0000","last_became_active":"2024-02-05T08:04:41.536019+0000","last_became_peered":"2024-02-05T08:04:41.536019+0000","last_unstale":"2024-02-05T08:10:50.177167+0000","last_undegraded":"2024-02-05T08:10:50.177167+0000","last_fullsized":"2024-02-05T08:10:50.177167+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:30.632785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:30.632785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.286348+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":220,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.942906+0000","last_change":"2024-02-05T08:08:40.082709+0000","last_active":"2024-02-05T08:10:48.942906+0000","last_peered":"2024-02-05T08:10:48.942906+0000","last_clean":"2024-02-05T08:10:48.942906+0000","last_became_active":"2024-02-05T08:06:39.560932+0000","last_became_peered":"2024-02-05T08:06:39.560932+0000","last_unstale":"2024-02-05T08:10:48.942906+0000","last_undegraded":"2024-02-05T08:10:48.942906+0000","last_fullsized":"2024-02-05T08:10:48.942906+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-02-05T08:08:40.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:40.082577+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.941196+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":208,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943397+0000","last_change":"2024-02-05T08:10:18.022780+0000","last_active":"2024-02-05T08:10:48.943397+0000","last_peered":"2024-02-05T08:10:48.943397+0000","last_clean":"2024-02-05T08:10:48.943397+0000","last_became_active":"2024-02-05T08:06:39.559212+0000","last_became_peered":"2024-02-05T08:06:39.559212+0000","last_unstale":"2024-02-05T08:10:48.943397+0000","last_undegraded":"2024-02-05T08:10:48.943397+0000","last_fullsized":"2024-02-05T08:10:48.943397+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.022670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.022670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:22.583988+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":213,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.458109+0000","last_change":"2024-02-05T08:09:36.353456+0000","last_active":"2024-02-05T08:10:50.458109+0000","last_peered":"2024-02-05T08:10:50.458109+0000","last_clean":"2024-02-05T08:10:50.458109+0000","last_became_active":"2024-02-05T08:06:39.555597+0000","last_became_peered":"2024-02-05T08:06:39.555597+0000","last_unstale":"2024-02-05T08:10:50.458109+0000","last_undegraded":"2024-02-05T08:10:50.458109+0000","last_fullsized":"2024-02-05T08:10:50.458109+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.353351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":214,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034510+0000","last_change":"2024-02-05T08:08:45.166139+0000","last_active":"2024-02-05T08:10:49.034510+0000","last_peered":"2024-02-05T08:10:49.034510+0000","last_clean":"2024-02-05T08:10:49.034510+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:10:49.034510+0000","last_undegraded":"2024-02-05T08:10:49.034510+0000","last_fullsized":"2024-02-05T08:10:49.034510+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:45.166006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:45.166006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"141'16","reported_seq":252,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943235+0000","last_change":"2024-02-05T08:09:39.113079+0000","last_active":"2024-02-05T08:10:48.943235+0000","last_peered":"2024-02-05T08:10:48.943235+0000","last_clean":"2024-02-05T08:10:48.943235+0000","last_became_active":"2024-02-05T08:06:39.556873+0000","last_became_peered":"2024-02-05T08:06:39.556873+0000","last_unstale":"2024-02-05T08:10:48.943235+0000","last_undegraded":"2024-02-05T08:10:48.943235+0000","last_fullsized":"2024-02-05T08:10:48.943235+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'16","last_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.112936+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.939176+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":194,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177212+0000","last_change":"2024-02-05T08:07:45.514975+0000","last_active":"2024-02-05T08:10:50.177212+0000","last_peered":"2024-02-05T08:10:50.177212+0000","last_clean":"2024-02-05T08:10:50.177212+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:10:50.177212+0000","last_undegraded":"2024-02-05T08:10:50.177212+0000","last_fullsized":"2024-02-05T08:10:50.177212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:45.514863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:45.514863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.286295+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":206,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943230+0000","last_change":"2024-02-05T08:10:26.987843+0000","last_active":"2024-02-05T08:10:48.943230+0000","last_peered":"2024-02-05T08:10:48.943230+0000","last_clean":"2024-02-05T08:10:48.943230+0000","last_became_active":"2024-02-05T08:04:45.568484+0000","last_became_peered":"2024-02-05T08:04:45.568484+0000","last_unstale":"2024-02-05T08:10:48.943230+0000","last_undegraded":"2024-02-05T08:10:48.943230+0000","last_fullsized":"2024-02-05T08:10:48.943230+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:26.987763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:26.987763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:37.413701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":215,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.457546+0000","last_change":"2024-02-05T08:10:31.270781+0000","last_active":"2024-02-05T08:10:50.457546+0000","last_peered":"2024-02-05T08:10:50.457546+0000","last_clean":"2024-02-05T08:10:50.457546+0000","last_became_active":"2024-02-05T08:04:41.539698+0000","last_became_peered":"2024-02-05T08:04:41.539698+0000","last_unstale":"2024-02-05T08:10:50.457546+0000","last_undegraded":"2024-02-05T08:10:50.457546+0000","last_fullsized":"2024-02-05T08:10:50.457546+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:31.270686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:31.270686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:35.517948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":243,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946252+0000","last_change":"2024-02-05T08:08:43.941119+0000","last_active":"2024-02-05T08:10:48.946252+0000","last_peered":"2024-02-05T08:10:48.946252+0000","last_clean":"2024-02-05T08:10:48.946252+0000","last_became_active":"2024-02-05T08:06:39.560259+0000","last_became_peered":"2024-02-05T08:06:39.560259+0000","last_unstale":"2024-02-05T08:10:48.946252+0000","last_undegraded":"2024-02-05T08:10:48.946252+0000","last_fullsized":"2024-02-05T08:10:48.946252+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:08:43.940980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:43.940980+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.486901+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":194,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.942897+0000","last_change":"2024-02-05T08:09:29.262843+0000","last_active":"2024-02-05T08:10:48.942897+0000","last_peered":"2024-02-05T08:10:48.942897+0000","last_clean":"2024-02-05T08:10:48.942897+0000","last_became_active":"2024-02-05T08:04:47.585496+0000","last_became_peered":"2024-02-05T08:04:47.585496+0000","last_unstale":"2024-02-05T08:10:48.942897+0000","last_undegraded":"2024-02-05T08:10:48.942897+0000","last_fullsized":"2024-02-05T08:10:48.942897+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:09:29.262582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.939201+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":207,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946043+0000","last_change":"2024-02-05T08:10:25.493102+0000","last_active":"2024-02-05T08:10:48.946043+0000","last_peered":"2024-02-05T08:10:48.946043+0000","last_clean":"2024-02-05T08:10:48.946043+0000","last_became_active":"2024-02-05T08:04:45.560815+0000","last_became_peered":"2024-02-05T08:04:45.560815+0000","last_unstale":"2024-02-05T08:10:48.946043+0000","last_undegraded":"2024-02-05T08:10:48.946043+0000","last_fullsized":"2024-02-05T08:10:48.946043+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:25.493013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:25.493013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:33.003654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":220,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946521+0000","last_change":"2024-02-05T08:09:25.486212+0000","last_active":"2024-02-05T08:10:48.946521+0000","last_peered":"2024-02-05T08:10:48.946521+0000","last_clean":"2024-02-05T08:10:48.946521+0000","last_became_active":"2024-02-05T08:06:39.545576+0000","last_became_peered":"2024-02-05T08:06:39.545576+0000","last_unstale":"2024-02-05T08:10:48.946521+0000","last_undegraded":"2024-02-05T08:10:48.946521+0000","last_fullsized":"2024-02-05T08:10:48.946521+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.486141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.966886+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.486141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.443643+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":247,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946397+0000","last_change":"2024-02-05T08:10:32.387139+0000","last_active":"2024-02-05T08:10:48.946397+0000","last_peered":"2024-02-05T08:10:48.946397+0000","last_clean":"2024-02-05T08:10:48.946397+0000","last_became_active":"2024-02-05T08:06:39.558257+0000","last_became_peered":"2024-02-05T08:06:39.558257+0000","last_unstale":"2024-02-05T08:10:48.946397+0000","last_undegraded":"2024-02-05T08:10:48.946397+0000","last_fullsized":"2024-02-05T08:10:48.946397+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:10:32.387029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:32.387029+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.428808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":194,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177899+0000","last_change":"2024-02-05T08:08:36.679472+0000","last_active":"2024-02-05T08:10:50.177899+0000","last_peered":"2024-02-05T08:10:50.177899+0000","last_clean":"2024-02-05T08:10:50.177899+0000","last_became_active":"2024-02-05T08:04:47.584235+0000","last_became_peered":"2024-02-05T08:04:47.584235+0000","last_unstale":"2024-02-05T08:10:50.177899+0000","last_undegraded":"2024-02-05T08:10:50.177899+0000","last_fullsized":"2024-02-05T08:10:50.177899+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:36.679368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:36.679368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.286445+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":289,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034463+0000","last_change":"2024-02-05T08:10:14.583618+0000","last_active":"2024-02-05T08:10:49.034463+0000","last_peered":"2024-02-05T08:10:49.034463+0000","last_clean":"2024-02-05T08:10:49.034463+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:10:49.034463+0000","last_undegraded":"2024-02-05T08:10:49.034463+0000","last_fullsized":"2024-02-05T08:10:49.034463+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:10:14.583489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:14.583489+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:28.694742+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":215,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.457403+0000","last_change":"2024-02-05T08:09:19.417311+0000","last_active":"2024-02-05T08:10:50.457403+0000","last_peered":"2024-02-05T08:10:50.457403+0000","last_clean":"2024-02-05T08:10:50.457403+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:10:50.457403+0000","last_undegraded":"2024-02-05T08:10:50.457403+0000","last_fullsized":"2024-02-05T08:10:50.457403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:19.417231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.417231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"129'15","reported_seq":222,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943637+0000","last_change":"2024-02-05T08:08:37.128945+0000","last_active":"2024-02-05T08:10:48.943637+0000","last_peered":"2024-02-05T08:10:48.943637+0000","last_clean":"2024-02-05T08:10:48.943637+0000","last_became_active":"2024-02-05T08:04:43.605981+0000","last_became_peered":"2024-02-05T08:04:43.605981+0000","last_unstale":"2024-02-05T08:10:48.943637+0000","last_undegraded":"2024-02-05T08:10:48.943637+0000","last_fullsized":"2024-02-05T08:10:48.943637+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'15","last_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:08:37.128817+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.941168+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":194,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946967+0000","last_change":"2024-02-05T08:09:18.397753+0000","last_active":"2024-02-05T08:10:48.946967+0000","last_peered":"2024-02-05T08:10:48.946967+0000","last_clean":"2024-02-05T08:10:48.946967+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:10:48.946967+0000","last_undegraded":"2024-02-05T08:10:48.946967+0000","last_fullsized":"2024-02-05T08:10:48.946967+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:18.397684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:18.397684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":289,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.944067+0000","last_change":"2024-02-05T08:10:33.947077+0000","last_active":"2024-02-05T08:10:48.944067+0000","last_peered":"2024-02-05T08:10:48.944067+0000","last_clean":"2024-02-05T08:10:48.944067+0000","last_became_active":"2024-02-05T08:04:45.565240+0000","last_became_peered":"2024-02-05T08:04:45.565240+0000","last_unstale":"2024-02-05T08:10:48.944067+0000","last_undegraded":"2024-02-05T08:10:48.944067+0000","last_fullsized":"2024-02-05T08:10:48.944067+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:10:33.946950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:33.946950+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:36.197109+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":206,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177060+0000","last_change":"2024-02-05T08:08:44.680464+0000","last_active":"2024-02-05T08:10:50.177060+0000","last_peered":"2024-02-05T08:10:50.177060+0000","last_clean":"2024-02-05T08:10:50.177060+0000","last_became_active":"2024-02-05T08:04:41.540252+0000","last_became_peered":"2024-02-05T08:04:41.540252+0000","last_unstale":"2024-02-05T08:10:50.177060+0000","last_undegraded":"2024-02-05T08:10:50.177060+0000","last_fullsized":"2024-02-05T08:10:50.177060+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:44.680353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.680353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.286415+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":226,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034560+0000","last_change":"2024-02-05T08:09:36.437558+0000","last_active":"2024-02-05T08:10:49.034560+0000","last_peered":"2024-02-05T08:10:49.034560+0000","last_clean":"2024-02-05T08:10:49.034560+0000","last_became_active":"2024-02-05T08:04:43.606414+0000","last_became_peered":"2024-02-05T08:04:43.606414+0000","last_unstale":"2024-02-05T08:10:49.034560+0000","last_undegraded":"2024-02-05T08:10:49.034560+0000","last_fullsized":"2024-02-05T08:10:49.034560+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:09:36.437435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.437435+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":211,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.360665+0000","last_change":"2024-02-05T08:10:20.671244+0000","last_active":"2024-02-05T08:10:50.360665+0000","last_peered":"2024-02-05T08:10:50.360665+0000","last_clean":"2024-02-05T08:10:50.360665+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:10:50.360665+0000","last_undegraded":"2024-02-05T08:10:50.360665+0000","last_fullsized":"2024-02-05T08:10:50.360665+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:20.671130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:20.671130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:39.009177+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":206,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943957+0000","last_change":"2024-02-05T08:10:15.994743+0000","last_active":"2024-02-05T08:10:48.943957+0000","last_peered":"2024-02-05T08:10:48.943957+0000","last_clean":"2024-02-05T08:10:48.943957+0000","last_became_active":"2024-02-05T08:04:45.567622+0000","last_became_peered":"2024-02-05T08:04:45.567622+0000","last_unstale":"2024-02-05T08:10:48.943957+0000","last_undegraded":"2024-02-05T08:10:48.943957+0000","last_fullsized":"2024-02-05T08:10:48.943957+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:15.994621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:15.994621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:33.060659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":213,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.458135+0000","last_change":"2024-02-05T08:08:35.480440+0000","last_active":"2024-02-05T08:10:50.458135+0000","last_peered":"2024-02-05T08:10:50.458135+0000","last_clean":"2024-02-05T08:10:50.458135+0000","last_became_active":"2024-02-05T08:06:39.557018+0000","last_became_peered":"2024-02-05T08:06:39.557018+0000","last_unstale":"2024-02-05T08:10:50.458135+0000","last_undegraded":"2024-02-05T08:10:50.458135+0000","last_fullsized":"2024-02-05T08:10:50.458135+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:35.480367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:35.480367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":218,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.178031+0000","last_change":"2024-02-05T08:08:45.712868+0000","last_active":"2024-02-05T08:10:50.178031+0000","last_peered":"2024-02-05T08:10:50.178031+0000","last_clean":"2024-02-05T08:10:50.178031+0000","last_became_active":"2024-02-05T08:04:43.608489+0000","last_became_peered":"2024-02-05T08:04:43.608489+0000","last_unstale":"2024-02-05T08:10:50.178031+0000","last_undegraded":"2024-02-05T08:10:50.178031+0000","last_fullsized":"2024-02-05T08:10:50.178031+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:08:45.712396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:45.712396+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.286400+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":209,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.458594+0000","last_change":"2024-02-05T08:09:33.391323+0000","last_active":"2024-02-05T08:10:50.458594+0000","last_peered":"2024-02-05T08:10:50.458594+0000","last_clean":"2024-02-05T08:10:50.458594+0000","last_became_active":"2024-02-05T08:06:39.548837+0000","last_became_peered":"2024-02-05T08:06:39.548837+0000","last_unstale":"2024-02-05T08:10:50.458594+0000","last_undegraded":"2024-02-05T08:10:50.458594+0000","last_fullsized":"2024-02-05T08:10:50.458594+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:33.391219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:33.391219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":206,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034516+0000","last_change":"2024-02-05T08:10:31.622795+0000","last_active":"2024-02-05T08:10:49.034516+0000","last_peered":"2024-02-05T08:10:49.034516+0000","last_clean":"2024-02-05T08:10:49.034516+0000","last_became_active":"2024-02-05T08:04:45.560241+0000","last_became_peered":"2024-02-05T08:04:45.560241+0000","last_unstale":"2024-02-05T08:10:49.034516+0000","last_undegraded":"2024-02-05T08:10:49.034516+0000","last_fullsized":"2024-02-05T08:10:49.034516+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:10:31.622683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:58.356161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":137,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177620+0000","last_change":"2024-02-05T08:07:36.550407+0000","last_active":"2024-02-05T08:10:50.177620+0000","last_peered":"2024-02-05T08:10:50.177620+0000","last_clean":"2024-02-05T08:10:50.177620+0000","last_became_active":"2024-02-05T08:06:39.554673+0000","last_became_peered":"2024-02-05T08:06:39.554673+0000","last_unstale":"2024-02-05T08:10:50.177620+0000","last_undegraded":"2024-02-05T08:10:50.177620+0000","last_fullsized":"2024-02-05T08:10:50.177620+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:36.550290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:36.550290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.286277+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"116'9","reported_seq":222,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.944043+0000","last_change":"2024-02-05T08:10:32.930947+0000","last_active":"2024-02-05T08:10:48.944043+0000","last_peered":"2024-02-05T08:10:48.944043+0000","last_clean":"2024-02-05T08:10:48.944043+0000","last_became_active":"2024-02-05T08:04:43.606624+0000","last_became_peered":"2024-02-05T08:04:43.606624+0000","last_unstale":"2024-02-05T08:10:48.944043+0000","last_undegraded":"2024-02-05T08:10:48.944043+0000","last_fullsized":"2024-02-05T08:10:48.944043+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-02-05T08:10:32.930856+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:10:32.930856+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:36.345351+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":208,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034160+0000","last_change":"2024-02-05T08:08:41.139144+0000","last_active":"2024-02-05T08:10:49.034160+0000","last_peered":"2024-02-05T08:10:49.034160+0000","last_clean":"2024-02-05T08:10:49.034160+0000","last_became_active":"2024-02-05T08:06:39.551759+0000","last_became_peered":"2024-02-05T08:06:39.551759+0000","last_unstale":"2024-02-05T08:10:49.034160+0000","last_undegraded":"2024-02-05T08:10:49.034160+0000","last_fullsized":"2024-02-05T08:10:49.034160+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:41.139067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:41.139067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":227,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.457528+0000","last_change":"2024-02-05T08:10:28.309205+0000","last_active":"2024-02-05T08:10:50.457528+0000","last_peered":"2024-02-05T08:10:50.457528+0000","last_clean":"2024-02-05T08:10:50.457528+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:10:50.457528+0000","last_undegraded":"2024-02-05T08:10:50.457528+0000","last_fullsized":"2024-02-05T08:10:50.457528+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:10:28.309092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:28.309092+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:44.852371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":249,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.033768+0000","last_change":"2024-02-05T08:08:33.007515+0000","last_active":"2024-02-05T08:10:49.033768+0000","last_peered":"2024-02-05T08:10:49.033768+0000","last_clean":"2024-02-05T08:10:49.033768+0000","last_became_active":"2024-02-05T08:06:39.546607+0000","last_became_peered":"2024-02-05T08:06:39.546607+0000","last_unstale":"2024-02-05T08:10:49.033768+0000","last_undegraded":"2024-02-05T08:10:49.033768+0000","last_fullsized":"2024-02-05T08:10:49.033768+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:08:33.007404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:33.007404+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":300,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.458420+0000","last_change":"2024-02-05T08:09:20.414916+0000","last_active":"2024-02-05T08:10:50.458420+0000","last_peered":"2024-02-05T08:10:50.458420+0000","last_clean":"2024-02-05T08:10:50.458420+0000","last_became_active":"2024-02-05T08:06:39.556562+0000","last_became_peered":"2024-02-05T08:06:39.556562+0000","last_unstale":"2024-02-05T08:10:50.458420+0000","last_undegraded":"2024-02-05T08:10:50.458420+0000","last_fullsized":"2024-02-05T08:10:50.458420+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:09:20.414837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:20.414837+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":194,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943387+0000","last_change":"2024-02-05T08:09:25.266619+0000","last_active":"2024-02-05T08:10:48.943387+0000","last_peered":"2024-02-05T08:10:48.943387+0000","last_clean":"2024-02-05T08:10:48.943387+0000","last_became_active":"2024-02-05T08:04:47.586044+0000","last_became_peered":"2024-02-05T08:04:47.586044+0000","last_unstale":"2024-02-05T08:10:48.943387+0000","last_undegraded":"2024-02-05T08:10:48.943387+0000","last_fullsized":"2024-02-05T08:10:48.943387+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.266513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.266513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:53.928249+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":215,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.457743+0000","last_change":"2024-02-05T08:09:39.244280+0000","last_active":"2024-02-05T08:10:50.457743+0000","last_peered":"2024-02-05T08:10:50.457743+0000","last_clean":"2024-02-05T08:10:50.457743+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:10:50.457743+0000","last_undegraded":"2024-02-05T08:10:50.457743+0000","last_fullsized":"2024-02-05T08:10:50.457743+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:39.244179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.244179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:55.149887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":225,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946157+0000","last_change":"2024-02-05T08:09:31.469380+0000","last_active":"2024-02-05T08:10:48.946157+0000","last_peered":"2024-02-05T08:10:48.946157+0000","last_clean":"2024-02-05T08:10:48.946157+0000","last_became_active":"2024-02-05T08:04:43.556580+0000","last_became_peered":"2024-02-05T08:04:43.556580+0000","last_unstale":"2024-02-05T08:10:48.946157+0000","last_undegraded":"2024-02-05T08:10:48.946157+0000","last_fullsized":"2024-02-05T08:10:48.946157+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:09:31.469295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:31.469295+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.626653+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":207,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.360125+0000","last_change":"2024-02-05T08:09:23.533364+0000","last_active":"2024-02-05T08:10:50.360125+0000","last_peered":"2024-02-05T08:10:50.360125+0000","last_clean":"2024-02-05T08:10:50.360125+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:10:50.360125+0000","last_undegraded":"2024-02-05T08:10:50.360125+0000","last_fullsized":"2024-02-05T08:10:50.360125+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:23.533259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_clean_scrub_stamp":"2024-02-05T08:09:23.533259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.414351+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":203,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.457711+0000","last_change":"2024-02-05T08:10:21.221033+0000","last_active":"2024-02-05T08:10:50.457711+0000","last_peered":"2024-02-05T08:10:50.457711+0000","last_clean":"2024-02-05T08:10:50.457711+0000","last_became_active":"2024-02-05T08:04:47.587082+0000","last_became_peered":"2024-02-05T08:04:47.587082+0000","last_unstale":"2024-02-05T08:10:50.457711+0000","last_undegraded":"2024-02-05T08:10:50.457711+0000","last_fullsized":"2024-02-05T08:10:50.457711+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:21.220941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:21.220941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:37.669661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":207,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.360429+0000","last_change":"2024-02-05T08:09:29.579918+0000","last_active":"2024-02-05T08:10:50.360429+0000","last_peered":"2024-02-05T08:10:50.360429+0000","last_clean":"2024-02-05T08:10:50.360429+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:10:50.360429+0000","last_undegraded":"2024-02-05T08:10:50.360429+0000","last_fullsized":"2024-02-05T08:10:50.360429+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:29.578521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:29.578521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.414426+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":221,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.458718+0000","last_change":"2024-02-05T08:09:31.480648+0000","last_active":"2024-02-05T08:10:50.458718+0000","last_peered":"2024-02-05T08:10:50.458718+0000","last_clean":"2024-02-05T08:10:50.458718+0000","last_became_active":"2024-02-05T08:06:39.554318+0000","last_became_peered":"2024-02-05T08:06:39.554318+0000","last_unstale":"2024-02-05T08:10:50.458718+0000","last_undegraded":"2024-02-05T08:10:50.458718+0000","last_fullsized":"2024-02-05T08:10:50.458718+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:31.480558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:31.480558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":220,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.360392+0000","last_change":"2024-02-05T08:09:34.530885+0000","last_active":"2024-02-05T08:10:50.360392+0000","last_peered":"2024-02-05T08:10:50.360392+0000","last_clean":"2024-02-05T08:10:50.360392+0000","last_became_active":"2024-02-05T08:06:39.552819+0000","last_became_peered":"2024-02-05T08:06:39.552819+0000","last_unstale":"2024-02-05T08:10:50.360392+0000","last_undegraded":"2024-02-05T08:10:50.360392+0000","last_fullsized":"2024-02-05T08:10:50.360392+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:09:34.530767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:34.530767+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.414390+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":194,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177636+0000","last_change":"2024-02-05T08:07:55.608340+0000","last_active":"2024-02-05T08:10:50.177636+0000","last_peered":"2024-02-05T08:10:50.177636+0000","last_clean":"2024-02-05T08:10:50.177636+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:10:50.177636+0000","last_undegraded":"2024-02-05T08:10:50.177636+0000","last_fullsized":"2024-02-05T08:10:50.177636+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:55.608083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:07:55.608083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.286313+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":207,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.360265+0000","last_change":"2024-02-05T08:09:21.583142+0000","last_active":"2024-02-05T08:10:50.360265+0000","last_peered":"2024-02-05T08:10:50.360265+0000","last_clean":"2024-02-05T08:10:50.360265+0000","last_became_active":"2024-02-05T08:04:45.567341+0000","last_became_peered":"2024-02-05T08:04:45.567341+0000","last_unstale":"2024-02-05T08:10:50.360265+0000","last_undegraded":"2024-02-05T08:10:50.360265+0000","last_fullsized":"2024-02-05T08:10:50.360265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:21.582910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.582910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.414316+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":220,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034050+0000","last_change":"2024-02-05T08:10:30.671004+0000","last_active":"2024-02-05T08:10:49.034050+0000","last_peered":"2024-02-05T08:10:49.034050+0000","last_clean":"2024-02-05T08:10:49.034050+0000","last_became_active":"2024-02-05T08:06:39.558993+0000","last_became_peered":"2024-02-05T08:06:39.558993+0000","last_unstale":"2024-02-05T08:10:49.034050+0000","last_undegraded":"2024-02-05T08:10:49.034050+0000","last_fullsized":"2024-02-05T08:10:49.034050+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:30.670916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.670916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:55.449333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":226,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177893+0000","last_change":"2024-02-05T08:10:34.198669+0000","last_active":"2024-02-05T08:10:50.177893+0000","last_peered":"2024-02-05T08:10:50.177893+0000","last_clean":"2024-02-05T08:10:50.177893+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:10:50.177893+0000","last_undegraded":"2024-02-05T08:10:50.177893+0000","last_fullsized":"2024-02-05T08:10:50.177893+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:10:34.198584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:34.198584+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:42.971407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":194,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943926+0000","last_change":"2024-02-05T08:10:17.009112+0000","last_active":"2024-02-05T08:10:48.943926+0000","last_peered":"2024-02-05T08:10:48.943926+0000","last_clean":"2024-02-05T08:10:48.943926+0000","last_became_active":"2024-02-05T08:04:47.588395+0000","last_became_peered":"2024-02-05T08:04:47.588395+0000","last_unstale":"2024-02-05T08:10:48.943926+0000","last_undegraded":"2024-02-05T08:10:48.943926+0000","last_fullsized":"2024-02-05T08:10:48.943926+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:17.008882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:17.008882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:17.659938+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":236,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.033890+0000","last_change":"2024-02-05T08:10:18.580307+0000","last_active":"2024-02-05T08:10:49.033890+0000","last_peered":"2024-02-05T08:10:49.033890+0000","last_clean":"2024-02-05T08:10:49.033890+0000","last_became_active":"2024-02-05T08:06:39.560631+0000","last_became_peered":"2024-02-05T08:06:39.560631+0000","last_unstale":"2024-02-05T08:10:49.033890+0000","last_undegraded":"2024-02-05T08:10:49.033890+0000","last_fullsized":"2024-02-05T08:10:49.033890+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:10:18.580209+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.580209+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:45.524064+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":212,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946018+0000","last_change":"2024-02-05T08:08:34.373002+0000","last_active":"2024-02-05T08:10:48.946018+0000","last_peered":"2024-02-05T08:10:48.946018+0000","last_clean":"2024-02-05T08:10:48.946018+0000","last_became_active":"2024-02-05T08:06:39.563051+0000","last_became_peered":"2024-02-05T08:06:39.563051+0000","last_unstale":"2024-02-05T08:10:48.946018+0000","last_undegraded":"2024-02-05T08:10:48.946018+0000","last_fullsized":"2024-02-05T08:10:48.946018+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:34.372891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:08:34.372891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.443444+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"85'38","reported_seq":344,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.947054+0000","last_change":"2024-02-05T08:09:35.108648+0000","last_active":"2024-02-05T08:10:48.947054+0000","last_peered":"2024-02-05T08:10:48.947054+0000","last_clean":"2024-02-05T08:10:48.947054+0000","last_became_active":"2024-02-05T08:04:43.554536+0000","last_became_peered":"2024-02-05T08:04:43.554536+0000","last_unstale":"2024-02-05T08:10:48.947054+0000","last_undegraded":"2024-02-05T08:10:48.947054+0000","last_fullsized":"2024-02-05T08:10:48.947054+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:09:35.108489+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:09:35.108489+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:56.622426+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":198,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177888+0000","last_change":"2024-02-05T08:10:19.265168+0000","last_active":"2024-02-05T08:10:50.177888+0000","last_peered":"2024-02-05T08:10:50.177888+0000","last_clean":"2024-02-05T08:10:50.177888+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:10:50.177888+0000","last_undegraded":"2024-02-05T08:10:50.177888+0000","last_fullsized":"2024-02-05T08:10:50.177888+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:19.265057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:19.265057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:45.883831+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":201,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.458827+0000","last_change":"2024-02-05T08:08:39.475407+0000","last_active":"2024-02-05T08:10:50.458827+0000","last_peered":"2024-02-05T08:10:50.458827+0000","last_clean":"2024-02-05T08:10:50.458827+0000","last_became_active":"2024-02-05T08:06:39.555147+0000","last_became_peered":"2024-02-05T08:06:39.555147+0000","last_unstale":"2024-02-05T08:10:50.458827+0000","last_undegraded":"2024-02-05T08:10:50.458827+0000","last_fullsized":"2024-02-05T08:10:50.458827+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:39.475316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.475316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":225,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.945785+0000","last_change":"2024-02-05T08:08:41.006239+0000","last_active":"2024-02-05T08:10:48.945785+0000","last_peered":"2024-02-05T08:10:48.945785+0000","last_clean":"2024-02-05T08:10:48.945785+0000","last_became_active":"2024-02-05T08:06:39.560285+0000","last_became_peered":"2024-02-05T08:06:39.560285+0000","last_unstale":"2024-02-05T08:10:48.945785+0000","last_undegraded":"2024-02-05T08:10:48.945785+0000","last_fullsized":"2024-02-05T08:10:48.945785+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:08:41.005907+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:08:41.005907+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.486995+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":232,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.360121+0000","last_change":"2024-02-05T08:10:34.494353+0000","last_active":"2024-02-05T08:10:50.360121+0000","last_peered":"2024-02-05T08:10:50.360121+0000","last_clean":"2024-02-05T08:10:50.360121+0000","last_became_active":"2024-02-05T08:04:40.529073+0000","last_became_peered":"2024-02-05T08:04:40.529073+0000","last_unstale":"2024-02-05T08:10:50.360121+0000","last_undegraded":"2024-02-05T08:10:50.360121+0000","last_fullsized":"2024-02-05T08:10:50.360121+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:34.494252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:10:34.494252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:01.739051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":215,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.360682+0000","last_change":"2024-02-05T08:10:24.681839+0000","last_active":"2024-02-05T08:10:50.360682+0000","last_peered":"2024-02-05T08:10:50.360682+0000","last_clean":"2024-02-05T08:10:50.360682+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:10:50.360682+0000","last_undegraded":"2024-02-05T08:10:50.360682+0000","last_fullsized":"2024-02-05T08:10:50.360682+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:10:24.681753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:24.681753+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:25.001232+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":207,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.360560+0000","last_change":"2024-02-05T08:09:25.547689+0000","last_active":"2024-02-05T08:10:50.360560+0000","last_peered":"2024-02-05T08:10:50.360560+0000","last_clean":"2024-02-05T08:10:50.360560+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:10:50.360560+0000","last_undegraded":"2024-02-05T08:10:50.360560+0000","last_fullsized":"2024-02-05T08:10:50.360560+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.547581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.547581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:55.227516+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":194,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034468+0000","last_change":"2024-02-05T08:08:34.051757+0000","last_active":"2024-02-05T08:10:49.034468+0000","last_peered":"2024-02-05T08:10:49.034468+0000","last_clean":"2024-02-05T08:10:49.034468+0000","last_became_active":"2024-02-05T08:04:47.586358+0000","last_became_peered":"2024-02-05T08:04:47.586358+0000","last_unstale":"2024-02-05T08:10:49.034468+0000","last_undegraded":"2024-02-05T08:10:49.034468+0000","last_fullsized":"2024-02-05T08:10:49.034468+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:34.051658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:34.051658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":480,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.457924+0000","last_change":"2024-02-05T08:08:49.423194+0000","last_active":"2024-02-05T08:10:50.457924+0000","last_peered":"2024-02-05T08:10:50.457924+0000","last_clean":"2024-02-05T08:10:50.457924+0000","last_became_active":"2024-02-05T08:06:42.566771+0000","last_became_peered":"2024-02-05T08:06:42.566771+0000","last_unstale":"2024-02-05T08:10:50.457924+0000","last_undegraded":"2024-02-05T08:10:50.457924+0000","last_fullsized":"2024-02-05T08:10:50.457924+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:08:49.423065+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:08:49.423065+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":212,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943131+0000","last_change":"2024-02-05T08:08:39.073558+0000","last_active":"2024-02-05T08:10:48.943131+0000","last_peered":"2024-02-05T08:10:48.943131+0000","last_clean":"2024-02-05T08:10:48.943131+0000","last_became_active":"2024-02-05T08:06:39.553704+0000","last_became_peered":"2024-02-05T08:06:39.553704+0000","last_unstale":"2024-02-05T08:10:48.943131+0000","last_undegraded":"2024-02-05T08:10:48.943131+0000","last_fullsized":"2024-02-05T08:10:48.943131+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.071354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.941143+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":258,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:51.662138+0000","last_change":"2024-02-05T08:10:19.453291+0000","last_active":"2024-02-05T08:10:51.662138+0000","last_peered":"2024-02-05T08:10:51.662138+0000","last_clean":"2024-02-05T08:10:51.662138+0000","last_became_active":"2024-02-05T08:06:39.563373+0000","last_became_peered":"2024-02-05T08:06:39.563373+0000","last_unstale":"2024-02-05T08:10:51.662138+0000","last_undegraded":"2024-02-05T08:10:51.662138+0000","last_fullsized":"2024-02-05T08:10:51.662138+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-02-05T08:10:19.453162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:19.453162+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:39.266463+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":199,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.458609+0000","last_change":"2024-02-05T08:08:38.472753+0000","last_active":"2024-02-05T08:10:50.458609+0000","last_peered":"2024-02-05T08:10:50.458609+0000","last_clean":"2024-02-05T08:10:50.458609+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:10:50.458609+0000","last_undegraded":"2024-02-05T08:10:50.458609+0000","last_fullsized":"2024-02-05T08:10:50.458609+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:38.472647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:38.472647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":203,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946879+0000","last_change":"2024-02-05T08:10:35.415472+0000","last_active":"2024-02-05T08:10:48.946879+0000","last_peered":"2024-02-05T08:10:48.946879+0000","last_clean":"2024-02-05T08:10:48.946879+0000","last_became_active":"2024-02-05T08:04:47.586774+0000","last_became_peered":"2024-02-05T08:04:47.586774+0000","last_unstale":"2024-02-05T08:10:48.946879+0000","last_undegraded":"2024-02-05T08:10:48.946879+0000","last_fullsized":"2024-02-05T08:10:48.946879+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:35.415402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:35.415402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:50.785267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":206,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177061+0000","last_change":"2024-02-05T08:08:38.705016+0000","last_active":"2024-02-05T08:10:50.177061+0000","last_peered":"2024-02-05T08:10:50.177061+0000","last_clean":"2024-02-05T08:10:50.177061+0000","last_became_active":"2024-02-05T08:04:41.537872+0000","last_became_peered":"2024-02-05T08:04:41.537872+0000","last_unstale":"2024-02-05T08:10:50.177061+0000","last_undegraded":"2024-02-05T08:10:50.177061+0000","last_fullsized":"2024-02-05T08:10:50.177061+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:38.704911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:38.704911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.286461+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":593,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:51.238789+0000","last_change":"2024-02-05T08:09:26.492000+0000","last_active":"2024-02-05T08:10:51.238789+0000","last_peered":"2024-02-05T08:10:51.238789+0000","last_clean":"2024-02-05T08:10:51.238789+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:10:51.238789+0000","last_undegraded":"2024-02-05T08:10:51.238789+0000","last_fullsized":"2024-02-05T08:10:51.238789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:09:26.491884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:09:26.491884+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":365,"num_read_kb":360,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":220,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177895+0000","last_change":"2024-02-05T08:09:19.836523+0000","last_active":"2024-02-05T08:10:50.177895+0000","last_peered":"2024-02-05T08:10:50.177895+0000","last_clean":"2024-02-05T08:10:50.177895+0000","last_became_active":"2024-02-05T08:04:43.549687+0000","last_became_peered":"2024-02-05T08:04:43.549687+0000","last_unstale":"2024-02-05T08:10:50.177895+0000","last_undegraded":"2024-02-05T08:10:50.177895+0000","last_fullsized":"2024-02-05T08:10:50.177895+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:09:19.836401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.836401+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.286480+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":198,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034711+0000","last_change":"2024-02-05T08:08:29.956921+0000","last_active":"2024-02-05T08:10:49.034711+0000","last_peered":"2024-02-05T08:10:49.034711+0000","last_clean":"2024-02-05T08:10:49.034711+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:10:49.034711+0000","last_undegraded":"2024-02-05T08:10:49.034711+0000","last_fullsized":"2024-02-05T08:10:49.034711+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:29.956845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:29.956845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":209,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946399+0000","last_change":"2024-02-05T08:10:18.609876+0000","last_active":"2024-02-05T08:10:48.946399+0000","last_peered":"2024-02-05T08:10:48.946399+0000","last_clean":"2024-02-05T08:10:48.946399+0000","last_became_active":"2024-02-05T08:06:39.559847+0000","last_became_peered":"2024-02-05T08:06:39.559847+0000","last_unstale":"2024-02-05T08:10:48.946399+0000","last_undegraded":"2024-02-05T08:10:48.946399+0000","last_fullsized":"2024-02-05T08:10:48.946399+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.609767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.609767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:43.852340+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":215,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946977+0000","last_change":"2024-02-05T08:08:44.964475+0000","last_active":"2024-02-05T08:10:48.946977+0000","last_peered":"2024-02-05T08:10:48.946977+0000","last_clean":"2024-02-05T08:10:48.946977+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:10:48.946977+0000","last_undegraded":"2024-02-05T08:10:48.946977+0000","last_fullsized":"2024-02-05T08:10:48.946977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:44.964357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.964357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.486960+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":246,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177870+0000","last_change":"2024-02-05T08:10:16.309686+0000","last_active":"2024-02-05T08:10:50.177870+0000","last_peered":"2024-02-05T08:10:50.177870+0000","last_clean":"2024-02-05T08:10:50.177870+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:10:50.177870+0000","last_undegraded":"2024-02-05T08:10:50.177870+0000","last_fullsized":"2024-02-05T08:10:50.177870+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:10:16.309551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:16.309551+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:43.442839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":138,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177693+0000","last_change":"2024-02-05T08:08:31.623904+0000","last_active":"2024-02-05T08:10:50.177693+0000","last_peered":"2024-02-05T08:10:50.177693+0000","last_clean":"2024-02-05T08:10:50.177693+0000","last_became_active":"2024-02-05T08:06:39.558100+0000","last_became_peered":"2024-02-05T08:06:39.558100+0000","last_unstale":"2024-02-05T08:10:50.177693+0000","last_undegraded":"2024-02-05T08:10:50.177693+0000","last_fullsized":"2024-02-05T08:10:50.177693+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:31.623748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:08:31.623748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:56.293698+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":203,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177395+0000","last_change":"2024-02-05T08:10:28.229516+0000","last_active":"2024-02-05T08:10:50.177395+0000","last_peered":"2024-02-05T08:10:50.177395+0000","last_clean":"2024-02-05T08:10:50.177395+0000","last_became_active":"2024-02-05T08:04:47.583866+0000","last_became_peered":"2024-02-05T08:04:47.583866+0000","last_unstale":"2024-02-05T08:10:50.177395+0000","last_undegraded":"2024-02-05T08:10:50.177395+0000","last_fullsized":"2024-02-05T08:10:50.177395+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:28.229441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:28.229441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:33.507208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":214,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034409+0000","last_change":"2024-02-05T08:09:40.402414+0000","last_active":"2024-02-05T08:10:49.034409+0000","last_peered":"2024-02-05T08:10:49.034409+0000","last_clean":"2024-02-05T08:10:49.034409+0000","last_became_active":"2024-02-05T08:04:41.537818+0000","last_became_peered":"2024-02-05T08:04:41.537818+0000","last_unstale":"2024-02-05T08:10:49.034409+0000","last_undegraded":"2024-02-05T08:10:49.034409+0000","last_fullsized":"2024-02-05T08:10:49.034409+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:40.402311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:40.402311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.704969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":226,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946128+0000","last_change":"2024-02-05T08:10:21.554686+0000","last_active":"2024-02-05T08:10:48.946128+0000","last_peered":"2024-02-05T08:10:48.946128+0000","last_clean":"2024-02-05T08:10:48.946128+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:10:48.946128+0000","last_undegraded":"2024-02-05T08:10:48.946128+0000","last_fullsized":"2024-02-05T08:10:48.946128+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:10:21.554300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:21.554300+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:39.632000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":291,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.178004+0000","last_change":"2024-02-05T08:07:48.538033+0000","last_active":"2024-02-05T08:10:50.178004+0000","last_peered":"2024-02-05T08:10:50.178004+0000","last_clean":"2024-02-05T08:10:50.178004+0000","last_became_active":"2024-02-05T08:06:39.554860+0000","last_became_peered":"2024-02-05T08:06:39.554860+0000","last_unstale":"2024-02-05T08:10:50.178004+0000","last_undegraded":"2024-02-05T08:10:50.178004+0000","last_fullsized":"2024-02-05T08:10:50.178004+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:07:48.537913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:48.537913+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:56.293669+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":201,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.458382+0000","last_change":"2024-02-05T08:08:43.475720+0000","last_active":"2024-02-05T08:10:50.458382+0000","last_peered":"2024-02-05T08:10:50.458382+0000","last_clean":"2024-02-05T08:10:50.458382+0000","last_became_active":"2024-02-05T08:06:39.554165+0000","last_became_peered":"2024-02-05T08:06:39.554165+0000","last_unstale":"2024-02-05T08:10:50.458382+0000","last_undegraded":"2024-02-05T08:10:50.458382+0000","last_fullsized":"2024-02-05T08:10:50.458382+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:43.475636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:43.475636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":215,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.177111+0000","last_change":"2024-02-05T08:10:30.240914+0000","last_active":"2024-02-05T08:10:50.177111+0000","last_peered":"2024-02-05T08:10:50.177111+0000","last_clean":"2024-02-05T08:10:50.177111+0000","last_became_active":"2024-02-05T08:04:41.534433+0000","last_became_peered":"2024-02-05T08:04:41.534433+0000","last_unstale":"2024-02-05T08:10:50.177111+0000","last_undegraded":"2024-02-05T08:10:50.177111+0000","last_fullsized":"2024-02-05T08:10:50.177111+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:10:30.240833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.240833+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.543762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":245,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.945951+0000","last_change":"2024-02-05T08:10:16.458012+0000","last_active":"2024-02-05T08:10:48.945951+0000","last_peered":"2024-02-05T08:10:48.945951+0000","last_clean":"2024-02-05T08:10:48.945951+0000","last_became_active":"2024-02-05T08:06:39.566434+0000","last_became_peered":"2024-02-05T08:06:39.566434+0000","last_unstale":"2024-02-05T08:10:48.945951+0000","last_undegraded":"2024-02-05T08:10:48.945951+0000","last_fullsized":"2024-02-05T08:10:48.945951+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-02-05T08:10:16.457895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:16.457895+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:40.130069+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":282,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943255+0000","last_change":"2024-02-05T08:07:47.765640+0000","last_active":"2024-02-05T08:10:48.943255+0000","last_peered":"2024-02-05T08:10:48.943255+0000","last_clean":"2024-02-05T08:10:48.943255+0000","last_became_active":"2024-02-05T08:04:45.562597+0000","last_became_peered":"2024-02-05T08:04:45.562597+0000","last_unstale":"2024-02-05T08:10:48.943255+0000","last_undegraded":"2024-02-05T08:10:48.943255+0000","last_fullsized":"2024-02-05T08:10:48.943255+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:07:47.765523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:47.765523+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:55.898150+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":202,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.945890+0000","last_change":"2024-02-05T08:10:15.488463+0000","last_active":"2024-02-05T08:10:48.945890+0000","last_peered":"2024-02-05T08:10:48.945890+0000","last_clean":"2024-02-05T08:10:48.945890+0000","last_became_active":"2024-02-05T08:04:47.582776+0000","last_became_peered":"2024-02-05T08:04:47.582776+0000","last_unstale":"2024-02-05T08:10:48.945890+0000","last_undegraded":"2024-02-05T08:10:48.945890+0000","last_fullsized":"2024-02-05T08:10:48.945890+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:15.488380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:15.488380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:31.489301+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":220,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943751+0000","last_change":"2024-02-05T08:09:36.111657+0000","last_active":"2024-02-05T08:10:48.943751+0000","last_peered":"2024-02-05T08:10:48.943751+0000","last_clean":"2024-02-05T08:10:48.943751+0000","last_became_active":"2024-02-05T08:04:41.540818+0000","last_became_peered":"2024-02-05T08:04:41.540818+0000","last_unstale":"2024-02-05T08:10:48.943751+0000","last_undegraded":"2024-02-05T08:10:48.943751+0000","last_fullsized":"2024-02-05T08:10:48.943751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:09:36.111537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.111537+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.119806+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":243,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943110+0000","last_change":"2024-02-05T08:09:33.222821+0000","last_active":"2024-02-05T08:10:48.943110+0000","last_peered":"2024-02-05T08:10:48.943110+0000","last_clean":"2024-02-05T08:10:48.943110+0000","last_became_active":"2024-02-05T08:04:43.607546+0000","last_became_peered":"2024-02-05T08:04:43.607546+0000","last_unstale":"2024-02-05T08:10:48.943110+0000","last_undegraded":"2024-02-05T08:10:48.943110+0000","last_fullsized":"2024-02-05T08:10:48.943110+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-05T08:09:33.222532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:33.222532+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:53.393561+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":198,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.945757+0000","last_change":"2024-02-05T08:09:20.449787+0000","last_active":"2024-02-05T08:10:48.945757+0000","last_peered":"2024-02-05T08:10:48.945757+0000","last_clean":"2024-02-05T08:10:48.945757+0000","last_became_active":"2024-02-05T08:04:45.561639+0000","last_became_peered":"2024-02-05T08:04:45.561639+0000","last_unstale":"2024-02-05T08:10:48.945757+0000","last_undegraded":"2024-02-05T08:10:48.945757+0000","last_fullsized":"2024-02-05T08:10:48.945757+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:20.449711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:20.449711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.443588+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":202,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034255+0000","last_change":"2024-02-05T08:08:43.142178+0000","last_active":"2024-02-05T08:10:49.034255+0000","last_peered":"2024-02-05T08:10:49.034255+0000","last_clean":"2024-02-05T08:10:49.034255+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:10:49.034255+0000","last_undegraded":"2024-02-05T08:10:49.034255+0000","last_fullsized":"2024-02-05T08:10:49.034255+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:43.142051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:43.142051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":281,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.947007+0000","last_change":"2024-02-05T08:09:17.391826+0000","last_active":"2024-02-05T08:10:48.947007+0000","last_peered":"2024-02-05T08:10:48.947007+0000","last_clean":"2024-02-05T08:10:48.947007+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:10:48.947007+0000","last_undegraded":"2024-02-05T08:10:48.947007+0000","last_fullsized":"2024-02-05T08:10:48.947007+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:09:17.391673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:17.391673+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.443607+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":209,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943011+0000","last_change":"2024-02-05T08:10:42.111871+0000","last_active":"2024-02-05T08:10:48.943011+0000","last_peered":"2024-02-05T08:10:48.943011+0000","last_clean":"2024-02-05T08:10:48.943011+0000","last_became_active":"2024-02-05T08:04:41.541327+0000","last_became_peered":"2024-02-05T08:04:41.541327+0000","last_unstale":"2024-02-05T08:10:48.943011+0000","last_undegraded":"2024-02-05T08:10:48.943011+0000","last_fullsized":"2024-02-05T08:10:48.943011+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:07:36.642926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:07:36.642926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.939154+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":236,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946906+0000","last_change":"2024-02-05T08:09:17.191152+0000","last_active":"2024-02-05T08:10:48.946906+0000","last_peered":"2024-02-05T08:10:48.946906+0000","last_clean":"2024-02-05T08:10:48.946906+0000","last_became_active":"2024-02-05T08:04:43.555028+0000","last_became_peered":"2024-02-05T08:04:43.555028+0000","last_unstale":"2024-02-05T08:10:48.946906+0000","last_undegraded":"2024-02-05T08:10:48.946906+0000","last_fullsized":"2024-02-05T08:10:48.946906+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:09:17.190290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:17.190290+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.487030+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":200,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943553+0000","last_change":"2024-02-05T08:09:32.213156+0000","last_active":"2024-02-05T08:10:48.943553+0000","last_peered":"2024-02-05T08:10:48.943553+0000","last_clean":"2024-02-05T08:10:48.943553+0000","last_became_active":"2024-02-05T08:06:39.558047+0000","last_became_peered":"2024-02-05T08:06:39.558047+0000","last_unstale":"2024-02-05T08:10:48.943553+0000","last_undegraded":"2024-02-05T08:10:48.943553+0000","last_fullsized":"2024-02-05T08:10:48.943553+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:32.213044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:09:32.213044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:54.206176+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":212,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:49.034653+0000","last_change":"2024-02-05T08:10:20.643993+0000","last_active":"2024-02-05T08:10:49.034653+0000","last_peered":"2024-02-05T08:10:49.034653+0000","last_clean":"2024-02-05T08:10:49.034653+0000","last_became_active":"2024-02-05T08:06:39.551933+0000","last_became_peered":"2024-02-05T08:06:39.551933+0000","last_unstale":"2024-02-05T08:10:49.034653+0000","last_undegraded":"2024-02-05T08:10:49.034653+0000","last_fullsized":"2024-02-05T08:10:49.034653+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:20.643918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:20.643918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:27.975950+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":214,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.945812+0000","last_change":"2024-02-05T08:09:22.454321+0000","last_active":"2024-02-05T08:10:48.945812+0000","last_peered":"2024-02-05T08:10:48.945812+0000","last_clean":"2024-02-05T08:10:48.945812+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:10:48.945812+0000","last_undegraded":"2024-02-05T08:10:48.945812+0000","last_fullsized":"2024-02-05T08:10:48.945812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:22.454247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.454247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.443625+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":239,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946793+0000","last_change":"2024-02-05T08:08:32.390911+0000","last_active":"2024-02-05T08:10:48.946793+0000","last_peered":"2024-02-05T08:10:48.946793+0000","last_clean":"2024-02-05T08:10:48.946793+0000","last_became_active":"2024-02-05T08:06:39.556433+0000","last_became_peered":"2024-02-05T08:06:39.556433+0000","last_unstale":"2024-02-05T08:10:48.946793+0000","last_undegraded":"2024-02-05T08:10:48.946793+0000","last_fullsized":"2024-02-05T08:10:48.946793+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:08:32.390785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:32.390785+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.443466+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":204,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.360798+0000","last_change":"2024-02-05T08:08:45.814521+0000","last_active":"2024-02-05T08:10:50.360798+0000","last_peered":"2024-02-05T08:10:50.360798+0000","last_clean":"2024-02-05T08:10:50.360798+0000","last_became_active":"2024-02-05T08:04:47.585962+0000","last_became_peered":"2024-02-05T08:04:47.585962+0000","last_unstale":"2024-02-05T08:10:50.360798+0000","last_undegraded":"2024-02-05T08:10:50.360798+0000","last_fullsized":"2024-02-05T08:10:50.360798+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:08:45.814384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:45.814384+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.414278+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":199,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.947104+0000","last_change":"2024-02-05T08:08:29.100467+0000","last_active":"2024-02-05T08:10:48.947104+0000","last_peered":"2024-02-05T08:10:48.947104+0000","last_clean":"2024-02-05T08:10:48.947104+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:10:48.947104+0000","last_undegraded":"2024-02-05T08:10:48.947104+0000","last_fullsized":"2024-02-05T08:10:48.947104+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:29.100364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:29.100364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:51.486867+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":145,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.945976+0000","last_change":"2024-02-05T08:10:25.446805+0000","last_active":"2024-02-05T08:10:48.945976+0000","last_peered":"2024-02-05T08:10:48.945976+0000","last_clean":"2024-02-05T08:10:48.945976+0000","last_became_active":"2024-02-05T08:06:39.559090+0000","last_became_peered":"2024-02-05T08:06:39.559090+0000","last_unstale":"2024-02-05T08:10:48.945976+0000","last_undegraded":"2024-02-05T08:10:48.945976+0000","last_fullsized":"2024-02-05T08:10:48.945976+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:25.446724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:10:25.446724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:50.729956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":230,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946590+0000","last_change":"2024-02-05T08:09:21.434983+0000","last_active":"2024-02-05T08:10:48.946590+0000","last_peered":"2024-02-05T08:10:48.946590+0000","last_clean":"2024-02-05T08:10:48.946590+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:10:48.946590+0000","last_undegraded":"2024-02-05T08:10:48.946590+0000","last_fullsized":"2024-02-05T08:10:48.946590+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-02-05T08:09:21.434891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.434891+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-02-05T08:10:53.958581+0000","last_change":"2024-02-05T08:10:53.958581+0000","last_active":"2024-02-05T08:10:53.958581+0000","last_peered":"2024-02-05T08:10:53.958581+0000","last_clean":"2024-02-05T08:10:53.958581+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-02-05T08:10:53.958581+0000","last_undegraded":"2024-02-05T08:10:53.958581+0000","last_fullsized":"2024-02-05T08:10:53.958581+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":194,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946998+0000","last_change":"2024-02-05T08:08:49.450529+0000","last_active":"2024-02-05T08:10:48.946998+0000","last_peered":"2024-02-05T08:10:48.946998+0000","last_clean":"2024-02-05T08:10:48.946998+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:10:48.946998+0000","last_undegraded":"2024-02-05T08:10:48.946998+0000","last_fullsized":"2024-02-05T08:10:48.946998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:49.450400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:49.450400+0000","last_clean_scrub_stamp":"2024-02-05T08:08:49.450400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.443508+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":299,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:50.360567+0000","last_change":"2024-02-05T08:10:31.516653+0000","last_active":"2024-02-05T08:10:50.360567+0000","last_peered":"2024-02-05T08:10:50.360567+0000","last_clean":"2024-02-05T08:10:50.360567+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:10:50.360567+0000","last_undegraded":"2024-02-05T08:10:50.360567+0000","last_fullsized":"2024-02-05T08:10:50.360567+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:10:31.516571+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:10:31.516571+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:01.515985+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":206,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.943140+0000","last_change":"2024-02-05T08:09:22.380135+0000","last_active":"2024-02-05T08:10:48.943140+0000","last_peered":"2024-02-05T08:10:48.943140+0000","last_clean":"2024-02-05T08:10:48.943140+0000","last_became_active":"2024-02-05T08:04:41.533838+0000","last_became_peered":"2024-02-05T08:04:41.533838+0000","last_unstale":"2024-02-05T08:10:48.943140+0000","last_undegraded":"2024-02-05T08:10:48.943140+0000","last_fullsized":"2024-02-05T08:10:48.943140+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:22.380016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.380016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:49.941251+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":235,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-02-05T08:10:48.946317+0000","last_change":"2024-02-05T08:08:35.372047+0000","last_active":"2024-02-05T08:10:48.946317+0000","last_peered":"2024-02-05T08:10:48.946317+0000","last_clean":"2024-02-05T08:10:48.946317+0000","last_became_active":"2024-02-05T08:06:39.555561+0000","last_became_peered":"2024-02-05T08:06:39.555561+0000","last_unstale":"2024-02-05T08:10:48.946317+0000","last_undegraded":"2024-02-05T08:10:48.946317+0000","last_fullsized":"2024-02-05T08:10:48.946317+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:08:35.371853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:35.371853+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:10:52.443490+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":365,"num_read_kb":360,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":1474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":433,"ondisk_log_size":433,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037839,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104784,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93640368,"statfs":{"total":95995035648,"available":95887736832,"internally_reserved":0,"allocated":1310720,"data_stored":1015811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.581,"15min":0.89700000000000002},"min":{"1min":0.254,"5min":0.218,"15min":0.217},"max":{"1min":3.8420000000000001,"5min":60.127000000000002,"15min":60.127000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.5920000000000001,"15min":0.88},"min":{"1min":0.28399999999999997,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.0860000000000003,"5min":58.283000000000001,"15min":58.283000000000001},"last":0.59799999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.411,"15min":0.84099999999999997},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1949999999999998,"5min":20.774000000000001,"15min":20.774000000000001},"last":0.443},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.095,"15min":0.73299999999999998},"min":{"1min":0.254,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":2.8149999999999999,"5min":20.715,"15min":20.715},"last":0.57199999999999995}]},{"osd":2,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":1.0349999999999999,"15min":0.73299999999999998},"min":{"1min":0.26900000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.2480000000000002,"5min":7.6779999999999999,"15min":7.6779999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.125,"5min":1.2170000000000001,"15min":0.76500000000000001},"min":{"1min":0.253,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.399,"5min":8.9100000000000001,"15min":8.9100000000000001},"last":0.48899999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.9910000000000001,"15min":1.0429999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.6630000000000003,"5min":58.515000000000001,"15min":58.515000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":1.607,"5min":1.923,"15min":1.0009999999999999},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":6.202,"5min":58.225999999999999,"15min":58.225999999999999},"last":0.39100000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.42299999999999999},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.58299999999999996,"5min":0.71099999999999997,"15min":0.91400000000000003},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.40999999999999998,"15min":0.42599999999999999},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.55900000000000005,"5min":0.70299999999999996,"15min":0.72999999999999998},"last":0.55700000000000005}]},{"osd":5,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.43099999999999999,"15min":0.438},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69999999999999996,"5min":1.2,"15min":1.2},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.44,"5min":0.45900000000000002,"15min":0.48099999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67900000000000005,"5min":0.81999999999999995,"15min":0.998},"last":0.64000000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47499999999999998,"15min":0.495},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67700000000000005,"5min":1.256,"15min":1.256},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47099999999999997,"15min":0.47599999999999998},"min":{"1min":0.32400000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.71799999999999997,"5min":0.72899999999999998,"15min":0.79300000000000004},"last":0.628}]}]},{"osd":6,"up_from":0,"seq":219043332181,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574958,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":185320,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93559832,"statfs":{"total":95995035648,"available":95805267968,"internally_reserved":0,"allocated":1335296,"data_stored":1015519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:10:03 2024","interfaces":[{"interface":"back","average":{"1min":2.7360000000000002,"5min":2.3450000000000002,"15min":0.98899999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.14699999999999999},"max":{"1min":7.7300000000000004,"5min":86.105000000000004,"15min":86.105000000000004},"last":0.375},{"interface":"front","average":{"1min":2.5880000000000001,"5min":2.6930000000000001,"15min":1.1140000000000001},"min":{"1min":0.28999999999999998,"5min":0.22900000000000001,"15min":0.16},"max":{"1min":13.800000000000001,"5min":85.665000000000006,"15min":85.665000000000006},"last":0.32700000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:10:34 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":2.0390000000000001,"15min":0.93000000000000005},"min":{"1min":0.20999999999999999,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":2.351,"5min":49.030999999999999,"15min":49.030999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.3320000000000001,"5min":2.3039999999999998,"15min":1.04},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":6.5590000000000002,"5min":48.984000000000002,"15min":48.984000000000002},"last":0.496}]},{"osd":3,"last update":"Mon Feb 5 08:09:59 2024","interfaces":[{"interface":"back","average":{"1min":2.3730000000000002,"5min":2.2349999999999999,"15min":1.04},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":12.468999999999999,"5min":49.067999999999998,"15min":49.067999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":2.0960000000000001,"5min":2.3130000000000002,"15min":1.0780000000000001},"min":{"1min":0.38,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.3940000000000001,"5min":48.734999999999999,"15min":48.734999999999999},"last":0.40699999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:10:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":1.093,"15min":0.755},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.8129999999999999,"5min":9.4269999999999996,"15min":9.4269999999999996},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.244,"5min":1.1819999999999999,"15min":0.77800000000000002},"min":{"1min":0.30399999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.9300000000000002,"5min":6.3890000000000002,"15min":6.3890000000000002},"last":0.72599999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":1.1499999999999999,"15min":0.752},"min":{"1min":0.308,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.1240000000000001,"5min":9.4559999999999995,"15min":9.4559999999999995},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.2010000000000001,"15min":0.77500000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.9660000000000002,"5min":13.776,"15min":13.776},"last":0.745}]},{"osd":6,"last update":"Mon Feb 5 08:10:18 2024","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.2,"15min":0.79500000000000004},"min":{"1min":0.36799999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.6319999999999997,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.369,"5min":1.1930000000000001,"15min":0.82099999999999995},"min":{"1min":0.41899999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":7.7409999999999997,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":0.68700000000000006}]},{"osd":7,"last update":"Mon Feb 5 08:10:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.3620000000000001,"15min":0.86799999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.0009999999999999,"5min":13.749000000000001,"15min":13.749000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.288,"15min":0.83499999999999996},"min":{"1min":0.39100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.4860000000000002,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.59499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738483,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163112,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93582040,"statfs":{"total":95995035648,"available":95828008960,"internally_reserved":0,"allocated":1335296,"data_stored":1016345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:10:15 2024","interfaces":[{"interface":"back","average":{"1min":2.9159999999999999,"5min":2.649,"15min":1.1080000000000001},"min":{"1min":0.21199999999999999,"5min":0.19,"15min":0.152},"max":{"1min":10.186,"5min":66.772999999999996,"15min":66.772999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":2.2599999999999998,"5min":2.0920000000000001,"15min":0.92100000000000004},"min":{"1min":0.33700000000000002,"5min":0.249,"15min":0.17199999999999999},"max":{"1min":6.2060000000000004,"5min":46.198999999999998,"15min":46.198999999999998},"last":0.34300000000000003}]},{"osd":2,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":2.1720000000000002,"15min":0.97399999999999998},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":6.2249999999999996,"5min":46.170999999999999,"15min":46.170999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":2.5870000000000002,"5min":2.5299999999999998,"15min":1.105},"min":{"1min":0.36099999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":11.07,"5min":46.543999999999997,"15min":46.543999999999997},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:10:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7869999999999999,"5min":2.419,"15min":1.113},"min":{"1min":0.42099999999999999,"5min":0.28399999999999997,"15min":0.21099999999999999},"max":{"1min":9.6829999999999998,"5min":67.888000000000005,"15min":67.888000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":2.7349999999999999,"5min":2.7229999999999999,"15min":1.22},"min":{"1min":0.41099999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":8.8979999999999997,"5min":66.891999999999996,"15min":66.891999999999996},"last":0.32700000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:10:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.1060000000000001,"15min":0.746},"min":{"1min":0.34599999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":3.5089999999999999,"5min":13.295,"15min":13.295},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.647,"5min":1.2290000000000001,"15min":0.81499999999999995},"min":{"1min":0.35399999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":9.5739999999999998,"5min":12.989000000000001,"15min":12.989000000000001},"last":0.70099999999999996}]},{"osd":5,"last update":"Mon Feb 5 08:10:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.3080000000000001,"15min":0.83999999999999997},"min":{"1min":0.29899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.375,"5min":13.073,"15min":13.073},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.998,"5min":1.2609999999999999,"15min":0.83299999999999996},"min":{"1min":0.42599999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.258,"5min":12.94,"15min":12.94},"last":0.67600000000000005}]},{"osd":6,"last update":"Mon Feb 5 08:10:15 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.292,"15min":0.83199999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":9.6039999999999992,"5min":12.967000000000001,"15min":12.967000000000001},"last":0.621},{"interface":"front","average":{"1min":1.472,"5min":1.3420000000000001,"15min":0.86199999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":6.335,"5min":13.108000000000001,"15min":13.108000000000001},"last":0.63200000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.1559999999999999,"15min":0.76200000000000001},"min":{"1min":0.38500000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.5459999999999998,"5min":13.263,"15min":13.263},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.151,"5min":1.3520000000000001,"15min":0.85699999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.2320000000000002,"5min":13.012,"15min":13.012},"last":0.64600000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411433,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148624,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93596528,"statfs":{"total":95995035648,"available":95842844672,"internally_reserved":0,"allocated":720896,"data_stored":424703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:10:40 2024","interfaces":[{"interface":"back","average":{"1min":2.589,"5min":2.1099999999999999,"15min":0.95199999999999996},"min":{"1min":0.25,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":8.0350000000000001,"5min":16.155999999999999,"15min":16.155999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":1.8089999999999999,"5min":2.105,"15min":0.95899999999999996},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.20699999999999999},"max":{"1min":3.9740000000000002,"5min":15.973000000000001,"15min":15.973000000000001},"last":0.371}]},{"osd":1,"last update":"Mon Feb 5 08:10:40 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":1.7969999999999999,"15min":0.83999999999999997},"min":{"1min":0.28599999999999998,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":6.4829999999999997,"5min":9.1340000000000003,"15min":9.1340000000000003},"last":0.25900000000000001},{"interface":"front","average":{"1min":2.1269999999999998,"5min":1.7649999999999999,"15min":0.84599999999999997},"min":{"1min":0.27100000000000002,"5min":0.193,"15min":0.193},"max":{"1min":7.9400000000000004,"5min":9.2769999999999992,"15min":9.2769999999999992},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Feb 5 08:09:59 2024","interfaces":[{"interface":"back","average":{"1min":5.7229999999999999,"5min":2.984,"15min":1.3069999999999999},"min":{"1min":0.314,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":68.108000000000004,"5min":68.108000000000004,"15min":68.108000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":5.7880000000000003,"5min":2.6589999999999998,"15min":1.1919999999999999},"min":{"1min":0.33500000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":67.945999999999998,"5min":67.945999999999998,"15min":67.945999999999998},"last":0.32700000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:10:23 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.3089999999999999,"15min":0.83699999999999997},"min":{"1min":0.34699999999999998,"5min":0.315,"15min":0.315},"max":{"1min":3.863,"5min":15.923999999999999,"15min":15.923999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.0529999999999999,"15min":0.73899999999999999},"min":{"1min":0.27200000000000002,"5min":0.252,"15min":0.252},"max":{"1min":3.5880000000000001,"5min":6.008,"15min":6.008},"last":0.48599999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.216,"15min":0.79800000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.6150000000000002,"5min":15.835000000000001,"15min":15.835000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.995,"5min":1.1160000000000001,"15min":0.755},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":3.2149999999999999,"5min":9.1809999999999992,"15min":9.1809999999999992},"last":0.56899999999999995}]},{"osd":6,"last update":"Mon Feb 5 08:10:20 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.359,"15min":0.878},"min":{"1min":0.32500000000000001,"5min":0.308,"15min":0.308},"max":{"1min":3.9449999999999998,"5min":15.859,"15min":15.859},"last":0.496},{"interface":"front","average":{"1min":1.284,"5min":1.129,"15min":0.80400000000000005},"min":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.6389999999999998,"5min":6.117,"15min":6.117},"last":0.55200000000000005}]},{"osd":7,"last update":"Mon Feb 5 08:10:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.337,"15min":0.85999999999999999},"min":{"1min":0.35699999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.258,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":1.343,"5min":1.4339999999999999,"15min":0.92600000000000005},"min":{"1min":0.46899999999999997,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":3.9140000000000001,"5min":15.951000000000001,"15min":15.951000000000001},"last":0.46100000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117092,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":212516,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93532636,"statfs":{"total":95995035648,"available":95777419264,"internally_reserved":0,"allocated":741376,"data_stored":425406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":216725961},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.1549999999999998,"5min":2.1379999999999999,"15min":0.98499999999999999},"min":{"1min":0.42099999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":6.4169999999999998,"5min":63.43,"15min":63.43},"last":0.20300000000000001},{"interface":"front","average":{"1min":2.5430000000000001,"5min":2.1829999999999998,"15min":1.028},"min":{"1min":0.34300000000000003,"5min":0.186,"15min":0.186},"max":{"1min":8.2650000000000006,"5min":63.048999999999999,"15min":63.048999999999999},"last":0.28199999999999997}]},{"osd":1,"last update":"Mon Feb 5 08:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.125,"5min":1.496,"15min":0.81999999999999995},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":6.085,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":0.23899999999999999},{"interface":"front","average":{"1min":3.7719999999999998,"5min":2.0539999999999998,"15min":0.995},"min":{"1min":0.34300000000000003,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":8.1760000000000002,"5min":12.087999999999999,"15min":12.087999999999999},"last":0.32700000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:09:54 2024","interfaces":[{"interface":"back","average":{"1min":2.3919999999999999,"5min":2.3559999999999999,"15min":1.1160000000000001},"min":{"1min":0.29999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":7.2800000000000002,"5min":62.902999999999999,"15min":62.902999999999999},"last":0.25700000000000001},{"interface":"front","average":{"1min":2.105,"5min":2.198,"15min":1.054},"min":{"1min":0.54100000000000004,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.3730000000000002,"5min":63.468000000000004,"15min":63.468000000000004},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:10:21 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.302,"15min":0.80800000000000005},"min":{"1min":0.40200000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.0760000000000001,"5min":7.7229999999999999,"15min":7.7229999999999999},"last":1.0609999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.399,"15min":0.86099999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.8039999999999998,"5min":8.7059999999999995,"15min":8.7059999999999995},"last":0.72699999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":1.294,"15min":0.82599999999999996},"min":{"1min":0.28000000000000003,"5min":0.255,"15min":0.255},"max":{"1min":1.792,"5min":7.6680000000000001,"15min":7.6680000000000001},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.81499999999999995,"5min":1.444,"15min":0.88700000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.5279999999999996,"5min":8.734,"15min":8.734},"last":0.86899999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:10:11 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.4339999999999999,"15min":0.876},"min":{"1min":0.372,"5min":0.372,"15min":0.36499999999999999},"max":{"1min":4.5,"5min":8.9429999999999996,"15min":8.9429999999999996},"last":0.80200000000000005},{"interface":"front","average":{"1min":2.2869999999999999,"5min":1.4450000000000001,"15min":0.86699999999999999},"min":{"1min":0.34300000000000003,"5min":0.308,"15min":0.308},"max":{"1min":6.9400000000000004,"5min":7.8639999999999999,"15min":7.8639999999999999},"last":0.69699999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.3440000000000001,"15min":0.84299999999999997},"min":{"1min":0.46999999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.1469999999999998,"5min":7.7000000000000002,"15min":7.7000000000000002},"last":1.0820000000000001},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.52,"15min":0.92800000000000005},"min":{"1min":0.39700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":4.4450000000000003,"5min":8.9900000000000002,"15min":8.9900000000000002},"last":0.93400000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953567,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100056,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93645096,"statfs":{"total":95995035648,"available":95892578304,"internally_reserved":0,"allocated":729088,"data_stored":424246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:10:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":2.2400000000000002,"15min":1.1140000000000001},"min":{"1min":0.27900000000000003,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":5.4409999999999998,"5min":73.078999999999994,"15min":73.078999999999994},"last":0.498},{"interface":"front","average":{"1min":1.25,"5min":2.0880000000000001,"15min":1.0609999999999999},"min":{"1min":0.21199999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":4.7709999999999999,"5min":73.412000000000006,"15min":73.412000000000006},"last":0.66600000000000004}]},{"osd":1,"last update":"Mon Feb 5 08:10:14 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":0.97699999999999998,"15min":0.68799999999999994},"min":{"1min":0.252,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":8.3409999999999993,"5min":8.3409999999999993,"15min":8.3409999999999993},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.74099999999999999,"15min":0.61199999999999999},"min":{"1min":0.25900000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":4.3120000000000003,"5min":4.3120000000000003,"15min":4.3120000000000003},"last":0.59899999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:10:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.8740000000000001,"15min":0.98699999999999999},"min":{"1min":0.26500000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.9640000000000004,"5min":96.849999999999994,"15min":96.849999999999994},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.94599999999999995,"5min":1.9490000000000001,"15min":1.0329999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.6309999999999998,"5min":96.143000000000001,"15min":96.143000000000001},"last":0.64200000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:10:14 2024","interfaces":[{"interface":"back","average":{"1min":2.8220000000000001,"5min":2.5489999999999999,"15min":1.2170000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":28.472999999999999,"5min":73.602999999999994,"15min":73.602999999999994},"last":0.61499999999999999},{"interface":"front","average":{"1min":3.2749999999999999,"5min":2.5680000000000001,"15min":1.222},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":28.861000000000001,"5min":74.817999999999998,"15min":74.817999999999998},"last":0.56599999999999995}]},{"osd":5,"last update":"Mon Feb 5 08:10:52 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.40200000000000002,"15min":0.38500000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.23999999999999999},"max":{"1min":0.70099999999999996,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.45200000000000001,"15min":0.41899999999999998},"min":{"1min":0.30599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.91000000000000003,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:10:14 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.46100000000000002,"15min":0.435},"min":{"1min":0.311,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.46899999999999997,"15min":0.433},"min":{"1min":0.34100000000000003,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.39200000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.95199999999999996,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.50800000000000001,"15min":0.505},"min":{"1min":0.33500000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.53300000000000003}]}]},{"osd":5,"up_from":42,"seq":180388626521,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108356,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636796,"statfs":{"total":95995035648,"available":95884079104,"internally_reserved":0,"allocated":708608,"data_stored":424173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:10:46 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":2.3919999999999999,"15min":1.1639999999999999},"min":{"1min":0.249,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":9.4659999999999993,"5min":143.06299999999999,"15min":143.06299999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.075,"5min":2.395,"15min":1.1539999999999999},"min":{"1min":0.23699999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.5330000000000004,"5min":143.09700000000001,"15min":143.09700000000001},"last":0.65100000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.77900000000000003,"15min":0.63900000000000001},"min":{"1min":0.252,"5min":0.215,"15min":0.215},"max":{"1min":2.71,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":0.495},{"interface":"front","average":{"1min":1.5549999999999999,"5min":0.96499999999999997,"15min":0.68200000000000005},"min":{"1min":0.29899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":7.0949999999999998,"5min":7.0949999999999998,"15min":7.0949999999999998},"last":0.56599999999999995}]},{"osd":2,"last update":"Mon Feb 5 08:10:46 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.069,"15min":0.73599999999999999},"min":{"1min":0.27600000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":10.733000000000001,"5min":10.733000000000001,"15min":10.733000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.034,"5min":1.109,"15min":0.747},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.8899999999999997,"5min":5.4089999999999998,"15min":5.4089999999999998},"last":0.52900000000000003}]},{"osd":3,"last update":"Mon Feb 5 08:10:46 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":2.5139999999999998,"15min":1.2110000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":4.5599999999999996,"5min":143.03399999999999,"15min":143.03399999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.8660000000000001,"5min":2.464,"15min":1.2150000000000001},"min":{"1min":0.32600000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.4509999999999996,"5min":143.46799999999999,"15min":143.46799999999999},"last":0.60999999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.44600000000000001,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.255,"15min":0.253},"max":{"1min":0.70999999999999996,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.432},"min":{"1min":0.29699999999999999,"5min":0.26900000000000002,"15min":0.26500000000000001},"max":{"1min":0.751,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.59299999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:10:09 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.45300000000000001,"15min":0.44500000000000001},"min":{"1min":0.32200000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.496,"15min":0.48699999999999999},"min":{"1min":0.40000000000000002,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":0.71099999999999997,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.65900000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.49299999999999999,"15min":0.47699999999999998},"min":{"1min":0.34399999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51600000000000001,"15min":0.49199999999999999},"min":{"1min":0.35399999999999998,"5min":0.33700000000000002,"15min":0.28199999999999997},"max":{"1min":0.78900000000000003,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:10:57.174 INFO:tasks.thrashosds.thrasher:Setting 3.1e to [3, 4, 1] 2024-02-05T08:10:57.175 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.1e', '3', '4', '1'] 2024-02-05T08:10:57.175 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pg-upmap 3.1e 3 4 1 2024-02-05T08:10:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:57.176 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:57.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:57.277 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:57.301 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:57.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:56 smithi196 ceph-mon[33018]: pgmap v246: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 768 B/s rd, 0 op/s 2024-02-05T08:10:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:56 smithi196 ceph-mon[33018]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:10:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:56 smithi196 ceph-mon[33018]: osdmap e159: 8 total, 8 up, 7 in 2024-02-05T08:10:57.403 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:57.427 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:57.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:57.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:57.553 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:57.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:57.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:57.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:57.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:57.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:57.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:57.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:57.915 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:10:57.939 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:57.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:57.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:10:58.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:10:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:10:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:10:58.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:58.068 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:58.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:58.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:58.170 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:58.194 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:58.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:58.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:58.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:58.320 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:58.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:58.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:58.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:10:58.449 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:58.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:58.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:10:58.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:10:58.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:58.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:58.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:10:58.680 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:58.704 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:58.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:58.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:58.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:10:58.832 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:58.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:58.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:10:58.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:58.957 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:58.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:58.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:10:59.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:59.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:59.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:59.178 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:10:58 smithi134 ceph-mon[30086]: pgmap v248: 148 pgs: 10 unknown, 138 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T08:10:59.178 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:10:58 smithi134 ceph-mon[39278]: pgmap v248: 148 pgs: 10 unknown, 138 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T08:10:59.190 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:10:59.202 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:10:59.217 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:59.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:59.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:10:59.219 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:10:59.219 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:10:59.219 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-02-05T08:10:59.219 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:10:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:10:58 smithi196 ceph-mon[33018]: pgmap v248: 148 pgs: 10 unknown, 138 active+clean; 582 KiB data, 999 MiB used, 625 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T08:10:59.319 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:59.343 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:59.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:59.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:59.444 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:59.470 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:59.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:59.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:59.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:59.600 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:59.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:59.702 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:10:59.725 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:59.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:59.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:10:59.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:10:59.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:10:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:59.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:10:59.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:10:59.980 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:10:59.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:10:59.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:00.082 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:00.106 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:00.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:00.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:00.208 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:00.214 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T08:11:00.231 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:00.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:00.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:00.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:00.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:00.418 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:00.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:00.487 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:00.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:00.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:00.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:00.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:00.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:00.718 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:00.742 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:00.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:00.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:00.844 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:00.868 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:00.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:00.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:00.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:00.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:00.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:01.003 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:11:01.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:00 smithi134 ceph-mon[30086]: pgmap v249: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:11:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:00 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/362527200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:11:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:00 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:11:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:00 smithi134 ceph-mon[39278]: pgmap v249: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:11:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:00 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/362527200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:11:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:00 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:11:01.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:01.126 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:01.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:01.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:01.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:01.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:01.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:00 smithi196 ceph-mon[33018]: pgmap v249: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:11:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:00 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/362527200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:11:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:00 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:11:01.358 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:01.385 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:01.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:01.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:01.510 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:01.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:01.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:01.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:01.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:01.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:01.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:01.766 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:01.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:01.868 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:01.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:01.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:01.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:01.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:02.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:02.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:02.126 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:02.141 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:11:02.141 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:11:02.141 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[30086]: osdmap e160: 8 total, 8 up, 7 in 2024-02-05T08:11:02.141 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[30086]: 3.10 scrub starts 2024-02-05T08:11:02.142 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[30086]: 3.10 scrub ok 2024-02-05T08:11:02.142 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[30086]: 6.12 scrub starts 2024-02-05T08:11:02.142 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[30086]: 6.12 scrub ok 2024-02-05T08:11:02.142 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:11:02.142 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:11:02.142 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[39278]: osdmap e160: 8 total, 8 up, 7 in 2024-02-05T08:11:02.142 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[39278]: 3.10 scrub starts 2024-02-05T08:11:02.142 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[39278]: 3.10 scrub ok 2024-02-05T08:11:02.142 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[39278]: 6.12 scrub starts 2024-02-05T08:11:02.142 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:01 smithi134 ceph-mon[39278]: 6.12 scrub ok 2024-02-05T08:11:02.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:02.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:02.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:02.277 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:02.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:02.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:01 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:11:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:01 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:11:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:01 smithi196 ceph-mon[33018]: osdmap e160: 8 total, 8 up, 7 in 2024-02-05T08:11:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:01 smithi196 ceph-mon[33018]: 3.10 scrub starts 2024-02-05T08:11:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:01 smithi196 ceph-mon[33018]: 3.10 scrub ok 2024-02-05T08:11:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:01 smithi196 ceph-mon[33018]: 6.12 scrub starts 2024-02-05T08:11:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:01 smithi196 ceph-mon[33018]: 6.12 scrub ok 2024-02-05T08:11:02.379 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:02.402 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:02.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:02.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:02.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:02.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:02.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:02.645 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:11:02.645 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:11:02.645 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:11:02.648 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:11:02.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:02.660 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:02.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:02.670 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:11:02.690 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:11:02.711 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:11:02.731 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:11:02.732 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:11:02.732 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:11:02.732 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:11:02.732 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:11:02.732 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:11:02.732 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:11:02.733 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:11:02.733 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:11:02.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:02.789 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:02.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:02.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:02.918 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:02.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:02.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:03.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:03.047 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:03.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:03.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:03.149 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:03.173 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:03.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:03.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:02 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:11:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:02 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:11:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:02 smithi134 ceph-mon[30086]: 4.8 scrub starts 2024-02-05T08:11:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:02 smithi134 ceph-mon[30086]: 4.8 scrub ok 2024-02-05T08:11:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:02 smithi134 ceph-mon[30086]: pgmap v251: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 771 B/s rd, 0 op/s 2024-02-05T08:11:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:02 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/47070761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:03.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:02 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:11:03.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:02 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:11:03.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:02 smithi134 ceph-mon[39278]: 4.8 scrub starts 2024-02-05T08:11:03.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:02 smithi134 ceph-mon[39278]: 4.8 scrub ok 2024-02-05T08:11:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:02 smithi134 ceph-mon[39278]: pgmap v251: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 771 B/s rd, 0 op/s 2024-02-05T08:11:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:02 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/47070761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:03.303 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:03.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:03.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:03.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:11:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:11:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:02 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:11:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:02 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:11:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:02 smithi196 ceph-mon[33018]: 4.8 scrub starts 2024-02-05T08:11:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:02 smithi196 ceph-mon[33018]: 4.8 scrub ok 2024-02-05T08:11:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:02 smithi196 ceph-mon[33018]: pgmap v251: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 771 B/s rd, 0 op/s 2024-02-05T08:11:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:02 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/47070761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:03.405 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:03.433 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:03.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:03.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:03.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:03.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:03.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:03.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:03.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:03.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:03.792 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:03.816 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:03.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:03.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:03.918 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:03.942 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:03.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:03.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:04.003 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-02-05T08:11:04.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:04.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:04.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:04.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:04.169 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:04.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:04.192 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:03 smithi134 ceph-mon[30086]: 6.7 scrub starts 2024-02-05T08:11:04.192 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:03 smithi134 ceph-mon[30086]: 6.7 scrub ok 2024-02-05T08:11:04.193 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:03 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:04.193 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:03 smithi134 ceph-mon[30086]: 3.8 scrub starts 2024-02-05T08:11:04.193 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:03 smithi134 ceph-mon[30086]: 3.8 scrub ok 2024-02-05T08:11:04.193 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:03 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/140065245' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:11:04.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:03 smithi134 ceph-mon[39278]: 6.7 scrub starts 2024-02-05T08:11:04.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:03 smithi134 ceph-mon[39278]: 6.7 scrub ok 2024-02-05T08:11:04.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:03 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:04.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:03 smithi134 ceph-mon[39278]: 3.8 scrub starts 2024-02-05T08:11:04.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:03 smithi134 ceph-mon[39278]: 3.8 scrub ok 2024-02-05T08:11:04.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:03 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/140065245' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:11:04.197 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:04.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:04.299 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:03 smithi196 ceph-mon[33018]: 6.7 scrub starts 2024-02-05T08:11:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:03 smithi196 ceph-mon[33018]: 6.7 scrub ok 2024-02-05T08:11:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:03 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:03 smithi196 ceph-mon[33018]: 3.8 scrub starts 2024-02-05T08:11:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:03 smithi196 ceph-mon[33018]: 3.8 scrub ok 2024-02-05T08:11:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:03 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/140065245' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:11:04.325 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:04.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:04.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:04.427 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:04.453 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:04.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:04.554 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:04.579 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:04.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:04.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:04.709 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:04.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:04.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:04.836 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:04.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:04.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:04.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:04.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:04.967 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:04.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:05.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:11:05.067 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:05.068 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[30086]: 5.7 scrub starts 2024-02-05T08:11:05.068 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[30086]: 5.7 scrub ok 2024-02-05T08:11:05.068 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[30086]: pgmap v252: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:11:05.068 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/140065245' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:11:05.069 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[30086]: osdmap e161: 8 total, 8 up, 7 in 2024-02-05T08:11:05.069 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[30086]: 5.4 scrub starts 2024-02-05T08:11:05.069 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[30086]: 5.4 scrub ok 2024-02-05T08:11:05.069 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[39278]: 5.7 scrub starts 2024-02-05T08:11:05.069 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[39278]: 5.7 scrub ok 2024-02-05T08:11:05.069 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[39278]: pgmap v252: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:11:05.069 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/140065245' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:11:05.069 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[39278]: osdmap e161: 8 total, 8 up, 7 in 2024-02-05T08:11:05.069 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[39278]: 5.4 scrub starts 2024-02-05T08:11:05.069 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:05 smithi134 ceph-mon[39278]: 5.4 scrub ok 2024-02-05T08:11:05.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:05.094 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:05.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:05.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:05.224 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:05.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:05 smithi196 ceph-mon[33018]: 5.7 scrub starts 2024-02-05T08:11:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:05 smithi196 ceph-mon[33018]: 5.7 scrub ok 2024-02-05T08:11:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:05 smithi196 ceph-mon[33018]: pgmap v252: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:11:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:05 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/140065245' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:11:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:05 smithi196 ceph-mon[33018]: osdmap e161: 8 total, 8 up, 7 in 2024-02-05T08:11:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:05 smithi196 ceph-mon[33018]: 5.4 scrub starts 2024-02-05T08:11:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:05 smithi196 ceph-mon[33018]: 5.4 scrub ok 2024-02-05T08:11:05.326 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:05.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:05.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:05.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:05.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:05.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:05.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:05.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:05.586 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:05.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:05.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:05.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:05.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:05.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:05.745 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:05.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:05.846 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:05.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:05.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:05.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:05.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:06.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:06.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:06.096 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:06 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:11:06.096 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:06 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:11:06.096 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:06 smithi134 ceph-mon[30086]: 3.16 scrub starts 2024-02-05T08:11:06.096 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:06 smithi134 ceph-mon[30086]: 3.16 scrub ok 2024-02-05T08:11:06.097 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:06 smithi134 ceph-mon[30086]: 1.0 scrub starts 2024-02-05T08:11:06.097 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:06 smithi134 ceph-mon[30086]: 1.0 scrub ok 2024-02-05T08:11:06.097 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:06 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:11:06.097 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:06 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:11:06.097 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:06 smithi134 ceph-mon[39278]: 3.16 scrub starts 2024-02-05T08:11:06.097 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:06 smithi134 ceph-mon[39278]: 3.16 scrub ok 2024-02-05T08:11:06.098 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:06 smithi134 ceph-mon[39278]: 1.0 scrub starts 2024-02-05T08:11:06.098 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:06 smithi134 ceph-mon[39278]: 1.0 scrub ok 2024-02-05T08:11:06.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:06.134 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:06.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:06.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:06.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:06.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:06.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:06 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:11:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:06 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:11:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:06 smithi196 ceph-mon[33018]: 3.16 scrub starts 2024-02-05T08:11:06.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:06 smithi196 ceph-mon[33018]: 3.16 scrub ok 2024-02-05T08:11:06.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:06 smithi196 ceph-mon[33018]: 1.0 scrub starts 2024-02-05T08:11:06.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:06 smithi196 ceph-mon[33018]: 1.0 scrub ok 2024-02-05T08:11:06.366 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:06.393 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:06.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:06.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:06.518 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:06.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:06.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:06.567 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:06.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:06.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:06.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:06.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:06.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:06.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:06.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:06.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:06.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:06.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:07.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[30086]: pgmap v254: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:11:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[30086]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:11:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[30086]: osdmap e162: 8 total, 8 up, 7 in 2024-02-05T08:11:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[30086]: 6.5 scrub starts 2024-02-05T08:11:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[30086]: 6.5 scrub ok 2024-02-05T08:11:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[30086]: 4.18 scrub starts 2024-02-05T08:11:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[30086]: 4.18 scrub ok 2024-02-05T08:11:07.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[39278]: pgmap v254: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:11:07.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[39278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:11:07.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[39278]: osdmap e162: 8 total, 8 up, 7 in 2024-02-05T08:11:07.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[39278]: 6.5 scrub starts 2024-02-05T08:11:07.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[39278]: 6.5 scrub ok 2024-02-05T08:11:07.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[39278]: 4.18 scrub starts 2024-02-05T08:11:07.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-mon[39278]: 4.18 scrub ok 2024-02-05T08:11:07.038 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:07.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:07.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:07.167 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:07.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:07.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:07.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:07.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:07.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:07.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:07 smithi196 ceph-mon[33018]: pgmap v254: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:11:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:07 smithi196 ceph-mon[33018]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:11:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:07 smithi196 ceph-mon[33018]: osdmap e162: 8 total, 8 up, 7 in 2024-02-05T08:11:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:07 smithi196 ceph-mon[33018]: 6.5 scrub starts 2024-02-05T08:11:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:07 smithi196 ceph-mon[33018]: 6.5 scrub ok 2024-02-05T08:11:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:07 smithi196 ceph-mon[33018]: 4.18 scrub starts 2024-02-05T08:11:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:07 smithi196 ceph-mon[33018]: 4.18 scrub ok 2024-02-05T08:11:07.398 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:11:07.398 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:11:07.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:07.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:07.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:07.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:07.557 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:07.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:07.603 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:11:07.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:07.686 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:07.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:07.788 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:07.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:07.813 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:07.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:07.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:07.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:07.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:07.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:07.953 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:11:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:11:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:11:08.015 INFO:teuthology.orchestra.run.smithi134.stderr:set 3.1e pg_upmap mapping to [3,4,1] 2024-02-05T08:11:08.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:08.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:08.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:08.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:08.200 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:08.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:11:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:11:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: 5.1d scrub starts 2024-02-05T08:11:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: 5.1d scrub ok 2024-02-05T08:11:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:11:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: 6.10 scrub starts 2024-02-05T08:11:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: 6.10 scrub ok 2024-02-05T08:11:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: 3.11 scrub starts 2024-02-05T08:11:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: 3.11 scrub ok 2024-02-05T08:11:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: 3.3 scrub starts 2024-02-05T08:11:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: 3.3 scrub ok 2024-02-05T08:11:08.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: 4.19 scrub starts 2024-02-05T08:11:08.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: 4.19 scrub ok 2024-02-05T08:11:08.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3406725533' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.1e", "id": [3, 4, 1]}]: dispatch 2024-02-05T08:11:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:11:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:11:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: 5.1d scrub starts 2024-02-05T08:11:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: 5.1d scrub ok 2024-02-05T08:11:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:11:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: 6.10 scrub starts 2024-02-05T08:11:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: 6.10 scrub ok 2024-02-05T08:11:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: 3.11 scrub starts 2024-02-05T08:11:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: 3.11 scrub ok 2024-02-05T08:11:08.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: 3.3 scrub starts 2024-02-05T08:11:08.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: 3.3 scrub ok 2024-02-05T08:11:08.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: 4.19 scrub starts 2024-02-05T08:11:08.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: 4.19 scrub ok 2024-02-05T08:11:08.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:08 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3406725533' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.1e", "id": [3, 4, 1]}]: dispatch 2024-02-05T08:11:08.301 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:11:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:11:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: 5.1d scrub starts 2024-02-05T08:11:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: 5.1d scrub ok 2024-02-05T08:11:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:11:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: 6.10 scrub starts 2024-02-05T08:11:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: 6.10 scrub ok 2024-02-05T08:11:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: 3.11 scrub starts 2024-02-05T08:11:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: 3.11 scrub ok 2024-02-05T08:11:08.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: 3.3 scrub starts 2024-02-05T08:11:08.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: 3.3 scrub ok 2024-02-05T08:11:08.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: 4.19 scrub starts 2024-02-05T08:11:08.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: 4.19 scrub ok 2024-02-05T08:11:08.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:08 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3406725533' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.1e", "id": [3, 4, 1]}]: dispatch 2024-02-05T08:11:08.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:08.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:08.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:08.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:08.451 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:08.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:08.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:08.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:08.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:08.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:08.683 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:08.708 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:08.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:08.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:08.833 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:08.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:08.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:08.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:08.964 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:08.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:09.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:09.092 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:09.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:09.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:09.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:09.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:09.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:09.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[30086]: pgmap v256: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T08:11:09.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3406725533' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.1e", "id": [3, 4, 1]}]': finished 2024-02-05T08:11:09.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[30086]: osdmap e163: 8 total, 8 up, 7 in 2024-02-05T08:11:09.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[30086]: 6.9 scrub starts 2024-02-05T08:11:09.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[30086]: 6.9 scrub ok 2024-02-05T08:11:09.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[30086]: 6.d scrub starts 2024-02-05T08:11:09.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[30086]: 6.d scrub ok 2024-02-05T08:11:09.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[39278]: pgmap v256: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T08:11:09.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3406725533' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.1e", "id": [3, 4, 1]}]': finished 2024-02-05T08:11:09.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[39278]: osdmap e163: 8 total, 8 up, 7 in 2024-02-05T08:11:09.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[39278]: 6.9 scrub starts 2024-02-05T08:11:09.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[39278]: 6.9 scrub ok 2024-02-05T08:11:09.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[39278]: 6.d scrub starts 2024-02-05T08:11:09.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:09 smithi134 ceph-mon[39278]: 6.d scrub ok 2024-02-05T08:11:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:09 smithi196 ceph-mon[33018]: pgmap v256: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T08:11:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:09 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3406725533' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.1e", "id": [3, 4, 1]}]': finished 2024-02-05T08:11:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:09 smithi196 ceph-mon[33018]: osdmap e163: 8 total, 8 up, 7 in 2024-02-05T08:11:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:09 smithi196 ceph-mon[33018]: 6.9 scrub starts 2024-02-05T08:11:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:09 smithi196 ceph-mon[33018]: 6.9 scrub ok 2024-02-05T08:11:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:09 smithi196 ceph-mon[33018]: 6.d scrub starts 2024-02-05T08:11:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:09 smithi196 ceph-mon[33018]: 6.d scrub ok 2024-02-05T08:11:09.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:09.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:09.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:09.394 DEBUG:teuthology.orchestra.run.smithi134:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-02-05T08:11:09.397 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T08:11:09.398 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T08:11:09.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:hints = 1 2024-02-05T08:11:09.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T08:11:09.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_115836 2024-02-05T08:11:09.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:11:09.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-02-05T08:11:09.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:09.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:09.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:09.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:09.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:09.624 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:09.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:09.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:09.786 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:09.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:09.888 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:09.928 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:09.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:10.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:10.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:10.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:10.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:10.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:10.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T08:11:10.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:10.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:10.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:10 smithi134 ceph-mon[30086]: osdmap e164: 8 total, 8 up, 7 in 2024-02-05T08:11:10.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:10 smithi134 ceph-mon[30086]: 6.3 scrub starts 2024-02-05T08:11:10.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:10 smithi134 ceph-mon[30086]: 6.3 scrub ok 2024-02-05T08:11:10.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:10 smithi134 ceph-mon[39278]: osdmap e164: 8 total, 8 up, 7 in 2024-02-05T08:11:10.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:10 smithi134 ceph-mon[39278]: 6.3 scrub starts 2024-02-05T08:11:10.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:10 smithi134 ceph-mon[39278]: 6.3 scrub ok 2024-02-05T08:11:10.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:10.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:10 smithi196 ceph-mon[33018]: osdmap e164: 8 total, 8 up, 7 in 2024-02-05T08:11:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:10 smithi196 ceph-mon[33018]: 6.3 scrub starts 2024-02-05T08:11:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:10 smithi196 ceph-mon[33018]: 6.3 scrub ok 2024-02-05T08:11:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:10.351 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:10.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:10.448 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 1 15 3294 3279 204.909 204.938 0.00259766 0.00485512 2024-02-05T08:11:10.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:10.483 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:10.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:10.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:10.585 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:10.629 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:10.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:10.731 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:10.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:10.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:10.819 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:10.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:10.923 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:10.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:10.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:11.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:11.061 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:11.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:11.127 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:11.162 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:11.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:11.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:11.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:11.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:11 smithi134 ceph-mon[30086]: pgmap v259: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 912 B/s rd, 0 op/s 2024-02-05T08:11:11.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:11 smithi134 ceph-mon[30086]: 6.1c scrub starts 2024-02-05T08:11:11.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:11 smithi134 ceph-mon[30086]: 6.1c scrub ok 2024-02-05T08:11:11.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:11 smithi134 ceph-mon[30086]: 4.e scrub starts 2024-02-05T08:11:11.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:11 smithi134 ceph-mon[30086]: 4.e scrub ok 2024-02-05T08:11:11.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:11 smithi134 ceph-mon[39278]: pgmap v259: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 912 B/s rd, 0 op/s 2024-02-05T08:11:11.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:11 smithi134 ceph-mon[39278]: 6.1c scrub starts 2024-02-05T08:11:11.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:11 smithi134 ceph-mon[39278]: 6.1c scrub ok 2024-02-05T08:11:11.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:11 smithi134 ceph-mon[39278]: 4.e scrub starts 2024-02-05T08:11:11.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:11 smithi134 ceph-mon[39278]: 4.e scrub ok 2024-02-05T08:11:11.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:11 smithi196 ceph-mon[33018]: pgmap v259: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 912 B/s rd, 0 op/s 2024-02-05T08:11:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:11 smithi196 ceph-mon[33018]: 6.1c scrub starts 2024-02-05T08:11:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:11 smithi196 ceph-mon[33018]: 6.1c scrub ok 2024-02-05T08:11:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:11 smithi196 ceph-mon[33018]: 4.e scrub starts 2024-02-05T08:11:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:11 smithi196 ceph-mon[33018]: 4.e scrub ok 2024-02-05T08:11:11.338 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:11.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:11.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:11.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:11.445 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 2 16 6264 6248 195.214 185.562 0.0097685 0.0051076 2024-02-05T08:11:11.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:11.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:11.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:11.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:11.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:11.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:11.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:11.706 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:11.746 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:11.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:11.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:11.848 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:11.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:11.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:11.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:11.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:12.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:12.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:12.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:12.113 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:12.155 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:12.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:12.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:12.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:12.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:12.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:12.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:12.310 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:12 smithi134 ceph-mon[30086]: 3.13 scrub starts 2024-02-05T08:11:12.310 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:12 smithi134 ceph-mon[30086]: 3.13 scrub ok 2024-02-05T08:11:12.311 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:12 smithi134 ceph-mon[39278]: 3.13 scrub starts 2024-02-05T08:11:12.311 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:12 smithi134 ceph-mon[39278]: 3.13 scrub ok 2024-02-05T08:11:12.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:12 smithi196 ceph-mon[33018]: 3.13 scrub starts 2024-02-05T08:11:12.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:12 smithi196 ceph-mon[33018]: 3.13 scrub ok 2024-02-05T08:11:12.393 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:12.419 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:12.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:12.446 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 3 16 8987 8971 186.868 170.188 0.00207381 0.00524288 2024-02-05T08:11:12.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:12.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:12.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:12.651 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:12.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:12.692 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:12.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:12.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:12.837 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:12.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:12.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:12.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:12.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:12.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:13.054 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:11:13.072 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:13.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:13.116 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:13.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:13.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:13.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:13.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:13.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:13.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:11:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:11:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:13 smithi196 ceph-mon[33018]: pgmap v260: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:11:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:13 smithi196 ceph-mon[33018]: 5.c scrub starts 2024-02-05T08:11:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:13 smithi196 ceph-mon[33018]: 5.c scrub ok 2024-02-05T08:11:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:13 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3625735305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:11:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:13 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:11:13.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:13.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:13.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:13.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:13.446 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 4 16 11456 11440 178.727 154.312 0.00626081 0.00558647 2024-02-05T08:11:13.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:13.510 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:13.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:13.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:13.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:13 smithi134 ceph-mon[30086]: pgmap v260: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:11:13.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:13 smithi134 ceph-mon[30086]: 5.c scrub starts 2024-02-05T08:11:13.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:13 smithi134 ceph-mon[30086]: 5.c scrub ok 2024-02-05T08:11:13.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:13 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3625735305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:11:13.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:13 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:11:13.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:13 smithi134 ceph-mon[39278]: pgmap v260: 148 pgs: 148 active+clean; 582 KiB data, 998 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:11:13.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:13 smithi134 ceph-mon[39278]: 5.c scrub starts 2024-02-05T08:11:13.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:13 smithi134 ceph-mon[39278]: 5.c scrub ok 2024-02-05T08:11:13.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:13 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3625735305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:11:13.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:13 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:11:13.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:13.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:13.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:13.747 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:13.787 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:13.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:13.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:13.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:13.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:13.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:14.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:14.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:14.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:14.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:14.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:14.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:14.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:14.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:14.295 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:14.340 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:14.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:14.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:14.445 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:14.447 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 5 16 14109 14093 176.107 165.812 0.00546403 0.00566755 2024-02-05T08:11:14.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:14.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:14.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:14.524 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:11:14.524 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:11:14.524 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:11:14.524 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[30086]: osdmap e165: 8 total, 8 up, 7 in 2024-02-05T08:11:14.524 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:14.524 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[30086]: 5.3 scrub starts 2024-02-05T08:11:14.525 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[30086]: 5.3 scrub ok 2024-02-05T08:11:14.525 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[30086]: 6.a scrub starts 2024-02-05T08:11:14.525 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[30086]: 6.a scrub ok 2024-02-05T08:11:14.525 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:11:14.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:11:14.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:11:14.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[39278]: osdmap e165: 8 total, 8 up, 7 in 2024-02-05T08:11:14.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:14.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[39278]: 5.3 scrub starts 2024-02-05T08:11:14.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[39278]: 5.3 scrub ok 2024-02-05T08:11:14.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[39278]: 6.a scrub starts 2024-02-05T08:11:14.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:14 smithi134 ceph-mon[39278]: 6.a scrub ok 2024-02-05T08:11:14.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:14 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:11:14.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:14 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:11:14.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:14 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:11:14.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:14 smithi196 ceph-mon[33018]: osdmap e165: 8 total, 8 up, 7 in 2024-02-05T08:11:14.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:14 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:14.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:14 smithi196 ceph-mon[33018]: 5.3 scrub starts 2024-02-05T08:11:14.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:14 smithi196 ceph-mon[33018]: 5.3 scrub ok 2024-02-05T08:11:14.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:14 smithi196 ceph-mon[33018]: 6.a scrub starts 2024-02-05T08:11:14.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:14 smithi196 ceph-mon[33018]: 6.a scrub ok 2024-02-05T08:11:14.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:14.618 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:14.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:14.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:14.707 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:11:14.708 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:11:14.708 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-02-05T08:11:14.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:11:14.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:14.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:14.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:14.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:14.906 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:14.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:15.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:15.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:15.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:15.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:15.136 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:15.186 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:15.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:15.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:15.288 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:15.312 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:11:15.313 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:11:15.313 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:11:15.313 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:11:15.313 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:11:15.313 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:11:15.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:15 smithi196 ceph-mon[33018]: pgmap v262: 148 pgs: 148 active+clean; 614 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-02-05T08:11:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:15 smithi196 ceph-mon[33018]: 4.12 scrub starts 2024-02-05T08:11:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:15 smithi196 ceph-mon[33018]: 4.12 scrub ok 2024-02-05T08:11:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:15 smithi196 ceph-mon[33018]: 4.c scrub starts 2024-02-05T08:11:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:15 smithi196 ceph-mon[33018]: 4.c scrub ok 2024-02-05T08:11:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:15 smithi196 ceph-mon[33018]: 5.9 scrub starts 2024-02-05T08:11:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:15 smithi196 ceph-mon[33018]: 5.9 scrub ok 2024-02-05T08:11:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:15 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2244987450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:15.315 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:15.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:15.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:15.344 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:11:15.344 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[30086]: pgmap v262: 148 pgs: 148 active+clean; 614 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-02-05T08:11:15.344 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[30086]: 4.12 scrub starts 2024-02-05T08:11:15.344 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[30086]: 4.12 scrub ok 2024-02-05T08:11:15.345 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[30086]: 4.c scrub starts 2024-02-05T08:11:15.345 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[30086]: 4.c scrub ok 2024-02-05T08:11:15.345 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[30086]: 5.9 scrub starts 2024-02-05T08:11:15.345 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[30086]: 5.9 scrub ok 2024-02-05T08:11:15.345 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2244987450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:15.345 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[39278]: pgmap v262: 148 pgs: 148 active+clean; 614 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-02-05T08:11:15.345 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[39278]: 4.12 scrub starts 2024-02-05T08:11:15.345 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[39278]: 4.12 scrub ok 2024-02-05T08:11:15.345 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[39278]: 4.c scrub starts 2024-02-05T08:11:15.346 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[39278]: 4.c scrub ok 2024-02-05T08:11:15.346 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[39278]: 5.9 scrub starts 2024-02-05T08:11:15.346 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[39278]: 5.9 scrub ok 2024-02-05T08:11:15.346 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:15 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2244987450' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:15.391 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:11:15.398 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:11:15.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:11:15.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:11:15.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:11:15.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:11:15.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:11:15.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:11:15.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:11:15.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:15.447 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 6 16 16201 16185 168.547 130.75 0.0089242 0.00592615 2024-02-05T08:11:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:15.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:15.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:15.549 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:15.591 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:15.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:15.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:15.693 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:15.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:15.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:15.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:15.874 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:15.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:15.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:16.006 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:16.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:16.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:16.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:16.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:16.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:16.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:16.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:16.293 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:16.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:16.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:16.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:16 smithi134 ceph-mon[30086]: 6.e deep-scrub starts 2024-02-05T08:11:16.323 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:16 smithi134 ceph-mon[30086]: 6.e deep-scrub ok 2024-02-05T08:11:16.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:16 smithi134 ceph-mon[30086]: 4.1f scrub starts 2024-02-05T08:11:16.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:16 smithi134 ceph-mon[30086]: 4.1f scrub ok 2024-02-05T08:11:16.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:16 smithi134 ceph-mon[30086]: 3.18 scrub starts 2024-02-05T08:11:16.324 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:16 smithi134 ceph-mon[30086]: 3.18 scrub ok 2024-02-05T08:11:16.324 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:16 smithi134 ceph-mon[39278]: 6.e deep-scrub starts 2024-02-05T08:11:16.324 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:16 smithi134 ceph-mon[39278]: 6.e deep-scrub ok 2024-02-05T08:11:16.324 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:16 smithi134 ceph-mon[39278]: 4.1f scrub starts 2024-02-05T08:11:16.324 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:16 smithi134 ceph-mon[39278]: 4.1f scrub ok 2024-02-05T08:11:16.324 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:16 smithi134 ceph-mon[39278]: 3.18 scrub starts 2024-02-05T08:11:16.324 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:16 smithi134 ceph-mon[39278]: 3.18 scrub ok 2024-02-05T08:11:16.395 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:16.439 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:16.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:16.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:16.448 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 7 15 18483 18468 164.851 142.688 0.00501004 0.00605894 2024-02-05T08:11:16.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:16 smithi196 ceph-mon[33018]: 6.e deep-scrub starts 2024-02-05T08:11:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:16 smithi196 ceph-mon[33018]: 6.e deep-scrub ok 2024-02-05T08:11:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:16 smithi196 ceph-mon[33018]: 4.1f scrub starts 2024-02-05T08:11:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:16 smithi196 ceph-mon[33018]: 4.1f scrub ok 2024-02-05T08:11:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:16 smithi196 ceph-mon[33018]: 3.18 scrub starts 2024-02-05T08:11:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:16 smithi196 ceph-mon[33018]: 3.18 scrub ok 2024-02-05T08:11:16.573 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:16.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:16.675 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:16.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:16.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:16.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:16.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:16.875 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:16.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:16.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:16.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:17.014 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:17.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:17.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:17.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:17.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:17.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:17.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:17.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:17.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:17.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:17.334 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:17 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:11:17.334 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:17 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:11:17.334 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:17 smithi134 ceph-mon[30086]: pgmap v263: 148 pgs: 148 active+clean; 614 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 673 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T08:11:17.334 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:17 smithi134 ceph-mon[30086]: 5.10 scrub starts 2024-02-05T08:11:17.334 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:17 smithi134 ceph-mon[30086]: 5.10 scrub ok 2024-02-05T08:11:17.334 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:17 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:11:17.334 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:17 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:11:17.334 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:17 smithi134 ceph-mon[39278]: pgmap v263: 148 pgs: 148 active+clean; 614 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 673 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T08:11:17.335 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:17 smithi134 ceph-mon[39278]: 5.10 scrub starts 2024-02-05T08:11:17.335 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:17 smithi134 ceph-mon[39278]: 5.10 scrub ok 2024-02-05T08:11:17.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:17.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:17.416 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:17.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:17.447 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 8 16 20712 20696 161.65 139.25 0.00457653 0.00618086 2024-02-05T08:11:17.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:17.546 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:17.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:17.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:17 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:11:17.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:17 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:11:17.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:17 smithi196 ceph-mon[33018]: pgmap v263: 148 pgs: 148 active+clean; 614 MiB data, 2.7 GiB used, 623 GiB / 626 GiB avail; 673 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T08:11:17.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:17 smithi196 ceph-mon[33018]: 5.10 scrub starts 2024-02-05T08:11:17.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:17 smithi196 ceph-mon[33018]: 5.10 scrub ok 2024-02-05T08:11:17.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:17.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:17.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:17.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:17.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:17.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:17.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:17.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:17.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:17.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:17.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:18.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:11:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:11:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:11:18.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:18.087 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:18.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:18.188 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:18.220 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:18.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:18.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:18.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:18.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:18.447 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 9 16 22749 22733 157.835 127.312 0.0124387 0.0063285 2024-02-05T08:11:18.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:18.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:18.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:18.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:18.595 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:18.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:18.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:18.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:18.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:18.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:18.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:18.797 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:11:18.862 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:18.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:18.888 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:18.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:18.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:19.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:19.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:19.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:19.136 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:19 smithi134 ceph-mon[30086]: pgmap v264: 148 pgs: 148 active+clean; 886 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-05T08:11:19.137 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:19 smithi134 ceph-mon[39278]: pgmap v264: 148 pgs: 148 active+clean; 886 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-05T08:11:19.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:19.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:19.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:19.298 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:19.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:19.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:19.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:19.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:19.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:19.447 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 10 16 24239 24223 151.363 93.125 0.00253952 0.00656517 2024-02-05T08:11:19.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:19.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:19 smithi134 ceph-mon[30086]: 5.19 scrub starts 2024-02-05T08:11:19.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:19 smithi134 ceph-mon[30086]: 5.19 scrub ok 2024-02-05T08:11:19.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:19 smithi134 ceph-mon[30086]: 6.15 scrub starts 2024-02-05T08:11:19.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:19 smithi134 ceph-mon[30086]: 6.15 scrub ok 2024-02-05T08:11:19.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:19 smithi134 ceph-mon[39278]: 5.19 scrub starts 2024-02-05T08:11:19.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:19 smithi134 ceph-mon[39278]: 5.19 scrub ok 2024-02-05T08:11:19.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:19 smithi134 ceph-mon[39278]: 6.15 scrub starts 2024-02-05T08:11:19.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:19 smithi134 ceph-mon[39278]: 6.15 scrub ok 2024-02-05T08:11:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:19 smithi196 ceph-mon[33018]: pgmap v264: 148 pgs: 148 active+clean; 886 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-05T08:11:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:19 smithi196 ceph-mon[33018]: 5.19 scrub starts 2024-02-05T08:11:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:19 smithi196 ceph-mon[33018]: 5.19 scrub ok 2024-02-05T08:11:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:19 smithi196 ceph-mon[33018]: 6.15 scrub starts 2024-02-05T08:11:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:19 smithi196 ceph-mon[33018]: 6.15 scrub ok 2024-02-05T08:11:19.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:19.567 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:19.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:19.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:19.699 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:19.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:19.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:19.801 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:19.824 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:19.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:19.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:19.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:19.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:20.056 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:20.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:20.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:20.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:20.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:20.214 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:20.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:20.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:20.357 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:20.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:20.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:20.448 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 11 16 25750 25734 146.187 94.4375 0.0114659 0.00681387 2024-02-05T08:11:20.460 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:20.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:20.484 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:20.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[39278]: 5.8 scrub starts 2024-02-05T08:11:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[39278]: 5.8 scrub ok 2024-02-05T08:11:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[39278]: 3.4 scrub starts 2024-02-05T08:11:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[39278]: 3.4 scrub ok 2024-02-05T08:11:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:11:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:11:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[39278]: 5.18 deep-scrub starts 2024-02-05T08:11:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[39278]: 5.18 deep-scrub ok 2024-02-05T08:11:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[39278]: 4.15 scrub starts 2024-02-05T08:11:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[39278]: 4.15 scrub ok 2024-02-05T08:11:20.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[30086]: 5.8 scrub starts 2024-02-05T08:11:20.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[30086]: 5.8 scrub ok 2024-02-05T08:11:20.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[30086]: 3.4 scrub starts 2024-02-05T08:11:20.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[30086]: 3.4 scrub ok 2024-02-05T08:11:20.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:11:20.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:11:20.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[30086]: 5.18 deep-scrub starts 2024-02-05T08:11:20.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[30086]: 5.18 deep-scrub ok 2024-02-05T08:11:20.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[30086]: 4.15 scrub starts 2024-02-05T08:11:20.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:20 smithi134 ceph-mon[30086]: 4.15 scrub ok 2024-02-05T08:11:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:20 smithi196 ceph-mon[33018]: 5.8 scrub starts 2024-02-05T08:11:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:20 smithi196 ceph-mon[33018]: 5.8 scrub ok 2024-02-05T08:11:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:20 smithi196 ceph-mon[33018]: 3.4 scrub starts 2024-02-05T08:11:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:20 smithi196 ceph-mon[33018]: 3.4 scrub ok 2024-02-05T08:11:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:20 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:11:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:20 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:11:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:20 smithi196 ceph-mon[33018]: 5.18 deep-scrub starts 2024-02-05T08:11:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:20 smithi196 ceph-mon[33018]: 5.18 deep-scrub ok 2024-02-05T08:11:20.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:20 smithi196 ceph-mon[33018]: 4.15 scrub starts 2024-02-05T08:11:20.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:20 smithi196 ceph-mon[33018]: 4.15 scrub ok 2024-02-05T08:11:20.585 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:20.625 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:20.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:20.727 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:20.768 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:20.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:20.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:20.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:20.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:20.901 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:20.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:21.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:21.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:21.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:21.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:21.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:21.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:21.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:21.322 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:21.365 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:21.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:21.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:21.448 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 12 15 27240 27225 141.77 93.1875 0.00822846 0.00704744 2024-02-05T08:11:21.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:21.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:21.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:21.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:11:21.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:11:21.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[30086]: pgmap v265: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-05T08:11:21.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[30086]: osdmap e166: 8 total, 8 up, 7 in 2024-02-05T08:11:21.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[30086]: 3.17 scrub starts 2024-02-05T08:11:21.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[30086]: 3.17 scrub ok 2024-02-05T08:11:21.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[30086]: 3.1d scrub starts 2024-02-05T08:11:21.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[30086]: 3.1d scrub ok 2024-02-05T08:11:21.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:11:21.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:11:21.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[39278]: pgmap v265: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-05T08:11:21.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[39278]: osdmap e166: 8 total, 8 up, 7 in 2024-02-05T08:11:21.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[39278]: 3.17 scrub starts 2024-02-05T08:11:21.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[39278]: 3.17 scrub ok 2024-02-05T08:11:21.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[39278]: 3.1d scrub starts 2024-02-05T08:11:21.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:21 smithi134 ceph-mon[39278]: 3.1d scrub ok 2024-02-05T08:11:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:21 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:11:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:21 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:11:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:21 smithi196 ceph-mon[33018]: pgmap v265: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 134 MiB/s wr, 2.14k op/s 2024-02-05T08:11:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:21 smithi196 ceph-mon[33018]: osdmap e166: 8 total, 8 up, 7 in 2024-02-05T08:11:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:21 smithi196 ceph-mon[33018]: 3.17 scrub starts 2024-02-05T08:11:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:21 smithi196 ceph-mon[33018]: 3.17 scrub ok 2024-02-05T08:11:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:21 smithi196 ceph-mon[33018]: 3.1d scrub starts 2024-02-05T08:11:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:21 smithi196 ceph-mon[33018]: 3.1d scrub ok 2024-02-05T08:11:21.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:21.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:21.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:21.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:21.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:21.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:21.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:21.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:21.897 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:21.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:21.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:22.016 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:11:22.016 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:11:22.016 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:11:22.016 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:11:22.016 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.016 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:19184 7.8 7:1698bfb9:::benchmark_data_smithi134_83016_object19183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:11:22.016 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.649970+0000", 2024-02-05T08:11:22.016 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 263.35983837600003, 2024-02-05T08:11:22.017 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12589771399999999, 2024-02-05T08:11:22.017 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.017 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.017 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.017 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.017 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.017 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 19184 2024-02-05T08:11:22.017 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.017 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.017 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.017 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.017 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649970+0000", 2024-02-05T08:11:22.017 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649970+0000", 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649971+0000", 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3900000000000003e-07 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.018 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649989+0000", 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8097999999999999e-05 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.649999+0000", 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0023999999999999e-05 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.650005+0000", 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7200000000000003e-06 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.019 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.020 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.650045+0000", 2024-02-05T08:11:22.020 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0070000000000001e-05 2024-02-05T08:11:22.020 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.020 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.020 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.020 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.650118+0000", 2024-02-05T08:11:22.020 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2837999999999996e-05 2024-02-05T08:11:22.020 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.020 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.020 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,7", 2024-02-05T08:11:22.020 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.650209+0000", 2024-02-05T08:11:22.020 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1215e-05 2024-02-05T08:11:22.020 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.021 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.021 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.021 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.650951+0000", 2024-02-05T08:11:22.021 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00074151200000000005 2024-02-05T08:11:22.021 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.021 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.021 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.021 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651227+0000", 2024-02-05T08:11:22.021 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00027632399999999999 2024-02-05T08:11:22.021 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.021 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.021 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.021 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651273+0000", 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6313999999999997e-05 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.775807+0000", 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12453357800000001 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.775868+0000", 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1081999999999996e-05 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.022 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:19184 7.8 e104/95 ondisk, result = 0)", 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:06:58.651219+0000", 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 263.35858927200002, 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.124660887, 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651219+0000", 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.023 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.024 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.024 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.024 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651219+0000", 2024-02-05T08:11:22.024 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.024 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.024 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.024 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.024 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651220+0000", 2024-02-05T08:11:22.024 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5700000000000002e-07 2024-02-05T08:11:22.024 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.024 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.024 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.024 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651224+0000", 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.732e-06 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651232+0000", 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4859999999999997e-06 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651237+0000", 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0930000000000002e-06 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651266+0000", 2024-02-05T08:11:22.025 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8612999999999999e-05 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.651270+0000", 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3499999999999999e-06 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:06:58.775880+0000", 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12460995599999999 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.026 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21740 7.e 7:702731a7:::benchmark_data_smithi134_83016_object21739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.327781+0000", 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 261.68202758899997, 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15303014300000001, 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21740 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.027 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.028 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327781+0000", 2024-02-05T08:11:22.028 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.028 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.028 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.028 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327781+0000", 2024-02-05T08:11:22.028 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.028 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.028 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.028 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327783+0000", 2024-02-05T08:11:22.028 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.688e-06 2024-02-05T08:11:22.028 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327805+0000", 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2300999999999999e-05 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327817+0000", 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2233e-05 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327825+0000", 2024-02-05T08:11:22.029 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1450000000000008e-06 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327885+0000", 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9546999999999998e-05 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.328002+0000", 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000117374 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.030 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.328125+0000", 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012256800000000001 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329306+0000", 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001181423 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330032+0000", 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00072612800000000004 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.031 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.032 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480676+0000", 2024-02-05T08:11:22.032 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15064364599999999 2024-02-05T08:11:22.032 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.032 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.032 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.032 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480734+0000", 2024-02-05T08:11:22.032 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7960000000000001e-05 2024-02-05T08:11:22.032 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.032 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.032 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.032 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480811+0000", 2024-02-05T08:11:22.032 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7130000000000002e-05 2024-02-05T08:11:22.032 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:21738 7.a e104/95 7:5f5817b2:::benchmark_data_smithi134_83016_object21737:head v 104'1353, mlcod=104'1351)", 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.329123+0000", 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 261.68068537599999, 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15121406700000001, 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329123+0000", 2024-02-05T08:11:22.033 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.034 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.034 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.034 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.034 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329123+0000", 2024-02-05T08:11:22.034 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.034 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.034 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.034 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.034 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329124+0000", 2024-02-05T08:11:22.034 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2500000000000001e-07 2024-02-05T08:11:22.034 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.034 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.034 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329156+0000", 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2563e-05 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329166+0000", 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6069999999999997e-06 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329172+0000", 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5339999999999996e-06 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.035 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329270+0000", 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7374999999999995e-05 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329289+0000", 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9258000000000001e-05 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330442+0000", 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00115318 2024-02-05T08:11:22.036 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480337+0000", 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14989512499999999 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21746 7.8 7:1fd2f339:::benchmark_data_smithi134_83016_object21745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.329211+0000", 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 261.680597901, 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15186722999999999, 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.037 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21746 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329211+0000", 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.038 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329211+0000", 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329212+0000", 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1200000000000001e-06 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329233+0000", 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1157000000000001e-05 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.039 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329245+0000", 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2095e-05 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329253+0000", 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0059999999999992e-06 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329312+0000", 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8953000000000003e-05 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.040 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329460+0000", 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014786199999999999 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,7", 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329592+0000", 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013251199999999999 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330443+0000", 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00085042699999999995 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480847+0000", 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15040416000000001 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480896+0000", 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9152e-05 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.481044+0000", 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014736299999999999 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.042 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.043 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.043 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.481078+0000", 2024-02-05T08:11:22.043 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4422999999999999e-05 2024-02-05T08:11:22.043 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.043 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.043 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.043 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.043 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.043 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:21746 7.8 e104/95 ondisk, result = 0)", 2024-02-05T08:11:22.044 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.330823+0000", 2024-02-05T08:11:22.044 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 261.67898577400001, 2024-02-05T08:11:22.044 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15025973000000001, 2024-02-05T08:11:22.044 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.045 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:11:22.045 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.045 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.045 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.045 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330823+0000", 2024-02-05T08:11:22.045 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.045 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.045 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.045 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.045 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330823+0000", 2024-02-05T08:11:22.045 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.045 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.045 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330823+0000", 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7100000000000002e-07 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330827+0000", 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7249999999999999e-06 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480582+0000", 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14975533799999999 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.046 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.047 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.047 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480591+0000", 2024-02-05T08:11:22.047 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8789999999999992e-06 2024-02-05T08:11:22.047 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.047 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.047 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.047 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480886+0000", 2024-02-05T08:11:22.047 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00029442999999999998 2024-02-05T08:11:22.047 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.047 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.047 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.047 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480892+0000", 2024-02-05T08:11:22.047 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1240000000000002e-06 2024-02-05T08:11:22.048 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.048 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.048 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.048 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.481083+0000", 2024-02-05T08:11:22.048 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019066299999999999 2024-02-05T08:11:22.048 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.048 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.048 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.048 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.048 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.048 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:21746 7.8 e104/95 ondisk, result = 0)", 2024-02-05T08:11:22.048 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.330832+0000", 2024-02-05T08:11:22.048 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 261.67897670500003, 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.150026044, 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330832+0000", 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330832+0000", 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330832+0000", 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3799999999999997e-07 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330835+0000", 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.971e-06 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480208+0000", 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.149372539 2024-02-05T08:11:22.050 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480221+0000", 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2695e-05 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480323+0000", 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010194799999999999 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480331+0000", 2024-02-05T08:11:22.051 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3499999999999997e-06 2024-02-05T08:11:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480858+0000", 2024-02-05T08:11:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052700299999999995 2024-02-05T08:11:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:99784 7.e 7:78fc162f:::benchmark_data_smithi134_83016_object99783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:11:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:08.939664+0000", 2024-02-05T08:11:22.052 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 193.07014469699999, 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13795306500000001, 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 99784 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939664+0000", 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.053 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.054 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.054 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.054 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939664+0000", 2024-02-05T08:11:22.054 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.054 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.054 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.054 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.054 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939664+0000", 2024-02-05T08:11:22.054 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7e-07 2024-02-05T08:11:22.054 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.054 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.054 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.054 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939688+0000", 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3336000000000002e-05 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939695+0000", 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5959999999999998e-06 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939701+0000", 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6119999999999998e-06 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939767+0000", 2024-02-05T08:11:22.055 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6088999999999997e-05 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.939882+0000", 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011451900000000001 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.940010+0000", 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012856599999999999 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.943195+0000", 2024-02-05T08:11:22.056 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003185121 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:08.947687+0000", 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.004491292 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.077504+0000", 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12981711600000001 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.077570+0000", 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6507999999999994e-05 2024-02-05T08:11:22.057 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:09.077617+0000", 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6839999999999999e-05 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106208 7.e 7:79fec55e:::benchmark_data_smithi134_83016_object106207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:15.904619+0000", 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 186.10519002800001, 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.120579906, 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.058 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106208 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904619+0000", 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904619+0000", 2024-02-05T08:11:22.059 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904620+0000", 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7579999999999999e-06 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904647+0000", 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6169e-05 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.060 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904661+0000", 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4321000000000001e-05 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904669+0000", 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7800000000000001e-06 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909475+0000", 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0048067120000000003 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.061 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909614+0000", 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013891899999999999 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909949+0000", 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00033502599999999999 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.922098+0000", 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012149139 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.062 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.922137+0000", 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9004000000000001e-05 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025064+0000", 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.102926244 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025154+0000", 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0464000000000007e-05 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.063 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.064 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.064 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025198+0000", 2024-02-05T08:11:22.064 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4369999999999997e-05 2024-02-05T08:11:22.064 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.064 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.064 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.064 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.064 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.064 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106209 7.e 7:78b87070:::benchmark_data_smithi134_83016_object106208:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:11:22.064 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:15.904686+0000", 2024-02-05T08:11:22.064 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 186.10512223800001, 2024-02-05T08:11:22.064 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.120675377, 2024-02-05T08:11:22.064 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106209 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904686+0000", 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.065 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904686+0000", 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904687+0000", 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7e-07 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904707+0000", 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0615000000000002e-05 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.066 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904715+0000", 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3690000000000001e-06 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.904720+0000", 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5550000000000002e-06 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.915304+0000", 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010583884 2024-02-05T08:11:22.067 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.915436+0000", 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013168000000000001 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.915587+0000", 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000150815 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.922149+0000", 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0065619090000000003 2024-02-05T08:11:22.068 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.069 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.069 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.069 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.017938+0000", 2024-02-05T08:11:22.069 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.095789183 2024-02-05T08:11:22.069 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.069 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.069 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.069 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025240+0000", 2024-02-05T08:11:22.069 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0073026159999999996 2024-02-05T08:11:22.069 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.069 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.069 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.069 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025325+0000", 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4917e-05 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025362+0000", 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6363999999999999e-05 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106211 7.e 7:7bf49e27:::benchmark_data_smithi134_83016_object106210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:15.906247+0000", 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 186.10356199500001, 2024-02-05T08:11:22.070 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12878605000000001, 2024-02-05T08:11:22.071 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.071 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.071 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.071 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.071 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.071 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106211 2024-02-05T08:11:22.071 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:22.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:22.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:22.073 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.073 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.073 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.073 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.906247+0000", 2024-02-05T08:11:22.073 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.073 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.073 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.073 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.073 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.906247+0000", 2024-02-05T08:11:22.073 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.906248+0000", 2024-02-05T08:11:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.06e-06 2024-02-05T08:11:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.906268+0000", 2024-02-05T08:11:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0755999999999999e-05 2024-02-05T08:11:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.074 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.906278+0000", 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8030000000000008e-06 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.906285+0000", 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0929999999999997e-06 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.918969+0000", 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012684140999999999 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.075 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.919104+0000", 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013453299999999999 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.919253+0000", 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014935 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.017948+0000", 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.098694226999999995 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.076 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.017981+0000", 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3278000000000002e-05 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.034565+0000", 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.016584216999999998 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.034974+0000", 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00040871100000000002 2024-02-05T08:11:22.077 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.035033+0000", 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8881000000000001e-05 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106217 7.e 7:74288038:::benchmark_data_smithi134_83016_object106216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:15.909843+0000", 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 186.099966112, 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12792978999999999, 2024-02-05T08:11:22.078 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106217 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909843+0000", 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.079 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909843+0000", 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909844+0000", 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.206e-06 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909867+0000", 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3632999999999999e-05 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.080 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909876+0000", 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8219999999999996e-06 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.909884+0000", 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3370000000000002e-06 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.018007+0000", 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.108123313 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.082 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.018147+0000", 2024-02-05T08:11:22.082 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014017599999999999 2024-02-05T08:11:22.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.082 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.082 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.082 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.018392+0000", 2024-02-05T08:11:22.082 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00024491200000000002 2024-02-05T08:11:22.082 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.082 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.082 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.082 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025444+0000", 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.007052221 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025502+0000", 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8044000000000002e-05 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.035086+0000", 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0095837630000000004 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.083 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.037726+0000", 2024-02-05T08:11:22.084 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0026399639999999999 2024-02-05T08:11:22.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.084 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.084 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.037772+0000", 2024-02-05T08:11:22.084 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6399000000000001e-05 2024-02-05T08:11:22.084 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.084 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.084 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.084 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.084 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.084 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106225 7.e 7:7e4a02c0:::benchmark_data_smithi134_83016_object106224:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:11:22.084 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:15.917206+0000", 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 186.09260275599999, 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.120823756, 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106225 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.917206+0000", 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.085 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.917206+0000", 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.917207+0000", 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5990000000000001e-06 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.917233+0000", 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5848000000000001e-05 2024-02-05T08:11:22.086 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.917243+0000", 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7040000000000006e-06 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:15.917249+0000", 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4649999999999999e-06 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.021506+0000", 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.104256852 2024-02-05T08:11:22.087 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.088 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.088 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.021656+0000", 2024-02-05T08:11:22.088 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015012700000000001 2024-02-05T08:11:22.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.088 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.088 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.021817+0000", 2024-02-05T08:11:22.088 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016057500000000001 2024-02-05T08:11:22.088 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.088 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.088 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.088 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.025489+0000", 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036717680000000002 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.029370+0000", 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00388128 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.037825+0000", 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0084547530000000006 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.089 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.037992+0000", 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016755700000000001 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.038030+0000", 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7228000000000003e-05 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106460 7.e 7:7b65a89c:::benchmark_data_smithi134_83016_object106459:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:11:22.090 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.272760+0000", 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 185.73704862, 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14252445999999999, 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106460 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272760+0000", 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.091 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.092 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.092 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272760+0000", 2024-02-05T08:11:22.092 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.092 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.092 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272761+0000", 2024-02-05T08:11:22.092 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3820000000000001e-06 2024-02-05T08:11:22.092 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.092 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.092 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.092 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272786+0000", 2024-02-05T08:11:22.092 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4306000000000001e-05 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272798+0000", 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2347999999999999e-05 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272807+0000", 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6130000000000004e-06 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.093 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.272875+0000", 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8631000000000001e-05 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.273003+0000", 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012771200000000001 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.273168+0000", 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016530000000000001 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.276443+0000", 2024-02-05T08:11:22.094 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032742489999999999 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.278626+0000", 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0021837380000000002 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.415201+0000", 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13657491199999999 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.415250+0000", 2024-02-05T08:11:22.095 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8547000000000001e-05 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.415284+0000", 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4721999999999997e-05 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106678 7.e 7:7f897189:::benchmark_data_smithi134_83016_object106677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.632287+0000", 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 185.37752165399999, 2024-02-05T08:11:22.096 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14517614000000001, 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106678 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632287+0000", 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.097 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632287+0000", 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632288+0000", 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3e-06 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632314+0000", 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5622e-05 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.098 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.099 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632326+0000", 2024-02-05T08:11:22.099 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1901e-05 2024-02-05T08:11:22.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.099 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.099 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632333+0000", 2024-02-05T08:11:22.099 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.08e-06 2024-02-05T08:11:22.099 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.099 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.099 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:22.100 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.100 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633044+0000", 2024-02-05T08:11:22.100 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071079200000000002 2024-02-05T08:11:22.100 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.100 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.100 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.100 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633761+0000", 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071729699999999999 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633976+0000", 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021524099999999999 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650870+0000", 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.016893551 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.101 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650906+0000", 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6427000000000002e-05 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777225+0000", 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.126318758 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777321+0000", 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6436999999999998e-05 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.102 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777463+0000", 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000141734 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106682 7.e 7:7cfa27eb:::benchmark_data_smithi134_83016_object106681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.633782+0000", 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 185.376026725, 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14385479700000001, 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.103 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106682 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633782+0000", 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633782+0000", 2024-02-05T08:11:22.104 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633783+0000", 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1090000000000001e-06 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633813+0000", 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9856999999999999e-05 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633823+0000", 2024-02-05T08:11:22.105 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0222e-05 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633830+0000", 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0639999999999996e-06 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642537+0000", 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0087064080000000005 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.106 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642683+0000", 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000146527 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642858+0000", 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017454499999999999 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650945+0000", 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0080870500000000001 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.107 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.769282+0000", 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.118337331 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777475+0000", 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0081926900000000007 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777545+0000", 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0735000000000006e-05 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.108 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.109 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.109 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777637+0000", 2024-02-05T08:11:22.109 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1259000000000002e-05 2024-02-05T08:11:22.109 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.109 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.109 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.109 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.109 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.109 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106688 7.e 7:758dcfa0:::benchmark_data_smithi134_83016_object106687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:11:22.109 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.642683+0000", 2024-02-05T08:11:22.109 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 185.36712518100001, 2024-02-05T08:11:22.109 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13519677799999999, 2024-02-05T08:11:22.109 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106688 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642683+0000", 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.110 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642683+0000", 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642685+0000", 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.279e-06 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642707+0000", 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2138999999999999e-05 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.111 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642717+0000", 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0385e-05 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642725+0000", 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5660000000000001e-06 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650974+0000", 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.008249269 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.112 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.651086+0000", 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111517 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.651232+0000", 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000146699 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.762167+0000", 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.110934479 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.113 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777649+0000", 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.015482666000000001 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777716+0000", 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6764e-05 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777816+0000", 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010012 2024-02-05T08:11:22.114 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777880+0000", 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3894999999999997e-05 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106691 7.e 7:7474d3c9:::benchmark_data_smithi134_83016_object106690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.647361+0000", 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 185.36244718200001, 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14090623099999999, 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.115 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.116 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.116 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.116 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.116 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106691 2024-02-05T08:11:22.116 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.116 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.116 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.116 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.116 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.647361+0000", 2024-02-05T08:11:22.116 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.116 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.116 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.116 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.647361+0000", 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.647363+0000", 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.336e-06 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.647484+0000", 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012124 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.117 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.647495+0000", 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0502999999999999e-05 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.647501+0000", 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8990000000000004e-06 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.655409+0000", 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0079076919999999992 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.118 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.655529+0000", 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000120269 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.655641+0000", 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111978 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.762224+0000", 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.10658280000000001 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.119 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777656+0000", 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.015431469 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.788030+0000", 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010374089 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.788223+0000", 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019300599999999999 2024-02-05T08:11:22.120 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.788268+0000", 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4950000000000002e-05 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106698 7.e 7:7004ec6e:::benchmark_data_smithi134_83016_object106697:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.650657+0000", 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 185.359151438, 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.140741798, 2024-02-05T08:11:22.121 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106698 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650657+0000", 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.122 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650657+0000", 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650658+0000", 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2810000000000001e-06 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650680+0000", 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1571000000000001e-05 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650689+0000", 2024-02-05T08:11:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3780000000000007e-06 2024-02-05T08:11:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650696+0000", 2024-02-05T08:11:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1430000000000003e-06 2024-02-05T08:11:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.762260+0000", 2024-02-05T08:11:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.11156439999999999 2024-02-05T08:11:22.124 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.762424+0000", 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016437999999999999 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.762591+0000", 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016713700000000001 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.769322+0000", 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0067301870000000003 2024-02-05T08:11:22.125 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777662+0000", 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0083403160000000004 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.791304+0000", 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.013641893 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.791362+0000", 2024-02-05T08:11:22.126 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7669e-05 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.791399+0000", 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7443e-05 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:107150 7.e 7:7956e5e6:::benchmark_data_smithi134_83016_object107149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:17.274871+0000", 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 184.73493780999999, 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13561259000000001, 2024-02-05T08:11:22.127 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:11:22.128 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:11:22.128 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:11:22.128 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:11:22.128 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:11:22.128 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 107150 2024-02-05T08:11:22.128 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.128 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:11:22.128 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.128 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:11:22.128 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274871+0000", 2024-02-05T08:11:22.128 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.128 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274871+0000", 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274873+0000", 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7269999999999999e-06 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274895+0000", 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2886000000000001e-05 2024-02-05T08:11:22.129 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274905+0000", 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1800000000000002e-06 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274911+0000", 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.674e-06 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.274978+0000", 2024-02-05T08:11:22.130 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6403999999999996e-05 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.275118+0000", 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014021699999999999 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.275286+0000", 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016760899999999999 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.278162+0000", 2024-02-05T08:11:22.132 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00287601 2024-02-05T08:11:22.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.132 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:11:22.132 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.278262+0000", 2024-02-05T08:11:22.132 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000100324 2024-02-05T08:11:22.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.132 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:11:22.132 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.410091+0000", 2024-02-05T08:11:22.132 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13182898700000001 2024-02-05T08:11:22.132 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.132 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.410282+0000", 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000191215 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:17.410483+0000", 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020135699999999999 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:22.133 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:11:22.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:22.218 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:22.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:22.320 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:22.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:22.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:22.448 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 13 16 28952 28936 139.09 106.938 0.00655402 0.00718311 2024-02-05T08:11:22.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:22.473 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:22.503 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:22.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:22.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[30086]: 2.0 scrub starts 2024-02-05T08:11:22.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[30086]: 2.0 scrub ok 2024-02-05T08:11:22.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:11:22.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:11:22.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[30086]: 5.2 scrub starts 2024-02-05T08:11:22.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[30086]: 5.2 scrub ok 2024-02-05T08:11:22.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]: dispatch 2024-02-05T08:11:22.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:11:22.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:11:22.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[39278]: 2.0 scrub starts 2024-02-05T08:11:22.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[39278]: 2.0 scrub ok 2024-02-05T08:11:22.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:11:22.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:11:22.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[39278]: 5.2 scrub starts 2024-02-05T08:11:22.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[39278]: 5.2 scrub ok 2024-02-05T08:11:22.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]: dispatch 2024-02-05T08:11:22.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:11:22.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:11:22.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:22 smithi196 ceph-mon[33018]: 2.0 scrub starts 2024-02-05T08:11:22.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:22 smithi196 ceph-mon[33018]: 2.0 scrub ok 2024-02-05T08:11:22.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:22 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:11:22.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:22 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:11:22.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:22 smithi196 ceph-mon[33018]: 5.2 scrub starts 2024-02-05T08:11:22.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:22 smithi196 ceph-mon[33018]: 5.2 scrub ok 2024-02-05T08:11:22.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]: dispatch 2024-02-05T08:11:22.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:11:22.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:11:22.604 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:22.646 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:22.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:22.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:22.787 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:22.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:22.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:22.888 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:22.926 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:22.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:23.028 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:23.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:23.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:23.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:23.161 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:23.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:23.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:23.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:23.287 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:23.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:23.329 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:23.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:23.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:23.448 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 14 16 30607 30591 136.543 103.438 0.00280253 0.00731671 2024-02-05T08:11:23.458 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:23.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:23.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:23 smithi134 ceph-mon[30086]: pgmap v267: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 597 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-02-05T08:11:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]': finished 2024-02-05T08:11:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:23 smithi134 ceph-mon[30086]: osdmap e167: 8 total, 8 up, 7 in 2024-02-05T08:11:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:23 smithi134 ceph-mon[30086]: 3.1b scrub starts 2024-02-05T08:11:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:23 smithi134 ceph-mon[30086]: 3.1b scrub ok 2024-02-05T08:11:23.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:23 smithi134 ceph-mon[39278]: pgmap v267: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 597 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-02-05T08:11:23.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]': finished 2024-02-05T08:11:23.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:23 smithi134 ceph-mon[39278]: osdmap e167: 8 total, 8 up, 7 in 2024-02-05T08:11:23.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:23 smithi134 ceph-mon[39278]: 3.1b scrub starts 2024-02-05T08:11:23.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:23 smithi134 ceph-mon[39278]: 3.1b scrub ok 2024-02-05T08:11:23.560 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:23.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:11:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:11:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:23 smithi196 ceph-mon[33018]: pgmap v267: 148 pgs: 148 active+clean; 1.3 GiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 597 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-02-05T08:11:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 2]}]': finished 2024-02-05T08:11:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:23 smithi196 ceph-mon[33018]: osdmap e167: 8 total, 8 up, 7 in 2024-02-05T08:11:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:23 smithi196 ceph-mon[33018]: 3.1b scrub starts 2024-02-05T08:11:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:23 smithi196 ceph-mon[33018]: 3.1b scrub ok 2024-02-05T08:11:23.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:23.603 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:23.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:23.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:23.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:23.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:23.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:23.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:23.865 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:23.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:23.967 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:24.010 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T08:11:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:24.016 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:24.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:24.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:24.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:11:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:24.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:24.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:24.243 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:24 smithi134 ceph-mon[30086]: 5.b deep-scrub starts 2024-02-05T08:11:24.243 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:24 smithi134 ceph-mon[30086]: 5.b deep-scrub ok 2024-02-05T08:11:24.243 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:24 smithi134 ceph-mon[30086]: osdmap e168: 8 total, 8 up, 7 in 2024-02-05T08:11:24.243 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:24 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:24.243 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:24 smithi134 ceph-mon[30086]: 4.16 scrub starts 2024-02-05T08:11:24.243 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:24 smithi134 ceph-mon[30086]: 4.16 scrub ok 2024-02-05T08:11:24.243 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:24 smithi134 ceph-mon[39278]: 5.b deep-scrub starts 2024-02-05T08:11:24.244 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:24 smithi134 ceph-mon[39278]: 5.b deep-scrub ok 2024-02-05T08:11:24.244 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:24 smithi134 ceph-mon[39278]: osdmap e168: 8 total, 8 up, 7 in 2024-02-05T08:11:24.244 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:24 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:24.244 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:24 smithi134 ceph-mon[39278]: 4.16 scrub starts 2024-02-05T08:11:24.244 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:24 smithi134 ceph-mon[39278]: 4.16 scrub ok 2024-02-05T08:11:24.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:24.273 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:24.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:24.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:24.375 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:24.421 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:24.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:24.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:24.451 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 15 16 32342 32326 134.639 108.438 0.00348908 0.00741977 2024-02-05T08:11:24.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:24.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:24 smithi196 ceph-mon[33018]: 5.b deep-scrub starts 2024-02-05T08:11:24.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:24 smithi196 ceph-mon[33018]: 5.b deep-scrub ok 2024-02-05T08:11:24.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:24 smithi196 ceph-mon[33018]: osdmap e168: 8 total, 8 up, 7 in 2024-02-05T08:11:24.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:24 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:24.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:24 smithi196 ceph-mon[33018]: 4.16 scrub starts 2024-02-05T08:11:24.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:24 smithi196 ceph-mon[33018]: 4.16 scrub ok 2024-02-05T08:11:24.585 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:24.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:24.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:24.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:24.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:24.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:24.817 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:24.860 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:24.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:24.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:25.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:25.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:25.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:25.136 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:25.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:25.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:25.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:25.279 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:25.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:25 smithi196 ceph-mon[33018]: pgmap v270: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 160 MiB/s wr, 2.56k op/s 2024-02-05T08:11:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:25 smithi196 ceph-mon[33018]: 3.0 scrub starts 2024-02-05T08:11:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:25 smithi196 ceph-mon[33018]: 3.0 scrub ok 2024-02-05T08:11:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:25 smithi196 ceph-mon[33018]: 3.12 deep-scrub starts 2024-02-05T08:11:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:25 smithi196 ceph-mon[33018]: 3.12 deep-scrub ok 2024-02-05T08:11:25.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:25.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:25.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:25.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:25.451 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 16 15 34205 34190 133.505 116.5 0.00510822 0.00748476 2024-02-05T08:11:25.496 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:25 smithi134 ceph-mon[30086]: pgmap v270: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 160 MiB/s wr, 2.56k op/s 2024-02-05T08:11:25.496 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:25 smithi134 ceph-mon[30086]: 3.0 scrub starts 2024-02-05T08:11:25.496 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:25 smithi134 ceph-mon[30086]: 3.0 scrub ok 2024-02-05T08:11:25.497 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:25 smithi134 ceph-mon[30086]: 3.12 deep-scrub starts 2024-02-05T08:11:25.497 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:25 smithi134 ceph-mon[30086]: 3.12 deep-scrub ok 2024-02-05T08:11:25.497 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:25 smithi134 ceph-mon[39278]: pgmap v270: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 160 MiB/s wr, 2.56k op/s 2024-02-05T08:11:25.497 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:25 smithi134 ceph-mon[39278]: 3.0 scrub starts 2024-02-05T08:11:25.497 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:25 smithi134 ceph-mon[39278]: 3.0 scrub ok 2024-02-05T08:11:25.497 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:25 smithi134 ceph-mon[39278]: 3.12 deep-scrub starts 2024-02-05T08:11:25.497 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:25 smithi134 ceph-mon[39278]: 3.12 deep-scrub ok 2024-02-05T08:11:25.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:25.590 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:25.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:25.691 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:25.739 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:25.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:25.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:25.841 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:25.879 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:25.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:25.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:25.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:26.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:26.024 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:26.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:26.125 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:26.170 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:26.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:26.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:26.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:26.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:26.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:26.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:26.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:26.441 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:26.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:26.444 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:26 smithi134 ceph-mon[30086]: 5.1e scrub starts 2024-02-05T08:11:26.444 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:26 smithi134 ceph-mon[30086]: 5.1e scrub ok 2024-02-05T08:11:26.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:26 smithi134 ceph-mon[39278]: 5.1e scrub starts 2024-02-05T08:11:26.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:26 smithi134 ceph-mon[39278]: 5.1e scrub ok 2024-02-05T08:11:26.451 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 17 16 35993 35977 132.219 111.688 0.0139132 0.00755572 2024-02-05T08:11:26.508 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:26.518 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:26.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:26.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:26 smithi196 ceph-mon[33018]: 5.1e scrub starts 2024-02-05T08:11:26.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:26 smithi196 ceph-mon[33018]: 5.1e scrub ok 2024-02-05T08:11:26.573 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:26.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:26.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:26.675 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:26.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:26.727 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:26.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:26.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:26.866 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:26.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:26.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:26.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:26.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:27.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:27.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:27.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:27.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:27.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:27.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:27.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:27.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:27.403 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:27.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:27.431 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[30086]: 4.7 scrub starts 2024-02-05T08:11:27.431 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[30086]: 4.7 scrub ok 2024-02-05T08:11:27.431 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[30086]: pgmap v271: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-05T08:11:27.431 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[30086]: 4.1a scrub starts 2024-02-05T08:11:27.431 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[30086]: 4.1a scrub ok 2024-02-05T08:11:27.431 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[30086]: 3.5 scrub starts 2024-02-05T08:11:27.431 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[30086]: 3.5 scrub ok 2024-02-05T08:11:27.432 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[39278]: 4.7 scrub starts 2024-02-05T08:11:27.432 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[39278]: 4.7 scrub ok 2024-02-05T08:11:27.432 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[39278]: pgmap v271: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-05T08:11:27.432 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[39278]: 4.1a scrub starts 2024-02-05T08:11:27.432 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[39278]: 4.1a scrub ok 2024-02-05T08:11:27.432 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[39278]: 3.5 scrub starts 2024-02-05T08:11:27.432 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-mon[39278]: 3.5 scrub ok 2024-02-05T08:11:27.452 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 18 16 37606 37590 130.474 100.812 0.0144105 0.0076572 2024-02-05T08:11:27.505 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:27.533 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:27.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:27 smithi196 ceph-mon[33018]: 4.7 scrub starts 2024-02-05T08:11:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:27 smithi196 ceph-mon[33018]: 4.7 scrub ok 2024-02-05T08:11:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:27 smithi196 ceph-mon[33018]: pgmap v271: 148 pgs: 1 peering, 147 active+clean; 1.8 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-05T08:11:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:27 smithi196 ceph-mon[33018]: 4.1a scrub starts 2024-02-05T08:11:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:27 smithi196 ceph-mon[33018]: 4.1a scrub ok 2024-02-05T08:11:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:27 smithi196 ceph-mon[33018]: 3.5 scrub starts 2024-02-05T08:11:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:27 smithi196 ceph-mon[33018]: 3.5 scrub ok 2024-02-05T08:11:27.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:27.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:27.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:27.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:27.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:27.799 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:27.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:27.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:27.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:27.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:27.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:28.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:11:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:11:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:11:28.031 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:28.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:28.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:28.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:28.159 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:11:28.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:28.188 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:28.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:28.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:28.216 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:11:28.265 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:11:28.285 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:28 smithi134 ceph-mon[30086]: 5.0 deep-scrub starts 2024-02-05T08:11:28.285 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:28 smithi134 ceph-mon[30086]: 5.0 deep-scrub ok 2024-02-05T08:11:28.285 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:28 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2968717388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:28.286 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:28 smithi134 ceph-mon[39278]: 5.0 deep-scrub starts 2024-02-05T08:11:28.286 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:28 smithi134 ceph-mon[39278]: 5.0 deep-scrub ok 2024-02-05T08:11:28.287 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:28 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2968717388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:28.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:28.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:28.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:28.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:28.322 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:11:28.364 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:11:28.406 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:11:28.420 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:28.445 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:11:28.452 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 19 16 39152 39136 128.692 96.625 0.0126493 0.00776232 2024-02-05T08:11:28.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:28.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:28.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:28.494 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:11:28.503 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:11:28.503 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:11:28.503 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:11:28.503 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:11:28.504 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:11:28.504 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:11:28.504 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:11:28.504 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:11:28.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:28 smithi196 ceph-mon[33018]: 5.0 deep-scrub starts 2024-02-05T08:11:28.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:28 smithi196 ceph-mon[33018]: 5.0 deep-scrub ok 2024-02-05T08:11:28.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:28 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2968717388' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:28.565 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:28.611 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:28.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:28.712 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:28.753 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:28.754 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:11:28.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:28.763 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:11:28.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:28.887 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:28.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:28.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:29.021 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:29.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:29.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:29.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:29.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:29.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:29.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:29.288 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:29 smithi134 ceph-mon[30086]: 3.d scrub starts 2024-02-05T08:11:29.288 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:29 smithi134 ceph-mon[30086]: 3.d scrub ok 2024-02-05T08:11:29.288 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:29 smithi134 ceph-mon[30086]: pgmap v272: 148 pgs: 1 active+clean+scrubbing, 1 peering, 146 active+clean; 2.0 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-02-05T08:11:29.288 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:29 smithi134 ceph-mon[30086]: 5.1 scrub starts 2024-02-05T08:11:29.288 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:29 smithi134 ceph-mon[30086]: 5.1 scrub ok 2024-02-05T08:11:29.289 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:29 smithi134 ceph-mon[39278]: 3.d scrub starts 2024-02-05T08:11:29.289 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:29 smithi134 ceph-mon[39278]: 3.d scrub ok 2024-02-05T08:11:29.289 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:29 smithi134 ceph-mon[39278]: pgmap v272: 148 pgs: 1 active+clean+scrubbing, 1 peering, 146 active+clean; 2.0 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-02-05T08:11:29.289 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:29 smithi134 ceph-mon[39278]: 5.1 scrub starts 2024-02-05T08:11:29.289 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:29 smithi134 ceph-mon[39278]: 5.1 scrub ok 2024-02-05T08:11:29.307 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:29.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:29.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:29.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:29.453 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:11:29.453472+0000 min lat: 0.00115834 max lat: 0.147925 avg lat: 0.00785612 2024-02-05T08:11:29.453 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:11:29.453 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 20 16 40733 40717 127.195 98.8125 0.0103163 0.00785612 2024-02-05T08:11:29.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:29.454 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:29.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:29.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:29.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:29 smithi196 ceph-mon[33018]: 3.d scrub starts 2024-02-05T08:11:29.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:29 smithi196 ceph-mon[33018]: 3.d scrub ok 2024-02-05T08:11:29.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:29 smithi196 ceph-mon[33018]: pgmap v272: 148 pgs: 1 active+clean+scrubbing, 1 peering, 146 active+clean; 2.0 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 1.3 KiB/s rd, 95 MiB/s wr, 1.51k op/s 2024-02-05T08:11:29.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:29 smithi196 ceph-mon[33018]: 5.1 scrub starts 2024-02-05T08:11:29.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:29 smithi196 ceph-mon[33018]: 5.1 scrub ok 2024-02-05T08:11:29.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:29.599 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:29.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:29.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:29.730 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:29.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:29.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:29.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:29.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:29.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:30.003 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:30.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:30.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:30.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:30.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:30.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:30.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:30.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:30.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:30.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:30.368 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:30.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:30.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:30.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:30.452 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 21 16 42401 42385 126.102 104.25 0.0103271 0.00792454 2024-02-05T08:11:30.499 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:30.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:30 smithi134 ceph-mon[30086]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:11:30.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:30 smithi134 ceph-mon[30086]: 5.f scrub starts 2024-02-05T08:11:30.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:30 smithi134 ceph-mon[30086]: 5.f scrub ok 2024-02-05T08:11:30.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:30 smithi134 ceph-mon[30086]: 3.2 scrub starts 2024-02-05T08:11:30.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:30 smithi134 ceph-mon[30086]: 3.2 scrub ok 2024-02-05T08:11:30.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:30 smithi134 ceph-mon[39278]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:11:30.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:30 smithi134 ceph-mon[39278]: 5.f scrub starts 2024-02-05T08:11:30.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:30 smithi134 ceph-mon[39278]: 5.f scrub ok 2024-02-05T08:11:30.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:30 smithi134 ceph-mon[39278]: 3.2 scrub starts 2024-02-05T08:11:30.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:30 smithi134 ceph-mon[39278]: 3.2 scrub ok 2024-02-05T08:11:30.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:30.529 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:30.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:30.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:30 smithi196 ceph-mon[33018]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:11:30.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:30 smithi196 ceph-mon[33018]: 5.f scrub starts 2024-02-05T08:11:30.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:30 smithi196 ceph-mon[33018]: 5.f scrub ok 2024-02-05T08:11:30.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:30 smithi196 ceph-mon[33018]: 3.2 scrub starts 2024-02-05T08:11:30.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:30 smithi196 ceph-mon[33018]: 3.2 scrub ok 2024-02-05T08:11:30.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:30.652 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-02-05T08:11:27.617202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2152470876,"num_objects":33033,"num_object_clones":0,"num_object_copies":99099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33033,"num_whiteouts":0,"num_read":1331,"num_read_kb":1177,"num_write":33450,"num_write_kb":2103901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":2363363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33484,"ondisk_log_size":33484,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7275900,"kb_used_data":6139952,"kb_used_omap":10,"kb_used_meta":1135797,"kb_avail":648940164,"statfs":{"total":671965249536,"available":664514727936,"internally_reserved":0,"allocated":6287310848,"data_stored":6284859333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11155,"internal_metadata":1163056237},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,8,4,4,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":747372544,"num_objects":11404,"num_object_clones":0,"num_object_copies":34212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11404,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11404,"num_write_kb":729856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.529194"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":226,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.113305+0000","last_change":"2024-02-05T08:10:26.432413+0000","last_active":"2024-02-05T08:11:23.113305+0000","last_peered":"2024-02-05T08:11:23.113305+0000","last_clean":"2024-02-05T08:11:23.113305+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:11:23.113305+0000","last_undegraded":"2024-02-05T08:11:23.113305+0000","last_fullsized":"2024-02-05T08:11:23.113305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:26.432294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:26.432294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.412548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":237,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.121491+0000","last_change":"2024-02-05T08:11:17.826990+0000","last_active":"2024-02-05T08:11:22.121491+0000","last_peered":"2024-02-05T08:11:22.121491+0000","last_clean":"2024-02-05T08:11:22.121491+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:11:22.121491+0000","last_undegraded":"2024-02-05T08:11:22.121491+0000","last_fullsized":"2024-02-05T08:11:22.121491+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:17.824305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:17.824305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:42.894736+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":237,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.122212+0000","last_change":"2024-02-05T08:10:18.254802+0000","last_active":"2024-02-05T08:11:22.122212+0000","last_peered":"2024-02-05T08:11:22.122212+0000","last_clean":"2024-02-05T08:11:22.122212+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:11:22.122212+0000","last_undegraded":"2024-02-05T08:11:22.122212+0000","last_fullsized":"2024-02-05T08:11:22.122212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.254568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.254568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:46.157955+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"90'13","reported_seq":270,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.112156+0000","last_change":"2024-02-05T08:11:06.482664+0000","last_active":"2024-02-05T08:11:23.112156+0000","last_peered":"2024-02-05T08:11:23.112156+0000","last_clean":"2024-02-05T08:11:23.112156+0000","last_became_active":"2024-02-05T08:06:39.560434+0000","last_became_peered":"2024-02-05T08:06:39.560434+0000","last_unstale":"2024-02-05T08:11:23.112156+0000","last_undegraded":"2024-02-05T08:11:23.112156+0000","last_fullsized":"2024-02-05T08:11:23.112156+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:11:06.482560+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:11:06.482560+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:21.899599+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":230,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.117515+0000","last_change":"2024-02-05T08:09:37.106387+0000","last_active":"2024-02-05T08:11:22.117515+0000","last_peered":"2024-02-05T08:11:22.117515+0000","last_clean":"2024-02-05T08:11:22.117515+0000","last_became_active":"2024-02-05T08:06:39.558790+0000","last_became_peered":"2024-02-05T08:06:39.558790+0000","last_unstale":"2024-02-05T08:11:22.117515+0000","last_undegraded":"2024-02-05T08:11:22.117515+0000","last_fullsized":"2024-02-05T08:11:22.117515+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:37.106147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:37.106147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":236,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.112121+0000","last_change":"2024-02-05T08:11:19.262689+0000","last_active":"2024-02-05T08:11:23.112121+0000","last_peered":"2024-02-05T08:11:23.112121+0000","last_clean":"2024-02-05T08:11:23.112121+0000","last_became_active":"2024-02-05T08:06:39.561731+0000","last_became_peered":"2024-02-05T08:06:39.561731+0000","last_unstale":"2024-02-05T08:11:23.112121+0000","last_undegraded":"2024-02-05T08:11:23.112121+0000","last_fullsized":"2024-02-05T08:11:23.112121+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:11:19.262611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:36.447755+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":249,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.123399+0000","last_change":"2024-02-05T08:11:09.444296+0000","last_active":"2024-02-05T08:11:22.123399+0000","last_peered":"2024-02-05T08:11:22.123399+0000","last_clean":"2024-02-05T08:11:22.123399+0000","last_became_active":"2024-02-05T08:11:09.443912+0000","last_became_peered":"2024-02-05T08:11:09.443912+0000","last_unstale":"2024-02-05T08:11:22.123399+0000","last_undegraded":"2024-02-05T08:11:22.123399+0000","last_fullsized":"2024-02-05T08:11:22.123399+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:28.007713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:28.007713+0000","last_clean_scrub_stamp":"2024-02-05T08:10:28.007713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:28.872467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"118'22","reported_seq":264,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.117157+0000","last_change":"2024-02-05T08:11:07.726805+0000","last_active":"2024-02-05T08:11:22.117157+0000","last_peered":"2024-02-05T08:11:22.117157+0000","last_clean":"2024-02-05T08:11:22.117157+0000","last_became_active":"2024-02-05T08:04:43.560660+0000","last_became_peered":"2024-02-05T08:04:43.560660+0000","last_unstale":"2024-02-05T08:11:22.117157+0000","last_undegraded":"2024-02-05T08:11:22.117157+0000","last_fullsized":"2024-02-05T08:11:22.117157+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-02-05T08:11:07.726635+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.726635+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:17.948247+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":224,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.122790+0000","last_change":"2024-02-05T08:10:29.223232+0000","last_active":"2024-02-05T08:11:22.122790+0000","last_peered":"2024-02-05T08:11:22.122790+0000","last_clean":"2024-02-05T08:11:22.122790+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:11:22.122790+0000","last_undegraded":"2024-02-05T08:11:22.122790+0000","last_fullsized":"2024-02-05T08:11:22.122790+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:29.223145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:29.223145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:55.416210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":230,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.117136+0000","last_change":"2024-02-05T08:10:35.612852+0000","last_active":"2024-02-05T08:11:23.117136+0000","last_peered":"2024-02-05T08:11:23.117136+0000","last_clean":"2024-02-05T08:11:23.117136+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:11:23.117136+0000","last_undegraded":"2024-02-05T08:11:23.117136+0000","last_fullsized":"2024-02-05T08:11:23.117136+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:10:35.612720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:52.008786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":252,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.116872+0000","last_change":"2024-02-05T08:11:20.316629+0000","last_active":"2024-02-05T08:11:23.116872+0000","last_peered":"2024-02-05T08:11:23.116872+0000","last_clean":"2024-02-05T08:11:23.116872+0000","last_became_active":"2024-02-05T08:06:39.559461+0000","last_became_peered":"2024-02-05T08:06:39.559461+0000","last_unstale":"2024-02-05T08:11:23.116872+0000","last_undegraded":"2024-02-05T08:11:23.116872+0000","last_fullsized":"2024-02-05T08:11:23.116872+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:20.316515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:20.316515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:23.278639+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":266,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:26.162366+0000","last_change":"2024-02-05T08:11:26.162414+0000","last_active":"2024-02-05T08:11:26.162366+0000","last_peered":"2024-02-05T08:11:26.162366+0000","last_clean":"2024-02-05T08:11:26.162366+0000","last_became_active":"2024-02-05T08:07:22.653892+0000","last_became_peered":"2024-02-05T08:07:22.653892+0000","last_unstale":"2024-02-05T08:11:26.162366+0000","last_undegraded":"2024-02-05T08:11:26.162366+0000","last_fullsized":"2024-02-05T08:11:26.162366+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:26.162293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:26.162293+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:36.094131+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":226,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.115963+0000","last_change":"2024-02-05T08:10:16.560606+0000","last_active":"2024-02-05T08:11:23.115963+0000","last_peered":"2024-02-05T08:11:23.115963+0000","last_clean":"2024-02-05T08:11:23.115963+0000","last_became_active":"2024-02-05T08:04:47.587515+0000","last_became_peered":"2024-02-05T08:04:47.587515+0000","last_unstale":"2024-02-05T08:11:23.115963+0000","last_undegraded":"2024-02-05T08:11:23.115963+0000","last_fullsized":"2024-02-05T08:11:23.115963+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:16.560531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:16.560531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:46.471717+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":231,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.193715+0000","last_change":"2024-02-05T08:10:24.239442+0000","last_active":"2024-02-05T08:11:23.193715+0000","last_peered":"2024-02-05T08:11:23.193715+0000","last_clean":"2024-02-05T08:11:23.193715+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:11:23.193715+0000","last_undegraded":"2024-02-05T08:11:23.193715+0000","last_fullsized":"2024-02-05T08:11:23.193715+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:24.239334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:24.239334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:53.342841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":238,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.116317+0000","last_change":"2024-02-05T08:10:23.663012+0000","last_active":"2024-02-05T08:11:23.116317+0000","last_peered":"2024-02-05T08:11:23.116317+0000","last_clean":"2024-02-05T08:11:23.116317+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:11:23.116317+0000","last_undegraded":"2024-02-05T08:11:23.116317+0000","last_fullsized":"2024-02-05T08:11:23.116317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:23.662919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:10:23.662919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:40.039778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":236,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.112205+0000","last_change":"2024-02-05T08:09:30.448604+0000","last_active":"2024-02-05T08:11:23.112205+0000","last_peered":"2024-02-05T08:11:23.112205+0000","last_clean":"2024-02-05T08:11:23.112205+0000","last_became_active":"2024-02-05T08:04:43.557559+0000","last_became_peered":"2024-02-05T08:04:43.557559+0000","last_unstale":"2024-02-05T08:11:23.112205+0000","last_undegraded":"2024-02-05T08:11:23.112205+0000","last_fullsized":"2024-02-05T08:11:23.112205+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:09:30.448493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:30.448493+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":233,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.113081+0000","last_change":"2024-02-05T08:09:26.466381+0000","last_active":"2024-02-05T08:11:23.113081+0000","last_peered":"2024-02-05T08:11:23.113081+0000","last_clean":"2024-02-05T08:11:23.113081+0000","last_became_active":"2024-02-05T08:06:39.547906+0000","last_became_peered":"2024-02-05T08:06:39.547906+0000","last_unstale":"2024-02-05T08:11:23.113081+0000","last_undegraded":"2024-02-05T08:11:23.113081+0000","last_fullsized":"2024-02-05T08:11:23.113081+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:26.466302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:09:26.466302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":229,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.122720+0000","last_change":"2024-02-05T08:11:06.715512+0000","last_active":"2024-02-05T08:11:22.122720+0000","last_peered":"2024-02-05T08:11:22.122720+0000","last_clean":"2024-02-05T08:11:22.122720+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:11:22.122720+0000","last_undegraded":"2024-02-05T08:11:22.122720+0000","last_fullsized":"2024-02-05T08:11:22.122720+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:06.715401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:11:06.715401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:28.763706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":236,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.303437+0000","last_change":"2024-02-05T08:11:22.303481+0000","last_active":"2024-02-05T08:11:22.303437+0000","last_peered":"2024-02-05T08:11:22.303437+0000","last_clean":"2024-02-05T08:11:22.303437+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:11:22.303437+0000","last_undegraded":"2024-02-05T08:11:22.303437+0000","last_fullsized":"2024-02-05T08:11:22.303437+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:22.303363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:22.303363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:45.613255+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":266,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.129382+0000","last_change":"2024-02-05T08:10:19.645020+0000","last_active":"2024-02-05T08:11:22.129382+0000","last_peered":"2024-02-05T08:11:22.129382+0000","last_clean":"2024-02-05T08:11:22.129382+0000","last_became_active":"2024-02-05T08:04:43.557496+0000","last_became_peered":"2024-02-05T08:04:43.557496+0000","last_unstale":"2024-02-05T08:11:22.129382+0000","last_undegraded":"2024-02-05T08:11:22.129382+0000","last_fullsized":"2024-02-05T08:11:22.129382+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:10:19.644900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:19.644900+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:31.573739+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":223,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.124139+0000","last_change":"2024-02-05T08:09:21.354430+0000","last_active":"2024-02-05T08:11:22.124139+0000","last_peered":"2024-02-05T08:11:22.124139+0000","last_clean":"2024-02-05T08:11:22.124139+0000","last_became_active":"2024-02-05T08:06:39.560948+0000","last_became_peered":"2024-02-05T08:06:39.560948+0000","last_unstale":"2024-02-05T08:11:22.124139+0000","last_undegraded":"2024-02-05T08:11:22.124139+0000","last_fullsized":"2024-02-05T08:11:22.124139+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:21.354158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.354158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":230,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.112599+0000","last_change":"2024-02-05T08:10:30.419053+0000","last_active":"2024-02-05T08:11:23.112599+0000","last_peered":"2024-02-05T08:11:23.112599+0000","last_clean":"2024-02-05T08:11:23.112599+0000","last_became_active":"2024-02-05T08:04:45.563948+0000","last_became_peered":"2024-02-05T08:04:45.563948+0000","last_unstale":"2024-02-05T08:11:23.112599+0000","last_undegraded":"2024-02-05T08:11:23.112599+0000","last_fullsized":"2024-02-05T08:11:23.112599+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:30.418931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.418931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:51.770800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":230,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.113643+0000","last_change":"2024-02-05T08:09:27.431319+0000","last_active":"2024-02-05T08:11:23.113643+0000","last_peered":"2024-02-05T08:11:23.113643+0000","last_clean":"2024-02-05T08:11:23.113643+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:11:23.113643+0000","last_undegraded":"2024-02-05T08:11:23.113643+0000","last_fullsized":"2024-02-05T08:11:23.113643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:27.431221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:27.431221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"90'17","reported_seq":260,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.119122+0000","last_change":"2024-02-05T08:10:30.977423+0000","last_active":"2024-02-05T08:11:22.119122+0000","last_peered":"2024-02-05T08:11:22.119122+0000","last_clean":"2024-02-05T08:11:22.119122+0000","last_became_active":"2024-02-05T08:04:43.557596+0000","last_became_peered":"2024-02-05T08:04:43.557596+0000","last_unstale":"2024-02-05T08:11:22.119122+0000","last_undegraded":"2024-02-05T08:11:22.119122+0000","last_fullsized":"2024-02-05T08:11:22.119122+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:10:30.977334+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.977334+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:45.611556+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":237,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.194023+0000","last_change":"2024-02-05T08:11:10.429120+0000","last_active":"2024-02-05T08:11:23.194023+0000","last_peered":"2024-02-05T08:11:23.194023+0000","last_clean":"2024-02-05T08:11:23.194023+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:11:23.194023+0000","last_undegraded":"2024-02-05T08:11:23.194023+0000","last_fullsized":"2024-02-05T08:11:23.194023+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:11:10.428967+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:11:10.428967+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:20.943263+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":263,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.113386+0000","last_change":"2024-02-05T08:11:21.208333+0000","last_active":"2024-02-05T08:11:23.113386+0000","last_peered":"2024-02-05T08:11:23.113386+0000","last_clean":"2024-02-05T08:11:23.113386+0000","last_became_active":"2024-02-05T08:06:39.552760+0000","last_became_peered":"2024-02-05T08:06:39.552760+0000","last_unstale":"2024-02-05T08:11:23.113386+0000","last_undegraded":"2024-02-05T08:11:23.113386+0000","last_fullsized":"2024-02-05T08:11:23.113386+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:11:21.208206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:21.208206+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:23.267303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":238,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.121395+0000","last_change":"2024-02-05T08:10:32.403469+0000","last_active":"2024-02-05T08:11:22.121395+0000","last_peered":"2024-02-05T08:11:22.121395+0000","last_clean":"2024-02-05T08:11:22.121395+0000","last_became_active":"2024-02-05T08:04:41.532339+0000","last_became_peered":"2024-02-05T08:04:41.532339+0000","last_unstale":"2024-02-05T08:11:22.121395+0000","last_undegraded":"2024-02-05T08:11:22.121395+0000","last_fullsized":"2024-02-05T08:11:22.121395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:10:32.403342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:32.403342+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:58.710603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":240,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.127382+0000","last_change":"2024-02-05T08:09:24.000902+0000","last_active":"2024-02-05T08:11:22.127382+0000","last_peered":"2024-02-05T08:11:22.127382+0000","last_clean":"2024-02-05T08:11:22.127382+0000","last_became_active":"2024-02-05T08:04:43.605952+0000","last_became_peered":"2024-02-05T08:04:43.605952+0000","last_unstale":"2024-02-05T08:11:22.127382+0000","last_undegraded":"2024-02-05T08:11:22.127382+0000","last_fullsized":"2024-02-05T08:11:22.127382+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:09:24.000675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:24.000675+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":225,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.122851+0000","last_change":"2024-02-05T08:09:25.266619+0000","last_active":"2024-02-05T08:11:22.122851+0000","last_peered":"2024-02-05T08:11:22.122851+0000","last_clean":"2024-02-05T08:11:22.122851+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:11:22.122851+0000","last_undegraded":"2024-02-05T08:11:22.122851+0000","last_fullsized":"2024-02-05T08:11:22.122851+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.266508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.266508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":221,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.127633+0000","last_change":"2024-02-05T08:08:42.676027+0000","last_active":"2024-02-05T08:11:22.127633+0000","last_peered":"2024-02-05T08:11:22.127633+0000","last_clean":"2024-02-05T08:11:22.127633+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:11:22.127633+0000","last_undegraded":"2024-02-05T08:11:22.127633+0000","last_fullsized":"2024-02-05T08:11:22.127633+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:42.675782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:08:42.675782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":236,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.117137+0000","last_change":"2024-02-05T08:11:15.831955+0000","last_active":"2024-02-05T08:11:22.117137+0000","last_peered":"2024-02-05T08:11:22.117137+0000","last_clean":"2024-02-05T08:11:22.117137+0000","last_became_active":"2024-02-05T08:04:41.534448+0000","last_became_peered":"2024-02-05T08:04:41.534448+0000","last_unstale":"2024-02-05T08:11:22.117137+0000","last_undegraded":"2024-02-05T08:11:22.117137+0000","last_fullsized":"2024-02-05T08:11:22.117137+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:15.831837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.831837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:27.579523+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":228,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.116845+0000","last_change":"2024-02-05T08:11:15.397484+0000","last_active":"2024-02-05T08:11:23.116845+0000","last_peered":"2024-02-05T08:11:23.116845+0000","last_clean":"2024-02-05T08:11:23.116845+0000","last_became_active":"2024-02-05T08:06:39.558747+0000","last_became_peered":"2024-02-05T08:06:39.558747+0000","last_unstale":"2024-02-05T08:11:23.116845+0000","last_undegraded":"2024-02-05T08:11:23.116845+0000","last_fullsized":"2024-02-05T08:11:23.116845+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:11:15.397335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.397335+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:27.975791+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":227,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.193707+0000","last_change":"2024-02-05T08:11:01.465783+0000","last_active":"2024-02-05T08:11:23.193707+0000","last_peered":"2024-02-05T08:11:23.193707+0000","last_clean":"2024-02-05T08:11:23.193707+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:11:23.193707+0000","last_undegraded":"2024-02-05T08:11:23.193707+0000","last_fullsized":"2024-02-05T08:11:23.193707+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:01.465657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:01.465657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:26.749290+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":170,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.111881+0000","last_change":"2024-02-05T08:09:36.434985+0000","last_active":"2024-02-05T08:11:23.111881+0000","last_peered":"2024-02-05T08:11:23.111881+0000","last_clean":"2024-02-05T08:11:23.111881+0000","last_became_active":"2024-02-05T08:06:39.551173+0000","last_became_peered":"2024-02-05T08:06:39.551173+0000","last_unstale":"2024-02-05T08:11:23.111881+0000","last_undegraded":"2024-02-05T08:11:23.111881+0000","last_fullsized":"2024-02-05T08:11:23.111881+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:36.434873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.434873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":236,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.122436+0000","last_change":"2024-02-05T08:11:20.295028+0000","last_active":"2024-02-05T08:11:22.122436+0000","last_peered":"2024-02-05T08:11:22.122436+0000","last_clean":"2024-02-05T08:11:22.122436+0000","last_became_active":"2024-02-05T08:04:41.536019+0000","last_became_peered":"2024-02-05T08:04:41.536019+0000","last_unstale":"2024-02-05T08:11:22.122436+0000","last_undegraded":"2024-02-05T08:11:22.122436+0000","last_fullsized":"2024-02-05T08:11:22.122436+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:20.294901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:20.294901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:38.830576+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":242,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.117304+0000","last_change":"2024-02-05T08:08:40.082709+0000","last_active":"2024-02-05T08:11:22.117304+0000","last_peered":"2024-02-05T08:11:22.117304+0000","last_clean":"2024-02-05T08:11:22.117304+0000","last_became_active":"2024-02-05T08:06:39.560932+0000","last_became_peered":"2024-02-05T08:06:39.560932+0000","last_unstale":"2024-02-05T08:11:22.117304+0000","last_undegraded":"2024-02-05T08:11:22.117304+0000","last_fullsized":"2024-02-05T08:11:22.117304+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-02-05T08:08:40.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:40.082577+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":230,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.117692+0000","last_change":"2024-02-05T08:10:18.022780+0000","last_active":"2024-02-05T08:11:22.117692+0000","last_peered":"2024-02-05T08:11:22.117692+0000","last_clean":"2024-02-05T08:11:22.117692+0000","last_became_active":"2024-02-05T08:06:39.559212+0000","last_became_peered":"2024-02-05T08:06:39.559212+0000","last_unstale":"2024-02-05T08:11:22.117692+0000","last_undegraded":"2024-02-05T08:11:22.117692+0000","last_fullsized":"2024-02-05T08:11:22.117692+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.022670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.022670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:22.583988+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":245,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.194352+0000","last_change":"2024-02-05T08:11:16.440695+0000","last_active":"2024-02-05T08:11:23.194352+0000","last_peered":"2024-02-05T08:11:23.194352+0000","last_clean":"2024-02-05T08:11:23.194352+0000","last_became_active":"2024-02-05T08:06:39.555597+0000","last_became_peered":"2024-02-05T08:06:39.555597+0000","last_unstale":"2024-02-05T08:11:23.194352+0000","last_undegraded":"2024-02-05T08:11:23.194352+0000","last_fullsized":"2024-02-05T08:11:23.194352+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:16.440585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:11:16.440585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:30.362929+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":246,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.116732+0000","last_change":"2024-02-05T08:11:05.463231+0000","last_active":"2024-02-05T08:11:23.116732+0000","last_peered":"2024-02-05T08:11:23.116732+0000","last_clean":"2024-02-05T08:11:23.116732+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:11:23.116732+0000","last_undegraded":"2024-02-05T08:11:23.116732+0000","last_fullsized":"2024-02-05T08:11:23.116732+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:05.463134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:05.463134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:10.311835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"141'16","reported_seq":274,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.117060+0000","last_change":"2024-02-05T08:09:39.113079+0000","last_active":"2024-02-05T08:11:22.117060+0000","last_peered":"2024-02-05T08:11:22.117060+0000","last_clean":"2024-02-05T08:11:22.117060+0000","last_became_active":"2024-02-05T08:06:39.556873+0000","last_became_peered":"2024-02-05T08:06:39.556873+0000","last_unstale":"2024-02-05T08:11:22.117060+0000","last_undegraded":"2024-02-05T08:11:22.117060+0000","last_fullsized":"2024-02-05T08:11:22.117060+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'16","last_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.112936+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":224,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.121884+0000","last_change":"2024-02-05T08:11:07.248032+0000","last_active":"2024-02-05T08:11:22.121884+0000","last_peered":"2024-02-05T08:11:22.121884+0000","last_clean":"2024-02-05T08:11:22.121884+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:11:22.121884+0000","last_undegraded":"2024-02-05T08:11:22.121884+0000","last_fullsized":"2024-02-05T08:11:22.121884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:07.247928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.247928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:12.667100+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":228,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.117440+0000","last_change":"2024-02-05T08:10:26.987843+0000","last_active":"2024-02-05T08:11:22.117440+0000","last_peered":"2024-02-05T08:11:22.117440+0000","last_clean":"2024-02-05T08:11:22.117440+0000","last_became_active":"2024-02-05T08:04:45.568484+0000","last_became_peered":"2024-02-05T08:04:45.568484+0000","last_unstale":"2024-02-05T08:11:22.117440+0000","last_undegraded":"2024-02-05T08:11:22.117440+0000","last_fullsized":"2024-02-05T08:11:22.117440+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:26.987763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:26.987763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:37.413701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":239,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.194924+0000","last_change":"2024-02-05T08:10:31.270781+0000","last_active":"2024-02-05T08:11:23.194924+0000","last_peered":"2024-02-05T08:11:23.194924+0000","last_clean":"2024-02-05T08:11:23.194924+0000","last_became_active":"2024-02-05T08:04:41.539698+0000","last_became_peered":"2024-02-05T08:04:41.539698+0000","last_unstale":"2024-02-05T08:11:23.194924+0000","last_undegraded":"2024-02-05T08:11:23.194924+0000","last_fullsized":"2024-02-05T08:11:23.194924+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:31.270686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:31.270686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:35.517948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":273,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.120164+0000","last_change":"2024-02-05T08:11:13.794030+0000","last_active":"2024-02-05T08:11:22.120164+0000","last_peered":"2024-02-05T08:11:22.120164+0000","last_clean":"2024-02-05T08:11:22.120164+0000","last_became_active":"2024-02-05T08:06:39.560259+0000","last_became_peered":"2024-02-05T08:06:39.560259+0000","last_unstale":"2024-02-05T08:11:22.120164+0000","last_undegraded":"2024-02-05T08:11:22.120164+0000","last_fullsized":"2024-02-05T08:11:22.120164+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:13.793895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:13.793895+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:27.120443+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":216,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.116773+0000","last_change":"2024-02-05T08:09:29.262843+0000","last_active":"2024-02-05T08:11:22.116773+0000","last_peered":"2024-02-05T08:11:22.116773+0000","last_clean":"2024-02-05T08:11:22.116773+0000","last_became_active":"2024-02-05T08:04:47.585496+0000","last_became_peered":"2024-02-05T08:04:47.585496+0000","last_unstale":"2024-02-05T08:11:22.116773+0000","last_undegraded":"2024-02-05T08:11:22.116773+0000","last_fullsized":"2024-02-05T08:11:22.116773+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:09:29.262582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":229,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.119981+0000","last_change":"2024-02-05T08:10:25.493102+0000","last_active":"2024-02-05T08:11:22.119981+0000","last_peered":"2024-02-05T08:11:22.119981+0000","last_clean":"2024-02-05T08:11:22.119981+0000","last_became_active":"2024-02-05T08:04:45.560815+0000","last_became_peered":"2024-02-05T08:04:45.560815+0000","last_unstale":"2024-02-05T08:11:22.119981+0000","last_undegraded":"2024-02-05T08:11:22.119981+0000","last_fullsized":"2024-02-05T08:11:22.119981+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:25.493013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:25.493013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:33.003654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":244,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.112437+0000","last_change":"2024-02-05T08:09:25.486212+0000","last_active":"2024-02-05T08:11:23.112437+0000","last_peered":"2024-02-05T08:11:23.112437+0000","last_clean":"2024-02-05T08:11:23.112437+0000","last_became_active":"2024-02-05T08:06:39.545576+0000","last_became_peered":"2024-02-05T08:06:39.545576+0000","last_unstale":"2024-02-05T08:11:23.112437+0000","last_undegraded":"2024-02-05T08:11:23.112437+0000","last_fullsized":"2024-02-05T08:11:23.112437+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.486141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.966886+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.486141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":271,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.112372+0000","last_change":"2024-02-05T08:10:32.387139+0000","last_active":"2024-02-05T08:11:23.112372+0000","last_peered":"2024-02-05T08:11:23.112372+0000","last_clean":"2024-02-05T08:11:23.112372+0000","last_became_active":"2024-02-05T08:06:39.558257+0000","last_became_peered":"2024-02-05T08:06:39.558257+0000","last_unstale":"2024-02-05T08:11:23.112372+0000","last_undegraded":"2024-02-05T08:11:23.112372+0000","last_fullsized":"2024-02-05T08:11:23.112372+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:10:32.387029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:32.387029+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.428808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":216,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.126132+0000","last_change":"2024-02-05T08:08:36.679472+0000","last_active":"2024-02-05T08:11:22.126132+0000","last_peered":"2024-02-05T08:11:22.126132+0000","last_clean":"2024-02-05T08:11:22.126132+0000","last_became_active":"2024-02-05T08:04:47.584235+0000","last_became_peered":"2024-02-05T08:04:47.584235+0000","last_unstale":"2024-02-05T08:11:22.126132+0000","last_undegraded":"2024-02-05T08:11:22.126132+0000","last_fullsized":"2024-02-05T08:11:22.126132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:36.679368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:36.679368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":320,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.117089+0000","last_change":"2024-02-05T08:10:14.583618+0000","last_active":"2024-02-05T08:11:23.117089+0000","last_peered":"2024-02-05T08:11:23.117089+0000","last_clean":"2024-02-05T08:11:23.117089+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:11:23.117089+0000","last_undegraded":"2024-02-05T08:11:23.117089+0000","last_fullsized":"2024-02-05T08:11:23.117089+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:10:14.583489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:14.583489+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:28.694742+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":247,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.193928+0000","last_change":"2024-02-05T08:11:11.422476+0000","last_active":"2024-02-05T08:11:23.193928+0000","last_peered":"2024-02-05T08:11:23.193928+0000","last_clean":"2024-02-05T08:11:23.193928+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:11:23.193928+0000","last_undegraded":"2024-02-05T08:11:23.193928+0000","last_fullsized":"2024-02-05T08:11:23.193928+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:11.422371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:11.422371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:36.238335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"129'15","reported_seq":244,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.118158+0000","last_change":"2024-02-05T08:08:37.128945+0000","last_active":"2024-02-05T08:11:22.118158+0000","last_peered":"2024-02-05T08:11:22.118158+0000","last_clean":"2024-02-05T08:11:22.118158+0000","last_became_active":"2024-02-05T08:04:43.605981+0000","last_became_peered":"2024-02-05T08:04:43.605981+0000","last_unstale":"2024-02-05T08:11:22.118158+0000","last_undegraded":"2024-02-05T08:11:22.118158+0000","last_fullsized":"2024-02-05T08:11:22.118158+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'15","last_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:08:37.128817+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":218,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.113190+0000","last_change":"2024-02-05T08:09:18.397753+0000","last_active":"2024-02-05T08:11:23.113190+0000","last_peered":"2024-02-05T08:11:23.113190+0000","last_clean":"2024-02-05T08:11:23.113190+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:11:23.113190+0000","last_undegraded":"2024-02-05T08:11:23.113190+0000","last_fullsized":"2024-02-05T08:11:23.113190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:18.397684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:18.397684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":317,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.118537+0000","last_change":"2024-02-05T08:10:33.947077+0000","last_active":"2024-02-05T08:11:22.118537+0000","last_peered":"2024-02-05T08:11:22.118537+0000","last_clean":"2024-02-05T08:11:22.118537+0000","last_became_active":"2024-02-05T08:04:45.565240+0000","last_became_peered":"2024-02-05T08:04:45.565240+0000","last_unstale":"2024-02-05T08:11:22.118537+0000","last_undegraded":"2024-02-05T08:11:22.118537+0000","last_fullsized":"2024-02-05T08:11:22.118537+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:10:33.946950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:33.946950+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:36.197109+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":228,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.126967+0000","last_change":"2024-02-05T08:08:44.680464+0000","last_active":"2024-02-05T08:11:22.126967+0000","last_peered":"2024-02-05T08:11:22.126967+0000","last_clean":"2024-02-05T08:11:22.126967+0000","last_became_active":"2024-02-05T08:04:41.540252+0000","last_became_peered":"2024-02-05T08:04:41.540252+0000","last_unstale":"2024-02-05T08:11:22.126967+0000","last_undegraded":"2024-02-05T08:11:22.126967+0000","last_fullsized":"2024-02-05T08:11:22.126967+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:44.680353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:44.680353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":258,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.117292+0000","last_change":"2024-02-05T08:11:19.287698+0000","last_active":"2024-02-05T08:11:23.117292+0000","last_peered":"2024-02-05T08:11:23.117292+0000","last_clean":"2024-02-05T08:11:23.117292+0000","last_became_active":"2024-02-05T08:04:43.606414+0000","last_became_peered":"2024-02-05T08:04:43.606414+0000","last_unstale":"2024-02-05T08:11:23.117292+0000","last_undegraded":"2024-02-05T08:11:23.117292+0000","last_fullsized":"2024-02-05T08:11:23.117292+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:19.287569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:19.287569+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:29.462060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":233,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.129688+0000","last_change":"2024-02-05T08:10:20.671244+0000","last_active":"2024-02-05T08:11:22.129688+0000","last_peered":"2024-02-05T08:11:22.129688+0000","last_clean":"2024-02-05T08:11:22.129688+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:11:22.129688+0000","last_undegraded":"2024-02-05T08:11:22.129688+0000","last_fullsized":"2024-02-05T08:11:22.129688+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:20.671130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:20.671130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:39.009177+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":228,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.124081+0000","last_change":"2024-02-05T08:10:15.994743+0000","last_active":"2024-02-05T08:11:22.124081+0000","last_peered":"2024-02-05T08:11:22.124081+0000","last_clean":"2024-02-05T08:11:22.124081+0000","last_became_active":"2024-02-05T08:04:45.567622+0000","last_became_peered":"2024-02-05T08:04:45.567622+0000","last_unstale":"2024-02-05T08:11:22.124081+0000","last_undegraded":"2024-02-05T08:11:22.124081+0000","last_fullsized":"2024-02-05T08:11:22.124081+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:15.994621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:15.994621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:33.060659+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":245,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.193707+0000","last_change":"2024-02-05T08:11:07.416059+0000","last_active":"2024-02-05T08:11:23.193707+0000","last_peered":"2024-02-05T08:11:23.193707+0000","last_clean":"2024-02-05T08:11:23.193707+0000","last_became_active":"2024-02-05T08:06:39.557018+0000","last_became_peered":"2024-02-05T08:06:39.557018+0000","last_unstale":"2024-02-05T08:11:23.193707+0000","last_undegraded":"2024-02-05T08:11:23.193707+0000","last_fullsized":"2024-02-05T08:11:23.193707+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:07.415963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.415963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:28.508089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":240,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.127475+0000","last_change":"2024-02-05T08:08:45.712868+0000","last_active":"2024-02-05T08:11:22.127475+0000","last_peered":"2024-02-05T08:11:22.127475+0000","last_clean":"2024-02-05T08:11:22.127475+0000","last_became_active":"2024-02-05T08:04:43.608489+0000","last_became_peered":"2024-02-05T08:04:43.608489+0000","last_unstale":"2024-02-05T08:11:22.127475+0000","last_undegraded":"2024-02-05T08:11:22.127475+0000","last_fullsized":"2024-02-05T08:11:22.127475+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:08:45.712396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:08:45.712396+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":241,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.194823+0000","last_change":"2024-02-05T08:11:18.424459+0000","last_active":"2024-02-05T08:11:23.194823+0000","last_peered":"2024-02-05T08:11:23.194823+0000","last_clean":"2024-02-05T08:11:23.194823+0000","last_became_active":"2024-02-05T08:06:39.548837+0000","last_became_peered":"2024-02-05T08:06:39.548837+0000","last_unstale":"2024-02-05T08:11:23.194823+0000","last_undegraded":"2024-02-05T08:11:23.194823+0000","last_fullsized":"2024-02-05T08:11:23.194823+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:18.424341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:18.424341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:34.523873+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":230,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.117042+0000","last_change":"2024-02-05T08:10:31.622795+0000","last_active":"2024-02-05T08:11:23.117042+0000","last_peered":"2024-02-05T08:11:23.117042+0000","last_clean":"2024-02-05T08:11:23.117042+0000","last_became_active":"2024-02-05T08:04:45.560241+0000","last_became_peered":"2024-02-05T08:04:45.560241+0000","last_unstale":"2024-02-05T08:11:23.117042+0000","last_undegraded":"2024-02-05T08:11:23.117042+0000","last_fullsized":"2024-02-05T08:11:23.117042+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:10:31.622683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:58.356161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":168,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.123027+0000","last_change":"2024-02-05T08:11:01.330524+0000","last_active":"2024-02-05T08:11:22.123027+0000","last_peered":"2024-02-05T08:11:22.123027+0000","last_clean":"2024-02-05T08:11:22.123027+0000","last_became_active":"2024-02-05T08:06:39.554673+0000","last_became_peered":"2024-02-05T08:06:39.554673+0000","last_unstale":"2024-02-05T08:11:22.123027+0000","last_undegraded":"2024-02-05T08:11:22.123027+0000","last_fullsized":"2024-02-05T08:11:22.123027+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:01.330449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:01.330449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:25.711003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"116'9","reported_seq":244,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.124167+0000","last_change":"2024-02-05T08:10:32.930947+0000","last_active":"2024-02-05T08:11:22.124167+0000","last_peered":"2024-02-05T08:11:22.124167+0000","last_clean":"2024-02-05T08:11:22.124167+0000","last_became_active":"2024-02-05T08:04:43.606624+0000","last_became_peered":"2024-02-05T08:04:43.606624+0000","last_unstale":"2024-02-05T08:11:22.124167+0000","last_undegraded":"2024-02-05T08:11:22.124167+0000","last_fullsized":"2024-02-05T08:11:22.124167+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-02-05T08:10:32.930856+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:10:32.930856+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:36.345351+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"168'2343","reported_seq":2375,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:27.542460+0000","last_change":"2024-02-05T08:10:54.986795+0000","last_active":"2024-02-05T08:11:27.542460+0000","last_peered":"2024-02-05T08:11:27.542460+0000","last_clean":"2024-02-05T08:11:27.542460+0000","last_became_active":"2024-02-05T08:10:54.986264+0000","last_became_peered":"2024-02-05T08:10:54.986264+0000","last_unstale":"2024-02-05T08:11:27.542460+0000","last_undegraded":"2024-02-05T08:11:27.542460+0000","last_fullsized":"2024-02-05T08:11:27.542460+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":2343,"log_dups_size":0,"ondisk_log_size":2343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:02.161612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153550848,"num_objects":2343,"num_object_clones":0,"num_object_copies":7029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2343,"num_write_kb":149952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":241,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.117095+0000","last_change":"2024-02-05T08:11:13.452209+0000","last_active":"2024-02-05T08:11:23.117095+0000","last_peered":"2024-02-05T08:11:23.117095+0000","last_clean":"2024-02-05T08:11:23.117095+0000","last_became_active":"2024-02-05T08:06:39.551759+0000","last_became_peered":"2024-02-05T08:06:39.551759+0000","last_unstale":"2024-02-05T08:11:23.117095+0000","last_undegraded":"2024-02-05T08:11:23.117095+0000","last_fullsized":"2024-02-05T08:11:23.117095+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:13.452127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:13.452127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:39.634517+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":251,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.194342+0000","last_change":"2024-02-05T08:10:28.309205+0000","last_active":"2024-02-05T08:11:23.194342+0000","last_peered":"2024-02-05T08:11:23.194342+0000","last_clean":"2024-02-05T08:11:23.194342+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:11:23.194342+0000","last_undegraded":"2024-02-05T08:11:23.194342+0000","last_fullsized":"2024-02-05T08:11:23.194342+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:10:28.309092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:28.309092+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:44.852371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":282,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.116221+0000","last_change":"2024-02-05T08:11:01.520516+0000","last_active":"2024-02-05T08:11:23.116221+0000","last_peered":"2024-02-05T08:11:23.116221+0000","last_clean":"2024-02-05T08:11:23.116221+0000","last_became_active":"2024-02-05T08:06:39.546607+0000","last_became_peered":"2024-02-05T08:06:39.546607+0000","last_unstale":"2024-02-05T08:11:23.116221+0000","last_undegraded":"2024-02-05T08:11:23.116221+0000","last_fullsized":"2024-02-05T08:11:23.116221+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:11:01.520393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:01.520393+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:04.614465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":340,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.193998+0000","last_change":"2024-02-05T08:11:14.422575+0000","last_active":"2024-02-05T08:11:23.193998+0000","last_peered":"2024-02-05T08:11:23.193998+0000","last_clean":"2024-02-05T08:11:23.193998+0000","last_became_active":"2024-02-05T08:06:39.556562+0000","last_became_peered":"2024-02-05T08:06:39.556562+0000","last_unstale":"2024-02-05T08:11:23.193998+0000","last_undegraded":"2024-02-05T08:11:23.193998+0000","last_fullsized":"2024-02-05T08:11:23.193998+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:11:14.422445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:14.422445+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:39.731435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"168'2426","reported_seq":2458,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:27.523278+0000","last_change":"2024-02-05T08:10:54.985646+0000","last_active":"2024-02-05T08:11:27.523278+0000","last_peered":"2024-02-05T08:11:27.523278+0000","last_clean":"2024-02-05T08:11:27.523278+0000","last_became_active":"2024-02-05T08:10:54.985285+0000","last_became_peered":"2024-02-05T08:10:54.985285+0000","last_unstale":"2024-02-05T08:11:27.523278+0000","last_undegraded":"2024-02-05T08:11:27.523278+0000","last_fullsized":"2024-02-05T08:11:27.523278+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":2426,"log_dups_size":0,"ondisk_log_size":2426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:00.726715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158990336,"num_objects":2426,"num_object_clones":0,"num_object_copies":7278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2426,"num_write_kb":155264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":216,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.117606+0000","last_change":"2024-02-05T08:09:25.266619+0000","last_active":"2024-02-05T08:11:22.117606+0000","last_peered":"2024-02-05T08:11:22.117606+0000","last_clean":"2024-02-05T08:11:22.117606+0000","last_became_active":"2024-02-05T08:04:47.586044+0000","last_became_peered":"2024-02-05T08:04:47.586044+0000","last_unstale":"2024-02-05T08:11:22.117606+0000","last_undegraded":"2024-02-05T08:11:22.117606+0000","last_fullsized":"2024-02-05T08:11:22.117606+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.266513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.266513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":240,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.194470+0000","last_change":"2024-02-05T08:09:39.244280+0000","last_active":"2024-02-05T08:11:23.194470+0000","last_peered":"2024-02-05T08:11:23.194470+0000","last_clean":"2024-02-05T08:11:23.194470+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:11:23.194470+0000","last_undegraded":"2024-02-05T08:11:23.194470+0000","last_fullsized":"2024-02-05T08:11:23.194470+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:39.244179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.244179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":250,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.112426+0000","last_change":"2024-02-05T08:09:31.469380+0000","last_active":"2024-02-05T08:11:23.112426+0000","last_peered":"2024-02-05T08:11:23.112426+0000","last_clean":"2024-02-05T08:11:23.112426+0000","last_became_active":"2024-02-05T08:04:43.556580+0000","last_became_peered":"2024-02-05T08:04:43.556580+0000","last_unstale":"2024-02-05T08:11:23.112426+0000","last_undegraded":"2024-02-05T08:11:23.112426+0000","last_fullsized":"2024-02-05T08:11:23.112426+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:09:31.469295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:31.469295+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":238,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.128621+0000","last_change":"2024-02-05T08:11:18.292754+0000","last_active":"2024-02-05T08:11:22.128621+0000","last_peered":"2024-02-05T08:11:22.128621+0000","last_clean":"2024-02-05T08:11:22.128621+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:11:22.128621+0000","last_undegraded":"2024-02-05T08:11:22.128621+0000","last_fullsized":"2024-02-05T08:11:22.128621+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:18.292640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_clean_scrub_stamp":"2024-02-05T08:11:18.292640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:23.693559+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"167'1935","reported_seq":1965,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.096464+0000","last_change":"2024-02-05T08:10:54.987729+0000","last_active":"2024-02-05T08:11:23.096464+0000","last_peered":"2024-02-05T08:11:23.096464+0000","last_clean":"2024-02-05T08:11:23.096464+0000","last_became_active":"2024-02-05T08:10:54.987175+0000","last_became_peered":"2024-02-05T08:10:54.987175+0000","last_unstale":"2024-02-05T08:11:23.096464+0000","last_undegraded":"2024-02-05T08:11:23.096464+0000","last_fullsized":"2024-02-05T08:11:23.096464+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":1935,"log_dups_size":0,"ondisk_log_size":1935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:56.138371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126812160,"num_objects":1935,"num_object_clones":0,"num_object_copies":5805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1935,"num_write_kb":123840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":228,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.193896+0000","last_change":"2024-02-05T08:10:21.221033+0000","last_active":"2024-02-05T08:11:23.193896+0000","last_peered":"2024-02-05T08:11:23.193896+0000","last_clean":"2024-02-05T08:11:23.193896+0000","last_became_active":"2024-02-05T08:04:47.587082+0000","last_became_peered":"2024-02-05T08:04:47.587082+0000","last_unstale":"2024-02-05T08:11:23.193896+0000","last_undegraded":"2024-02-05T08:11:23.193896+0000","last_fullsized":"2024-02-05T08:11:23.193896+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:21.220941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:21.220941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:37.669661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":237,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.231240+0000","last_change":"2024-02-05T08:11:22.231278+0000","last_active":"2024-02-05T08:11:22.231240+0000","last_peered":"2024-02-05T08:11:22.231240+0000","last_clean":"2024-02-05T08:11:22.231240+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:11:22.231240+0000","last_undegraded":"2024-02-05T08:11:22.231240+0000","last_fullsized":"2024-02-05T08:11:22.231240+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:11:22.231148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:51.378207+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":249,"reported_epoch":168,"state":"active+clean+scrubbing","last_fresh":"2024-02-05T08:11:27.537755+0000","last_change":"2024-02-05T08:11:27.534449+0000","last_active":"2024-02-05T08:11:27.537755+0000","last_peered":"2024-02-05T08:11:27.537755+0000","last_clean":"2024-02-05T08:11:27.537755+0000","last_became_active":"2024-02-05T08:06:39.554318+0000","last_became_peered":"2024-02-05T08:06:39.554318+0000","last_unstale":"2024-02-05T08:11:27.537755+0000","last_undegraded":"2024-02-05T08:11:27.537755+0000","last_fullsized":"2024-02-05T08:11:27.537755+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:31.480558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:31.480558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":251,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.129541+0000","last_change":"2024-02-05T08:11:19.277720+0000","last_active":"2024-02-05T08:11:22.129541+0000","last_peered":"2024-02-05T08:11:22.129541+0000","last_clean":"2024-02-05T08:11:22.129541+0000","last_became_active":"2024-02-05T08:06:39.552819+0000","last_became_peered":"2024-02-05T08:06:39.552819+0000","last_unstale":"2024-02-05T08:11:22.129541+0000","last_undegraded":"2024-02-05T08:11:22.129541+0000","last_fullsized":"2024-02-05T08:11:22.129541+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:11:19.277464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:19.277464+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:36.607221+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"167'1796","reported_seq":1825,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.896855+0000","last_change":"2024-02-05T08:10:54.989610+0000","last_active":"2024-02-05T08:11:22.896855+0000","last_peered":"2024-02-05T08:11:22.896855+0000","last_clean":"2024-02-05T08:11:22.896855+0000","last_became_active":"2024-02-05T08:10:54.989337+0000","last_became_peered":"2024-02-05T08:10:54.989337+0000","last_unstale":"2024-02-05T08:11:22.896855+0000","last_undegraded":"2024-02-05T08:11:22.896855+0000","last_fullsized":"2024-02-05T08:11:22.896855+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":1796,"log_dups_size":0,"ondisk_log_size":1796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:08.496244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117702656,"num_objects":1796,"num_object_clones":0,"num_object_copies":5388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1796,"num_write_kb":114944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":225,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.126975+0000","last_change":"2024-02-05T08:11:08.235398+0000","last_active":"2024-02-05T08:11:22.126975+0000","last_peered":"2024-02-05T08:11:22.126975+0000","last_clean":"2024-02-05T08:11:22.126975+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:11:22.126975+0000","last_undegraded":"2024-02-05T08:11:22.126975+0000","last_fullsized":"2024-02-05T08:11:22.126975+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:08.235281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:08.235281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:09.928779+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":238,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.128970+0000","last_change":"2024-02-05T08:11:15.272413+0000","last_active":"2024-02-05T08:11:22.128970+0000","last_peered":"2024-02-05T08:11:22.128970+0000","last_clean":"2024-02-05T08:11:22.128970+0000","last_became_active":"2024-02-05T08:04:45.567341+0000","last_became_peered":"2024-02-05T08:04:45.567341+0000","last_unstale":"2024-02-05T08:11:22.128970+0000","last_undegraded":"2024-02-05T08:11:22.128970+0000","last_fullsized":"2024-02-05T08:11:22.128970+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:15.272287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.272287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:39.542308+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":245,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.116676+0000","last_change":"2024-02-05T08:10:30.671004+0000","last_active":"2024-02-05T08:11:23.116676+0000","last_peered":"2024-02-05T08:11:23.116676+0000","last_clean":"2024-02-05T08:11:23.116676+0000","last_became_active":"2024-02-05T08:06:39.558993+0000","last_became_peered":"2024-02-05T08:06:39.558993+0000","last_unstale":"2024-02-05T08:11:23.116676+0000","last_undegraded":"2024-02-05T08:11:23.116676+0000","last_fullsized":"2024-02-05T08:11:23.116676+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:30.670916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.670916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:55.449333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":248,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.122721+0000","last_change":"2024-02-05T08:10:34.198669+0000","last_active":"2024-02-05T08:11:22.122721+0000","last_peered":"2024-02-05T08:11:22.122721+0000","last_clean":"2024-02-05T08:11:22.122721+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:11:22.122721+0000","last_undegraded":"2024-02-05T08:11:22.122721+0000","last_fullsized":"2024-02-05T08:11:22.122721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:10:34.198584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:34.198584+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:42.971407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"168'2366","reported_seq":2398,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:27.513319+0000","last_change":"2024-02-05T08:10:54.991561+0000","last_active":"2024-02-05T08:11:27.513319+0000","last_peered":"2024-02-05T08:11:27.513319+0000","last_clean":"2024-02-05T08:11:27.513319+0000","last_became_active":"2024-02-05T08:10:54.991198+0000","last_became_peered":"2024-02-05T08:10:54.991198+0000","last_unstale":"2024-02-05T08:11:27.513319+0000","last_undegraded":"2024-02-05T08:11:27.513319+0000","last_fullsized":"2024-02-05T08:11:27.513319+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":2366,"log_dups_size":0,"ondisk_log_size":2366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:05.226680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155058176,"num_objects":2366,"num_object_clones":0,"num_object_copies":7098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2366,"num_write_kb":151424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":217,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.123541+0000","last_change":"2024-02-05T08:10:17.009112+0000","last_active":"2024-02-05T08:11:22.123541+0000","last_peered":"2024-02-05T08:11:22.123541+0000","last_clean":"2024-02-05T08:11:22.123541+0000","last_became_active":"2024-02-05T08:04:47.588395+0000","last_became_peered":"2024-02-05T08:04:47.588395+0000","last_unstale":"2024-02-05T08:11:22.123541+0000","last_undegraded":"2024-02-05T08:11:22.123541+0000","last_fullsized":"2024-02-05T08:11:22.123541+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:17.008882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:17.008882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":261,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.116796+0000","last_change":"2024-02-05T08:10:18.580307+0000","last_active":"2024-02-05T08:11:23.116796+0000","last_peered":"2024-02-05T08:11:23.116796+0000","last_clean":"2024-02-05T08:11:23.116796+0000","last_became_active":"2024-02-05T08:06:39.560631+0000","last_became_peered":"2024-02-05T08:06:39.560631+0000","last_unstale":"2024-02-05T08:11:23.116796+0000","last_undegraded":"2024-02-05T08:11:23.116796+0000","last_fullsized":"2024-02-05T08:11:23.116796+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:10:18.580209+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.580209+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:45.524064+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":245,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.112251+0000","last_change":"2024-02-05T08:11:07.502507+0000","last_active":"2024-02-05T08:11:23.112251+0000","last_peered":"2024-02-05T08:11:23.112251+0000","last_clean":"2024-02-05T08:11:23.112251+0000","last_became_active":"2024-02-05T08:06:39.563051+0000","last_became_peered":"2024-02-05T08:06:39.563051+0000","last_unstale":"2024-02-05T08:11:23.112251+0000","last_undegraded":"2024-02-05T08:11:23.112251+0000","last_fullsized":"2024-02-05T08:11:23.112251+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:07.502393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.502393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:36.220738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"85'38","reported_seq":373,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.120013+0000","last_change":"2024-02-05T08:09:35.108648+0000","last_active":"2024-02-05T08:11:22.120013+0000","last_peered":"2024-02-05T08:11:22.120013+0000","last_clean":"2024-02-05T08:11:22.120013+0000","last_became_active":"2024-02-05T08:04:43.554536+0000","last_became_peered":"2024-02-05T08:04:43.554536+0000","last_unstale":"2024-02-05T08:11:22.120013+0000","last_undegraded":"2024-02-05T08:11:22.120013+0000","last_fullsized":"2024-02-05T08:11:22.120013+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:09:35.108489+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:09:35.108489+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":221,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.123295+0000","last_change":"2024-02-05T08:10:19.265168+0000","last_active":"2024-02-05T08:11:22.123295+0000","last_peered":"2024-02-05T08:11:22.123295+0000","last_clean":"2024-02-05T08:11:22.123295+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:11:22.123295+0000","last_undegraded":"2024-02-05T08:11:22.123295+0000","last_fullsized":"2024-02-05T08:11:22.123295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:19.265057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:19.265057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:45.883831+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"167'1818","reported_seq":1848,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.867354+0000","last_change":"2024-02-05T08:10:54.985946+0000","last_active":"2024-02-05T08:11:22.867354+0000","last_peered":"2024-02-05T08:11:22.867354+0000","last_clean":"2024-02-05T08:11:22.867354+0000","last_became_active":"2024-02-05T08:10:54.984063+0000","last_became_peered":"2024-02-05T08:10:54.984063+0000","last_unstale":"2024-02-05T08:11:22.867354+0000","last_undegraded":"2024-02-05T08:11:22.867354+0000","last_fullsized":"2024-02-05T08:11:22.867354+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":1818,"log_dups_size":0,"ondisk_log_size":1818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:17.275769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119144448,"num_objects":1818,"num_object_clones":0,"num_object_copies":5454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1818,"num_write_kb":116352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":234,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.195142+0000","last_change":"2024-02-05T08:11:06.450399+0000","last_active":"2024-02-05T08:11:23.195142+0000","last_peered":"2024-02-05T08:11:23.195142+0000","last_clean":"2024-02-05T08:11:23.195142+0000","last_became_active":"2024-02-05T08:06:39.555147+0000","last_became_peered":"2024-02-05T08:06:39.555147+0000","last_unstale":"2024-02-05T08:11:23.195142+0000","last_undegraded":"2024-02-05T08:11:23.195142+0000","last_fullsized":"2024-02-05T08:11:23.195142+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:06.450276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:06.450276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:15.727823+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":247,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.121002+0000","last_change":"2024-02-05T08:08:41.006239+0000","last_active":"2024-02-05T08:11:22.121002+0000","last_peered":"2024-02-05T08:11:22.121002+0000","last_clean":"2024-02-05T08:11:22.121002+0000","last_became_active":"2024-02-05T08:06:39.560285+0000","last_became_peered":"2024-02-05T08:06:39.560285+0000","last_unstale":"2024-02-05T08:11:22.121002+0000","last_undegraded":"2024-02-05T08:11:22.121002+0000","last_fullsized":"2024-02-05T08:11:22.121002+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:08:41.005907+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:08:41.005907+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":255,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.130810+0000","last_change":"2024-02-05T08:10:34.494353+0000","last_active":"2024-02-05T08:11:22.130810+0000","last_peered":"2024-02-05T08:11:22.130810+0000","last_clean":"2024-02-05T08:11:22.130810+0000","last_became_active":"2024-02-05T08:04:40.529073+0000","last_became_peered":"2024-02-05T08:04:40.529073+0000","last_unstale":"2024-02-05T08:11:22.130810+0000","last_undegraded":"2024-02-05T08:11:22.130810+0000","last_fullsized":"2024-02-05T08:11:22.130810+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:34.494252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:10:34.494252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:01.739051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":237,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.130077+0000","last_change":"2024-02-05T08:10:24.681839+0000","last_active":"2024-02-05T08:11:22.130077+0000","last_peered":"2024-02-05T08:11:22.130077+0000","last_clean":"2024-02-05T08:11:22.130077+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:11:22.130077+0000","last_undegraded":"2024-02-05T08:11:22.130077+0000","last_fullsized":"2024-02-05T08:11:22.130077+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:10:24.681753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:24.681753+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:25.001232+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":238,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.131325+0000","last_change":"2024-02-05T08:11:01.337800+0000","last_active":"2024-02-05T08:11:22.131325+0000","last_peered":"2024-02-05T08:11:22.131325+0000","last_clean":"2024-02-05T08:11:22.131325+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:11:22.131325+0000","last_undegraded":"2024-02-05T08:11:22.131325+0000","last_fullsized":"2024-02-05T08:11:22.131325+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:01.337730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:01.337730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:03.588977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"168'2279","reported_seq":2311,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:27.565857+0000","last_change":"2024-02-05T08:10:54.989478+0000","last_active":"2024-02-05T08:11:27.565857+0000","last_peered":"2024-02-05T08:11:27.565857+0000","last_clean":"2024-02-05T08:11:27.565857+0000","last_became_active":"2024-02-05T08:10:54.989246+0000","last_became_peered":"2024-02-05T08:10:54.989246+0000","last_unstale":"2024-02-05T08:11:27.565857+0000","last_undegraded":"2024-02-05T08:11:27.565857+0000","last_fullsized":"2024-02-05T08:11:27.565857+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":2279,"log_dups_size":0,"ondisk_log_size":2279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:10.507860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149356544,"num_objects":2279,"num_object_clones":0,"num_object_copies":6837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2279,"num_write_kb":145856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":227,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.116798+0000","last_change":"2024-02-05T08:11:02.526794+0000","last_active":"2024-02-05T08:11:23.116798+0000","last_peered":"2024-02-05T08:11:23.116798+0000","last_clean":"2024-02-05T08:11:23.116798+0000","last_became_active":"2024-02-05T08:04:47.586358+0000","last_became_peered":"2024-02-05T08:04:47.586358+0000","last_unstale":"2024-02-05T08:11:23.116798+0000","last_undegraded":"2024-02-05T08:11:23.116798+0000","last_fullsized":"2024-02-05T08:11:23.116798+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:02.526668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:02.526668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:03.167722+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":512,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.194121+0000","last_change":"2024-02-05T08:11:05.478409+0000","last_active":"2024-02-05T08:11:23.194121+0000","last_peered":"2024-02-05T08:11:23.194121+0000","last_clean":"2024-02-05T08:11:23.194121+0000","last_became_active":"2024-02-05T08:06:42.566771+0000","last_became_peered":"2024-02-05T08:06:42.566771+0000","last_unstale":"2024-02-05T08:11:23.194121+0000","last_undegraded":"2024-02-05T08:11:23.194121+0000","last_fullsized":"2024-02-05T08:11:23.194121+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:11:05.478284+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:11:05.478284+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:23.502339+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":235,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.118139+0000","last_change":"2024-02-05T08:08:39.073558+0000","last_active":"2024-02-05T08:11:22.118139+0000","last_peered":"2024-02-05T08:11:22.118139+0000","last_clean":"2024-02-05T08:11:22.118139+0000","last_became_active":"2024-02-05T08:06:39.553704+0000","last_became_peered":"2024-02-05T08:06:39.553704+0000","last_unstale":"2024-02-05T08:11:22.118139+0000","last_undegraded":"2024-02-05T08:11:22.118139+0000","last_fullsized":"2024-02-05T08:11:22.118139+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:08:39.071354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":289,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.113720+0000","last_change":"2024-02-05T08:10:19.453291+0000","last_active":"2024-02-05T08:11:23.113720+0000","last_peered":"2024-02-05T08:11:23.113720+0000","last_clean":"2024-02-05T08:11:23.113720+0000","last_became_active":"2024-02-05T08:06:39.563373+0000","last_became_peered":"2024-02-05T08:06:39.563373+0000","last_unstale":"2024-02-05T08:11:23.113720+0000","last_undegraded":"2024-02-05T08:11:23.113720+0000","last_fullsized":"2024-02-05T08:11:23.113720+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-02-05T08:10:19.453162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:19.453162+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:39.266463+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":232,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.194078+0000","last_change":"2024-02-05T08:11:04.446955+0000","last_active":"2024-02-05T08:11:23.194078+0000","last_peered":"2024-02-05T08:11:23.194078+0000","last_clean":"2024-02-05T08:11:23.194078+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:11:23.194078+0000","last_undegraded":"2024-02-05T08:11:23.194078+0000","last_fullsized":"2024-02-05T08:11:23.194078+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:04.446828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:04.446828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:21.009839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"167'1871","reported_seq":1901,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.888560+0000","last_change":"2024-02-05T08:10:54.985737+0000","last_active":"2024-02-05T08:11:22.888560+0000","last_peered":"2024-02-05T08:11:22.888560+0000","last_clean":"2024-02-05T08:11:22.888560+0000","last_became_active":"2024-02-05T08:10:54.985443+0000","last_became_peered":"2024-02-05T08:10:54.985443+0000","last_unstale":"2024-02-05T08:11:22.888560+0000","last_undegraded":"2024-02-05T08:11:22.888560+0000","last_fullsized":"2024-02-05T08:11:22.888560+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":1871,"log_dups_size":0,"ondisk_log_size":1871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:00.190681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":122617856,"num_objects":1871,"num_object_clones":0,"num_object_copies":5613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1871,"num_write_kb":119744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":226,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.119761+0000","last_change":"2024-02-05T08:10:35.415472+0000","last_active":"2024-02-05T08:11:22.119761+0000","last_peered":"2024-02-05T08:11:22.119761+0000","last_clean":"2024-02-05T08:11:22.119761+0000","last_became_active":"2024-02-05T08:04:47.586774+0000","last_became_peered":"2024-02-05T08:04:47.586774+0000","last_unstale":"2024-02-05T08:11:22.119761+0000","last_undegraded":"2024-02-05T08:11:22.119761+0000","last_fullsized":"2024-02-05T08:11:22.119761+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:35.415402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:35.415402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:50.785267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":229,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.125517+0000","last_change":"2024-02-05T08:08:38.705016+0000","last_active":"2024-02-05T08:11:22.125517+0000","last_peered":"2024-02-05T08:11:22.125517+0000","last_clean":"2024-02-05T08:11:22.125517+0000","last_became_active":"2024-02-05T08:04:41.537872+0000","last_became_peered":"2024-02-05T08:04:41.537872+0000","last_unstale":"2024-02-05T08:11:22.125517+0000","last_undegraded":"2024-02-05T08:11:22.125517+0000","last_fullsized":"2024-02-05T08:11:22.125517+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:08:38.704911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:08:38.704911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":660,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:26.284696+0000","last_change":"2024-02-05T08:11:20.552800+0000","last_active":"2024-02-05T08:11:26.284696+0000","last_peered":"2024-02-05T08:11:26.284696+0000","last_clean":"2024-02-05T08:11:26.284696+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:11:26.284696+0000","last_undegraded":"2024-02-05T08:11:26.284696+0000","last_fullsized":"2024-02-05T08:11:26.284696+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:11:20.552682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:11:20.552682+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:33.685592+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":400,"num_read_kb":395,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":243,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.129782+0000","last_change":"2024-02-05T08:09:19.836523+0000","last_active":"2024-02-05T08:11:22.129782+0000","last_peered":"2024-02-05T08:11:22.129782+0000","last_clean":"2024-02-05T08:11:22.129782+0000","last_became_active":"2024-02-05T08:04:43.549687+0000","last_became_peered":"2024-02-05T08:04:43.549687+0000","last_unstale":"2024-02-05T08:11:22.129782+0000","last_undegraded":"2024-02-05T08:11:22.129782+0000","last_fullsized":"2024-02-05T08:11:22.129782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:09:19.836401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.836401+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":231,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.116948+0000","last_change":"2024-02-05T08:11:03.475345+0000","last_active":"2024-02-05T08:11:23.116948+0000","last_peered":"2024-02-05T08:11:23.116948+0000","last_clean":"2024-02-05T08:11:23.116948+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:11:23.116948+0000","last_undegraded":"2024-02-05T08:11:23.116948+0000","last_fullsized":"2024-02-05T08:11:23.116948+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:03.475206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:03.475206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:03.647627+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"167'1924","reported_seq":1953,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.859737+0000","last_change":"2024-02-05T08:10:54.988633+0000","last_active":"2024-02-05T08:11:22.859737+0000","last_peered":"2024-02-05T08:11:22.859737+0000","last_clean":"2024-02-05T08:11:22.859737+0000","last_became_active":"2024-02-05T08:10:54.988338+0000","last_became_peered":"2024-02-05T08:10:54.988338+0000","last_unstale":"2024-02-05T08:11:22.859737+0000","last_undegraded":"2024-02-05T08:11:22.859737+0000","last_fullsized":"2024-02-05T08:11:22.859737+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":1924,"log_dups_size":0,"ondisk_log_size":1924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:02.573105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126091264,"num_objects":1924,"num_object_clones":0,"num_object_copies":5772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1924,"num_write_kb":123136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":232,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.121574+0000","last_change":"2024-02-05T08:10:18.609876+0000","last_active":"2024-02-05T08:11:22.121574+0000","last_peered":"2024-02-05T08:11:22.121574+0000","last_clean":"2024-02-05T08:11:22.121574+0000","last_became_active":"2024-02-05T08:06:39.559847+0000","last_became_peered":"2024-02-05T08:06:39.559847+0000","last_unstale":"2024-02-05T08:11:22.121574+0000","last_undegraded":"2024-02-05T08:11:22.121574+0000","last_fullsized":"2024-02-05T08:11:22.121574+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.609767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.609767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:43.852340+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":246,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.119938+0000","last_change":"2024-02-05T08:11:18.828143+0000","last_active":"2024-02-05T08:11:22.119938+0000","last_peered":"2024-02-05T08:11:22.119938+0000","last_clean":"2024-02-05T08:11:22.119938+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:11:22.119938+0000","last_undegraded":"2024-02-05T08:11:22.119938+0000","last_fullsized":"2024-02-05T08:11:22.119938+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:18.828029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:18.828029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:19.010299+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":269,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.122574+0000","last_change":"2024-02-05T08:10:16.309686+0000","last_active":"2024-02-05T08:11:22.122574+0000","last_peered":"2024-02-05T08:11:22.122574+0000","last_clean":"2024-02-05T08:11:22.122574+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:11:22.122574+0000","last_undegraded":"2024-02-05T08:11:22.122574+0000","last_fullsized":"2024-02-05T08:11:22.122574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:10:16.309551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:16.309551+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:43.442839+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":168,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.126885+0000","last_change":"2024-02-05T08:11:21.296201+0000","last_active":"2024-02-05T08:11:22.126885+0000","last_peered":"2024-02-05T08:11:22.126885+0000","last_clean":"2024-02-05T08:11:22.126885+0000","last_became_active":"2024-02-05T08:06:39.558100+0000","last_became_peered":"2024-02-05T08:06:39.558100+0000","last_unstale":"2024-02-05T08:11:22.126885+0000","last_undegraded":"2024-02-05T08:11:22.126885+0000","last_fullsized":"2024-02-05T08:11:22.126885+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:21.296072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:11:21.296072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:23.327017+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"167'1901","reported_seq":1929,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.088057+0000","last_change":"2024-02-05T08:10:54.987464+0000","last_active":"2024-02-05T08:11:23.088057+0000","last_peered":"2024-02-05T08:11:23.088057+0000","last_clean":"2024-02-05T08:11:23.088057+0000","last_became_active":"2024-02-05T08:10:54.985442+0000","last_became_peered":"2024-02-05T08:10:54.985442+0000","last_unstale":"2024-02-05T08:11:23.088057+0000","last_undegraded":"2024-02-05T08:11:23.088057+0000","last_fullsized":"2024-02-05T08:11:23.088057+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":1901,"log_dups_size":0,"ondisk_log_size":1901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.099986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124583936,"num_objects":1901,"num_object_clones":0,"num_object_copies":5703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1901,"num_write_kb":121664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":225,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.121981+0000","last_change":"2024-02-05T08:10:28.229516+0000","last_active":"2024-02-05T08:11:22.121981+0000","last_peered":"2024-02-05T08:11:22.121981+0000","last_clean":"2024-02-05T08:11:22.121981+0000","last_became_active":"2024-02-05T08:04:47.583866+0000","last_became_peered":"2024-02-05T08:04:47.583866+0000","last_unstale":"2024-02-05T08:11:22.121981+0000","last_undegraded":"2024-02-05T08:11:22.121981+0000","last_fullsized":"2024-02-05T08:11:22.121981+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:28.229441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:28.229441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:33.507208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":247,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:26.212445+0000","last_change":"2024-02-05T08:11:26.212482+0000","last_active":"2024-02-05T08:11:26.212445+0000","last_peered":"2024-02-05T08:11:26.212445+0000","last_clean":"2024-02-05T08:11:26.212445+0000","last_became_active":"2024-02-05T08:04:41.537818+0000","last_became_peered":"2024-02-05T08:04:41.537818+0000","last_unstale":"2024-02-05T08:11:26.212445+0000","last_undegraded":"2024-02-05T08:11:26.212445+0000","last_fullsized":"2024-02-05T08:11:26.212445+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:26.212363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:26.212363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:35.994909+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":249,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.120066+0000","last_change":"2024-02-05T08:10:21.554686+0000","last_active":"2024-02-05T08:11:22.120066+0000","last_peered":"2024-02-05T08:11:22.120066+0000","last_clean":"2024-02-05T08:11:22.120066+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:11:22.120066+0000","last_undegraded":"2024-02-05T08:11:22.120066+0000","last_fullsized":"2024-02-05T08:11:22.120066+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:10:21.554300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:21.554300+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:39.632000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":328,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.122827+0000","last_change":"2024-02-05T08:11:13.252802+0000","last_active":"2024-02-05T08:11:22.122827+0000","last_peered":"2024-02-05T08:11:22.122827+0000","last_clean":"2024-02-05T08:11:22.122827+0000","last_became_active":"2024-02-05T08:06:39.554860+0000","last_became_peered":"2024-02-05T08:06:39.554860+0000","last_unstale":"2024-02-05T08:11:22.122827+0000","last_undegraded":"2024-02-05T08:11:22.122827+0000","last_fullsized":"2024-02-05T08:11:22.122827+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:11:13.252683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:13.252683+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:33.309577+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"168'2387","reported_seq":2419,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:27.544298+0000","last_change":"2024-02-05T08:10:54.986461+0000","last_active":"2024-02-05T08:11:27.544298+0000","last_peered":"2024-02-05T08:11:27.544298+0000","last_clean":"2024-02-05T08:11:27.544298+0000","last_became_active":"2024-02-05T08:10:54.986149+0000","last_became_peered":"2024-02-05T08:10:54.986149+0000","last_unstale":"2024-02-05T08:11:27.544298+0000","last_undegraded":"2024-02-05T08:11:27.544298+0000","last_fullsized":"2024-02-05T08:11:27.544298+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":2387,"log_dups_size":0,"ondisk_log_size":2387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:21.583807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156434432,"num_objects":2387,"num_object_clones":0,"num_object_copies":7161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2387,"num_write_kb":152768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":234,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.194237+0000","last_change":"2024-02-05T08:11:09.466858+0000","last_active":"2024-02-05T08:11:23.194237+0000","last_peered":"2024-02-05T08:11:23.194237+0000","last_clean":"2024-02-05T08:11:23.194237+0000","last_became_active":"2024-02-05T08:06:39.554165+0000","last_became_peered":"2024-02-05T08:06:39.554165+0000","last_unstale":"2024-02-05T08:11:23.194237+0000","last_undegraded":"2024-02-05T08:11:23.194237+0000","last_fullsized":"2024-02-05T08:11:23.194237+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:09.466721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:09.466721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:19.223486+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":237,"reported_epoch":166,"state":"peering","last_fresh":"2024-02-05T08:11:20.096847+0000","last_change":"2024-02-05T08:11:22.125643+0000","last_active":"2024-02-05T08:11:22.125643+0000","last_peered":"2024-02-05T08:11:20.096847+0000","last_clean":"2024-02-05T08:11:20.096847+0000","last_became_active":"2024-02-05T08:04:41.534433+0000","last_became_peered":"2024-02-05T08:04:41.534433+0000","last_unstale":"2024-02-05T08:11:20.096847+0000","last_undegraded":"2024-02-05T08:11:20.096847+0000","last_fullsized":"2024-02-05T08:11:20.096847+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:10:30.240833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.240833+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":270,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.112507+0000","last_change":"2024-02-05T08:10:16.458012+0000","last_active":"2024-02-05T08:11:23.112507+0000","last_peered":"2024-02-05T08:11:23.112507+0000","last_clean":"2024-02-05T08:11:23.112507+0000","last_became_active":"2024-02-05T08:06:39.566434+0000","last_became_peered":"2024-02-05T08:06:39.566434+0000","last_unstale":"2024-02-05T08:11:23.112507+0000","last_undegraded":"2024-02-05T08:11:23.112507+0000","last_fullsized":"2024-02-05T08:11:23.112507+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-02-05T08:10:16.457895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:16.457895+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:40.130069+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":310,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.117626+0000","last_change":"2024-02-05T08:07:47.765640+0000","last_active":"2024-02-05T08:11:22.117626+0000","last_peered":"2024-02-05T08:11:22.117626+0000","last_clean":"2024-02-05T08:11:22.117626+0000","last_became_active":"2024-02-05T08:04:45.562597+0000","last_became_peered":"2024-02-05T08:04:45.562597+0000","last_unstale":"2024-02-05T08:11:22.117626+0000","last_undegraded":"2024-02-05T08:11:22.117626+0000","last_fullsized":"2024-02-05T08:11:22.117626+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:07:47.765523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:07:47.765523+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"167'1852","reported_seq":1882,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.083900+0000","last_change":"2024-02-05T08:10:54.990174+0000","last_active":"2024-02-05T08:11:23.083900+0000","last_peered":"2024-02-05T08:11:23.083900+0000","last_clean":"2024-02-05T08:11:23.083900+0000","last_became_active":"2024-02-05T08:10:54.989454+0000","last_became_peered":"2024-02-05T08:10:54.989454+0000","last_unstale":"2024-02-05T08:11:23.083900+0000","last_undegraded":"2024-02-05T08:11:23.083900+0000","last_fullsized":"2024-02-05T08:11:23.083900+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":1852,"log_dups_size":0,"ondisk_log_size":1852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.180834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121372672,"num_objects":1852,"num_object_clones":0,"num_object_copies":5556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1852,"num_write_kb":118528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":227,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.113393+0000","last_change":"2024-02-05T08:10:15.488463+0000","last_active":"2024-02-05T08:11:23.113393+0000","last_peered":"2024-02-05T08:11:23.113393+0000","last_clean":"2024-02-05T08:11:23.113393+0000","last_became_active":"2024-02-05T08:04:47.582776+0000","last_became_peered":"2024-02-05T08:04:47.582776+0000","last_unstale":"2024-02-05T08:11:23.113393+0000","last_undegraded":"2024-02-05T08:11:23.113393+0000","last_fullsized":"2024-02-05T08:11:23.113393+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:15.488380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:15.488380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:31.489301+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":243,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.123864+0000","last_change":"2024-02-05T08:09:36.111657+0000","last_active":"2024-02-05T08:11:22.123864+0000","last_peered":"2024-02-05T08:11:22.123864+0000","last_clean":"2024-02-05T08:11:22.123864+0000","last_became_active":"2024-02-05T08:04:41.540818+0000","last_became_peered":"2024-02-05T08:04:41.540818+0000","last_unstale":"2024-02-05T08:11:22.123864+0000","last_undegraded":"2024-02-05T08:11:22.123864+0000","last_fullsized":"2024-02-05T08:11:22.123864+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:09:36.111537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.111537+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":265,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.117544+0000","last_change":"2024-02-05T08:09:33.222821+0000","last_active":"2024-02-05T08:11:22.117544+0000","last_peered":"2024-02-05T08:11:22.117544+0000","last_clean":"2024-02-05T08:11:22.117544+0000","last_became_active":"2024-02-05T08:04:43.607546+0000","last_became_peered":"2024-02-05T08:04:43.607546+0000","last_unstale":"2024-02-05T08:11:22.117544+0000","last_undegraded":"2024-02-05T08:11:22.117544+0000","last_fullsized":"2024-02-05T08:11:22.117544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-05T08:09:33.222532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:33.222532+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":223,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.111979+0000","last_change":"2024-02-05T08:09:20.449787+0000","last_active":"2024-02-05T08:11:23.111979+0000","last_peered":"2024-02-05T08:11:23.111979+0000","last_clean":"2024-02-05T08:11:23.111979+0000","last_became_active":"2024-02-05T08:04:45.561639+0000","last_became_peered":"2024-02-05T08:04:45.561639+0000","last_unstale":"2024-02-05T08:11:23.111979+0000","last_undegraded":"2024-02-05T08:11:23.111979+0000","last_fullsized":"2024-02-05T08:11:23.111979+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:20.449711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:20.449711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"167'1841","reported_seq":1871,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.888795+0000","last_change":"2024-02-05T08:10:54.984735+0000","last_active":"2024-02-05T08:11:22.888795+0000","last_peered":"2024-02-05T08:11:22.888795+0000","last_clean":"2024-02-05T08:11:22.888795+0000","last_became_active":"2024-02-05T08:10:54.984306+0000","last_became_peered":"2024-02-05T08:10:54.984306+0000","last_unstale":"2024-02-05T08:11:22.888795+0000","last_undegraded":"2024-02-05T08:11:22.888795+0000","last_fullsized":"2024-02-05T08:11:22.888795+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":1841,"log_dups_size":0,"ondisk_log_size":1841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:17.368690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120651776,"num_objects":1841,"num_object_clones":0,"num_object_copies":5523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1841,"num_write_kb":117824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":234,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.116750+0000","last_change":"2024-02-05T08:11:08.491427+0000","last_active":"2024-02-05T08:11:23.116750+0000","last_peered":"2024-02-05T08:11:23.116750+0000","last_clean":"2024-02-05T08:11:23.116750+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:11:23.116750+0000","last_undegraded":"2024-02-05T08:11:23.116750+0000","last_fullsized":"2024-02-05T08:11:23.116750+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:08.491314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:08.491314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:11.472763+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":312,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.113199+0000","last_change":"2024-02-05T08:09:17.391826+0000","last_active":"2024-02-05T08:11:23.113199+0000","last_peered":"2024-02-05T08:11:23.113199+0000","last_clean":"2024-02-05T08:11:23.113199+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:11:23.113199+0000","last_undegraded":"2024-02-05T08:11:23.113199+0000","last_fullsized":"2024-02-05T08:11:23.113199+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:09:17.391673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:09:17.391673+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":240,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.117470+0000","last_change":"2024-02-05T08:11:03.761651+0000","last_active":"2024-02-05T08:11:22.117470+0000","last_peered":"2024-02-05T08:11:22.117470+0000","last_clean":"2024-02-05T08:11:22.117470+0000","last_became_active":"2024-02-05T08:04:41.541327+0000","last_became_peered":"2024-02-05T08:04:41.541327+0000","last_unstale":"2024-02-05T08:11:22.117470+0000","last_undegraded":"2024-02-05T08:11:22.117470+0000","last_fullsized":"2024-02-05T08:11:22.117470+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:03.761546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:03.761546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:30.728931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":258,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.123333+0000","last_change":"2024-02-05T08:09:17.191152+0000","last_active":"2024-02-05T08:11:22.123333+0000","last_peered":"2024-02-05T08:11:22.123333+0000","last_clean":"2024-02-05T08:11:22.123333+0000","last_became_active":"2024-02-05T08:04:43.555028+0000","last_became_peered":"2024-02-05T08:04:43.555028+0000","last_unstale":"2024-02-05T08:11:22.123333+0000","last_undegraded":"2024-02-05T08:11:22.123333+0000","last_fullsized":"2024-02-05T08:11:22.123333+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:09:17.190290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:17.190290+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"168'2414","reported_seq":2446,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:27.526490+0000","last_change":"2024-02-05T08:10:54.985813+0000","last_active":"2024-02-05T08:11:27.526490+0000","last_peered":"2024-02-05T08:11:27.526490+0000","last_clean":"2024-02-05T08:11:27.526490+0000","last_became_active":"2024-02-05T08:10:54.985501+0000","last_became_peered":"2024-02-05T08:10:54.985501+0000","last_unstale":"2024-02-05T08:11:27.526490+0000","last_undegraded":"2024-02-05T08:11:27.526490+0000","last_fullsized":"2024-02-05T08:11:27.526490+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":2414,"log_dups_size":0,"ondisk_log_size":2414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:19.562319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158203904,"num_objects":2414,"num_object_clones":0,"num_object_copies":7242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2414,"num_write_kb":154496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":222,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.118067+0000","last_change":"2024-02-05T08:09:32.213156+0000","last_active":"2024-02-05T08:11:22.118067+0000","last_peered":"2024-02-05T08:11:22.118067+0000","last_clean":"2024-02-05T08:11:22.118067+0000","last_became_active":"2024-02-05T08:06:39.558047+0000","last_became_peered":"2024-02-05T08:06:39.558047+0000","last_unstale":"2024-02-05T08:11:22.118067+0000","last_undegraded":"2024-02-05T08:11:22.118067+0000","last_fullsized":"2024-02-05T08:11:22.118067+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:32.213044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:09:32.213044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":236,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.116962+0000","last_change":"2024-02-05T08:10:20.643993+0000","last_active":"2024-02-05T08:11:23.116962+0000","last_peered":"2024-02-05T08:11:23.116962+0000","last_clean":"2024-02-05T08:11:23.116962+0000","last_became_active":"2024-02-05T08:06:39.551933+0000","last_became_peered":"2024-02-05T08:06:39.551933+0000","last_unstale":"2024-02-05T08:11:23.116962+0000","last_undegraded":"2024-02-05T08:11:23.116962+0000","last_fullsized":"2024-02-05T08:11:23.116962+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:20.643918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:20.643918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:27.975950+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":239,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.112075+0000","last_change":"2024-02-05T08:09:22.454321+0000","last_active":"2024-02-05T08:11:23.112075+0000","last_peered":"2024-02-05T08:11:23.112075+0000","last_clean":"2024-02-05T08:11:23.112075+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:11:23.112075+0000","last_undegraded":"2024-02-05T08:11:23.112075+0000","last_fullsized":"2024-02-05T08:11:23.112075+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:22.454247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.454247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":271,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.112993+0000","last_change":"2024-02-05T08:11:10.470892+0000","last_active":"2024-02-05T08:11:23.112993+0000","last_peered":"2024-02-05T08:11:23.112993+0000","last_clean":"2024-02-05T08:11:23.112993+0000","last_became_active":"2024-02-05T08:06:39.556433+0000","last_became_peered":"2024-02-05T08:06:39.556433+0000","last_unstale":"2024-02-05T08:11:23.112993+0000","last_undegraded":"2024-02-05T08:11:23.112993+0000","last_fullsized":"2024-02-05T08:11:23.112993+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:11:10.470749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:10.470749+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:36.153341+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"167'1794","reported_seq":1824,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.897865+0000","last_change":"2024-02-05T08:10:54.982668+0000","last_active":"2024-02-05T08:11:22.897865+0000","last_peered":"2024-02-05T08:11:22.897865+0000","last_clean":"2024-02-05T08:11:22.897865+0000","last_became_active":"2024-02-05T08:10:54.982372+0000","last_became_peered":"2024-02-05T08:10:54.982372+0000","last_unstale":"2024-02-05T08:11:22.897865+0000","last_undegraded":"2024-02-05T08:11:22.897865+0000","last_fullsized":"2024-02-05T08:11:22.897865+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":1794,"log_dups_size":0,"ondisk_log_size":1794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.063257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117571584,"num_objects":1794,"num_object_clones":0,"num_object_copies":5382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1794,"num_write_kb":114816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":234,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.130281+0000","last_change":"2024-02-05T08:11:06.399741+0000","last_active":"2024-02-05T08:11:22.130281+0000","last_peered":"2024-02-05T08:11:22.130281+0000","last_clean":"2024-02-05T08:11:22.130281+0000","last_became_active":"2024-02-05T08:04:47.585962+0000","last_became_peered":"2024-02-05T08:04:47.585962+0000","last_unstale":"2024-02-05T08:11:22.130281+0000","last_undegraded":"2024-02-05T08:11:22.130281+0000","last_fullsized":"2024-02-05T08:11:22.130281+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:11:06.399635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:06.399635+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:11.606714+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":229,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.120140+0000","last_change":"2024-02-05T08:11:11.789034+0000","last_active":"2024-02-05T08:11:22.120140+0000","last_peered":"2024-02-05T08:11:22.120140+0000","last_clean":"2024-02-05T08:11:22.120140+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:11:22.120140+0000","last_undegraded":"2024-02-05T08:11:22.120140+0000","last_fullsized":"2024-02-05T08:11:22.120140+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:11.788902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:11.788902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:31.941871+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":170,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.113475+0000","last_change":"2024-02-05T08:10:25.446805+0000","last_active":"2024-02-05T08:11:23.113475+0000","last_peered":"2024-02-05T08:11:23.113475+0000","last_clean":"2024-02-05T08:11:23.113475+0000","last_became_active":"2024-02-05T08:06:39.559090+0000","last_became_peered":"2024-02-05T08:06:39.559090+0000","last_unstale":"2024-02-05T08:11:23.113475+0000","last_undegraded":"2024-02-05T08:11:23.113475+0000","last_fullsized":"2024-02-05T08:11:23.113475+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:25.446724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:10:25.446724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:50.729956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":255,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.113639+0000","last_change":"2024-02-05T08:09:21.434983+0000","last_active":"2024-02-05T08:11:23.113639+0000","last_peered":"2024-02-05T08:11:23.113639+0000","last_clean":"2024-02-05T08:11:23.113639+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:11:23.113639+0000","last_undegraded":"2024-02-05T08:11:23.113639+0000","last_fullsized":"2024-02-05T08:11:23.113639+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-02-05T08:09:21.434891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.434891+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"167'1888","reported_seq":1918,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.869305+0000","last_change":"2024-02-05T08:10:54.989334+0000","last_active":"2024-02-05T08:11:22.869305+0000","last_peered":"2024-02-05T08:11:22.869305+0000","last_clean":"2024-02-05T08:11:22.869305+0000","last_became_active":"2024-02-05T08:10:54.988993+0000","last_became_peered":"2024-02-05T08:10:54.988993+0000","last_unstale":"2024-02-05T08:11:22.869305+0000","last_undegraded":"2024-02-05T08:11:22.869305+0000","last_fullsized":"2024-02-05T08:11:22.869305+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":1888,"log_dups_size":0,"ondisk_log_size":1888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:14.963803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123731968,"num_objects":1888,"num_object_clones":0,"num_object_copies":5664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1888,"num_write_kb":120832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":226,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.113242+0000","last_change":"2024-02-05T08:11:15.360217+0000","last_active":"2024-02-05T08:11:23.113242+0000","last_peered":"2024-02-05T08:11:23.113242+0000","last_clean":"2024-02-05T08:11:23.113242+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:11:23.113242+0000","last_undegraded":"2024-02-05T08:11:23.113242+0000","last_fullsized":"2024-02-05T08:11:23.113242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.360090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:39.893177+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":327,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.129558+0000","last_change":"2024-02-05T08:10:31.516653+0000","last_active":"2024-02-05T08:11:22.129558+0000","last_peered":"2024-02-05T08:11:22.129558+0000","last_clean":"2024-02-05T08:11:22.129558+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:11:22.129558+0000","last_undegraded":"2024-02-05T08:11:22.129558+0000","last_fullsized":"2024-02-05T08:11:22.129558+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:10:31.516571+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:10:31.516571+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:01.515985+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":228,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-02-05T08:11:22.117362+0000","last_change":"2024-02-05T08:09:22.380135+0000","last_active":"2024-02-05T08:11:22.117362+0000","last_peered":"2024-02-05T08:11:22.117362+0000","last_clean":"2024-02-05T08:11:22.117362+0000","last_became_active":"2024-02-05T08:04:41.533838+0000","last_became_peered":"2024-02-05T08:04:41.533838+0000","last_unstale":"2024-02-05T08:11:22.117362+0000","last_undegraded":"2024-02-05T08:11:22.117362+0000","last_fullsized":"2024-02-05T08:11:22.117362+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:22.380016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.380016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":268,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-02-05T08:11:23.112606+0000","last_change":"2024-02-05T08:11:14.388931+0000","last_active":"2024-02-05T08:11:23.112606+0000","last_peered":"2024-02-05T08:11:23.112606+0000","last_clean":"2024-02-05T08:11:23.112606+0000","last_became_active":"2024-02-05T08:06:39.555561+0000","last_became_peered":"2024-02-05T08:06:39.555561+0000","last_unstale":"2024-02-05T08:11:23.112606+0000","last_undegraded":"2024-02-05T08:11:23.112606+0000","last_fullsized":"2024-02-05T08:11:23.112606+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:11:14.388825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:14.388825+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:39.267898+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2151874560,"num_objects":32835,"num_object_clones":0,"num_object_copies":98505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32835,"num_write_kb":2101440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6439174144,"data_stored":6439174144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32835,"ondisk_log_size":32835,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":400,"num_read_kb":395,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":1474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":433,"ondisk_log_size":433,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037846,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":881816,"kb_used_data":764560,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":92863336,"statfs":{"total":95995035648,"available":95092056064,"internally_reserved":0,"allocated":782909440,"data_stored":782565810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":120060362},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.581,"15min":0.89700000000000002},"min":{"1min":0.254,"5min":0.218,"15min":0.217},"max":{"1min":3.8420000000000001,"5min":60.127000000000002,"15min":60.127000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.5920000000000001,"15min":0.88},"min":{"1min":0.28399999999999997,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.0860000000000003,"5min":58.283000000000001,"15min":58.283000000000001},"last":0.251}]},{"osd":1,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.411,"15min":0.84099999999999997},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.1949999999999998,"5min":20.774000000000001,"15min":20.774000000000001},"last":1.671},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.095,"15min":0.73299999999999998},"min":{"1min":0.254,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":2.8149999999999999,"5min":20.715,"15min":20.715},"last":0.22500000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":1.837,"5min":1.0349999999999999,"15min":0.73299999999999998},"min":{"1min":0.26900000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":6.2480000000000002,"5min":7.6779999999999999,"15min":7.6779999999999999},"last":1.325},{"interface":"front","average":{"1min":1.125,"5min":1.2170000000000001,"15min":0.76500000000000001},"min":{"1min":0.253,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":6.399,"5min":8.9100000000000001,"15min":8.9100000000000001},"last":1.9059999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.9910000000000001,"15min":1.0429999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.6630000000000003,"5min":58.515000000000001,"15min":58.515000000000001},"last":1.415},{"interface":"front","average":{"1min":1.607,"5min":1.923,"15min":1.0009999999999999},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":6.202,"5min":58.225999999999999,"15min":58.225999999999999},"last":0.50900000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.39300000000000002,"15min":0.42299999999999999},"min":{"1min":0.23799999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.58299999999999996,"5min":0.71099999999999997,"15min":0.91400000000000003},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.40999999999999998,"15min":0.42599999999999999},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.55900000000000005,"5min":0.70299999999999996,"15min":0.72999999999999998},"last":0.33700000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.43099999999999999,"15min":0.438},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69999999999999996,"5min":1.2,"15min":1.2},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.45900000000000002,"15min":0.48099999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67900000000000005,"5min":0.81999999999999995,"15min":0.998},"last":0.372}]},{"osd":6,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47499999999999998,"15min":0.495},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67700000000000005,"5min":1.256,"15min":1.256},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47099999999999997,"15min":0.47599999999999998},"min":{"1min":0.32400000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.71799999999999997,"5min":0.72899999999999998,"15min":0.79300000000000004},"last":0.441}]}]},{"osd":6,"up_from":0,"seq":219043332188,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574964,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":959708,"kb_used_data":784144,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92785444,"statfs":{"total":95995035648,"available":95012294656,"internally_reserved":0,"allocated":802963456,"data_stored":802603169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":2.3929999999999998,"15min":1.01},"min":{"1min":0.253,"5min":0.186,"15min":0.14699999999999999},"max":{"1min":2.5499999999999998,"5min":86.105000000000004,"15min":86.105000000000004},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.0269999999999999,"5min":2.8140000000000001,"15min":1.1619999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.16},"max":{"1min":8.4510000000000005,"5min":85.665000000000006,"15min":85.665000000000006},"last":1.647}]},{"osd":2,"last update":"Mon Feb 5 08:10:34 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":2.0390000000000001,"15min":0.93000000000000005},"min":{"1min":0.20999999999999999,"5min":0.19700000000000001,"15min":0.17799999999999999},"max":{"1min":2.351,"5min":49.030999999999999,"15min":49.030999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":2.3039999999999998,"15min":1.04},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":6.5590000000000002,"5min":48.984000000000002,"15min":48.984000000000002},"last":1.4219999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:10:59 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":2.2709999999999999,"15min":1.05},"min":{"1min":0.26600000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":1.9530000000000001,"5min":49.067999999999998,"15min":49.067999999999998},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":2.3370000000000002,"15min":1.0860000000000001},"min":{"1min":0.27800000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.5109999999999999,"5min":48.734999999999999,"15min":48.734999999999999},"last":0.379}]},{"osd":4,"last update":"Mon Feb 5 08:10:24 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":1.093,"15min":0.755},"min":{"1min":0.28000000000000003,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.8129999999999999,"5min":9.4269999999999996,"15min":9.4269999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.244,"5min":1.1819999999999999,"15min":0.77800000000000002},"min":{"1min":0.30399999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.9300000000000002,"5min":6.3890000000000002,"15min":6.3890000000000002},"last":0.52100000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":1.1499999999999999,"15min":0.752},"min":{"1min":0.308,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.1240000000000001,"5min":9.4559999999999995,"15min":9.4559999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.2010000000000001,"15min":0.77500000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.9660000000000002,"5min":13.776,"15min":13.776},"last":1.387}]},{"osd":6,"last update":"Mon Feb 5 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":1.2170000000000001,"15min":0.80100000000000005},"min":{"1min":0.38,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.3660000000000001,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.63500000000000001,"5min":1.1910000000000001,"15min":0.82099999999999995},"min":{"1min":0.42399999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86699999999999999,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":0.58299999999999996}]},{"osd":7,"last update":"Mon Feb 5 08:10:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.3620000000000001,"15min":0.86799999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.0009999999999999,"5min":13.749000000000001,"15min":13.749000000000001},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.288,"15min":0.83499999999999996},"min":{"1min":0.39100000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.4860000000000002,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.621}]}]},{"osd":0,"up_from":8,"seq":34359738489,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965736,"kb_used_data":799900,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92779416,"statfs":{"total":95995035648,"available":95006121984,"internally_reserved":0,"allocated":819097600,"data_stored":818734044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.2559999999999998,"5min":3.028,"15min":1.2370000000000001},"min":{"1min":0.28799999999999998,"5min":0.19,"15min":0.152},"max":{"1min":32.066000000000003,"5min":66.772999999999996,"15min":66.772999999999996},"last":2.2509999999999999},{"interface":"front","average":{"1min":0.497,"5min":2.1110000000000002,"15min":0.93300000000000005},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.17199999999999999},"max":{"1min":2.2610000000000001,"5min":46.198999999999998,"15min":46.198999999999998},"last":0.69699999999999995}]},{"osd":2,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":2.1720000000000002,"15min":0.97399999999999998},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":6.2249999999999996,"5min":46.170999999999999,"15min":46.170999999999999},"last":1.5189999999999999},{"interface":"front","average":{"1min":2.5870000000000002,"5min":2.5299999999999998,"15min":1.105},"min":{"1min":0.36099999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":11.07,"5min":46.543999999999997,"15min":46.543999999999997},"last":2.3959999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":2.5960000000000001,"15min":1.1699999999999999},"min":{"1min":0.34499999999999997,"5min":0.28399999999999997,"15min":0.21099999999999999},"max":{"1min":7.5970000000000004,"5min":67.888000000000005,"15min":67.888000000000005},"last":0.77500000000000002},{"interface":"front","average":{"1min":3.512,"5min":3.3340000000000001,"15min":1.4219999999999999},"min":{"1min":0.28599999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":37.262999999999998,"5min":66.891999999999996,"15min":66.891999999999996},"last":0.89100000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:10:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.1060000000000001,"15min":0.746},"min":{"1min":0.34599999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":3.5089999999999999,"5min":13.295,"15min":13.295},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.647,"5min":1.2290000000000001,"15min":0.81499999999999995},"min":{"1min":0.35399999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":9.5739999999999998,"5min":12.989000000000001,"15min":12.989000000000001},"last":1.496}]},{"osd":5,"last update":"Mon Feb 5 08:10:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.3080000000000001,"15min":0.83999999999999997},"min":{"1min":0.29899999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.375,"5min":13.073,"15min":13.073},"last":2.2890000000000001},{"interface":"front","average":{"1min":0.998,"5min":1.2609999999999999,"15min":0.83299999999999996},"min":{"1min":0.42599999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.258,"5min":12.94,"15min":12.94},"last":1.4399999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:11:15 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":1.321,"15min":0.84099999999999997},"min":{"1min":0.45600000000000002,"5min":0.378,"15min":0.378},"max":{"1min":2.1549999999999998,"5min":12.967000000000001,"15min":12.967000000000001},"last":1.6140000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.363,"15min":0.86899999999999999},"min":{"1min":0.47699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.9490000000000001,"5min":13.108000000000001,"15min":13.108000000000001},"last":0.628}]},{"osd":7,"last update":"Mon Feb 5 08:10:39 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.1559999999999999,"15min":0.76200000000000001},"min":{"1min":0.38500000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.5459999999999998,"5min":13.263,"15min":13.263},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.151,"5min":1.3520000000000001,"15min":0.85699999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.2320000000000002,"5min":13.012,"15min":13.012},"last":1.47}]}]},{"osd":2,"up_from":18,"seq":77309411439,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1172756,"kb_used_data":1029124,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92572396,"statfs":{"total":95995035648,"available":94794133504,"internally_reserved":0,"allocated":1053822976,"data_stored":1053482178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":147061190},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:10:40 2024","interfaces":[{"interface":"back","average":{"1min":2.589,"5min":2.1099999999999999,"15min":0.95199999999999996},"min":{"1min":0.25,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":8.0350000000000001,"5min":16.155999999999999,"15min":16.155999999999999},"last":0.308},{"interface":"front","average":{"1min":1.8089999999999999,"5min":2.105,"15min":0.95899999999999996},"min":{"1min":0.24199999999999999,"5min":0.23300000000000001,"15min":0.20699999999999999},"max":{"1min":3.9740000000000002,"5min":15.973000000000001,"15min":15.973000000000001},"last":0.41599999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:10:40 2024","interfaces":[{"interface":"back","average":{"1min":2.2669999999999999,"5min":1.7969999999999999,"15min":0.83999999999999997},"min":{"1min":0.28599999999999998,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":6.4829999999999997,"5min":9.1340000000000003,"15min":9.1340000000000003},"last":0.35099999999999998},{"interface":"front","average":{"1min":2.1269999999999998,"5min":1.7649999999999999,"15min":0.84599999999999997},"min":{"1min":0.27100000000000002,"5min":0.193,"15min":0.193},"max":{"1min":7.9400000000000004,"5min":9.2769999999999992,"15min":9.2769999999999992},"last":0.505}]},{"osd":3,"last update":"Mon Feb 5 08:11:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":3.1389999999999998,"15min":1.3580000000000001},"min":{"1min":0.32800000000000001,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":5.4850000000000003,"5min":68.108000000000004,"15min":68.108000000000004},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.776,"5min":2.9319999999999999,"15min":1.28},"min":{"1min":0.32700000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":7.8879999999999999,"5min":67.945999999999998,"15min":67.945999999999998},"last":0.39700000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:10:23 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.3089999999999999,"15min":0.83699999999999997},"min":{"1min":0.34699999999999998,"5min":0.315,"15min":0.315},"max":{"1min":3.863,"5min":15.923999999999999,"15min":15.923999999999999},"last":31.596},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.0529999999999999,"15min":0.73899999999999999},"min":{"1min":0.27200000000000002,"5min":0.252,"15min":0.252},"max":{"1min":3.5880000000000001,"5min":6.008,"15min":6.008},"last":31.57}]},{"osd":5,"last update":"Mon Feb 5 08:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.216,"15min":0.79800000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.6150000000000002,"5min":15.835000000000001,"15min":15.835000000000001},"last":58.799999999999997},{"interface":"front","average":{"1min":0.995,"5min":1.1160000000000001,"15min":0.755},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":3.2149999999999999,"5min":9.1809999999999992,"15min":9.1809999999999992},"last":58.615000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:10:20 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.359,"15min":0.878},"min":{"1min":0.32500000000000001,"5min":0.308,"15min":0.308},"max":{"1min":3.9449999999999998,"5min":15.859,"15min":15.859},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.284,"5min":1.129,"15min":0.80400000000000005},"min":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.6389999999999998,"5min":6.117,"15min":6.117},"last":0.45500000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:10:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.337,"15min":0.85999999999999999},"min":{"1min":0.35699999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":3.258,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.441},{"interface":"front","average":{"1min":1.343,"5min":1.4339999999999999,"15min":0.92600000000000005},"min":{"1min":0.46899999999999997,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":3.9140000000000001,"5min":15.951000000000001,"15min":15.951000000000001},"last":0.72099999999999997}]}]},{"osd":3,"up_from":27,"seq":115964117098,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1093924,"kb_used_data":819736,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92651228,"statfs":{"total":95995035648,"available":94874857472,"internally_reserved":0,"allocated":839409664,"data_stored":839049089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":280689097},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":2.2490000000000001,"15min":1.0229999999999999},"min":{"1min":0.20300000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":6.2199999999999998,"5min":63.43,"15min":63.43},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.754,"5min":2.2490000000000001,"15min":1.048},"min":{"1min":0.25600000000000001,"5min":0.186,"15min":0.186},"max":{"1min":4.5700000000000003,"5min":63.048999999999999,"15min":63.048999999999999},"last":2.2010000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.609,"15min":0.85499999999999998},"min":{"1min":0.23899999999999999,"5min":0.217,"15min":0.217},"max":{"1min":4.5979999999999999,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":4.0149999999999997},{"interface":"front","average":{"1min":1.286,"5min":2.2290000000000001,"15min":1.0489999999999999},"min":{"1min":0.22800000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":6.7130000000000001,"5min":12.087999999999999,"15min":12.087999999999999},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:10:58 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":2.484,"15min":1.1579999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":6.1760000000000002,"5min":62.902999999999999,"15min":62.902999999999999},"last":3.7200000000000002},{"interface":"front","average":{"1min":0.96999999999999997,"5min":2.2959999999999998,"15min":1.0860000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":7.992,"5min":63.468000000000004,"15min":63.468000000000004},"last":3.4990000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:10:21 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.302,"15min":0.80800000000000005},"min":{"1min":0.40200000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.0760000000000001,"5min":7.7229999999999999,"15min":7.7229999999999999},"last":3.5339999999999998},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.399,"15min":0.86099999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.8039999999999998,"5min":8.7059999999999995,"15min":8.7059999999999995},"last":2.1379999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:10:51 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":1.294,"15min":0.82599999999999996},"min":{"1min":0.28000000000000003,"5min":0.255,"15min":0.255},"max":{"1min":1.792,"5min":7.6680000000000001,"15min":7.6680000000000001},"last":3.5699999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":1.444,"15min":0.88700000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":4.5279999999999996,"5min":8.734,"15min":8.734},"last":2.0209999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:11:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":1.454,"15min":0.88400000000000001},"min":{"1min":0.39400000000000002,"5min":0.372,"15min":0.36499999999999999},"max":{"1min":2.2360000000000002,"5min":8.9429999999999996,"15min":8.9429999999999996},"last":2.0640000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.476,"15min":0.88},"min":{"1min":0.39300000000000002,"5min":0.308,"15min":0.308},"max":{"1min":2.6949999999999998,"5min":7.8639999999999999,"15min":7.8639999999999999},"last":0.54400000000000004}]},{"osd":7,"last update":"Mon Feb 5 08:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.3440000000000001,"15min":0.84299999999999997},"min":{"1min":0.46999999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.1469999999999998,"5min":7.7000000000000002,"15min":7.7000000000000002},"last":3.6070000000000002},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.52,"15min":0.92800000000000005},"min":{"1min":0.39700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":4.4450000000000003,"5min":8.9900000000000002,"15min":8.9900000000000002},"last":2.105}]}]},{"osd":4,"up_from":32,"seq":137438953574,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":879644,"kb_used_data":749908,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92865508,"statfs":{"total":95995035648,"available":95094280192,"internally_reserved":0,"allocated":767905792,"data_stored":767556279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:11:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":2.2509999999999999,"15min":1.115},"min":{"1min":0.29499999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.76700000000000002,"5min":73.078999999999994,"15min":73.078999999999994},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":2.0950000000000002,"15min":1.0620000000000001},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.0089999999999999,"5min":73.412000000000006,"15min":73.412000000000006},"last":0.81399999999999995}]},{"osd":1,"last update":"Mon Feb 5 08:11:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.98399999999999999,"15min":0.68899999999999995},"min":{"1min":0.39300000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.181,"5min":8.3409999999999993,"15min":8.3409999999999993},"last":1.859},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.78100000000000003,"15min":0.623},"min":{"1min":0.39100000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":3.367,"5min":4.3120000000000003,"15min":4.3120000000000003},"last":0.249}]},{"osd":2,"last update":"Mon Feb 5 08:11:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":1.883,"15min":0.98899999999999999},"min":{"1min":0.32200000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.119,"5min":96.849999999999994,"15min":96.849999999999994},"last":4.1180000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.9770000000000001,"15min":1.04},"min":{"1min":0.309,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.9510000000000001,"5min":96.143000000000001,"15min":96.143000000000001},"last":4.1299999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:11:14 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":2.5609999999999999,"15min":1.22},"min":{"1min":0.33700000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.82999999999999996,"5min":73.602999999999994,"15min":73.602999999999994},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.63200000000000001,"5min":2.5830000000000002,"15min":1.2270000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.2949999999999999,"5min":74.817999999999998,"15min":74.817999999999998},"last":0.68000000000000005}]},{"osd":5,"last update":"Mon Feb 5 08:10:52 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.40200000000000002,"15min":0.38500000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.23999999999999999},"max":{"1min":0.70099999999999996,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":1.369},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.45200000000000001,"15min":0.41899999999999998},"min":{"1min":0.30599999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.91000000000000003,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.40799999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:11:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.46400000000000002,"15min":0.443},"min":{"1min":0.33900000000000002,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":0.77100000000000002,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.46800000000000003,"15min":0.442},"min":{"1min":0.35399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.752,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.47699999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46999999999999997,"15min":0.47299999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.95199999999999996,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.50800000000000001,"15min":0.505},"min":{"1min":0.33500000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.437}]}]},{"osd":5,"up_from":42,"seq":180388626528,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1322316,"kb_used_data":1192580,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92422836,"statfs":{"total":95995035648,"available":94640984064,"internally_reserved":0,"allocated":1221201920,"data_stored":1220868764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:10:46 2024","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":2.3919999999999999,"15min":1.1639999999999999},"min":{"1min":0.249,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":9.4659999999999993,"5min":143.06299999999999,"15min":143.06299999999999},"last":1.794},{"interface":"front","average":{"1min":1.075,"5min":2.395,"15min":1.1539999999999999},"min":{"1min":0.23699999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.5330000000000004,"5min":143.09700000000001,"15min":143.09700000000001},"last":2.5710000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.77900000000000003,"15min":0.63900000000000001},"min":{"1min":0.252,"5min":0.215,"15min":0.215},"max":{"1min":2.71,"5min":3.0449999999999999,"15min":3.0449999999999999},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.5549999999999999,"5min":0.96499999999999997,"15min":0.68200000000000005},"min":{"1min":0.29899999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":7.0949999999999998,"5min":7.0949999999999998,"15min":7.0949999999999998},"last":3.0659999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:10:46 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.069,"15min":0.73599999999999999},"min":{"1min":0.27600000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":10.733000000000001,"5min":10.733000000000001,"15min":10.733000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.109,"15min":0.747},"min":{"1min":0.28399999999999997,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.8899999999999997,"5min":5.4089999999999998,"15min":5.4089999999999998},"last":0.624}]},{"osd":3,"last update":"Mon Feb 5 08:10:46 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":2.5139999999999998,"15min":1.2110000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":4.5599999999999996,"5min":143.03399999999999,"15min":143.03399999999999},"last":3.2839999999999998},{"interface":"front","average":{"1min":1.8660000000000001,"5min":2.464,"15min":1.2150000000000001},"min":{"1min":0.32600000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":5.4509999999999996,"5min":143.46799999999999,"15min":143.46799999999999},"last":0.71399999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.44600000000000001,"15min":0.41299999999999998},"min":{"1min":0.28000000000000003,"5min":0.255,"15min":0.253},"max":{"1min":0.70999999999999996,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.45800000000000002,"15min":0.432},"min":{"1min":0.29699999999999999,"5min":0.26900000000000002,"15min":0.26500000000000001},"max":{"1min":0.751,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.56599999999999995}]},{"osd":6,"last update":"Mon Feb 5 08:11:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46500000000000002,"15min":0.45300000000000001},"min":{"1min":0.38900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.50900000000000001,"15min":0.496},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.68000000000000005}]},{"osd":7,"last update":"Mon Feb 5 08:10:40 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.49299999999999999,"15min":0.47699999999999998},"min":{"1min":0.34399999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51600000000000001,"15min":0.49199999999999999},"min":{"1min":0.35399999999999998,"5min":0.33700000000000002,"15min":0.28199999999999997},"max":{"1min":0.78900000000000003,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":860028928,"data_stored":860028928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":847773696,"data_stored":847773696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1086717952,"data_stored":1086717952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":844365824,"data_stored":844365824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":779157504,"data_stored":779157504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1239482368,"data_stored":1239482368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":781647872,"data_stored":781647872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:11:30.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-02-05T08:11:30.672 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:30.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:30.775 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:30.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:30.829 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:30.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:30.930 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:30.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:30.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:30.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:31.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:31.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:31.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:31.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:31.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:31.231 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:31.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:31.290 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:31 smithi134 ceph-mon[30086]: pgmap v273: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.09k op/s; 5 B/s, 0 objects/s recovering 2024-02-05T08:11:31.291 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:31 smithi134 ceph-mon[30086]: 5.e scrub starts 2024-02-05T08:11:31.291 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:31 smithi134 ceph-mon[30086]: 5.e scrub ok 2024-02-05T08:11:31.291 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:31 smithi134 ceph-mon[30086]: 4.14 scrub starts 2024-02-05T08:11:31.291 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:31 smithi134 ceph-mon[30086]: 4.14 scrub ok 2024-02-05T08:11:31.291 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:31 smithi134 ceph-mon[39278]: pgmap v273: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.09k op/s; 5 B/s, 0 objects/s recovering 2024-02-05T08:11:31.291 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:31 smithi134 ceph-mon[39278]: 5.e scrub starts 2024-02-05T08:11:31.291 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:31 smithi134 ceph-mon[39278]: 5.e scrub ok 2024-02-05T08:11:31.291 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:31 smithi134 ceph-mon[39278]: 4.14 scrub starts 2024-02-05T08:11:31.291 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:31 smithi134 ceph-mon[39278]: 4.14 scrub ok 2024-02-05T08:11:31.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:31.374 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:31.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:31.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:31.453 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 22 16 43900 43884 124.629 93.6875 0.0106207 0.00800929 2024-02-05T08:11:31.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:31.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:31.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:31 smithi196 ceph-mon[33018]: pgmap v273: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.09k op/s; 5 B/s, 0 objects/s recovering 2024-02-05T08:11:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:31 smithi196 ceph-mon[33018]: 5.e scrub starts 2024-02-05T08:11:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:31 smithi196 ceph-mon[33018]: 5.e scrub ok 2024-02-05T08:11:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:31 smithi196 ceph-mon[33018]: 4.14 scrub starts 2024-02-05T08:11:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:31 smithi196 ceph-mon[33018]: 4.14 scrub ok 2024-02-05T08:11:31.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:31.637 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:31.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:31.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:31.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:31.767 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:31.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:31.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:31.869 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:31.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:31.896 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:31.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:31.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:32.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:32.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:32.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:32.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:32.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:32.274 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:32.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:32.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:32.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:32.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:32.452 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 23 16 45414 45398 123.324 94.625 0.0141535 0.00810228 2024-02-05T08:11:32.462 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:32.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:32.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[30086]: 4.f scrub starts 2024-02-05T08:11:32.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[30086]: 4.f scrub ok 2024-02-05T08:11:32.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[30086]: 3.9 deep-scrub starts 2024-02-05T08:11:32.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[30086]: 3.9 deep-scrub ok 2024-02-05T08:11:32.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[30086]: 6.16 scrub starts 2024-02-05T08:11:32.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[30086]: 3.e scrub starts 2024-02-05T08:11:32.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[30086]: 3.e scrub ok 2024-02-05T08:11:32.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[30086]: 6.16 scrub ok 2024-02-05T08:11:32.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[39278]: 4.f scrub starts 2024-02-05T08:11:32.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[39278]: 4.f scrub ok 2024-02-05T08:11:32.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[39278]: 3.9 deep-scrub starts 2024-02-05T08:11:32.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[39278]: 3.9 deep-scrub ok 2024-02-05T08:11:32.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[39278]: 6.16 scrub starts 2024-02-05T08:11:32.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[39278]: 3.e scrub starts 2024-02-05T08:11:32.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[39278]: 3.e scrub ok 2024-02-05T08:11:32.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:32 smithi134 ceph-mon[39278]: 6.16 scrub ok 2024-02-05T08:11:32.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:32 smithi196 ceph-mon[33018]: 4.f scrub starts 2024-02-05T08:11:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:32 smithi196 ceph-mon[33018]: 4.f scrub ok 2024-02-05T08:11:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:32 smithi196 ceph-mon[33018]: 3.9 deep-scrub starts 2024-02-05T08:11:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:32 smithi196 ceph-mon[33018]: 3.9 deep-scrub ok 2024-02-05T08:11:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:32 smithi196 ceph-mon[33018]: 6.16 scrub starts 2024-02-05T08:11:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:32 smithi196 ceph-mon[33018]: 3.e scrub starts 2024-02-05T08:11:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:32 smithi196 ceph-mon[33018]: 3.e scrub ok 2024-02-05T08:11:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:32 smithi196 ceph-mon[33018]: 6.16 scrub ok 2024-02-05T08:11:32.605 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:32.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:32.707 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:32.748 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:32.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:32.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:32.879 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:32.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:32.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:33.018 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:33.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:33.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:33.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:11:33.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:33.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:33.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:33.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:33.305 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:33.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:33.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:33.348 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[30086]: pgmap v274: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 4 B/s, 0 objects/s recovering 2024-02-05T08:11:33.348 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[30086]: 3.14 scrub starts 2024-02-05T08:11:33.348 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[30086]: 3.14 scrub ok 2024-02-05T08:11:33.348 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[30086]: 5.15 scrub starts 2024-02-05T08:11:33.348 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[30086]: 5.15 scrub ok 2024-02-05T08:11:33.348 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[30086]: 4.10 scrub starts 2024-02-05T08:11:33.349 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[30086]: 4.10 scrub ok 2024-02-05T08:11:33.349 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:11:33.349 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:11:33.349 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[39278]: pgmap v274: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 4 B/s, 0 objects/s recovering 2024-02-05T08:11:33.349 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[39278]: 3.14 scrub starts 2024-02-05T08:11:33.349 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[39278]: 3.14 scrub ok 2024-02-05T08:11:33.349 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[39278]: 5.15 scrub starts 2024-02-05T08:11:33.349 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[39278]: 5.15 scrub ok 2024-02-05T08:11:33.349 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[39278]: 4.10 scrub starts 2024-02-05T08:11:33.349 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[39278]: 4.10 scrub ok 2024-02-05T08:11:33.349 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:11:33.349 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:33 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:11:33.407 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:33.453 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 24 16 46798 46782 121.789 86.5 0.00932581 0.00820495 2024-02-05T08:11:33.455 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:33.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:33.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:33 smithi196 ceph-mon[33018]: pgmap v274: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.3 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1.1 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 4 B/s, 0 objects/s recovering 2024-02-05T08:11:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:33 smithi196 ceph-mon[33018]: 3.14 scrub starts 2024-02-05T08:11:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:33 smithi196 ceph-mon[33018]: 3.14 scrub ok 2024-02-05T08:11:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:33 smithi196 ceph-mon[33018]: 5.15 scrub starts 2024-02-05T08:11:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:33 smithi196 ceph-mon[33018]: 5.15 scrub ok 2024-02-05T08:11:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:33 smithi196 ceph-mon[33018]: 4.10 scrub starts 2024-02-05T08:11:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:33 smithi196 ceph-mon[33018]: 4.10 scrub ok 2024-02-05T08:11:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:33 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:11:33.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:33 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:11:33.565 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:11:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:11:33.593 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:33.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:33.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:33.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:33.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:33.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:33.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:33.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:33.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:33.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:34.006 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:34.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:34.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:34.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:34.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:34.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:34.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:34.181 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:11:34.181 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:11:34.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:34.267 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:34.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:34.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:34.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:34.385 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:34.395 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:34.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:34.455 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 25 16 47939 47923 119.759 71.3125 0.0240093 0.00834304 2024-02-05T08:11:34.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:34 smithi134 ceph-mon[30086]: 4.1b scrub starts 2024-02-05T08:11:34.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:34 smithi134 ceph-mon[30086]: 4.1b scrub ok 2024-02-05T08:11:34.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:34 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:34.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:34 smithi134 ceph-mon[30086]: osdmap e169: 8 total, 8 up, 7 in 2024-02-05T08:11:34.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:34 smithi134 ceph-mon[30086]: 4.1c scrub starts 2024-02-05T08:11:34.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:34 smithi134 ceph-mon[30086]: 4.1c scrub ok 2024-02-05T08:11:34.466 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:34 smithi134 ceph-mon[39278]: 4.1b scrub starts 2024-02-05T08:11:34.466 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:34 smithi134 ceph-mon[39278]: 4.1b scrub ok 2024-02-05T08:11:34.466 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:34 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:34.466 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:34 smithi134 ceph-mon[39278]: osdmap e169: 8 total, 8 up, 7 in 2024-02-05T08:11:34.466 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:34 smithi134 ceph-mon[39278]: 4.1c scrub starts 2024-02-05T08:11:34.466 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:34 smithi134 ceph-mon[39278]: 4.1c scrub ok 2024-02-05T08:11:34.480 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:34.497 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:34.556 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:34.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:34 smithi196 ceph-mon[33018]: 4.1b scrub starts 2024-02-05T08:11:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:34 smithi196 ceph-mon[33018]: 4.1b scrub ok 2024-02-05T08:11:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:34 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:34 smithi196 ceph-mon[33018]: osdmap e169: 8 total, 8 up, 7 in 2024-02-05T08:11:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:34 smithi196 ceph-mon[33018]: 4.1c scrub starts 2024-02-05T08:11:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:34 smithi196 ceph-mon[33018]: 4.1c scrub ok 2024-02-05T08:11:34.658 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:34.698 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:34.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:34.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:34.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:34.844 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:34.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:34.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:34.945 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:34.986 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:34.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:34.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:35.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:35.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:35.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:35.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:35.232 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:11:35.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:35.263 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:35.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:35.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:35.333 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[30086]: pgmap v276: 148 pgs: 148 active+clean; 2.8 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.65k op/s; 4 B/s, 0 objects/s recovering 2024-02-05T08:11:35.333 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[30086]: 6.1d scrub starts 2024-02-05T08:11:35.333 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[30086]: 6.1d scrub ok 2024-02-05T08:11:35.333 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[30086]: 3.1 scrub starts 2024-02-05T08:11:35.333 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[30086]: 3.1 scrub ok 2024-02-05T08:11:35.333 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2063940841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:11:35.333 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[30086]: 6.1f scrub starts 2024-02-05T08:11:35.333 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[30086]: 6.1f scrub ok 2024-02-05T08:11:35.334 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[39278]: pgmap v276: 148 pgs: 148 active+clean; 2.8 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.65k op/s; 4 B/s, 0 objects/s recovering 2024-02-05T08:11:35.334 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[39278]: 6.1d scrub starts 2024-02-05T08:11:35.334 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[39278]: 6.1d scrub ok 2024-02-05T08:11:35.334 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[39278]: 3.1 scrub starts 2024-02-05T08:11:35.334 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[39278]: 3.1 scrub ok 2024-02-05T08:11:35.334 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2063940841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:11:35.334 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[39278]: 6.1f scrub starts 2024-02-05T08:11:35.334 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:35 smithi134 ceph-mon[39278]: 6.1f scrub ok 2024-02-05T08:11:35.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:35.404 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:35.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:35.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:35.455 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 26 16 49188 49172 118.156 78.0625 0.00532818 0.00845896 2024-02-05T08:11:35.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:35.533 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:35.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:35 smithi196 ceph-mon[33018]: pgmap v276: 148 pgs: 148 active+clean; 2.8 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.65k op/s; 4 B/s, 0 objects/s recovering 2024-02-05T08:11:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:35 smithi196 ceph-mon[33018]: 6.1d scrub starts 2024-02-05T08:11:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:35 smithi196 ceph-mon[33018]: 6.1d scrub ok 2024-02-05T08:11:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:35 smithi196 ceph-mon[33018]: 3.1 scrub starts 2024-02-05T08:11:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:35 smithi196 ceph-mon[33018]: 3.1 scrub ok 2024-02-05T08:11:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:35 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2063940841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:11:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:35 smithi196 ceph-mon[33018]: 6.1f scrub starts 2024-02-05T08:11:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:35 smithi196 ceph-mon[33018]: 6.1f scrub ok 2024-02-05T08:11:35.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:35.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:35.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:35.796 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:35.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:35.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:35.933 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:35.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:35.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:36.035 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:36.077 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:36.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:36.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:36.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:36.218 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:36.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:36.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:36.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:36.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:36.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:36.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:36.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:36.455 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 27 15 50703 50688 117.289 94.75 0.0135618 0.00852007 2024-02-05T08:11:36.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:36.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:36.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:36.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:36 smithi134 ceph-mon[30086]: 4.4 scrub starts 2024-02-05T08:11:36.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:36 smithi134 ceph-mon[30086]: 4.4 scrub ok 2024-02-05T08:11:36.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:36 smithi134 ceph-mon[30086]: 6.17 scrub starts 2024-02-05T08:11:36.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:36 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:11:36.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2063940841' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:11:36.511 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:36 smithi134 ceph-mon[30086]: osdmap e170: 8 total, 8 up, 7 in 2024-02-05T08:11:36.511 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:36 smithi134 ceph-mon[39278]: 4.4 scrub starts 2024-02-05T08:11:36.511 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:36 smithi134 ceph-mon[39278]: 4.4 scrub ok 2024-02-05T08:11:36.511 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:36 smithi134 ceph-mon[39278]: 6.17 scrub starts 2024-02-05T08:11:36.511 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:36 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:11:36.511 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2063940841' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:11:36.511 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:36 smithi134 ceph-mon[39278]: osdmap e170: 8 total, 8 up, 7 in 2024-02-05T08:11:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:36 smithi196 ceph-mon[33018]: 4.4 scrub starts 2024-02-05T08:11:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:36 smithi196 ceph-mon[33018]: 4.4 scrub ok 2024-02-05T08:11:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:36 smithi196 ceph-mon[33018]: 6.17 scrub starts 2024-02-05T08:11:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:36 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:11:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2063940841' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:11:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:36 smithi196 ceph-mon[33018]: osdmap e170: 8 total, 8 up, 7 in 2024-02-05T08:11:36.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:36.648 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:36.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:36.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:36.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:36.782 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:36.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:36.884 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:36.923 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:36.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:36.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:36.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T08:11:37.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:37.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:37.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:37.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:37.154 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:37.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:37.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:37.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:37.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:37.313 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:37.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:37.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:37.414 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:37.457 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 28 16 52112 52096 116.236 88 0.0209839 0.00859841 2024-02-05T08:11:37.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:37.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:37.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:37 smithi134 ceph-mon[30086]: pgmap v278: 148 pgs: 148 active+clean; 2.8 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 103 MiB/s wr, 1.64k op/s; 5 B/s, 0 objects/s recovering 2024-02-05T08:11:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:37 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:11:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:37 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:11:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:37 smithi134 ceph-mon[39278]: pgmap v278: 148 pgs: 148 active+clean; 2.8 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 103 MiB/s wr, 1.64k op/s; 5 B/s, 0 objects/s recovering 2024-02-05T08:11:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:37 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:11:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:37 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:11:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:37 smithi196 ceph-mon[33018]: pgmap v278: 148 pgs: 148 active+clean; 2.8 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 103 MiB/s wr, 1.64k op/s; 5 B/s, 0 objects/s recovering 2024-02-05T08:11:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:37 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:11:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:37 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:11:37.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:37.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:37.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:37.699 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:37.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:37.745 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:37.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:37.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:37.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:37.889 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:37.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:37.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:38.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:11:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:11:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:11:38.034 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:38.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:38.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:38.164 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:38.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:38.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:38.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:38.294 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:38.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:38.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:38.395 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:38.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:38.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:38.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:38.457 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 29 16 53422 53406 115.052 81.875 0.0106247 0.00868611 2024-02-05T08:11:38.529 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:38.558 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:38.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:38.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:38.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:38.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:38.701 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:38.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:38.802 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:38.843 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:38.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:38.945 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:38.984 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:38.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:39.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:39.115 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:39.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:39.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:39.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:39.251 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:39.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:39.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:39.300 INFO:teuthology.orchestra.run.smithi134.stderr:set pool 1 pgp_num to 1 2024-02-05T08:11:39.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:39.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:39.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:39.457 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 30 16 54810 54794 114.108 86.75 0.00589654 0.00875802 2024-02-05T08:11:39.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:39.513 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:39.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:39 smithi196 ceph-mon[33018]: pgmap v279: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T08:11:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:39 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:11:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:39 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:11:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:39 smithi196 ceph-mon[33018]: osdmap e171: 8 total, 8 up, 7 in 2024-02-05T08:11:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:39 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2865754759' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-05T08:11:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:39 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-05T08:11:39.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:39.666 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:39.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:39.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:39.670 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:39 smithi134 ceph-mon[30086]: pgmap v279: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T08:11:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:39 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:11:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:39 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:11:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:39 smithi134 ceph-mon[30086]: osdmap e171: 8 total, 8 up, 7 in 2024-02-05T08:11:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:39 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2865754759' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-05T08:11:39.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:39 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-05T08:11:39.671 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:39 smithi134 ceph-mon[39278]: pgmap v279: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.0 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T08:11:39.671 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:39 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:11:39.671 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:39 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:11:39.671 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:39 smithi134 ceph-mon[39278]: osdmap e171: 8 total, 8 up, 7 in 2024-02-05T08:11:39.671 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:39 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2865754759' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-05T08:11:39.671 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:39 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-02-05T08:11:39.768 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:39.807 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:39.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:39.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:39.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:39.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:39.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:40.041 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:40.077 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:40.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:40.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:40.150 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:40.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:40.230 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:40.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:40.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:40.332 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:40.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:40.381 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:40.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:40.457 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 31 16 56178 56162 113.185 85.5 0.0105694 0.00882921 2024-02-05T08:11:40.460 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:40 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-02-05T08:11:40.460 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:40 smithi134 ceph-mon[30086]: osdmap e172: 8 total, 8 up, 7 in 2024-02-05T08:11:40.460 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:40 smithi134 ceph-mon[30086]: osdmap e173: 8 total, 8 up, 7 in 2024-02-05T08:11:40.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:40 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-02-05T08:11:40.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:40 smithi134 ceph-mon[39278]: osdmap e172: 8 total, 8 up, 7 in 2024-02-05T08:11:40.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:40 smithi134 ceph-mon[39278]: osdmap e173: 8 total, 8 up, 7 in 2024-02-05T08:11:40.482 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:40.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:40.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:40 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-02-05T08:11:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:40 smithi196 ceph-mon[33018]: osdmap e172: 8 total, 8 up, 7 in 2024-02-05T08:11:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:40 smithi196 ceph-mon[33018]: osdmap e173: 8 total, 8 up, 7 in 2024-02-05T08:11:40.632 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:40.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:40.674 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:40.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:40.775 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:40.824 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:40.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:40.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:40.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:40.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:41.014 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:11:41.055 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:41.058 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:11:41.098 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:41.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:41.107 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:11:41.148 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:11:41.189 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:11:41.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:41.228 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:11:41.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:41.241 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:41.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:41.265 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:11:41.305 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:11:41.311 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:11:41.311 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:11:41.311 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:11:41.311 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:11:41.311 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:11:41.311 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:11:41.311 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:11:41.312 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:11:41.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:41.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:41.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:41.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:41.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:11:41.459 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 32 16 57425 57409 112.078 77.9375 0.0158399 0.00891593 2024-02-05T08:11:41.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:41.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:41.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:41.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:41 smithi134 ceph-mon[30086]: pgmap v282: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-02-05T08:11:41.551 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:41 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/602755256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:41.552 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:41 smithi134 ceph-mon[39278]: pgmap v282: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-02-05T08:11:41.552 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:41 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/602755256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:41.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:41 smithi196 ceph-mon[33018]: pgmap v282: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-02-05T08:11:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:41 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/602755256' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:41.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:41.657 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:41.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:41.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:41.781 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:41.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:41.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:41.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:41.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:41.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:41.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:42.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:42.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:42.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:42.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:42.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:42.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:42.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:42.276 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:42.320 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:42.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:42.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:42.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:42.451 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:42.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:42.459 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 33 16 58603 58587 110.913 73.625 0.00935782 0.00901175 2024-02-05T08:11:42.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:42.583 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:42.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:42.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:42.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:42.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:42.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:42.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:42.816 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:42.861 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:42.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:42.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:43.025 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:43.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:43.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:43.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:43.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:43.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:43.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:43.297 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:43.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:43.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:43.443 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:43.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:43.459 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 34 16 59884 59868 110.006 80.0625 0.00880312 0.00908467 2024-02-05T08:11:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:43 smithi134 ceph-mon[30086]: pgmap v284: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-02-05T08:11:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:43 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:11:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:43 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:11:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:43 smithi134 ceph-mon[39278]: pgmap v284: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-02-05T08:11:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:43 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:11:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:43 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:11:43.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:43 smithi196 ceph-mon[33018]: pgmap v284: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-02-05T08:11:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:43 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:11:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:43 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:11:43.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:11:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:11:43.576 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:43.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:43.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:43.678 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:43.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:43.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:43.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:43.839 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:43.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:43.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:43.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:43.986 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:43.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:44.087 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:44.117 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:44.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:44.219 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:44.245 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:44.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:44.306 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:44.332 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:11:44.332 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:11:44.332 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-02-05T08:11:44.332 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-02-05T08:11:44.332 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:11:44.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:44.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:44.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:44.459 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 35 16 61154 61138 109.131 79.375 0.0101706 0.00915843 2024-02-05T08:11:44.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:44.525 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:44.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:44.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:44.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:44 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:11:44.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:44 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:11:44.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:44 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:44.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:44 smithi134 ceph-mon[30086]: osdmap e174: 8 total, 8 up, 7 in 2024-02-05T08:11:44.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:44 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:11:44.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:44 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:11:44.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:44 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:44.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:44 smithi134 ceph-mon[39278]: osdmap e174: 8 total, 8 up, 7 in 2024-02-05T08:11:44.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:44.658 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:44.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:44.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:44.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:44.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:44.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:44.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:44 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:11:44.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:44 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:11:44.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:44 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:44.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:44 smithi196 ceph-mon[33018]: osdmap e174: 8 total, 8 up, 7 in 2024-02-05T08:11:44.889 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:44.946 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:44.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:45.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:45.074 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:45.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:45.134 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:11:45.134 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:11:45.134 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-05T08:11:45.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T08:11:45.176 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:45.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:45.240 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:45.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:45.341 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:45.399 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:45.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:45.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:45.460 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 36 16 62195 62179 107.905 65.0625 0.047065 0.0092627 2024-02-05T08:11:45.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:45 smithi134 ceph-mon[30086]: pgmap v286: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 958 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:11:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:45 smithi134 ceph-mon[30086]: osdmap e175: 8 total, 8 up, 7 in 2024-02-05T08:11:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:45 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:11:45.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:45 smithi134 ceph-mon[39278]: pgmap v286: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 958 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:11:45.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:45 smithi134 ceph-mon[39278]: osdmap e175: 8 total, 8 up, 7 in 2024-02-05T08:11:45.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:45 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:11:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:45.558 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:45.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:45.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:45.701 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:45.701 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:11:45.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:45.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:45.803 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:45.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:45 smithi196 ceph-mon[33018]: pgmap v286: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 958 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:11:45.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:45 smithi196 ceph-mon[33018]: osdmap e175: 8 total, 8 up, 7 in 2024-02-05T08:11:45.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:45 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:11:45.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:45.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:45.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:45.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:45.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:45.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:45.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:46.064 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:46.113 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:46.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:46.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:46.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:46.243 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:46.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:46.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:46.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:46.385 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:46.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:46.460 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 37 16 63369 63353 106.972 73.375 0.0134728 0.00934326 2024-02-05T08:11:46.487 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:46.509 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T08:11:46.567 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:46.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:46.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:46.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:46.696 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:46.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:46.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:46.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:46.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:46.914 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:46.930 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:46.958 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:46.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:47.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:47.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:47.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:47.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:47.243 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:47.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:47.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:47.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:47.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:47.460 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 38 16 64442 64426 105.922 67.0625 0.0219068 0.00943629 2024-02-05T08:11:47.473 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:47.510 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:47.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:47.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:47.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:47.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:47.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:47.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:47 smithi134 ceph-mon[30086]: pgmap v288: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T08:11:47.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:47 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:11:47.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:47 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:11:47.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:47 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:11:47.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:47 smithi134 ceph-mon[39278]: pgmap v288: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T08:11:47.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:47 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:11:47.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:47 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:11:47.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:47 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:11:47.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:47.746 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:47.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:47.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:47.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:47 smithi196 ceph-mon[33018]: pgmap v288: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T08:11:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:47 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:11:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:47 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:11:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:47 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:11:47.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:47.915 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:47.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:48.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:48.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:11:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:11:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:11:48.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:48.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:48.148 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:48.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:48.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:48.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:48.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:48.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:48.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:48.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:48.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:48.462 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 39 16 65655 65639 105.145 75.8125 0.0139123 0.00950434 2024-02-05T08:11:48.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:48.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:48.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:48.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:48.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:48.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:48.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:48.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:48.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:48.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:48.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:48.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:48.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:48.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:48.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:48.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:49.003 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:49.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:49.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:49.133 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:49.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:49.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:49.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:49.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:49.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:49.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:49.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:49.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:49.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:49.462 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:11:49.463292+0000 min lat: 0.00115834 max lat: 0.147925 avg lat: 0.00956404 2024-02-05T08:11:49.462 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:11:49.462 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 40 15 66921 66906 104.496 79.1875 0.0119501 0.00956404 2024-02-05T08:11:49.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:49.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:49.533 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:49.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:49.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:49.681 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:49.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:49.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:49 smithi134 ceph-mon[30086]: pgmap v289: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T08:11:49.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:49 smithi134 ceph-mon[39278]: pgmap v289: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T08:11:49.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:49.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:49 smithi196 ceph-mon[33018]: pgmap v289: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T08:11:49.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:49.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:49.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:49.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:49.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:49.960 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:49.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:50.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:50.089 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:50.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:50.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:50.191 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:50.233 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:50.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:50.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:50.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:50.371 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:50.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:50.462 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 41 16 68119 68103 103.772 74.8125 0.00926676 0.00963154 2024-02-05T08:11:50.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:50.506 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:50.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:50.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:50.637 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:50.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:50.738 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:50.787 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:50.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:50.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:50.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:50.919 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:50.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:51.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:51.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:50 smithi134 ceph-mon[30086]: pgmap v290: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-02-05T08:11:51.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:50 smithi134 ceph-mon[30086]: osdmap e176: 8 total, 8 up, 7 in 2024-02-05T08:11:51.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:50 smithi134 ceph-mon[39278]: pgmap v290: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-02-05T08:11:51.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:50 smithi134 ceph-mon[39278]: osdmap e176: 8 total, 8 up, 7 in 2024-02-05T08:11:51.062 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:51.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:51.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:50 smithi196 ceph-mon[33018]: pgmap v290: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-02-05T08:11:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:50 smithi196 ceph-mon[33018]: osdmap e176: 8 total, 8 up, 7 in 2024-02-05T08:11:51.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:51.190 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:11:51.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:51.213 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:51.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:51.314 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:51.341 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:51.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:51.442 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:51.462 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 42 16 69297 69281 103.055 73.625 0.0160297 0.00969832 2024-02-05T08:11:51.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:51.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:51.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:51.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:51.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:51.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:51.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:51.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:51.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:51.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:51.837 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:51.877 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:51.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:51.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:51.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:52.019 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:52.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:52.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:52.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:52.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:52.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:52.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:52.294 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:52.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:52.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:52.396 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:52.449 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:52.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:52.462 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 43 16 70424 70408 102.296 70.4375 0.0115499 0.0097687 2024-02-05T08:11:52.551 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:52.591 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:11:52.600 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:52.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:52.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:52.640 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:11:52.689 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:11:52.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:52.730 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:52.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:52.757 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:11:52.804 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:11:52.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:52.852 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:11:52.852 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:52 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:11:52.852 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:52 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:11:52.852 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:52 smithi134 ceph-mon[39278]: pgmap v292: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s 2024-02-05T08:11:52.853 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:52 smithi134 ceph-mon[39278]: osdmap e177: 8 total, 8 up, 7 in 2024-02-05T08:11:52.853 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:52 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:11:52.853 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:52 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/752465201' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:52.853 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:52 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:11:52.853 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:52 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:11:52.853 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:52 smithi134 ceph-mon[30086]: pgmap v292: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s 2024-02-05T08:11:52.853 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:52 smithi134 ceph-mon[30086]: osdmap e177: 8 total, 8 up, 7 in 2024-02-05T08:11:52.853 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:52 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:11:52.853 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:52 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/752465201' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:52.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:52.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:52.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:52.891 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:11:52.931 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:11:52.933 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:11:52.933 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:11:52.933 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:11:52.933 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:11:52.933 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:11:52.933 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:11:52.934 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:11:52.934 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:11:52.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:53.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:53.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:52 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:11:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:52 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:11:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:52 smithi196 ceph-mon[33018]: pgmap v292: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 45 MiB/s wr, 725 op/s 2024-02-05T08:11:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:52 smithi196 ceph-mon[33018]: osdmap e177: 8 total, 8 up, 7 in 2024-02-05T08:11:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:52 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:11:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:52 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/752465201' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:11:53.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:53.135 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:53.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:53.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:53.267 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:53.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:53.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:53.369 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:53.411 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:53.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:53.462 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 44 16 71432 71416 101.403 63 0.0266407 0.00985644 2024-02-05T08:11:53.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:53.541 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:53.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:53.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:11:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:11:53.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:53.672 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:53.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:53.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:53.677 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:11:53.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:53.812 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:53.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:53.913 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:53.951 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:53.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:54.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:53 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4245775302' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:11:54.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:53 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:54.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:53 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4245775302' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:11:54.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:53 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:54.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:54.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:53 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4245775302' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:11:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:53 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:11:54.080 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:54.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:54.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:54.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:54.221 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:54.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:54.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:54.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:54.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:54.351 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:54.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:54.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:54.462 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 45 16 72514 72498 100.653 67.625 0.00683454 0.0099292 2024-02-05T08:11:54.494 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:54.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:54.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:54.596 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:54.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:54.637 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:54.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:54.738 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:54.781 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:54.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:54.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:54.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:54.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:55.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:55.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:54 smithi134 ceph-mon[30086]: pgmap v294: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T08:11:55.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:54 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:11:55.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:54 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4245775302' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:11:55.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:54 smithi134 ceph-mon[30086]: osdmap e178: 8 total, 8 up, 7 in 2024-02-05T08:11:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:54 smithi134 ceph-mon[39278]: pgmap v294: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T08:11:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:54 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:11:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:54 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4245775302' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:11:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:54 smithi134 ceph-mon[39278]: osdmap e178: 8 total, 8 up, 7 in 2024-02-05T08:11:55.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:55.051 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:55.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:54 smithi196 ceph-mon[33018]: pgmap v294: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T08:11:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:54 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:11:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:54 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4245775302' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:11:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:54 smithi196 ceph-mon[33018]: osdmap e178: 8 total, 8 up, 7 in 2024-02-05T08:11:55.152 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:55.183 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:55.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:55.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:55.284 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:55.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:55.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:55.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:55.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:55.450 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:55.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:55.463 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 46 16 73599 73583 99.9388 67.8125 0.00600761 0.0100015 2024-02-05T08:11:55.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:55.576 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:55.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:55.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:55.723 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:55.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:55.824 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:55.862 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:55.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:55.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:11:56.001 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:56.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:11:56.102 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:56.145 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:56.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:56.245 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:56.284 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:56.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:56.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:56.414 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:56.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:56.463 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 47 16 74742 74726 99.3324 71.4375 0.00571222 0.0100606 2024-02-05T08:11:56.516 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:56.541 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:56.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:56.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:56.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:56.670 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:56.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:56.772 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:56.812 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:56.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:56.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:56.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:56.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:56.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:56.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:57.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:56 smithi134 ceph-mon[30086]: pgmap v296: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 976 op/s 2024-02-05T08:11:57.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:56 smithi134 ceph-mon[39278]: pgmap v296: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 976 op/s 2024-02-05T08:11:57.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:56 smithi196 ceph-mon[33018]: pgmap v296: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 976 op/s 2024-02-05T08:11:57.100 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:57.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:57.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:57.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:57.258 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:57.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:57.360 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:57.410 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:57.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:57.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:57.463 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 48 15 75775 75760 98.6095 64.625 0.0068148 0.0101356 2024-02-05T08:11:57.512 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:57.555 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:57.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:57.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:57.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:11:57.685 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:57.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:57.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:11:57.787 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:57.829 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:57.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:57.931 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:57.973 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:57.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:57.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:58.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:11:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:11:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:11:58.076 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:58.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:11:58.118 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:58.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:58.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:58.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:58.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:58.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:58.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:58.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:58.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:58.464 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 49 16 76881 76865 98.0065 69.0625 0.0073495 0.0101981 2024-02-05T08:11:58.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:58.505 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:58.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:58.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:11:58.646 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:58.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:58.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:11:58.749 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:58.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:58.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:58.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:58.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:11:58.912 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:58.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:58.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:11:58.970 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:11:58 smithi134 ceph-mon[30086]: pgmap v297: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 61 MiB/s wr, 984 op/s 2024-02-05T08:11:58.970 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:11:58 smithi134 ceph-mon[39278]: pgmap v297: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 61 MiB/s wr, 984 op/s 2024-02-05T08:11:59.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:59.053 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:59.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:11:58 smithi196 ceph-mon[33018]: pgmap v297: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 61 MiB/s wr, 984 op/s 2024-02-05T08:11:59.154 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:59.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:59.198 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:59.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:59.298 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:11:59.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:59.341 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:59.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:11:59.442 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:11:59.463 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 50 16 77929 77913 97.3564 65.5 0.0300677 0.0102666 2024-02-05T08:11:59.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:59.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:59.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:11:59.505 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:11:59.505 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:11:59.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:11:59.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:11:59.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:59.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:11:59.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:59.745 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:59.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:11:59.834 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:11:59.834 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:11:59.834 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 178, 2024-02-05T08:11:59.834 INFO:teuthology.orchestra.run.smithi134.stdout: "fsid": "43e13aa4-c3fc-11ee-95b6-87774f69a715", 2024-02-05T08:11:59.834 INFO:teuthology.orchestra.run.smithi134.stdout: "created": "2024-02-05T07:59:09.321262+0000", 2024-02-05T08:11:59.834 INFO:teuthology.orchestra.run.smithi134.stdout: "modified": "2024-02-05T08:11:53.654417+0000", 2024-02-05T08:11:59.834 INFO:teuthology.orchestra.run.smithi134.stdout: "last_up_change": "2024-02-05T08:04:26.849889+0000", 2024-02-05T08:11:59.834 INFO:teuthology.orchestra.run.smithi134.stdout: "last_in_change": "2024-02-05T08:06:38.610101+0000", 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_num": 5806080, 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_set": [ 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "nodeep-scrub", 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "noscrub", 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "pglog_hardlimit", 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "purged_snapdirs", 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_deletes", 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "sortbitwise" 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_version": 18, 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_snapshot": "", 2024-02-05T08:11:59.835 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_max": 8, 2024-02-05T08:11:59.836 INFO:teuthology.orchestra.run.smithi134.stdout: "max_osd": 8, 2024-02-05T08:11:59.836 INFO:teuthology.orchestra.run.smithi134.stdout: "require_min_compat_client": "luminous", 2024-02-05T08:11:59.836 INFO:teuthology.orchestra.run.smithi134.stdout: "min_compat_client": "luminous", 2024-02-05T08:11:59.836 INFO:teuthology.orchestra.run.smithi134.stdout: "require_osd_release": "squid", 2024-02-05T08:11:59.836 INFO:teuthology.orchestra.run.smithi134.stdout: "allow_crimson": false, 2024-02-05T08:11:59.836 INFO:teuthology.orchestra.run.smithi134.stdout: "pools": [ 2024-02-05T08:11:59.836 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.836 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 1, 2024-02-05T08:11:59.836 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": ".mgr", 2024-02-05T08:11:59.836 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:02:21.034012+0000", 2024-02-05T08:11:59.836 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:11:59.836 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:11:59.836 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:11:59.837 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:11:59.837 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:11:59.837 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:11:59.837 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:11:59.837 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:11:59.837 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:11:59.837 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:11:59.837 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:11:59.837 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:11:59.837 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 1, 2024-02-05T08:11:59.837 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 1, 2024-02-05T08:11:59.837 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 1, 2024-02-05T08:11:59.837 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 1, 2024-02-05T08:11:59.838 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 1, 2024-02-05T08:11:59.838 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:11:59.838 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:11:59.838 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:11:59.838 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:11:59.838 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:11:59.838 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:11:59.838 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:11:59.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.838 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "172", 2024-02-05T08:11:59.838 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:11:59.838 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:11:59.838 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:11:59.839 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:11:59.840 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "options": { 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_max": 32, 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_min": 1 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr": {} 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:11:59.841 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:11:59.842 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 7.8899998664855957, 2024-02-05T08:11:59.842 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 7.8899998664855957, 2024-02-05T08:11:59.842 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.37999999523162842, 2024-02-05T08:11:59.842 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 3, 2024-02-05T08:11:59.842 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 3, 2024-02-05T08:11:59.842 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:11:59.842 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:11:59.842 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:11:59.842 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.842 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.842 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 2, 2024-02-05T08:11:59.842 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "datapool", 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:39.400746+0000", 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 8193, 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 3, 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 3, 2024-02-05T08:11:59.843 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 3, 2024-02-05T08:11:59.844 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 3, 2024-02-05T08:11:59.844 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 3, 2024-02-05T08:11:59.844 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:11:59.844 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:11:59.844 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:11:59.844 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:11:59.844 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:11:59.844 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:11:59.844 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:11:59.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.844 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "70", 2024-02-05T08:11:59.844 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:11:59.844 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 2, 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 70, 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:11:59.845 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:11:59.846 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd": {} 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.6700000762939453, 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.6700000762939453, 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.75, 2024-02-05T08:11:59.847 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 2, 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 2, 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 3, 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": ".rgw.root", 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:40.005769+0000", 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:11:59.848 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:11:59.849 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:11:59.849 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:11:59.849 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:11:59.849 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:11:59.849 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:11:59.849 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:11:59.849 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:11:59.849 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:11:59.849 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:11:59.849 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:11:59.849 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:11:59.849 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:11:59.849 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "68", 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:11:59.850 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:11:59.851 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:11:59.852 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:11:59.852 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:11:59.852 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:11:59.852 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:11:59.852 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:11:59.852 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:11:59.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.852 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:11:59.852 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:11:59.852 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:11:59.852 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:11:59.852 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:11:59.852 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.7400000095367432, 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.7400000095367432, 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:11:59.853 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.5299999713897705, 2024-02-05T08:11:59.854 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.5299999713897705, 2024-02-05T08:11:59.854 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:11:59.854 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:11:59.854 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:11:59.854 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.854 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.854 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.854 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 4, 2024-02-05T08:11:59.854 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.log", 2024-02-05T08:11:59.854 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:41.851891+0000", 2024-02-05T08:11:59.854 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:11:59.854 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:11:59.854 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:11:59.855 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:11:59.855 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:11:59.855 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:11:59.855 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:11:59.855 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:11:59.855 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:11:59.855 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:11:59.855 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:11:59.855 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:11:59.855 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:11:59.855 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:11:59.855 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:11:59.855 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "70", 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:11:59.856 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:11:59.857 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:11:59.858 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.4900000095367432, 2024-02-05T08:11:59.859 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.4900000095367432, 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 2.190000057220459, 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 2.190000057220459, 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 5, 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.control", 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:43.793804+0000", 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:11:59.860 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:11:59.861 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "72", 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:11:59.862 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:11:59.863 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:11:59.864 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:11:59.865 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:11:59.865 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:11:59.865 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:11:59.865 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:11:59.865 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:11:59.865 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:11:59.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.865 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:11:59.865 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:11:59.865 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.4900000095367432, 2024-02-05T08:11:59.865 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.4900000095367432, 2024-02-05T08:11:59.865 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:11:59.865 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.309999942779541, 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.309999942779541, 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 6, 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.meta", 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:45.853999+0000", 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:11:59.866 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:11:59.867 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "74", 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:11:59.868 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:11:59.869 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:11:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: "options": { 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_bias": 4 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.7400000095367432, 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.7400000095367432, 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:11:59.871 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.5299999713897705, 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.5299999713897705, 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 8, 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "unique_pool_1", 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:10:53.553654+0000", 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:11:59.872 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 16, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 16, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 16, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 16, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 16, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:11:59.873 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "161", 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:11:59.874 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:11:59.875 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:11:59.876 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:11:59.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.876 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:11:59.876 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:11:59.876 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:11:59.876 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:11:59.876 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:11:59.876 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:11:59.876 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:11:59.876 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:11:59.876 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:11:59.876 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:11:59.876 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: "rados": {} 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.9900000095367432, 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.9900000095367432, 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.75, 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.75, 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:11:59.877 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:11:59.878 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.878 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.878 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:11:59.878 INFO:teuthology.orchestra.run.smithi134.stdout: "osds": [ 2024-02-05T08:11:59.878 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.878 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 0, 2024-02-05T08:11:59.878 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "d90a5ea0-9a19-452e-9564-ca7b7a7c45e4", 2024-02-05T08:11:59.878 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:11:59.878 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:11:59.878 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:11:59.878 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:11:59.878 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:11:59.879 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:11:59.879 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 8, 2024-02-05T08:11:59.879 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 167, 2024-02-05T08:11:59.879 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:11:59.879 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:11:59.879 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:11:59.879 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.879 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.879 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.879 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6802", 2024-02-05T08:11:59.879 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:11:59.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.879 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.880 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.880 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6803", 2024-02-05T08:11:59.880 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:11:59.880 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.880 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.880 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:11:59.880 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.880 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.880 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.880 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6804", 2024-02-05T08:11:59.880 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:11:59.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.881 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.881 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.881 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6805", 2024-02-05T08:11:59.881 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:11:59.881 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.881 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.881 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:11:59.881 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.881 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.881 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.881 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6808", 2024-02-05T08:11:59.881 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6809", 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6806", 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:11:59.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6807", 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6803/2540271081", 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6805/2540271081", 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6809/2540271081", 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6807/2540271081", 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:11:59.883 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 1, 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4", 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 13, 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 157, 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:11:59.884 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6810", 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6811", 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.885 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6812", 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6813", 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:11:59.886 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6816", 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6817", 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:11:59.887 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.888 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.888 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.888 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6814", 2024-02-05T08:11:59.888 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:11:59.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.888 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.888 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.888 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6815", 2024-02-05T08:11:59.888 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:11:59.888 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.888 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.888 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6811/2032802960", 2024-02-05T08:11:59.889 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6813/2032802960", 2024-02-05T08:11:59.889 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6817/2032802960", 2024-02-05T08:11:59.889 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6815/2032802960", 2024-02-05T08:11:59.889 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:11:59.889 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:11:59.889 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:11:59.889 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.889 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.889 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 2, 2024-02-05T08:11:59.889 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708", 2024-02-05T08:11:59.889 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:11:59.889 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 18, 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 97, 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6818", 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:11:59.890 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6819", 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6820", 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.891 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6821", 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6824", 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.892 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.893 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.893 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6825", 2024-02-05T08:11:59.893 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:11:59.893 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.893 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.893 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:11:59.893 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.893 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.893 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.893 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6822", 2024-02-05T08:11:59.893 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:11:59.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.894 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.894 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.894 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6823", 2024-02-05T08:11:59.894 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:11:59.894 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.894 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.894 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6819/153827987", 2024-02-05T08:11:59.894 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6821/153827987", 2024-02-05T08:11:59.894 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6825/153827987", 2024-02-05T08:11:59.894 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6823/153827987", 2024-02-05T08:11:59.894 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 3, 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "6774e330-3202-4671-894e-172cc2f6189d", 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:11:59.895 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 27, 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 163, 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6826", 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6827", 2024-02-05T08:11:59.896 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6828", 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6829", 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.897 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6832", 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6833", 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6830", 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6831", 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.899 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6827/1527507056", 2024-02-05T08:11:59.900 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6829/1527507056", 2024-02-05T08:11:59.900 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6833/1527507056", 2024-02-05T08:11:59.900 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6831/1527507056", 2024-02-05T08:11:59.900 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:11:59.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:11:59.901 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:11:59.901 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:11:59.901 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.901 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.901 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 4, 2024-02-05T08:11:59.901 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693", 2024-02-05T08:11:59.901 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:11:59.902 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:11:59.902 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:11:59.902 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:11:59.902 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:11:59.902 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:11:59.902 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 32, 2024-02-05T08:11:59.902 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 157, 2024-02-05T08:11:59.902 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:11:59.902 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:11:59.902 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:11:59.902 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.902 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.902 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6800", 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6801", 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.903 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6802", 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6803", 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.904 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6806", 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6807", 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.905 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6804", 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6805", 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6801/3005156808", 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6803/3005156808", 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3005156808", 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3005156808", 2024-02-05T08:11:59.906 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 5, 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955", 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:11:59.907 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 42, 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 157, 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6808", 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6809", 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:11:59.908 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.909 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.909 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:11:59.909 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.909 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.909 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.909 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6810", 2024-02-05T08:11:59.909 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:11:59.909 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.909 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.909 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.909 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6811", 2024-02-05T08:11:59.909 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6814", 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6815", 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:11:59.910 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.911 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.911 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:11:59.911 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.911 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.911 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.911 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6812", 2024-02-05T08:11:59.911 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:11:59.911 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.911 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.911 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.911 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6813", 2024-02-05T08:11:59.911 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:11:59.912 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.912 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.912 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6809/1603224792", 2024-02-05T08:11:59.912 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6811/1603224792", 2024-02-05T08:11:59.912 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6815/1603224792", 2024-02-05T08:11:59.912 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6813/1603224792", 2024-02-05T08:11:59.912 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:11:59.912 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:11:59.912 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:11:59.912 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.912 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.912 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 6, 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a", 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 0, 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 0, 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 51, 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 95, 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:11:59.913 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6816", 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6817", 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:11:59.914 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6818", 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6819", 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.915 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6822", 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6823", 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6820", 2024-02-05T08:11:59.916 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6821", 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6817/3727413143", 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6819/3727413143", 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6823/3727413143", 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6821/3727413143", 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:11:59.917 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 7, 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057", 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 60, 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 100, 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:11:59.918 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6824", 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6825", 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:11:59.919 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6826", 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6827", 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:11:59.920 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6830", 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6831", 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:11:59.921 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6828", 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6829", 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6825/4206571010", 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6827/4206571010", 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6831/4206571010", 2024-02-05T08:11:59.922 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6829/4206571010", 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_xinfo": [ 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 0, 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:11:59.923 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:11:19.143195+0000", 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 1, 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:11:05.058869+0000", 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.924 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 2, 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:10:54.740063+0000", 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 3, 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:11:59.925 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:11:32.944359+0000", 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 4, 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:11:43.025464+0000", 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.926 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 5, 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:11:42.970092+0000", 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 6, 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:11:59.927 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:11:59.928 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:11:59.928 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 65536, 2024-02-05T08:11:59.928 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:11:51.290727+0000", 2024-02-05T08:11:59.928 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:11:59.928 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.928 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.928 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 7, 2024-02-05T08:11:59.928 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:11:59.928 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:11:59.928 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:11:59.928 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:11:59.928 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:11:59.928 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:11:38.093946+0000", 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap": [ 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "3.1e", 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: "osds": [ 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: 3, 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: 4, 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: 1 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap_items": [ 2024-02-05T08:11:59.929 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "3.6", 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 4, 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.7", 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:11:59.930 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.a", 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.1a", 2024-02-05T08:11:59.931 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 3, 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.1f", 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:11:59.932 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap_primaries": [], 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_temp": [], 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_temp": [], 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "blocklist": { 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/788812699": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/1046500326": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/464208089": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/301941854": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/3607168099": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/673876301": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/4225724311": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/1525062017": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/3607168099": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:11:59.933 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/673876301": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3200884805": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/2952444615": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3325835814": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3161126612": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/69097091": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/1525062017": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/1645475987": "2024-02-06T08:05:21.465844+0000" 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "range_blocklist": {}, 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profiles": { 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "default": { 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "crush-failure-domain": "osd", 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "k": "2", 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "m": "1", 2024-02-05T08:11:59.934 INFO:teuthology.orchestra.run.smithi134.stdout: "plugin": "jerasure", 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: "technique": "reed_sol_van" 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps_queue": [], 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: "new_removed_snaps": [], 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: "new_purged_snaps": [], 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_node_flags": {}, 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: "device_class_flags": {}, 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode": { 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode_enabled": false, 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_bucket_count": 0, 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: "degraded_stretch_mode": 0, 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: "recovering_stretch_mode": 0, 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode_bucket": 0 2024-02-05T08:11:59.935 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:11:59.936 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:11:59.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:11:59.971 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:11:59.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:00.072 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:00.126 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:00.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:00.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:00.228 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:00.275 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:00.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:00.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:00.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:00.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:00.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:00.463 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 51 16 78997 78981 96.7564 66.75 0.00909759 0.0103305 2024-02-05T08:12:00.508 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:00.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:00.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:00.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:00.639 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:00.680 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:00.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:00.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:00.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:00.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:00.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:00.917 INFO:tasks.thrashosds.thrasher:j is {'epoch': 178, 'fsid': '43e13aa4-c3fc-11ee-95b6-87774f69a715', 'created': '2024-02-05T07:59:09.321262+0000', 'modified': '2024-02-05T08:11:53.654417+0000', 'last_up_change': '2024-02-05T08:04:26.849889+0000', 'last_in_change': '2024-02-05T08:06:38.610101+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T08:02:21.034012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T08:04:39.400746+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T08:04:40.005769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T08:04:41.851891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T08:04:43.793804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T08:04:45.853999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-02-05T08:10:53.553654+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '161', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'd90a5ea0-9a19-452e-9564-ca7b7a7c45e4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 167, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6802', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 2540271081}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6804', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 2540271081}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6808', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2540271081}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6806', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 2540271081}]}, 'public_addr': '172.21.15.134:6803/2540271081', 'cluster_addr': '172.21.15.134:6805/2540271081', 'heartbeat_back_addr': '172.21.15.134:6809/2540271081', 'heartbeat_front_addr': '172.21.15.134:6807/2540271081', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 157, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6810', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2032802960}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6812', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 2032802960}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6816', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6817', 'nonce': 2032802960}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6814', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 2032802960}]}, 'public_addr': '172.21.15.134:6811/2032802960', 'cluster_addr': '172.21.15.134:6813/2032802960', 'heartbeat_back_addr': '172.21.15.134:6817/2032802960', 'heartbeat_front_addr': '172.21.15.134:6815/2032802960', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '078e578b-855b-49ff-83bd-0548a9c5a708', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 97, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6818', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6819', 'nonce': 153827987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6820', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6821', 'nonce': 153827987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 153827987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6822', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6823', 'nonce': 153827987}]}, 'public_addr': '172.21.15.134:6819/153827987', 'cluster_addr': '172.21.15.134:6821/153827987', 'heartbeat_back_addr': '172.21.15.134:6825/153827987', 'heartbeat_front_addr': '172.21.15.134:6823/153827987', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6774e330-3202-4671-894e-172cc2f6189d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 1527507056}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6828', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6829', 'nonce': 1527507056}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6832', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6833', 'nonce': 1527507056}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6830', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6831', 'nonce': 1527507056}]}, 'public_addr': '172.21.15.134:6827/1527507056', 'cluster_addr': '172.21.15.134:6829/1527507056', 'heartbeat_back_addr': '172.21.15.134:6833/1527507056', 'heartbeat_front_addr': '172.21.15.134:6831/1527507056', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '52f70cfa-ab0e-4751-a5f8-1045c61c1693', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 157, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3005156808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3005156808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3005156808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3005156808}]}, 'public_addr': '172.21.15.196:6801/3005156808', 'cluster_addr': '172.21.15.196:6803/3005156808', 'heartbeat_back_addr': '172.21.15.196:6807/3005156808', 'heartbeat_front_addr': '172.21.15.196:6805/3005156808', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b4b21769-d40f-43fd-b7e9-2477b74d3955', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 157, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 1603224792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 1603224792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 1603224792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 1603224792}]}, 'public_addr': '172.21.15.196:6809/1603224792', 'cluster_addr': '172.21.15.196:6811/1603224792', 'heartbeat_back_addr': '172.21.15.196:6815/1603224792', 'heartbeat_front_addr': '172.21.15.196:6813/1603224792', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'af1cced4-0015-47f5-b105-27641bfe4f4a', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 3727413143}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 3727413143}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 3727413143}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 3727413143}]}, 'public_addr': '172.21.15.196:6817/3727413143', 'cluster_addr': '172.21.15.196:6819/3727413143', 'heartbeat_back_addr': '172.21.15.196:6823/3727413143', 'heartbeat_front_addr': '172.21.15.196:6821/3727413143', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c54beeae-9f7c-4723-a369-13bb531b4057', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 100, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 4206571010}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 4206571010}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 4206571010}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 4206571010}]}, 'public_addr': '172.21.15.196:6825/4206571010', 'cluster_addr': '172.21.15.196:6827/4206571010', 'heartbeat_back_addr': '172.21.15.196:6831/4206571010', 'heartbeat_front_addr': '172.21.15.196:6829/4206571010', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:11:19.143195+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:11:05.058869+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:10:54.740063+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:11:32.944359+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:11:43.025464+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:11:42.970092+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T08:11:51.290727+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:11:38.093946+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.1e', 'osds': [3, 4, 1]}], 'pg_upmap_items': [{'pgid': '3.6', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1a', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.134:0/788812699': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/1046500326': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/464208089': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/301941854': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:6800/3607168099': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6800/673876301': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/4225724311': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6800/1525062017': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:6801/3607168099': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6801/673876301': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/3200884805': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/2952444615': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:0/3325835814': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/3161126612': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:0/69097091': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:6801/1525062017': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/1645475987': '2024-02-06T08:05:21.465844+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T08:12:00.917 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:12:00.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:00.995 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:00.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:01.003 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:00 smithi134 ceph-mon[39278]: pgmap v298: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-02-05T08:12:01.003 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:00 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1787299940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:12:01.003 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:00 smithi134 ceph-mon[30086]: pgmap v298: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-02-05T08:12:01.003 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:00 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1787299940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:12:01.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:00 smithi196 ceph-mon[33018]: pgmap v298: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-02-05T08:12:01.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:00 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1787299940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:12:01.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:01.146 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:01.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:01.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:01.247 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:01.300 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:01.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:01.332 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:01.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T08:12:01.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:01.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:12:01.463 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 52 16 80076 80060 96.1926 67.4375 0.0138862 0.0103912 2024-02-05T08:12:01.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:01.473 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:01.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:01.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:01.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:01.635 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:01.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:01.675 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:12:01.736 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:01.796 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:01.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:01.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:01.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:01.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:01.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:01.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:02.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:02.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:02.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:02.161 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:02.178 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:02.209 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:02.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:02.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:02.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:02.356 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:02.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:02.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:02.458 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:02.465 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 53 16 81166 81150 95.6598 68.125 0.0134585 0.0104487 2024-02-05T08:12:02.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:02.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:02.588 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:02.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:02.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:02.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:02.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:02.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:02.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:02.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:02.892 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:02.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:02.976 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:02 smithi134 ceph-mon[30086]: pgmap v299: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-02-05T08:12:02.976 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:02 smithi134 ceph-mon[39278]: pgmap v299: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-02-05T08:12:02.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:03.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:02 smithi196 ceph-mon[33018]: pgmap v299: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.13k op/s 2024-02-05T08:12:03.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:03.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:03.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:03.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:03.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:03.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:03.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:03.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:03.300 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:03.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:03.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:12:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:12:03.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:03.431 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:03.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:03.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:03.465 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 54 16 82293 82277 95.1926 70.4375 0.014505 0.0104999 2024-02-05T08:12:03.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:03.564 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:03.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:03.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:03.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:03.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:03.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:03.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:03 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:03.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:03 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:03.796 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:03.847 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:03.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:03.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:03.949 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:03.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:03.989 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:03.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:03.990 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:03 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:04.062 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:12:04.062 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:12:04.090 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:04.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:04.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:04.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:04.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:04.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:04.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:04.318 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:04.364 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:04.420 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:04.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:04.438 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:12:04.465 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 55 16 83327 83311 94.6369 64.625 0.0163443 0.0105611 2024-02-05T08:12:04.483 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:12:04.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:04.531 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:12:04.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:04.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:04.574 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:12:04.617 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:12:04.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:04.668 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:12:04.694 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:04.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:04.707 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:12:04.747 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:12:04.758 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:12:04.759 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:12:04.759 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:12:04.759 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:12:04.759 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:12:04.759 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:12:04.759 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:12:04.759 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:12:04.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T08:12:04.796 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:04.839 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:04.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:04.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:04.941 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:04.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:04.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:04.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:05.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:04 smithi134 ceph-mon[30086]: pgmap v300: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T08:12:05.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:04 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/316180665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:05.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:04 smithi134 ceph-mon[39278]: pgmap v300: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T08:12:05.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:04 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/316180665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:05.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:04 smithi196 ceph-mon[33018]: pgmap v300: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T08:12:05.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:04 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/316180665' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:05.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:05.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:05.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:05.106 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:12:05.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:05.245 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:05.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:05.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:05.347 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:05.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:05.391 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:05.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:05.467 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 56 16 84250 84234 93.9772 57.6875 0.0188171 0.0106358 2024-02-05T08:12:05.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:05.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:05.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:05.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:05.621 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:05.658 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:05.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:05.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:05.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:05.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:05.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:05.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:05.889 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:05.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:05.937 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:05.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:06.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:06.069 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:06.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:06.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:06.200 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:06.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:06.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:06.346 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:06.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:06.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:06.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:06.466 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 57 16 85225 85209 93.3967 60.9375 0.00908637 0.0107011 2024-02-05T08:12:06.485 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:06.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:06.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:06.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:06.629 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:06.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:06.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:06.731 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:06.769 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:06.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:06.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:06.894 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:06.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:06.934 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:12:06.935 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:12:06.935 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T08:12:06.935 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T08:12:06.935 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:12:06.935 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:06 smithi196 ceph-mon[33018]: pgmap v301: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 855 B/s rd, 57 MiB/s wr, 905 op/s 2024-02-05T08:12:06.997 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:07.031 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:06 smithi134 ceph-mon[30086]: pgmap v301: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 855 B/s rd, 57 MiB/s wr, 905 op/s 2024-02-05T08:12:07.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:06 smithi134 ceph-mon[39278]: pgmap v301: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 855 B/s rd, 57 MiB/s wr, 905 op/s 2024-02-05T08:12:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:07.040 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:07.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:07.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:07.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:07.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:07.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:07.303 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:07.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:07.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:07.405 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:07.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:07.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:07.466 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 58 15 86321 86306 92.9685 68.5625 0.00957573 0.010751 2024-02-05T08:12:07.536 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:07.580 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:07.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:07.682 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:07.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:07.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:07.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:07.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:07.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:07.863 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:07.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:07.936 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:12:07.939 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:12:07.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:07.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T08:12:07.992 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:12:08.008 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:08.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:08.014 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:12:08.015 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:12:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:12:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:12:08.015 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:12:08.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:08.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:08.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:08.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:08.275 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:12:08.281 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:08.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:08.383 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:08.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:08.422 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:08.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:08.466 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 59 16 87351 87335 92.4829 64.3125 0.010098 0.0108083 2024-02-05T08:12:08.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:08.552 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:08.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:08.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:08.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:08.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:08.784 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:08.824 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:08.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:08.926 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:08.967 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:08.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:08.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:09.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:08 smithi134 ceph-mon[30086]: pgmap v302: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-05T08:12:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:08 smithi134 ceph-mon[30086]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:12:09.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:08 smithi134 ceph-mon[39278]: pgmap v302: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-05T08:12:09.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:08 smithi134 ceph-mon[39278]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:12:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:08 smithi196 ceph-mon[33018]: pgmap v302: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-05T08:12:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:08 smithi196 ceph-mon[33018]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:12:09.069 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:09.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:09.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:09.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:09.114 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-02-05T08:12:07.636075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5473442140,"num_objects":83707,"num_object_clones":0,"num_object_copies":251121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83707,"num_whiteouts":0,"num_read":1371,"num_read_kb":1217,"num_write":84124,"num_write_kb":5347037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":2363409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75259,"ondisk_log_size":75259,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17422320,"kb_used_data":15937976,"kb_used_omap":10,"kb_used_meta":1484277,"kb_avail":638793744,"statfs":{"total":671965249536,"available":654124793856,"internally_reserved":0,"allocated":16320487424,"data_stored":16317726960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11155,"internal_metadata":1519899757},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":835518464,"num_objects":12749,"num_object_clones":0,"num_object_copies":38247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12749,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":12749,"num_write_kb":815936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003477"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":246,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.666872+0000","last_change":"2024-02-05T08:10:26.432413+0000","last_active":"2024-02-05T08:11:53.666872+0000","last_peered":"2024-02-05T08:11:53.666872+0000","last_clean":"2024-02-05T08:11:53.666872+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:11:53.666872+0000","last_undegraded":"2024-02-05T08:11:53.666872+0000","last_fullsized":"2024-02-05T08:11:53.666872+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:26.432294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:26.432294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.412548+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.683326+0000","last_change":"2024-02-05T08:11:17.826990+0000","last_active":"2024-02-05T08:11:53.683326+0000","last_peered":"2024-02-05T08:11:53.683326+0000","last_clean":"2024-02-05T08:11:53.683326+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:11:53.683326+0000","last_undegraded":"2024-02-05T08:11:53.683326+0000","last_fullsized":"2024-02-05T08:11:53.683326+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:17.824305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:17.824305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:42.894736+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":259,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.680278+0000","last_change":"2024-02-05T08:10:18.254802+0000","last_active":"2024-02-05T08:11:53.680278+0000","last_peered":"2024-02-05T08:11:53.680278+0000","last_clean":"2024-02-05T08:11:53.680278+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:11:53.680278+0000","last_undegraded":"2024-02-05T08:11:53.680278+0000","last_fullsized":"2024-02-05T08:11:53.680278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.254568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.254568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"90'13","reported_seq":291,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.666550+0000","last_change":"2024-02-05T08:11:06.482664+0000","last_active":"2024-02-05T08:11:53.666550+0000","last_peered":"2024-02-05T08:11:53.666550+0000","last_clean":"2024-02-05T08:11:53.666550+0000","last_became_active":"2024-02-05T08:06:39.560434+0000","last_became_peered":"2024-02-05T08:06:39.560434+0000","last_unstale":"2024-02-05T08:11:53.666550+0000","last_undegraded":"2024-02-05T08:11:53.666550+0000","last_fullsized":"2024-02-05T08:11:53.666550+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:11:06.482560+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:11:06.482560+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:21.899599+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":252,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.678049+0000","last_change":"2024-02-05T08:09:37.106387+0000","last_active":"2024-02-05T08:11:53.678049+0000","last_peered":"2024-02-05T08:11:53.678049+0000","last_clean":"2024-02-05T08:11:53.678049+0000","last_became_active":"2024-02-05T08:06:39.558790+0000","last_became_peered":"2024-02-05T08:06:39.558790+0000","last_unstale":"2024-02-05T08:11:53.678049+0000","last_undegraded":"2024-02-05T08:11:53.678049+0000","last_fullsized":"2024-02-05T08:11:53.678049+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:37.106147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:37.106147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.980054+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":257,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.666702+0000","last_change":"2024-02-05T08:11:19.262689+0000","last_active":"2024-02-05T08:11:53.666702+0000","last_peered":"2024-02-05T08:11:53.666702+0000","last_clean":"2024-02-05T08:11:53.666702+0000","last_became_active":"2024-02-05T08:06:39.561731+0000","last_became_peered":"2024-02-05T08:06:39.561731+0000","last_unstale":"2024-02-05T08:11:53.666702+0000","last_undegraded":"2024-02-05T08:11:53.666702+0000","last_fullsized":"2024-02-05T08:11:53.666702+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:11:19.262611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:36.447755+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":271,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676046+0000","last_change":"2024-02-05T08:11:09.444296+0000","last_active":"2024-02-05T08:11:53.676046+0000","last_peered":"2024-02-05T08:11:53.676046+0000","last_clean":"2024-02-05T08:11:53.676046+0000","last_became_active":"2024-02-05T08:11:09.443912+0000","last_became_peered":"2024-02-05T08:11:09.443912+0000","last_unstale":"2024-02-05T08:11:53.676046+0000","last_undegraded":"2024-02-05T08:11:53.676046+0000","last_fullsized":"2024-02-05T08:11:53.676046+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:28.007713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:28.007713+0000","last_clean_scrub_stamp":"2024-02-05T08:10:28.007713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.980130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"118'22","reported_seq":287,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.683617+0000","last_change":"2024-02-05T08:11:07.726805+0000","last_active":"2024-02-05T08:11:53.683617+0000","last_peered":"2024-02-05T08:11:53.683617+0000","last_clean":"2024-02-05T08:11:53.683617+0000","last_became_active":"2024-02-05T08:04:43.560660+0000","last_became_peered":"2024-02-05T08:04:43.560660+0000","last_unstale":"2024-02-05T08:11:53.683617+0000","last_undegraded":"2024-02-05T08:11:53.683617+0000","last_fullsized":"2024-02-05T08:11:53.683617+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-02-05T08:11:07.726635+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.726635+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:17.948247+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":246,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679576+0000","last_change":"2024-02-05T08:10:29.223232+0000","last_active":"2024-02-05T08:11:53.679576+0000","last_peered":"2024-02-05T08:11:53.679576+0000","last_clean":"2024-02-05T08:11:53.679576+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:11:53.679576+0000","last_undegraded":"2024-02-05T08:11:53.679576+0000","last_fullsized":"2024-02-05T08:11:53.679576+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:29.223145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:29.223145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:55.416210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":250,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679125+0000","last_change":"2024-02-05T08:10:35.612852+0000","last_active":"2024-02-05T08:11:53.679125+0000","last_peered":"2024-02-05T08:11:53.679125+0000","last_clean":"2024-02-05T08:11:53.679125+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:11:53.679125+0000","last_undegraded":"2024-02-05T08:11:53.679125+0000","last_fullsized":"2024-02-05T08:11:53.679125+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:10:35.612720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":272,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679592+0000","last_change":"2024-02-05T08:11:20.316629+0000","last_active":"2024-02-05T08:11:53.679592+0000","last_peered":"2024-02-05T08:11:53.679592+0000","last_clean":"2024-02-05T08:11:53.679592+0000","last_became_active":"2024-02-05T08:06:39.559461+0000","last_became_peered":"2024-02-05T08:06:39.559461+0000","last_unstale":"2024-02-05T08:11:53.679592+0000","last_undegraded":"2024-02-05T08:11:53.679592+0000","last_fullsized":"2024-02-05T08:11:53.679592+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:20.316515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:20.316515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:23.278639+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":286,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667002+0000","last_change":"2024-02-05T08:11:26.162414+0000","last_active":"2024-02-05T08:11:53.667002+0000","last_peered":"2024-02-05T08:11:53.667002+0000","last_clean":"2024-02-05T08:11:53.667002+0000","last_became_active":"2024-02-05T08:07:22.653892+0000","last_became_peered":"2024-02-05T08:07:22.653892+0000","last_unstale":"2024-02-05T08:11:53.667002+0000","last_undegraded":"2024-02-05T08:11:53.667002+0000","last_fullsized":"2024-02-05T08:11:53.667002+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:26.162293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:26.162293+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:36.094131+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.678932+0000","last_change":"2024-02-05T08:10:16.560606+0000","last_active":"2024-02-05T08:11:53.678932+0000","last_peered":"2024-02-05T08:11:53.678932+0000","last_clean":"2024-02-05T08:11:53.678932+0000","last_became_active":"2024-02-05T08:04:47.587515+0000","last_became_peered":"2024-02-05T08:04:47.587515+0000","last_unstale":"2024-02-05T08:11:53.678932+0000","last_undegraded":"2024-02-05T08:11:53.678932+0000","last_fullsized":"2024-02-05T08:11:53.678932+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:16.560531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:16.560531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676938+0000","last_change":"2024-02-05T08:10:24.239442+0000","last_active":"2024-02-05T08:11:53.676938+0000","last_peered":"2024-02-05T08:11:53.676938+0000","last_clean":"2024-02-05T08:11:53.676938+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:11:53.676938+0000","last_undegraded":"2024-02-05T08:11:53.676938+0000","last_fullsized":"2024-02-05T08:11:53.676938+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:24.239334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:24.239334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":258,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679127+0000","last_change":"2024-02-05T08:10:23.663012+0000","last_active":"2024-02-05T08:11:53.679127+0000","last_peered":"2024-02-05T08:11:53.679127+0000","last_clean":"2024-02-05T08:11:53.679127+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:11:53.679127+0000","last_undegraded":"2024-02-05T08:11:53.679127+0000","last_fullsized":"2024-02-05T08:11:53.679127+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:23.662919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:10:23.662919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":264,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667584+0000","last_change":"2024-02-05T08:11:33.210562+0000","last_active":"2024-02-05T08:11:53.667584+0000","last_peered":"2024-02-05T08:11:53.667584+0000","last_clean":"2024-02-05T08:11:53.667584+0000","last_became_active":"2024-02-05T08:04:43.557559+0000","last_became_peered":"2024-02-05T08:04:43.557559+0000","last_unstale":"2024-02-05T08:11:53.667584+0000","last_undegraded":"2024-02-05T08:11:53.667584+0000","last_fullsized":"2024-02-05T08:11:53.667584+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:11:33.210478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:33.210478+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:59.600947+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":253,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667670+0000","last_change":"2024-02-05T08:09:26.466381+0000","last_active":"2024-02-05T08:11:53.667670+0000","last_peered":"2024-02-05T08:11:53.667670+0000","last_clean":"2024-02-05T08:11:53.667670+0000","last_became_active":"2024-02-05T08:06:39.547906+0000","last_became_peered":"2024-02-05T08:06:39.547906+0000","last_unstale":"2024-02-05T08:11:53.667670+0000","last_undegraded":"2024-02-05T08:11:53.667670+0000","last_fullsized":"2024-02-05T08:11:53.667670+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:26.466302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:09:26.466302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.124592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.677908+0000","last_change":"2024-02-05T08:11:06.715512+0000","last_active":"2024-02-05T08:11:53.677908+0000","last_peered":"2024-02-05T08:11:53.677908+0000","last_clean":"2024-02-05T08:11:53.677908+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:11:53.677908+0000","last_undegraded":"2024-02-05T08:11:53.677908+0000","last_fullsized":"2024-02-05T08:11:53.677908+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:06.715401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:11:06.715401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:28.763706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":258,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679811+0000","last_change":"2024-02-05T08:11:22.303481+0000","last_active":"2024-02-05T08:11:53.679811+0000","last_peered":"2024-02-05T08:11:53.679811+0000","last_clean":"2024-02-05T08:11:53.679811+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:11:53.679811+0000","last_undegraded":"2024-02-05T08:11:53.679811+0000","last_fullsized":"2024-02-05T08:11:53.679811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:22.303363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:22.303363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:45.613255+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":296,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676512+0000","last_change":"2024-02-05T08:11:33.391854+0000","last_active":"2024-02-05T08:11:53.676512+0000","last_peered":"2024-02-05T08:11:53.676512+0000","last_clean":"2024-02-05T08:11:53.676512+0000","last_became_active":"2024-02-05T08:04:43.557496+0000","last_became_peered":"2024-02-05T08:04:43.557496+0000","last_unstale":"2024-02-05T08:11:53.676512+0000","last_undegraded":"2024-02-05T08:11:53.676512+0000","last_fullsized":"2024-02-05T08:11:53.676512+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:11:33.391736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:33.391736+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:40.588709+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":253,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676920+0000","last_change":"2024-02-05T08:11:34.947889+0000","last_active":"2024-02-05T08:11:53.676920+0000","last_peered":"2024-02-05T08:11:53.676920+0000","last_clean":"2024-02-05T08:11:53.676920+0000","last_became_active":"2024-02-05T08:06:39.560948+0000","last_became_peered":"2024-02-05T08:06:39.560948+0000","last_unstale":"2024-02-05T08:11:53.676920+0000","last_undegraded":"2024-02-05T08:11:53.676920+0000","last_fullsized":"2024-02-05T08:11:53.676920+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:34.947466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:34.947466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:38.791108+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":250,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667624+0000","last_change":"2024-02-05T08:10:30.419053+0000","last_active":"2024-02-05T08:11:53.667624+0000","last_peered":"2024-02-05T08:11:53.667624+0000","last_clean":"2024-02-05T08:11:53.667624+0000","last_became_active":"2024-02-05T08:04:45.563948+0000","last_became_peered":"2024-02-05T08:04:45.563948+0000","last_unstale":"2024-02-05T08:11:53.667624+0000","last_undegraded":"2024-02-05T08:11:53.667624+0000","last_fullsized":"2024-02-05T08:11:53.667624+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:30.418931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.418931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.124691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":250,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667072+0000","last_change":"2024-02-05T08:09:27.431319+0000","last_active":"2024-02-05T08:11:53.667072+0000","last_peered":"2024-02-05T08:11:53.667072+0000","last_clean":"2024-02-05T08:11:53.667072+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:11:53.667072+0000","last_undegraded":"2024-02-05T08:11:53.667072+0000","last_fullsized":"2024-02-05T08:11:53.667072+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:27.431221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:27.431221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.124508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"90'17","reported_seq":282,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.675531+0000","last_change":"2024-02-05T08:10:30.977423+0000","last_active":"2024-02-05T08:11:53.675531+0000","last_peered":"2024-02-05T08:11:53.675531+0000","last_clean":"2024-02-05T08:11:53.675531+0000","last_became_active":"2024-02-05T08:04:43.557596+0000","last_became_peered":"2024-02-05T08:04:43.557596+0000","last_unstale":"2024-02-05T08:11:53.675531+0000","last_undegraded":"2024-02-05T08:11:53.675531+0000","last_fullsized":"2024-02-05T08:11:53.675531+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:10:30.977334+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.977334+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.980278+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":257,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.677407+0000","last_change":"2024-02-05T08:11:10.429120+0000","last_active":"2024-02-05T08:11:53.677407+0000","last_peered":"2024-02-05T08:11:53.677407+0000","last_clean":"2024-02-05T08:11:53.677407+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:11:53.677407+0000","last_undegraded":"2024-02-05T08:11:53.677407+0000","last_fullsized":"2024-02-05T08:11:53.677407+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:11:10.428967+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:11:10.428967+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:20.943263+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":291,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:01.675774+0000","last_change":"2024-02-05T08:11:21.208333+0000","last_active":"2024-02-05T08:12:01.675774+0000","last_peered":"2024-02-05T08:12:01.675774+0000","last_clean":"2024-02-05T08:12:01.675774+0000","last_became_active":"2024-02-05T08:06:39.552760+0000","last_became_peered":"2024-02-05T08:06:39.552760+0000","last_unstale":"2024-02-05T08:12:01.675774+0000","last_undegraded":"2024-02-05T08:12:01.675774+0000","last_fullsized":"2024-02-05T08:12:01.675774+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:11:21.208206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:21.208206+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:23.267303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":261,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.683570+0000","last_change":"2024-02-05T08:10:32.403469+0000","last_active":"2024-02-05T08:11:53.683570+0000","last_peered":"2024-02-05T08:11:53.683570+0000","last_clean":"2024-02-05T08:11:53.683570+0000","last_became_active":"2024-02-05T08:04:41.532339+0000","last_became_peered":"2024-02-05T08:04:41.532339+0000","last_unstale":"2024-02-05T08:11:53.683570+0000","last_undegraded":"2024-02-05T08:11:53.683570+0000","last_fullsized":"2024-02-05T08:11:53.683570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:10:32.403342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:32.403342+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:03.879468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":262,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.681248+0000","last_change":"2024-02-05T08:09:24.000902+0000","last_active":"2024-02-05T08:11:53.681248+0000","last_peered":"2024-02-05T08:11:53.681248+0000","last_clean":"2024-02-05T08:11:53.681248+0000","last_became_active":"2024-02-05T08:04:43.605952+0000","last_became_peered":"2024-02-05T08:04:43.605952+0000","last_unstale":"2024-02-05T08:11:53.681248+0000","last_undegraded":"2024-02-05T08:11:53.681248+0000","last_fullsized":"2024-02-05T08:11:53.681248+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:09:24.000675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:24.000675+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":255,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.678060+0000","last_change":"2024-02-05T08:11:33.834287+0000","last_active":"2024-02-05T08:11:53.678060+0000","last_peered":"2024-02-05T08:11:53.678060+0000","last_clean":"2024-02-05T08:11:53.678060+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:11:53.678060+0000","last_undegraded":"2024-02-05T08:11:53.678060+0000","last_fullsized":"2024-02-05T08:11:53.678060+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:33.834034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:33.834034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:52.764333+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.681496+0000","last_change":"2024-02-05T08:11:25.316698+0000","last_active":"2024-02-05T08:11:53.681496+0000","last_peered":"2024-02-05T08:11:53.681496+0000","last_clean":"2024-02-05T08:11:53.681496+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:11:53.681496+0000","last_undegraded":"2024-02-05T08:11:53.681496+0000","last_fullsized":"2024-02-05T08:11:53.681496+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:25.316598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:25.316598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:38.269059+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":259,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.689227+0000","last_change":"2024-02-05T08:11:15.831955+0000","last_active":"2024-02-05T08:11:53.689227+0000","last_peered":"2024-02-05T08:11:53.689227+0000","last_clean":"2024-02-05T08:11:53.689227+0000","last_became_active":"2024-02-05T08:04:41.534448+0000","last_became_peered":"2024-02-05T08:04:41.534448+0000","last_unstale":"2024-02-05T08:11:53.689227+0000","last_undegraded":"2024-02-05T08:11:53.689227+0000","last_fullsized":"2024-02-05T08:11:53.689227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:15.831837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.831837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:27.579523+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679712+0000","last_change":"2024-02-05T08:11:15.397484+0000","last_active":"2024-02-05T08:11:53.679712+0000","last_peered":"2024-02-05T08:11:53.679712+0000","last_clean":"2024-02-05T08:11:53.679712+0000","last_became_active":"2024-02-05T08:06:39.558747+0000","last_became_peered":"2024-02-05T08:06:39.558747+0000","last_unstale":"2024-02-05T08:11:53.679712+0000","last_undegraded":"2024-02-05T08:11:53.679712+0000","last_fullsized":"2024-02-05T08:11:53.679712+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:11:15.397335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.397335+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:27.975791+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676877+0000","last_change":"2024-02-05T08:11:01.465783+0000","last_active":"2024-02-05T08:11:53.676877+0000","last_peered":"2024-02-05T08:11:53.676877+0000","last_clean":"2024-02-05T08:11:53.676877+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:11:53.676877+0000","last_undegraded":"2024-02-05T08:11:53.676877+0000","last_fullsized":"2024-02-05T08:11:53.676877+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:01.465657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:01.465657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:26.749290+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":190,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.666785+0000","last_change":"2024-02-05T08:09:36.434985+0000","last_active":"2024-02-05T08:11:53.666785+0000","last_peered":"2024-02-05T08:11:53.666785+0000","last_clean":"2024-02-05T08:11:53.666785+0000","last_became_active":"2024-02-05T08:06:39.551173+0000","last_became_peered":"2024-02-05T08:06:39.551173+0000","last_unstale":"2024-02-05T08:11:53.666785+0000","last_undegraded":"2024-02-05T08:11:53.666785+0000","last_fullsized":"2024-02-05T08:11:53.666785+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:36.434873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.434873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.124608+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":259,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.680533+0000","last_change":"2024-02-05T08:11:20.295028+0000","last_active":"2024-02-05T08:11:53.680533+0000","last_peered":"2024-02-05T08:11:53.680533+0000","last_clean":"2024-02-05T08:11:53.680533+0000","last_became_active":"2024-02-05T08:04:41.536019+0000","last_became_peered":"2024-02-05T08:04:41.536019+0000","last_unstale":"2024-02-05T08:11:53.680533+0000","last_undegraded":"2024-02-05T08:11:53.680533+0000","last_fullsized":"2024-02-05T08:11:53.680533+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:20.294901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:20.294901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:38.830576+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":273,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.689271+0000","last_change":"2024-02-05T08:11:32.786164+0000","last_active":"2024-02-05T08:11:53.689271+0000","last_peered":"2024-02-05T08:11:53.689271+0000","last_clean":"2024-02-05T08:11:53.689271+0000","last_became_active":"2024-02-05T08:06:39.560932+0000","last_became_peered":"2024-02-05T08:06:39.560932+0000","last_unstale":"2024-02-05T08:11:53.689271+0000","last_undegraded":"2024-02-05T08:11:53.689271+0000","last_fullsized":"2024-02-05T08:11:53.689271+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-02-05T08:11:32.785856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:32.785856+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:42.171053+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":253,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.680065+0000","last_change":"2024-02-05T08:10:18.022780+0000","last_active":"2024-02-05T08:11:53.680065+0000","last_peered":"2024-02-05T08:11:53.680065+0000","last_clean":"2024-02-05T08:11:53.680065+0000","last_became_active":"2024-02-05T08:06:39.559212+0000","last_became_peered":"2024-02-05T08:06:39.559212+0000","last_unstale":"2024-02-05T08:11:53.680065+0000","last_undegraded":"2024-02-05T08:11:53.680065+0000","last_fullsized":"2024-02-05T08:11:53.680065+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.022670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.022670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.980104+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":266,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.677040+0000","last_change":"2024-02-05T08:11:16.440695+0000","last_active":"2024-02-05T08:11:53.677040+0000","last_peered":"2024-02-05T08:11:53.677040+0000","last_clean":"2024-02-05T08:11:53.677040+0000","last_became_active":"2024-02-05T08:06:39.555597+0000","last_became_peered":"2024-02-05T08:06:39.555597+0000","last_unstale":"2024-02-05T08:11:53.677040+0000","last_undegraded":"2024-02-05T08:11:53.677040+0000","last_fullsized":"2024-02-05T08:11:53.677040+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:16.440585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:11:16.440585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:30.362929+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":267,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679618+0000","last_change":"2024-02-05T08:11:05.463231+0000","last_active":"2024-02-05T08:11:53.679618+0000","last_peered":"2024-02-05T08:11:53.679618+0000","last_clean":"2024-02-05T08:11:53.679618+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:11:53.679618+0000","last_undegraded":"2024-02-05T08:11:53.679618+0000","last_fullsized":"2024-02-05T08:11:53.679618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:05.463134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:05.463134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:10.311835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"141'16","reported_seq":297,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.683486+0000","last_change":"2024-02-05T08:09:39.113079+0000","last_active":"2024-02-05T08:11:53.683486+0000","last_peered":"2024-02-05T08:11:53.683486+0000","last_clean":"2024-02-05T08:11:53.683486+0000","last_became_active":"2024-02-05T08:06:39.556873+0000","last_became_peered":"2024-02-05T08:06:39.556873+0000","last_unstale":"2024-02-05T08:11:53.683486+0000","last_undegraded":"2024-02-05T08:11:53.683486+0000","last_fullsized":"2024-02-05T08:11:53.683486+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'16","last_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:09:39.112936+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.979891+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679339+0000","last_change":"2024-02-05T08:11:07.248032+0000","last_active":"2024-02-05T08:11:53.679339+0000","last_peered":"2024-02-05T08:11:53.679339+0000","last_clean":"2024-02-05T08:11:53.679339+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:11:53.679339+0000","last_undegraded":"2024-02-05T08:11:53.679339+0000","last_fullsized":"2024-02-05T08:11:53.679339+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:07.247928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.247928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:12.667100+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676393+0000","last_change":"2024-02-05T08:10:26.987843+0000","last_active":"2024-02-05T08:11:53.676393+0000","last_peered":"2024-02-05T08:11:53.676393+0000","last_clean":"2024-02-05T08:11:53.676393+0000","last_became_active":"2024-02-05T08:04:45.568484+0000","last_became_peered":"2024-02-05T08:04:45.568484+0000","last_unstale":"2024-02-05T08:11:53.676393+0000","last_undegraded":"2024-02-05T08:11:53.676393+0000","last_fullsized":"2024-02-05T08:11:53.676393+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:26.987763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:26.987763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.980249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.678266+0000","last_change":"2024-02-05T08:10:31.270781+0000","last_active":"2024-02-05T08:11:53.678266+0000","last_peered":"2024-02-05T08:11:53.678266+0000","last_clean":"2024-02-05T08:11:53.678266+0000","last_became_active":"2024-02-05T08:04:41.539698+0000","last_became_peered":"2024-02-05T08:04:41.539698+0000","last_unstale":"2024-02-05T08:11:53.678266+0000","last_undegraded":"2024-02-05T08:11:53.678266+0000","last_fullsized":"2024-02-05T08:11:53.678266+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:31.270686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:31.270686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:02.410609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":296,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.683070+0000","last_change":"2024-02-05T08:11:13.794030+0000","last_active":"2024-02-05T08:11:53.683070+0000","last_peered":"2024-02-05T08:11:53.683070+0000","last_clean":"2024-02-05T08:11:53.683070+0000","last_became_active":"2024-02-05T08:06:39.560259+0000","last_became_peered":"2024-02-05T08:06:39.560259+0000","last_unstale":"2024-02-05T08:11:53.683070+0000","last_undegraded":"2024-02-05T08:11:53.683070+0000","last_fullsized":"2024-02-05T08:11:53.683070+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:13.793895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:13.793895+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:27.120443+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":239,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.683014+0000","last_change":"2024-02-05T08:09:29.262843+0000","last_active":"2024-02-05T08:11:53.683014+0000","last_peered":"2024-02-05T08:11:53.683014+0000","last_clean":"2024-02-05T08:11:53.683014+0000","last_became_active":"2024-02-05T08:04:47.585496+0000","last_became_peered":"2024-02-05T08:04:47.585496+0000","last_unstale":"2024-02-05T08:11:53.683014+0000","last_undegraded":"2024-02-05T08:11:53.683014+0000","last_fullsized":"2024-02-05T08:11:53.683014+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:09:29.262582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.979917+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":252,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.682974+0000","last_change":"2024-02-05T08:10:25.493102+0000","last_active":"2024-02-05T08:11:53.682974+0000","last_peered":"2024-02-05T08:11:53.682974+0000","last_clean":"2024-02-05T08:11:53.682974+0000","last_became_active":"2024-02-05T08:04:45.560815+0000","last_became_peered":"2024-02-05T08:04:45.560815+0000","last_unstale":"2024-02-05T08:11:53.682974+0000","last_undegraded":"2024-02-05T08:11:53.682974+0000","last_fullsized":"2024-02-05T08:11:53.682974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:25.493013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:25.493013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.850481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":273,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667394+0000","last_change":"2024-02-05T08:11:32.160444+0000","last_active":"2024-02-05T08:11:53.667394+0000","last_peered":"2024-02-05T08:11:53.667394+0000","last_clean":"2024-02-05T08:11:53.667394+0000","last_became_active":"2024-02-05T08:06:39.545576+0000","last_became_peered":"2024-02-05T08:06:39.545576+0000","last_unstale":"2024-02-05T08:11:53.667394+0000","last_undegraded":"2024-02-05T08:11:53.667394+0000","last_fullsized":"2024-02-05T08:11:53.667394+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:32.160333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.966886+0000","last_clean_scrub_stamp":"2024-02-05T08:11:32.160333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:35.137296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":292,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667759+0000","last_change":"2024-02-05T08:10:32.387139+0000","last_active":"2024-02-05T08:11:53.667759+0000","last_peered":"2024-02-05T08:11:53.667759+0000","last_clean":"2024-02-05T08:11:53.667759+0000","last_became_active":"2024-02-05T08:06:39.558257+0000","last_became_peered":"2024-02-05T08:06:39.558257+0000","last_unstale":"2024-02-05T08:11:53.667759+0000","last_undegraded":"2024-02-05T08:11:53.667759+0000","last_fullsized":"2024-02-05T08:11:53.667759+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:10:32.387029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:32.387029+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.428808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.681003+0000","last_change":"2024-02-05T08:11:31.475090+0000","last_active":"2024-02-05T08:11:53.681003+0000","last_peered":"2024-02-05T08:11:53.681003+0000","last_clean":"2024-02-05T08:11:53.681003+0000","last_became_active":"2024-02-05T08:04:47.584235+0000","last_became_peered":"2024-02-05T08:04:47.584235+0000","last_unstale":"2024-02-05T08:11:53.681003+0000","last_undegraded":"2024-02-05T08:11:53.681003+0000","last_fullsized":"2024-02-05T08:11:53.681003+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:31.474957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:31.474957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:44.860106+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":357,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:01.675963+0000","last_change":"2024-02-05T08:11:32.363524+0000","last_active":"2024-02-05T08:12:01.675963+0000","last_peered":"2024-02-05T08:12:01.675963+0000","last_clean":"2024-02-05T08:12:01.675963+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:12:01.675963+0000","last_undegraded":"2024-02-05T08:12:01.675963+0000","last_fullsized":"2024-02-05T08:12:01.675963+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:11:32.363440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:32.363440+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:39.688176+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.677050+0000","last_change":"2024-02-05T08:11:11.422476+0000","last_active":"2024-02-05T08:11:53.677050+0000","last_peered":"2024-02-05T08:11:53.677050+0000","last_clean":"2024-02-05T08:11:53.677050+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:11:53.677050+0000","last_undegraded":"2024-02-05T08:11:53.677050+0000","last_fullsized":"2024-02-05T08:11:53.677050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:11.422371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:11.422371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:36.238335+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"129'15","reported_seq":275,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.675502+0000","last_change":"2024-02-05T08:11:30.787282+0000","last_active":"2024-02-05T08:11:53.675502+0000","last_peered":"2024-02-05T08:11:53.675502+0000","last_clean":"2024-02-05T08:11:53.675502+0000","last_became_active":"2024-02-05T08:04:43.605981+0000","last_became_peered":"2024-02-05T08:04:43.605981+0000","last_unstale":"2024-02-05T08:11:53.675502+0000","last_undegraded":"2024-02-05T08:11:53.675502+0000","last_fullsized":"2024-02-05T08:11:53.675502+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'15","last_scrub_stamp":"2024-02-05T08:11:30.787149+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:11:30.787149+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:45.453499+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":242,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667435+0000","last_change":"2024-02-05T08:11:38.280175+0000","last_active":"2024-02-05T08:11:53.667435+0000","last_peered":"2024-02-05T08:11:53.667435+0000","last_clean":"2024-02-05T08:11:53.667435+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:11:53.667435+0000","last_undegraded":"2024-02-05T08:11:53.667435+0000","last_fullsized":"2024-02-05T08:11:53.667435+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:18.397684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:18.397684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:02.985463+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":348,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:56.675810+0000","last_change":"2024-02-05T08:10:33.947077+0000","last_active":"2024-02-05T08:11:56.675810+0000","last_peered":"2024-02-05T08:11:56.675810+0000","last_clean":"2024-02-05T08:11:56.675810+0000","last_became_active":"2024-02-05T08:04:45.565240+0000","last_became_peered":"2024-02-05T08:04:45.565240+0000","last_unstale":"2024-02-05T08:11:56.675810+0000","last_undegraded":"2024-02-05T08:11:56.675810+0000","last_fullsized":"2024-02-05T08:11:56.675810+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:10:33.946950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:33.946950+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":259,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.680847+0000","last_change":"2024-02-05T08:11:24.316478+0000","last_active":"2024-02-05T08:11:53.680847+0000","last_peered":"2024-02-05T08:11:53.680847+0000","last_clean":"2024-02-05T08:11:53.680847+0000","last_became_active":"2024-02-05T08:04:41.540252+0000","last_became_peered":"2024-02-05T08:04:41.540252+0000","last_unstale":"2024-02-05T08:11:53.680847+0000","last_undegraded":"2024-02-05T08:11:53.680847+0000","last_fullsized":"2024-02-05T08:11:53.680847+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:11:24.316208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:52.711369+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":279,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679840+0000","last_change":"2024-02-05T08:11:19.287698+0000","last_active":"2024-02-05T08:11:53.679840+0000","last_peered":"2024-02-05T08:11:53.679840+0000","last_clean":"2024-02-05T08:11:53.679840+0000","last_became_active":"2024-02-05T08:04:43.606414+0000","last_became_peered":"2024-02-05T08:04:43.606414+0000","last_unstale":"2024-02-05T08:11:53.679840+0000","last_undegraded":"2024-02-05T08:11:53.679840+0000","last_fullsized":"2024-02-05T08:11:53.679840+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:19.287569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:19.287569+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:29.462060+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":256,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676794+0000","last_change":"2024-02-05T08:10:20.671244+0000","last_active":"2024-02-05T08:11:53.676794+0000","last_peered":"2024-02-05T08:11:53.676794+0000","last_clean":"2024-02-05T08:11:53.676794+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:11:53.676794+0000","last_undegraded":"2024-02-05T08:11:53.676794+0000","last_fullsized":"2024-02-05T08:11:53.676794+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:20.671130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:20.671130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.500424+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676873+0000","last_change":"2024-02-05T08:10:15.994743+0000","last_active":"2024-02-05T08:11:53.676873+0000","last_peered":"2024-02-05T08:11:53.676873+0000","last_clean":"2024-02-05T08:11:53.676873+0000","last_became_active":"2024-02-05T08:04:45.567622+0000","last_became_peered":"2024-02-05T08:04:45.567622+0000","last_unstale":"2024-02-05T08:11:53.676873+0000","last_undegraded":"2024-02-05T08:11:53.676873+0000","last_fullsized":"2024-02-05T08:11:53.676873+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:15.994621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:15.994621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":266,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676966+0000","last_change":"2024-02-05T08:11:07.416059+0000","last_active":"2024-02-05T08:11:53.676966+0000","last_peered":"2024-02-05T08:11:53.676966+0000","last_clean":"2024-02-05T08:11:53.676966+0000","last_became_active":"2024-02-05T08:06:39.557018+0000","last_became_peered":"2024-02-05T08:06:39.557018+0000","last_unstale":"2024-02-05T08:11:53.676966+0000","last_undegraded":"2024-02-05T08:11:53.676966+0000","last_fullsized":"2024-02-05T08:11:53.676966+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:07.415963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.415963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:28.508089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":271,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.681378+0000","last_change":"2024-02-05T08:11:23.324482+0000","last_active":"2024-02-05T08:11:53.681378+0000","last_peered":"2024-02-05T08:11:53.681378+0000","last_clean":"2024-02-05T08:11:53.681378+0000","last_became_active":"2024-02-05T08:04:43.608489+0000","last_became_peered":"2024-02-05T08:04:43.608489+0000","last_unstale":"2024-02-05T08:11:53.681378+0000","last_undegraded":"2024-02-05T08:11:53.681378+0000","last_fullsized":"2024-02-05T08:11:53.681378+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:23.324165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:23.324165+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:26.923669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":262,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.678183+0000","last_change":"2024-02-05T08:11:18.424459+0000","last_active":"2024-02-05T08:11:53.678183+0000","last_peered":"2024-02-05T08:11:53.678183+0000","last_clean":"2024-02-05T08:11:53.678183+0000","last_became_active":"2024-02-05T08:06:39.548837+0000","last_became_peered":"2024-02-05T08:06:39.548837+0000","last_unstale":"2024-02-05T08:11:53.678183+0000","last_undegraded":"2024-02-05T08:11:53.678183+0000","last_fullsized":"2024-02-05T08:11:53.678183+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:18.424341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:18.424341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:34.523873+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679621+0000","last_change":"2024-02-05T08:10:31.622795+0000","last_active":"2024-02-05T08:11:53.679621+0000","last_peered":"2024-02-05T08:11:53.679621+0000","last_clean":"2024-02-05T08:11:53.679621+0000","last_became_active":"2024-02-05T08:04:45.560241+0000","last_became_peered":"2024-02-05T08:04:45.560241+0000","last_unstale":"2024-02-05T08:11:53.679621+0000","last_undegraded":"2024-02-05T08:11:53.679621+0000","last_fullsized":"2024-02-05T08:11:53.679621+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:10:31.622683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:58.356161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":190,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679756+0000","last_change":"2024-02-05T08:11:01.330524+0000","last_active":"2024-02-05T08:11:53.679756+0000","last_peered":"2024-02-05T08:11:53.679756+0000","last_clean":"2024-02-05T08:11:53.679756+0000","last_became_active":"2024-02-05T08:06:39.554673+0000","last_became_peered":"2024-02-05T08:06:39.554673+0000","last_unstale":"2024-02-05T08:11:53.679756+0000","last_undegraded":"2024-02-05T08:11:53.679756+0000","last_fullsized":"2024-02-05T08:11:53.679756+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:01.330449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:01.330449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:25.711003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"116'9","reported_seq":267,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.677049+0000","last_change":"2024-02-05T08:10:32.930947+0000","last_active":"2024-02-05T08:11:53.677049+0000","last_peered":"2024-02-05T08:11:53.677049+0000","last_clean":"2024-02-05T08:11:53.677049+0000","last_became_active":"2024-02-05T08:04:43.606624+0000","last_became_peered":"2024-02-05T08:04:43.606624+0000","last_unstale":"2024-02-05T08:11:53.677049+0000","last_undegraded":"2024-02-05T08:11:53.677049+0000","last_fullsized":"2024-02-05T08:11:53.677049+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-02-05T08:10:32.930856+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:10:32.930856+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"178'5397","reported_seq":5449,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:07.552547+0000","last_change":"2024-02-05T08:10:54.986795+0000","last_active":"2024-02-05T08:12:07.552547+0000","last_peered":"2024-02-05T08:12:07.552547+0000","last_clean":"2024-02-05T08:12:07.552547+0000","last_became_active":"2024-02-05T08:10:54.986264+0000","last_became_peered":"2024-02-05T08:10:54.986264+0000","last_unstale":"2024-02-05T08:12:07.552547+0000","last_undegraded":"2024-02-05T08:12:07.552547+0000","last_fullsized":"2024-02-05T08:12:07.552547+0000","mapping_epoch":157,"log_start":"164'300","ondisk_log_start":"164'300","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5097,"log_dups_size":0,"ondisk_log_size":5097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:09.209651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353697792,"num_objects":5397,"num_object_clones":0,"num_object_copies":16191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5397,"num_write_kb":345408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":261,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679211+0000","last_change":"2024-02-05T08:11:13.452209+0000","last_active":"2024-02-05T08:11:53.679211+0000","last_peered":"2024-02-05T08:11:53.679211+0000","last_clean":"2024-02-05T08:11:53.679211+0000","last_became_active":"2024-02-05T08:06:39.551759+0000","last_became_peered":"2024-02-05T08:06:39.551759+0000","last_unstale":"2024-02-05T08:11:53.679211+0000","last_undegraded":"2024-02-05T08:11:53.679211+0000","last_fullsized":"2024-02-05T08:11:53.679211+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:13.452127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:13.452127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:39.634517+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":272,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.677610+0000","last_change":"2024-02-05T08:10:28.309205+0000","last_active":"2024-02-05T08:11:53.677610+0000","last_peered":"2024-02-05T08:11:53.677610+0000","last_clean":"2024-02-05T08:11:53.677610+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:11:53.677610+0000","last_undegraded":"2024-02-05T08:11:53.677610+0000","last_fullsized":"2024-02-05T08:11:53.677610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:10:28.309092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:28.309092+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":302,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.678911+0000","last_change":"2024-02-05T08:11:01.520516+0000","last_active":"2024-02-05T08:11:53.678911+0000","last_peered":"2024-02-05T08:11:53.678911+0000","last_clean":"2024-02-05T08:11:53.678911+0000","last_became_active":"2024-02-05T08:06:39.546607+0000","last_became_peered":"2024-02-05T08:06:39.546607+0000","last_unstale":"2024-02-05T08:11:53.678911+0000","last_undegraded":"2024-02-05T08:11:53.678911+0000","last_fullsized":"2024-02-05T08:11:53.678911+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:11:01.520393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:01.520393+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:04.614465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":368,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:01.675828+0000","last_change":"2024-02-05T08:11:14.422575+0000","last_active":"2024-02-05T08:12:01.675828+0000","last_peered":"2024-02-05T08:12:01.675828+0000","last_clean":"2024-02-05T08:12:01.675828+0000","last_became_active":"2024-02-05T08:06:39.556562+0000","last_became_peered":"2024-02-05T08:06:39.556562+0000","last_unstale":"2024-02-05T08:12:01.675828+0000","last_undegraded":"2024-02-05T08:12:01.675828+0000","last_fullsized":"2024-02-05T08:12:01.675828+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:11:14.422445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:14.422445+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:39.731435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"178'5436","reported_seq":5488,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:07.546373+0000","last_change":"2024-02-05T08:10:54.985646+0000","last_active":"2024-02-05T08:12:07.546373+0000","last_peered":"2024-02-05T08:12:07.546373+0000","last_clean":"2024-02-05T08:12:07.546373+0000","last_became_active":"2024-02-05T08:10:54.985285+0000","last_became_peered":"2024-02-05T08:10:54.985285+0000","last_unstale":"2024-02-05T08:12:07.546373+0000","last_undegraded":"2024-02-05T08:12:07.546373+0000","last_fullsized":"2024-02-05T08:12:07.546373+0000","mapping_epoch":157,"log_start":"165'800","ondisk_log_start":"165'800","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356253696,"num_objects":5436,"num_object_clones":0,"num_object_copies":16308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5436,"num_write_kb":347904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":239,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.680074+0000","last_change":"2024-02-05T08:09:25.266619+0000","last_active":"2024-02-05T08:11:53.680074+0000","last_peered":"2024-02-05T08:11:53.680074+0000","last_clean":"2024-02-05T08:11:53.680074+0000","last_became_active":"2024-02-05T08:04:47.586044+0000","last_became_peered":"2024-02-05T08:04:47.586044+0000","last_unstale":"2024-02-05T08:11:53.680074+0000","last_undegraded":"2024-02-05T08:11:53.680074+0000","last_fullsized":"2024-02-05T08:11:53.680074+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:25.266513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:09:25.266513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.979943+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.677553+0000","last_change":"2024-02-05T08:11:31.474056+0000","last_active":"2024-02-05T08:11:53.677553+0000","last_peered":"2024-02-05T08:11:53.677553+0000","last_clean":"2024-02-05T08:11:53.677553+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:11:53.677553+0000","last_undegraded":"2024-02-05T08:11:53.677553+0000","last_fullsized":"2024-02-05T08:11:53.677553+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:31.473976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:11:31.473976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.557497+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":270,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667314+0000","last_change":"2024-02-05T08:09:31.469380+0000","last_active":"2024-02-05T08:11:53.667314+0000","last_peered":"2024-02-05T08:11:53.667314+0000","last_clean":"2024-02-05T08:11:53.667314+0000","last_became_active":"2024-02-05T08:04:43.556580+0000","last_became_peered":"2024-02-05T08:04:43.556580+0000","last_unstale":"2024-02-05T08:11:53.667314+0000","last_undegraded":"2024-02-05T08:11:53.667314+0000","last_fullsized":"2024-02-05T08:11:53.667314+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:09:31.469295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:31.469295+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.124576+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.678663+0000","last_change":"2024-02-05T08:11:18.292754+0000","last_active":"2024-02-05T08:11:53.678663+0000","last_peered":"2024-02-05T08:11:53.678663+0000","last_clean":"2024-02-05T08:11:53.678663+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:11:53.678663+0000","last_undegraded":"2024-02-05T08:11:53.678663+0000","last_fullsized":"2024-02-05T08:11:53.678663+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:18.292640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_clean_scrub_stamp":"2024-02-05T08:11:18.292640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:23.693559+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"178'5167","reported_seq":5219,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:03.168365+0000","last_change":"2024-02-05T08:10:54.987729+0000","last_active":"2024-02-05T08:12:03.168365+0000","last_peered":"2024-02-05T08:12:03.168365+0000","last_clean":"2024-02-05T08:12:03.168365+0000","last_became_active":"2024-02-05T08:10:54.987175+0000","last_became_peered":"2024-02-05T08:10:54.987175+0000","last_unstale":"2024-02-05T08:12:03.168365+0000","last_undegraded":"2024-02-05T08:12:03.168365+0000","last_fullsized":"2024-02-05T08:12:03.168365+0000","mapping_epoch":157,"log_start":"164'300","ondisk_log_start":"164'300","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4867,"log_dups_size":0,"ondisk_log_size":4867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:05.506648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338624512,"num_objects":5167,"num_object_clones":0,"num_object_copies":15501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5167,"num_write_kb":330688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676796+0000","last_change":"2024-02-05T08:10:21.221033+0000","last_active":"2024-02-05T08:11:53.676796+0000","last_peered":"2024-02-05T08:11:53.676796+0000","last_clean":"2024-02-05T08:11:53.676796+0000","last_became_active":"2024-02-05T08:04:47.587082+0000","last_became_peered":"2024-02-05T08:04:47.587082+0000","last_unstale":"2024-02-05T08:11:53.676796+0000","last_undegraded":"2024-02-05T08:11:53.676796+0000","last_fullsized":"2024-02-05T08:11:53.676796+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:21.220941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:21.220941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676174+0000","last_change":"2024-02-05T08:11:22.231278+0000","last_active":"2024-02-05T08:11:53.676174+0000","last_peered":"2024-02-05T08:11:53.676174+0000","last_clean":"2024-02-05T08:11:53.676174+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:11:53.676174+0000","last_undegraded":"2024-02-05T08:11:53.676174+0000","last_fullsized":"2024-02-05T08:11:53.676174+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:11:22.231148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:51.378207+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":274,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.678380+0000","last_change":"2024-02-05T08:11:27.540590+0000","last_active":"2024-02-05T08:11:53.678380+0000","last_peered":"2024-02-05T08:11:53.678380+0000","last_clean":"2024-02-05T08:11:53.678380+0000","last_became_active":"2024-02-05T08:06:39.554318+0000","last_became_peered":"2024-02-05T08:06:39.554318+0000","last_unstale":"2024-02-05T08:11:53.678380+0000","last_undegraded":"2024-02-05T08:11:53.678380+0000","last_fullsized":"2024-02-05T08:11:53.678380+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:27.540472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:27.540472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:57.050482+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":273,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.677254+0000","last_change":"2024-02-05T08:11:19.277720+0000","last_active":"2024-02-05T08:11:53.677254+0000","last_peered":"2024-02-05T08:11:53.677254+0000","last_clean":"2024-02-05T08:11:53.677254+0000","last_became_active":"2024-02-05T08:06:39.552819+0000","last_became_peered":"2024-02-05T08:06:39.552819+0000","last_unstale":"2024-02-05T08:11:53.677254+0000","last_undegraded":"2024-02-05T08:11:53.677254+0000","last_fullsized":"2024-02-05T08:11:53.677254+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:11:19.277464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:19.277464+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:36.607221+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.7","version":"178'4928","reported_seq":4980,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:02.944086+0000","last_change":"2024-02-05T08:10:54.989610+0000","last_active":"2024-02-05T08:12:02.944086+0000","last_peered":"2024-02-05T08:12:02.944086+0000","last_clean":"2024-02-05T08:12:02.944086+0000","last_became_active":"2024-02-05T08:10:54.989337+0000","last_became_peered":"2024-02-05T08:10:54.989337+0000","last_unstale":"2024-02-05T08:12:02.944086+0000","last_undegraded":"2024-02-05T08:12:02.944086+0000","last_fullsized":"2024-02-05T08:12:02.944086+0000","mapping_epoch":157,"log_start":"165'700","ondisk_log_start":"165'700","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4228,"log_dups_size":0,"ondisk_log_size":4228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:08.496244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322961408,"num_objects":4928,"num_object_clones":0,"num_object_copies":14784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4928,"num_write_kb":315392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.686505+0000","last_change":"2024-02-05T08:11:08.235398+0000","last_active":"2024-02-05T08:11:53.686505+0000","last_peered":"2024-02-05T08:11:53.686505+0000","last_clean":"2024-02-05T08:11:53.686505+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:11:53.686505+0000","last_undegraded":"2024-02-05T08:11:53.686505+0000","last_fullsized":"2024-02-05T08:11:53.686505+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:08.235281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:08.235281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:09.928779+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.677018+0000","last_change":"2024-02-05T08:11:15.272413+0000","last_active":"2024-02-05T08:11:53.677018+0000","last_peered":"2024-02-05T08:11:53.677018+0000","last_clean":"2024-02-05T08:11:53.677018+0000","last_became_active":"2024-02-05T08:04:45.567341+0000","last_became_peered":"2024-02-05T08:04:45.567341+0000","last_unstale":"2024-02-05T08:11:53.677018+0000","last_undegraded":"2024-02-05T08:11:53.677018+0000","last_fullsized":"2024-02-05T08:11:53.677018+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:15.272287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.272287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:39.542308+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":265,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679147+0000","last_change":"2024-02-05T08:10:30.671004+0000","last_active":"2024-02-05T08:11:53.679147+0000","last_peered":"2024-02-05T08:11:53.679147+0000","last_clean":"2024-02-05T08:11:53.679147+0000","last_became_active":"2024-02-05T08:06:39.558993+0000","last_became_peered":"2024-02-05T08:06:39.558993+0000","last_unstale":"2024-02-05T08:11:53.679147+0000","last_undegraded":"2024-02-05T08:11:53.679147+0000","last_fullsized":"2024-02-05T08:11:53.679147+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:30.670916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.670916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:55.449333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":271,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.680019+0000","last_change":"2024-02-05T08:10:34.198669+0000","last_active":"2024-02-05T08:11:53.680019+0000","last_peered":"2024-02-05T08:11:53.680019+0000","last_clean":"2024-02-05T08:11:53.680019+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:11:53.680019+0000","last_undegraded":"2024-02-05T08:11:53.680019+0000","last_fullsized":"2024-02-05T08:11:53.680019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:10:34.198584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:34.198584+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"178'5380","reported_seq":5432,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:07.530258+0000","last_change":"2024-02-05T08:10:54.991561+0000","last_active":"2024-02-05T08:12:07.530258+0000","last_peered":"2024-02-05T08:12:07.530258+0000","last_clean":"2024-02-05T08:12:07.530258+0000","last_became_active":"2024-02-05T08:10:54.991198+0000","last_became_peered":"2024-02-05T08:10:54.991198+0000","last_unstale":"2024-02-05T08:12:07.530258+0000","last_undegraded":"2024-02-05T08:12:07.530258+0000","last_fullsized":"2024-02-05T08:12:07.530258+0000","mapping_epoch":157,"log_start":"164'200","ondisk_log_start":"164'200","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5180,"log_dups_size":0,"ondisk_log_size":5180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352583680,"num_objects":5380,"num_object_clones":0,"num_object_copies":16140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5380,"num_write_kb":344320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":239,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676188+0000","last_change":"2024-02-05T08:10:17.009112+0000","last_active":"2024-02-05T08:11:53.676188+0000","last_peered":"2024-02-05T08:11:53.676188+0000","last_clean":"2024-02-05T08:11:53.676188+0000","last_became_active":"2024-02-05T08:04:47.588395+0000","last_became_peered":"2024-02-05T08:04:47.588395+0000","last_unstale":"2024-02-05T08:11:53.676188+0000","last_undegraded":"2024-02-05T08:11:53.676188+0000","last_fullsized":"2024-02-05T08:11:53.676188+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:17.008882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:17.008882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.980079+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":281,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679094+0000","last_change":"2024-02-05T08:10:18.580307+0000","last_active":"2024-02-05T08:11:53.679094+0000","last_peered":"2024-02-05T08:11:53.679094+0000","last_clean":"2024-02-05T08:11:53.679094+0000","last_became_active":"2024-02-05T08:06:39.560631+0000","last_became_peered":"2024-02-05T08:06:39.560631+0000","last_unstale":"2024-02-05T08:11:53.679094+0000","last_undegraded":"2024-02-05T08:11:53.679094+0000","last_fullsized":"2024-02-05T08:11:53.679094+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:10:18.580209+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.580209+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":265,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667392+0000","last_change":"2024-02-05T08:11:07.502507+0000","last_active":"2024-02-05T08:11:53.667392+0000","last_peered":"2024-02-05T08:11:53.667392+0000","last_clean":"2024-02-05T08:11:53.667392+0000","last_became_active":"2024-02-05T08:06:39.563051+0000","last_became_peered":"2024-02-05T08:06:39.563051+0000","last_unstale":"2024-02-05T08:11:53.667392+0000","last_undegraded":"2024-02-05T08:11:53.667392+0000","last_fullsized":"2024-02-05T08:11:53.667392+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:07.502393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.502393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:36.220738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"85'38","reported_seq":411,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:56.676250+0000","last_change":"2024-02-05T08:11:34.819707+0000","last_active":"2024-02-05T08:11:56.676250+0000","last_peered":"2024-02-05T08:11:56.676250+0000","last_clean":"2024-02-05T08:11:56.676250+0000","last_became_active":"2024-02-05T08:04:43.554536+0000","last_became_peered":"2024-02-05T08:04:43.554536+0000","last_unstale":"2024-02-05T08:11:56.676250+0000","last_undegraded":"2024-02-05T08:11:56.676250+0000","last_fullsized":"2024-02-05T08:11:56.676250+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:11:34.819564+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:11:34.819564+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:43.712077+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":243,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.680676+0000","last_change":"2024-02-05T08:10:19.265168+0000","last_active":"2024-02-05T08:11:53.680676+0000","last_peered":"2024-02-05T08:11:53.680676+0000","last_clean":"2024-02-05T08:11:53.680676+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:11:53.680676+0000","last_undegraded":"2024-02-05T08:11:53.680676+0000","last_fullsized":"2024-02-05T08:11:53.680676+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:19.265057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:19.265057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"178'5092","reported_seq":5144,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:02.891510+0000","last_change":"2024-02-05T08:10:54.985946+0000","last_active":"2024-02-05T08:12:02.891510+0000","last_peered":"2024-02-05T08:12:02.891510+0000","last_clean":"2024-02-05T08:12:02.891510+0000","last_became_active":"2024-02-05T08:10:54.984063+0000","last_became_peered":"2024-02-05T08:10:54.984063+0000","last_unstale":"2024-02-05T08:12:02.891510+0000","last_undegraded":"2024-02-05T08:12:02.891510+0000","last_fullsized":"2024-02-05T08:12:02.891510+0000","mapping_epoch":157,"log_start":"165'700","ondisk_log_start":"165'700","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:17.275769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333709312,"num_objects":5092,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":325888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":254,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.678462+0000","last_change":"2024-02-05T08:11:06.450399+0000","last_active":"2024-02-05T08:11:53.678462+0000","last_peered":"2024-02-05T08:11:53.678462+0000","last_clean":"2024-02-05T08:11:53.678462+0000","last_became_active":"2024-02-05T08:06:39.555147+0000","last_became_peered":"2024-02-05T08:06:39.555147+0000","last_unstale":"2024-02-05T08:11:53.678462+0000","last_undegraded":"2024-02-05T08:11:53.678462+0000","last_fullsized":"2024-02-05T08:11:53.678462+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:06.450276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:06.450276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:15.727823+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":277,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.682031+0000","last_change":"2024-02-05T08:11:23.935034+0000","last_active":"2024-02-05T08:11:53.682031+0000","last_peered":"2024-02-05T08:11:53.682031+0000","last_clean":"2024-02-05T08:11:53.682031+0000","last_became_active":"2024-02-05T08:06:39.560285+0000","last_became_peered":"2024-02-05T08:06:39.560285+0000","last_unstale":"2024-02-05T08:11:53.682031+0000","last_undegraded":"2024-02-05T08:11:53.682031+0000","last_fullsized":"2024-02-05T08:11:53.682031+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:11:23.934912+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:11:23.934912+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:43.872940+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":277,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676587+0000","last_change":"2024-02-05T08:10:34.494353+0000","last_active":"2024-02-05T08:11:53.676587+0000","last_peered":"2024-02-05T08:11:53.676587+0000","last_clean":"2024-02-05T08:11:53.676587+0000","last_became_active":"2024-02-05T08:04:40.529073+0000","last_became_peered":"2024-02-05T08:04:40.529073+0000","last_unstale":"2024-02-05T08:11:53.676587+0000","last_undegraded":"2024-02-05T08:11:53.676587+0000","last_fullsized":"2024-02-05T08:11:53.676587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:34.494252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:10:34.494252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:01.739051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.680554+0000","last_change":"2024-02-05T08:11:25.289984+0000","last_active":"2024-02-05T08:11:53.680554+0000","last_peered":"2024-02-05T08:11:53.680554+0000","last_clean":"2024-02-05T08:11:53.680554+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:11:53.680554+0000","last_undegraded":"2024-02-05T08:11:53.680554+0000","last_fullsized":"2024-02-05T08:11:53.680554+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:11:25.289752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:25.289752+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:40.953433+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":260,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.680340+0000","last_change":"2024-02-05T08:11:01.337800+0000","last_active":"2024-02-05T08:11:53.680340+0000","last_peered":"2024-02-05T08:11:53.680340+0000","last_clean":"2024-02-05T08:11:53.680340+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:11:53.680340+0000","last_undegraded":"2024-02-05T08:11:53.680340+0000","last_fullsized":"2024-02-05T08:11:53.680340+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:01.337730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:01.337730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:03.588977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"178'5334","reported_seq":5386,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:07.555334+0000","last_change":"2024-02-05T08:10:54.989478+0000","last_active":"2024-02-05T08:12:07.555334+0000","last_peered":"2024-02-05T08:12:07.555334+0000","last_clean":"2024-02-05T08:12:07.555334+0000","last_became_active":"2024-02-05T08:10:54.989246+0000","last_became_peered":"2024-02-05T08:10:54.989246+0000","last_unstale":"2024-02-05T08:12:07.555334+0000","last_undegraded":"2024-02-05T08:12:07.555334+0000","last_fullsized":"2024-02-05T08:12:07.555334+0000","mapping_epoch":157,"log_start":"164'200","ondisk_log_start":"164'200","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5134,"log_dups_size":0,"ondisk_log_size":5134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:10.507860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349569024,"num_objects":5334,"num_object_clones":0,"num_object_copies":16002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5334,"num_write_kb":341376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679513+0000","last_change":"2024-02-05T08:11:02.526794+0000","last_active":"2024-02-05T08:11:53.679513+0000","last_peered":"2024-02-05T08:11:53.679513+0000","last_clean":"2024-02-05T08:11:53.679513+0000","last_became_active":"2024-02-05T08:04:47.586358+0000","last_became_peered":"2024-02-05T08:04:47.586358+0000","last_unstale":"2024-02-05T08:11:53.679513+0000","last_undegraded":"2024-02-05T08:11:53.679513+0000","last_fullsized":"2024-02-05T08:11:53.679513+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:02.526668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:02.526668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:03.167722+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":533,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676922+0000","last_change":"2024-02-05T08:11:05.478409+0000","last_active":"2024-02-05T08:11:53.676922+0000","last_peered":"2024-02-05T08:11:53.676922+0000","last_clean":"2024-02-05T08:11:53.676922+0000","last_became_active":"2024-02-05T08:06:42.566771+0000","last_became_peered":"2024-02-05T08:06:42.566771+0000","last_unstale":"2024-02-05T08:11:53.676922+0000","last_undegraded":"2024-02-05T08:11:53.676922+0000","last_fullsized":"2024-02-05T08:11:53.676922+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:11:05.478284+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:11:05.478284+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:22.817135+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":265,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.672980+0000","last_change":"2024-02-05T08:11:29.759917+0000","last_active":"2024-02-05T08:11:53.672980+0000","last_peered":"2024-02-05T08:11:53.672980+0000","last_clean":"2024-02-05T08:11:53.672980+0000","last_became_active":"2024-02-05T08:06:39.553704+0000","last_became_peered":"2024-02-05T08:06:39.553704+0000","last_unstale":"2024-02-05T08:11:53.672980+0000","last_undegraded":"2024-02-05T08:11:53.672980+0000","last_fullsized":"2024-02-05T08:11:53.672980+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:29.759802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:29.759802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:32.319049+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":317,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:01.675819+0000","last_change":"2024-02-05T08:10:19.453291+0000","last_active":"2024-02-05T08:12:01.675819+0000","last_peered":"2024-02-05T08:12:01.675819+0000","last_clean":"2024-02-05T08:12:01.675819+0000","last_became_active":"2024-02-05T08:06:39.563373+0000","last_became_peered":"2024-02-05T08:06:39.563373+0000","last_unstale":"2024-02-05T08:12:01.675819+0000","last_undegraded":"2024-02-05T08:12:01.675819+0000","last_fullsized":"2024-02-05T08:12:01.675819+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-02-05T08:10:19.453162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:19.453162+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:07.989563+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":252,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.677452+0000","last_change":"2024-02-05T08:11:04.446955+0000","last_active":"2024-02-05T08:11:53.677452+0000","last_peered":"2024-02-05T08:11:53.677452+0000","last_clean":"2024-02-05T08:11:53.677452+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:11:53.677452+0000","last_undegraded":"2024-02-05T08:11:53.677452+0000","last_fullsized":"2024-02-05T08:11:53.677452+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:04.446828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:04.446828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:21.009839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"178'5096","reported_seq":5148,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:02.898335+0000","last_change":"2024-02-05T08:10:54.985737+0000","last_active":"2024-02-05T08:12:02.898335+0000","last_peered":"2024-02-05T08:12:02.898335+0000","last_clean":"2024-02-05T08:12:02.898335+0000","last_became_active":"2024-02-05T08:10:54.985443+0000","last_became_peered":"2024-02-05T08:10:54.985443+0000","last_unstale":"2024-02-05T08:12:02.898335+0000","last_undegraded":"2024-02-05T08:12:02.898335+0000","last_fullsized":"2024-02-05T08:12:02.898335+0000","mapping_epoch":157,"log_start":"165'800","ondisk_log_start":"165'800","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4296,"log_dups_size":0,"ondisk_log_size":4296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:05.845861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333971456,"num_objects":5096,"num_object_clones":0,"num_object_copies":15288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5096,"num_write_kb":326144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":248,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.677672+0000","last_change":"2024-02-05T08:10:35.415472+0000","last_active":"2024-02-05T08:11:53.677672+0000","last_peered":"2024-02-05T08:11:53.677672+0000","last_clean":"2024-02-05T08:11:53.677672+0000","last_became_active":"2024-02-05T08:04:47.586774+0000","last_became_peered":"2024-02-05T08:04:47.586774+0000","last_unstale":"2024-02-05T08:11:53.677672+0000","last_undegraded":"2024-02-05T08:11:53.677672+0000","last_fullsized":"2024-02-05T08:11:53.677672+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:35.415402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:35.415402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.850594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":259,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.686237+0000","last_change":"2024-02-05T08:11:34.336191+0000","last_active":"2024-02-05T08:11:53.686237+0000","last_peered":"2024-02-05T08:11:53.686237+0000","last_clean":"2024-02-05T08:11:53.686237+0000","last_became_active":"2024-02-05T08:04:41.537872+0000","last_became_peered":"2024-02-05T08:04:41.537872+0000","last_unstale":"2024-02-05T08:11:53.686237+0000","last_undegraded":"2024-02-05T08:11:53.686237+0000","last_fullsized":"2024-02-05T08:11:53.686237+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:34.336040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:34.336040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:36.256864+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":720,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:06.325681+0000","last_change":"2024-02-05T08:11:20.552800+0000","last_active":"2024-02-05T08:12:06.325681+0000","last_peered":"2024-02-05T08:12:06.325681+0000","last_clean":"2024-02-05T08:12:06.325681+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:12:06.325681+0000","last_undegraded":"2024-02-05T08:12:06.325681+0000","last_fullsized":"2024-02-05T08:12:06.325681+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:11:20.552682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:11:20.552682+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:33.685592+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":440,"num_read_kb":435,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":265,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.681054+0000","last_change":"2024-02-05T08:09:19.836523+0000","last_active":"2024-02-05T08:11:53.681054+0000","last_peered":"2024-02-05T08:11:53.681054+0000","last_clean":"2024-02-05T08:11:53.681054+0000","last_became_active":"2024-02-05T08:04:43.549687+0000","last_became_peered":"2024-02-05T08:04:43.549687+0000","last_unstale":"2024-02-05T08:11:53.681054+0000","last_undegraded":"2024-02-05T08:11:53.681054+0000","last_fullsized":"2024-02-05T08:11:53.681054+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:09:19.836401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:19.836401+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679854+0000","last_change":"2024-02-05T08:11:03.475345+0000","last_active":"2024-02-05T08:11:53.679854+0000","last_peered":"2024-02-05T08:11:53.679854+0000","last_clean":"2024-02-05T08:11:53.679854+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:11:53.679854+0000","last_undegraded":"2024-02-05T08:11:53.679854+0000","last_fullsized":"2024-02-05T08:11:53.679854+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:03.475206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:03.475206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:03.647627+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"178'5154","reported_seq":5206,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:02.903379+0000","last_change":"2024-02-05T08:10:54.988633+0000","last_active":"2024-02-05T08:12:02.903379+0000","last_peered":"2024-02-05T08:12:02.903379+0000","last_clean":"2024-02-05T08:12:02.903379+0000","last_became_active":"2024-02-05T08:10:54.988338+0000","last_became_peered":"2024-02-05T08:10:54.988338+0000","last_unstale":"2024-02-05T08:12:02.903379+0000","last_undegraded":"2024-02-05T08:12:02.903379+0000","last_fullsized":"2024-02-05T08:12:02.903379+0000","mapping_epoch":157,"log_start":"165'800","ondisk_log_start":"165'800","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4354,"log_dups_size":0,"ondisk_log_size":4354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337772544,"num_objects":5154,"num_object_clones":0,"num_object_copies":15462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5154,"num_write_kb":329856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":254,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.683380+0000","last_change":"2024-02-05T08:10:18.609876+0000","last_active":"2024-02-05T08:11:53.683380+0000","last_peered":"2024-02-05T08:11:53.683380+0000","last_clean":"2024-02-05T08:11:53.683380+0000","last_became_active":"2024-02-05T08:06:39.559847+0000","last_became_peered":"2024-02-05T08:06:39.559847+0000","last_unstale":"2024-02-05T08:11:53.683380+0000","last_undegraded":"2024-02-05T08:11:53.683380+0000","last_fullsized":"2024-02-05T08:11:53.683380+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.609767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.609767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.850564+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":268,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.681723+0000","last_change":"2024-02-05T08:11:18.828143+0000","last_active":"2024-02-05T08:11:53.681723+0000","last_peered":"2024-02-05T08:11:53.681723+0000","last_clean":"2024-02-05T08:11:53.681723+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:11:53.681723+0000","last_undegraded":"2024-02-05T08:11:53.681723+0000","last_fullsized":"2024-02-05T08:11:53.681723+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:18.828029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:18.828029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:19.010299+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":291,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679889+0000","last_change":"2024-02-05T08:10:16.309686+0000","last_active":"2024-02-05T08:11:53.679889+0000","last_peered":"2024-02-05T08:11:53.679889+0000","last_clean":"2024-02-05T08:11:53.679889+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:11:53.679889+0000","last_undegraded":"2024-02-05T08:11:53.679889+0000","last_fullsized":"2024-02-05T08:11:53.679889+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:10:16.309551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:16.309551+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":190,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.680757+0000","last_change":"2024-02-05T08:11:21.296201+0000","last_active":"2024-02-05T08:11:53.680757+0000","last_peered":"2024-02-05T08:11:53.680757+0000","last_clean":"2024-02-05T08:11:53.680757+0000","last_became_active":"2024-02-05T08:06:39.558100+0000","last_became_peered":"2024-02-05T08:06:39.558100+0000","last_unstale":"2024-02-05T08:11:53.680757+0000","last_undegraded":"2024-02-05T08:11:53.680757+0000","last_fullsized":"2024-02-05T08:11:53.680757+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:21.296072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:11:21.296072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:23.327017+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"178'5225","reported_seq":5277,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:03.186820+0000","last_change":"2024-02-05T08:10:54.987464+0000","last_active":"2024-02-05T08:12:03.186820+0000","last_peered":"2024-02-05T08:12:03.186820+0000","last_clean":"2024-02-05T08:12:03.186820+0000","last_became_active":"2024-02-05T08:10:54.985442+0000","last_became_peered":"2024-02-05T08:10:54.985442+0000","last_unstale":"2024-02-05T08:12:03.186820+0000","last_undegraded":"2024-02-05T08:12:03.186820+0000","last_fullsized":"2024-02-05T08:12:03.186820+0000","mapping_epoch":157,"log_start":"164'300","ondisk_log_start":"164'300","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4925,"log_dups_size":0,"ondisk_log_size":4925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:05.506595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342425600,"num_objects":5225,"num_object_clones":0,"num_object_copies":15675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5225,"num_write_kb":334400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679431+0000","last_change":"2024-02-05T08:10:28.229516+0000","last_active":"2024-02-05T08:11:53.679431+0000","last_peered":"2024-02-05T08:11:53.679431+0000","last_clean":"2024-02-05T08:11:53.679431+0000","last_became_active":"2024-02-05T08:04:47.583866+0000","last_became_peered":"2024-02-05T08:04:47.583866+0000","last_unstale":"2024-02-05T08:11:53.679431+0000","last_undegraded":"2024-02-05T08:11:53.679431+0000","last_fullsized":"2024-02-05T08:11:53.679431+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:28.229441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:28.229441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":267,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679689+0000","last_change":"2024-02-05T08:11:26.212482+0000","last_active":"2024-02-05T08:11:53.679689+0000","last_peered":"2024-02-05T08:11:53.679689+0000","last_clean":"2024-02-05T08:11:53.679689+0000","last_became_active":"2024-02-05T08:04:41.537818+0000","last_became_peered":"2024-02-05T08:04:41.537818+0000","last_unstale":"2024-02-05T08:11:53.679689+0000","last_undegraded":"2024-02-05T08:11:53.679689+0000","last_fullsized":"2024-02-05T08:11:53.679689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:26.212363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:26.212363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:35.994909+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":271,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.683020+0000","last_change":"2024-02-05T08:10:21.554686+0000","last_active":"2024-02-05T08:11:53.683020+0000","last_peered":"2024-02-05T08:11:53.683020+0000","last_clean":"2024-02-05T08:11:53.683020+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:11:53.683020+0000","last_undegraded":"2024-02-05T08:11:53.683020+0000","last_fullsized":"2024-02-05T08:11:53.683020+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:10:21.554300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:21.554300+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.850527+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":358,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:56.676043+0000","last_change":"2024-02-05T08:11:13.252802+0000","last_active":"2024-02-05T08:11:56.676043+0000","last_peered":"2024-02-05T08:11:56.676043+0000","last_clean":"2024-02-05T08:11:56.676043+0000","last_became_active":"2024-02-05T08:06:39.554860+0000","last_became_peered":"2024-02-05T08:06:39.554860+0000","last_unstale":"2024-02-05T08:11:56.676043+0000","last_undegraded":"2024-02-05T08:11:56.676043+0000","last_fullsized":"2024-02-05T08:11:56.676043+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:11:13.252683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:13.252683+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:33.309577+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"178'5536","reported_seq":5588,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:07.539758+0000","last_change":"2024-02-05T08:10:54.986461+0000","last_active":"2024-02-05T08:12:07.539758+0000","last_peered":"2024-02-05T08:12:07.539758+0000","last_clean":"2024-02-05T08:12:07.539758+0000","last_became_active":"2024-02-05T08:10:54.986149+0000","last_became_peered":"2024-02-05T08:10:54.986149+0000","last_unstale":"2024-02-05T08:12:07.539758+0000","last_undegraded":"2024-02-05T08:12:07.539758+0000","last_fullsized":"2024-02-05T08:12:07.539758+0000","mapping_epoch":157,"log_start":"164'400","ondisk_log_start":"164'400","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5136,"log_dups_size":0,"ondisk_log_size":5136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:21.583807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362807296,"num_objects":5536,"num_object_clones":0,"num_object_copies":16608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5536,"num_write_kb":354304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":254,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.677298+0000","last_change":"2024-02-05T08:11:09.466858+0000","last_active":"2024-02-05T08:11:53.677298+0000","last_peered":"2024-02-05T08:11:53.677298+0000","last_clean":"2024-02-05T08:11:53.677298+0000","last_became_active":"2024-02-05T08:06:39.554165+0000","last_became_peered":"2024-02-05T08:06:39.554165+0000","last_unstale":"2024-02-05T08:11:53.677298+0000","last_undegraded":"2024-02-05T08:11:53.677298+0000","last_fullsized":"2024-02-05T08:11:53.677298+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:09.466721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:09.466721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:19.223486+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":270,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.681148+0000","last_change":"2024-02-05T08:11:23.215153+0000","last_active":"2024-02-05T08:11:53.681148+0000","last_peered":"2024-02-05T08:11:53.681148+0000","last_clean":"2024-02-05T08:11:53.681148+0000","last_became_active":"2024-02-05T08:11:23.200753+0000","last_became_peered":"2024-02-05T08:11:23.200753+0000","last_unstale":"2024-02-05T08:11:53.681148+0000","last_undegraded":"2024-02-05T08:11:53.681148+0000","last_fullsized":"2024-02-05T08:11:53.681148+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:10:30.240833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.240833+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":290,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667101+0000","last_change":"2024-02-05T08:10:16.458012+0000","last_active":"2024-02-05T08:11:53.667101+0000","last_peered":"2024-02-05T08:11:53.667101+0000","last_clean":"2024-02-05T08:11:53.667101+0000","last_became_active":"2024-02-05T08:06:39.566434+0000","last_became_peered":"2024-02-05T08:06:39.566434+0000","last_unstale":"2024-02-05T08:11:53.667101+0000","last_undegraded":"2024-02-05T08:11:53.667101+0000","last_fullsized":"2024-02-05T08:11:53.667101+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-02-05T08:10:16.457895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:10:16.457895+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.124660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"168'9","reported_seq":349,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:56.675690+0000","last_change":"2024-02-05T08:11:27.751314+0000","last_active":"2024-02-05T08:11:56.675690+0000","last_peered":"2024-02-05T08:11:56.675690+0000","last_clean":"2024-02-05T08:11:56.675690+0000","last_became_active":"2024-02-05T08:04:45.562597+0000","last_became_peered":"2024-02-05T08:04:45.562597+0000","last_unstale":"2024-02-05T08:11:56.675690+0000","last_undegraded":"2024-02-05T08:11:56.675690+0000","last_fullsized":"2024-02-05T08:11:56.675690+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'9","last_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:11:27.751188+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:49.016719+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"178'5042","reported_seq":5093,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:03.175006+0000","last_change":"2024-02-05T08:10:54.990174+0000","last_active":"2024-02-05T08:12:03.175006+0000","last_peered":"2024-02-05T08:12:03.175006+0000","last_clean":"2024-02-05T08:12:03.175006+0000","last_became_active":"2024-02-05T08:10:54.989454+0000","last_became_peered":"2024-02-05T08:10:54.989454+0000","last_unstale":"2024-02-05T08:12:03.175006+0000","last_undegraded":"2024-02-05T08:12:03.175006+0000","last_fullsized":"2024-02-05T08:12:03.175006+0000","mapping_epoch":157,"log_start":"164'200","ondisk_log_start":"164'200","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4842,"log_dups_size":0,"ondisk_log_size":4842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:05.506677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330432512,"num_objects":5042,"num_object_clones":0,"num_object_copies":15126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5042,"num_write_kb":322688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.666909+0000","last_change":"2024-02-05T08:10:15.488463+0000","last_active":"2024-02-05T08:11:53.666909+0000","last_peered":"2024-02-05T08:11:53.666909+0000","last_clean":"2024-02-05T08:11:53.666909+0000","last_became_active":"2024-02-05T08:04:47.582776+0000","last_became_peered":"2024-02-05T08:04:47.582776+0000","last_unstale":"2024-02-05T08:11:53.666909+0000","last_undegraded":"2024-02-05T08:11:53.666909+0000","last_fullsized":"2024-02-05T08:11:53.666909+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:15.488380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:15.488380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.124623+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":265,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676644+0000","last_change":"2024-02-05T08:09:36.111657+0000","last_active":"2024-02-05T08:11:53.676644+0000","last_peered":"2024-02-05T08:11:53.676644+0000","last_clean":"2024-02-05T08:11:53.676644+0000","last_became_active":"2024-02-05T08:04:41.540818+0000","last_became_peered":"2024-02-05T08:04:41.540818+0000","last_unstale":"2024-02-05T08:11:53.676644+0000","last_undegraded":"2024-02-05T08:11:53.676644+0000","last_fullsized":"2024-02-05T08:11:53.676644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:09:36.111537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:36.111537+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.979973+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":287,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.689485+0000","last_change":"2024-02-05T08:09:33.222821+0000","last_active":"2024-02-05T08:11:53.689485+0000","last_peered":"2024-02-05T08:11:53.689485+0000","last_clean":"2024-02-05T08:11:53.689485+0000","last_became_active":"2024-02-05T08:04:43.607546+0000","last_became_peered":"2024-02-05T08:04:43.607546+0000","last_unstale":"2024-02-05T08:11:53.689485+0000","last_undegraded":"2024-02-05T08:11:53.689485+0000","last_fullsized":"2024-02-05T08:11:53.689485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-02-05T08:09:33.222532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:33.222532+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.980002+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.666868+0000","last_change":"2024-02-05T08:11:28.185998+0000","last_active":"2024-02-05T08:11:53.666868+0000","last_peered":"2024-02-05T08:11:53.666868+0000","last_clean":"2024-02-05T08:11:53.666868+0000","last_became_active":"2024-02-05T08:04:45.561639+0000","last_became_peered":"2024-02-05T08:04:45.561639+0000","last_unstale":"2024-02-05T08:11:53.666868+0000","last_undegraded":"2024-02-05T08:11:53.666868+0000","last_fullsized":"2024-02-05T08:11:53.666868+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:28.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:28.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:41.074874+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"178'5142","reported_seq":5194,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:02.950739+0000","last_change":"2024-02-05T08:10:54.984735+0000","last_active":"2024-02-05T08:12:02.950739+0000","last_peered":"2024-02-05T08:12:02.950739+0000","last_clean":"2024-02-05T08:12:02.950739+0000","last_became_active":"2024-02-05T08:10:54.984306+0000","last_became_peered":"2024-02-05T08:10:54.984306+0000","last_unstale":"2024-02-05T08:12:02.950739+0000","last_undegraded":"2024-02-05T08:12:02.950739+0000","last_fullsized":"2024-02-05T08:12:02.950739+0000","mapping_epoch":157,"log_start":"165'900","ondisk_log_start":"165'900","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:17.368690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336986112,"num_objects":5142,"num_object_clones":0,"num_object_copies":15426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5142,"num_write_kb":329088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":255,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679375+0000","last_change":"2024-02-05T08:11:08.491427+0000","last_active":"2024-02-05T08:11:53.679375+0000","last_peered":"2024-02-05T08:11:53.679375+0000","last_clean":"2024-02-05T08:11:53.679375+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:11:53.679375+0000","last_undegraded":"2024-02-05T08:11:53.679375+0000","last_fullsized":"2024-02-05T08:11:53.679375+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:08.491314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:08.491314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:11.472763+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":349,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:01.675918+0000","last_change":"2024-02-05T08:11:30.179068+0000","last_active":"2024-02-05T08:12:01.675918+0000","last_peered":"2024-02-05T08:12:01.675918+0000","last_clean":"2024-02-05T08:12:01.675918+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:12:01.675918+0000","last_undegraded":"2024-02-05T08:12:01.675918+0000","last_fullsized":"2024-02-05T08:12:01.675918+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:11:30.178959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:30.178959+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:51.082949+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":262,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.689405+0000","last_change":"2024-02-05T08:11:03.761651+0000","last_active":"2024-02-05T08:11:53.689405+0000","last_peered":"2024-02-05T08:11:53.689405+0000","last_clean":"2024-02-05T08:11:53.689405+0000","last_became_active":"2024-02-05T08:04:41.541327+0000","last_became_peered":"2024-02-05T08:04:41.541327+0000","last_unstale":"2024-02-05T08:11:53.689405+0000","last_undegraded":"2024-02-05T08:11:53.689405+0000","last_fullsized":"2024-02-05T08:11:53.689405+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:03.761546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:03.761546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:30.728931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":289,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.678269+0000","last_change":"2024-02-05T08:11:30.843753+0000","last_active":"2024-02-05T08:11:53.678269+0000","last_peered":"2024-02-05T08:11:53.678269+0000","last_clean":"2024-02-05T08:11:53.678269+0000","last_became_active":"2024-02-05T08:04:43.555028+0000","last_became_peered":"2024-02-05T08:04:43.555028+0000","last_unstale":"2024-02-05T08:11:53.678269+0000","last_undegraded":"2024-02-05T08:11:53.678269+0000","last_fullsized":"2024-02-05T08:11:53.678269+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:11:30.843620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:30.843620+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:40.001755+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"178'5375","reported_seq":5427,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:07.530534+0000","last_change":"2024-02-05T08:10:54.985813+0000","last_active":"2024-02-05T08:12:07.530534+0000","last_peered":"2024-02-05T08:12:07.530534+0000","last_clean":"2024-02-05T08:12:07.530534+0000","last_became_active":"2024-02-05T08:10:54.985501+0000","last_became_peered":"2024-02-05T08:10:54.985501+0000","last_unstale":"2024-02-05T08:12:07.530534+0000","last_undegraded":"2024-02-05T08:12:07.530534+0000","last_fullsized":"2024-02-05T08:12:07.530534+0000","mapping_epoch":157,"log_start":"165'800","ondisk_log_start":"165'800","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4575,"log_dups_size":0,"ondisk_log_size":4575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:19.562319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352256000,"num_objects":5375,"num_object_clones":0,"num_object_copies":16125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5375,"num_write_kb":344000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":245,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.675409+0000","last_change":"2024-02-05T08:09:32.213156+0000","last_active":"2024-02-05T08:11:53.675409+0000","last_peered":"2024-02-05T08:11:53.675409+0000","last_clean":"2024-02-05T08:11:53.675409+0000","last_became_active":"2024-02-05T08:06:39.558047+0000","last_became_peered":"2024-02-05T08:06:39.558047+0000","last_unstale":"2024-02-05T08:11:53.675409+0000","last_undegraded":"2024-02-05T08:11:53.675409+0000","last_fullsized":"2024-02-05T08:11:53.675409+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:32.213044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:09:32.213044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.980029+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":265,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.679760+0000","last_change":"2024-02-05T08:11:29.241996+0000","last_active":"2024-02-05T08:11:53.679760+0000","last_peered":"2024-02-05T08:11:53.679760+0000","last_clean":"2024-02-05T08:11:53.679760+0000","last_became_active":"2024-02-05T08:06:39.551933+0000","last_became_peered":"2024-02-05T08:06:39.551933+0000","last_unstale":"2024-02-05T08:11:53.679760+0000","last_undegraded":"2024-02-05T08:11:53.679760+0000","last_fullsized":"2024-02-05T08:11:53.679760+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:29.241689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:29.241689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:35.925371+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":267,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.666952+0000","last_change":"2024-02-05T08:11:31.201157+0000","last_active":"2024-02-05T08:11:53.666952+0000","last_peered":"2024-02-05T08:11:53.666952+0000","last_clean":"2024-02-05T08:11:53.666952+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:11:53.666952+0000","last_undegraded":"2024-02-05T08:11:53.666952+0000","last_fullsized":"2024-02-05T08:11:53.666952+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:11:31.201062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:41.625348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":292,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667585+0000","last_change":"2024-02-05T08:11:10.470892+0000","last_active":"2024-02-05T08:11:53.667585+0000","last_peered":"2024-02-05T08:11:53.667585+0000","last_clean":"2024-02-05T08:11:53.667585+0000","last_became_active":"2024-02-05T08:06:39.556433+0000","last_became_peered":"2024-02-05T08:06:39.556433+0000","last_unstale":"2024-02-05T08:11:53.667585+0000","last_undegraded":"2024-02-05T08:11:53.667585+0000","last_fullsized":"2024-02-05T08:11:53.667585+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:11:10.470749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:10.470749+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:36.153341+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"178'5080","reported_seq":5132,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:02.966621+0000","last_change":"2024-02-05T08:10:54.982668+0000","last_active":"2024-02-05T08:12:02.966621+0000","last_peered":"2024-02-05T08:12:02.966621+0000","last_clean":"2024-02-05T08:12:02.966621+0000","last_became_active":"2024-02-05T08:10:54.982372+0000","last_became_peered":"2024-02-05T08:10:54.982372+0000","last_unstale":"2024-02-05T08:12:02.966621+0000","last_undegraded":"2024-02-05T08:12:02.966621+0000","last_fullsized":"2024-02-05T08:12:02.966621+0000","mapping_epoch":157,"log_start":"165'800","ondisk_log_start":"165'800","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4280,"log_dups_size":0,"ondisk_log_size":4280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:03.879428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332922880,"num_objects":5080,"num_object_clones":0,"num_object_copies":15240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5080,"num_write_kb":325120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":257,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.680806+0000","last_change":"2024-02-05T08:11:06.399741+0000","last_active":"2024-02-05T08:11:53.680806+0000","last_peered":"2024-02-05T08:11:53.680806+0000","last_clean":"2024-02-05T08:11:53.680806+0000","last_became_active":"2024-02-05T08:04:47.585962+0000","last_became_peered":"2024-02-05T08:04:47.585962+0000","last_unstale":"2024-02-05T08:11:53.680806+0000","last_undegraded":"2024-02-05T08:11:53.680806+0000","last_fullsized":"2024-02-05T08:11:53.680806+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:11:06.399635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:06.399635+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:11.606714+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":252,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.681396+0000","last_change":"2024-02-05T08:11:11.789034+0000","last_active":"2024-02-05T08:11:53.681396+0000","last_peered":"2024-02-05T08:11:53.681396+0000","last_clean":"2024-02-05T08:11:53.681396+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:11:53.681396+0000","last_undegraded":"2024-02-05T08:11:53.681396+0000","last_fullsized":"2024-02-05T08:11:53.681396+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:11.788902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:11.788902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:31.941871+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":190,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.666977+0000","last_change":"2024-02-05T08:10:25.446805+0000","last_active":"2024-02-05T08:11:53.666977+0000","last_peered":"2024-02-05T08:11:53.666977+0000","last_clean":"2024-02-05T08:11:53.666977+0000","last_became_active":"2024-02-05T08:06:39.559090+0000","last_became_peered":"2024-02-05T08:06:39.559090+0000","last_unstale":"2024-02-05T08:11:53.666977+0000","last_undegraded":"2024-02-05T08:11:53.666977+0000","last_fullsized":"2024-02-05T08:11:53.666977+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:25.446724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:10:25.446724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.124675+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":275,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.668731+0000","last_change":"2024-02-05T08:09:21.434983+0000","last_active":"2024-02-05T08:11:53.668731+0000","last_peered":"2024-02-05T08:11:53.668731+0000","last_clean":"2024-02-05T08:11:53.668731+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:11:53.668731+0000","last_undegraded":"2024-02-05T08:11:53.668731+0000","last_fullsized":"2024-02-05T08:11:53.668731+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-02-05T08:09:21.434891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:09:21.434891+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:57.124551+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"178'5125","reported_seq":5177,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:12:02.875299+0000","last_change":"2024-02-05T08:10:54.989334+0000","last_active":"2024-02-05T08:12:02.875299+0000","last_peered":"2024-02-05T08:12:02.875299+0000","last_clean":"2024-02-05T08:12:02.875299+0000","last_became_active":"2024-02-05T08:10:54.988993+0000","last_became_peered":"2024-02-05T08:10:54.988993+0000","last_unstale":"2024-02-05T08:12:02.875299+0000","last_undegraded":"2024-02-05T08:12:02.875299+0000","last_fullsized":"2024-02-05T08:12:02.875299+0000","mapping_epoch":157,"log_start":"165'700","ondisk_log_start":"165'700","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4425,"log_dups_size":0,"ondisk_log_size":4425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:14.963803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335872000,"num_objects":5125,"num_object_clones":0,"num_object_copies":15375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5125,"num_write_kb":328000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":247,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667841+0000","last_change":"2024-02-05T08:11:15.360217+0000","last_active":"2024-02-05T08:11:53.667841+0000","last_peered":"2024-02-05T08:11:53.667841+0000","last_clean":"2024-02-05T08:11:53.667841+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:11:53.667841+0000","last_undegraded":"2024-02-05T08:11:53.667841+0000","last_fullsized":"2024-02-05T08:11:53.667841+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.360090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:39.893177+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":358,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:56.678344+0000","last_change":"2024-02-05T08:10:31.516653+0000","last_active":"2024-02-05T08:11:56.678344+0000","last_peered":"2024-02-05T08:11:56.678344+0000","last_clean":"2024-02-05T08:11:56.678344+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:11:56.678344+0000","last_undegraded":"2024-02-05T08:11:56.678344+0000","last_fullsized":"2024-02-05T08:11:56.678344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:10:31.516571+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:10:31.516571+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":251,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.676318+0000","last_change":"2024-02-05T08:09:22.380135+0000","last_active":"2024-02-05T08:11:53.676318+0000","last_peered":"2024-02-05T08:11:53.676318+0000","last_clean":"2024-02-05T08:11:53.676318+0000","last_became_active":"2024-02-05T08:04:41.533838+0000","last_became_peered":"2024-02-05T08:04:41.533838+0000","last_unstale":"2024-02-05T08:11:53.676318+0000","last_undegraded":"2024-02-05T08:11:53.676318+0000","last_fullsized":"2024-02-05T08:11:53.676318+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:09:22.380016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:09:22.380016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:11:54.979858+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":288,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-02-05T08:11:53.667531+0000","last_change":"2024-02-05T08:11:14.388931+0000","last_active":"2024-02-05T08:11:53.667531+0000","last_peered":"2024-02-05T08:11:53.667531+0000","last_clean":"2024-02-05T08:11:53.667531+0000","last_became_active":"2024-02-05T08:06:39.555561+0000","last_became_peered":"2024-02-05T08:06:39.555561+0000","last_unstale":"2024-02-05T08:11:53.667531+0000","last_undegraded":"2024-02-05T08:11:53.667531+0000","last_fullsized":"2024-02-05T08:11:53.667531+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:11:14.388825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:14.388825+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:39.267898+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5472845824,"num_objects":83509,"num_object_clones":0,"num_object_copies":250527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":83509,"num_write_kb":5344576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16411721728,"data_stored":16411721728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74609,"ondisk_log_size":74609,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":440,"num_read_kb":435,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":1474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":433,"ondisk_log_size":433,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037854,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1879952,"kb_used_data":1732168,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":91865200,"statfs":{"total":95995035648,"available":94069964800,"internally_reserved":0,"allocated":1773740032,"data_stored":1773355782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":151321034},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4929999999999999,"5min":1.9810000000000001,"15min":1.026},"min":{"1min":0.30099999999999999,"5min":0.218,"15min":0.217},"max":{"1min":37.514000000000003,"5min":60.127000000000002,"15min":60.127000000000002},"last":1.292},{"interface":"front","average":{"1min":2.556,"5min":2.0009999999999999,"15min":1.0149999999999999},"min":{"1min":0.251,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":38.360999999999997,"5min":58.283000000000001,"15min":58.283000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon Feb 5 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":2.5990000000000002,"5min":1.823,"15min":0.97699999999999998},"min":{"1min":0.41599999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":38.399999999999999,"5min":38.399999999999999,"15min":38.399999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":2.6779999999999999,"5min":1.528,"15min":0.874},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":38.195999999999998,"5min":38.195999999999998,"15min":38.195999999999998},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":1.0589999999999999,"15min":0.73699999999999999},"min":{"1min":0.33200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.9259999999999999,"5min":7.6779999999999999,"15min":7.6779999999999999},"last":0.752},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.254,"15min":0.77600000000000002},"min":{"1min":0.314,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.9900000000000002,"5min":8.9100000000000001,"15min":8.9100000000000001},"last":0.46000000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":2.0459999999999998,"15min":1.0589999999999999},"min":{"1min":0.29099999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.7050000000000001,"5min":58.515000000000001,"15min":58.515000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.9830000000000001,"15min":1.02},"min":{"1min":0.35599999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.8090000000000002,"5min":58.225999999999999,"15min":58.225999999999999},"last":0.372}]},{"osd":4,"last update":"Mon Feb 5 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.39200000000000002,"15min":0.42299999999999999},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.91400000000000003},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.40600000000000003,"15min":0.42899999999999999},"min":{"1min":0.33700000000000002,"5min":0.249,"15min":0.249},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.42099999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.45400000000000001,"15min":0.44700000000000001},"min":{"1min":0.29799999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.46600000000000003,"15min":0.48499999999999999},"min":{"1min":0.34899999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.998},"last":0.436}]},{"osd":6,"last update":"Mon Feb 5 08:11:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.48399999999999999,"15min":0.499},"min":{"1min":0.35799999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.0269999999999999,"5min":1.256,"15min":1.256},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.39500000000000002,"5min":0.308,"15min":0.308},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.50700000000000001}]}]},{"osd":6,"up_from":0,"seq":219043332196,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574972,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2539688,"kb_used_data":2278048,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":91205464,"statfs":{"total":95995035648,"available":93394395136,"internally_reserved":0,"allocated":2332721152,"data_stored":2332312034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":2.3929999999999998,"15min":1.01},"min":{"1min":0.253,"5min":0.186,"15min":0.14699999999999999},"max":{"1min":2.5499999999999998,"5min":86.105000000000004,"15min":86.105000000000004},"last":2.0350000000000001},{"interface":"front","average":{"1min":1.0269999999999999,"5min":2.8140000000000001,"15min":1.1619999999999999},"min":{"1min":0.28699999999999998,"5min":0.245,"15min":0.16},"max":{"1min":8.4510000000000005,"5min":85.665000000000006,"15min":85.665000000000006},"last":1.345}]},{"osd":2,"last update":"Mon Feb 5 08:11:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":2.1829999999999998,"15min":0.97799999999999998},"min":{"1min":0.191,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":8.9540000000000006,"5min":49.030999999999999,"15min":49.030999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.292,"5min":2.4910000000000001,"15min":1.1000000000000001},"min":{"1min":0.253,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.3109999999999999,"5min":48.984000000000002,"15min":48.984000000000002},"last":2.052}]},{"osd":3,"last update":"Mon Feb 5 08:12:00 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":2.4950000000000001,"15min":1.121},"min":{"1min":0.34499999999999997,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":2.8380000000000001,"5min":49.067999999999998,"15min":49.067999999999998},"last":1.444},{"interface":"front","average":{"1min":1.3660000000000001,"5min":2.5230000000000001,"15min":1.1459999999999999},"min":{"1min":0.31,"5min":0.27800000000000002,"15min":0.24199999999999999},"max":{"1min":6.8029999999999999,"5min":48.734999999999999,"15min":48.734999999999999},"last":1.3899999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:11:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":1.0920000000000001,"15min":0.75700000000000001},"min":{"1min":0.35399999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.278,"5min":9.4269999999999996,"15min":9.4269999999999996},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.2270000000000001,"15min":0.79400000000000004},"min":{"1min":0.38500000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.3380000000000001,"5min":6.3890000000000002,"15min":6.3890000000000002},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.1599999999999999,"15min":0.755},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.24,"5min":9.4559999999999995,"15min":9.4559999999999995},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.6970000000000001,"5min":1.4330000000000001,"15min":0.85099999999999998},"min":{"1min":0.62,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.8310000000000004,"5min":13.776,"15min":13.776},"last":1.3080000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:11:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":1.2170000000000001,"15min":0.80100000000000005},"min":{"1min":0.38,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":2.3660000000000001,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":1.1910000000000001,"15min":0.82099999999999995},"min":{"1min":0.42399999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86699999999999999,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":0.56999999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:11:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.5269999999999999,"15min":0.92100000000000004},"min":{"1min":0.39500000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.9059999999999997,"5min":13.749000000000001,"15min":13.749000000000001},"last":0.628},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.381,"15min":0.86599999999999999},"min":{"1min":0.436,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":2.9119999999999999,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.53200000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738497,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2525736,"kb_used_data":2273824,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91219416,"statfs":{"total":95995035648,"available":93408681984,"internally_reserved":0,"allocated":2328395776,"data_stored":2327987484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:11:15 2024","interfaces":[{"interface":"back","average":{"1min":2.2559999999999998,"5min":3.028,"15min":1.2370000000000001},"min":{"1min":0.28799999999999998,"5min":0.19,"15min":0.152},"max":{"1min":32.066000000000003,"5min":66.772999999999996,"15min":66.772999999999996},"last":6.774},{"interface":"front","average":{"1min":0.497,"5min":2.1110000000000002,"15min":0.93300000000000005},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.17199999999999999},"max":{"1min":2.2610000000000001,"5min":46.198999999999998,"15min":46.198999999999998},"last":1.4339999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:11:43 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":2.3109999999999999,"15min":1.0209999999999999},"min":{"1min":0.28699999999999998,"5min":0.216,"15min":0.216},"max":{"1min":4.6230000000000002,"5min":46.170999999999999,"15min":46.170999999999999},"last":0.24399999999999999},{"interface":"front","average":{"1min":1.161,"5min":2.6850000000000001,"15min":1.157},"min":{"1min":0.26800000000000002,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.0410000000000004,"5min":46.543999999999997,"15min":46.543999999999997},"last":6.8209999999999997}]},{"osd":3,"last update":"Mon Feb 5 08:11:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":2.5960000000000001,"15min":1.1699999999999999},"min":{"1min":0.34499999999999997,"5min":0.28399999999999997,"15min":0.21099999999999999},"max":{"1min":7.5970000000000004,"5min":67.888000000000005,"15min":67.888000000000005},"last":2.8889999999999998},{"interface":"front","average":{"1min":3.512,"5min":3.3340000000000001,"15min":1.4219999999999999},"min":{"1min":0.28599999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":37.262999999999998,"5min":66.891999999999996,"15min":66.891999999999996},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:11:32 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.1859999999999999,"15min":0.77100000000000002},"min":{"1min":0.34499999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":3.3039999999999998,"5min":13.295,"15min":13.295},"last":1.3620000000000001},{"interface":"front","average":{"1min":0.80900000000000005,"5min":1.2749999999999999,"15min":0.82899999999999996},"min":{"1min":0.28999999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.8220000000000001,"5min":12.989000000000001,"15min":12.989000000000001},"last":0.377}]},{"osd":5,"last update":"Mon Feb 5 08:11:51 2024","interfaces":[{"interface":"back","average":{"1min":1.768,"5min":1.538,"15min":0.91700000000000004},"min":{"1min":0.41599999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":9.1660000000000004,"5min":13.073,"15min":13.073},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3640000000000001,"15min":0.86799999999999999},"min":{"1min":0.38,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.7839999999999998,"5min":12.94,"15min":12.94},"last":0.34899999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:11:15 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":1.321,"15min":0.84099999999999997},"min":{"1min":0.45600000000000002,"5min":0.378,"15min":0.378},"max":{"1min":2.1549999999999998,"5min":12.967000000000001,"15min":12.967000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.363,"15min":0.86899999999999999},"min":{"1min":0.47699999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.9490000000000001,"5min":13.108000000000001,"15min":13.108000000000001},"last":0.497}]},{"osd":7,"last update":"Mon Feb 5 08:11:43 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.228,"15min":0.78600000000000003},"min":{"1min":0.33600000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.2759999999999998,"5min":13.263,"15min":13.263},"last":1.4039999999999999},{"interface":"front","average":{"1min":0.90400000000000003,"5min":1.4099999999999999,"15min":0.877},"min":{"1min":0.45000000000000001,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":3.8660000000000001,"5min":13.012,"15min":13.012},"last":2.8439999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411447,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3160084,"kb_used_data":2926220,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":90585068,"statfs":{"total":95995035648,"available":92759109632,"internally_reserved":0,"allocated":2996449280,"data_stored":2996059696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":239466950},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:11:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":2.1930000000000001,"15min":0.97899999999999998},"min":{"1min":0.27500000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":2.121,"5min":16.155999999999999,"15min":16.155999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.3859999999999999,"5min":2.306,"15min":1.026},"min":{"1min":0.32600000000000001,"5min":0.23300000000000001,"15min":0.20699999999999999},"max":{"1min":5.3520000000000003,"5min":15.973000000000001,"15min":15.973000000000001},"last":1.7969999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:11:44 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":2.0150000000000001,"15min":0.91200000000000003},"min":{"1min":0.25900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":5.1970000000000001,"5min":9.1340000000000003,"15min":9.1340000000000003},"last":13.935},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.998,"15min":0.92500000000000004},"min":{"1min":0.34100000000000003,"5min":0.193,"15min":0.193},"max":{"1min":6.8049999999999997,"5min":9.2769999999999992,"15min":9.2769999999999992},"last":2.597}]},{"osd":3,"last update":"Mon Feb 5 08:11:01 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":3.1389999999999998,"15min":1.3580000000000001},"min":{"1min":0.32800000000000001,"5min":0.307,"15min":0.27700000000000002},"max":{"1min":5.4850000000000003,"5min":68.108000000000004,"15min":68.108000000000004},"last":5.2880000000000003},{"interface":"front","average":{"1min":1.776,"5min":2.9319999999999999,"15min":1.28},"min":{"1min":0.32700000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":7.8879999999999999,"5min":67.945999999999998,"15min":67.945999999999998},"last":0.36799999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:11:24 2024","interfaces":[{"interface":"back","average":{"1min":3.0640000000000001,"5min":1.802,"15min":1.0009999999999999},"min":{"1min":0.443,"5min":0.315,"15min":0.315},"max":{"1min":31.596,"5min":31.596,"15min":31.596},"last":1.839},{"interface":"front","average":{"1min":2.8730000000000002,"5min":1.5149999999999999,"15min":0.89200000000000002},"min":{"1min":0.42899999999999999,"5min":0.252,"15min":0.252},"max":{"1min":31.57,"5min":31.57,"15min":31.57},"last":0.42799999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:11:52 2024","interfaces":[{"interface":"back","average":{"1min":4.3220000000000001,"5min":1.968,"15min":1.0469999999999999},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":58.799999999999997,"5min":58.799999999999997,"15min":58.799999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":4.8440000000000003,"5min":1.9750000000000001,"15min":1.04},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":58.615000000000002,"5min":58.615000000000002,"15min":58.615000000000002},"last":1.9350000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:11:24 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":1.4019999999999999,"15min":0.89100000000000001},"min":{"1min":0.496,"5min":0.308,"15min":0.308},"max":{"1min":2.9220000000000002,"5min":15.859,"15min":15.859},"last":1.8700000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":1.133,"15min":0.80300000000000005},"min":{"1min":0.45500000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.5349999999999999,"5min":6.117,"15min":6.117},"last":0.45400000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:11:44 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.502,"15min":0.91200000000000003},"min":{"1min":0.34100000000000003,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":6.343,"5min":15.986000000000001,"15min":15.986000000000001},"last":1.9650000000000001},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.5649999999999999,"15min":0.96499999999999997},"min":{"1min":0.38,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":3.4049999999999998,"5min":15.951000000000001,"15min":15.951000000000001},"last":1.899}]}]},{"osd":3,"up_from":27,"seq":115964117106,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2545188,"kb_used_data":2247452,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":91199964,"statfs":{"total":95995035648,"available":93388763136,"internally_reserved":0,"allocated":2301390848,"data_stored":2300985537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":304871881},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:12:01 2024","interfaces":[{"interface":"back","average":{"1min":1.8049999999999999,"5min":2.5409999999999999,"15min":1.1160000000000001},"min":{"1min":0.21299999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.7599999999999998,"5min":63.43,"15min":63.43},"last":3.6419999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":2.456,"15min":1.1100000000000001},"min":{"1min":0.317,"5min":0.186,"15min":0.186},"max":{"1min":3.4089999999999998,"5min":63.048999999999999,"15min":63.048999999999999},"last":0.82299999999999995}]},{"osd":1,"last update":"Mon Feb 5 08:12:01 2024","interfaces":[{"interface":"back","average":{"1min":2.137,"5min":1.958,"15min":0.96499999999999997},"min":{"1min":0.23799999999999999,"5min":0.217,"15min":0.217},"max":{"1min":10.452999999999999,"5min":10.452999999999999,"15min":10.452999999999999},"last":2.0990000000000002},{"interface":"front","average":{"1min":1.417,"5min":2.4369999999999998,"15min":1.1120000000000001},"min":{"1min":0.26500000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":4.3550000000000004,"5min":12.087999999999999,"15min":12.087999999999999},"last":0.64600000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:12:01 2024","interfaces":[{"interface":"back","average":{"1min":1.788,"5min":2.762,"15min":1.244},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":6.2370000000000001,"5min":62.902999999999999,"15min":62.902999999999999},"last":0.378},{"interface":"front","average":{"1min":1.744,"5min":2.5710000000000002,"15min":1.1699999999999999},"min":{"1min":0.27200000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.8660000000000001,"5min":63.468000000000004,"15min":63.468000000000004},"last":3.27}]},{"osd":4,"last update":"Mon Feb 5 08:11:24 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.3520000000000001,"15min":0.82599999999999996},"min":{"1min":0.32600000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.5339999999999998,"5min":7.7229999999999999,"15min":7.7229999999999999},"last":3.3079999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":1.413,"15min":0.86499999999999999},"min":{"1min":0.38300000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.1379999999999999,"5min":8.7059999999999995,"15min":8.7059999999999995},"last":0.29999999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:11:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.4159999999999999,"15min":0.86499999999999999},"min":{"1min":0.36699999999999999,"5min":0.255,"15min":0.255},"max":{"1min":3.5699999999999998,"5min":7.6680000000000001,"15min":7.6680000000000001},"last":3.855},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.512,"15min":0.90700000000000003},"min":{"1min":0.39300000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.0840000000000001,"5min":8.734,"15min":8.734},"last":0.39600000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:11:15 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":1.454,"15min":0.88400000000000001},"min":{"1min":0.39400000000000002,"5min":0.372,"15min":0.36499999999999999},"max":{"1min":2.2360000000000002,"5min":8.9429999999999996,"15min":8.9429999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.476,"15min":0.88},"min":{"1min":0.39300000000000002,"5min":0.308,"15min":0.308},"max":{"1min":2.6949999999999998,"5min":7.8639999999999999,"15min":7.8639999999999999},"last":0.60699999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:11:33 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.4370000000000001,"15min":0.874},"min":{"1min":0.39500000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.6070000000000002,"5min":7.7000000000000002,"15min":7.7000000000000002},"last":3.3580000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":1.571,"15min":0.94299999999999995},"min":{"1min":0.36399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.044,"5min":8.9900000000000002,"15min":8.9900000000000002},"last":0.42899999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953582,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1863892,"kb_used_data":1720268,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91881260,"statfs":{"total":95995035648,"available":94086410240,"internally_reserved":0,"allocated":1761554432,"data_stored":1761164299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:11:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":2.2509999999999999,"15min":1.115},"min":{"1min":0.29499999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.76700000000000002,"5min":73.078999999999994,"15min":73.078999999999994},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":2.0950000000000002,"15min":1.0620000000000001},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.0089999999999999,"5min":73.412000000000006,"15min":73.412000000000006},"last":5.5750000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:11:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.98399999999999999,"15min":0.68899999999999995},"min":{"1min":0.39300000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.181,"5min":8.3409999999999993,"15min":8.3409999999999993},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.78100000000000003,"15min":0.623},"min":{"1min":0.39100000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":3.367,"5min":4.3120000000000003,"15min":4.3120000000000003},"last":0.313}]},{"osd":2,"last update":"Mon Feb 5 08:11:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":1.883,"15min":0.98899999999999999},"min":{"1min":0.32200000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.119,"5min":96.849999999999994,"15min":96.849999999999994},"last":5.649},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.9770000000000001,"15min":1.04},"min":{"1min":0.309,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.9510000000000001,"5min":96.143000000000001,"15min":96.143000000000001},"last":1.6100000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:11:14 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":2.5609999999999999,"15min":1.22},"min":{"1min":0.33700000000000002,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.82999999999999996,"5min":73.602999999999994,"15min":73.602999999999994},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":2.5830000000000002,"15min":1.2270000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.2949999999999999,"5min":74.817999999999998,"15min":74.817999999999998},"last":0.56599999999999995}]},{"osd":5,"last update":"Mon Feb 5 08:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.41099999999999998,"15min":0.39600000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.45400000000000001,"15min":0.42699999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.252,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.63100000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:11:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.46400000000000002,"15min":0.443},"min":{"1min":0.33900000000000002,"5min":0.30499999999999999,"15min":0.29299999999999998},"max":{"1min":0.77100000000000002,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.46800000000000003,"15min":0.442},"min":{"1min":0.35399999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.752,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.72899999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.46700000000000003,"15min":0.47699999999999998},"min":{"1min":0.35099999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.78200000000000003,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.49299999999999999,"15min":0.50700000000000001},"min":{"1min":0.34399999999999997,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.80500000000000005,"5min":1.147,"15min":1.147},"last":0.69599999999999995}]}]},{"osd":5,"up_from":42,"seq":180388626536,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2907780,"kb_used_data":2759996,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":90837372,"statfs":{"total":95995035648,"available":93017468928,"internally_reserved":0,"allocated":2826235904,"data_stored":2825862128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:11:47 2024","interfaces":[{"interface":"back","average":{"1min":3.5369999999999999,"5min":2.988,"15min":1.363},"min":{"1min":0.22800000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":46.962000000000003,"5min":143.06299999999999,"15min":143.06299999999999},"last":0.28999999999999998},{"interface":"front","average":{"1min":3.4860000000000002,"5min":2.9910000000000001,"15min":1.351},"min":{"1min":0.26700000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":47.067,"5min":143.09700000000001,"15min":143.09700000000001},"last":0.40200000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:11:47 2024","interfaces":[{"interface":"back","average":{"1min":3.4340000000000002,"5min":1.3600000000000001,"15min":0.82999999999999996},"min":{"1min":0.28899999999999998,"5min":0.215,"15min":0.215},"max":{"1min":46.200000000000003,"5min":46.200000000000003,"15min":46.200000000000003},"last":0.25},{"interface":"front","average":{"1min":3.7229999999999999,"5min":1.6020000000000001,"15min":0.89400000000000002},"min":{"1min":0.30299999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":46.343000000000004,"5min":46.343000000000004,"15min":46.343000000000004},"last":1.9610000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:11:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.179,"15min":0.77200000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":3.1389999999999998,"5min":10.733000000000001,"15min":10.733000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.216,"15min":0.77900000000000003},"min":{"1min":0.31900000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.3879999999999999,"5min":5.4089999999999998,"15min":5.4089999999999998},"last":8.0220000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:11:47 2024","interfaces":[{"interface":"back","average":{"1min":3.7839999999999998,"5min":3.1549999999999998,"15min":1.4259999999999999},"min":{"1min":0.32000000000000001,"5min":0.309,"15min":0.309},"max":{"1min":46.569000000000003,"5min":143.03399999999999,"15min":143.03399999999999},"last":0.501},{"interface":"front","average":{"1min":3.3250000000000002,"5min":3.0099999999999998,"15min":1.397},"min":{"1min":0.41799999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":47.820999999999998,"5min":143.46799999999999,"15min":143.46799999999999},"last":0.32600000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:11:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.44400000000000001,"15min":0.41799999999999998},"min":{"1min":0.31900000000000001,"5min":0.255,"15min":0.253},"max":{"1min":1.1759999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.45300000000000001,"15min":0.435},"min":{"1min":0.28399999999999997,"5min":0.26900000000000002,"15min":0.26500000000000001},"max":{"1min":0.69999999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.309}]},{"osd":6,"last update":"Mon Feb 5 08:11:09 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.46500000000000002,"15min":0.45300000000000001},"min":{"1min":0.38900000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.50900000000000001,"15min":0.496},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.35399999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:11:42 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.51200000000000001,"15min":0.48799999999999999},"min":{"1min":0.37,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.53500000000000003,"15min":0.502},"min":{"1min":0.35999999999999999,"5min":0.34499999999999997,"15min":0.28199999999999997},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.38100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2350907392,"data_stored":2350907392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2335113216,"data_stored":2335113216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3013672960,"data_stored":3013672960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2321874944,"data_stored":2321874944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1775370240,"data_stored":1775370240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2838167552,"data_stored":2838167552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1776615424,"data_stored":1776615424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:12:09.121 INFO:tasks.thrashosds.thrasher:Setting 4.1e to [7, 1, 5, 0, 3, 6] 2024-02-05T08:12:09.121 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.1e', '7', '1', '5', '0', '3', '6'] 2024-02-05T08:12:09.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pg-upmap-items 4.1e 7 1 5 0 3 6 2024-02-05T08:12:09.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:09.223 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:09.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:09.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:09.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:09.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:09.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:09.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:09.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:09.466 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:12:09.467445+0000 min lat: 0.00115834 max lat: 0.147925 avg lat: 0.0108557 2024-02-05T08:12:09.467 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:12:09.467 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 60 16 88435 88419 92.0707 67.75 0.0124018 0.0108557 2024-02-05T08:12:09.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:09.496 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:09.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:09.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:09.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:09.636 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:09.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:09.737 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:09.782 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:09.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:09.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:09.883 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:09.928 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:09.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:10.031 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:10.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:10.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:10.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:10.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:10.215 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:10.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:10.284 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:10.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:10.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:10.371 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:10.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:10.408 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:12:10.408 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:12:10.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:12:10.408 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:12:10.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.409 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:19178 7.1 7:843e4ad0:::benchmark_data_smithi134_83016_object19177:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:12:10.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:06:58.646746+0000", 2024-02-05T08:12:10.409 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 311.75869992600002, 2024-02-05T08:12:10.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.137073784, 2024-02-05T08:12:10.409 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.409 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:12:10.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:12:10.409 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 19178 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646746+0000", 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646746+0000", 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646747+0000", 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.415e-06 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646769+0000", 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2388999999999999e-05 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646783+0000", 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3696000000000001e-05 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646791+0000", 2024-02-05T08:12:10.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1300000000000001e-06 2024-02-05T08:12:10.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646834+0000", 2024-02-05T08:12:10.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3006999999999997e-05 2024-02-05T08:12:10.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646936+0000", 2024-02-05T08:12:10.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101811 2024-02-05T08:12:10.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.647049+0000", 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113516 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.648395+0000", 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001345572 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.648785+0000", 2024-02-05T08:12:10.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00039036099999999998 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.783734+0000", 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13494893399999999 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.783780+0000", 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5284000000000002e-05 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.783819+0000", 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9669000000000003e-05 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:20343 7.1 7:8ad7240f:::benchmark_data_smithi134_83016_object20342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:06:59.425830+0000", 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 310.97961561, 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.089606505000000003, 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:12:10.415 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 20343 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.425830+0000", 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.425830+0000", 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.425831+0000", 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1200000000000002e-07 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.425885+0000", 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3940999999999997e-05 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.425895+0000", 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0356e-05 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.425900+0000", 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.378e-06 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.425946+0000", 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5274000000000001e-05 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.426020+0000", 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4834999999999998e-05 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.426120+0000", 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9443000000000003e-05 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.426769+0000", 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00064914599999999997 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.428451+0000", 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001682062 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.515263+0000", 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.086812332000000006 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.515382+0000", 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000118187 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.515436+0000", 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4738999999999999e-05 2024-02-05T08:12:10.420 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.421 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.421 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.421 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:20690 7.1 7:85067d19:::benchmark_data_smithi134_83016_object20689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:12:10.421 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:06:59.674363+0000", 2024-02-05T08:12:10.421 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 310.731082141, 2024-02-05T08:12:10.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14902151899999999, 2024-02-05T08:12:10.421 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.421 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.421 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.421 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:12:10.421 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:12:10.422 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 20690 2024-02-05T08:12:10.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.422 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674363+0000", 2024-02-05T08:12:10.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674363+0000", 2024-02-05T08:12:10.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674365+0000", 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.748e-06 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674395+0000", 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9620000000000001e-05 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674405+0000", 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0220999999999999e-05 2024-02-05T08:12:10.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674412+0000", 2024-02-05T08:12:10.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5739999999999998e-06 2024-02-05T08:12:10.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674509+0000", 2024-02-05T08:12:10.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6210000000000002e-05 2024-02-05T08:12:10.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674634+0000", 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000125241 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674750+0000", 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116585 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.675575+0000", 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000824236 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.676042+0000", 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000466995 2024-02-05T08:12:10.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.823321+0000", 2024-02-05T08:12:10.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14727914 2024-02-05T08:12:10.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.823361+0000", 2024-02-05T08:12:10.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0445999999999999e-05 2024-02-05T08:12:10.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.823385+0000", 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3502999999999998e-05 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:99792 7.1 7:8fc2463e:::benchmark_data_smithi134_83016_object99791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:08.947855+0000", 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 241.45759046399999, 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13887300799999999, 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.427 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 99792 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.947855+0000", 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.947855+0000", 2024-02-05T08:12:10.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.947856+0000", 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.091e-06 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.947913+0000", 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7017e-05 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.947925+0000", 2024-02-05T08:12:10.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1654999999999999e-05 2024-02-05T08:12:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.947931+0000", 2024-02-05T08:12:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3770000000000001e-06 2024-02-05T08:12:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.948005+0000", 2024-02-05T08:12:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4165999999999994e-05 2024-02-05T08:12:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.948103+0000", 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8139000000000003e-05 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.948203+0000", 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9871000000000007e-05 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.950138+0000", 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019349269999999999 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:09.077895+0000", 2024-02-05T08:12:10.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12775708699999999 2024-02-05T08:12:10.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:09.086623+0000", 2024-02-05T08:12:10.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087272589999999994 2024-02-05T08:12:10.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:09.086687+0000", 2024-02-05T08:12:10.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4357999999999996e-05 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:09.086728+0000", 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1060999999999999e-05 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:106040 7.1 7:803076d3:::benchmark_data_smithi134_83016_object106039:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:15.632160+0000", 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 234.773285362, 2024-02-05T08:12:10.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12624469199999999, 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 106040 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.632160+0000", 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.632160+0000", 2024-02-05T08:12:10.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.632161+0000", 2024-02-05T08:12:10.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.313e-06 2024-02-05T08:12:10.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.632208+0000", 2024-02-05T08:12:10.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6804000000000001e-05 2024-02-05T08:12:10.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.632221+0000", 2024-02-05T08:12:10.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3071000000000001e-05 2024-02-05T08:12:10.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.632229+0000", 2024-02-05T08:12:10.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7409999999999992e-06 2024-02-05T08:12:10.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.632277+0000", 2024-02-05T08:12:10.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8488000000000001e-05 2024-02-05T08:12:10.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.632425+0000", 2024-02-05T08:12:10.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014778600000000001 2024-02-05T08:12:10.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:12:10.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.632584+0000", 2024-02-05T08:12:10.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015841999999999999 2024-02-05T08:12:10.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.634538+0000", 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019540460000000001 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.642864+0000", 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0083259170000000004 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.758339+0000", 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11547571399999999 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.758378+0000", 2024-02-05T08:12:10.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8312000000000002e-05 2024-02-05T08:12:10.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.758405+0000", 2024-02-05T08:12:10.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7080000000000002e-05 2024-02-05T08:12:10.440 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.440 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.440 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.440 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:106044 7.1 7:8d145210:::benchmark_data_smithi134_83016_object106043:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:12:10.441 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:15.634815+0000", 2024-02-05T08:12:10.441 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 234.77063003500001, 2024-02-05T08:12:10.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12368595, 2024-02-05T08:12:10.441 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.441 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.441 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.441 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:12:10.441 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:12:10.441 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 106044 2024-02-05T08:12:10.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.441 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.634815+0000", 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.634815+0000", 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.634816+0000", 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1699999999999997e-07 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.634873+0000", 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7148000000000002e-05 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.634884+0000", 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0413e-05 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.634890+0000", 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9549999999999997e-06 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.634925+0000", 2024-02-05T08:12:10.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5098999999999997e-05 2024-02-05T08:12:10.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.635010+0000", 2024-02-05T08:12:10.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4683000000000002e-05 2024-02-05T08:12:10.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:12:10.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.635277+0000", 2024-02-05T08:12:10.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026762200000000002 2024-02-05T08:12:10.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.637160+0000", 2024-02-05T08:12:10.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001882966 2024-02-05T08:12:10.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.642898+0000", 2024-02-05T08:12:10.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057377599999999997 2024-02-05T08:12:10.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.758440+0000", 2024-02-05T08:12:10.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11554234200000001 2024-02-05T08:12:10.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.758478+0000", 2024-02-05T08:12:10.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7596999999999999e-05 2024-02-05T08:12:10.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.758501+0000", 2024-02-05T08:12:10.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3547999999999999e-05 2024-02-05T08:12:10.447 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.447 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.447 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.447 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:106223 7.1 7:8f829678:::benchmark_data_smithi134_83016_object106222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:12:10.447 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:15.917091+0000", 2024-02-05T08:12:10.447 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 234.48835410199999, 2024-02-05T08:12:10.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.116607476, 2024-02-05T08:12:10.447 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.447 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.447 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.448 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:12:10.448 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:12:10.448 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 106223 2024-02-05T08:12:10.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.448 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.917091+0000", 2024-02-05T08:12:10.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.917091+0000", 2024-02-05T08:12:10.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.917093+0000", 2024-02-05T08:12:10.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.812e-06 2024-02-05T08:12:10.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.917131+0000", 2024-02-05T08:12:10.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7725000000000002e-05 2024-02-05T08:12:10.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.917143+0000", 2024-02-05T08:12:10.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2078e-05 2024-02-05T08:12:10.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.917151+0000", 2024-02-05T08:12:10.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3510000000000002e-06 2024-02-05T08:12:10.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.917230+0000", 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8608000000000004e-05 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.917368+0000", 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000138011 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.917499+0000", 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130875 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.920020+0000", 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025213369999999998 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:15.921202+0000", 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001181887 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.033626+0000", 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11242371499999999 2024-02-05T08:12:10.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.033662+0000", 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6259e-05 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.033699+0000", 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6817999999999999e-05 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:106464 7.1 7:89c15f15:::benchmark_data_smithi134_83016_object106463:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:16.273083+0000", 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 234.13236277999999, 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13271049600000001, 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 106464 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.273083+0000", 2024-02-05T08:12:10.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.273083+0000", 2024-02-05T08:12:10.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.273084+0000", 2024-02-05T08:12:10.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.252e-06 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.273119+0000", 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5352000000000003e-05 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.273131+0000", 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1965e-05 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.273137+0000", 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8839999999999999e-06 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.273195+0000", 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8254000000000002e-05 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.273313+0000", 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117291 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:12:10.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.273412+0000", 2024-02-05T08:12:10.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9223999999999997e-05 2024-02-05T08:12:10.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.275452+0000", 2024-02-05T08:12:10.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020397229999999998 2024-02-05T08:12:10.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.281113+0000", 2024-02-05T08:12:10.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0056609379999999999 2024-02-05T08:12:10.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.405723+0000", 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.124610308 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.405765+0000", 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1980999999999997e-05 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.405793+0000", 2024-02-05T08:12:10.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8323999999999999e-05 2024-02-05T08:12:10.460 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.460 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.460 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.460 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:106467 7.1 7:86287060:::benchmark_data_smithi134_83016_object106466:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:12:10.460 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:16.278286+0000", 2024-02-05T08:12:10.460 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 234.12715893500001, 2024-02-05T08:12:10.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13027485499999999, 2024-02-05T08:12:10.460 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.460 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.460 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 106467 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.278286+0000", 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.278286+0000", 2024-02-05T08:12:10.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.278288+0000", 2024-02-05T08:12:10.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5290000000000001e-06 2024-02-05T08:12:10.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.278334+0000", 2024-02-05T08:12:10.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5682000000000001e-05 2024-02-05T08:12:10.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.278348+0000", 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4066000000000001e-05 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.278355+0000", 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6079999999999998e-06 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.278396+0000", 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1014999999999999e-05 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.278504+0000", 2024-02-05T08:12:10.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010809199999999999 2024-02-05T08:12:10.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:12:10.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.278617+0000", 2024-02-05T08:12:10.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112546 2024-02-05T08:12:10.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.280492+0000", 2024-02-05T08:12:10.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001874695 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.281380+0000", 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00088822900000000004 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.408499+0000", 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12711889600000001 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.408533+0000", 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4408e-05 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:16.408561+0000", 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8089000000000002e-05 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:166246 7.1 7:8f84926c:::benchmark_data_smithi134_83016_object56312:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:22.632559+0000", 2024-02-05T08:12:10.466 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 167.77288640099999, 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11209184, 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 166246 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.632559+0000", 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.632559+0000", 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.632560+0000", 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3289999999999999e-06 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.632564+0000", 2024-02-05T08:12:10.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8909999999999997e-06 2024-02-05T08:12:10.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.632574+0000", 2024-02-05T08:12:10.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5170000000000005e-06 2024-02-05T08:12:10.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.632579+0000", 2024-02-05T08:12:10.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3600000000000004e-06 2024-02-05T08:12:10.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.632622+0000", 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2688000000000002e-05 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.632711+0000", 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9315999999999997e-05 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.632792+0000", 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1100000000000006e-05 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.636210+0000", 2024-02-05T08:12:10.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034181110000000002 2024-02-05T08:12:10.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.640724+0000", 2024-02-05T08:12:10.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045136300000000002 2024-02-05T08:12:10.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.744534+0000", 2024-02-05T08:12:10.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.103810339 2024-02-05T08:12:10.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.744610+0000", 2024-02-05T08:12:10.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5753999999999993e-05 2024-02-05T08:12:10.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:22.744651+0000", 2024-02-05T08:12:10.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0805e-05 2024-02-05T08:12:10.472 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.472 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.473 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.473 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:167118 7.1 7:8db30ba2:::benchmark_data_smithi134_83016_object57184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T08:12:10.473 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:23.653867+0000", 2024-02-05T08:12:10.473 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 166.75157829, 2024-02-05T08:12:10.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.124243702, 2024-02-05T08:12:10.473 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.473 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.473 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.473 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:12:10.473 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:12:10.473 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 167118 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.653867+0000", 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.653867+0000", 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.653869+0000", 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.035e-06 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.653876+0000", 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5250000000000002e-06 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.653896+0000", 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0097000000000001e-05 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.653906+0000", 2024-02-05T08:12:10.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0213e-05 2024-02-05T08:12:10.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.653968+0000", 2024-02-05T08:12:10.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2008000000000007e-05 2024-02-05T08:12:10.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.654118+0000", 2024-02-05T08:12:10.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000150245 2024-02-05T08:12:10.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:12:10.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.654237+0000", 2024-02-05T08:12:10.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000118816 2024-02-05T08:12:10.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.656263+0000", 2024-02-05T08:12:10.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002025967 2024-02-05T08:12:10.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.660515+0000", 2024-02-05T08:12:10.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042517869999999999 2024-02-05T08:12:10.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.777988+0000", 2024-02-05T08:12:10.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.117473564 2024-02-05T08:12:10.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.778052+0000", 2024-02-05T08:12:10.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3522999999999995e-05 2024-02-05T08:12:10.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.778111+0000", 2024-02-05T08:12:10.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8922000000000003e-05 2024-02-05T08:12:10.479 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.479 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.479 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.479 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:167124 7.1 7:8ac20bd7:::benchmark_data_smithi134_83016_object57190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-02-05T08:12:10.479 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:09:23.658346+0000", 2024-02-05T08:12:10.479 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 166.74709965700001, 2024-02-05T08:12:10.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.119936578, 2024-02-05T08:12:10.480 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.480 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:12:10.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:12:10.480 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 167124 2024-02-05T08:12:10.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.480 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.658346+0000", 2024-02-05T08:12:10.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.658346+0000", 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.658347+0000", 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.553e-06 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.658353+0000", 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4199999999999998e-06 2024-02-05T08:12:10.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.658365+0000", 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1847e-05 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.658371+0000", 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7089999999999996e-06 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.658416+0000", 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4533000000000002e-05 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.658519+0000", 2024-02-05T08:12:10.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010289 2024-02-05T08:12:10.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:12:10.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.658607+0000", 2024-02-05T08:12:10.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7879000000000003e-05 2024-02-05T08:12:10.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.660650+0000", 2024-02-05T08:12:10.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020432940000000002 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.661114+0000", 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00046385099999999999 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.778178+0000", 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.117063906 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.778236+0000", 2024-02-05T08:12:10.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8505000000000003e-05 2024-02-05T08:12:10.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:09:23.778282+0000", 2024-02-05T08:12:10.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6190999999999999e-05 2024-02-05T08:12:10.485 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.485 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.485 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.485 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:23476 8.2 8:4131261d:::benchmark_data_smithi134_115836_object23475:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T08:12:10.485 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:11:18.832157+0000", 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 51.573287979, 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.096639155000000004, 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 23476 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832157+0000", 2024-02-05T08:12:10.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832157+0000", 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832160+0000", 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.114e-06 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832205+0000", 2024-02-05T08:12:10.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5816000000000003e-05 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832230+0000", 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4586000000000001e-05 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832245+0000", 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4973e-05 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832347+0000", 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010175699999999999 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832425+0000", 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8545000000000001e-05 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832504+0000", 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9060999999999994e-05 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.833875+0000", 2024-02-05T08:12:10.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001370312 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.835534+0000", 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0016593619999999999 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.928696+0000", 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.093162166000000005 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.928753+0000", 2024-02-05T08:12:10.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6919999999999997e-05 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.928797+0000", 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3543e-05 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:23481 8.2 8:4a26eba5:::benchmark_data_smithi134_115836_object23480:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:11:18.832821+0000", 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 51.572624838999999, 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.096786289999999997, 2024-02-05T08:12:10.491 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.492 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:10.493 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 61 16 89499 89483 91.6515 66.5 0.0343202 0.0109054 2024-02-05T08:12:10.494 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.494 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.494 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:10.494 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:10.494 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 23481 2024-02-05T08:12:10.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.494 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832821+0000", 2024-02-05T08:12:10.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832821+0000", 2024-02-05T08:12:10.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832821+0000", 2024-02-05T08:12:10.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5499999999999998e-07 2024-02-05T08:12:10.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832905+0000", 2024-02-05T08:12:10.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3912999999999994e-05 2024-02-05T08:12:10.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832917+0000", 2024-02-05T08:12:10.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1953000000000001e-05 2024-02-05T08:12:10.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.496 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.496 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.496 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.832926+0000", 2024-02-05T08:12:10.496 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7229999999999993e-06 2024-02-05T08:12:10.496 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.833211+0000", 2024-02-05T08:12:10.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028567499999999999 2024-02-05T08:12:10.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.497 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.833289+0000", 2024-02-05T08:12:10.497 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7856999999999997e-05 2024-02-05T08:12:10.497 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.497 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.497 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.833377+0000", 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7653999999999998e-05 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.834009+0000", 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00063161700000000001 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.840813+0000", 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068047899999999998 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.498 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.929521+0000", 2024-02-05T08:12:10.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.088707217000000005 2024-02-05T08:12:10.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.929571+0000", 2024-02-05T08:12:10.499 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9979999999999999e-05 2024-02-05T08:12:10.499 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.499 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.499 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.499 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.929607+0000", 2024-02-05T08:12:10.500 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6356000000000002e-05 2024-02-05T08:12:10.500 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.500 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.500 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.500 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.500 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.500 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:23486 8.2 8:4f1c75c8:::benchmark_data_smithi134_115836_object23485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T08:12:10.500 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:11:18.836240+0000", 2024-02-05T08:12:10.500 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 51.569205732, 2024-02-05T08:12:10.500 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.093507357999999999, 2024-02-05T08:12:10.500 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.500 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.500 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.501 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:10.501 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:10.501 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 23486 2024-02-05T08:12:10.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.501 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.501 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.836240+0000", 2024-02-05T08:12:10.501 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.501 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.501 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.501 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.501 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.836240+0000", 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.836241+0000", 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7969999999999999e-06 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.836298+0000", 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6017999999999998e-05 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.502 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.836307+0000", 2024-02-05T08:12:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0589999999999992e-06 2024-02-05T08:12:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.836312+0000", 2024-02-05T08:12:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1200000000000001e-06 2024-02-05T08:12:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.836385+0000", 2024-02-05T08:12:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2848000000000005e-05 2024-02-05T08:12:10.503 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.836455+0000", 2024-02-05T08:12:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9987999999999995e-05 2024-02-05T08:12:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:12:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.836533+0000", 2024-02-05T08:12:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8831999999999993e-05 2024-02-05T08:12:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.504 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.837162+0000", 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00062816599999999999 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.840896+0000", 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003734177 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.929658+0000", 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.088762530000000006 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.505 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.929715+0000", 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6406999999999997e-05 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:18.929747+0000", 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2416000000000003e-05 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:23691 8.5 8:a5ec77e3:::benchmark_data_smithi134_115836_object23690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:11:19.016593+0000", 2024-02-05T08:12:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 51.388851977999998, 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.124456833, 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 23691 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.016593+0000", 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.507 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.016593+0000", 2024-02-05T08:12:10.508 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.016602+0000", 2024-02-05T08:12:10.508 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0490000000000004e-06 2024-02-05T08:12:10.508 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.508 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.508 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.508 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.016662+0000", 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0837e-05 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.016677+0000", 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5124e-05 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.016686+0000", 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1580000000000005e-06 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.509 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.016755+0000", 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9838000000000005e-05 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.016895+0000", 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013969500000000001 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.017028+0000", 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013249 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.510 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.018093+0000", 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010653380000000001 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.019291+0000", 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011977229999999999 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.140935+0000", 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12164391400000001 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.511 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.140999+0000", 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4579999999999995e-05 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.141050+0000", 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1087000000000001e-05 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.512 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:24234 8.5 8:a5eac373:::benchmark_data_smithi134_115836_object24233:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:11:19.392580+0000", 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 51.012865224999999, 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.094569940000000005, 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 24234 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.513 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392580+0000", 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392580+0000", 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392584+0000", 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8469999999999998e-06 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.514 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392633+0000", 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8912999999999997e-05 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392646+0000", 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3217999999999999e-05 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392653+0000", 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4189999999999999e-06 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.515 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.516 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.516 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392699+0000", 2024-02-05T08:12:10.516 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6615e-05 2024-02-05T08:12:10.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.516 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.516 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.516 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392790+0000", 2024-02-05T08:12:10.516 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0482999999999994e-05 2024-02-05T08:12:10.516 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.516 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.516 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:12:10.516 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392888+0000", 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8349000000000003e-05 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.394874+0000", 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00198547 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.394902+0000", 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8975999999999998e-05 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.517 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.487041+0000", 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.092138154999999999 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.487105+0000", 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4524999999999996e-05 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.487150+0000", 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4969999999999998e-05 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.518 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.519 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.519 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.519 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.519 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:24235 8.5 8:a7939b68:::benchmark_data_smithi134_115836_object24234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T08:12:10.519 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:11:19.392668+0000", 2024-02-05T08:12:10.519 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 51.012777006999997, 2024-02-05T08:12:10.519 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.094618357, 2024-02-05T08:12:10.519 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.519 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.519 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.519 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:10.519 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:10.519 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 24235 2024-02-05T08:12:10.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.520 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.520 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.520 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.520 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392668+0000", 2024-02-05T08:12:10.520 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.520 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.520 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.520 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392668+0000", 2024-02-05T08:12:10.520 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.520 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392669+0000", 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0999999999999999e-07 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392694+0000", 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.535e-05 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392699+0000", 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.147e-06 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.521 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.392704+0000", 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3290000000000004e-06 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.393824+0000", 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001120172 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.393928+0000", 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010383299999999999 2024-02-05T08:12:10.522 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.394051+0000", 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012322099999999999 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.394995+0000", 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00094419700000000003 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.395239+0000", 2024-02-05T08:12:10.523 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00024356000000000001 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.487201+0000", 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.091962074000000005 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.487260+0000", 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9058000000000003e-05 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.524 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:19.487287+0000", 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6906e-05 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:25746 8.2 8:4ade417d:::benchmark_data_smithi134_115836_object25745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:11:20.413491+0000", 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 49.991954671000002, 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14147770600000001, 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.525 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 25746 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413491+0000", 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413491+0000", 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.526 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413491+0000", 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0299999999999998e-07 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413556+0000", 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4455000000000005e-05 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413567+0000", 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0749e-05 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.527 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413575+0000", 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9710000000000004e-06 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413621+0000", 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6288000000000001e-05 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413721+0000", 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100373 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.528 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413813+0000", 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1324999999999997e-05 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414683+0000", 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00087058100000000005 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414951+0000", 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026772299999999998 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.529 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.554907+0000", 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13995589999999999 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.554944+0000", 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7314000000000002e-05 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.554968+0000", 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4324e-05 2024-02-05T08:12:10.530 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:25750 8.5 8:a65889c9:::benchmark_data_smithi134_115836_object25749:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:11:20.414049+0000", 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 49.991396223999999, 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13532624500000001, 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:10.531 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 25750 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414049+0000", 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414049+0000", 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.532 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414050+0000", 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0400000000000005e-07 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414098+0000", 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8439999999999997e-05 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414111+0000", 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.22e-05 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.533 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:10.534 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414118+0000", 2024-02-05T08:12:10.534 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0829999999999998e-06 2024-02-05T08:12:10.534 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.534 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.534 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:10.534 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414159+0000", 2024-02-05T08:12:10.534 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1511999999999998e-05 2024-02-05T08:12:10.534 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.534 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.534 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:10.534 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414251+0000", 2024-02-05T08:12:10.534 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1366999999999994e-05 2024-02-05T08:12:10.534 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414346+0000", 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5629000000000002e-05 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.415357+0000", 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010108809999999999 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.415535+0000", 2024-02-05T08:12:10.535 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000178004 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.549281+0000", 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13374603199999999 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.549336+0000", 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5210999999999997e-05 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.549375+0000", 2024-02-05T08:12:10.536 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9082000000000003e-05 2024-02-05T08:12:10.537 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.537 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.537 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.537 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:10.537 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:10.537 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:12:10.541 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:10.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:10.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:10.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:10.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:10.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:10.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:10.773 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:10.792 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:12:10.815 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:10.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:10.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:10.959 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:10.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:10.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:10 smithi134 ceph-mon[30086]: pgmap v303: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-05T08:12:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:10 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/315524137' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:12:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:10 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:12:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:10 smithi134 ceph-mon[39278]: pgmap v303: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-05T08:12:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:10 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/315524137' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:12:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:10 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:12:11.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:10 smithi196 ceph-mon[33018]: pgmap v303: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-05T08:12:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:10 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/315524137' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:12:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:10 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:12:11.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:11.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:11.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:11.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:11.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:11.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:11.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:11.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:11.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:11.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:11.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:11.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T08:12:11.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:11.466 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 62 16 90522 90506 91.2045 63.9375 0.0575355 0.0109576 2024-02-05T08:12:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:11.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:11.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:11.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:11.632 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:11.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:11.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:11.734 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:11.737 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:12:11.772 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:11.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:11.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:11.903 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:11.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:11.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:11.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:12:12.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:12.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:11 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:12:12.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:11 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:12:12.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:11 smithi134 ceph-mon[30086]: osdmap e179: 8 total, 8 up, 7 in 2024-02-05T08:12:12.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:11 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:12:12.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:11 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:12:12.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:11 smithi134 ceph-mon[39278]: osdmap e179: 8 total, 8 up, 7 in 2024-02-05T08:12:12.032 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:11 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:12:12.032 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:11 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:12:12.032 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:11 smithi196 ceph-mon[33018]: osdmap e179: 8 total, 8 up, 7 in 2024-02-05T08:12:12.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:12.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:12.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:12.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:12.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:12.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:12.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:12.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:12.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:12.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:12.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:12.422 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:12.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:12.466 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 63 16 91602 91586 90.8282 67.5 0.00658411 0.0110054 2024-02-05T08:12:12.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:12.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:12.569 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:12.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:12.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:12.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:12.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:12.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:12.842 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:12.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:12.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:12.885 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:12 smithi196 ceph-mon[33018]: pgmap v305: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T08:12:12.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:12.988 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:12.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:13.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:13.122 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:13.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:13.134 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:12 smithi134 ceph-mon[30086]: pgmap v305: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T08:12:13.135 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:12 smithi134 ceph-mon[39278]: pgmap v305: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T08:12:13.223 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:13.260 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:13.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:13.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:13.361 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:13.405 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:13.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:13.466 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 64 16 92696 92680 90.4774 68.375 0.0126887 0.011048 2024-02-05T08:12:13.507 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:13.544 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:13.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:13.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:13.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:12:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:12:13.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:13.673 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:13.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:13.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:13.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:13.814 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:13.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:13.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:13.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:13.947 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:13.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:13.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[30086]: 3.1c scrub starts 2024-02-05T08:12:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[30086]: 3.1c scrub ok 2024-02-05T08:12:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:12:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:12:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[30086]: osdmap e180: 8 total, 8 up, 7 in 2024-02-05T08:12:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[30086]: 5.12 scrub starts 2024-02-05T08:12:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[30086]: 5.12 scrub ok 2024-02-05T08:12:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:12:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:12:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[39278]: 3.1c scrub starts 2024-02-05T08:12:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[39278]: 3.1c scrub ok 2024-02-05T08:12:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:12:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:12:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[39278]: osdmap e180: 8 total, 8 up, 7 in 2024-02-05T08:12:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[39278]: 5.12 scrub starts 2024-02-05T08:12:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[39278]: 5.12 scrub ok 2024-02-05T08:12:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:12:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:13 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:12:14.049 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:14.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:13 smithi196 ceph-mon[33018]: 3.1c scrub starts 2024-02-05T08:12:14.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:13 smithi196 ceph-mon[33018]: 3.1c scrub ok 2024-02-05T08:12:14.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:13 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:12:14.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:13 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:12:14.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:13 smithi196 ceph-mon[33018]: osdmap e180: 8 total, 8 up, 7 in 2024-02-05T08:12:14.061 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:13 smithi196 ceph-mon[33018]: 5.12 scrub starts 2024-02-05T08:12:14.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:13 smithi196 ceph-mon[33018]: 5.12 scrub ok 2024-02-05T08:12:14.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:13 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:14.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:13 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:12:14.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:13 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:12:14.062 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:12:14.062 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:12:14.062 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:93284 8.d 8:b8d3085c:::benchmark_data_smithi134_115836_object93283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:14.050298+0000", 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.011101175, 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011119031, 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 93284 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.063 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:14.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.050298+0000", 2024-02-05T08:12:14.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:14.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:14.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.050298+0000", 2024-02-05T08:12:14.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:14.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.064 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.064 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:14.064 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.050299+0000", 2024-02-05T08:12:14.064 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6099999999999999e-07 2024-02-05T08:12:14.064 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.050349+0000", 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0277e-05 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.050358+0000", 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4210000000000001e-06 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.050364+0000", 2024-02-05T08:12:14.065 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2190000000000003e-06 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.050413+0000", 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9836000000000002e-05 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.050548+0000", 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013448900000000001 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,7", 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.050705+0000", 2024-02-05T08:12:14.066 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015672700000000001 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.053121+0000", 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024167220000000001 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.053942+0000", 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00082107999999999997 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:14.067 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.061315+0000", 2024-02-05T08:12:14.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073729700000000004 2024-02-05T08:12:14.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:14.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.061378+0000", 2024-02-05T08:12:14.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2748000000000003e-05 2024-02-05T08:12:14.068 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.068 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.068 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:14.068 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.061417+0000", 2024-02-05T08:12:14.068 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8581000000000002e-05 2024-02-05T08:12:14.068 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:14.068 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:93293 8.9 8:99cc8064:::benchmark_data_smithi134_115836_object93292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e181)", 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:14.060647+0000", 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.00075178500000000004, 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00087376200000000004, 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 93293 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.069 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.060647+0000", 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.060647+0000", 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.060648+0000", 2024-02-05T08:12:14.070 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0270000000000001e-06 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.060703+0000", 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5408000000000002e-05 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.060713+0000", 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0240000000000003e-06 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:14.071 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.060718+0000", 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3709999999999999e-06 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.060776+0000", 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7707000000000002e-05 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.060886+0000", 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110226 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:12:14.072 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.060980+0000", 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3938999999999996e-05 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.15087.0:93291 8.3 e181/157 8:c3b264c1:::benchmark_data_smithi134_115836_object93290:head v 181'5870, mlcod=181'5867)", 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:14.061283+0000", 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.000116221, 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028122500000000002, 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.073 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.061283+0000", 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.061283+0000", 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.061283+0000", 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1100000000000002e-07 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.061354+0000", 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1202e-05 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.061362+0000", 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4290000000000004e-06 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.061368+0000", 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0140000000000004e-06 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.061412+0000", 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4762000000000003e-05 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:14.061430+0000", 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8068999999999999e-05 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-02-05T08:12:14.076 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 3 2024-02-05T08:12:14.077 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:12:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:14.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:14.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:14.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:14.229 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:14.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:14.304 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:14.331 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:14.386 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:14.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:14.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:14.467 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 65 16 93716 93700 90.0662 63.75 0.0100294 0.0110988 2024-02-05T08:12:14.493 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:14.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:14.535 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:14.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:14.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:14.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:14.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:14.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:14.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:14.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T08:12:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:14.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:14.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:14.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:14.946 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:14.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:15.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:15.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:14 smithi196 ceph-mon[33018]: 2.2 scrub starts 2024-02-05T08:12:15.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:14 smithi196 ceph-mon[33018]: 2.2 scrub ok 2024-02-05T08:12:15.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:14 smithi196 ceph-mon[33018]: pgmap v307: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T08:12:15.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:14 smithi196 ceph-mon[33018]: osdmap e181: 8 total, 8 up, 7 in 2024-02-05T08:12:15.089 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:15.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:15.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:15.135 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:12:15.156 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:14 smithi134 ceph-mon[30086]: 2.2 scrub starts 2024-02-05T08:12:15.156 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:14 smithi134 ceph-mon[30086]: 2.2 scrub ok 2024-02-05T08:12:15.156 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:14 smithi134 ceph-mon[30086]: pgmap v307: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T08:12:15.156 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:14 smithi134 ceph-mon[30086]: osdmap e181: 8 total, 8 up, 7 in 2024-02-05T08:12:15.157 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:14 smithi134 ceph-mon[39278]: 2.2 scrub starts 2024-02-05T08:12:15.157 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:14 smithi134 ceph-mon[39278]: 2.2 scrub ok 2024-02-05T08:12:15.157 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:14 smithi134 ceph-mon[39278]: pgmap v307: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T08:12:15.157 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:14 smithi134 ceph-mon[39278]: osdmap e181: 8 total, 8 up, 7 in 2024-02-05T08:12:15.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:15.238 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:15.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:15.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:15.340 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:15.380 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:15.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:15.467 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 66 16 94707 94691 89.64 61.9375 0.025232 0.0111507 2024-02-05T08:12:15.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:15.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:15.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:15.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:15.643 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:15.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:15.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:15.691 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:12:15.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:15.777 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:15.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:15.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:15.911 INFO:teuthology.orchestra.run.smithi134.stderr:set 4.1e pg_upmap_items mapping to [7->1,5->0,3->6] 2024-02-05T08:12:15.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:15.942 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:15.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:16.043 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:16.108 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:16.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:16.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:15 smithi134 ceph-mon[30086]: 6.19 scrub starts 2024-02-05T08:12:16.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:15 smithi134 ceph-mon[30086]: 6.19 scrub ok 2024-02-05T08:12:16.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:15 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2981972581' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1e", "id": [7, 1, 5, 0, 3, 6]}]: dispatch 2024-02-05T08:12:16.176 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:15 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1e", "id": [7, 1, 5, 0, 3, 6]}]: dispatch 2024-02-05T08:12:16.177 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:15 smithi134 ceph-mon[39278]: 6.19 scrub starts 2024-02-05T08:12:16.177 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:15 smithi134 ceph-mon[39278]: 6.19 scrub ok 2024-02-05T08:12:16.177 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:15 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2981972581' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1e", "id": [7, 1, 5, 0, 3, 6]}]: dispatch 2024-02-05T08:12:16.177 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:15 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1e", "id": [7, 1, 5, 0, 3, 6]}]: dispatch 2024-02-05T08:12:16.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:16.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:16.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:16.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:15 smithi196 ceph-mon[33018]: 6.19 scrub starts 2024-02-05T08:12:16.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:15 smithi196 ceph-mon[33018]: 6.19 scrub ok 2024-02-05T08:12:16.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:15 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2981972581' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1e", "id": [7, 1, 5, 0, 3, 6]}]: dispatch 2024-02-05T08:12:16.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:15 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1e", "id": [7, 1, 5, 0, 3, 6]}]: dispatch 2024-02-05T08:12:16.341 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:16.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:16.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:16.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:16.467 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 67 16 95759 95743 89.2834 65.75 0.00563963 0.0111959 2024-02-05T08:12:16.475 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:16.511 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:16.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:16.613 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:16.651 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:16.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:16.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:16.753 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:16.788 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:16.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:16.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:16.930 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:16.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:17.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:17.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:16 smithi196 ceph-mon[33018]: 5.1b scrub starts 2024-02-05T08:12:17.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:16 smithi196 ceph-mon[33018]: 5.1b scrub ok 2024-02-05T08:12:17.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:16 smithi196 ceph-mon[33018]: pgmap v309: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 56 MiB/s wr, 889 op/s 2024-02-05T08:12:17.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:16 smithi196 ceph-mon[33018]: 6.17 scrub starts 2024-02-05T08:12:17.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:16 smithi196 ceph-mon[33018]: 6.17 scrub ok 2024-02-05T08:12:17.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:16 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1e", "id": [7, 1, 5, 0, 3, 6]}]': finished 2024-02-05T08:12:17.035 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:16 smithi196 ceph-mon[33018]: osdmap e182: 8 total, 8 up, 7 in 2024-02-05T08:12:17.035 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:12:17.035 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:12:17.035 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T08:12:17.035 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T08:12:17.036 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:12:17.072 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:17.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:17.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:17.204 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:17.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:17.207 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[30086]: 5.1b scrub starts 2024-02-05T08:12:17.207 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[30086]: 5.1b scrub ok 2024-02-05T08:12:17.207 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[30086]: pgmap v309: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 56 MiB/s wr, 889 op/s 2024-02-05T08:12:17.207 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[30086]: 6.17 scrub starts 2024-02-05T08:12:17.207 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[30086]: 6.17 scrub ok 2024-02-05T08:12:17.207 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1e", "id": [7, 1, 5, 0, 3, 6]}]': finished 2024-02-05T08:12:17.207 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[30086]: osdmap e182: 8 total, 8 up, 7 in 2024-02-05T08:12:17.209 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[39278]: 5.1b scrub starts 2024-02-05T08:12:17.209 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[39278]: 5.1b scrub ok 2024-02-05T08:12:17.209 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[39278]: pgmap v309: 148 pgs: 148 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 852 B/s rd, 56 MiB/s wr, 889 op/s 2024-02-05T08:12:17.209 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[39278]: 6.17 scrub starts 2024-02-05T08:12:17.209 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[39278]: 6.17 scrub ok 2024-02-05T08:12:17.209 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1e", "id": [7, 1, 5, 0, 3, 6]}]': finished 2024-02-05T08:12:17.209 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:16 smithi134 ceph-mon[39278]: osdmap e182: 8 total, 8 up, 7 in 2024-02-05T08:12:17.287 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:17.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:17.341 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:17.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:17.342 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:12:17.385 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:12:17.433 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:12:17.442 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:17.467 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 68 16 96826 96810 88.9511 66.6875 0.00701119 0.0112371 2024-02-05T08:12:17.479 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:12:17.487 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:17.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:17.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:17.516 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:12:17.558 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:12:17.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:17.599 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:12:17.618 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:17.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:17.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:17.661 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:12:17.674 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:12:17.674 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:12:17.674 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:12:17.674 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:12:17.674 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:12:17.674 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:12:17.674 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:12:17.674 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:12:17.720 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:17.759 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:17.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:17.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T08:12:17.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:17.893 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:17.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:17.995 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:18.000 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[30086]: 4.6 scrub starts 2024-02-05T08:12:18.001 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[30086]: 3.b scrub starts 2024-02-05T08:12:18.001 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[30086]: 4.6 scrub ok 2024-02-05T08:12:18.001 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[30086]: 3.b scrub ok 2024-02-05T08:12:18.001 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[30086]: 3.1a scrub starts 2024-02-05T08:12:18.001 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[30086]: 3.1a scrub ok 2024-02-05T08:12:18.001 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[30086]: osdmap e183: 8 total, 8 up, 7 in 2024-02-05T08:12:18.001 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/452541971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:18.001 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:12:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:12:18.002 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[39278]: 4.6 scrub starts 2024-02-05T08:12:18.002 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[39278]: 3.b scrub starts 2024-02-05T08:12:18.002 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[39278]: 4.6 scrub ok 2024-02-05T08:12:18.002 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[39278]: 3.b scrub ok 2024-02-05T08:12:18.002 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[39278]: 3.1a scrub starts 2024-02-05T08:12:18.002 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[39278]: 3.1a scrub ok 2024-02-05T08:12:18.002 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[39278]: osdmap e183: 8 total, 8 up, 7 in 2024-02-05T08:12:18.002 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:17 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/452541971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:18.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:18.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:18.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:17 smithi196 ceph-mon[33018]: 4.6 scrub starts 2024-02-05T08:12:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:17 smithi196 ceph-mon[33018]: 3.b scrub starts 2024-02-05T08:12:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:17 smithi196 ceph-mon[33018]: 4.6 scrub ok 2024-02-05T08:12:18.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:17 smithi196 ceph-mon[33018]: 3.b scrub ok 2024-02-05T08:12:18.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:17 smithi196 ceph-mon[33018]: 3.1a scrub starts 2024-02-05T08:12:18.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:17 smithi196 ceph-mon[33018]: 3.1a scrub ok 2024-02-05T08:12:18.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:17 smithi196 ceph-mon[33018]: osdmap e183: 8 total, 8 up, 7 in 2024-02-05T08:12:18.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:17 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/452541971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:18.109 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:12:18.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:18.178 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:18.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:18.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:18.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:18.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:18.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:18.407 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:18.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:18.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:18.467 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 69 16 97895 97879 88.6303 66.8125 0.00857911 0.011278 2024-02-05T08:12:18.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:18.572 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:18.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:18.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:18.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:18.702 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:18.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:18.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:18.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:18.842 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:18.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:18.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:18.971 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:18.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:19.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:18 smithi134 ceph-mon[30086]: 4.11 scrub starts 2024-02-05T08:12:19.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:18 smithi134 ceph-mon[30086]: 4.11 scrub ok 2024-02-05T08:12:19.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:18 smithi134 ceph-mon[30086]: pgmap v312: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T08:12:19.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:18 smithi134 ceph-mon[30086]: 4.d scrub starts 2024-02-05T08:12:19.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:18 smithi134 ceph-mon[30086]: 4.d scrub ok 2024-02-05T08:12:19.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:18 smithi134 ceph-mon[39278]: 4.11 scrub starts 2024-02-05T08:12:19.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:18 smithi134 ceph-mon[39278]: 4.11 scrub ok 2024-02-05T08:12:19.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:18 smithi134 ceph-mon[39278]: pgmap v312: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T08:12:19.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:18 smithi134 ceph-mon[39278]: 4.d scrub starts 2024-02-05T08:12:19.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:18 smithi134 ceph-mon[39278]: 4.d scrub ok 2024-02-05T08:12:19.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:19.110 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:19.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:19.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:19.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:19.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:19.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:19.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:18 smithi196 ceph-mon[33018]: 4.11 scrub starts 2024-02-05T08:12:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:18 smithi196 ceph-mon[33018]: 4.11 scrub ok 2024-02-05T08:12:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:18 smithi196 ceph-mon[33018]: pgmap v312: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T08:12:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:18 smithi196 ceph-mon[33018]: 4.d scrub starts 2024-02-05T08:12:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:18 smithi196 ceph-mon[33018]: 4.d scrub ok 2024-02-05T08:12:19.341 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:19.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:19.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:19.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:19.467 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 70 16 99070 99054 88.4132 73.4375 0.0199334 0.0113059 2024-02-05T08:12:19.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:19.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:19.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:19.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:19.620 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:19.646 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:19.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:19.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:19.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:19.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:19.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:19.912 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:19.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:19.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:19 smithi196 ceph-mon[33018]: 3.c scrub starts 2024-02-05T08:12:19.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:19 smithi196 ceph-mon[33018]: 3.c scrub ok 2024-02-05T08:12:19.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:19 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:12:19.970 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:19 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:12:19.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:19 smithi196 ceph-mon[33018]: 4.1e scrub starts 2024-02-05T08:12:19.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:19 smithi196 ceph-mon[33018]: 4.1e scrub ok 2024-02-05T08:12:19.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:19 smithi196 ceph-mon[33018]: 4.2 scrub starts 2024-02-05T08:12:19.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:19 smithi196 ceph-mon[33018]: 4.2 scrub ok 2024-02-05T08:12:19.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:19 smithi196 ceph-mon[33018]: 3.15 scrub starts 2024-02-05T08:12:19.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:19 smithi196 ceph-mon[33018]: 3.15 scrub ok 2024-02-05T08:12:20.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:20.041 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:12:20.041 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:12:20.054 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:20.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:20.156 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:20.199 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:20.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:20.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[30086]: 3.c scrub starts 2024-02-05T08:12:20.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[30086]: 3.c scrub ok 2024-02-05T08:12:20.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:12:20.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:12:20.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[30086]: 4.1e scrub starts 2024-02-05T08:12:20.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[30086]: 4.1e scrub ok 2024-02-05T08:12:20.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[30086]: 4.2 scrub starts 2024-02-05T08:12:20.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[30086]: 4.2 scrub ok 2024-02-05T08:12:20.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[30086]: 3.15 scrub starts 2024-02-05T08:12:20.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[30086]: 3.15 scrub ok 2024-02-05T08:12:20.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[39278]: 3.c scrub starts 2024-02-05T08:12:20.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[39278]: 3.c scrub ok 2024-02-05T08:12:20.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:12:20.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:12:20.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[39278]: 4.1e scrub starts 2024-02-05T08:12:20.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[39278]: 4.1e scrub ok 2024-02-05T08:12:20.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[39278]: 4.2 scrub starts 2024-02-05T08:12:20.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[39278]: 4.2 scrub ok 2024-02-05T08:12:20.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[39278]: 3.15 scrub starts 2024-02-05T08:12:20.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:19 smithi134 ceph-mon[39278]: 3.15 scrub ok 2024-02-05T08:12:20.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:20.334 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:20.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:20.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:20.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:20.467 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 71 16 100101 100085 88.0751 64.4375 0.0197359 0.0113478 2024-02-05T08:12:20.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:20.473 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:20.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:20.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:20.610 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:20.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:20.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:20.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:20.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:20.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:20.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:20.774 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T08:12:20.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:20.904 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:20.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:21.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:21.047 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:21.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:21.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:20 smithi196 ceph-mon[33018]: 5.d scrub starts 2024-02-05T08:12:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:20 smithi196 ceph-mon[33018]: 5.d scrub ok 2024-02-05T08:12:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:20 smithi196 ceph-mon[33018]: pgmap v313: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 54 MiB/s wr, 866 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:20 smithi196 ceph-mon[33018]: 4.9 scrub starts 2024-02-05T08:12:21.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:20 smithi196 ceph-mon[33018]: 4.9 scrub ok 2024-02-05T08:12:21.129 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:12:21.149 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:21.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:21.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:21.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:20 smithi134 ceph-mon[30086]: 5.d scrub starts 2024-02-05T08:12:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:20 smithi134 ceph-mon[30086]: 5.d scrub ok 2024-02-05T08:12:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:20 smithi134 ceph-mon[30086]: pgmap v313: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 54 MiB/s wr, 866 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:20 smithi134 ceph-mon[30086]: 4.9 scrub starts 2024-02-05T08:12:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:20 smithi134 ceph-mon[30086]: 4.9 scrub ok 2024-02-05T08:12:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:20 smithi134 ceph-mon[39278]: 5.d scrub starts 2024-02-05T08:12:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:20 smithi134 ceph-mon[39278]: 5.d scrub ok 2024-02-05T08:12:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:20 smithi134 ceph-mon[39278]: pgmap v313: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 54 MiB/s wr, 866 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:20 smithi134 ceph-mon[39278]: 4.9 scrub starts 2024-02-05T08:12:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:20 smithi134 ceph-mon[39278]: 4.9 scrub ok 2024-02-05T08:12:21.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:21.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:21.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:21.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:21.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:21.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:21.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:21.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:21.468 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 72 16 101180 101164 87.7884 67.4375 0.0102751 0.0113854 2024-02-05T08:12:21.557 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:21.598 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:21.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:21.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:21.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:21.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:21.742 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:21.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:21.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:21.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:21.874 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:21.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:21.958 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:12:21.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:21.985 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:12:21.985 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:12:21.985 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-02-05T08:12:21.986 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd in 6 2024-02-05T08:12:22.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:22.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:22.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:22.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:22.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:22.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:22.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:22.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:22.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:22.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:22.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[30086]: 5.16 scrub starts 2024-02-05T08:12:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[30086]: 5.16 scrub ok 2024-02-05T08:12:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[30086]: 2.1 scrub starts 2024-02-05T08:12:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[30086]: 2.1 scrub ok 2024-02-05T08:12:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[30086]: 6.11 scrub starts 2024-02-05T08:12:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[30086]: 6.11 scrub ok 2024-02-05T08:12:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[30086]: 6.0 scrub starts 2024-02-05T08:12:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[30086]: 6.0 scrub ok 2024-02-05T08:12:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[30086]: 6.1e scrub starts 2024-02-05T08:12:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[30086]: 6.1e scrub ok 2024-02-05T08:12:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[30086]: 6.8 scrub starts 2024-02-05T08:12:22.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[30086]: 6.8 scrub ok 2024-02-05T08:12:22.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2554419917' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:12:22.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[39278]: 5.16 scrub starts 2024-02-05T08:12:22.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[39278]: 5.16 scrub ok 2024-02-05T08:12:22.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[39278]: 2.1 scrub starts 2024-02-05T08:12:22.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[39278]: 2.1 scrub ok 2024-02-05T08:12:22.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[39278]: 6.11 scrub starts 2024-02-05T08:12:22.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[39278]: 6.11 scrub ok 2024-02-05T08:12:22.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[39278]: 6.0 scrub starts 2024-02-05T08:12:22.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[39278]: 6.0 scrub ok 2024-02-05T08:12:22.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[39278]: 6.1e scrub starts 2024-02-05T08:12:22.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[39278]: 6.1e scrub ok 2024-02-05T08:12:22.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[39278]: 6.8 scrub starts 2024-02-05T08:12:22.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[39278]: 6.8 scrub ok 2024-02-05T08:12:22.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:21 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2554419917' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:12:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:21 smithi196 ceph-mon[33018]: 5.16 scrub starts 2024-02-05T08:12:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:21 smithi196 ceph-mon[33018]: 5.16 scrub ok 2024-02-05T08:12:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:21 smithi196 ceph-mon[33018]: 2.1 scrub starts 2024-02-05T08:12:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:21 smithi196 ceph-mon[33018]: 2.1 scrub ok 2024-02-05T08:12:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:21 smithi196 ceph-mon[33018]: 6.11 scrub starts 2024-02-05T08:12:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:21 smithi196 ceph-mon[33018]: 6.11 scrub ok 2024-02-05T08:12:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:21 smithi196 ceph-mon[33018]: 6.0 scrub starts 2024-02-05T08:12:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:21 smithi196 ceph-mon[33018]: 6.0 scrub ok 2024-02-05T08:12:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:21 smithi196 ceph-mon[33018]: 6.1e scrub starts 2024-02-05T08:12:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:21 smithi196 ceph-mon[33018]: 6.1e scrub ok 2024-02-05T08:12:22.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:21 smithi196 ceph-mon[33018]: 6.8 scrub starts 2024-02-05T08:12:22.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:21 smithi196 ceph-mon[33018]: 6.8 scrub ok 2024-02-05T08:12:22.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:21 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2554419917' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:12:22.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:22.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:22.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:22.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:12:22.468 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 73 16 102234 102218 87.4883 65.875 0.0106826 0.0114245 2024-02-05T08:12:22.680 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:22.743 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:22.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:22.844 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:22.901 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:22.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:23.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:23.049 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:23.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:23.125 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:12:23.125 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:12:23.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:23.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:23.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:23.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:23.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: 6.b scrub starts 2024-02-05T08:12:23.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: 6.b scrub ok 2024-02-05T08:12:23.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: 3.6 scrub starts 2024-02-05T08:12:23.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: 3.6 scrub ok 2024-02-05T08:12:23.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: pgmap v314: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 658 B/s rd, 42 MiB/s wr, 669 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:23.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:12:23.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:12:23.266 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2554419917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:12:23.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: osdmap e184: 8 total, 8 up, 7 in 2024-02-05T08:12:23.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: 3.f scrub starts 2024-02-05T08:12:23.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: 3.f scrub ok 2024-02-05T08:12:23.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1e"}]: dispatch 2024-02-05T08:12:23.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:12:23.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:12:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: 6.b scrub starts 2024-02-05T08:12:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: 6.b scrub ok 2024-02-05T08:12:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: 3.6 scrub starts 2024-02-05T08:12:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: 3.6 scrub ok 2024-02-05T08:12:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: pgmap v314: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 658 B/s rd, 42 MiB/s wr, 669 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:12:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:12:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2554419917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:12:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: osdmap e184: 8 total, 8 up, 7 in 2024-02-05T08:12:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: 3.f scrub starts 2024-02-05T08:12:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: 3.f scrub ok 2024-02-05T08:12:23.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1e"}]: dispatch 2024-02-05T08:12:23.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:12:23.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:12:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: 6.b scrub starts 2024-02-05T08:12:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: 6.b scrub ok 2024-02-05T08:12:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: 3.6 scrub starts 2024-02-05T08:12:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: 3.6 scrub ok 2024-02-05T08:12:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: pgmap v314: 148 pgs: 148 active+clean; 5.9 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 658 B/s rd, 42 MiB/s wr, 669 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:12:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:12:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2554419917' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:12:23.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: osdmap e184: 8 total, 8 up, 7 in 2024-02-05T08:12:23.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: 3.f scrub starts 2024-02-05T08:12:23.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: 3.f scrub ok 2024-02-05T08:12:23.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1e"}]: dispatch 2024-02-05T08:12:23.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:12:23.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:12:23.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:23.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:23.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:23.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:23.465 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:23.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:23.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:23.468 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 74 16 103273 103257 87.1835 64.9375 0.0166236 0.0114645 2024-02-05T08:12:23.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:12:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:12:23.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:23.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:23.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:23.624 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:23.648 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:23.701 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:23.749 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:23.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:23.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:23.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:23.893 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:23.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:23.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:23.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:24.023 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:24.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:24.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[30086]: 8.4 scrub starts 2024-02-05T08:12:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[30086]: 4.b scrub starts 2024-02-05T08:12:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[30086]: 4.b scrub ok 2024-02-05T08:12:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1e"}]': finished 2024-02-05T08:12:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[30086]: osdmap e185: 8 total, 8 up, 7 in 2024-02-05T08:12:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:12:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:12:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[30086]: 5.1a scrub starts 2024-02-05T08:12:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[30086]: 5.1a scrub ok 2024-02-05T08:12:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[39278]: 8.4 scrub starts 2024-02-05T08:12:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[39278]: 4.b scrub starts 2024-02-05T08:12:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[39278]: 4.b scrub ok 2024-02-05T08:12:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1e"}]': finished 2024-02-05T08:12:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[39278]: osdmap e185: 8 total, 8 up, 7 in 2024-02-05T08:12:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:12:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:12:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[39278]: 5.1a scrub starts 2024-02-05T08:12:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[39278]: 5.1a scrub ok 2024-02-05T08:12:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:23 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:24.125 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:24.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:24.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:24.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:24.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:24.313 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:24.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:23 smithi196 ceph-mon[33018]: 8.4 scrub starts 2024-02-05T08:12:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:23 smithi196 ceph-mon[33018]: 4.b scrub starts 2024-02-05T08:12:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:23 smithi196 ceph-mon[33018]: 4.b scrub ok 2024-02-05T08:12:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "format": "json", "pgid": "4.1e"}]': finished 2024-02-05T08:12:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:23 smithi196 ceph-mon[33018]: osdmap e185: 8 total, 8 up, 7 in 2024-02-05T08:12:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:23 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:12:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:23 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:12:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:23 smithi196 ceph-mon[33018]: 5.1a scrub starts 2024-02-05T08:12:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:23 smithi196 ceph-mon[33018]: 5.1a scrub ok 2024-02-05T08:12:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:23 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:24.414 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:24.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:24.463 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:24.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:24.469 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 75 16 104295 104279 86.8727 63.875 0.00841392 0.0115062 2024-02-05T08:12:24.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:24.592 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:24.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:24.694 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:24.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:24.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:24.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:24.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:24.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:24.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:24.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:24.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:24.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:25.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[30086]: 6.7 scrub starts 2024-02-05T08:12:25.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[30086]: 6.7 scrub ok 2024-02-05T08:12:25.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[30086]: 3.7 scrub starts 2024-02-05T08:12:25.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[30086]: 3.7 scrub ok 2024-02-05T08:12:25.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[30086]: pgmap v317: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.5 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:25.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[30086]: 8.4 scrub ok 2024-02-05T08:12:25.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[30086]: osdmap e186: 8 total, 8 up, 7 in 2024-02-05T08:12:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[39278]: 6.7 scrub starts 2024-02-05T08:12:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[39278]: 6.7 scrub ok 2024-02-05T08:12:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[39278]: 3.7 scrub starts 2024-02-05T08:12:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[39278]: 3.7 scrub ok 2024-02-05T08:12:25.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[39278]: pgmap v317: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.5 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:25.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[39278]: 8.4 scrub ok 2024-02-05T08:12:25.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:24 smithi134 ceph-mon[39278]: osdmap e186: 8 total, 8 up, 7 in 2024-02-05T08:12:25.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:25.113 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:25.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:25.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:25.243 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:25.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:24 smithi196 ceph-mon[33018]: 6.7 scrub starts 2024-02-05T08:12:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: 6.7 scrub ok 2024-02-05T08:12:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: 3.7 scrub starts 2024-02-05T08:12:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: 3.7 scrub ok 2024-02-05T08:12:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: pgmap v317: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.5 KiB/s rd, 98 MiB/s wr, 1.57k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: 8.4 scrub ok 2024-02-05T08:12:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: osdmap e186: 8 total, 8 up, 7 in 2024-02-05T08:12:25.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:25.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:25.376 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:25.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:25.468 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 76 16 105205 105189 86.4781 56.875 0.0126769 0.0115582 2024-02-05T08:12:25.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:25.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:12:25.520 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:25.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:25.622 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:25.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:25.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:25.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:25.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:25.844 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:25.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:25.945 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:25.986 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:25.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:26.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:26.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:26.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:26.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:26.129 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: 5.7 scrub starts 2024-02-05T08:12:26.129 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: 5.7 scrub ok 2024-02-05T08:12:26.129 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: 4.0 deep-scrub starts 2024-02-05T08:12:26.129 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: 4.0 deep-scrub ok 2024-02-05T08:12:26.129 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: 5.11 scrub starts 2024-02-05T08:12:26.130 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: 5.11 scrub ok 2024-02-05T08:12:26.130 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: 6.1 scrub starts 2024-02-05T08:12:26.130 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:25 smithi196 ceph-mon[33018]: 6.1 scrub ok 2024-02-05T08:12:26.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T08:12:26.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[30086]: 5.7 scrub starts 2024-02-05T08:12:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[30086]: 5.7 scrub ok 2024-02-05T08:12:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[30086]: 4.0 deep-scrub starts 2024-02-05T08:12:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[30086]: 4.0 deep-scrub ok 2024-02-05T08:12:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[30086]: 5.11 scrub starts 2024-02-05T08:12:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[30086]: 5.11 scrub ok 2024-02-05T08:12:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[30086]: 6.1 scrub starts 2024-02-05T08:12:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[30086]: 6.1 scrub ok 2024-02-05T08:12:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[39278]: 5.7 scrub starts 2024-02-05T08:12:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[39278]: 5.7 scrub ok 2024-02-05T08:12:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[39278]: 4.0 deep-scrub starts 2024-02-05T08:12:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[39278]: 4.0 deep-scrub ok 2024-02-05T08:12:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[39278]: 5.11 scrub starts 2024-02-05T08:12:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[39278]: 5.11 scrub ok 2024-02-05T08:12:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[39278]: 6.1 scrub starts 2024-02-05T08:12:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:25 smithi134 ceph-mon[39278]: 6.1 scrub ok 2024-02-05T08:12:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:26.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:26.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:26.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:26.431 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:26.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:26.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:26.468 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 77 16 106300 106284 86.2437 68.4375 0.00846444 0.0115899 2024-02-05T08:12:26.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:26.548 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:12:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:26.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:26.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:26.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:26.710 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:26.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:26.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:26.849 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:26.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:26.950 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:27.005 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:27.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:27.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:27.107 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:27.166 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:27.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:27.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:27.196 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:12:27.196 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:12:27.196 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[30086]: pgmap v319: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 900 op/s 2024-02-05T08:12:27.196 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[30086]: 6.2 scrub starts 2024-02-05T08:12:27.196 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[30086]: 6.2 scrub ok 2024-02-05T08:12:27.197 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[30086]: 6.4 scrub starts 2024-02-05T08:12:27.197 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[30086]: 6.4 scrub ok 2024-02-05T08:12:27.197 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:12:27.197 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:12:27.197 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[39278]: pgmap v319: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 900 op/s 2024-02-05T08:12:27.197 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[39278]: 6.2 scrub starts 2024-02-05T08:12:27.197 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[39278]: 6.2 scrub ok 2024-02-05T08:12:27.197 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[39278]: 6.4 scrub starts 2024-02-05T08:12:27.197 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:26 smithi134 ceph-mon[39278]: 6.4 scrub ok 2024-02-05T08:12:27.269 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:27.290 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:12:27.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:27.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:27.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:26 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:12:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:26 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:12:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:26 smithi196 ceph-mon[33018]: pgmap v319: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 853 B/s rd, 56 MiB/s wr, 900 op/s 2024-02-05T08:12:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:26 smithi196 ceph-mon[33018]: 6.2 scrub starts 2024-02-05T08:12:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:26 smithi196 ceph-mon[33018]: 6.2 scrub ok 2024-02-05T08:12:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:26 smithi196 ceph-mon[33018]: 6.4 scrub starts 2024-02-05T08:12:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:26 smithi196 ceph-mon[33018]: 6.4 scrub ok 2024-02-05T08:12:27.331 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:12:27.376 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:12:27.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:27.421 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:12:27.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:27.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:27.464 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:12:27.468 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 78 16 107347 107331 85.9768 65.4375 0.0303329 0.0116261 2024-02-05T08:12:27.510 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:12:27.527 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:27.557 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:12:27.574 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:27.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:27.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:27.604 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:12:27.611 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:12:27.611 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:12:27.611 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:12:27.612 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:12:27.612 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:12:27.613 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:12:27.613 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:12:27.613 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:12:27.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:27.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:27.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:27.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:27.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:27.861 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:27.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:27.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:27.950 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:27.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:27.994 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:12:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:12:28.032 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:28.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:28.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:28.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:28.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:28.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:28.170 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:27 smithi196 ceph-mon[33018]: 4.8 scrub starts 2024-02-05T08:12:28.170 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:27 smithi196 ceph-mon[33018]: 4.8 scrub ok 2024-02-05T08:12:28.170 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:27 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:12:28.170 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:27 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:12:28.170 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:27 smithi196 ceph-mon[33018]: 6.c scrub starts 2024-02-05T08:12:28.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:27 smithi196 ceph-mon[33018]: 6.c scrub ok 2024-02-05T08:12:28.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:27 smithi196 ceph-mon[33018]: 4.5 scrub starts 2024-02-05T08:12:28.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:27 smithi196 ceph-mon[33018]: 4.5 scrub ok 2024-02-05T08:12:28.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:27 smithi196 ceph-mon[33018]: 6.5 scrub starts 2024-02-05T08:12:28.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:27 smithi196 ceph-mon[33018]: 6.5 scrub ok 2024-02-05T08:12:28.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:27 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3416521477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:28.268 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[30086]: 4.8 scrub starts 2024-02-05T08:12:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[30086]: 4.8 scrub ok 2024-02-05T08:12:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:12:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:12:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[30086]: 6.c scrub starts 2024-02-05T08:12:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[30086]: 6.c scrub ok 2024-02-05T08:12:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[30086]: 4.5 scrub starts 2024-02-05T08:12:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[30086]: 4.5 scrub ok 2024-02-05T08:12:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[30086]: 6.5 scrub starts 2024-02-05T08:12:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[30086]: 6.5 scrub ok 2024-02-05T08:12:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3416521477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:28 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a[30063]: 2024-02-05T08:12:28.215+0000 7f4523bf3700 -1 mon.a@0(leader).osd e186 definitely_dead 0 2024-02-05T08:12:28.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[39278]: 4.8 scrub starts 2024-02-05T08:12:28.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[39278]: 4.8 scrub ok 2024-02-05T08:12:28.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:12:28.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:12:28.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[39278]: 6.c scrub starts 2024-02-05T08:12:28.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[39278]: 6.c scrub ok 2024-02-05T08:12:28.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[39278]: 4.5 scrub starts 2024-02-05T08:12:28.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[39278]: 4.5 scrub ok 2024-02-05T08:12:28.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[39278]: 6.5 scrub starts 2024-02-05T08:12:28.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[39278]: 6.5 scrub ok 2024-02-05T08:12:28.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:27 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3416521477' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:28.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:28.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:28.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:28.399 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:28.438 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:28.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:28.468 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 79 16 108314 108298 85.6535 60.4375 0.0111187 0.0116702 2024-02-05T08:12:28.487 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:12:28.487 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:12:28.488 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T08:12:28.488 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T08:12:28.488 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:12:28.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:28.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:28.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:28.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:28.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:28.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:28.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:28.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:28.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:28.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:28.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:28.928 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:28.966 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:28.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:29.013 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[30086]: pgmap v320: 148 pgs: 1 peering, 147 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:12:29.013 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[30086]: 4.3 scrub starts 2024-02-05T08:12:29.013 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[30086]: 4.3 scrub ok 2024-02-05T08:12:29.013 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[30086]: 4.1 scrub starts 2024-02-05T08:12:29.043 INFO:teuthology.orchestra.run.smithi134.stderr:marked in osd.6. 2024-02-05T08:12:29.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:29.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:29.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:29.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:29.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:29 smithi196 ceph-mon[33018]: pgmap v320: 148 pgs: 1 peering, 147 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:12:29.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:29 smithi196 ceph-mon[33018]: 4.3 scrub starts 2024-02-05T08:12:29.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:29 smithi196 ceph-mon[33018]: 4.3 scrub ok 2024-02-05T08:12:29.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:29 smithi196 ceph-mon[33018]: 4.1 scrub starts 2024-02-05T08:12:29.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:29 smithi196 ceph-mon[33018]: 4.1 scrub ok 2024-02-05T08:12:29.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:29 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1557441510' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-05T08:12:29.172 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:29 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-05T08:12:29.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:29.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T08:12:29.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:29.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:29.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:29.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[30086]: 4.1 scrub ok 2024-02-05T08:12:29.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1557441510' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-05T08:12:29.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-05T08:12:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[39278]: pgmap v320: 148 pgs: 1 peering, 147 active+clean; 6.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:12:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[39278]: 4.3 scrub starts 2024-02-05T08:12:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[39278]: 4.3 scrub ok 2024-02-05T08:12:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[39278]: 4.1 scrub starts 2024-02-05T08:12:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[39278]: 4.1 scrub ok 2024-02-05T08:12:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1557441510' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-05T08:12:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:29 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-02-05T08:12:29.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:29.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:29.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:29.433 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:12:29.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:29.468 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:12:29.469687+0000 min lat: 0.00115834 max lat: 0.159276 avg lat: 0.0116838 2024-02-05T08:12:29.468 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:12:29.468 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 80 16 109000 108984 85.1189 42.875 0.00390222 0.0116838 2024-02-05T08:12:29.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:29.474 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:29.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:29.575 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:29.598 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:29.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:29.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:29.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:29.727 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:29.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:29.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:29.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:29.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:29.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:29.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:29.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:29.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:29.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:29.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:30.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:30.106 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:30.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:30.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:30.208 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:30.233 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:30.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:30.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[30086]: 5.5 scrub starts 2024-02-05T08:12:30.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[30086]: 5.5 scrub ok 2024-02-05T08:12:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-05T08:12:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[30086]: osdmap e187: 8 total, 8 up, 8 in 2024-02-05T08:12:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[30086]: 6.1c scrub starts 2024-02-05T08:12:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[30086]: 6.1c scrub ok 2024-02-05T08:12:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[30086]: osdmap e188: 8 total, 8 up, 8 in 2024-02-05T08:12:30.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[39278]: 5.5 scrub starts 2024-02-05T08:12:30.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[39278]: 5.5 scrub ok 2024-02-05T08:12:30.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-05T08:12:30.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[39278]: osdmap e187: 8 total, 8 up, 8 in 2024-02-05T08:12:30.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[39278]: 6.1c scrub starts 2024-02-05T08:12:30.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[39278]: 6.1c scrub ok 2024-02-05T08:12:30.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:30 smithi134 ceph-mon[39278]: osdmap e188: 8 total, 8 up, 8 in 2024-02-05T08:12:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:30 smithi196 ceph-mon[33018]: 5.5 scrub starts 2024-02-05T08:12:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:30 smithi196 ceph-mon[33018]: 5.5 scrub ok 2024-02-05T08:12:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:30 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-02-05T08:12:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:30 smithi196 ceph-mon[33018]: osdmap e187: 8 total, 8 up, 8 in 2024-02-05T08:12:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:30 smithi196 ceph-mon[33018]: 6.1c scrub starts 2024-02-05T08:12:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:30 smithi196 ceph-mon[33018]: 6.1c scrub ok 2024-02-05T08:12:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:30 smithi196 ceph-mon[33018]: osdmap e188: 8 total, 8 up, 8 in 2024-02-05T08:12:30.334 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:30.359 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:30.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:30.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:30.468 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 81 16 109000 108984 84.0682 0 - 0.0116838 2024-02-05T08:12:30.486 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:30.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:30.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:30.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:30.612 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:30.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:30.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:30.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:30.737 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:30.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:30.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:30.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:30.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:30.969 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:31.009 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:31.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:31.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:31.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:31.144 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:31.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:31.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:31.231 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:12:31.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:31.271 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[30086]: 5.8 scrub starts 2024-02-05T08:12:31.271 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[30086]: 5.8 scrub ok 2024-02-05T08:12:31.271 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[30086]: pgmap v322: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 765 B/s rd, 70 MiB/s wr, 1.11k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:31.271 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[30086]: 4.13 scrub starts 2024-02-05T08:12:31.272 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[30086]: 4.13 scrub ok 2024-02-05T08:12:31.272 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[30086]: osdmap e189: 8 total, 8 up, 8 in 2024-02-05T08:12:31.272 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[30086]: 8.6 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6783 2024-02-05T08:12:31.272 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[30086]: 8.5 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6808 2024-02-05T08:12:31.272 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[39278]: 5.8 scrub starts 2024-02-05T08:12:31.272 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[39278]: 5.8 scrub ok 2024-02-05T08:12:31.272 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[39278]: pgmap v322: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 765 B/s rd, 70 MiB/s wr, 1.11k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:31.272 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[39278]: 4.13 scrub starts 2024-02-05T08:12:31.272 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[39278]: 4.13 scrub ok 2024-02-05T08:12:31.273 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[39278]: osdmap e189: 8 total, 8 up, 8 in 2024-02-05T08:12:31.273 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[39278]: 8.6 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6783 2024-02-05T08:12:31.273 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:31 smithi134 ceph-mon[39278]: 8.5 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6808 2024-02-05T08:12:31.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:31 smithi196 ceph-mon[33018]: 5.8 scrub starts 2024-02-05T08:12:31.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:31 smithi196 ceph-mon[33018]: 5.8 scrub ok 2024-02-05T08:12:31.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:31 smithi196 ceph-mon[33018]: pgmap v322: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 765 B/s rd, 70 MiB/s wr, 1.11k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:31.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:31 smithi196 ceph-mon[33018]: 4.13 scrub starts 2024-02-05T08:12:31.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:31 smithi196 ceph-mon[33018]: 4.13 scrub ok 2024-02-05T08:12:31.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:31 smithi196 ceph-mon[33018]: osdmap e189: 8 total, 8 up, 8 in 2024-02-05T08:12:31.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:31 smithi196 ceph-mon[33018]: 8.6 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6783 2024-02-05T08:12:31.306 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:31 smithi196 ceph-mon[33018]: 8.5 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6808 2024-02-05T08:12:31.327 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:31.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:31.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:31.374 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-02-05T08:12:31.429 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:31.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:31.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:31.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:31.469 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 82 16 109674 109658 83.5567 21.0625 0.0173282 0.0119624 2024-02-05T08:12:31.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:31.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:31.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:31.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:31.742 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:31.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:31.760 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:12:31.760 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:12:31.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:12:31.760 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:12:31.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.760 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14481.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-02-05T08:12:31.760 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:04:45.556099+0000", 2024-02-05T08:12:31.760 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 466.20268743499997, 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012799511, 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14481", 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2578723559", 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 46 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556099+0000", 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556099+0000", 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556100+0000", 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5910000000000001e-06 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556105+0000", 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1019999999999996e-06 2024-02-05T08:12:31.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556118+0000", 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2788e-05 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556128+0000", 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0334e-05 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556227+0000", 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8481999999999997e-05 2024-02-05T08:12:31.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.764 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:12:31.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556244+0000", 2024-02-05T08:12:31.764 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7425999999999999e-05 2024-02-05T08:12:31.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.764 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.565681+0000", 2024-02-05T08:12:31.764 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094368549999999992 2024-02-05T08:12:31.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.764 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.565756+0000", 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4908999999999997e-05 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.565877+0000", 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012119499999999999 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.566533+0000", 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00065580100000000002 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:31.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.568576+0000", 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020428600000000001 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.568849+0000", 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00027359999999999998 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.568877+0000", 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7923e-05 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.568898+0000", 2024-02-05T08:12:31.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0645e-05 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24344.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:04:45.556413+0000", 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 466.20237282099998, 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012406772, 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24344", 2024-02-05T08:12:31.767 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1272526039", 2024-02-05T08:12:31.768 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 46 2024-02-05T08:12:31.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.768 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556413+0000", 2024-02-05T08:12:31.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556413+0000", 2024-02-05T08:12:31.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556414+0000", 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0189999999999999e-06 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556419+0000", 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0390000000000004e-06 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556432+0000", 2024-02-05T08:12:31.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2706e-05 2024-02-05T08:12:31.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556441+0000", 2024-02-05T08:12:31.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4350000000000003e-06 2024-02-05T08:12:31.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556530+0000", 2024-02-05T08:12:31.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8701000000000003e-05 2024-02-05T08:12:31.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:12:31.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.556537+0000", 2024-02-05T08:12:31.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.323e-06 2024-02-05T08:12:31.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.565182+0000", 2024-02-05T08:12:31.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0086446800000000001 2024-02-05T08:12:31.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.565322+0000", 2024-02-05T08:12:31.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013962499999999999 2024-02-05T08:12:31.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 4,7", 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.565456+0000", 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013403300000000001 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.566229+0000", 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00077307199999999997 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.568518+0000", 2024-02-05T08:12:31.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022892070000000001 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.568730+0000", 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000212289 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.568791+0000", 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0906000000000003e-05 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:04:45.568820+0000", 2024-02-05T08:12:31.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8736999999999999e-05 2024-02-05T08:12:31.774 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.774 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.774 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:108961 8.9 8.107267e9 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:29.041581+0000", 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 2.717205179, 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62651763699999996, 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108961 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.775 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041581+0000", 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041581+0000", 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041582+0000", 2024-02-05T08:12:31.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5570000000000001e-06 2024-02-05T08:12:31.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041733+0000", 2024-02-05T08:12:31.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015074700000000001 2024-02-05T08:12:31.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041745+0000", 2024-02-05T08:12:31.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1508999999999999e-05 2024-02-05T08:12:31.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041755+0000", 2024-02-05T08:12:31.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0533000000000001e-05 2024-02-05T08:12:31.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041803+0000", 2024-02-05T08:12:31.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8022999999999997e-05 2024-02-05T08:12:31.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:12:31.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041811+0000", 2024-02-05T08:12:31.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0460000000000006e-06 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668084+0000", 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62627265600000004 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668098+0000", 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4566000000000001e-05 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.780 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:108978 8.9 8.d5065aa9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:12:31.780 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:29.058870+0000", 2024-02-05T08:12:31.780 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 2.6999159530000001, 2024-02-05T08:12:31.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.60925642300000005, 2024-02-05T08:12:31.780 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.780 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:12:31.780 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:31.780 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:31.780 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:31.780 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108978 2024-02-05T08:12:31.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.780 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058870+0000", 2024-02-05T08:12:31.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058870+0000", 2024-02-05T08:12:31.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058872+0000", 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.655e-06 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058938+0000", 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6568000000000004e-05 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058949+0000", 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.058e-05 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058957+0000", 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8099999999999998e-06 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058997+0000", 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0797999999999998e-05 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059004+0000", 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7510000000000002e-06 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668120+0000", 2024-02-05T08:12:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.60911546800000005 2024-02-05T08:12:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668126+0000", 2024-02-05T08:12:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.793e-06 2024-02-05T08:12:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.785 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:108980 8.5 8.acde60b5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:12:31.785 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:29.059192+0000", 2024-02-05T08:12:31.785 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 2.6995936039999999, 2024-02-05T08:12:31.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.61158693799999997, 2024-02-05T08:12:31.785 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.785 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:12:31.785 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:31.785 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:31.785 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:31.785 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108980 2024-02-05T08:12:31.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.785 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.786 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059192+0000", 2024-02-05T08:12:31.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.786 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059192+0000", 2024-02-05T08:12:31.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.786 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059193+0000", 2024-02-05T08:12:31.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4900000000000005e-07 2024-02-05T08:12:31.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059259+0000", 2024-02-05T08:12:31.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5699999999999998e-05 2024-02-05T08:12:31.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059272+0000", 2024-02-05T08:12:31.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3658e-05 2024-02-05T08:12:31.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059281+0000", 2024-02-05T08:12:31.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9469999999999992e-06 2024-02-05T08:12:31.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059332+0000", 2024-02-05T08:12:31.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0247999999999997e-05 2024-02-05T08:12:31.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:12:31.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059338+0000", 2024-02-05T08:12:31.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4659999999999996e-06 2024-02-05T08:12:31.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.670768+0000", 2024-02-05T08:12:31.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.61143003200000001 2024-02-05T08:12:31.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.670779+0000", 2024-02-05T08:12:31.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1138000000000001e-05 2024-02-05T08:12:31.789 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.789 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.789 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.790 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:109000 8.9 8.6538e8d9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:12:31.790 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:29.089503+0000", 2024-02-05T08:12:31.790 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 2.6692828579999999, 2024-02-05T08:12:31.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.57864895999999999, 2024-02-05T08:12:31.790 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.790 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:12:31.790 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:12:31.790 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:12:31.790 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:31.790 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 109000 2024-02-05T08:12:31.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.790 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089503+0000", 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089503+0000", 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089505+0000", 2024-02-05T08:12:31.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.559e-06 2024-02-05T08:12:31.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.792 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.792 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089552+0000", 2024-02-05T08:12:31.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7407000000000002e-05 2024-02-05T08:12:31.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.792 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.792 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089568+0000", 2024-02-05T08:12:31.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5722999999999998e-05 2024-02-05T08:12:31.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.792 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.792 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089576+0000", 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1759999999999993e-06 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089638+0000", 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2442999999999998e-05 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089646+0000", 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2629999999999997e-06 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668145+0000", 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.57849936599999996 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668152+0000", 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0230000000000004e-06 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.15087.0:108979 8.3 e189/188 8:cfa8e567:::benchmark_data_smithi134_115836_object108978:head v 189'6852, mlcod=0'0)", 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:30.755474+0000", 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 1.003311963, 2024-02-05T08:12:31.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058019259999999998, 2024-02-05T08:12:31.795 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.795 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.795 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.755474+0000", 2024-02-05T08:12:31.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.755474+0000", 2024-02-05T08:12:31.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.755475+0000", 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.279e-06 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.755485+0000", 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3530000000000001e-06 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.755505+0000", 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0732000000000001e-05 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.755515+0000", 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9979999999999998e-06 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.755583+0000", 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7630000000000001e-05 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.755608+0000", 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5014000000000001e-05 2024-02-05T08:12:31.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.798 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.798 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.761190+0000", 2024-02-05T08:12:31.798 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055823069999999999 2024-02-05T08:12:31.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.798 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.798 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.761276+0000", 2024-02-05T08:12:31.798 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5612999999999995e-05 2024-02-05T08:12:31.798 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.798 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.798 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.15087.0:108987 8.3 e189/188 8:c7b15d84:::benchmark_data_smithi134_115836_object108986:head v 189'6853, mlcod=0'0)", 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:30.759322+0000", 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.99946360300000003, 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058269369999999999, 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.759322+0000", 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.759322+0000", 2024-02-05T08:12:31.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.800 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.759323+0000", 2024-02-05T08:12:31.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5300000000000002e-07 2024-02-05T08:12:31.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.800 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.759343+0000", 2024-02-05T08:12:31.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9281000000000001e-05 2024-02-05T08:12:31.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.759373+0000", 2024-02-05T08:12:31.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0465999999999999e-05 2024-02-05T08:12:31.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.759383+0000", 2024-02-05T08:12:31.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6609999999999995e-06 2024-02-05T08:12:31.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.759475+0000", 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2219999999999995e-05 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.759493+0000", 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8110000000000001e-05 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.765040+0000", 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005546835 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.765149+0000", 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109411 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.15087.0:108994 8.6 e189/188 8:69b35254:::benchmark_data_smithi134_115836_object108993:head v 189'6787, mlcod=189'6784)", 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:30.782705+0000", 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.97608119500000001, 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006496801, 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.782705+0000", 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.782705+0000", 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.782705+0000", 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2399999999999998e-07 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.782710+0000", 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7149999999999997e-06 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.782718+0000", 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7300000000000005e-06 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.782723+0000", 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5929999999999996e-06 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.783271+0000", 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00054793799999999998 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.783291+0000", 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0154999999999999e-05 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.789147+0000", 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058552179999999997 2024-02-05T08:12:31.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:30.789202+0000", 2024-02-05T08:12:31.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4928e-05 2024-02-05T08:12:31.807 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.807 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.807 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.807 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "MOSDPGPush(8.5 189/188 [PushOp(8:a0134928:::benchmark_data_smithi134_115836_object35304:head, version: 168'2261, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:a0134928:::benchmark_data_smithi134_115836_object35304:head@168'2261, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T08:12:31.807 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:31.024967+0000", 2024-02-05T08:12:31.807 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.733818572, 2024-02-05T08:12:31.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012673772999999999, 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.024967+0000", 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.024967+0000", 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.024968+0000", 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8999999999999995e-07 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.025040+0000", 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2082999999999994e-05 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.025067+0000", 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6740000000000001e-05 2024-02-05T08:12:31.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.025084+0000", 2024-02-05T08:12:31.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6589999999999999e-05 2024-02-05T08:12:31.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.037388+0000", 2024-02-05T08:12:31.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012304344 2024-02-05T08:12:31.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.037401+0000", 2024-02-05T08:12:31.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3343000000000001e-05 2024-02-05T08:12:31.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.811 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.037641+0000", 2024-02-05T08:12:31.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00023988300000000001 2024-02-05T08:12:31.811 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.811 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.811 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.811 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.811 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "pg_backfill(progress 8.5 e 189/189 lb 8:a01205fc:::benchmark_data_smithi134_115836_object35766:head)", 2024-02-05T08:12:31.811 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:31.025110+0000", 2024-02-05T08:12:31.812 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.73367570400000004, 2024-02-05T08:12:31.812 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012204646, 2024-02-05T08:12:31.812 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.812 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T08:12:31.812 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.812 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.812 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.025110+0000", 2024-02-05T08:12:31.812 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.812 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.025110+0000", 2024-02-05T08:12:31.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.813 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.025111+0000", 2024-02-05T08:12:31.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2199999999999999e-07 2024-02-05T08:12:31.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.813 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.025118+0000", 2024-02-05T08:12:31.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2169999999999997e-06 2024-02-05T08:12:31.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.025127+0000", 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0920000000000004e-06 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.025133+0000", 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8780000000000003e-06 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.025194+0000", 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1179000000000005e-05 2024-02-05T08:12:31.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.025209+0000", 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4581e-05 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.037315+0000", 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012106277 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.15087.0:109260 8.5 e189/188 8:acfbe2a6:::benchmark_data_smithi134_115836_object109259:head v 189'6823, mlcod=189'6821)", 2024-02-05T08:12:31.815 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:31.026085+0000", 2024-02-05T08:12:31.816 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.73270079399999999, 2024-02-05T08:12:31.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013496889, 2024-02-05T08:12:31.816 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.816 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.816 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.026085+0000", 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.026085+0000", 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.026086+0000", 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4499999999999997e-07 2024-02-05T08:12:31.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.026091+0000", 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.366e-06 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.026102+0000", 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1080000000000001e-05 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.026109+0000", 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6540000000000002e-06 2024-02-05T08:12:31.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.037669+0000", 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011559699 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.037694+0000", 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5486e-05 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.039550+0000", 2024-02-05T08:12:31.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0018554349999999999 2024-02-05T08:12:31.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.039582+0000", 2024-02-05T08:12:31.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2524000000000002e-05 2024-02-05T08:12:31.820 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.820 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.820 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.820 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.15087.0:109266 8.7 e189/188 8:efa68f23:::benchmark_data_smithi134_115836_object109265:head v 189'6672, mlcod=189'6670)", 2024-02-05T08:12:31.820 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:31.034396+0000", 2024-02-05T08:12:31.820 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.72438963300000003, 2024-02-05T08:12:31.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050958360000000003, 2024-02-05T08:12:31.821 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.821 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.821 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.034396+0000", 2024-02-05T08:12:31.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.034396+0000", 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.034397+0000", 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9400000000000005e-07 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.034402+0000", 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5709999999999997e-06 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.034414+0000", 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1963999999999999e-05 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.034423+0000", 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6950000000000006e-06 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.037664+0000", 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032405020000000001 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.037682+0000", 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8853000000000001e-05 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.039446+0000", 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017632979999999999 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.039492+0000", 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6359000000000001e-05 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.824 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.15087.0:109389 8.5 e189/188 8:a846e00e:::benchmark_data_smithi134_115836_object109388:head v 189'6830, mlcod=189'6828)", 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:31.147521+0000", 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.61126507200000002, 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053038060000000003, 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.147521+0000", 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.147521+0000", 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.147522+0000", 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6599999999999996e-07 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.147528+0000", 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9800000000000003e-06 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.826 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.147541+0000", 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.361e-05 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.147549+0000", 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6580000000000002e-06 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.149020+0000", 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001470938 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.149043+0000", 2024-02-05T08:12:31.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3374e-05 2024-02-05T08:12:31.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.152773+0000", 2024-02-05T08:12:31.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0037302400000000001 2024-02-05T08:12:31.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.152825+0000", 2024-02-05T08:12:31.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.134e-05 2024-02-05T08:12:31.828 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.828 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.828 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.15087.0:109777 8.5 e189/188 8:af1f47d3:::benchmark_data_smithi134_115836_object109776:head v 189'6854, mlcod=189'6852)", 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:31.582129+0000", 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.17665704600000001, 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057936630000000001, 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.582129+0000", 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.829 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.582129+0000", 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.582130+0000", 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2300000000000001e-06 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.582143+0000", 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2583e-05 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.582164+0000", 2024-02-05T08:12:31.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1486999999999999e-05 2024-02-05T08:12:31.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.831 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.582176+0000", 2024-02-05T08:12:31.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1420000000000001e-05 2024-02-05T08:12:31.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.831 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.584298+0000", 2024-02-05T08:12:31.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021219059999999998 2024-02-05T08:12:31.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.584321+0000", 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3425999999999999e-05 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.587875+0000", 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003554167 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.587923+0000", 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7444000000000002e-05 2024-02-05T08:12:31.832 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.15087.0:109778 8.7 e189/188 8:ed4b01fe:::benchmark_data_smithi134_115836_object109777:head v 189'6711, mlcod=189'6709)", 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:31.585011+0000", 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.17377477899999999, 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0069189400000000002, 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.585011+0000", 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.585011+0000", 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.585012+0000", 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6400000000000002e-07 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.585017+0000", 2024-02-05T08:12:31.834 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4550000000000003e-06 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.585027+0000", 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5570000000000001e-06 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.585034+0000", 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1520000000000003e-06 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.587960+0000", 2024-02-05T08:12:31.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002925619 2024-02-05T08:12:31.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.587980+0000", 2024-02-05T08:12:31.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0302e-05 2024-02-05T08:12:31.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.591867+0000", 2024-02-05T08:12:31.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0038869920000000001 2024-02-05T08:12:31.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.591930+0000", 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3299000000000005e-05 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.15087.0:109786 8.3 e189/188 8:c109c4ed:::benchmark_data_smithi134_115836_object109785:head v 189'6887, mlcod=189'6885)", 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:31.586180+0000", 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.17260578600000001, 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055993559999999998, 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.837 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.586180+0000", 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.586180+0000", 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.838 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.586181+0000", 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5299999999999999e-07 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.586186+0000", 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3090000000000003e-06 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.586194+0000", 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2279999999999998e-06 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.586200+0000", 2024-02-05T08:12:31.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8370000000000002e-06 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.586268+0000", 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8318999999999994e-05 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.586289+0000", 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0511000000000001e-05 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.591729+0000", 2024-02-05T08:12:31.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054398210000000001 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.591780+0000", 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0878000000000003e-05 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.15087.0:109791 8.9 e189/188 8:9ab381b8:::benchmark_data_smithi134_115836_object109790:head v 189'6792, mlcod=189'6790)", 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:31.590400+0000", 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.16838545999999999, 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006173843, 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.841 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.590400+0000", 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.590400+0000", 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.590401+0000", 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4799999999999997e-07 2024-02-05T08:12:31.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.590413+0000", 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1601e-05 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.590425+0000", 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2411e-05 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.590433+0000", 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2760000000000003e-06 2024-02-05T08:12:31.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.590511+0000", 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8861000000000003e-05 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.590534+0000", 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2583e-05 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.596472+0000", 2024-02-05T08:12:31.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059378529999999999 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.596574+0000", 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010241 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.15087.0:109868 8.5 e190/188 8:a2309611:::benchmark_data_smithi134_115836_object109867:head v 190'6857, mlcod=189'6855)", 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:31.678190+0000", 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.080596341000000002, 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059073210000000001, 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.845 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678190+0000", 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678190+0000", 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678190+0000", 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0499999999999999e-07 2024-02-05T08:12:31.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678199+0000", 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6330000000000002e-06 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678205+0000", 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6610000000000003e-06 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678211+0000", 2024-02-05T08:12:31.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4469999999999997e-06 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678279+0000", 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7892999999999995e-05 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678308+0000", 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9094999999999998e-05 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.684047+0000", 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057389590000000001 2024-02-05T08:12:31.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.684097+0000", 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0328000000000003e-05 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.15087.0:109876 8.6 e190/188 8:6bec7e42:::benchmark_data_smithi134_115836_object109875:head v 190'6846, mlcod=189'6844)", 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:31.678226+0000", 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.080559926000000004, 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051304209999999996, 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:12:31.849 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678226+0000", 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678226+0000", 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678226+0000", 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5899999999999998e-07 2024-02-05T08:12:31.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678237+0000", 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0550999999999999e-05 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678247+0000", 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0219e-05 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678252+0000", 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9849999999999997e-06 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.852 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:12:31.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678360+0000", 2024-02-05T08:12:31.852 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107704 2024-02-05T08:12:31.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.852 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.852 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:12:31.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.678380+0000", 2024-02-05T08:12:31.852 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0299e-05 2024-02-05T08:12:31.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.852 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.852 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:12:31.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.683319+0000", 2024-02-05T08:12:31.853 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049384959999999997 2024-02-05T08:12:31.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:12:31.853 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:12:31.853 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:12:31.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:31.683356+0000", 2024-02-05T08:12:31.853 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7907999999999997e-05 2024-02-05T08:12:31.853 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.853 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.853 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.853 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:12:31.853 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:12:31.853 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:12:31.853 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:31.904 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:31.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:32.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:32.042 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:32.085 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:32.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:32.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:32 smithi134 ceph-mon[30086]: 6.1b scrub starts 2024-02-05T08:12:32.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:32 smithi134 ceph-mon[30086]: 6.1b scrub ok 2024-02-05T08:12:32.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:32 smithi134 ceph-mon[30086]: 8.3 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6851 2024-02-05T08:12:32.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:32 smithi134 ceph-mon[30086]: 8.7 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6660 2024-02-05T08:12:32.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:32 smithi134 ceph-mon[30086]: 8.9 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6734 2024-02-05T08:12:32.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:32 smithi134 ceph-mon[30086]: osdmap e190: 8 total, 8 up, 8 in 2024-02-05T08:12:32.153 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:32 smithi134 ceph-mon[39278]: 6.1b scrub starts 2024-02-05T08:12:32.153 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:32 smithi134 ceph-mon[39278]: 6.1b scrub ok 2024-02-05T08:12:32.153 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:32 smithi134 ceph-mon[39278]: 8.3 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6851 2024-02-05T08:12:32.153 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:32 smithi134 ceph-mon[39278]: 8.7 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6660 2024-02-05T08:12:32.153 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:32 smithi134 ceph-mon[39278]: 8.9 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6734 2024-02-05T08:12:32.153 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:32 smithi134 ceph-mon[39278]: osdmap e190: 8 total, 8 up, 8 in 2024-02-05T08:12:32.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:32.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:32.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:32 smithi196 ceph-mon[33018]: 6.1b scrub starts 2024-02-05T08:12:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:32 smithi196 ceph-mon[33018]: 6.1b scrub ok 2024-02-05T08:12:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:32 smithi196 ceph-mon[33018]: 8.3 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6851 2024-02-05T08:12:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:32 smithi196 ceph-mon[33018]: 8.7 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6660 2024-02-05T08:12:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:32 smithi196 ceph-mon[33018]: 8.9 starting backfill to osd.6 from (0'0,0'0] MAX to 186'6734 2024-02-05T08:12:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:32 smithi196 ceph-mon[33018]: osdmap e190: 8 total, 8 up, 8 in 2024-02-05T08:12:32.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:32.346 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:32.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:32.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:32.469 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 83 15 110615 110600 83.2593 58.875 0.0151934 0.0120056 2024-02-05T08:12:32.489 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:32.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:32.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:32.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:32.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:32.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:32.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:32.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:32.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:32.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:32.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:32.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:32.887 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:32.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:32.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:33.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T08:12:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:33.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:33.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:33.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:33.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:33.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:33.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:33.257 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:33.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:33 smithi196 ceph-mon[33018]: pgmap v325: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 53 MiB/s wr, 851 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:33.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:33 smithi196 ceph-mon[33018]: 5.1c scrub starts 2024-02-05T08:12:33.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:33 smithi196 ceph-mon[33018]: 5.1c scrub ok 2024-02-05T08:12:33.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:33 smithi196 ceph-mon[33018]: osdmap e191: 8 total, 8 up, 8 in 2024-02-05T08:12:33.294 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:33.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:33.364 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:12:33.395 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:33.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:33.434 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:33.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:33.469 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 84 16 111506 111490 82.9304 55.625 0.00563168 0.0120525 2024-02-05T08:12:33.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:33 smithi134 ceph-mon[30086]: pgmap v325: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 53 MiB/s wr, 851 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:33.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:33 smithi134 ceph-mon[30086]: 5.1c scrub starts 2024-02-05T08:12:33.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:33 smithi134 ceph-mon[30086]: 5.1c scrub ok 2024-02-05T08:12:33.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:33 smithi134 ceph-mon[30086]: osdmap e191: 8 total, 8 up, 8 in 2024-02-05T08:12:33.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:33 smithi134 ceph-mon[39278]: pgmap v325: 148 pgs: 148 active+clean; 6.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 53 MiB/s wr, 851 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:12:33.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:33 smithi134 ceph-mon[39278]: 5.1c scrub starts 2024-02-05T08:12:33.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:33 smithi134 ceph-mon[39278]: 5.1c scrub ok 2024-02-05T08:12:33.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:33 smithi134 ceph-mon[39278]: osdmap e191: 8 total, 8 up, 8 in 2024-02-05T08:12:33.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:33.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:12:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:12:33.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:33.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:33.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:33.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:33.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:33.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:33.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:33.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:33.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:33.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:33.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:34.000 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:34.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:34.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:34.101 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:34.116 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:12:34.144 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:34.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:34.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:34.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:34.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:34 smithi134 ceph-mon[30086]: 6.6 scrub starts 2024-02-05T08:12:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:34 smithi134 ceph-mon[30086]: 6.6 scrub ok 2024-02-05T08:12:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:34 smithi134 ceph-mon[30086]: 6.1a scrub starts 2024-02-05T08:12:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:34 smithi134 ceph-mon[30086]: 6.1a scrub ok 2024-02-05T08:12:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:34 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:34 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/283055217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:12:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:34 smithi134 ceph-mon[39278]: 6.6 scrub starts 2024-02-05T08:12:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:34 smithi134 ceph-mon[39278]: 6.6 scrub ok 2024-02-05T08:12:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:34 smithi134 ceph-mon[39278]: 6.1a scrub starts 2024-02-05T08:12:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:34 smithi134 ceph-mon[39278]: 6.1a scrub ok 2024-02-05T08:12:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:34 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:34 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/283055217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:12:34.278 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:34.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:34.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:34 smithi196 ceph-mon[33018]: 6.6 scrub starts 2024-02-05T08:12:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:34 smithi196 ceph-mon[33018]: 6.6 scrub ok 2024-02-05T08:12:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:34 smithi196 ceph-mon[33018]: 6.1a scrub starts 2024-02-05T08:12:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:34 smithi196 ceph-mon[33018]: 6.1a scrub ok 2024-02-05T08:12:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:34 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:34 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/283055217' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:12:34.380 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:34.425 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:34.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:34.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:34.469 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 85 16 112357 112341 82.5805 53.1875 0.0274181 0.0121037 2024-02-05T08:12:34.527 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:34.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:34.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:34.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:34.713 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:34.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:34.815 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:34.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:34.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:34.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:34.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:34.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:34.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:35.075 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:35.117 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:35.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:35.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:35.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:35 smithi196 ceph-mon[33018]: 8.8 scrub starts 2024-02-05T08:12:35.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:35 smithi196 ceph-mon[33018]: 6.13 scrub starts 2024-02-05T08:12:35.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:35 smithi196 ceph-mon[33018]: 6.13 scrub ok 2024-02-05T08:12:35.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:35 smithi196 ceph-mon[33018]: pgmap v328: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 1 peering, 142 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 823 op/s; 33690/333684 objects misplaced (10.096%); 13 MiB/s, 214 objects/s recovering 2024-02-05T08:12:35.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:35 smithi196 ceph-mon[33018]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:12:35.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:35 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:12:35.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:35 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/283055217' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:12:35.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:35 smithi196 ceph-mon[33018]: osdmap e192: 8 total, 8 up, 8 in 2024-02-05T08:12:35.263 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:35.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:35.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[30086]: 8.8 scrub starts 2024-02-05T08:12:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[30086]: 6.13 scrub starts 2024-02-05T08:12:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[30086]: 6.13 scrub ok 2024-02-05T08:12:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[30086]: pgmap v328: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 1 peering, 142 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 823 op/s; 33690/333684 objects misplaced (10.096%); 13 MiB/s, 214 objects/s recovering 2024-02-05T08:12:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[30086]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:12:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:12:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/283055217' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:12:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[30086]: osdmap e192: 8 total, 8 up, 8 in 2024-02-05T08:12:35.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[39278]: 8.8 scrub starts 2024-02-05T08:12:35.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[39278]: 6.13 scrub starts 2024-02-05T08:12:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[39278]: 6.13 scrub ok 2024-02-05T08:12:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[39278]: pgmap v328: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 1 peering, 142 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 51 MiB/s wr, 823 op/s; 33690/333684 objects misplaced (10.096%); 13 MiB/s, 214 objects/s recovering 2024-02-05T08:12:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[39278]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:12:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:12:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/283055217' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:12:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:35 smithi134 ceph-mon[39278]: osdmap e192: 8 total, 8 up, 8 in 2024-02-05T08:12:35.297 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:12:35.298 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:12:35.298 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T08:12:35.298 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:12:35.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:35.399 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:35.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:35.469 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 86 16 113178 113162 82.2169 51.3125 0.0289443 0.0121573 2024-02-05T08:12:35.500 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:35.536 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:35.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:35.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:35.637 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:12:35.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:35.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:35.675 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:12:35.721 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:12:35.761 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:12:35.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:35.796 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:35.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:35.800 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:12:35.839 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:12:35.878 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:12:35.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:35.917 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:12:35.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:12:35.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:12:35.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:12:35.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:12:35.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:12:35.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:12:35.923 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:12:35.924 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:12:35.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:35.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:36.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:36.057 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:36.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:36.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:36.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T08:12:36.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:36.205 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:36.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:36.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2300437943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:36.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2300437943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:36.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:36.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2300437943' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:36.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:36.355 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:36.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:36.374 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:12:36.375 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:12:36.375 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 7, 6] 2024-02-05T08:12:36.375 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd out 3 2024-02-05T08:12:36.456 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:36.469 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 87 15 114024 114009 81.8804 52.9375 0.0128788 0.0122083 2024-02-05T08:12:36.473 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:12:36.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:36.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:36.589 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:36.630 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:36.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:36.731 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:36.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:36.762 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:36.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:36.863 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:36.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:12:36.875 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:36.915 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:36.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:36.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:37.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:37.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:37.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:37.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:37.176 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:37.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:37.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:37.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:37 smithi134 ceph-mon[30086]: pgmap v330: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 1 peering, 142 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 41 MiB/s wr, 659 op/s; 33690/333684 objects misplaced (10.096%); 10 MiB/s, 172 objects/s recovering 2024-02-05T08:12:37.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:37 smithi134 ceph-mon[39278]: pgmap v330: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 1 peering, 142 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 41 MiB/s wr, 659 op/s; 33690/333684 objects misplaced (10.096%); 10 MiB/s, 172 objects/s recovering 2024-02-05T08:12:37.278 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:37.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:37.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:37.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:37.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:37 smithi196 ceph-mon[33018]: pgmap v330: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 1 peering, 142 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.0 KiB/s rd, 41 MiB/s wr, 659 op/s; 33690/333684 objects misplaced (10.096%); 10 MiB/s, 172 objects/s recovering 2024-02-05T08:12:37.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:37.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:37.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:37.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:37.469 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 88 16 114877 114861 81.555 53.25 0.0226578 0.0122571 2024-02-05T08:12:37.541 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:37.581 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:37.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:37.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:37.713 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:37.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:37.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:37.756 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:37.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:37.880 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:37.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:37.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:38.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:38.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:38.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:38.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:12:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:12:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:12:38.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:38.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:38.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:38.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:38.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:38.299 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:38.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:38.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:38.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:38.446 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:38.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:38.470 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 89 16 115827 115811 81.3052 59.375 0.010951 0.0122942 2024-02-05T08:12:38.482 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:12:38.482 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:12:38.482 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:12:38.482 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:12:38.547 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:12:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:12:38.589 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:38.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:38.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:38.691 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:38.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:38.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:38.769 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:12:38.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:38.917 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:38.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:38.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:39.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:39.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:39.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:39.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:39.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:39.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:39.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:39.242 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:12:39.242 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:12:39.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:39.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:39.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:39.458 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:39.470 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Total time run: 90.0102 2024-02-05T08:12:39.470 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Total writes made: 116657 2024-02-05T08:12:39.470 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Write size: 65536 2024-02-05T08:12:39.470 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Object size: 65536 2024-02-05T08:12:39.470 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Bandwidth (MB/sec): 81.0026 2024-02-05T08:12:39.470 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev Bandwidth: 32.2428 2024-02-05T08:12:39.470 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max bandwidth (MB/sec): 204.938 2024-02-05T08:12:39.471 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min bandwidth (MB/sec): 0 2024-02-05T08:12:39.471 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Average IOPS: 1296 2024-02-05T08:12:39.471 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev IOPS: 515.885 2024-02-05T08:12:39.471 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max IOPS: 3279 2024-02-05T08:12:39.471 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min IOPS: 0 2024-02-05T08:12:39.471 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Average Latency(s): 0.0123412 2024-02-05T08:12:39.471 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev Latency(s): 0.0218205 2024-02-05T08:12:39.471 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max latency(s): 1.76818 2024-02-05T08:12:39.471 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min latency(s): 0.00115834 2024-02-05T08:12:39.475 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T08:12:39.489 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:39.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:39.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:39 smithi134 ceph-mon[30086]: pgmap v331: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 143 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 45 MiB/s wr, 724 op/s; 33690/336825 objects misplaced (10.002%); 8.4 MiB/s, 143 objects/s recovering 2024-02-05T08:12:39.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:39 smithi134 ceph-mon[39278]: pgmap v331: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 143 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 45 MiB/s wr, 724 op/s; 33690/336825 objects misplaced (10.002%); 8.4 MiB/s, 143 objects/s recovering 2024-02-05T08:12:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:39 smithi196 ceph-mon[33018]: pgmap v331: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 143 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 45 MiB/s wr, 724 op/s; 33690/336825 objects misplaced (10.002%); 8.4 MiB/s, 143 objects/s recovering 2024-02-05T08:12:39.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:39.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:39.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:39.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:39.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:39.760 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:39.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:39.861 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:39.905 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:39.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:39.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:39.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T08:12:40.008 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:40.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:39 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a[30063]: 2024-02-05T08:12:39.714+0000 7f4523bf3700 -1 mon.a@0(leader).osd e192 definitely_dead 0 2024-02-05T08:12:40.053 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:40.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:40.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:40.156 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:40.194 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:40.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:40.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:40.263 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:12:40.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:40.333 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:40.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:40.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:40.434 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:40.476 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:40.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:40.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:40.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:40.611 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:40.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:40.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:40.700 INFO:teuthology.orchestra.run.smithi134.stderr:marked out osd.3. 2024-02-05T08:12:40.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:40.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:40.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:40.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:40.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:40 smithi134 ceph-mon[30086]: pgmap v332: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 143 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 59 MiB/s wr, 953 op/s; 33266/346629 objects misplaced (9.597%); 20 MiB/s, 327 objects/s recovering 2024-02-05T08:12:40.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:40 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1296947354' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-05T08:12:40.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:40 smithi134 ceph-mon[30086]: Client client.admin marked osd.3 out, while it was still marked up 2024-02-05T08:12:40.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:40 smithi134 ceph-mon[39278]: pgmap v332: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 143 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 59 MiB/s wr, 953 op/s; 33266/346629 objects misplaced (9.597%); 20 MiB/s, 327 objects/s recovering 2024-02-05T08:12:40.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:40 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1296947354' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-05T08:12:40.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:40 smithi134 ceph-mon[39278]: Client client.admin marked osd.3 out, while it was still marked up 2024-02-05T08:12:40.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:40.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:40.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:40.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:40.996 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:40.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:40 smithi196 ceph-mon[33018]: pgmap v332: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 143 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 59 MiB/s wr, 953 op/s; 33266/346629 objects misplaced (9.597%); 20 MiB/s, 327 objects/s recovering 2024-02-05T08:12:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:40 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1296947354' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-02-05T08:12:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:40 smithi196 ceph-mon[33018]: Client client.admin marked osd.3 out, while it was still marked up 2024-02-05T08:12:41.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:41.121 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:41.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:41.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:41.223 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:41.251 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:41.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:41.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:41.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:41.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:41.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:41.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:41.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:41.507 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:41.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:41.608 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:41.634 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:41.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:41.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:41.736 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:41.776 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:41.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:41.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:41.808 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:41 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1296947354' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-05T08:12:41.808 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:41 smithi196 ceph-mon[33018]: osdmap e193: 8 total, 8 up, 7 in 2024-02-05T08:12:41.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:41.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:41.901 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:41.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:42.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:42.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:41 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1296947354' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-05T08:12:42.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:41 smithi134 ceph-mon[30086]: osdmap e193: 8 total, 8 up, 7 in 2024-02-05T08:12:42.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:41 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1296947354' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-02-05T08:12:42.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:41 smithi134 ceph-mon[39278]: osdmap e193: 8 total, 8 up, 7 in 2024-02-05T08:12:42.029 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:42.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:42.131 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:42.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:42.156 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:42.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:42.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:42.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:42.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:42.327 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:12:42.327 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:12:42.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:42.411 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:42.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:42.513 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:42.539 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:42.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:42.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:42.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:42.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:42.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:42.768 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:42.804 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:42.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:42.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:42.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:42.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:42.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:43.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:42 smithi196 ceph-mon[33018]: pgmap v334: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 143 active+clean; 7.0 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 34 MiB/s wr, 539 op/s; 33266/346629 objects misplaced (9.597%); 14 MiB/s, 219 objects/s recovering 2024-02-05T08:12:43.016 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:42 smithi196 ceph-mon[33018]: osdmap e194: 8 total, 8 up, 7 in 2024-02-05T08:12:43.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:42 smithi134 ceph-mon[30086]: pgmap v334: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 143 active+clean; 7.0 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 34 MiB/s wr, 539 op/s; 33266/346629 objects misplaced (9.597%); 14 MiB/s, 219 objects/s recovering 2024-02-05T08:12:43.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:42 smithi134 ceph-mon[30086]: osdmap e194: 8 total, 8 up, 7 in 2024-02-05T08:12:43.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:42 smithi134 ceph-mon[39278]: pgmap v334: 148 pgs: 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 143 active+clean; 7.0 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 34 MiB/s wr, 539 op/s; 33266/346629 objects misplaced (9.597%); 14 MiB/s, 219 objects/s recovering 2024-02-05T08:12:43.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:42 smithi134 ceph-mon[39278]: osdmap e194: 8 total, 8 up, 7 in 2024-02-05T08:12:43.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:43.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T08:12:43.077 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:43.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:43.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:43.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:43.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:43.233 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:43.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:43.334 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:43.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:43.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:43.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:43.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:43.505 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:43.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:43.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:12:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:12:43.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:43.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:43.643 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:43.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:43.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:43.776 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:43.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:43.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:43.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[30086]: osdmap e195: 8 total, 8 up, 7 in 2024-02-05T08:12:43.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[30086]: 8.0 starting backfill to osd.4 from (0'0,0'0] MAX to 192'7387 2024-02-05T08:12:43.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[30086]: 8.f starting backfill to osd.4 from (0'0,0'0] MAX to 192'7409 2024-02-05T08:12:43.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[30086]: 8.7 continuing backfill to osd.6 from (168'2400,192'7231] 8:e2c5bfd9:::benchmark_data_smithi134_115836_object37821:head to 192'7231 2024-02-05T08:12:43.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[30086]: 8.8 starting backfill to osd.0 from (0'0,0'0] MAX to 192'7331 2024-02-05T08:12:43.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[30086]: 8.c starting backfill to osd.5 from (0'0,0'0] MAX to 192'7301 2024-02-05T08:12:43.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[30086]: 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 192'7298 2024-02-05T08:12:43.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:43.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[39278]: osdmap e195: 8 total, 8 up, 7 in 2024-02-05T08:12:43.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[39278]: 8.0 starting backfill to osd.4 from (0'0,0'0] MAX to 192'7387 2024-02-05T08:12:43.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[39278]: 8.f starting backfill to osd.4 from (0'0,0'0] MAX to 192'7409 2024-02-05T08:12:43.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[39278]: 8.7 continuing backfill to osd.6 from (168'2400,192'7231] 8:e2c5bfd9:::benchmark_data_smithi134_115836_object37821:head to 192'7231 2024-02-05T08:12:43.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[39278]: 8.8 starting backfill to osd.0 from (0'0,0'0] MAX to 192'7331 2024-02-05T08:12:43.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[39278]: 8.c starting backfill to osd.5 from (0'0,0'0] MAX to 192'7301 2024-02-05T08:12:43.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[39278]: 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 192'7298 2024-02-05T08:12:43.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:43 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:43.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:43.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:43.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:43.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:44.011 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:44.053 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:44.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:43 smithi196 ceph-mon[33018]: osdmap e195: 8 total, 8 up, 7 in 2024-02-05T08:12:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:43 smithi196 ceph-mon[33018]: 8.0 starting backfill to osd.4 from (0'0,0'0] MAX to 192'7387 2024-02-05T08:12:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:43 smithi196 ceph-mon[33018]: 8.f starting backfill to osd.4 from (0'0,0'0] MAX to 192'7409 2024-02-05T08:12:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:43 smithi196 ceph-mon[33018]: 8.7 continuing backfill to osd.6 from (168'2400,192'7231] 8:e2c5bfd9:::benchmark_data_smithi134_115836_object37821:head to 192'7231 2024-02-05T08:12:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:43 smithi196 ceph-mon[33018]: 8.8 starting backfill to osd.0 from (0'0,0'0] MAX to 192'7331 2024-02-05T08:12:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:43 smithi196 ceph-mon[33018]: 8.c starting backfill to osd.5 from (0'0,0'0] MAX to 192'7301 2024-02-05T08:12:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:43 smithi196 ceph-mon[33018]: 8.b starting backfill to osd.2 from (0'0,0'0] MAX to 192'7298 2024-02-05T08:12:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:43 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:44.154 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:44.202 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:44.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:44.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:44.337 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:44.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:44.438 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:44.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:44.479 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:44.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:44.532 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:44.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:44.611 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:44.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:44.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:44.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:44.775 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:44.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:44.877 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:44.933 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:44.933 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:44 smithi134 ceph-mon[30086]: pgmap v337: 148 pgs: 3 remapped+peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 138 active+clean; 7.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 48 MiB/s wr, 965 op/s; 43897/346920 objects misplaced (12.653%); 58 MiB/s, 934 objects/s recovering 2024-02-05T08:12:44.934 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:44 smithi134 ceph-mon[30086]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:12:44.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:44 smithi134 ceph-mon[39278]: pgmap v337: 148 pgs: 3 remapped+peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 138 active+clean; 7.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 48 MiB/s wr, 965 op/s; 43897/346920 objects misplaced (12.653%); 58 MiB/s, 934 objects/s recovering 2024-02-05T08:12:44.934 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:44 smithi134 ceph-mon[39278]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:12:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:44.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:45.036 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:44 smithi196 ceph-mon[33018]: pgmap v337: 148 pgs: 3 remapped+peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 138 active+clean; 7.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 48 MiB/s wr, 965 op/s; 43897/346920 objects misplaced (12.653%); 58 MiB/s, 934 objects/s recovering 2024-02-05T08:12:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:44 smithi196 ceph-mon[33018]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:12:45.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:45.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:45.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:45.212 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:45.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:45.314 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:45.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:45.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:45.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:45.366 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:45.449 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:45.497 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:45.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:45.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:45.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:45.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:45.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:45.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:45.754 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:12:45.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:45.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:45.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:45.774 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:45 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2063751410' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:12:45.774 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:45 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:12:45.775 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:45 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2063751410' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:12:45.775 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:45 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:12:45.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:45.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:45.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:45.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:46.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:46.043 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:46.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:46.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:45 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2063751410' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:12:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:45 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:12:46.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:46.179 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:46.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:46.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:46.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:46.313 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:46.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:46.415 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:46.477 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:46.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:46.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:46.619 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:46.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:46.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:46.762 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:46.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:46.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:46.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:46.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:46.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:46.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:47.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:46 smithi134 ceph-mon[30086]: pgmap v338: 148 pgs: 3 remapped+peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 138 active+clean; 7.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 14 MiB/s wr, 421 op/s; 43897/346920 objects misplaced (12.653%); 39 MiB/s, 641 objects/s recovering 2024-02-05T08:12:47.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:46 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:12:47.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:46 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:12:47.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:46 smithi134 ceph-mon[30086]: osdmap e196: 8 total, 8 up, 7 in 2024-02-05T08:12:47.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:46 smithi134 ceph-mon[39278]: pgmap v338: 148 pgs: 3 remapped+peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 138 active+clean; 7.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 14 MiB/s wr, 421 op/s; 43897/346920 objects misplaced (12.653%); 39 MiB/s, 641 objects/s recovering 2024-02-05T08:12:47.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:46 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:12:47.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:46 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:12:47.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:46 smithi134 ceph-mon[39278]: osdmap e196: 8 total, 8 up, 7 in 2024-02-05T08:12:47.031 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:47.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:47.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:46 smithi196 ceph-mon[33018]: pgmap v338: 148 pgs: 3 remapped+peering, 4 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 138 active+clean; 7.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 14 MiB/s wr, 421 op/s; 43897/346920 objects misplaced (12.653%); 39 MiB/s, 641 objects/s recovering 2024-02-05T08:12:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:46 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:12:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:46 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:12:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:46 smithi196 ceph-mon[33018]: osdmap e196: 8 total, 8 up, 7 in 2024-02-05T08:12:47.133 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:47.180 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:47.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:47.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:47.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:47.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:47.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:47.463 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:47.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:47.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:47.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:47.600 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:47.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:47.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:47.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:47.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:47.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:47.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:47.860 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:47.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:47.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:47.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:47.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:47.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:48.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:47 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:12:48.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:47 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:12:48.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:12:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:12:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:12:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:47 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:12:48.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:48.143 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:48.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:48.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:48.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:48.276 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:48.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:48.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:48.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:48.416 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:48.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:48.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:48.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:48.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:48.583 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:12:48.583 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:12:48.583 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-02-05T08:12:48.583 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-02-05T08:12:48.584 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:12:48.643 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:48.686 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:48.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:48.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:48.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:48.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:48.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:48.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:48.947 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:48.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:48.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:49.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:48 smithi134 ceph-mon[30086]: pgmap v340: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 511 B/s rd, 14 MiB/s wr, 703 op/s; 63474/341856 objects misplaced (18.567%); 52 MiB/s, 845 objects/s recovering 2024-02-05T08:12:49.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:48 smithi134 ceph-mon[39278]: pgmap v340: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 511 B/s rd, 14 MiB/s wr, 703 op/s; 63474/341856 objects misplaced (18.567%); 52 MiB/s, 845 objects/s recovering 2024-02-05T08:12:49.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:49.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:48 smithi196 ceph-mon[33018]: pgmap v340: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 511 B/s rd, 14 MiB/s wr, 703 op/s; 63474/341856 objects misplaced (18.567%); 52 MiB/s, 845 objects/s recovering 2024-02-05T08:12:49.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:49.085 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:49.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:49.186 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:49.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:49.227 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:49.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:49.328 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:49.387 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:49.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:49.488 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:49.545 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:49.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:49.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:49.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:49.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:49.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:49.809 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:49.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:49.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:49.911 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:49.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:49.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:49.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:50.045 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:49 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:12:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:49 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:12:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:49 smithi196 ceph-mon[33018]: osdmap e197: 8 total, 8 up, 7 in 2024-02-05T08:12:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:49 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:12:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:49 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:12:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:49 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:12:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:49 smithi196 ceph-mon[33018]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:12:50.071 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:50.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:50.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:50.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:50.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:50.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:12:50.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:12:50.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[30086]: osdmap e197: 8 total, 8 up, 7 in 2024-02-05T08:12:50.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:12:50.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:12:50.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:12:50.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[30086]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:12:50.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:12:50.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:12:50.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[39278]: osdmap e197: 8 total, 8 up, 7 in 2024-02-05T08:12:50.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:12:50.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:12:50.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:12:50.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:49 smithi134 ceph-mon[39278]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:12:50.307 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:50.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:50.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:50.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:50.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:50.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:50.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:50.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:50.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:50.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:50.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:50.739 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:50.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:50.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:50.873 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:50.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:50.974 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:51.013 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:51.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:50 smithi196 ceph-mon[33018]: pgmap v342: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 589 B/s rd, 6.8 MiB/s wr, 881 op/s; 59442/333291 objects misplaced (17.835%); 66 MiB/s, 1.06k objects/s recovering 2024-02-05T08:12:51.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:51.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:51.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:51.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:51.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:50 smithi134 ceph-mon[30086]: pgmap v342: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 589 B/s rd, 6.8 MiB/s wr, 881 op/s; 59442/333291 objects misplaced (17.835%); 66 MiB/s, 1.06k objects/s recovering 2024-02-05T08:12:51.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:50 smithi134 ceph-mon[39278]: pgmap v342: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 589 B/s rd, 6.8 MiB/s wr, 881 op/s; 59442/333291 objects misplaced (17.835%); 66 MiB/s, 1.06k objects/s recovering 2024-02-05T08:12:51.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:51.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:51.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:51.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:51.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:51.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:51.518 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:51.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:51.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:51.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:51.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:51.686 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:51.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:51.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:51.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:51.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:51.826 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:51.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:51.927 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:51.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:51.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:51.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:52.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:52.094 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:52.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:52.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:52.196 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:52.240 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:52.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:52.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:52.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:52.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:52.390 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:52.473 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:12:52.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:52.508 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:52.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:52.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:52.522 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:12:52.565 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:12:52.608 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:12:52.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:52.642 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:52.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:52.645 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:12:52.688 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:12:52.728 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:12:52.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:52.769 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:12:52.774 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:52.774 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:12:52.775 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:12:52.775 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:12:52.775 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:12:52.775 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:12:52.775 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:12:52.775 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:12:52.775 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:12:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:52.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:52.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:52.916 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:52.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:52.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:53.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:52 smithi134 ceph-mon[30086]: pgmap v343: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 568 op/s; 59442/333291 objects misplaced (17.835%); 28 MiB/s, 442 objects/s recovering 2024-02-05T08:12:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:52 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:12:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:52 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2922935691' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:53.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:52 smithi134 ceph-mon[39278]: pgmap v343: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 568 op/s; 59442/333291 objects misplaced (17.835%); 28 MiB/s, 442 objects/s recovering 2024-02-05T08:12:53.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:52 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:12:53.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:52 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2922935691' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:53.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:53.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:52 smithi196 ceph-mon[33018]: pgmap v343: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 568 op/s; 59442/333291 objects misplaced (17.835%); 28 MiB/s, 442 objects/s recovering 2024-02-05T08:12:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:52 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:12:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:52 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2922935691' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:12:53.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:53.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:53.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:53.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:53.466 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:53.505 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:53.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:53.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:53.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:12:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:12:53.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:53.657 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:53.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:53.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:53.759 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:53.814 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:53.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:53.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:53.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:53.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:53.952 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:53.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:54.049 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:12:54.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:54.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:53 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:54.085 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:54.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:54.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:54.095 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:53 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:54.095 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:53 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:12:54.187 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:54.236 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:54.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:54.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:54.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:54.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:54.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:54.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:54.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:54.509 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:54.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:54.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:54.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:54.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:54.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:54.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:54.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:54.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:54.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:54.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:54.912 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:54.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:55.015 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:55.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:54 smithi134 ceph-mon[30086]: pgmap v344: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 53366/320322 objects misplaced (16.660%); 57 MiB/s, 917 objects/s recovering 2024-02-05T08:12:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:54 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:12:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:54 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:12:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:54 smithi134 ceph-mon[39278]: pgmap v344: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 53366/320322 objects misplaced (16.660%); 57 MiB/s, 917 objects/s recovering 2024-02-05T08:12:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:54 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:12:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:54 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:12:55.054 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:55.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:54 smithi196 ceph-mon[33018]: pgmap v344: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 53366/320322 objects misplaced (16.660%); 57 MiB/s, 917 objects/s recovering 2024-02-05T08:12:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:54 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:12:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:54 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:12:55.155 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:55.191 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:55.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:55.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:55.337 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:55.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:55.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:55.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:55.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:55.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:55.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:55.612 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:55.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:55.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:55.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:55.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:55.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:55.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:55.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:55.906 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:55.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:56.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:56.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:56.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:56.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:56.104 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:55 smithi134 ceph-mon[30086]: osdmap e198: 8 total, 8 up, 7 in 2024-02-05T08:12:56.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:55 smithi134 ceph-mon[39278]: osdmap e198: 8 total, 8 up, 7 in 2024-02-05T08:12:56.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:12:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:56.203 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:56.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:12:56.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:55 smithi196 ceph-mon[33018]: osdmap e198: 8 total, 8 up, 7 in 2024-02-05T08:12:56.218 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:12:56.218 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:12:56.218 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.218 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:129835 8.e 8:7a77a78e:::benchmark_data_smithi134_115836_object13175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:12:56.218 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:56.212516+0000", 2024-02-05T08:12:56.218 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0048601909999999998, 2024-02-05T08:12:56.218 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0048918679999999997, 2024-02-05T08:12:56.219 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:12:56.219 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:12:56.219 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:12:56.219 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:12:56.219 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:56.219 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 129835 2024-02-05T08:12:56.219 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.219 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:12:56.219 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.219 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:12:56.219 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212516+0000", 2024-02-05T08:12:56.220 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:12:56.220 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.220 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.220 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:12:56.220 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212516+0000", 2024-02-05T08:12:56.220 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:12:56.220 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.220 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.220 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:12:56.220 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212517+0000", 2024-02-05T08:12:56.220 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4929999999999999e-06 2024-02-05T08:12:56.220 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212523+0000", 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5219999999999998e-06 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212535+0000", 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2622999999999999e-05 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212544+0000", 2024-02-05T08:12:56.221 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5459999999999992e-06 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.216404+0000", 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0038603079999999998 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.216539+0000", 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000134714 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-02-05T08:12:56.222 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.216664+0000", 2024-02-05T08:12:56.223 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012543499999999999 2024-02-05T08:12:56.223 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:12:56.223 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:12:56.223 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:12:56.223 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.223 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.223 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:129837 8.c 8:3f7e3bad:::benchmark_data_smithi134_115836_object13177:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:12:56.223 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:56.212560+0000", 2024-02-05T08:12:56.223 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0048152430000000003, 2024-02-05T08:12:56.223 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0049327490000000002, 2024-02-05T08:12:56.223 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:12:56.223 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:12:56.223 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 129837 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212560+0000", 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212560+0000", 2024-02-05T08:12:56.224 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:12:56.225 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.225 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.225 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:12:56.225 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212561+0000", 2024-02-05T08:12:56.225 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6199999999999998e-07 2024-02-05T08:12:56.225 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.225 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.225 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:12:56.225 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212564+0000", 2024-02-05T08:12:56.225 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5639999999999999e-06 2024-02-05T08:12:56.225 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.225 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.225 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:12:56.226 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212572+0000", 2024-02-05T08:12:56.226 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0160000000000008e-06 2024-02-05T08:12:56.226 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.226 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.226 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:12:56.226 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212578+0000", 2024-02-05T08:12:56.226 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9970000000000004e-06 2024-02-05T08:12:56.226 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.226 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.226 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:12:56.226 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212640+0000", 2024-02-05T08:12:56.226 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2891999999999993e-05 2024-02-05T08:12:56.226 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212782+0000", 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014195599999999999 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,5,7", 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.212934+0000", 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015131500000000001 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.227 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15087.0:129836 8.2 e198/157 8:4dbc64d8:::benchmark_data_smithi134_115836_object13176:head v 198'8059, mlcod=198'8057)", 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:56.213187+0000", 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0041888940000000003, 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0043838230000000002, 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.213187+0000", 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:12:56.228 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.213187+0000", 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.213188+0000", 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5199999999999996e-07 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.213194+0000", 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.866e-06 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:12:56.230 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.213204+0000", 2024-02-05T08:12:56.230 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8239999999999995e-06 2024-02-05T08:12:56.230 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.230 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.230 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:12:56.230 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.213211+0000", 2024-02-05T08:12:56.230 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2099999999999996e-06 2024-02-05T08:12:56.230 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.230 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.230 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:12:56.230 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.213280+0000", 2024-02-05T08:12:56.230 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9023000000000006e-05 2024-02-05T08:12:56.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.231 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.231 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:12:56.231 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.213300+0000", 2024-02-05T08:12:56.231 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9726999999999999e-05 2024-02-05T08:12:56.231 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:12:56.231 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:12:56.231 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:12:56.231 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.231 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.231 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15087.0:129839 8.8 e198/194)", 2024-02-05T08:12:56.231 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:56.215869+0000", 2024-02-05T08:12:56.231 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.001506472, 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001758297, 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.215869+0000", 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.215869+0000", 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:12:56.232 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.233 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.233 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:12:56.233 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.215870+0000", 2024-02-05T08:12:56.233 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7199999999999997e-07 2024-02-05T08:12:56.233 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.233 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.233 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:12:56.233 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.215878+0000", 2024-02-05T08:12:56.233 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.926e-06 2024-02-05T08:12:56.233 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.233 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.233 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:12:56.234 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.215894+0000", 2024-02-05T08:12:56.234 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6194000000000001e-05 2024-02-05T08:12:56.234 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:12:56.234 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:12:56.234 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:12:56.234 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:56.215903+0000", 2024-02-05T08:12:56.234 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4060000000000005e-06 2024-02-05T08:12:56.234 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:12:56.234 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:12:56.234 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:12:56.234 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:12:56.234 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:12:56.234 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 4 2024-02-05T08:12:56.235 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:12:56.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:56.337 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:56.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:56.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:56.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:56.473 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:56.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:56.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:56.575 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:56.618 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:56.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:56.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:56.722 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:12:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:56.772 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:56.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:56.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:56.904 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:56.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:56.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:56.949 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:12:57.006 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:57.031 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:56 smithi134 ceph-mon[30086]: pgmap v346: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 897 op/s; 53366/320322 objects misplaced (16.660%); 48 MiB/s, 765 objects/s recovering 2024-02-05T08:12:57.031 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:56 smithi134 ceph-mon[39278]: pgmap v346: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 897 op/s; 53366/320322 objects misplaced (16.660%); 48 MiB/s, 765 objects/s recovering 2024-02-05T08:12:57.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:57.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:57.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:57.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:57.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:57.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:57.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:57.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:57.303 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:57.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:57.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:57.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:56 smithi196 ceph-mon[33018]: pgmap v346: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 897 op/s; 53366/320322 objects misplaced (16.660%); 48 MiB/s, 765 objects/s recovering 2024-02-05T08:12:57.360 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:12:57.371 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:12:57.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T08:12:57.405 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:12:57.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:57.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:57.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:12:57.594 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:57.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:57.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:57.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:57.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:57.762 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:57.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:57.863 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:57.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:57.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:57.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:57.951 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:57 smithi134 ceph-mon[30086]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:12:57.952 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:12:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:12:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:12:57.952 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:57 smithi134 ceph-mon[39278]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:12:58.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:58.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:58.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:58.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:58.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:58.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:58.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:58.305 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:58.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:58.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:57 smithi196 ceph-mon[33018]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:12:58.406 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:58.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:58.450 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:58.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:58.551 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:58.589 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:58.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:58.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:58.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:58.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:58.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:58.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:58.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:12:58.850 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:58.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:12:58.952 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:58.985 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:58.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:58.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:59.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:12:59.117 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:59.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:59.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:12:59.219 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:12:59.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:59.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:12:59.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:12:58 smithi134 ceph-mon[30086]: pgmap v347: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 679 op/s; 51193/315243 objects misplaced (16.239%); 37 MiB/s, 594 objects/s recovering 2024-02-05T08:12:59.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:12:58 smithi134 ceph-mon[39278]: pgmap v347: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 679 op/s; 51193/315243 objects misplaced (16.239%); 37 MiB/s, 594 objects/s recovering 2024-02-05T08:12:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:12:58 smithi196 ceph-mon[33018]: pgmap v347: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 679 op/s; 51193/315243 objects misplaced (16.239%); 37 MiB/s, 594 objects/s recovering 2024-02-05T08:12:59.354 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:59.384 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:59.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:59.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:59.486 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:12:59.517 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:12:59.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:59.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:12:59.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:12:59.660 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:59.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:59.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:12:59.762 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:59.799 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:59.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:12:59.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:12:59.925 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:12:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:12:59.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:00.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:00.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:00.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:00.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:00.202 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:00.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:00.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:00.327 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:00.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:00.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:00.468 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:00.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:00.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:00.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:00.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:00.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:00.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:00.699 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:00.742 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:00.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:00.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:00.844 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:00.867 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:00.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:00.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:01.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:01.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:01.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:01.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:01.136 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:01.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:01.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:01.271 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:01.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:01.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:01.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:00 smithi134 ceph-mon[30086]: pgmap v348: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 891 op/s; 47316/306570 objects misplaced (15.434%); 48 MiB/s, 763 objects/s recovering 2024-02-05T08:13:01.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:00 smithi134 ceph-mon[39278]: pgmap v348: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 891 op/s; 47316/306570 objects misplaced (15.434%); 48 MiB/s, 763 objects/s recovering 2024-02-05T08:13:01.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:00 smithi196 ceph-mon[33018]: pgmap v348: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 891 op/s; 47316/306570 objects misplaced (15.434%); 48 MiB/s, 763 objects/s recovering 2024-02-05T08:13:01.373 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:01.395 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:01.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:01.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:01.497 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:01.537 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:01.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:01.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:01.639 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:01.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:01.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:01.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:01.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:01.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:01.816 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:01.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:01.917 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:01.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:01.961 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:01.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:02.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:02.107 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:02.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:02.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:02.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:02.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:02.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:01 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:13:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:01 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:13:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:01 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:13:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:01 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:13:02.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:01 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:13:02.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:01 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:13:02.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:01 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:13:02.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:01 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:13:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:01 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:13:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:01 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:13:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:01 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:13:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:01 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:13:02.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:02.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:02.380 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:02.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:02.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:02.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:02.519 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:02.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:02.620 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:02.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:02.644 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:02.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:02.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:02.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:02.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:02.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:02.882 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:02.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:02.925 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:02.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:03.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:03.050 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:03.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:03.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:03.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:03.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:03.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:03.228 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:02 smithi134 ceph-mon[30086]: pgmap v349: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 891 op/s; 47316/306570 objects misplaced (15.434%); 48 MiB/s, 763 objects/s recovering 2024-02-05T08:13:03.229 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:02 smithi134 ceph-mon[30086]: osdmap e199: 8 total, 8 up, 7 in 2024-02-05T08:13:03.229 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:02 smithi134 ceph-mon[39278]: pgmap v349: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 891 op/s; 47316/306570 objects misplaced (15.434%); 48 MiB/s, 763 objects/s recovering 2024-02-05T08:13:03.229 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:02 smithi134 ceph-mon[39278]: osdmap e199: 8 total, 8 up, 7 in 2024-02-05T08:13:03.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:03.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:02 smithi196 ceph-mon[33018]: pgmap v349: 148 pgs: 5 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 138 active+clean; 6.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 891 op/s; 47316/306570 objects misplaced (15.434%); 48 MiB/s, 763 objects/s recovering 2024-02-05T08:13:03.310 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:02 smithi196 ceph-mon[33018]: osdmap e199: 8 total, 8 up, 7 in 2024-02-05T08:13:03.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:03.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:03.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:03.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:03.460 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:03.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:03.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:03.562 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:03.565 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:13:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:13:03.593 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:03.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:03.599 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-02-05T08:12:55.655295+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6985095540,"num_objects":106774,"num_object_clones":0,"num_object_copies":320322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":53366,"num_objects_unfound":0,"num_objects_dirty":106774,"num_whiteouts":0,"num_read":1409,"num_read_kb":1255,"num_write":127355,"num_write_kb":7468510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13910,"num_bytes_recovered":898769159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76567,"ondisk_log_size":76567,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":447,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":19912644,"kb_used_data":18373900,"kb_used_omap":10,"kb_used_meta":1538421,"kb_avail":636303420,"statfs":{"total":671965249536,"available":651574702080,"internally_reserved":0,"allocated":18814873600,"data_stored":18812556095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":1575343210},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-470417408,"num_objects":-7178,"num_object_clones":0,"num_object_copies":-21534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-10108,"num_objects_unfound":0,"num_objects_dirty":-7178,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":7178,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6125,"num_bytes_recovered":401408000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002728"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":293,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.783998+0000","last_change":"2024-02-05T08:12:32.976357+0000","last_active":"2024-02-05T08:12:48.783998+0000","last_peered":"2024-02-05T08:12:48.783998+0000","last_clean":"2024-02-05T08:12:48.783998+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:12:48.783998+0000","last_undegraded":"2024-02-05T08:12:48.783998+0000","last_fullsized":"2024-02-05T08:12:48.783998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:32.976264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:32.976264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:39.644642+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.871514+0000","last_change":"2024-02-05T08:11:17.826990+0000","last_active":"2024-02-05T08:12:48.871514+0000","last_peered":"2024-02-05T08:12:48.871514+0000","last_clean":"2024-02-05T08:12:48.871514+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:12:48.871514+0000","last_undegraded":"2024-02-05T08:12:48.871514+0000","last_fullsized":"2024-02-05T08:12:48.871514+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:17.824305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:17.824305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":297,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.869127+0000","last_change":"2024-02-05T08:10:18.254802+0000","last_active":"2024-02-05T08:12:48.869127+0000","last_peered":"2024-02-05T08:12:48.869127+0000","last_clean":"2024-02-05T08:12:48.869127+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:12:48.869127+0000","last_undegraded":"2024-02-05T08:12:48.869127+0000","last_fullsized":"2024-02-05T08:12:48.869127+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.254568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.254568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"90'13","reported_seq":345,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.783367+0000","last_change":"2024-02-05T08:12:29.723024+0000","last_active":"2024-02-05T08:12:48.783367+0000","last_peered":"2024-02-05T08:12:48.783367+0000","last_clean":"2024-02-05T08:12:48.783367+0000","last_became_active":"2024-02-05T08:12:29.675863+0000","last_became_peered":"2024-02-05T08:12:29.675863+0000","last_unstale":"2024-02-05T08:12:48.783367+0000","last_undegraded":"2024-02-05T08:12:48.783367+0000","last_fullsized":"2024-02-05T08:12:48.783367+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:11:06.482560+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:11:06.482560+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":288,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790183+0000","last_change":"2024-02-05T08:12:41.723242+0000","last_active":"2024-02-05T08:12:48.790183+0000","last_peered":"2024-02-05T08:12:48.790183+0000","last_clean":"2024-02-05T08:12:48.790183+0000","last_became_active":"2024-02-05T08:12:41.722414+0000","last_became_peered":"2024-02-05T08:12:41.722414+0000","last_unstale":"2024-02-05T08:12:48.790183+0000","last_undegraded":"2024-02-05T08:12:48.790183+0000","last_fullsized":"2024-02-05T08:12:48.790183+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:30.552741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:30.552741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:54.257872+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":301,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.783651+0000","last_change":"2024-02-05T08:12:29.676957+0000","last_active":"2024-02-05T08:12:48.783651+0000","last_peered":"2024-02-05T08:12:48.783651+0000","last_clean":"2024-02-05T08:12:48.783651+0000","last_became_active":"2024-02-05T08:12:29.676482+0000","last_became_peered":"2024-02-05T08:12:29.676482+0000","last_unstale":"2024-02-05T08:12:48.783651+0000","last_undegraded":"2024-02-05T08:12:48.783651+0000","last_fullsized":"2024-02-05T08:12:48.783651+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:11:19.262611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.794124+0000","last_change":"2024-02-05T08:12:41.727776+0000","last_active":"2024-02-05T08:12:48.794124+0000","last_peered":"2024-02-05T08:12:48.794124+0000","last_clean":"2024-02-05T08:12:48.794124+0000","last_became_active":"2024-02-05T08:12:41.727450+0000","last_became_peered":"2024-02-05T08:12:41.727450+0000","last_unstale":"2024-02-05T08:12:48.794124+0000","last_undegraded":"2024-02-05T08:12:48.794124+0000","last_fullsized":"2024-02-05T08:12:48.794124+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:28.007713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:28.007713+0000","last_clean_scrub_stamp":"2024-02-05T08:10:28.007713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"118'22","reported_seq":179,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.870567+0000","last_change":"2024-02-05T08:12:41.771457+0000","last_active":"2024-02-05T08:12:48.870567+0000","last_peered":"2024-02-05T08:12:48.870567+0000","last_clean":"2024-02-05T08:12:48.870567+0000","last_became_active":"2024-02-05T08:12:41.745917+0000","last_became_peered":"2024-02-05T08:12:41.745917+0000","last_unstale":"2024-02-05T08:12:48.870567+0000","last_undegraded":"2024-02-05T08:12:48.870567+0000","last_fullsized":"2024-02-05T08:12:48.870567+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-02-05T08:11:07.726635+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.726635+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":285,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.867630+0000","last_change":"2024-02-05T08:10:29.223232+0000","last_active":"2024-02-05T08:12:48.867630+0000","last_peered":"2024-02-05T08:12:48.867630+0000","last_clean":"2024-02-05T08:12:48.867630+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:12:48.867630+0000","last_undegraded":"2024-02-05T08:12:48.867630+0000","last_fullsized":"2024-02-05T08:12:48.867630+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:29.223145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:29.223145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":297,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.787064+0000","last_change":"2024-02-05T08:12:15.282974+0000","last_active":"2024-02-05T08:12:48.787064+0000","last_peered":"2024-02-05T08:12:48.787064+0000","last_clean":"2024-02-05T08:12:48.787064+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:12:48.787064+0000","last_undegraded":"2024-02-05T08:12:48.787064+0000","last_fullsized":"2024-02-05T08:12:48.787064+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:15.282863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:12:15.282863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:25.141357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":317,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.787766+0000","last_change":"2024-02-05T08:12:29.684488+0000","last_active":"2024-02-05T08:12:48.787766+0000","last_peered":"2024-02-05T08:12:48.787766+0000","last_clean":"2024-02-05T08:12:48.787766+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:12:48.787766+0000","last_undegraded":"2024-02-05T08:12:48.787766+0000","last_fullsized":"2024-02-05T08:12:48.787766+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:20.316515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:20.316515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":331,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.784306+0000","last_change":"2024-02-05T08:12:41.731788+0000","last_active":"2024-02-05T08:12:48.784306+0000","last_peered":"2024-02-05T08:12:48.784306+0000","last_clean":"2024-02-05T08:12:48.784306+0000","last_became_active":"2024-02-05T08:12:41.731452+0000","last_became_peered":"2024-02-05T08:12:41.731452+0000","last_unstale":"2024-02-05T08:12:48.784306+0000","last_undegraded":"2024-02-05T08:12:48.784306+0000","last_fullsized":"2024-02-05T08:12:48.784306+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:26.162293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:26.162293+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.786791+0000","last_change":"2024-02-05T08:12:41.722105+0000","last_active":"2024-02-05T08:12:48.786791+0000","last_peered":"2024-02-05T08:12:48.786791+0000","last_clean":"2024-02-05T08:12:48.786791+0000","last_became_active":"2024-02-05T08:12:41.721611+0000","last_became_peered":"2024-02-05T08:12:41.721611+0000","last_unstale":"2024-02-05T08:12:48.786791+0000","last_undegraded":"2024-02-05T08:12:48.786791+0000","last_fullsized":"2024-02-05T08:12:48.786791+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:14.301656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:14.301656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:36.614851+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790763+0000","last_change":"2024-02-05T08:12:23.136907+0000","last_active":"2024-02-05T08:12:48.790763+0000","last_peered":"2024-02-05T08:12:48.790763+0000","last_clean":"2024-02-05T08:12:48.790763+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:12:48.790763+0000","last_undegraded":"2024-02-05T08:12:48.790763+0000","last_fullsized":"2024-02-05T08:12:48.790763+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:23.136787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:23.136787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:49.647033+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":305,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.787783+0000","last_change":"2024-02-05T08:12:12.261136+0000","last_active":"2024-02-05T08:12:48.787783+0000","last_peered":"2024-02-05T08:12:48.787783+0000","last_clean":"2024-02-05T08:12:48.787783+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:12:48.787783+0000","last_undegraded":"2024-02-05T08:12:48.787783+0000","last_fullsized":"2024-02-05T08:12:48.787783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:12.261000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:12:12.261000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:16.750433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":315,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.784154+0000","last_change":"2024-02-05T08:12:41.747033+0000","last_active":"2024-02-05T08:12:48.784154+0000","last_peered":"2024-02-05T08:12:48.784154+0000","last_clean":"2024-02-05T08:12:48.784154+0000","last_became_active":"2024-02-05T08:12:41.730485+0000","last_became_peered":"2024-02-05T08:12:41.730485+0000","last_unstale":"2024-02-05T08:12:48.784154+0000","last_undegraded":"2024-02-05T08:12:48.784154+0000","last_fullsized":"2024-02-05T08:12:48.784154+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:11:33.210478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:33.210478+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":306,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.784333+0000","last_change":"2024-02-05T08:12:29.675196+0000","last_active":"2024-02-05T08:12:48.784333+0000","last_peered":"2024-02-05T08:12:48.784333+0000","last_clean":"2024-02-05T08:12:48.784333+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:12:48.784333+0000","last_undegraded":"2024-02-05T08:12:48.784333+0000","last_fullsized":"2024-02-05T08:12:48.784333+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.904127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.904127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:34.241676+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":290,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.869525+0000","last_change":"2024-02-05T08:11:06.715512+0000","last_active":"2024-02-05T08:12:48.869525+0000","last_peered":"2024-02-05T08:12:48.869525+0000","last_clean":"2024-02-05T08:12:48.869525+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:12:48.869525+0000","last_undegraded":"2024-02-05T08:12:48.869525+0000","last_fullsized":"2024-02-05T08:12:48.869525+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:06.715401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:11:06.715401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":297,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.868476+0000","last_change":"2024-02-05T08:11:22.303481+0000","last_active":"2024-02-05T08:12:48.868476+0000","last_peered":"2024-02-05T08:12:48.868476+0000","last_clean":"2024-02-05T08:12:48.868476+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:12:48.868476+0000","last_undegraded":"2024-02-05T08:12:48.868476+0000","last_fullsized":"2024-02-05T08:12:48.868476+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:22.303363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:22.303363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":357,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.869732+0000","last_change":"2024-02-05T08:12:41.771382+0000","last_active":"2024-02-05T08:12:48.869732+0000","last_peered":"2024-02-05T08:12:48.869732+0000","last_clean":"2024-02-05T08:12:48.869732+0000","last_became_active":"2024-02-05T08:12:41.727027+0000","last_became_peered":"2024-02-05T08:12:41.727027+0000","last_unstale":"2024-02-05T08:12:48.869732+0000","last_undegraded":"2024-02-05T08:12:48.869732+0000","last_fullsized":"2024-02-05T08:12:48.869732+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:11:33.391736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:33.391736+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":288,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.793905+0000","last_change":"2024-02-05T08:12:41.713865+0000","last_active":"2024-02-05T08:12:48.793905+0000","last_peered":"2024-02-05T08:12:48.793905+0000","last_clean":"2024-02-05T08:12:48.793905+0000","last_became_active":"2024-02-05T08:12:41.713564+0000","last_became_peered":"2024-02-05T08:12:41.713564+0000","last_unstale":"2024-02-05T08:12:48.793905+0000","last_undegraded":"2024-02-05T08:12:48.793905+0000","last_fullsized":"2024-02-05T08:12:48.793905+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:34.947466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:34.947466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.389408+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":303,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.783767+0000","last_change":"2024-02-05T08:12:41.735590+0000","last_active":"2024-02-05T08:12:48.783767+0000","last_peered":"2024-02-05T08:12:48.783767+0000","last_clean":"2024-02-05T08:12:48.783767+0000","last_became_active":"2024-02-05T08:12:41.735240+0000","last_became_peered":"2024-02-05T08:12:41.735240+0000","last_unstale":"2024-02-05T08:12:48.783767+0000","last_undegraded":"2024-02-05T08:12:48.783767+0000","last_fullsized":"2024-02-05T08:12:48.783767+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:31.934762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:12:31.934762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:51.198468+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":297,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.784414+0000","last_change":"2024-02-05T08:12:16.855738+0000","last_active":"2024-02-05T08:12:48.784414+0000","last_peered":"2024-02-05T08:12:48.784414+0000","last_clean":"2024-02-05T08:12:48.784414+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:12:48.784414+0000","last_undegraded":"2024-02-05T08:12:48.784414+0000","last_fullsized":"2024-02-05T08:12:48.784414+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:16.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:16.855647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:42.807659+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"90'17","reported_seq":108,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.869313+0000","last_change":"2024-02-05T08:12:41.756597+0000","last_active":"2024-02-05T08:12:48.869313+0000","last_peered":"2024-02-05T08:12:48.869313+0000","last_clean":"2024-02-05T08:12:48.869313+0000","last_became_active":"2024-02-05T08:12:41.716794+0000","last_became_peered":"2024-02-05T08:12:41.716794+0000","last_unstale":"2024-02-05T08:12:48.869313+0000","last_undegraded":"2024-02-05T08:12:48.869313+0000","last_fullsized":"2024-02-05T08:12:48.869313+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:10:30.977334+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.977334+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":304,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790866+0000","last_change":"2024-02-05T08:12:29.117226+0000","last_active":"2024-02-05T08:12:48.790866+0000","last_peered":"2024-02-05T08:12:48.790866+0000","last_clean":"2024-02-05T08:12:48.790866+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:12:48.790866+0000","last_undegraded":"2024-02-05T08:12:48.790866+0000","last_fullsized":"2024-02-05T08:12:48.790866+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:12:29.117142+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:12:29.117142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:36.497586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":38,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.794201+0000","last_change":"2024-02-05T08:12:29.693990+0000","last_active":"2024-02-05T08:12:48.794201+0000","last_peered":"2024-02-05T08:12:48.794201+0000","last_clean":"2024-02-05T08:12:48.794201+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:12:48.794201+0000","last_undegraded":"2024-02-05T08:12:48.794201+0000","last_fullsized":"2024-02-05T08:12:48.794201+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:11:21.208206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:21.208206+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":309,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.871424+0000","last_change":"2024-02-05T08:12:41.749863+0000","last_active":"2024-02-05T08:12:48.871424+0000","last_peered":"2024-02-05T08:12:48.871424+0000","last_clean":"2024-02-05T08:12:48.871424+0000","last_became_active":"2024-02-05T08:12:41.719010+0000","last_became_peered":"2024-02-05T08:12:41.719010+0000","last_unstale":"2024-02-05T08:12:48.871424+0000","last_undegraded":"2024-02-05T08:12:48.871424+0000","last_fullsized":"2024-02-05T08:12:48.871424+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:10:32.403342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:32.403342+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":363,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.872584+0000","last_change":"2024-02-05T08:12:41.775561+0000","last_active":"2024-02-05T08:12:48.872584+0000","last_peered":"2024-02-05T08:12:48.872584+0000","last_clean":"2024-02-05T08:12:48.872584+0000","last_became_active":"2024-02-05T08:12:41.757718+0000","last_became_peered":"2024-02-05T08:12:41.757718+0000","last_unstale":"2024-02-05T08:12:48.872584+0000","last_undegraded":"2024-02-05T08:12:48.872584+0000","last_fullsized":"2024-02-05T08:12:48.872584+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:12:18.538594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:18.538594+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:23.144318+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":294,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.869654+0000","last_change":"2024-02-05T08:11:33.834287+0000","last_active":"2024-02-05T08:12:48.869654+0000","last_peered":"2024-02-05T08:12:48.869654+0000","last_clean":"2024-02-05T08:12:48.869654+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:12:48.869654+0000","last_undegraded":"2024-02-05T08:12:48.869654+0000","last_fullsized":"2024-02-05T08:12:48.869654+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:33.834034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:33.834034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:52.764333+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":290,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.872828+0000","last_change":"2024-02-05T08:11:25.316698+0000","last_active":"2024-02-05T08:12:48.872828+0000","last_peered":"2024-02-05T08:12:48.872828+0000","last_clean":"2024-02-05T08:12:48.872828+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:12:48.872828+0000","last_undegraded":"2024-02-05T08:12:48.872828+0000","last_fullsized":"2024-02-05T08:12:48.872828+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:25.316598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:25.316598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.867561+0000","last_change":"2024-02-05T08:12:41.727708+0000","last_active":"2024-02-05T08:12:48.867561+0000","last_peered":"2024-02-05T08:12:48.867561+0000","last_clean":"2024-02-05T08:12:48.867561+0000","last_became_active":"2024-02-05T08:12:41.727406+0000","last_became_peered":"2024-02-05T08:12:41.727406+0000","last_unstale":"2024-02-05T08:12:48.867561+0000","last_undegraded":"2024-02-05T08:12:48.867561+0000","last_fullsized":"2024-02-05T08:12:48.867561+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:15.831837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.831837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":41,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.794012+0000","last_change":"2024-02-05T08:12:29.712505+0000","last_active":"2024-02-05T08:12:48.794012+0000","last_peered":"2024-02-05T08:12:48.794012+0000","last_clean":"2024-02-05T08:12:48.794012+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:12:48.794012+0000","last_undegraded":"2024-02-05T08:12:48.794012+0000","last_fullsized":"2024-02-05T08:12:48.794012+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:11:15.397335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.397335+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":286,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790683+0000","last_change":"2024-02-05T08:11:01.465783+0000","last_active":"2024-02-05T08:12:48.790683+0000","last_peered":"2024-02-05T08:12:48.790683+0000","last_clean":"2024-02-05T08:12:48.790683+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:12:48.790683+0000","last_undegraded":"2024-02-05T08:12:48.790683+0000","last_fullsized":"2024-02-05T08:12:48.790683+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:01.465657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:01.465657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.131797+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":29,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.793409+0000","last_change":"2024-02-05T08:12:29.678505+0000","last_active":"2024-02-05T08:12:48.793409+0000","last_peered":"2024-02-05T08:12:48.793409+0000","last_clean":"2024-02-05T08:12:48.793409+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:12:48.793409+0000","last_undegraded":"2024-02-05T08:12:48.793409+0000","last_fullsized":"2024-02-05T08:12:48.793409+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:24.812126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:12:24.812126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:51.087313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":303,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.869337+0000","last_change":"2024-02-05T08:12:41.737429+0000","last_active":"2024-02-05T08:12:48.869337+0000","last_peered":"2024-02-05T08:12:48.869337+0000","last_clean":"2024-02-05T08:12:48.869337+0000","last_became_active":"2024-02-05T08:12:41.736991+0000","last_became_peered":"2024-02-05T08:12:41.736991+0000","last_unstale":"2024-02-05T08:12:48.869337+0000","last_undegraded":"2024-02-05T08:12:48.869337+0000","last_fullsized":"2024-02-05T08:12:48.869337+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:20.294901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:20.294901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":49,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.873596+0000","last_change":"2024-02-05T08:12:41.743649+0000","last_active":"2024-02-05T08:12:48.873596+0000","last_peered":"2024-02-05T08:12:48.873596+0000","last_clean":"2024-02-05T08:12:48.873596+0000","last_became_active":"2024-02-05T08:12:41.717459+0000","last_became_peered":"2024-02-05T08:12:41.717459+0000","last_unstale":"2024-02-05T08:12:48.873596+0000","last_undegraded":"2024-02-05T08:12:48.873596+0000","last_fullsized":"2024-02-05T08:12:48.873596+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-02-05T08:11:32.785856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:32.785856+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":288,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.789894+0000","last_change":"2024-02-05T08:12:41.738206+0000","last_active":"2024-02-05T08:12:48.789894+0000","last_peered":"2024-02-05T08:12:48.789894+0000","last_clean":"2024-02-05T08:12:48.789894+0000","last_became_active":"2024-02-05T08:12:41.737803+0000","last_became_peered":"2024-02-05T08:12:41.737803+0000","last_unstale":"2024-02-05T08:12:48.789894+0000","last_undegraded":"2024-02-05T08:12:48.789894+0000","last_fullsized":"2024-02-05T08:12:48.789894+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:33.610823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:33.610823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:39.697997+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":310,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790534+0000","last_change":"2024-02-05T08:12:29.682425+0000","last_active":"2024-02-05T08:12:48.790534+0000","last_peered":"2024-02-05T08:12:48.790534+0000","last_clean":"2024-02-05T08:12:48.790534+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:12:48.790534+0000","last_undegraded":"2024-02-05T08:12:48.790534+0000","last_fullsized":"2024-02-05T08:12:48.790534+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:16.440585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:11:16.440585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.131657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":305,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.787416+0000","last_change":"2024-02-05T08:11:05.463231+0000","last_active":"2024-02-05T08:12:48.787416+0000","last_peered":"2024-02-05T08:12:48.787416+0000","last_clean":"2024-02-05T08:12:48.787416+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:12:48.787416+0000","last_undegraded":"2024-02-05T08:12:48.787416+0000","last_fullsized":"2024-02-05T08:12:48.787416+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:05.463134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:05.463134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"141'16","reported_seq":242,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.791863+0000","last_change":"2024-02-05T08:12:41.735407+0000","last_active":"2024-02-05T08:12:48.791863+0000","last_peered":"2024-02-05T08:12:48.791863+0000","last_clean":"2024-02-05T08:12:48.791863+0000","last_became_active":"2024-02-05T08:12:41.718651+0000","last_became_peered":"2024-02-05T08:12:41.718651+0000","last_unstale":"2024-02-05T08:12:48.791863+0000","last_undegraded":"2024-02-05T08:12:48.791863+0000","last_fullsized":"2024-02-05T08:12:48.791863+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'16","last_scrub_stamp":"2024-02-05T08:12:17.493187+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:12:17.493187+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:20.359082+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":285,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.789983+0000","last_change":"2024-02-05T08:11:07.248032+0000","last_active":"2024-02-05T08:12:48.789983+0000","last_peered":"2024-02-05T08:12:48.789983+0000","last_clean":"2024-02-05T08:12:48.789983+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:12:48.789983+0000","last_undegraded":"2024-02-05T08:12:48.789983+0000","last_fullsized":"2024-02-05T08:12:48.789983+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:07.247928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.247928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":19,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790049+0000","last_change":"2024-02-05T08:12:41.758242+0000","last_active":"2024-02-05T08:12:48.790049+0000","last_peered":"2024-02-05T08:12:48.790049+0000","last_clean":"2024-02-05T08:12:48.790049+0000","last_became_active":"2024-02-05T08:12:41.757901+0000","last_became_peered":"2024-02-05T08:12:41.757901+0000","last_unstale":"2024-02-05T08:12:48.790049+0000","last_undegraded":"2024-02-05T08:12:48.790049+0000","last_fullsized":"2024-02-05T08:12:48.790049+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:26.987763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:26.987763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":312,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.791167+0000","last_change":"2024-02-05T08:12:41.734449+0000","last_active":"2024-02-05T08:12:48.791167+0000","last_peered":"2024-02-05T08:12:48.791167+0000","last_clean":"2024-02-05T08:12:48.791167+0000","last_became_active":"2024-02-05T08:12:41.734072+0000","last_became_peered":"2024-02-05T08:12:41.734072+0000","last_unstale":"2024-02-05T08:12:48.791167+0000","last_undegraded":"2024-02-05T08:12:48.791167+0000","last_fullsized":"2024-02-05T08:12:48.791167+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:19.108762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:19.108762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:41.388316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":350,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.868489+0000","last_change":"2024-02-05T08:12:29.700891+0000","last_active":"2024-02-05T08:12:48.868489+0000","last_peered":"2024-02-05T08:12:48.868489+0000","last_clean":"2024-02-05T08:12:48.868489+0000","last_became_active":"2024-02-05T08:12:29.676152+0000","last_became_peered":"2024-02-05T08:12:29.676152+0000","last_unstale":"2024-02-05T08:12:48.868489+0000","last_undegraded":"2024-02-05T08:12:48.868489+0000","last_fullsized":"2024-02-05T08:12:48.868489+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:13.793895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:13.793895+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":18,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.868724+0000","last_change":"2024-02-05T08:12:41.742137+0000","last_active":"2024-02-05T08:12:48.868724+0000","last_peered":"2024-02-05T08:12:48.868724+0000","last_clean":"2024-02-05T08:12:48.868724+0000","last_became_active":"2024-02-05T08:12:41.737246+0000","last_became_peered":"2024-02-05T08:12:41.737246+0000","last_unstale":"2024-02-05T08:12:48.868724+0000","last_undegraded":"2024-02-05T08:12:48.868724+0000","last_fullsized":"2024-02-05T08:12:48.868724+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.507800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.507800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:28.052173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":304,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.868312+0000","last_change":"2024-02-05T08:12:41.717743+0000","last_active":"2024-02-05T08:12:48.868312+0000","last_peered":"2024-02-05T08:12:48.868312+0000","last_clean":"2024-02-05T08:12:48.868312+0000","last_became_active":"2024-02-05T08:12:41.717285+0000","last_became_peered":"2024-02-05T08:12:41.717285+0000","last_unstale":"2024-02-05T08:12:48.868312+0000","last_undegraded":"2024-02-05T08:12:48.868312+0000","last_fullsized":"2024-02-05T08:12:48.868312+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:12.952558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:12.952558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:29.694413+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":317,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.783582+0000","last_change":"2024-02-05T08:12:29.677640+0000","last_active":"2024-02-05T08:12:48.783582+0000","last_peered":"2024-02-05T08:12:48.783582+0000","last_clean":"2024-02-05T08:12:48.783582+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:12:48.783582+0000","last_undegraded":"2024-02-05T08:12:48.783582+0000","last_fullsized":"2024-02-05T08:12:48.783582+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:32.160333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.966886+0000","last_clean_scrub_stamp":"2024-02-05T08:11:32.160333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:50.727750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":356,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.784306+0000","last_change":"2024-02-05T08:12:29.939886+0000","last_active":"2024-02-05T08:12:48.784306+0000","last_peered":"2024-02-05T08:12:48.784306+0000","last_clean":"2024-02-05T08:12:48.784306+0000","last_became_active":"2024-02-05T08:12:29.675657+0000","last_became_peered":"2024-02-05T08:12:29.675657+0000","last_unstale":"2024-02-05T08:12:48.784306+0000","last_undegraded":"2024-02-05T08:12:48.784306+0000","last_fullsized":"2024-02-05T08:12:48.784306+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:12:29.939772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:29.939772+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:49.888863+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":291,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.872342+0000","last_change":"2024-02-05T08:12:41.737779+0000","last_active":"2024-02-05T08:12:48.872342+0000","last_peered":"2024-02-05T08:12:48.872342+0000","last_clean":"2024-02-05T08:12:48.872342+0000","last_became_active":"2024-02-05T08:12:41.736845+0000","last_became_peered":"2024-02-05T08:12:41.736845+0000","last_unstale":"2024-02-05T08:12:48.872342+0000","last_undegraded":"2024-02-05T08:12:48.872342+0000","last_fullsized":"2024-02-05T08:12:48.872342+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:31.474957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:31.474957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:54.541375+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":404,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.788039+0000","last_change":"2024-02-05T08:11:32.363524+0000","last_active":"2024-02-05T08:12:48.788039+0000","last_peered":"2024-02-05T08:12:48.788039+0000","last_clean":"2024-02-05T08:12:48.788039+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:12:48.788039+0000","last_undegraded":"2024-02-05T08:12:48.788039+0000","last_fullsized":"2024-02-05T08:12:48.788039+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:11:32.363440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:32.363440+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":306,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790274+0000","last_change":"2024-02-05T08:11:11.422476+0000","last_active":"2024-02-05T08:12:48.790274+0000","last_peered":"2024-02-05T08:12:48.790274+0000","last_clean":"2024-02-05T08:12:48.790274+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:12:48.790274+0000","last_undegraded":"2024-02-05T08:12:48.790274+0000","last_fullsized":"2024-02-05T08:12:48.790274+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:11.422371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:11.422371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:50.065521+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"129'15","reported_seq":187,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.870664+0000","last_change":"2024-02-05T08:12:41.781948+0000","last_active":"2024-02-05T08:12:48.870664+0000","last_peered":"2024-02-05T08:12:48.870664+0000","last_clean":"2024-02-05T08:12:48.870664+0000","last_became_active":"2024-02-05T08:12:41.726861+0000","last_became_peered":"2024-02-05T08:12:41.726861+0000","last_unstale":"2024-02-05T08:12:48.870664+0000","last_undegraded":"2024-02-05T08:12:48.870664+0000","last_fullsized":"2024-02-05T08:12:48.870664+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'15","last_scrub_stamp":"2024-02-05T08:11:30.787149+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:11:30.787149+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:50.487748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":288,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.784208+0000","last_change":"2024-02-05T08:12:15.881540+0000","last_active":"2024-02-05T08:12:48.784208+0000","last_peered":"2024-02-05T08:12:48.784208+0000","last_clean":"2024-02-05T08:12:48.784208+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:12:48.784208+0000","last_undegraded":"2024-02-05T08:12:48.784208+0000","last_fullsized":"2024-02-05T08:12:48.784208+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:15.881453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:15.881453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:30.969474+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"194'9","reported_seq":50,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790801+0000","last_change":"2024-02-05T08:12:41.752244+0000","last_active":"2024-02-05T08:12:48.790801+0000","last_peered":"2024-02-05T08:12:48.790801+0000","last_clean":"2024-02-05T08:12:48.790801+0000","last_became_active":"2024-02-05T08:12:41.728687+0000","last_became_peered":"2024-02-05T08:12:41.728687+0000","last_unstale":"2024-02-05T08:12:48.790801+0000","last_undegraded":"2024-02-05T08:12:48.790801+0000","last_fullsized":"2024-02-05T08:12:48.790801+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:10:33.946950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:33.946950+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.131512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":303,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.868278+0000","last_change":"2024-02-05T08:12:41.723729+0000","last_active":"2024-02-05T08:12:48.868278+0000","last_peered":"2024-02-05T08:12:48.868278+0000","last_clean":"2024-02-05T08:12:48.868278+0000","last_became_active":"2024-02-05T08:12:41.723383+0000","last_became_peered":"2024-02-05T08:12:41.723383+0000","last_unstale":"2024-02-05T08:12:48.868278+0000","last_undegraded":"2024-02-05T08:12:48.868278+0000","last_fullsized":"2024-02-05T08:12:48.868278+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:11:24.316208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:50.923554+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":333,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.788100+0000","last_change":"2024-02-05T08:12:41.736804+0000","last_active":"2024-02-05T08:12:48.788100+0000","last_peered":"2024-02-05T08:12:48.788100+0000","last_clean":"2024-02-05T08:12:48.788100+0000","last_became_active":"2024-02-05T08:12:41.723526+0000","last_became_peered":"2024-02-05T08:12:41.723526+0000","last_unstale":"2024-02-05T08:12:48.788100+0000","last_undegraded":"2024-02-05T08:12:48.788100+0000","last_fullsized":"2024-02-05T08:12:48.788100+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:19.287569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:19.287569+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":302,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.870407+0000","last_change":"2024-02-05T08:12:18.337618+0000","last_active":"2024-02-05T08:12:48.870407+0000","last_peered":"2024-02-05T08:12:48.870407+0000","last_clean":"2024-02-05T08:12:48.870407+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:12:48.870407+0000","last_undegraded":"2024-02-05T08:12:48.870407+0000","last_fullsized":"2024-02-05T08:12:48.870407+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:18.337499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:18.337499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:36.647778+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":19,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.872410+0000","last_change":"2024-02-05T08:12:41.722172+0000","last_active":"2024-02-05T08:12:48.872410+0000","last_peered":"2024-02-05T08:12:48.872410+0000","last_clean":"2024-02-05T08:12:48.872410+0000","last_became_active":"2024-02-05T08:12:41.721852+0000","last_became_peered":"2024-02-05T08:12:41.721852+0000","last_unstale":"2024-02-05T08:12:48.872410+0000","last_undegraded":"2024-02-05T08:12:48.872410+0000","last_fullsized":"2024-02-05T08:12:48.872410+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:15.994621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:15.994621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":310,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.789971+0000","last_change":"2024-02-05T08:12:29.684885+0000","last_active":"2024-02-05T08:12:48.789971+0000","last_peered":"2024-02-05T08:12:48.789971+0000","last_clean":"2024-02-05T08:12:48.789971+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:12:48.789971+0000","last_undegraded":"2024-02-05T08:12:48.789971+0000","last_fullsized":"2024-02-05T08:12:48.789971+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:07.415963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.415963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.131588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":325,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.872668+0000","last_change":"2024-02-05T08:12:41.770999+0000","last_active":"2024-02-05T08:12:48.872668+0000","last_peered":"2024-02-05T08:12:48.872668+0000","last_clean":"2024-02-05T08:12:48.872668+0000","last_became_active":"2024-02-05T08:12:41.736414+0000","last_became_peered":"2024-02-05T08:12:41.736414+0000","last_unstale":"2024-02-05T08:12:48.872668+0000","last_undegraded":"2024-02-05T08:12:48.872668+0000","last_fullsized":"2024-02-05T08:12:48.872668+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:23.324165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:23.324165+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":306,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790993+0000","last_change":"2024-02-05T08:12:29.677397+0000","last_active":"2024-02-05T08:12:48.790993+0000","last_peered":"2024-02-05T08:12:48.790993+0000","last_clean":"2024-02-05T08:12:48.790993+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:12:48.790993+0000","last_undegraded":"2024-02-05T08:12:48.790993+0000","last_fullsized":"2024-02-05T08:12:48.790993+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:18.424341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:18.424341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.131767+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":303,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.787643+0000","last_change":"2024-02-05T08:12:41.721999+0000","last_active":"2024-02-05T08:12:48.787643+0000","last_peered":"2024-02-05T08:12:48.787643+0000","last_clean":"2024-02-05T08:12:48.787643+0000","last_became_active":"2024-02-05T08:12:41.721486+0000","last_became_peered":"2024-02-05T08:12:41.721486+0000","last_unstale":"2024-02-05T08:12:48.787643+0000","last_undegraded":"2024-02-05T08:12:48.787643+0000","last_fullsized":"2024-02-05T08:12:48.787643+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.242229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.242229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:32.796493+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.793689+0000","last_change":"2024-02-05T08:12:29.684768+0000","last_active":"2024-02-05T08:12:48.793689+0000","last_peered":"2024-02-05T08:12:48.793689+0000","last_clean":"2024-02-05T08:12:48.793689+0000","last_became_active":"2024-02-05T08:12:29.684444+0000","last_became_peered":"2024-02-05T08:12:29.684444+0000","last_unstale":"2024-02-05T08:12:48.793689+0000","last_undegraded":"2024-02-05T08:12:48.793689+0000","last_fullsized":"2024-02-05T08:12:48.793689+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:01.330449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:01.330449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"116'9","reported_seq":151,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.791448+0000","last_change":"2024-02-05T08:12:41.763268+0000","last_active":"2024-02-05T08:12:48.791448+0000","last_peered":"2024-02-05T08:12:48.791448+0000","last_clean":"2024-02-05T08:12:48.791448+0000","last_became_active":"2024-02-05T08:12:41.732369+0000","last_became_peered":"2024-02-05T08:12:41.732369+0000","last_unstale":"2024-02-05T08:12:48.791448+0000","last_undegraded":"2024-02-05T08:12:48.791448+0000","last_fullsized":"2024-02-05T08:12:48.791448+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-02-05T08:10:32.930856+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:10:32.930856+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.131558+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"197'7840","reported_seq":8003,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:52.563790+0000","last_change":"2024-02-05T08:12:23.713036+0000","last_active":"2024-02-05T08:12:52.563790+0000","last_peered":"2024-02-05T08:12:52.563790+0000","last_clean":"2024-02-05T08:12:52.563790+0000","last_became_active":"2024-02-05T08:10:54.986264+0000","last_became_peered":"2024-02-05T08:10:54.986264+0000","last_unstale":"2024-02-05T08:12:52.563790+0000","last_undegraded":"2024-02-05T08:12:52.563790+0000","last_fullsized":"2024-02-05T08:12:52.563790+0000","mapping_epoch":157,"log_start":"169'2948","ondisk_log_start":"169'2948","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'6431","last_scrub_stamp":"2024-02-05T08:12:23.712829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:12:23.712829+0000","objects_scrubbed":6374,"log_size":4892,"log_dups_size":0,"ondisk_log_size":4892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:27.342344+0000","scrub_duration":1445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432537600,"num_objects":6600,"num_object_clones":0,"num_object_copies":19800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7840,"num_write_kb":462080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":305,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.787354+0000","last_change":"2024-02-05T08:12:29.680463+0000","last_active":"2024-02-05T08:12:48.787354+0000","last_peered":"2024-02-05T08:12:48.787354+0000","last_clean":"2024-02-05T08:12:48.787354+0000","last_became_active":"2024-02-05T08:12:29.679849+0000","last_became_peered":"2024-02-05T08:12:29.679849+0000","last_unstale":"2024-02-05T08:12:48.787354+0000","last_undegraded":"2024-02-05T08:12:48.787354+0000","last_fullsized":"2024-02-05T08:12:48.787354+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:13.452127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:13.452127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":318,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.791535+0000","last_change":"2024-02-05T08:12:22.085674+0000","last_active":"2024-02-05T08:12:48.791535+0000","last_peered":"2024-02-05T08:12:48.791535+0000","last_clean":"2024-02-05T08:12:48.791535+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:12:48.791535+0000","last_undegraded":"2024-02-05T08:12:48.791535+0000","last_fullsized":"2024-02-05T08:12:48.791535+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:12:22.085596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:22.085596+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:23.913734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":370,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.788042+0000","last_change":"2024-02-05T08:12:29.719785+0000","last_active":"2024-02-05T08:12:48.788042+0000","last_peered":"2024-02-05T08:12:48.788042+0000","last_clean":"2024-02-05T08:12:48.788042+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:12:48.788042+0000","last_undegraded":"2024-02-05T08:12:48.788042+0000","last_fullsized":"2024-02-05T08:12:48.788042+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:12:26.319958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.319958+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:48.768788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"190'10","reported_seq":425,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790147+0000","last_change":"2024-02-05T08:12:29.714750+0000","last_active":"2024-02-05T08:12:48.790147+0000","last_peered":"2024-02-05T08:12:48.790147+0000","last_clean":"2024-02-05T08:12:48.790147+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:12:48.790147+0000","last_undegraded":"2024-02-05T08:12:48.790147+0000","last_fullsized":"2024-02-05T08:12:48.790147+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:11:14.422445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:14.422445+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.131868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"197'7963","reported_seq":15421,"reported_epoch":197,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:12:53.324680+0000","last_change":"2024-02-05T08:12:41.759343+0000","last_active":"2024-02-05T08:12:53.324680+0000","last_peered":"2024-02-05T08:12:53.324680+0000","last_clean":"2024-02-05T08:12:28.935587+0000","last_became_active":"2024-02-05T08:12:30.757813+0000","last_became_peered":"2024-02-05T08:12:30.757813+0000","last_unstale":"2024-02-05T08:12:53.324680+0000","last_undegraded":"2024-02-05T08:12:53.324680+0000","last_fullsized":"2024-02-05T08:12:53.324680+0000","mapping_epoch":188,"log_start":"168'2864","ondisk_log_start":"168'2864","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434700288,"num_objects":6633,"num_object_clones":0,"num_object_copies":19899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3189,"num_objects_unfound":0,"num_objects_dirty":6633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7963,"num_write_kb":467072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3681,"num_bytes_recovered":241238016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[0,4,7],"avail_no_missing":["0","4","7"],"object_location_counts":[{"shards":"0,4,7","objects":6633}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":18,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790361+0000","last_change":"2024-02-05T08:12:41.723494+0000","last_active":"2024-02-05T08:12:48.790361+0000","last_peered":"2024-02-05T08:12:48.790361+0000","last_clean":"2024-02-05T08:12:48.790361+0000","last_became_active":"2024-02-05T08:12:41.722853+0000","last_became_peered":"2024-02-05T08:12:41.722853+0000","last_unstale":"2024-02-05T08:12:48.790361+0000","last_undegraded":"2024-02-05T08:12:48.790361+0000","last_fullsized":"2024-02-05T08:12:48.790361+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:21.562331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:21.562331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:39.534804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":306,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.791554+0000","last_change":"2024-02-05T08:11:31.474056+0000","last_active":"2024-02-05T08:12:48.791554+0000","last_peered":"2024-02-05T08:12:48.791554+0000","last_clean":"2024-02-05T08:12:48.791554+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:12:48.791554+0000","last_undegraded":"2024-02-05T08:12:48.791554+0000","last_fullsized":"2024-02-05T08:12:48.791554+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:31.473976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:11:31.473976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.557497+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":334,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.783637+0000","last_change":"2024-02-05T08:12:41.750244+0000","last_active":"2024-02-05T08:12:48.783637+0000","last_peered":"2024-02-05T08:12:48.783637+0000","last_clean":"2024-02-05T08:12:48.783637+0000","last_became_active":"2024-02-05T08:12:41.728925+0000","last_became_peered":"2024-02-05T08:12:41.728925+0000","last_unstale":"2024-02-05T08:12:48.783637+0000","last_undegraded":"2024-02-05T08:12:48.783637+0000","last_fullsized":"2024-02-05T08:12:48.783637+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:12:19.888170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:19.888170+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:27.404627+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":306,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.788861+0000","last_change":"2024-02-05T08:12:29.353477+0000","last_active":"2024-02-05T08:12:48.788861+0000","last_peered":"2024-02-05T08:12:48.788861+0000","last_clean":"2024-02-05T08:12:48.788861+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:12:48.788861+0000","last_undegraded":"2024-02-05T08:12:48.788861+0000","last_fullsized":"2024-02-05T08:12:48.788861+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:29.353397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_clean_scrub_stamp":"2024-02-05T08:12:29.353397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:42.610728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"197'7965","reported_seq":14101,"reported_epoch":197,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:12:53.324598+0000","last_change":"2024-02-05T08:12:41.770370+0000","last_active":"2024-02-05T08:12:53.324598+0000","last_peered":"2024-02-05T08:12:53.324598+0000","last_clean":"2024-02-05T08:12:28.993658+0000","last_became_active":"2024-02-05T08:12:30.762738+0000","last_became_peered":"2024-02-05T08:12:30.762738+0000","last_unstale":"2024-02-05T08:12:53.324598+0000","last_undegraded":"2024-02-05T08:12:53.324598+0000","last_fullsized":"2024-02-05T08:12:53.324598+0000","mapping_epoch":188,"log_start":"168'2953","ondisk_log_start":"168'2953","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5012,"log_dups_size":0,"ondisk_log_size":5012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432472064,"num_objects":6599,"num_object_clones":0,"num_object_copies":19797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3772,"num_objects_unfound":0,"num_objects_dirty":6599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7965,"num_write_kb":466048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3020,"num_bytes_recovered":197918720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,7,2],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":6599}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":300,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.789892+0000","last_change":"2024-02-05T08:12:41.725243+0000","last_active":"2024-02-05T08:12:48.789892+0000","last_peered":"2024-02-05T08:12:48.789892+0000","last_clean":"2024-02-05T08:12:48.789892+0000","last_became_active":"2024-02-05T08:12:41.724830+0000","last_became_peered":"2024-02-05T08:12:41.724830+0000","last_unstale":"2024-02-05T08:12:48.789892+0000","last_undegraded":"2024-02-05T08:12:48.789892+0000","last_fullsized":"2024-02-05T08:12:48.789892+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:21.070242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:21.070242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:42.972126+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.788822+0000","last_change":"2024-02-05T08:11:22.231278+0000","last_active":"2024-02-05T08:12:48.788822+0000","last_peered":"2024-02-05T08:12:48.788822+0000","last_clean":"2024-02-05T08:12:48.788822+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:12:48.788822+0000","last_undegraded":"2024-02-05T08:12:48.788822+0000","last_fullsized":"2024-02-05T08:12:48.788822+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:11:22.231148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:51.378207+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":318,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.791299+0000","last_change":"2024-02-05T08:12:29.679032+0000","last_active":"2024-02-05T08:12:48.791299+0000","last_peered":"2024-02-05T08:12:48.791299+0000","last_clean":"2024-02-05T08:12:48.791299+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:12:48.791299+0000","last_undegraded":"2024-02-05T08:12:48.791299+0000","last_fullsized":"2024-02-05T08:12:48.791299+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:27.540472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:27.540472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.131908+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":49,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.793195+0000","last_change":"2024-02-05T08:12:29.716681+0000","last_active":"2024-02-05T08:12:48.793195+0000","last_peered":"2024-02-05T08:12:48.793195+0000","last_clean":"2024-02-05T08:12:48.793195+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:12:48.793195+0000","last_undegraded":"2024-02-05T08:12:48.793195+0000","last_fullsized":"2024-02-05T08:12:48.793195+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:11:19.277464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:19.277464+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"197'7803","reported_seq":12980,"reported_epoch":197,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:12:53.048455+0000","last_change":"2024-02-05T08:12:42.740253+0000","last_active":"2024-02-05T08:12:53.048455+0000","last_peered":"2024-02-05T08:12:53.048455+0000","last_clean":"2024-02-05T08:12:29.026043+0000","last_became_active":"2024-02-05T08:12:42.725152+0000","last_became_peered":"2024-02-05T08:12:42.725152+0000","last_unstale":"2024-02-05T08:12:53.048455+0000","last_undegraded":"2024-02-05T08:12:53.048455+0000","last_fullsized":"2024-02-05T08:12:53.048455+0000","mapping_epoch":194,"log_start":"174'3622","ondisk_log_start":"174'3622","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4181,"log_dups_size":0,"ondisk_log_size":4181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426967040,"num_objects":6515,"num_object_clones":0,"num_object_copies":19545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4106,"num_objects_unfound":0,"num_objects_dirty":6515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7803,"num_write_kb":458176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2538,"num_bytes_recovered":166330368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,2,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":6515}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":285,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.871062+0000","last_change":"2024-02-05T08:11:08.235398+0000","last_active":"2024-02-05T08:12:48.871062+0000","last_peered":"2024-02-05T08:12:48.871062+0000","last_clean":"2024-02-05T08:12:48.871062+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:12:48.871062+0000","last_undegraded":"2024-02-05T08:12:48.871062+0000","last_fullsized":"2024-02-05T08:12:48.871062+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:08.235281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:08.235281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":304,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.788451+0000","last_change":"2024-02-05T08:12:41.744632+0000","last_active":"2024-02-05T08:12:48.788451+0000","last_peered":"2024-02-05T08:12:48.788451+0000","last_clean":"2024-02-05T08:12:48.788451+0000","last_became_active":"2024-02-05T08:12:41.743888+0000","last_became_peered":"2024-02-05T08:12:41.743888+0000","last_unstale":"2024-02-05T08:12:48.788451+0000","last_undegraded":"2024-02-05T08:12:48.788451+0000","last_fullsized":"2024-02-05T08:12:48.788451+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:15.272287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.272287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":323,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.787865+0000","last_change":"2024-02-05T08:12:41.719156+0000","last_active":"2024-02-05T08:12:48.787865+0000","last_peered":"2024-02-05T08:12:48.787865+0000","last_clean":"2024-02-05T08:12:48.787865+0000","last_became_active":"2024-02-05T08:12:41.718843+0000","last_became_peered":"2024-02-05T08:12:41.718843+0000","last_unstale":"2024-02-05T08:12:48.787865+0000","last_undegraded":"2024-02-05T08:12:48.787865+0000","last_fullsized":"2024-02-05T08:12:48.787865+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:18.291829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:18.291829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:38.961092+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":317,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.868867+0000","last_change":"2024-02-05T08:12:22.599613+0000","last_active":"2024-02-05T08:12:48.868867+0000","last_peered":"2024-02-05T08:12:48.868867+0000","last_clean":"2024-02-05T08:12:48.868867+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:12:48.868867+0000","last_undegraded":"2024-02-05T08:12:48.868867+0000","last_fullsized":"2024-02-05T08:12:48.868867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:12:22.599493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:22.599493+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:42.403843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"197'7894","reported_seq":8022,"reported_epoch":197,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T08:12:52.553707+0000","last_change":"2024-02-05T08:12:42.858463+0000","last_active":"2024-02-05T08:12:52.553707+0000","last_peered":"2024-02-05T08:12:52.553707+0000","last_clean":"2024-02-05T08:12:40.696594+0000","last_became_active":"2024-02-05T08:12:42.836170+0000","last_became_peered":"2024-02-05T08:12:42.836170+0000","last_unstale":"2024-02-05T08:12:52.553707+0000","last_undegraded":"2024-02-05T08:12:52.553707+0000","last_fullsized":"2024-02-05T08:12:52.553707+0000","mapping_epoch":194,"log_start":"169'2993","ondisk_log_start":"169'2993","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":2849,"log_size":4901,"log_dups_size":0,"ondisk_log_size":4901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434634752,"num_objects":6632,"num_object_clones":0,"num_object_copies":19896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6632,"num_objects_unfound":0,"num_objects_dirty":6632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7894,"num_write_kb":464832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,3],"avail_no_missing":["5","3","7"],"object_location_counts":[{"shards":"3,5,7","objects":6632}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":18,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.784048+0000","last_change":"2024-02-05T08:12:41.727102+0000","last_active":"2024-02-05T08:12:48.784048+0000","last_peered":"2024-02-05T08:12:48.784048+0000","last_clean":"2024-02-05T08:12:48.784048+0000","last_became_active":"2024-02-05T08:12:41.726776+0000","last_became_peered":"2024-02-05T08:12:41.726776+0000","last_unstale":"2024-02-05T08:12:48.784048+0000","last_undegraded":"2024-02-05T08:12:48.784048+0000","last_fullsized":"2024-02-05T08:12:48.784048+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:32.545208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:32.545208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:54.224769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":337,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.787082+0000","last_change":"2024-02-05T08:12:29.715120+0000","last_active":"2024-02-05T08:12:48.787082+0000","last_peered":"2024-02-05T08:12:48.787082+0000","last_clean":"2024-02-05T08:12:48.787082+0000","last_became_active":"2024-02-05T08:12:29.679395+0000","last_became_peered":"2024-02-05T08:12:29.679395+0000","last_unstale":"2024-02-05T08:12:48.787082+0000","last_undegraded":"2024-02-05T08:12:48.787082+0000","last_fullsized":"2024-02-05T08:12:48.787082+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:12:13.266830+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:12:13.266830+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:16.282151+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":309,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.783959+0000","last_change":"2024-02-05T08:12:29.671185+0000","last_active":"2024-02-05T08:12:48.783959+0000","last_peered":"2024-02-05T08:12:48.783959+0000","last_clean":"2024-02-05T08:12:48.783959+0000","last_became_active":"2024-02-05T08:12:29.670859+0000","last_became_peered":"2024-02-05T08:12:29.670859+0000","last_unstale":"2024-02-05T08:12:48.783959+0000","last_undegraded":"2024-02-05T08:12:48.783959+0000","last_fullsized":"2024-02-05T08:12:48.783959+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:07.502393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.502393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"194'39","reported_seq":487,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.870615+0000","last_change":"2024-02-05T08:12:41.775200+0000","last_active":"2024-02-05T08:12:48.870615+0000","last_peered":"2024-02-05T08:12:48.870615+0000","last_clean":"2024-02-05T08:12:48.870615+0000","last_became_active":"2024-02-05T08:12:41.719820+0000","last_became_peered":"2024-02-05T08:12:41.719820+0000","last_unstale":"2024-02-05T08:12:48.870615+0000","last_undegraded":"2024-02-05T08:12:48.870615+0000","last_fullsized":"2024-02-05T08:12:48.870615+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:11:34.819564+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:11:34.819564+0000","objects_scrubbed":10,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":289,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.871404+0000","last_change":"2024-02-05T08:12:28.727498+0000","last_active":"2024-02-05T08:12:48.871404+0000","last_peered":"2024-02-05T08:12:48.871404+0000","last_clean":"2024-02-05T08:12:48.871404+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:12:48.871404+0000","last_undegraded":"2024-02-05T08:12:48.871404+0000","last_fullsized":"2024-02-05T08:12:48.871404+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:28.721402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:28.721402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:51.573749+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"197'7871","reported_seq":7977,"reported_epoch":197,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T08:12:52.538715+0000","last_change":"2024-02-05T08:12:42.843779+0000","last_active":"2024-02-05T08:12:52.538715+0000","last_peered":"2024-02-05T08:12:52.538715+0000","last_clean":"2024-02-05T08:12:40.665520+0000","last_became_active":"2024-02-05T08:12:42.797892+0000","last_became_peered":"2024-02-05T08:12:42.797892+0000","last_unstale":"2024-02-05T08:12:52.538715+0000","last_undegraded":"2024-02-05T08:12:52.538715+0000","last_fullsized":"2024-02-05T08:12:52.538715+0000","mapping_epoch":194,"log_start":"171'3387","ondisk_log_start":"171'3387","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4484,"log_dups_size":0,"ondisk_log_size":4484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":431685632,"num_objects":6587,"num_object_clones":0,"num_object_copies":19761,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6587,"num_objects_unfound":0,"num_objects_dirty":6587,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7871,"num_write_kb":462656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,3],"avail_no_missing":["4","1","3"],"object_location_counts":[{"shards":"1,3,4","objects":6587}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":312,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.791389+0000","last_change":"2024-02-05T08:12:41.733166+0000","last_active":"2024-02-05T08:12:48.791389+0000","last_peered":"2024-02-05T08:12:48.791389+0000","last_clean":"2024-02-05T08:12:48.791389+0000","last_became_active":"2024-02-05T08:12:41.732338+0000","last_became_peered":"2024-02-05T08:12:41.732338+0000","last_unstale":"2024-02-05T08:12:48.791389+0000","last_undegraded":"2024-02-05T08:12:48.791389+0000","last_fullsized":"2024-02-05T08:12:48.791389+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:27.114442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:27.114442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:53.398218+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":326,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.873502+0000","last_change":"2024-02-05T08:12:29.682711+0000","last_active":"2024-02-05T08:12:48.873502+0000","last_peered":"2024-02-05T08:12:48.873502+0000","last_clean":"2024-02-05T08:12:48.873502+0000","last_became_active":"2024-02-05T08:12:29.673554+0000","last_became_peered":"2024-02-05T08:12:29.673554+0000","last_unstale":"2024-02-05T08:12:48.873502+0000","last_undegraded":"2024-02-05T08:12:48.873502+0000","last_fullsized":"2024-02-05T08:12:48.873502+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:11:23.934912+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:11:23.934912+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":329,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.788898+0000","last_change":"2024-02-05T08:12:41.746703+0000","last_active":"2024-02-05T08:12:48.788898+0000","last_peered":"2024-02-05T08:12:48.788898+0000","last_clean":"2024-02-05T08:12:48.788898+0000","last_became_active":"2024-02-05T08:12:41.746289+0000","last_became_peered":"2024-02-05T08:12:41.746289+0000","last_unstale":"2024-02-05T08:12:48.788898+0000","last_undegraded":"2024-02-05T08:12:48.788898+0000","last_fullsized":"2024-02-05T08:12:48.788898+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.404533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.404533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:41.027208+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":306,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.870662+0000","last_change":"2024-02-05T08:11:25.289984+0000","last_active":"2024-02-05T08:12:48.870662+0000","last_peered":"2024-02-05T08:12:48.870662+0000","last_clean":"2024-02-05T08:12:48.870662+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:12:48.870662+0000","last_undegraded":"2024-02-05T08:12:48.870662+0000","last_fullsized":"2024-02-05T08:12:48.870662+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:11:25.289752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:25.289752+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":306,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.877648+0000","last_change":"2024-02-05T08:12:25.317266+0000","last_active":"2024-02-05T08:12:48.877648+0000","last_peered":"2024-02-05T08:12:48.877648+0000","last_clean":"2024-02-05T08:12:48.877648+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:12:48.877648+0000","last_undegraded":"2024-02-05T08:12:48.877648+0000","last_fullsized":"2024-02-05T08:12:48.877648+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:25.317043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:25.317043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:46.301280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"197'7832","reported_seq":7933,"reported_epoch":197,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T08:12:52.992070+0000","last_change":"2024-02-05T08:12:30.813287+0000","last_active":"2024-02-05T08:12:52.992070+0000","last_peered":"2024-02-05T08:12:52.992070+0000","last_clean":"2024-02-05T08:12:29.005556+0000","last_became_active":"2024-02-05T08:12:30.765777+0000","last_became_peered":"2024-02-05T08:12:30.765777+0000","last_unstale":"2024-02-05T08:12:52.992070+0000","last_undegraded":"2024-02-05T08:12:52.992070+0000","last_fullsized":"2024-02-05T08:12:52.992070+0000","mapping_epoch":188,"log_start":"168'2396","ondisk_log_start":"168'2396","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5436,"log_dups_size":0,"ondisk_log_size":5436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433586176,"num_objects":6616,"num_object_clones":0,"num_object_copies":19848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6616,"num_objects_unfound":0,"num_objects_dirty":6616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7832,"num_write_kb":462336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[3,5,0],"avail_no_missing":["3","0","5"],"object_location_counts":[{"shards":"0,3,5","objects":6616}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.788111+0000","last_change":"2024-02-05T08:12:41.717519+0000","last_active":"2024-02-05T08:12:48.788111+0000","last_peered":"2024-02-05T08:12:48.788111+0000","last_clean":"2024-02-05T08:12:48.788111+0000","last_became_active":"2024-02-05T08:12:41.717170+0000","last_became_peered":"2024-02-05T08:12:41.717170+0000","last_unstale":"2024-02-05T08:12:48.788111+0000","last_undegraded":"2024-02-05T08:12:48.788111+0000","last_fullsized":"2024-02-05T08:12:48.788111+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:23.303613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:23.303613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:28.548919+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":587,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790332+0000","last_change":"2024-02-05T08:12:32.677767+0000","last_active":"2024-02-05T08:12:48.790332+0000","last_peered":"2024-02-05T08:12:48.790332+0000","last_clean":"2024-02-05T08:12:48.790332+0000","last_became_active":"2024-02-05T08:12:32.677426+0000","last_became_peered":"2024-02-05T08:12:32.677426+0000","last_unstale":"2024-02-05T08:12:48.790332+0000","last_undegraded":"2024-02-05T08:12:48.790332+0000","last_fullsized":"2024-02-05T08:12:48.790332+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:11:05.478284+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:11:05.478284+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.131729+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":300,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.786995+0000","last_change":"2024-02-05T08:12:41.722635+0000","last_active":"2024-02-05T08:12:48.786995+0000","last_peered":"2024-02-05T08:12:48.786995+0000","last_clean":"2024-02-05T08:12:48.786995+0000","last_became_active":"2024-02-05T08:12:41.722331+0000","last_became_peered":"2024-02-05T08:12:41.722331+0000","last_unstale":"2024-02-05T08:12:48.786995+0000","last_undegraded":"2024-02-05T08:12:48.786995+0000","last_fullsized":"2024-02-05T08:12:48.786995+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:29.759802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:29.759802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"190'18","reported_seq":52,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.793948+0000","last_change":"2024-02-05T08:12:29.711770+0000","last_active":"2024-02-05T08:12:48.793948+0000","last_peered":"2024-02-05T08:12:48.793948+0000","last_clean":"2024-02-05T08:12:48.793948+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:12:48.793948+0000","last_undegraded":"2024-02-05T08:12:48.793948+0000","last_fullsized":"2024-02-05T08:12:48.793948+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-02-05T08:12:26.852734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.852734+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:51.240118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":290,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790957+0000","last_change":"2024-02-05T08:11:04.446955+0000","last_active":"2024-02-05T08:12:48.790957+0000","last_peered":"2024-02-05T08:12:48.790957+0000","last_clean":"2024-02-05T08:12:48.790957+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:12:48.790957+0000","last_undegraded":"2024-02-05T08:12:48.790957+0000","last_fullsized":"2024-02-05T08:12:48.790957+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:04.446828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:04.446828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.131688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"197'8009","reported_seq":8099,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:53.031876+0000","last_change":"2024-02-05T08:10:54.985737+0000","last_active":"2024-02-05T08:12:53.031876+0000","last_peered":"2024-02-05T08:12:53.031876+0000","last_clean":"2024-02-05T08:12:53.031876+0000","last_became_active":"2024-02-05T08:10:54.985443+0000","last_became_peered":"2024-02-05T08:10:54.985443+0000","last_unstale":"2024-02-05T08:12:53.031876+0000","last_undegraded":"2024-02-05T08:12:53.031876+0000","last_fullsized":"2024-02-05T08:12:53.031876+0000","mapping_epoch":157,"log_start":"174'3815","ondisk_log_start":"174'3815","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4194,"log_dups_size":0,"ondisk_log_size":4194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439681024,"num_objects":6709,"num_object_clones":0,"num_object_copies":20127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8009,"num_write_kb":470976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":300,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.870248+0000","last_change":"2024-02-05T08:12:41.716722+0000","last_active":"2024-02-05T08:12:48.870248+0000","last_peered":"2024-02-05T08:12:48.870248+0000","last_clean":"2024-02-05T08:12:48.870248+0000","last_became_active":"2024-02-05T08:12:41.716394+0000","last_became_peered":"2024-02-05T08:12:41.716394+0000","last_unstale":"2024-02-05T08:12:48.870248+0000","last_undegraded":"2024-02-05T08:12:48.870248+0000","last_fullsized":"2024-02-05T08:12:48.870248+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:26.178219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.178219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:28.307031+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":303,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.868630+0000","last_change":"2024-02-05T08:12:41.742379+0000","last_active":"2024-02-05T08:12:48.868630+0000","last_peered":"2024-02-05T08:12:48.868630+0000","last_clean":"2024-02-05T08:12:48.868630+0000","last_became_active":"2024-02-05T08:12:41.741846+0000","last_became_peered":"2024-02-05T08:12:41.741846+0000","last_unstale":"2024-02-05T08:12:48.868630+0000","last_undegraded":"2024-02-05T08:12:48.868630+0000","last_fullsized":"2024-02-05T08:12:48.868630+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:34.336040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:34.336040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":804,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:51.383757+0000","last_change":"2024-02-05T08:11:20.552800+0000","last_active":"2024-02-05T08:12:51.383757+0000","last_peered":"2024-02-05T08:12:51.383757+0000","last_clean":"2024-02-05T08:12:51.383757+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:12:51.383757+0000","last_undegraded":"2024-02-05T08:12:51.383757+0000","last_fullsized":"2024-02-05T08:12:51.383757+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:11:20.552682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:11:20.552682+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.131830+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":485,"num_read_kb":480,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":311,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.872425+0000","last_change":"2024-02-05T08:12:16.517917+0000","last_active":"2024-02-05T08:12:48.872425+0000","last_peered":"2024-02-05T08:12:48.872425+0000","last_clean":"2024-02-05T08:12:48.872425+0000","last_became_active":"2024-02-05T08:04:43.549687+0000","last_became_peered":"2024-02-05T08:04:43.549687+0000","last_unstale":"2024-02-05T08:12:48.872425+0000","last_undegraded":"2024-02-05T08:12:48.872425+0000","last_fullsized":"2024-02-05T08:12:48.872425+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:12:16.517774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:16.517774+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:29.388717+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":297,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.788826+0000","last_change":"2024-02-05T08:12:24.353439+0000","last_active":"2024-02-05T08:12:48.788826+0000","last_peered":"2024-02-05T08:12:48.788826+0000","last_clean":"2024-02-05T08:12:48.788826+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:12:48.788826+0000","last_undegraded":"2024-02-05T08:12:48.788826+0000","last_fullsized":"2024-02-05T08:12:48.788826+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:24.353354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:24.353354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:41.429618+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"197'8010","reported_seq":10382,"reported_epoch":197,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:12:52.913064+0000","last_change":"2024-02-05T08:12:42.801000+0000","last_active":"2024-02-05T08:12:52.913064+0000","last_peered":"2024-02-05T08:12:52.913064+0000","last_clean":"2024-02-05T08:12:40.643159+0000","last_became_active":"2024-02-05T08:12:42.786875+0000","last_became_peered":"2024-02-05T08:12:42.786875+0000","last_unstale":"2024-02-05T08:12:52.913064+0000","last_undegraded":"2024-02-05T08:12:52.913064+0000","last_fullsized":"2024-02-05T08:12:52.913064+0000","mapping_epoch":194,"log_start":"169'3000","ondisk_log_start":"169'3000","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5010,"log_dups_size":0,"ondisk_log_size":5010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438304768,"num_objects":6688,"num_object_clones":0,"num_object_copies":20064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5600,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8010,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1138,"num_bytes_recovered":74579968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,3],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":6688}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":306,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.871594+0000","last_change":"2024-02-05T08:12:29.673477+0000","last_active":"2024-02-05T08:12:48.871594+0000","last_peered":"2024-02-05T08:12:48.871594+0000","last_clean":"2024-02-05T08:12:48.871594+0000","last_became_active":"2024-02-05T08:12:29.672973+0000","last_became_peered":"2024-02-05T08:12:29.672973+0000","last_unstale":"2024-02-05T08:12:48.871594+0000","last_undegraded":"2024-02-05T08:12:48.871594+0000","last_fullsized":"2024-02-05T08:12:48.871594+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:25.219172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:25.219172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:43.872593+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":306,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.870440+0000","last_change":"2024-02-05T08:11:18.828143+0000","last_active":"2024-02-05T08:12:48.870440+0000","last_peered":"2024-02-05T08:12:48.870440+0000","last_clean":"2024-02-05T08:12:48.870440+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:12:48.870440+0000","last_undegraded":"2024-02-05T08:12:48.870440+0000","last_fullsized":"2024-02-05T08:12:48.870440+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:18.828029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:18.828029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":337,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.868568+0000","last_change":"2024-02-05T08:12:27.724075+0000","last_active":"2024-02-05T08:12:48.868568+0000","last_peered":"2024-02-05T08:12:48.868568+0000","last_clean":"2024-02-05T08:12:48.868568+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:12:48.868568+0000","last_undegraded":"2024-02-05T08:12:48.868568+0000","last_fullsized":"2024-02-05T08:12:48.868568+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:12:27.723924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:27.723924+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:30.337919+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":29,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.793113+0000","last_change":"2024-02-05T08:12:29.688850+0000","last_active":"2024-02-05T08:12:48.793113+0000","last_peered":"2024-02-05T08:12:48.793113+0000","last_clean":"2024-02-05T08:12:48.793113+0000","last_became_active":"2024-02-05T08:12:29.688534+0000","last_became_peered":"2024-02-05T08:12:29.688534+0000","last_unstale":"2024-02-05T08:12:48.793113+0000","last_undegraded":"2024-02-05T08:12:48.793113+0000","last_fullsized":"2024-02-05T08:12:48.793113+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:21.296072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:11:21.296072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"197'8017","reported_seq":8107,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:53.276849+0000","last_change":"2024-02-05T08:10:54.987464+0000","last_active":"2024-02-05T08:12:53.276849+0000","last_peered":"2024-02-05T08:12:53.276849+0000","last_clean":"2024-02-05T08:12:53.276849+0000","last_became_active":"2024-02-05T08:10:54.985442+0000","last_became_peered":"2024-02-05T08:10:54.985442+0000","last_unstale":"2024-02-05T08:12:53.276849+0000","last_undegraded":"2024-02-05T08:12:53.276849+0000","last_fullsized":"2024-02-05T08:12:53.276849+0000","mapping_epoch":157,"log_start":"169'2944","ondisk_log_start":"169'2944","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442171392,"num_objects":6747,"num_object_clones":0,"num_object_copies":20241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8017,"num_write_kb":472448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":300,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790106+0000","last_change":"2024-02-05T08:12:41.736005+0000","last_active":"2024-02-05T08:12:48.790106+0000","last_peered":"2024-02-05T08:12:48.790106+0000","last_clean":"2024-02-05T08:12:48.790106+0000","last_became_active":"2024-02-05T08:12:41.735706+0000","last_became_peered":"2024-02-05T08:12:41.735706+0000","last_unstale":"2024-02-05T08:12:48.790106+0000","last_undegraded":"2024-02-05T08:12:48.790106+0000","last_fullsized":"2024-02-05T08:12:48.790106+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.607056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.607056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:31.817287+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":311,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.787991+0000","last_change":"2024-02-05T08:12:41.730442+0000","last_active":"2024-02-05T08:12:48.787991+0000","last_peered":"2024-02-05T08:12:48.787991+0000","last_clean":"2024-02-05T08:12:48.787991+0000","last_became_active":"2024-02-05T08:12:41.730010+0000","last_became_peered":"2024-02-05T08:12:41.730010+0000","last_unstale":"2024-02-05T08:12:48.787991+0000","last_undegraded":"2024-02-05T08:12:48.787991+0000","last_fullsized":"2024-02-05T08:12:48.787991+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:26.212363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:26.212363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":317,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.868396+0000","last_change":"2024-02-05T08:12:19.056472+0000","last_active":"2024-02-05T08:12:48.868396+0000","last_peered":"2024-02-05T08:12:48.868396+0000","last_clean":"2024-02-05T08:12:48.868396+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:12:48.868396+0000","last_undegraded":"2024-02-05T08:12:48.868396+0000","last_fullsized":"2024-02-05T08:12:48.868396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:12:19.056345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:19.056345+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:20.775637+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"190'10","reported_seq":416,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.869011+0000","last_change":"2024-02-05T08:12:29.716459+0000","last_active":"2024-02-05T08:12:48.869011+0000","last_peered":"2024-02-05T08:12:48.869011+0000","last_clean":"2024-02-05T08:12:48.869011+0000","last_became_active":"2024-02-05T08:12:29.683091+0000","last_became_peered":"2024-02-05T08:12:29.683091+0000","last_unstale":"2024-02-05T08:12:48.869011+0000","last_undegraded":"2024-02-05T08:12:48.869011+0000","last_fullsized":"2024-02-05T08:12:48.869011+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:11:13.252683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:13.252683+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"197'8099","reported_seq":8190,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:52.596028+0000","last_change":"2024-02-05T08:10:54.986461+0000","last_active":"2024-02-05T08:12:52.596028+0000","last_peered":"2024-02-05T08:12:52.596028+0000","last_clean":"2024-02-05T08:12:52.596028+0000","last_became_active":"2024-02-05T08:10:54.986149+0000","last_became_peered":"2024-02-05T08:10:54.986149+0000","last_unstale":"2024-02-05T08:12:52.596028+0000","last_undegraded":"2024-02-05T08:12:52.596028+0000","last_fullsized":"2024-02-05T08:12:52.596028+0000","mapping_epoch":157,"log_start":"170'3211","ondisk_log_start":"170'3211","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449708056,"num_objects":6863,"num_object_clones":0,"num_object_copies":20589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6863,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8099,"num_write_kb":478721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790589+0000","last_change":"2024-02-05T08:12:29.678526+0000","last_active":"2024-02-05T08:12:48.790589+0000","last_peered":"2024-02-05T08:12:48.790589+0000","last_clean":"2024-02-05T08:12:48.790589+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:12:48.790589+0000","last_undegraded":"2024-02-05T08:12:48.790589+0000","last_fullsized":"2024-02-05T08:12:48.790589+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:09.466721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:09.466721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:12:53.131617+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":316,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.872504+0000","last_change":"2024-02-05T08:12:21.623410+0000","last_active":"2024-02-05T08:12:48.872504+0000","last_peered":"2024-02-05T08:12:48.872504+0000","last_clean":"2024-02-05T08:12:48.872504+0000","last_became_active":"2024-02-05T08:11:23.200753+0000","last_became_peered":"2024-02-05T08:11:23.200753+0000","last_unstale":"2024-02-05T08:12:48.872504+0000","last_undegraded":"2024-02-05T08:12:48.872504+0000","last_fullsized":"2024-02-05T08:12:48.872504+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:12:21.623291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:21.623291+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:47.267748+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":358,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.783542+0000","last_change":"2024-02-05T08:12:29.712640+0000","last_active":"2024-02-05T08:12:48.783542+0000","last_peered":"2024-02-05T08:12:48.783542+0000","last_clean":"2024-02-05T08:12:48.783542+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:12:48.783542+0000","last_undegraded":"2024-02-05T08:12:48.783542+0000","last_fullsized":"2024-02-05T08:12:48.783542+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-02-05T08:12:27.883928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:27.883928+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:48.176622+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"194'10","reported_seq":341,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.873281+0000","last_change":"2024-02-05T08:12:41.773528+0000","last_active":"2024-02-05T08:12:48.873281+0000","last_peered":"2024-02-05T08:12:48.873281+0000","last_clean":"2024-02-05T08:12:48.873281+0000","last_became_active":"2024-02-05T08:12:41.721686+0000","last_became_peered":"2024-02-05T08:12:41.721686+0000","last_unstale":"2024-02-05T08:12:48.873281+0000","last_undegraded":"2024-02-05T08:12:48.873281+0000","last_fullsized":"2024-02-05T08:12:48.873281+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'9","last_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:11:27.751188+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"197'7843","reported_seq":7945,"reported_epoch":197,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T08:12:53.320188+0000","last_change":"2024-02-05T08:12:42.833342+0000","last_active":"2024-02-05T08:12:53.320188+0000","last_peered":"2024-02-05T08:12:53.320188+0000","last_clean":"2024-02-05T08:12:40.686058+0000","last_became_active":"2024-02-05T08:12:42.799583+0000","last_became_peered":"2024-02-05T08:12:42.799583+0000","last_unstale":"2024-02-05T08:12:53.320188+0000","last_undegraded":"2024-02-05T08:12:53.320188+0000","last_fullsized":"2024-02-05T08:12:53.320188+0000","mapping_epoch":194,"log_start":"168'2801","ondisk_log_start":"168'2801","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434962432,"num_objects":6637,"num_object_clones":0,"num_object_copies":19911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6637,"num_objects_unfound":0,"num_objects_dirty":6637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7843,"num_write_kb":463360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":6637}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.784153+0000","last_change":"2024-02-05T08:12:41.731865+0000","last_active":"2024-02-05T08:12:48.784153+0000","last_peered":"2024-02-05T08:12:48.784153+0000","last_clean":"2024-02-05T08:12:48.784153+0000","last_became_active":"2024-02-05T08:12:41.731253+0000","last_became_peered":"2024-02-05T08:12:41.731253+0000","last_unstale":"2024-02-05T08:12:48.784153+0000","last_undegraded":"2024-02-05T08:12:48.784153+0000","last_fullsized":"2024-02-05T08:12:48.784153+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:25.851312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:25.851312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:44.639876+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":30,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.791734+0000","last_change":"2024-02-05T08:12:41.761180+0000","last_active":"2024-02-05T08:12:48.791734+0000","last_peered":"2024-02-05T08:12:48.791734+0000","last_clean":"2024-02-05T08:12:48.791734+0000","last_became_active":"2024-02-05T08:12:41.723699+0000","last_became_peered":"2024-02-05T08:12:41.723699+0000","last_unstale":"2024-02-05T08:12:48.791734+0000","last_undegraded":"2024-02-05T08:12:48.791734+0000","last_fullsized":"2024-02-05T08:12:48.791734+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:12:23.575903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:23.575903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:44.725076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"186'26","reported_seq":342,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.790484+0000","last_change":"2024-02-05T08:12:41.783220+0000","last_active":"2024-02-05T08:12:48.790484+0000","last_peered":"2024-02-05T08:12:48.790484+0000","last_clean":"2024-02-05T08:12:48.790484+0000","last_became_active":"2024-02-05T08:12:41.728044+0000","last_became_peered":"2024-02-05T08:12:41.728044+0000","last_unstale":"2024-02-05T08:12:48.790484+0000","last_undegraded":"2024-02-05T08:12:48.790484+0000","last_fullsized":"2024-02-05T08:12:48.790484+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'26","last_scrub_stamp":"2024-02-05T08:12:24.579532+0000","last_deep_scrub":"186'26","last_deep_scrub_stamp":"2024-02-05T08:12:24.579532+0000","last_clean_scrub_stamp":"2024-02-05T08:12:24.579532+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:28.490849+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":295,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.783434+0000","last_change":"2024-02-05T08:12:41.720557+0000","last_active":"2024-02-05T08:12:48.783434+0000","last_peered":"2024-02-05T08:12:48.783434+0000","last_clean":"2024-02-05T08:12:48.783434+0000","last_became_active":"2024-02-05T08:12:41.720247+0000","last_became_peered":"2024-02-05T08:12:41.720247+0000","last_unstale":"2024-02-05T08:12:48.783434+0000","last_undegraded":"2024-02-05T08:12:48.783434+0000","last_fullsized":"2024-02-05T08:12:48.783434+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:28.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:28.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"197'7940","reported_seq":8965,"reported_epoch":197,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T08:12:52.984192+0000","last_change":"2024-02-05T08:12:41.776443+0000","last_active":"2024-02-05T08:12:52.984192+0000","last_peered":"2024-02-05T08:12:52.984192+0000","last_clean":"2024-02-05T08:12:29.029449+0000","last_became_active":"2024-02-05T08:12:30.750855+0000","last_became_peered":"2024-02-05T08:12:30.750855+0000","last_unstale":"2024-02-05T08:12:52.984192+0000","last_undegraded":"2024-02-05T08:12:52.984192+0000","last_fullsized":"2024-02-05T08:12:52.984192+0000","mapping_epoch":188,"log_start":"174'3768","ondisk_log_start":"174'3768","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437125120,"num_objects":6670,"num_object_clones":0,"num_object_copies":20010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6243,"num_objects_unfound":0,"num_objects_dirty":6670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7940,"num_write_kb":467520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":30212096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,2],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":6670}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":293,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.787289+0000","last_change":"2024-02-05T08:11:08.491427+0000","last_active":"2024-02-05T08:12:48.787289+0000","last_peered":"2024-02-05T08:12:48.787289+0000","last_clean":"2024-02-05T08:12:48.787289+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:12:48.787289+0000","last_undegraded":"2024-02-05T08:12:48.787289+0000","last_fullsized":"2024-02-05T08:12:48.787289+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:08.491314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:08.491314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":396,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.784420+0000","last_change":"2024-02-05T08:11:30.179068+0000","last_active":"2024-02-05T08:12:48.784420+0000","last_peered":"2024-02-05T08:12:48.784420+0000","last_clean":"2024-02-05T08:12:48.784420+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:12:48.784420+0000","last_undegraded":"2024-02-05T08:12:48.784420+0000","last_fullsized":"2024-02-05T08:12:48.784420+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:11:30.178959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:30.178959+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.873706+0000","last_change":"2024-02-05T08:12:41.725451+0000","last_active":"2024-02-05T08:12:48.873706+0000","last_peered":"2024-02-05T08:12:48.873706+0000","last_clean":"2024-02-05T08:12:48.873706+0000","last_became_active":"2024-02-05T08:12:41.724969+0000","last_became_peered":"2024-02-05T08:12:41.724969+0000","last_unstale":"2024-02-05T08:12:48.873706+0000","last_undegraded":"2024-02-05T08:12:48.873706+0000","last_fullsized":"2024-02-05T08:12:48.873706+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:03.761546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:03.761546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":349,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.872274+0000","last_change":"2024-02-05T08:12:41.775704+0000","last_active":"2024-02-05T08:12:48.872274+0000","last_peered":"2024-02-05T08:12:48.872274+0000","last_clean":"2024-02-05T08:12:48.872274+0000","last_became_active":"2024-02-05T08:12:41.725007+0000","last_became_peered":"2024-02-05T08:12:41.725007+0000","last_unstale":"2024-02-05T08:12:48.872274+0000","last_undegraded":"2024-02-05T08:12:48.872274+0000","last_fullsized":"2024-02-05T08:12:48.872274+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:11:30.843620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:30.843620+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"197'7840","reported_seq":7930,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:52.564985+0000","last_change":"2024-02-05T08:10:54.985813+0000","last_active":"2024-02-05T08:12:52.564985+0000","last_peered":"2024-02-05T08:12:52.564985+0000","last_clean":"2024-02-05T08:12:52.564985+0000","last_became_active":"2024-02-05T08:10:54.985501+0000","last_became_peered":"2024-02-05T08:10:54.985501+0000","last_unstale":"2024-02-05T08:12:52.564985+0000","last_undegraded":"2024-02-05T08:12:52.564985+0000","last_fullsized":"2024-02-05T08:12:52.564985+0000","mapping_epoch":157,"log_start":"171'3410","ondisk_log_start":"171'3410","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4430,"log_dups_size":0,"ondisk_log_size":4430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437518336,"num_objects":6676,"num_object_clones":0,"num_object_copies":20028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7840,"num_write_kb":464512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":288,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.793432+0000","last_change":"2024-02-05T08:12:41.720396+0000","last_active":"2024-02-05T08:12:48.793432+0000","last_peered":"2024-02-05T08:12:48.793432+0000","last_clean":"2024-02-05T08:12:48.793432+0000","last_became_active":"2024-02-05T08:12:41.719835+0000","last_became_peered":"2024-02-05T08:12:41.719835+0000","last_unstale":"2024-02-05T08:12:48.793432+0000","last_undegraded":"2024-02-05T08:12:48.793432+0000","last_fullsized":"2024-02-05T08:12:48.793432+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:26.583813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.583813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:29.636244+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":309,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.788753+0000","last_change":"2024-02-05T08:12:29.686189+0000","last_active":"2024-02-05T08:12:48.788753+0000","last_peered":"2024-02-05T08:12:48.788753+0000","last_clean":"2024-02-05T08:12:48.788753+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:12:48.788753+0000","last_undegraded":"2024-02-05T08:12:48.788753+0000","last_fullsized":"2024-02-05T08:12:48.788753+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:29.241689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:29.241689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":305,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.783403+0000","last_change":"2024-02-05T08:11:31.201157+0000","last_active":"2024-02-05T08:12:48.783403+0000","last_peered":"2024-02-05T08:12:48.783403+0000","last_clean":"2024-02-05T08:12:48.783403+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:12:48.783403+0000","last_undegraded":"2024-02-05T08:12:48.783403+0000","last_fullsized":"2024-02-05T08:12:48.783403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:11:31.201062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":348,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.784196+0000","last_change":"2024-02-05T08:12:29.702808+0000","last_active":"2024-02-05T08:12:48.784196+0000","last_peered":"2024-02-05T08:12:48.784196+0000","last_clean":"2024-02-05T08:12:48.784196+0000","last_became_active":"2024-02-05T08:12:29.666284+0000","last_became_peered":"2024-02-05T08:12:29.666284+0000","last_unstale":"2024-02-05T08:12:48.784196+0000","last_undegraded":"2024-02-05T08:12:48.784196+0000","last_fullsized":"2024-02-05T08:12:48.784196+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:11:10.470749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:10.470749+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"197'7888","reported_seq":7978,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:53.034540+0000","last_change":"2024-02-05T08:10:54.982668+0000","last_active":"2024-02-05T08:12:53.034540+0000","last_peered":"2024-02-05T08:12:53.034540+0000","last_clean":"2024-02-05T08:12:53.034540+0000","last_became_active":"2024-02-05T08:10:54.982372+0000","last_became_peered":"2024-02-05T08:10:54.982372+0000","last_unstale":"2024-02-05T08:12:53.034540+0000","last_undegraded":"2024-02-05T08:12:53.034540+0000","last_fullsized":"2024-02-05T08:12:53.034540+0000","mapping_epoch":157,"log_start":"174'3726","ondisk_log_start":"174'3726","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4162,"log_dups_size":0,"ondisk_log_size":4162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437780480,"num_objects":6680,"num_object_clones":0,"num_object_copies":20040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7888,"num_write_kb":466176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":313,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.870800+0000","last_change":"2024-02-05T08:12:41.755839+0000","last_active":"2024-02-05T08:12:48.870800+0000","last_peered":"2024-02-05T08:12:48.870800+0000","last_clean":"2024-02-05T08:12:48.870800+0000","last_became_active":"2024-02-05T08:12:41.737224+0000","last_became_peered":"2024-02-05T08:12:41.737224+0000","last_unstale":"2024-02-05T08:12:48.870800+0000","last_undegraded":"2024-02-05T08:12:48.870800+0000","last_fullsized":"2024-02-05T08:12:48.870800+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:12:26.409403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.409403+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:47.963356+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":290,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.870749+0000","last_change":"2024-02-05T08:11:11.789034+0000","last_active":"2024-02-05T08:12:48.870749+0000","last_peered":"2024-02-05T08:12:48.870749+0000","last_clean":"2024-02-05T08:12:48.870749+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:12:48.870749+0000","last_undegraded":"2024-02-05T08:12:48.870749+0000","last_fullsized":"2024-02-05T08:12:48.870749+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:11.788902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:11.788902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.793282+0000","last_change":"2024-02-05T08:12:29.678813+0000","last_active":"2024-02-05T08:12:48.793282+0000","last_peered":"2024-02-05T08:12:48.793282+0000","last_clean":"2024-02-05T08:12:48.793282+0000","last_became_active":"2024-02-05T08:12:29.678501+0000","last_became_peered":"2024-02-05T08:12:29.678501+0000","last_unstale":"2024-02-05T08:12:48.793282+0000","last_undegraded":"2024-02-05T08:12:48.793282+0000","last_fullsized":"2024-02-05T08:12:48.793282+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:25.446724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:10:25.446724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":321,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.784051+0000","last_change":"2024-02-05T08:12:17.870737+0000","last_active":"2024-02-05T08:12:48.784051+0000","last_peered":"2024-02-05T08:12:48.784051+0000","last_clean":"2024-02-05T08:12:48.784051+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:12:48.784051+0000","last_undegraded":"2024-02-05T08:12:48.784051+0000","last_fullsized":"2024-02-05T08:12:48.784051+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-02-05T08:12:17.870617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:17.870617+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:18.886680+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"197'7926","reported_seq":13681,"reported_epoch":197,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:12:52.613436+0000","last_change":"2024-02-05T08:12:42.805364+0000","last_active":"2024-02-05T08:12:52.613436+0000","last_peered":"2024-02-05T08:12:52.613436+0000","last_clean":"2024-02-05T08:12:40.679251+0000","last_became_active":"2024-02-05T08:12:42.794989+0000","last_became_peered":"2024-02-05T08:12:42.794989+0000","last_unstale":"2024-02-05T08:12:52.613436+0000","last_undegraded":"2024-02-05T08:12:52.613436+0000","last_fullsized":"2024-02-05T08:12:52.613436+0000","mapping_epoch":194,"log_start":"169'3000","ondisk_log_start":"169'3000","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440664064,"num_objects":6724,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3984,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7926,"num_write_kb":468800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2830,"num_bytes_recovered":185466880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":6724}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":285,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.784446+0000","last_change":"2024-02-05T08:11:15.360217+0000","last_active":"2024-02-05T08:12:48.784446+0000","last_peered":"2024-02-05T08:12:48.784446+0000","last_clean":"2024-02-05T08:12:48.784446+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:12:48.784446+0000","last_undegraded":"2024-02-05T08:12:48.784446+0000","last_fullsized":"2024-02-05T08:12:48.784446+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.360090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":414,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.871433+0000","last_change":"2024-02-05T08:12:19.354343+0000","last_active":"2024-02-05T08:12:48.871433+0000","last_peered":"2024-02-05T08:12:48.871433+0000","last_clean":"2024-02-05T08:12:48.871433+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:12:48.871433+0000","last_undegraded":"2024-02-05T08:12:48.871433+0000","last_fullsized":"2024-02-05T08:12:48.871433+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:12:19.351324+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:12:19.351324+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:22.424067+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.868706+0000","last_change":"2024-02-05T08:12:41.757481+0000","last_active":"2024-02-05T08:12:48.868706+0000","last_peered":"2024-02-05T08:12:48.868706+0000","last_clean":"2024-02-05T08:12:48.868706+0000","last_became_active":"2024-02-05T08:12:41.757159+0000","last_became_peered":"2024-02-05T08:12:41.757159+0000","last_unstale":"2024-02-05T08:12:48.868706+0000","last_undegraded":"2024-02-05T08:12:48.868706+0000","last_fullsized":"2024-02-05T08:12:48.868706+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:16.521723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:16.521723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:23.352820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":362,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-02-05T08:12:48.783637+0000","last_change":"2024-02-05T08:12:41.770037+0000","last_active":"2024-02-05T08:12:48.783637+0000","last_peered":"2024-02-05T08:12:48.783637+0000","last_clean":"2024-02-05T08:12:48.783637+0000","last_became_active":"2024-02-05T08:12:41.727271+0000","last_became_peered":"2024-02-05T08:12:41.727271+0000","last_unstale":"2024-02-05T08:12:48.783637+0000","last_undegraded":"2024-02-05T08:12:48.783637+0000","last_fullsized":"2024-02-05T08:12:48.783637+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:11:14.388825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:14.388825+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6984499224,"num_objects":106576,"num_object_clones":0,"num_object_copies":319728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":53366,"num_objects_unfound":0,"num_objects_dirty":106576,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126740,"num_write_kb":7466049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13672,"num_bytes_recovered":895811584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21797023744,"data_stored":21797011528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75902,"ondisk_log_size":75902,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":485,"num_read_kb":480,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":3922,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":12894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":443,"ondisk_log_size":443,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037863,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2338960,"kb_used_data":2155080,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91406192,"statfs":{"total":95995035648,"available":93599940608,"internally_reserved":0,"allocated":2206801920,"data_stored":2206474637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":188283338},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.946,"15min":1.0620000000000001},"min":{"1min":0.22,"5min":0.218,"15min":0.217},"max":{"1min":4.819,"5min":60.127000000000002,"15min":60.127000000000002},"last":2.2410000000000001},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.9790000000000001,"15min":1.0429999999999999},"min":{"1min":0.248,"5min":0.23300000000000001,"15min":0.22500000000000001},"max":{"1min":4.2300000000000004,"5min":58.283000000000001,"15min":58.283000000000001},"last":1.5049999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":2.7349999999999999,"5min":2.1909999999999998,"15min":1.123},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":32.682000000000002,"5min":38.399999999999999,"15min":38.399999999999999},"last":2.4380000000000002},{"interface":"front","average":{"1min":2.7410000000000001,"5min":1.9219999999999999,"15min":1.02},"min":{"1min":0.312,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":32.412999999999997,"5min":38.195999999999998,"15min":38.195999999999998},"last":0.70699999999999996}]},{"osd":2,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.139,"15min":0.76500000000000001},"min":{"1min":0.26800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.2629999999999999,"5min":7.6779999999999999,"15min":7.6779999999999999},"last":5.0590000000000002},{"interface":"front","average":{"1min":1.575,"5min":1.3049999999999999,"15min":0.84499999999999997},"min":{"1min":0.28499999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.8600000000000003,"5min":8.9100000000000001,"15min":8.9100000000000001},"last":0.92200000000000004}]},{"osd":3,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":2.0009999999999999,"15min":1.101},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":2.7050000000000001,"5min":58.515000000000001,"15min":58.515000000000001},"last":1.028},{"interface":"front","average":{"1min":1.4079999999999999,"5min":2.0430000000000001,"15min":1.0780000000000001},"min":{"1min":0.27400000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.6879999999999997,"5min":58.225999999999999,"15min":58.225999999999999},"last":1.274}]},{"osd":4,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":0.624,"15min":0.498},"min":{"1min":0.23699999999999999,"5min":0.23100000000000001,"15min":0.22},"max":{"1min":19.838999999999999,"5min":19.838999999999999,"15min":19.838999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.6419999999999999,"5min":0.65400000000000003,"15min":0.50900000000000001},"min":{"1min":0.28999999999999998,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":20.113,"5min":20.113,"15min":20.113},"last":0.95699999999999996}]},{"osd":5,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.46999999999999997,"15min":0.45400000000000001},"min":{"1min":0.251,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.224,"5min":1.752,"15min":1.752},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.47799999999999998,"15min":0.48699999999999999},"min":{"1min":0.33900000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.80300000000000005,"5min":0.99199999999999999,"15min":0.998},"last":0.67300000000000004}]},{"osd":6,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47399999999999998,"15min":0.496},"min":{"1min":0.32800000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72299999999999998,"5min":1.0269999999999999,"15min":1.256},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.5,"15min":0.48699999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":1.0049999999999999}]}]},{"osd":6,"up_from":51,"seq":219043332205,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":643488,"kb_used_data":558168,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93101664,"statfs":{"total":95995035648,"available":95336103936,"internally_reserved":0,"allocated":571564032,"data_stored":571219671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87357894},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:12:15 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":1.0309999999999999,"15min":0.76700000000000002},"min":{"1min":0.23899999999999999,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":2.3090000000000002,"5min":12.096,"15min":12.096},"last":0.22900000000000001},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.448,"15min":0.88800000000000001},"min":{"1min":0.309,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":6.4100000000000001,"5min":12.675000000000001,"15min":12.675000000000001},"last":0.33900000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:12:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.135,"15min":0.78200000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.4609999999999999,"5min":6.0739999999999998,"15min":6.0739999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.032,"5min":1.143,"15min":0.75800000000000001},"min":{"1min":0.30199999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.4940000000000002,"5min":4.4349999999999996,"15min":4.4349999999999996},"last":0.59999999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:12:15 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.23,"15min":0.80000000000000004},"min":{"1min":0.38100000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.9800000000000004,"5min":6.282,"15min":6.282},"last":1.361},{"interface":"front","average":{"1min":1.482,"5min":1.1639999999999999,"15min":0.78400000000000003},"min":{"1min":0.33500000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.2699999999999996,"5min":6.2699999999999996,"15min":6.2699999999999996},"last":1.175}]},{"osd":3,"last update":"Mon Feb 5 08:12:15 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":1.3140000000000001,"15min":0.85199999999999998},"min":{"1min":0.28599999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.1030000000000002,"5min":12.32,"15min":12.32},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":1.294,"15min":0.83099999999999996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.3080000000000001,"5min":12.762,"15min":12.762},"last":0.83599999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:12:15 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.42099999999999999,"15min":0.433},"min":{"1min":0.26100000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.60199999999999998,"5min":1.034,"15min":1.034},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.47199999999999998,"15min":0.45700000000000002},"min":{"1min":0.30199999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.6399999999999999,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.1240000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:12:15 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.51200000000000001,"15min":0.49199999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.165,"5min":6.165,"15min":6.165},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.57799999999999996,"15min":0.51800000000000002},"min":{"1min":0.33900000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.2229999999999999,"5min":6.2229999999999999,"15min":6.2229999999999999},"last":0.43099999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53500000000000003,"15min":0.52600000000000002},"min":{"1min":0.34799999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.5329999999999999,"5min":3.5529999999999999,"15min":3.5529999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.33700000000000002,"15min":0.29299999999999998},"max":{"1min":0.59299999999999997,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.75600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574982,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3297316,"kb_used_data":2999580,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":90447836,"statfs":{"total":95995035648,"available":92618584064,"internally_reserved":0,"allocated":3071569920,"data_stored":3071221407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:12:08 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":2.6619999999999999,"15min":1.1060000000000001},"min":{"1min":0.254,"5min":0.186,"15min":0.14699999999999999},"max":{"1min":5.2699999999999996,"5min":86.105000000000004,"15min":86.105000000000004},"last":4.4089999999999998},{"interface":"front","average":{"1min":2.4809999999999999,"5min":3.2320000000000002,"15min":1.3069999999999999},"min":{"1min":0.34499999999999997,"5min":0.245,"15min":0.16},"max":{"1min":6.9500000000000002,"5min":85.665000000000006,"15min":85.665000000000006},"last":1.732}]},{"osd":2,"last update":"Mon Feb 5 08:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.6200000000000001,"15min":1.0529999999999999},"min":{"1min":0.219,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":3.9750000000000001,"5min":9.3840000000000003,"15min":49.030999999999999},"last":1.627},{"interface":"front","average":{"1min":1.431,"5min":1.831,"15min":1.1679999999999999},"min":{"1min":0.23599999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.1459999999999999,"5min":13.707000000000001,"15min":48.984000000000002},"last":1.6890000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:12:00 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":2.4950000000000001,"15min":1.121},"min":{"1min":0.34499999999999997,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":2.8380000000000001,"5min":49.067999999999998,"15min":49.067999999999998},"last":1.8919999999999999},{"interface":"front","average":{"1min":1.3660000000000001,"5min":2.5230000000000001,"15min":1.1459999999999999},"min":{"1min":0.31,"5min":0.27800000000000002,"15min":0.24199999999999999},"max":{"1min":6.8029999999999999,"5min":48.734999999999999,"15min":48.734999999999999},"last":2.04}]},{"osd":4,"last update":"Mon Feb 5 08:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":1.087,"15min":0.76300000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.593,"5min":9.4269999999999996,"15min":9.4269999999999996},"last":1.6739999999999999},{"interface":"front","average":{"1min":1.224,"5min":1.3420000000000001,"15min":0.83699999999999997},"min":{"1min":0.30199999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.258,"5min":6.3890000000000002,"15min":6.3890000000000002},"last":1.8520000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:12:51 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.173,"15min":0.80800000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.4430000000000001,"5min":9.4559999999999995,"15min":9.4559999999999995},"last":1.9850000000000001},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.355,"15min":0.878},"min":{"1min":0.47999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.9770000000000001,"5min":13.776,"15min":13.776},"last":1.5980000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:12:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.373,"15min":0.85499999999999998},"min":{"1min":0.38900000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.3449999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":1.708},{"interface":"front","average":{"1min":0.73099999999999998,"5min":1.2030000000000001,"15min":0.82799999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.6519999999999999,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":2.323}]},{"osd":7,"last update":"Mon Feb 5 08:12:37 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.494,"15min":0.94299999999999995},"min":{"1min":0.55500000000000005,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.974,"5min":13.749000000000001,"15min":13.749000000000001},"last":1.875},{"interface":"front","average":{"1min":1.466,"5min":1.4550000000000001,"15min":0.92300000000000004},"min":{"1min":0.48399999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.585,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":2.1970000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738507,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3278544,"kb_used_data":2990344,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":90466608,"statfs":{"total":95995035648,"available":92637806592,"internally_reserved":0,"allocated":3062112256,"data_stored":3061784692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:12:17 2024","interfaces":[{"interface":"back","average":{"1min":3.3100000000000001,"5min":3.5270000000000001,"15min":1.4359999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.152},"max":{"1min":16.228000000000002,"5min":66.772999999999996,"15min":66.772999999999996},"last":3.7229999999999999},{"interface":"front","average":{"1min":2.7719999999999998,"5min":2.5569999999999999,"15min":1.097},"min":{"1min":0.218,"5min":0.218,"15min":0.17199999999999999},"max":{"1min":11.247999999999999,"5min":46.198999999999998,"15min":46.198999999999998},"last":3.4180000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:12:43 2024","interfaces":[{"interface":"back","average":{"1min":2.0270000000000001,"5min":2.3010000000000002,"15min":1.1319999999999999},"min":{"1min":0.23699999999999999,"5min":0.216,"15min":0.216},"max":{"1min":7.8579999999999997,"5min":46.170999999999999,"15min":46.170999999999999},"last":2.2759999999999998},{"interface":"front","average":{"1min":2.5619999999999998,"5min":2.75,"15min":1.302},"min":{"1min":0.20399999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":9.1210000000000004,"5min":46.543999999999997,"15min":46.543999999999997},"last":3.7599999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:12:10 2024","interfaces":[{"interface":"back","average":{"1min":3.4660000000000002,"5min":3.1549999999999998,"15min":1.3700000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.21099999999999999},"max":{"1min":13.013,"5min":67.888000000000005,"15min":67.888000000000005},"last":2.8719999999999999},{"interface":"front","average":{"1min":3.157,"5min":3.855,"15min":1.601},"min":{"1min":0.38700000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":16.289999999999999,"5min":66.891999999999996,"15min":66.891999999999996},"last":2.4580000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:12:34 2024","interfaces":[{"interface":"back","average":{"1min":1.7669999999999999,"5min":1.415,"15min":0.85099999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":7.5709999999999997,"5min":13.295,"15min":13.295},"last":2.0390000000000001},{"interface":"front","average":{"1min":1.7210000000000001,"5min":1.458,"15min":0.90300000000000002},"min":{"1min":0.29999999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":10.928000000000001,"5min":12.989000000000001,"15min":12.989000000000001},"last":2.3490000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:12:52 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.454,"15min":0.96299999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.2190000000000003,"5min":9.1660000000000004,"15min":13.073},"last":2.1080000000000001},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.3260000000000001,"15min":0.93200000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.782,"5min":9.5289999999999999,"15min":12.94},"last":2.4169999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:12:17 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.4650000000000001,"15min":0.90700000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":5.8559999999999999,"5min":12.967000000000001,"15min":12.967000000000001},"last":2.3889999999999998},{"interface":"front","average":{"1min":1.9710000000000001,"5min":1.6040000000000001,"15min":0.95899999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":9.1929999999999996,"5min":13.108000000000001,"15min":13.108000000000001},"last":2.141}]},{"osd":7,"last update":"Mon Feb 5 08:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.9379999999999999,"5min":1.3300000000000001,"15min":0.878},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":7.6020000000000003,"5min":7.6020000000000003,"15min":13.263},"last":1.99},{"interface":"front","average":{"1min":2.1869999999999998,"5min":1.4650000000000001,"15min":0.98199999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":6.718,"5min":9.6300000000000008,"15min":13.012},"last":2.4470000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411457,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4139608,"kb_used_data":3865424,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":89605544,"statfs":{"total":95995035648,"available":91756077056,"internally_reserved":0,"allocated":3958194176,"data_stored":3957857573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:12:46 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":3.073,"15min":1.3640000000000001},"min":{"1min":0.26200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":40.354999999999997,"5min":40.354999999999997,"15min":40.354999999999997},"last":0.64200000000000002},{"interface":"front","average":{"1min":6.2599999999999998,"5min":3.206,"15min":1.419},"min":{"1min":0.35099999999999998,"5min":0.24199999999999999,"15min":0.20699999999999999},"max":{"1min":40.566000000000003,"5min":40.566000000000003,"15min":40.566000000000003},"last":5.7350000000000003}]},{"osd":1,"last update":"Mon Feb 5 08:12:46 2024","interfaces":[{"interface":"back","average":{"1min":4.875,"5min":2.6560000000000001,"15min":1.2130000000000001},"min":{"1min":0.20000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":55.942,"5min":55.942,"15min":55.942},"last":0.71299999999999997},{"interface":"front","average":{"1min":5.5519999999999996,"5min":2.8359999999999999,"15min":1.2709999999999999},"min":{"1min":0.22900000000000001,"5min":0.193,"15min":0.193},"max":{"1min":55.960999999999999,"5min":55.960999999999999,"15min":55.960999999999999},"last":5.6890000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:12:02 2024","interfaces":[{"interface":"back","average":{"1min":2.2570000000000001,"5min":3.5059999999999998,"15min":1.4770000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.5490000000000004,"5min":68.108000000000004,"15min":68.108000000000004},"last":5.7569999999999997},{"interface":"front","average":{"1min":1.597,"5min":3.1680000000000001,"15min":1.3560000000000001},"min":{"1min":0.309,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":6.9279999999999999,"5min":67.945999999999998,"15min":67.945999999999998},"last":0.55800000000000005}]},{"osd":4,"last update":"Mon Feb 5 08:12:24 2024","interfaces":[{"interface":"back","average":{"1min":2.0150000000000001,"5min":2.0470000000000002,"15min":1.095},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":7.4359999999999999,"5min":31.596,"15min":31.596},"last":5.6299999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.617,"15min":0.93999999999999995},"min":{"1min":0.33100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":6.149,"5min":31.57,"15min":31.57},"last":1.01}]},{"osd":5,"last update":"Mon Feb 5 08:11:52 2024","interfaces":[{"interface":"back","average":{"1min":4.3220000000000001,"5min":1.968,"15min":1.0469999999999999},"min":{"1min":0.29999999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":58.799999999999997,"5min":58.799999999999997,"15min":58.799999999999997},"last":4.2080000000000002},{"interface":"front","average":{"1min":4.8440000000000003,"5min":1.9750000000000001,"15min":1.04},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":58.615000000000002,"5min":58.615000000000002,"15min":58.615000000000002},"last":3.8879999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:12:24 2024","interfaces":[{"interface":"back","average":{"1min":2.0110000000000001,"5min":1.641,"15min":0.98299999999999998},"min":{"1min":0.36299999999999999,"5min":0.308,"15min":0.308},"max":{"1min":7.5099999999999998,"5min":15.859,"15min":15.859},"last":5.6639999999999997},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.2729999999999999,"15min":0.86199999999999999},"min":{"1min":0.41199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":6.1820000000000004,"5min":6.1820000000000004,"15min":6.1820000000000004},"last":0.91300000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:12:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8320000000000001,"5min":1.6870000000000001,"15min":0.99299999999999999},"min":{"1min":0.40100000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":6.5490000000000004,"5min":15.986000000000001,"15min":15.986000000000001},"last":1.042},{"interface":"front","average":{"1min":1.9970000000000001,"5min":1.7410000000000001,"15min":1.0529999999999999},"min":{"1min":0.40400000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":7.4779999999999998,"5min":15.951000000000001,"15min":15.951000000000001},"last":5.7119999999999997}]}]},{"osd":3,"up_from":0,"seq":115964117116,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953591,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2546632,"kb_used_data":2371072,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91198520,"statfs":{"total":95995035648,"available":93387284480,"internally_reserved":0,"allocated":2427977728,"data_stored":2427656606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:12:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":1.343,"15min":1.1240000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.0939999999999999,"5min":27.966000000000001,"15min":73.078999999999994},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.268,"15min":1.0980000000000001},"min":{"1min":0.28000000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.5750000000000002,"5min":26.097999999999999,"15min":73.412000000000006},"last":0.433}]},{"osd":1,"last update":"Mon Feb 5 08:12:14 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.113,"15min":0.73399999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":3.3069999999999999,"5min":8.3409999999999993,"15min":8.3409999999999993},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.80900000000000005,"15min":0.63400000000000001},"min":{"1min":0.249,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.621,"5min":4.3120000000000003,"15min":4.3120000000000003},"last":0.27600000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:12:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":0.92700000000000005,"15min":1.0369999999999999},"min":{"1min":0.252,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.649,"5min":5.9640000000000004,"15min":96.849999999999994},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0069999999999999,"15min":1.069},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.1299999999999999,"5min":7.032,"15min":96.143000000000001},"last":2.1589999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:12:14 2024","interfaces":[{"interface":"back","average":{"1min":3.8100000000000001,"5min":2.4009999999999998,"15min":1.4370000000000001},"min":{"1min":0.29099999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":57.204000000000001,"5min":57.204000000000001,"15min":73.602999999999994},"last":1.9610000000000001},{"interface":"front","average":{"1min":4.2110000000000003,"5min":2.4820000000000002,"15min":1.4710000000000001},"min":{"1min":0.27900000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":57.606999999999999,"5min":57.606999999999999,"15min":74.817999999999998},"last":1.792}]},{"osd":5,"last update":"Mon Feb 5 08:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.41099999999999998,"15min":0.39600000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.45400000000000001,"15min":0.42699999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.252,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.48799999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:12:14 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.46600000000000003,"15min":0.44800000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29299999999999998},"max":{"1min":0.78000000000000003,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.46899999999999997,"15min":0.44700000000000001},"min":{"1min":0.32400000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.74099999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.40899999999999997}]},{"osd":7,"last update":"Mon Feb 5 08:12:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.33100000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.88800000000000001,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.5,"15min":0.51000000000000001},"min":{"1min":0.30099999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0349999999999999,"5min":1.147,"15min":1.147},"last":0.92600000000000005}]}]},{"osd":5,"up_from":42,"seq":180388626545,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3668096,"kb_used_data":3434232,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":90077056,"statfs":{"total":95995035648,"available":92238905344,"internally_reserved":0,"allocated":3516653568,"data_stored":3516341509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":239466949},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:12:48 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.831,"15min":1.409},"min":{"1min":0.252,"5min":0.22800000000000001,"15min":0.19400000000000001},"max":{"1min":3.7370000000000001,"5min":46.962000000000003,"15min":143.06299999999999},"last":2.077},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.8320000000000001,"15min":1.411},"min":{"1min":0.30299999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4859999999999998,"5min":47.067,"15min":143.09700000000001},"last":3.2599999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.379,"15min":0.84199999999999997},"min":{"1min":0.25,"5min":0.23200000000000001,"15min":0.215},"max":{"1min":2.5099999999999998,"5min":46.200000000000003,"15min":46.200000000000003},"last":0.27100000000000002},{"interface":"front","average":{"1min":1.903,"5min":1.863,"15min":0.98399999999999999},"min":{"1min":0.44500000000000001,"5min":0.246,"15min":0.22900000000000001},"max":{"1min":5.4809999999999999,"5min":46.343000000000004,"15min":46.343000000000004},"last":2.3399999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:12:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.256,"15min":0.81000000000000005},"min":{"1min":0.26700000000000002,"5min":0.246,"15min":0.246},"max":{"1min":4.0629999999999997,"5min":10.733000000000001,"15min":10.733000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.389,"15min":0.84099999999999997},"min":{"1min":0.309,"5min":0.28399999999999997,"15min":0.26400000000000001},"max":{"1min":8.0220000000000002,"5min":8.0220000000000002,"15min":8.0220000000000002},"last":2.0379999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:12:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.9099999999999999,"15min":1.472},"min":{"1min":0.36199999999999999,"5min":0.309,"15min":0.309},"max":{"1min":4.3330000000000002,"5min":46.569000000000003,"15min":143.03399999999999},"last":1.21},{"interface":"front","average":{"1min":1.8069999999999999,"5min":1.9750000000000001,"15min":1.4770000000000001},"min":{"1min":0.32600000000000001,"5min":0.28899999999999998,"15min":0.28100000000000003},"max":{"1min":7.5270000000000001,"5min":47.820999999999998,"15min":143.46799999999999},"last":2.0070000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.47199999999999998,"15min":0.42599999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.839,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.309},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.45200000000000001,"15min":0.434},"min":{"1min":0.28299999999999997,"5min":0.26900000000000002,"15min":0.26500000000000001},"max":{"1min":0.53100000000000003,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.39800000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:12:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47699999999999998,"15min":0.45800000000000002},"min":{"1min":0.313,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.51900000000000002,"15min":0.5},"min":{"1min":0.34200000000000003,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.78800000000000003,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.45200000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53200000000000003,"15min":0.49399999999999999},"min":{"1min":0.34699999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.0089999999999999,"5min":1.26,"15min":1.26},"last":0.44},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.53600000000000003,"15min":0.502},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.28199999999999997},"max":{"1min":0.82399999999999995,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.46700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3048996864,"data_stored":3048996864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3048341504,"data_stored":3048341504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3938848768,"data_stored":3938844696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3037462528,"data_stored":3037462528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2432237568,"data_stored":2432237568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3509653504,"data_stored":3509649432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":586743808,"data_stored":586743808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2194739200,"data_stored":2194735128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:13:03.606 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-02-05T08:13:03.695 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:03.736 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:03.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:03.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:03.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:03.880 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:03.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:03.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:04.019 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:04.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:04.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:04.156 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:04.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:04.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:04.175 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:03 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:04.177 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:03 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:04.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:04.291 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:04.297 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:04.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:03 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:04.355 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:04.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:04.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:04.457 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:04.500 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:04.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:04.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:04.668 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:04.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:04.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:04.798 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:04.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:04.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:04.932 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:04.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:05.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:04 smithi134 ceph-mon[30086]: pgmap v351: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 953 op/s; 41655/295038 objects misplaced (14.119%); 55 MiB/s, 879 objects/s recovering 2024-02-05T08:13:05.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:04 smithi134 ceph-mon[30086]: osdmap e200: 8 total, 8 up, 7 in 2024-02-05T08:13:05.007 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:04 smithi134 ceph-mon[39278]: pgmap v351: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 953 op/s; 41655/295038 objects misplaced (14.119%); 55 MiB/s, 879 objects/s recovering 2024-02-05T08:13:05.007 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:04 smithi134 ceph-mon[39278]: osdmap e200: 8 total, 8 up, 7 in 2024-02-05T08:13:05.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:05.061 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:05.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:05.162 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:05.190 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:05.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:05.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:05.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:04 smithi196 ceph-mon[33018]: pgmap v351: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 953 op/s; 41655/295038 objects misplaced (14.119%); 55 MiB/s, 879 objects/s recovering 2024-02-05T08:13:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:04 smithi196 ceph-mon[33018]: osdmap e200: 8 total, 8 up, 7 in 2024-02-05T08:13:05.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:05.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:05.427 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:05.459 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:05.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:05.560 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:05.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:05.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:05.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:05.677 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:13:05.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:05.720 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:05.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:05.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:05.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:05.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:05.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:05.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:05.989 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:05.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:05.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:06.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:05 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1596829454' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:13:06.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:05 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:13:06.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:05 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1596829454' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:13:06.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:05 smithi134 ceph-mon[30086]: osdmap e201: 8 total, 8 up, 7 in 2024-02-05T08:13:06.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:05 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1596829454' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:13:06.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:05 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:13:06.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:05 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1596829454' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:13:06.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:05 smithi134 ceph-mon[39278]: osdmap e201: 8 total, 8 up, 7 in 2024-02-05T08:13:06.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:06.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:06.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:06.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:06.222 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:06.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:06.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:06.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:05 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1596829454' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:13:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:05 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:13:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:05 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1596829454' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:13:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:05 smithi196 ceph-mon[33018]: osdmap e201: 8 total, 8 up, 7 in 2024-02-05T08:13:06.358 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:06.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:06.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:06.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:06.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:06.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:06.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:06.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:06.660 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:06.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:06.762 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:06.801 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:06.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:06.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:06.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:06.942 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:06.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:06.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:07.044 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:07.089 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:07.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:07.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:07.224 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:07.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:07.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:07.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:06 smithi134 ceph-mon[30086]: pgmap v353: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 842 op/s; 41655/295038 objects misplaced (14.119%); 49 MiB/s, 788 objects/s recovering 2024-02-05T08:13:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:06 smithi134 ceph-mon[39278]: pgmap v353: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 842 op/s; 41655/295038 objects misplaced (14.119%); 49 MiB/s, 788 objects/s recovering 2024-02-05T08:13:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:06 smithi196 ceph-mon[33018]: pgmap v353: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 842 op/s; 41655/295038 objects misplaced (14.119%); 49 MiB/s, 788 objects/s recovering 2024-02-05T08:13:07.326 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:07.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:07.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:07.458 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:07.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:07.501 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:07.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:07.602 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:07.668 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:07.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:07.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:07.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:07.815 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:07.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:07.890 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:13:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:13:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:13:07.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:07.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:07.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:07.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:08.054 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:08.108 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:08.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:08.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:08.210 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:08.248 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:08.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:07 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:13:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:07 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:13:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:07 smithi134 ceph-mon[30086]: osdmap e202: 8 total, 8 up, 7 in 2024-02-05T08:13:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:07 smithi134 ceph-mon[30086]: 8.0 scrub starts 2024-02-05T08:13:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:13:08.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:07 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:13:08.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:07 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:13:08.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:07 smithi134 ceph-mon[39278]: osdmap e202: 8 total, 8 up, 7 in 2024-02-05T08:13:08.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:07 smithi134 ceph-mon[39278]: 8.0 scrub starts 2024-02-05T08:13:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:13:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:07 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:13:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:07 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:13:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:07 smithi196 ceph-mon[33018]: osdmap e202: 8 total, 8 up, 7 in 2024-02-05T08:13:08.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:07 smithi196 ceph-mon[33018]: 8.0 scrub starts 2024-02-05T08:13:08.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:13:08.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:08.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:08.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:08.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:08.524 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:08.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:08.607 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:13:08.608 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:13:08.608 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:13:08.609 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:13:08.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:08.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:08.657 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:08.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:08.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:08.845 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:08.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:08.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:08.904 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:08 smithi134 ceph-mon[30086]: pgmap v356: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 879 B/s rd, 0 B/s wr, 661 op/s; 41655/295038 objects misplaced (14.119%); 42 MiB/s, 679 objects/s recovering 2024-02-05T08:13:08.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:09.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:09.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:09.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:09.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:09.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:09.241 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:09.264 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:09.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:09.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:08 smithi134 ceph-mon[39278]: pgmap v356: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 879 B/s rd, 0 B/s wr, 661 op/s; 41655/295038 objects misplaced (14.119%); 42 MiB/s, 679 objects/s recovering 2024-02-05T08:13:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:08 smithi196 ceph-mon[33018]: pgmap v356: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 879 B/s rd, 0 B/s wr, 661 op/s; 41655/295038 objects misplaced (14.119%); 42 MiB/s, 679 objects/s recovering 2024-02-05T08:13:09.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:09.408 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:09.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:09.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:09.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:09.547 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:09.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:09.648 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:09.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:09.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:09.784 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:09.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:09.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:09.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:09.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:09.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:09.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:09.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:10.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:10.082 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:10.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:09 smithi134 ceph-mon[30086]: 8.0 scrub ok 2024-02-05T08:13:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:09 smithi134 ceph-mon[30086]: 5.14 scrub starts 2024-02-05T08:13:10.172 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:09 smithi134 ceph-mon[30086]: 5.14 scrub ok 2024-02-05T08:13:10.172 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:09 smithi134 ceph-mon[39278]: 8.0 scrub ok 2024-02-05T08:13:10.172 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:09 smithi134 ceph-mon[39278]: 5.14 scrub starts 2024-02-05T08:13:10.172 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:09 smithi134 ceph-mon[39278]: 5.14 scrub ok 2024-02-05T08:13:10.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:10.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:10.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:10.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:09 smithi196 ceph-mon[33018]: 8.0 scrub ok 2024-02-05T08:13:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:09 smithi196 ceph-mon[33018]: 5.14 scrub starts 2024-02-05T08:13:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:09 smithi196 ceph-mon[33018]: 5.14 scrub ok 2024-02-05T08:13:10.314 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:10.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:10.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:10.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:10.446 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:10.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:10.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:10.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:10.602 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:10.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:10.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:10.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:10.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:10.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:10.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:10.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:10.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:10.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:11.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:11.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:11.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:11.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:11.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:11.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:11.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:11.277 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:11.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:11.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:10 smithi134 ceph-mon[30086]: pgmap v357: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 851 B/s rd, 0 B/s wr, 602 op/s; 35783/284187 objects misplaced (12.591%); 46 MiB/s, 743 objects/s recovering 2024-02-05T08:13:11.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:10 smithi134 ceph-mon[30086]: 4.17 scrub starts 2024-02-05T08:13:11.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:10 smithi134 ceph-mon[30086]: 4.17 scrub ok 2024-02-05T08:13:11.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:10 smithi134 ceph-mon[39278]: pgmap v357: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 851 B/s rd, 0 B/s wr, 602 op/s; 35783/284187 objects misplaced (12.591%); 46 MiB/s, 743 objects/s recovering 2024-02-05T08:13:11.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:10 smithi134 ceph-mon[39278]: 4.17 scrub starts 2024-02-05T08:13:11.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:10 smithi134 ceph-mon[39278]: 4.17 scrub ok 2024-02-05T08:13:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:10 smithi196 ceph-mon[33018]: pgmap v357: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 851 B/s rd, 0 B/s wr, 602 op/s; 35783/284187 objects misplaced (12.591%); 46 MiB/s, 743 objects/s recovering 2024-02-05T08:13:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:10 smithi196 ceph-mon[33018]: 4.17 scrub starts 2024-02-05T08:13:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:10 smithi196 ceph-mon[33018]: 4.17 scrub ok 2024-02-05T08:13:11.378 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:11.429 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:11.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:11.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:11.474 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:13:11.475 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:13:11.475 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-02-05T08:13:11.475 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-02-05T08:13:11.475 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:13:11.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:11.570 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:11.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:11.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:11.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:11.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:11.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:11.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:11.850 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:11.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:11.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:11.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:12.001 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:12.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:12.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:12.141 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:12.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:12.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:12.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:11 smithi134 ceph-mon[30086]: 3.16 scrub starts 2024-02-05T08:13:12.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:11 smithi134 ceph-mon[30086]: 3.16 scrub ok 2024-02-05T08:13:12.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:11 smithi134 ceph-mon[39278]: 3.16 scrub starts 2024-02-05T08:13:12.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:11 smithi134 ceph-mon[39278]: 3.16 scrub ok 2024-02-05T08:13:12.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:12.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:12.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:11 smithi196 ceph-mon[33018]: 3.16 scrub starts 2024-02-05T08:13:12.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:11 smithi196 ceph-mon[33018]: 3.16 scrub ok 2024-02-05T08:13:12.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:12.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:12.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:12.524 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:12.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:12.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:12.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:12.689 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:12.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:12.790 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:12.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T08:13:12.830 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:12.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:12.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:12.932 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:13.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:13.003 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:13.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:13.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:12 smithi134 ceph-mon[30086]: pgmap v358: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 732 B/s rd, 0 B/s wr, 518 op/s; 35783/284187 objects misplaced (12.591%); 40 MiB/s, 639 objects/s recovering 2024-02-05T08:13:13.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:12 smithi134 ceph-mon[30086]: 3.11 scrub starts 2024-02-05T08:13:13.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:12 smithi134 ceph-mon[30086]: 3.11 scrub ok 2024-02-05T08:13:13.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:12 smithi134 ceph-mon[39278]: pgmap v358: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 732 B/s rd, 0 B/s wr, 518 op/s; 35783/284187 objects misplaced (12.591%); 40 MiB/s, 639 objects/s recovering 2024-02-05T08:13:13.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:12 smithi134 ceph-mon[39278]: 3.11 scrub starts 2024-02-05T08:13:13.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:12 smithi134 ceph-mon[39278]: 3.11 scrub ok 2024-02-05T08:13:13.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:13.144 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:13.160 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:13.166 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:13:13.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:13.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:13.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:12 smithi196 ceph-mon[33018]: pgmap v358: 148 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+clean+scrubbing, 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 138 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 732 B/s rd, 0 B/s wr, 518 op/s; 35783/284187 objects misplaced (12.591%); 40 MiB/s, 639 objects/s recovering 2024-02-05T08:13:13.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:12 smithi196 ceph-mon[33018]: 3.11 scrub starts 2024-02-05T08:13:13.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:12 smithi196 ceph-mon[33018]: 3.11 scrub ok 2024-02-05T08:13:13.216 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:13:13.261 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:13:13.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:13.298 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:13:13.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:13.322 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:13.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:13.337 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:13:13.377 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:13:13.418 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:13:13.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:13.458 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:13:13.469 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:13:13.469 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:13:13.469 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:13:13.469 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:13:13.469 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:13:13.469 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:13:13.470 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:13:13.470 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:13:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:13.473 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:13.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:13.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:13:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:13:13.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:13.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:13.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:13.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:13.716 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:13.820 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:13.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:13.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:13.966 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:13.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:14.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:14.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:14.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:14.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:14.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:14.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:14.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:14.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:14.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:13 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/882326700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:13:14.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:13 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:14.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:13 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/882326700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:13:14.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:13 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:13 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/882326700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:13:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:13 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:14.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:14.362 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:14.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:14.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:14.495 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:14.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:14.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:14.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:14.620 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:14.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:14.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:14.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:14.754 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:14.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:14.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:14.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:14.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:14.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:14.988 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:15.022 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:15.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:15.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:15.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:15.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:15.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:15.259 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:15.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:14 smithi134 ceph-mon[30086]: pgmap v359: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 139 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 31930/270858 objects misplaced (11.788%); 52 MiB/s, 839 objects/s recovering 2024-02-05T08:13:15.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:14 smithi134 ceph-mon[30086]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:13:15.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:14 smithi134 ceph-mon[30086]: osdmap e203: 8 total, 8 up, 7 in 2024-02-05T08:13:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:14 smithi134 ceph-mon[39278]: pgmap v359: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 139 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 31930/270858 objects misplaced (11.788%); 52 MiB/s, 839 objects/s recovering 2024-02-05T08:13:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:14 smithi134 ceph-mon[39278]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:13:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:14 smithi134 ceph-mon[39278]: osdmap e203: 8 total, 8 up, 7 in 2024-02-05T08:13:15.292 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:15.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:15.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:14 smithi196 ceph-mon[33018]: pgmap v359: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 139 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 31930/270858 objects misplaced (11.788%); 52 MiB/s, 839 objects/s recovering 2024-02-05T08:13:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:14 smithi196 ceph-mon[33018]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:13:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:14 smithi196 ceph-mon[33018]: osdmap e203: 8 total, 8 up, 7 in 2024-02-05T08:13:15.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:15.432 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:15.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:15.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:15.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:15.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:15.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:15.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:15.714 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:15.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:15.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:15.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:15.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:15.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:15.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:15.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:15.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:16.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:16.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:16.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:16.220 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:16.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:15 smithi134 ceph-mon[30086]: osdmap e204: 8 total, 8 up, 7 in 2024-02-05T08:13:16.237 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:15 smithi134 ceph-mon[39278]: osdmap e204: 8 total, 8 up, 7 in 2024-02-05T08:13:16.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:16.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:16.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:15 smithi196 ceph-mon[33018]: osdmap e204: 8 total, 8 up, 7 in 2024-02-05T08:13:16.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:16.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:16.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:16.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:16.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:16.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:16.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:16.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:16.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:16.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:16.767 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:16.828 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:16.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:16.929 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:16.974 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:16.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:16.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:17.075 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:17.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:17.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:17.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:17.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:17.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:17.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:16 smithi134 ceph-mon[30086]: pgmap v362: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 139 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 31930/270858 objects misplaced (11.788%); 52 MiB/s, 839 objects/s recovering 2024-02-05T08:13:17.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:16 smithi134 ceph-mon[39278]: pgmap v362: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 139 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 31930/270858 objects misplaced (11.788%); 52 MiB/s, 839 objects/s recovering 2024-02-05T08:13:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:16 smithi196 ceph-mon[33018]: pgmap v362: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 5 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 139 active+clean; 5.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 31930/270858 objects misplaced (11.788%); 52 MiB/s, 839 objects/s recovering 2024-02-05T08:13:17.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:17.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:17.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:17.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:17.513 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:17.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:17.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:17.647 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:17.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:17.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:17.749 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:17.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:17.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:17.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:17.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:17.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:17.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:18.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:18.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:13:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:13:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:13:18.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:18.072 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:18.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:18.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:18.211 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:18.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:18.313 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:18.342 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:18.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:18.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:18.473 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:18.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:18.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:18.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:18.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:18.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:18.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:18.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:18.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:18.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:18.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:18.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:18.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:18.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:19.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:19.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:19.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:19.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:19.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:19.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:19.239 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:19.257 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:18 smithi134 ceph-mon[30086]: pgmap v363: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfilling, 140 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 760 op/s; 30021/265959 objects misplaced (11.288%); 29 MiB/s, 464 objects/s recovering 2024-02-05T08:13:19.257 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:18 smithi134 ceph-mon[30086]: osdmap e205: 8 total, 8 up, 7 in 2024-02-05T08:13:19.258 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:18 smithi134 ceph-mon[30086]: 6.3 scrub starts 2024-02-05T08:13:19.258 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:18 smithi134 ceph-mon[30086]: 6.3 scrub ok 2024-02-05T08:13:19.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:18 smithi134 ceph-mon[39278]: pgmap v363: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfilling, 140 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 760 op/s; 30021/265959 objects misplaced (11.288%); 29 MiB/s, 464 objects/s recovering 2024-02-05T08:13:19.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:18 smithi134 ceph-mon[39278]: osdmap e205: 8 total, 8 up, 7 in 2024-02-05T08:13:19.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:18 smithi134 ceph-mon[39278]: 6.3 scrub starts 2024-02-05T08:13:19.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:18 smithi134 ceph-mon[39278]: 6.3 scrub ok 2024-02-05T08:13:19.279 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:19.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:19.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:18 smithi196 ceph-mon[33018]: pgmap v363: 148 pgs: 2 active+remapped+backfill_wait+backfill_toofull, 6 active+remapped+backfilling, 140 active+clean; 5.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 760 op/s; 30021/265959 objects misplaced (11.288%); 29 MiB/s, 464 objects/s recovering 2024-02-05T08:13:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:18 smithi196 ceph-mon[33018]: osdmap e205: 8 total, 8 up, 7 in 2024-02-05T08:13:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:18 smithi196 ceph-mon[33018]: 6.3 scrub starts 2024-02-05T08:13:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:18 smithi196 ceph-mon[33018]: 6.3 scrub ok 2024-02-05T08:13:19.381 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:19.405 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:19.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:19.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:19.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:19.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:19.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:19.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:19.638 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:19.671 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:19.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:19.773 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:19.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:19.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:19.905 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:19.939 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:19.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:19.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:20.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:20.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:20.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:20.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:20.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:20.202 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:20 smithi134 ceph-mon[30086]: osdmap e206: 8 total, 8 up, 7 in 2024-02-05T08:13:20.202 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:20 smithi134 ceph-mon[30086]: 5.10 scrub starts 2024-02-05T08:13:20.202 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:20 smithi134 ceph-mon[30086]: 5.10 scrub ok 2024-02-05T08:13:20.202 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:20 smithi134 ceph-mon[30086]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:13:20.203 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:20 smithi134 ceph-mon[39278]: osdmap e206: 8 total, 8 up, 7 in 2024-02-05T08:13:20.203 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:20 smithi134 ceph-mon[39278]: 5.10 scrub starts 2024-02-05T08:13:20.203 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:20 smithi134 ceph-mon[39278]: 5.10 scrub ok 2024-02-05T08:13:20.203 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:20 smithi134 ceph-mon[39278]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:13:20.204 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:20.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:20.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:20 smithi196 ceph-mon[33018]: osdmap e206: 8 total, 8 up, 7 in 2024-02-05T08:13:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:20 smithi196 ceph-mon[33018]: 5.10 scrub starts 2024-02-05T08:13:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:20 smithi196 ceph-mon[33018]: 5.10 scrub ok 2024-02-05T08:13:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:20 smithi196 ceph-mon[33018]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:13:20.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:20.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:20.440 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:20.481 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:20.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:20.583 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:20.625 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:20.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:20.648 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:20.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:20.749 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:20.797 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:20.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:20.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:20.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:20.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:20.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:20.964 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:13:20.970 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:13:21.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:21 smithi134 ceph-mon[30086]: pgmap v366: 148 pgs: 1 peering, 7 active+remapped+backfilling, 140 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 894 B/s rd, 0 B/s wr, 652 op/s; 25458/259671 objects misplaced (9.804%); 57 MiB/s, 919 objects/s recovering 2024-02-05T08:13:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:21 smithi134 ceph-mon[30086]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-02-05T08:13:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:21 smithi134 ceph-mon[30086]: osdmap e207: 8 total, 8 up, 7 in 2024-02-05T08:13:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:21 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3251763827' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:13:21.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:21 smithi134 ceph-mon[39278]: pgmap v366: 148 pgs: 1 peering, 7 active+remapped+backfilling, 140 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 894 B/s rd, 0 B/s wr, 652 op/s; 25458/259671 objects misplaced (9.804%); 57 MiB/s, 919 objects/s recovering 2024-02-05T08:13:21.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:21 smithi134 ceph-mon[39278]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-02-05T08:13:21.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:21 smithi134 ceph-mon[39278]: osdmap e207: 8 total, 8 up, 7 in 2024-02-05T08:13:21.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:21 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3251763827' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:13:21.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:21.035 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:13:21.073 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:21.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:21.175 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:21.219 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:21.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:21.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:21 smithi196 ceph-mon[33018]: pgmap v366: 148 pgs: 1 peering, 7 active+remapped+backfilling, 140 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 894 B/s rd, 0 B/s wr, 652 op/s; 25458/259671 objects misplaced (9.804%); 57 MiB/s, 919 objects/s recovering 2024-02-05T08:13:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:21 smithi196 ceph-mon[33018]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-02-05T08:13:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:21 smithi196 ceph-mon[33018]: osdmap e207: 8 total, 8 up, 7 in 2024-02-05T08:13:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:21 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3251763827' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:13:21.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:21.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:21.361 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:21.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:21.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:21.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:21.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:21.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:21.638 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:21.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:21.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:21.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:13:21.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:21.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:21.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:21.871 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:21.924 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:21.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:21.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:22.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:22.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:22.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:22.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:22.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:22.230 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:22.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:22.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:22 smithi134 ceph-mon[30086]: from='client.25088 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:13:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:22 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:13:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:22 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:13:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:22 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3251763827' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:13:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:22 smithi134 ceph-mon[30086]: osdmap e208: 8 total, 8 up, 7 in 2024-02-05T08:13:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:22 smithi134 ceph-mon[39278]: from='client.25088 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:13:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:22 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:13:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:22 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:13:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:22 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3251763827' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:13:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:22 smithi134 ceph-mon[39278]: osdmap e208: 8 total, 8 up, 7 in 2024-02-05T08:13:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:22 smithi196 ceph-mon[33018]: from='client.25088 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:13:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:22 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:13:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:22 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:13:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:22 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3251763827' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:13:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:22 smithi196 ceph-mon[33018]: osdmap e208: 8 total, 8 up, 7 in 2024-02-05T08:13:22.331 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:22.372 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:22.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:22.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:22.507 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:22.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:22.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:22.608 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:22.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:22.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:22.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:22.767 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-02-05T08:13:20.009414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5660154228,"num_objects":86557,"num_object_clones":0,"num_object_copies":259671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":25458,"num_objects_unfound":0,"num_objects_dirty":86557,"num_whiteouts":0,"num_read":1434,"num_read_kb":1280,"num_write":147572,"num_write_kb":7468510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33682,"num_bytes_recovered":2194481415,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77528,"ondisk_log_size":77528,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":447,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17732248,"kb_used_data":16026964,"kb_used_omap":10,"kb_used_meta":1705013,"kb_avail":638483816,"statfs":{"total":671965249536,"available":653807427584,"internally_reserved":0,"allocated":16411611136,"data_stored":16409336176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":1745933418},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-244383744,"num_objects":-3729,"num_object_clones":0,"num_object_copies":-11187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6472,"num_objects_unfound":0,"num_objects_dirty":-3729,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3729,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5265,"num_bytes_recovered":344981504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.062218"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":307,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.950803+0000","last_change":"2024-02-05T08:12:32.976357+0000","last_active":"2024-02-05T08:13:14.950803+0000","last_peered":"2024-02-05T08:13:14.950803+0000","last_clean":"2024-02-05T08:13:14.950803+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:13:14.950803+0000","last_undegraded":"2024-02-05T08:13:14.950803+0000","last_fullsized":"2024-02-05T08:13:14.950803+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:32.976264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:32.976264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:39.644642+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":314,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.002054+0000","last_change":"2024-02-05T08:11:17.826990+0000","last_active":"2024-02-05T08:13:18.002054+0000","last_peered":"2024-02-05T08:13:18.002054+0000","last_clean":"2024-02-05T08:13:18.002054+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:13:18.002054+0000","last_undegraded":"2024-02-05T08:13:18.002054+0000","last_fullsized":"2024-02-05T08:13:18.002054+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:17.824305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:17.824305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":314,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.001339+0000","last_change":"2024-02-05T08:10:18.254802+0000","last_active":"2024-02-05T08:13:18.001339+0000","last_peered":"2024-02-05T08:13:18.001339+0000","last_clean":"2024-02-05T08:13:18.001339+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:13:18.001339+0000","last_undegraded":"2024-02-05T08:13:18.001339+0000","last_fullsized":"2024-02-05T08:13:18.001339+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:18.254568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:18.254568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"90'13","reported_seq":359,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.950445+0000","last_change":"2024-02-05T08:12:29.723024+0000","last_active":"2024-02-05T08:13:14.950445+0000","last_peered":"2024-02-05T08:13:14.950445+0000","last_clean":"2024-02-05T08:13:14.950445+0000","last_became_active":"2024-02-05T08:12:29.675863+0000","last_became_peered":"2024-02-05T08:12:29.675863+0000","last_unstale":"2024-02-05T08:13:14.950445+0000","last_undegraded":"2024-02-05T08:13:14.950445+0000","last_fullsized":"2024-02-05T08:13:14.950445+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:11:06.482560+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:11:06.482560+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":302,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117112+0000","last_change":"2024-02-05T08:12:41.723242+0000","last_active":"2024-02-05T08:13:15.117112+0000","last_peered":"2024-02-05T08:13:15.117112+0000","last_clean":"2024-02-05T08:13:15.117112+0000","last_became_active":"2024-02-05T08:12:41.722414+0000","last_became_peered":"2024-02-05T08:12:41.722414+0000","last_unstale":"2024-02-05T08:13:15.117112+0000","last_undegraded":"2024-02-05T08:13:15.117112+0000","last_fullsized":"2024-02-05T08:13:15.117112+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:30.552741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:30.552741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:54.257872+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":315,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.950564+0000","last_change":"2024-02-05T08:12:29.676957+0000","last_active":"2024-02-05T08:13:14.950564+0000","last_peered":"2024-02-05T08:13:14.950564+0000","last_clean":"2024-02-05T08:13:14.950564+0000","last_became_active":"2024-02-05T08:12:29.676482+0000","last_became_peered":"2024-02-05T08:12:29.676482+0000","last_unstale":"2024-02-05T08:13:14.950564+0000","last_undegraded":"2024-02-05T08:13:14.950564+0000","last_fullsized":"2024-02-05T08:13:14.950564+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:11:19.262611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":31,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953063+0000","last_change":"2024-02-05T08:12:41.727776+0000","last_active":"2024-02-05T08:13:14.953063+0000","last_peered":"2024-02-05T08:13:14.953063+0000","last_clean":"2024-02-05T08:13:14.953063+0000","last_became_active":"2024-02-05T08:12:41.727450+0000","last_became_peered":"2024-02-05T08:12:41.727450+0000","last_unstale":"2024-02-05T08:13:14.953063+0000","last_undegraded":"2024-02-05T08:13:14.953063+0000","last_fullsized":"2024-02-05T08:13:14.953063+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:28.007713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:28.007713+0000","last_clean_scrub_stamp":"2024-02-05T08:10:28.007713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"118'22","reported_seq":193,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953040+0000","last_change":"2024-02-05T08:12:41.771457+0000","last_active":"2024-02-05T08:13:14.953040+0000","last_peered":"2024-02-05T08:13:14.953040+0000","last_clean":"2024-02-05T08:13:14.953040+0000","last_became_active":"2024-02-05T08:12:41.745917+0000","last_became_peered":"2024-02-05T08:12:41.745917+0000","last_unstale":"2024-02-05T08:13:14.953040+0000","last_undegraded":"2024-02-05T08:13:14.953040+0000","last_fullsized":"2024-02-05T08:13:14.953040+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-02-05T08:11:07.726635+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.726635+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":301,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.001131+0000","last_change":"2024-02-05T08:10:29.223232+0000","last_active":"2024-02-05T08:13:18.001131+0000","last_peered":"2024-02-05T08:13:18.001131+0000","last_clean":"2024-02-05T08:13:18.001131+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:13:18.001131+0000","last_undegraded":"2024-02-05T08:13:18.001131+0000","last_fullsized":"2024-02-05T08:13:18.001131+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:29.223145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:10:29.223145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":311,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.962878+0000","last_change":"2024-02-05T08:12:15.282974+0000","last_active":"2024-02-05T08:13:14.962878+0000","last_peered":"2024-02-05T08:13:14.962878+0000","last_clean":"2024-02-05T08:13:14.962878+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:13:14.962878+0000","last_undegraded":"2024-02-05T08:13:14.962878+0000","last_fullsized":"2024-02-05T08:13:14.962878+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:15.282863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:12:15.282863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:25.141357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":331,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.963830+0000","last_change":"2024-02-05T08:12:29.684488+0000","last_active":"2024-02-05T08:13:14.963830+0000","last_peered":"2024-02-05T08:13:14.963830+0000","last_clean":"2024-02-05T08:13:14.963830+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:13:14.963830+0000","last_undegraded":"2024-02-05T08:13:14.963830+0000","last_fullsized":"2024-02-05T08:13:14.963830+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:20.316515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:20.316515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":345,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.950931+0000","last_change":"2024-02-05T08:12:41.731788+0000","last_active":"2024-02-05T08:13:14.950931+0000","last_peered":"2024-02-05T08:13:14.950931+0000","last_clean":"2024-02-05T08:13:14.950931+0000","last_became_active":"2024-02-05T08:12:41.731452+0000","last_became_peered":"2024-02-05T08:12:41.731452+0000","last_unstale":"2024-02-05T08:13:14.950931+0000","last_undegraded":"2024-02-05T08:13:14.950931+0000","last_fullsized":"2024-02-05T08:13:14.950931+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:26.162293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:26.162293+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":313,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.961115+0000","last_change":"2024-02-05T08:12:41.722105+0000","last_active":"2024-02-05T08:13:14.961115+0000","last_peered":"2024-02-05T08:13:14.961115+0000","last_clean":"2024-02-05T08:13:14.961115+0000","last_became_active":"2024-02-05T08:12:41.721611+0000","last_became_peered":"2024-02-05T08:12:41.721611+0000","last_unstale":"2024-02-05T08:13:14.961115+0000","last_undegraded":"2024-02-05T08:13:14.961115+0000","last_fullsized":"2024-02-05T08:13:14.961115+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:14.301656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:14.301656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:36.614851+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":312,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117004+0000","last_change":"2024-02-05T08:12:23.136907+0000","last_active":"2024-02-05T08:13:15.117004+0000","last_peered":"2024-02-05T08:13:15.117004+0000","last_clean":"2024-02-05T08:13:15.117004+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:13:15.117004+0000","last_undegraded":"2024-02-05T08:13:15.117004+0000","last_fullsized":"2024-02-05T08:13:15.117004+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:23.136787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:23.136787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:49.647033+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":319,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.963243+0000","last_change":"2024-02-05T08:12:12.261136+0000","last_active":"2024-02-05T08:13:14.963243+0000","last_peered":"2024-02-05T08:13:14.963243+0000","last_clean":"2024-02-05T08:13:14.963243+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:13:14.963243+0000","last_undegraded":"2024-02-05T08:13:14.963243+0000","last_fullsized":"2024-02-05T08:13:14.963243+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:12.261000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:12:12.261000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:16.750433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":329,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.951314+0000","last_change":"2024-02-05T08:12:41.747033+0000","last_active":"2024-02-05T08:13:14.951314+0000","last_peered":"2024-02-05T08:13:14.951314+0000","last_clean":"2024-02-05T08:13:14.951314+0000","last_became_active":"2024-02-05T08:12:41.730485+0000","last_became_peered":"2024-02-05T08:12:41.730485+0000","last_unstale":"2024-02-05T08:13:14.951314+0000","last_undegraded":"2024-02-05T08:13:14.951314+0000","last_fullsized":"2024-02-05T08:13:14.951314+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:11:33.210478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:33.210478+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.951940+0000","last_change":"2024-02-05T08:12:29.675196+0000","last_active":"2024-02-05T08:13:14.951940+0000","last_peered":"2024-02-05T08:13:14.951940+0000","last_clean":"2024-02-05T08:13:14.951940+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:13:14.951940+0000","last_undegraded":"2024-02-05T08:13:14.951940+0000","last_fullsized":"2024-02-05T08:13:14.951940+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.904127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.904127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:34.241676+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":306,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.000564+0000","last_change":"2024-02-05T08:11:06.715512+0000","last_active":"2024-02-05T08:13:18.000564+0000","last_peered":"2024-02-05T08:13:18.000564+0000","last_clean":"2024-02-05T08:13:18.000564+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:13:18.000564+0000","last_undegraded":"2024-02-05T08:13:18.000564+0000","last_fullsized":"2024-02-05T08:13:18.000564+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:06.715401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:11:06.715401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":313,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.999295+0000","last_change":"2024-02-05T08:11:22.303481+0000","last_active":"2024-02-05T08:13:17.999295+0000","last_peered":"2024-02-05T08:13:17.999295+0000","last_clean":"2024-02-05T08:13:17.999295+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:13:17.999295+0000","last_undegraded":"2024-02-05T08:13:17.999295+0000","last_fullsized":"2024-02-05T08:13:17.999295+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:22.303363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:22.303363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":371,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953657+0000","last_change":"2024-02-05T08:12:41.771382+0000","last_active":"2024-02-05T08:13:14.953657+0000","last_peered":"2024-02-05T08:13:14.953657+0000","last_clean":"2024-02-05T08:13:14.953657+0000","last_became_active":"2024-02-05T08:12:41.727027+0000","last_became_peered":"2024-02-05T08:12:41.727027+0000","last_unstale":"2024-02-05T08:13:14.953657+0000","last_undegraded":"2024-02-05T08:13:14.953657+0000","last_fullsized":"2024-02-05T08:13:14.953657+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:11:33.391736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:33.391736+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":302,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953966+0000","last_change":"2024-02-05T08:12:41.713865+0000","last_active":"2024-02-05T08:13:14.953966+0000","last_peered":"2024-02-05T08:13:14.953966+0000","last_clean":"2024-02-05T08:13:14.953966+0000","last_became_active":"2024-02-05T08:12:41.713564+0000","last_became_peered":"2024-02-05T08:12:41.713564+0000","last_unstale":"2024-02-05T08:13:14.953966+0000","last_undegraded":"2024-02-05T08:13:14.953966+0000","last_fullsized":"2024-02-05T08:13:14.953966+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:34.947466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:34.947466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":317,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.951423+0000","last_change":"2024-02-05T08:12:41.735590+0000","last_active":"2024-02-05T08:13:14.951423+0000","last_peered":"2024-02-05T08:13:14.951423+0000","last_clean":"2024-02-05T08:13:14.951423+0000","last_became_active":"2024-02-05T08:12:41.735240+0000","last_became_peered":"2024-02-05T08:12:41.735240+0000","last_unstale":"2024-02-05T08:13:14.951423+0000","last_undegraded":"2024-02-05T08:13:14.951423+0000","last_fullsized":"2024-02-05T08:13:14.951423+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:31.934762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:12:31.934762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:51.198468+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":311,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.950977+0000","last_change":"2024-02-05T08:12:16.855738+0000","last_active":"2024-02-05T08:13:14.950977+0000","last_peered":"2024-02-05T08:13:14.950977+0000","last_clean":"2024-02-05T08:13:14.950977+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:13:14.950977+0000","last_undegraded":"2024-02-05T08:13:14.950977+0000","last_fullsized":"2024-02-05T08:13:14.950977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:16.855647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:16.855647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:42.807659+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"90'17","reported_seq":124,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.000357+0000","last_change":"2024-02-05T08:12:41.756597+0000","last_active":"2024-02-05T08:13:18.000357+0000","last_peered":"2024-02-05T08:13:18.000357+0000","last_clean":"2024-02-05T08:13:18.000357+0000","last_became_active":"2024-02-05T08:12:41.716794+0000","last_became_peered":"2024-02-05T08:12:41.716794+0000","last_unstale":"2024-02-05T08:13:18.000357+0000","last_undegraded":"2024-02-05T08:13:18.000357+0000","last_fullsized":"2024-02-05T08:13:18.000357+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:10:30.977334+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:10:30.977334+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":318,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.118197+0000","last_change":"2024-02-05T08:12:29.117226+0000","last_active":"2024-02-05T08:13:15.118197+0000","last_peered":"2024-02-05T08:13:15.118197+0000","last_clean":"2024-02-05T08:13:15.118197+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:13:15.118197+0000","last_undegraded":"2024-02-05T08:13:15.118197+0000","last_fullsized":"2024-02-05T08:13:15.118197+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:12:29.117142+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:12:29.117142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:36.497586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":57,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.955491+0000","last_change":"2024-02-05T08:12:29.693990+0000","last_active":"2024-02-05T08:13:14.955491+0000","last_peered":"2024-02-05T08:13:14.955491+0000","last_clean":"2024-02-05T08:13:14.955491+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:13:14.955491+0000","last_undegraded":"2024-02-05T08:13:14.955491+0000","last_fullsized":"2024-02-05T08:13:14.955491+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:11:21.208206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:21.208206+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":325,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.001967+0000","last_change":"2024-02-05T08:12:41.749863+0000","last_active":"2024-02-05T08:13:18.001967+0000","last_peered":"2024-02-05T08:13:18.001967+0000","last_clean":"2024-02-05T08:13:18.001967+0000","last_became_active":"2024-02-05T08:12:41.719010+0000","last_became_peered":"2024-02-05T08:12:41.719010+0000","last_unstale":"2024-02-05T08:13:18.001967+0000","last_undegraded":"2024-02-05T08:13:18.001967+0000","last_fullsized":"2024-02-05T08:13:18.001967+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:10:32.403342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:10:32.403342+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":379,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.002530+0000","last_change":"2024-02-05T08:12:41.775561+0000","last_active":"2024-02-05T08:13:18.002530+0000","last_peered":"2024-02-05T08:13:18.002530+0000","last_clean":"2024-02-05T08:13:18.002530+0000","last_became_active":"2024-02-05T08:12:41.757718+0000","last_became_peered":"2024-02-05T08:12:41.757718+0000","last_unstale":"2024-02-05T08:13:18.002530+0000","last_undegraded":"2024-02-05T08:13:18.002530+0000","last_fullsized":"2024-02-05T08:13:18.002530+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:12:18.538594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:18.538594+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:23.144318+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":310,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.000689+0000","last_change":"2024-02-05T08:11:33.834287+0000","last_active":"2024-02-05T08:13:18.000689+0000","last_peered":"2024-02-05T08:13:18.000689+0000","last_clean":"2024-02-05T08:13:18.000689+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:13:18.000689+0000","last_undegraded":"2024-02-05T08:13:18.000689+0000","last_fullsized":"2024-02-05T08:13:18.000689+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:33.834034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:33.834034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":306,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.002766+0000","last_change":"2024-02-05T08:11:25.316698+0000","last_active":"2024-02-05T08:13:18.002766+0000","last_peered":"2024-02-05T08:13:18.002766+0000","last_clean":"2024-02-05T08:13:18.002766+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:13:18.002766+0000","last_undegraded":"2024-02-05T08:13:18.002766+0000","last_fullsized":"2024-02-05T08:13:18.002766+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:25.316598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:25.316598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":34,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.001051+0000","last_change":"2024-02-05T08:12:41.727708+0000","last_active":"2024-02-05T08:13:18.001051+0000","last_peered":"2024-02-05T08:13:18.001051+0000","last_clean":"2024-02-05T08:13:18.001051+0000","last_became_active":"2024-02-05T08:12:41.727406+0000","last_became_peered":"2024-02-05T08:12:41.727406+0000","last_unstale":"2024-02-05T08:13:18.001051+0000","last_undegraded":"2024-02-05T08:13:18.001051+0000","last_fullsized":"2024-02-05T08:13:18.001051+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:15.831837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.831837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":55,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953968+0000","last_change":"2024-02-05T08:12:29.712505+0000","last_active":"2024-02-05T08:13:14.953968+0000","last_peered":"2024-02-05T08:13:14.953968+0000","last_clean":"2024-02-05T08:13:14.953968+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:13:14.953968+0000","last_undegraded":"2024-02-05T08:13:14.953968+0000","last_fullsized":"2024-02-05T08:13:14.953968+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:11:15.397335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.397335+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":300,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.116910+0000","last_change":"2024-02-05T08:11:01.465783+0000","last_active":"2024-02-05T08:13:15.116910+0000","last_peered":"2024-02-05T08:13:15.116910+0000","last_clean":"2024-02-05T08:13:15.116910+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:13:15.116910+0000","last_undegraded":"2024-02-05T08:13:15.116910+0000","last_fullsized":"2024-02-05T08:13:15.116910+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:01.465657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:01.465657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":43,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953775+0000","last_change":"2024-02-05T08:12:29.678505+0000","last_active":"2024-02-05T08:13:14.953775+0000","last_peered":"2024-02-05T08:13:14.953775+0000","last_clean":"2024-02-05T08:13:14.953775+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:13:14.953775+0000","last_undegraded":"2024-02-05T08:13:14.953775+0000","last_fullsized":"2024-02-05T08:13:14.953775+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:24.812126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:12:24.812126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:51.087313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":319,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.001526+0000","last_change":"2024-02-05T08:12:41.737429+0000","last_active":"2024-02-05T08:13:18.001526+0000","last_peered":"2024-02-05T08:13:18.001526+0000","last_clean":"2024-02-05T08:13:18.001526+0000","last_became_active":"2024-02-05T08:12:41.736991+0000","last_became_peered":"2024-02-05T08:12:41.736991+0000","last_unstale":"2024-02-05T08:13:18.001526+0000","last_undegraded":"2024-02-05T08:13:18.001526+0000","last_fullsized":"2024-02-05T08:13:18.001526+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:20.294901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:20.294901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":65,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.001596+0000","last_change":"2024-02-05T08:12:41.743649+0000","last_active":"2024-02-05T08:13:18.001596+0000","last_peered":"2024-02-05T08:13:18.001596+0000","last_clean":"2024-02-05T08:13:18.001596+0000","last_became_active":"2024-02-05T08:12:41.717459+0000","last_became_peered":"2024-02-05T08:12:41.717459+0000","last_unstale":"2024-02-05T08:13:18.001596+0000","last_undegraded":"2024-02-05T08:13:18.001596+0000","last_fullsized":"2024-02-05T08:13:18.001596+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-02-05T08:11:32.785856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:32.785856+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":304,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.998980+0000","last_change":"2024-02-05T08:12:41.738206+0000","last_active":"2024-02-05T08:13:17.998980+0000","last_peered":"2024-02-05T08:13:17.998980+0000","last_clean":"2024-02-05T08:13:17.998980+0000","last_became_active":"2024-02-05T08:12:41.737803+0000","last_became_peered":"2024-02-05T08:12:41.737803+0000","last_unstale":"2024-02-05T08:13:17.998980+0000","last_undegraded":"2024-02-05T08:13:17.998980+0000","last_fullsized":"2024-02-05T08:13:17.998980+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:33.610823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:33.610823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:39.697997+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":324,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117213+0000","last_change":"2024-02-05T08:12:29.682425+0000","last_active":"2024-02-05T08:13:15.117213+0000","last_peered":"2024-02-05T08:13:15.117213+0000","last_clean":"2024-02-05T08:13:15.117213+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:13:15.117213+0000","last_undegraded":"2024-02-05T08:13:15.117213+0000","last_fullsized":"2024-02-05T08:13:15.117213+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:16.440585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:11:16.440585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":327,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.963114+0000","last_change":"2024-02-05T08:13:10.189057+0000","last_active":"2024-02-05T08:13:14.963114+0000","last_peered":"2024-02-05T08:13:14.963114+0000","last_clean":"2024-02-05T08:13:14.963114+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:13:14.963114+0000","last_undegraded":"2024-02-05T08:13:14.963114+0000","last_fullsized":"2024-02-05T08:13:14.963114+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:10.188977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:13:10.188977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:14:11.641053+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"141'16","reported_seq":256,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117350+0000","last_change":"2024-02-05T08:12:41.735407+0000","last_active":"2024-02-05T08:13:15.117350+0000","last_peered":"2024-02-05T08:13:15.117350+0000","last_clean":"2024-02-05T08:13:15.117350+0000","last_became_active":"2024-02-05T08:12:41.718651+0000","last_became_peered":"2024-02-05T08:12:41.718651+0000","last_unstale":"2024-02-05T08:13:15.117350+0000","last_undegraded":"2024-02-05T08:13:15.117350+0000","last_fullsized":"2024-02-05T08:13:15.117350+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'16","last_scrub_stamp":"2024-02-05T08:12:17.493187+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:12:17.493187+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:20.359082+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":301,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.999774+0000","last_change":"2024-02-05T08:11:07.248032+0000","last_active":"2024-02-05T08:13:17.999774+0000","last_peered":"2024-02-05T08:13:17.999774+0000","last_clean":"2024-02-05T08:13:17.999774+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:13:17.999774+0000","last_undegraded":"2024-02-05T08:13:17.999774+0000","last_fullsized":"2024-02-05T08:13:17.999774+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:07.247928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.247928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":35,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.999116+0000","last_change":"2024-02-05T08:12:41.758242+0000","last_active":"2024-02-05T08:13:17.999116+0000","last_peered":"2024-02-05T08:13:17.999116+0000","last_clean":"2024-02-05T08:13:17.999116+0000","last_became_active":"2024-02-05T08:12:41.757901+0000","last_became_peered":"2024-02-05T08:12:41.757901+0000","last_unstale":"2024-02-05T08:13:17.999116+0000","last_undegraded":"2024-02-05T08:13:17.999116+0000","last_fullsized":"2024-02-05T08:13:17.999116+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:26.987763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:26.987763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":326,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.125306+0000","last_change":"2024-02-05T08:12:41.734449+0000","last_active":"2024-02-05T08:13:15.125306+0000","last_peered":"2024-02-05T08:13:15.125306+0000","last_clean":"2024-02-05T08:13:15.125306+0000","last_became_active":"2024-02-05T08:12:41.734072+0000","last_became_peered":"2024-02-05T08:12:41.734072+0000","last_unstale":"2024-02-05T08:13:15.125306+0000","last_undegraded":"2024-02-05T08:13:15.125306+0000","last_fullsized":"2024-02-05T08:13:15.125306+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:19.108762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:19.108762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:41.388316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":366,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.999980+0000","last_change":"2024-02-05T08:12:29.700891+0000","last_active":"2024-02-05T08:13:17.999980+0000","last_peered":"2024-02-05T08:13:17.999980+0000","last_clean":"2024-02-05T08:13:17.999980+0000","last_became_active":"2024-02-05T08:12:29.676152+0000","last_became_peered":"2024-02-05T08:12:29.676152+0000","last_unstale":"2024-02-05T08:13:17.999980+0000","last_undegraded":"2024-02-05T08:13:17.999980+0000","last_fullsized":"2024-02-05T08:13:17.999980+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:13.793895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:13.793895+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.011296+0000","last_change":"2024-02-05T08:12:41.742137+0000","last_active":"2024-02-05T08:13:18.011296+0000","last_peered":"2024-02-05T08:13:18.011296+0000","last_clean":"2024-02-05T08:13:18.011296+0000","last_became_active":"2024-02-05T08:12:41.737246+0000","last_became_peered":"2024-02-05T08:12:41.737246+0000","last_unstale":"2024-02-05T08:13:18.011296+0000","last_undegraded":"2024-02-05T08:13:18.011296+0000","last_fullsized":"2024-02-05T08:13:18.011296+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.507800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.507800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:28.052173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":320,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.999810+0000","last_change":"2024-02-05T08:12:41.717743+0000","last_active":"2024-02-05T08:13:17.999810+0000","last_peered":"2024-02-05T08:13:17.999810+0000","last_clean":"2024-02-05T08:13:17.999810+0000","last_became_active":"2024-02-05T08:12:41.717285+0000","last_became_peered":"2024-02-05T08:12:41.717285+0000","last_unstale":"2024-02-05T08:13:17.999810+0000","last_undegraded":"2024-02-05T08:13:17.999810+0000","last_fullsized":"2024-02-05T08:13:17.999810+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:12.952558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:12.952558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:29.694413+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":331,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.951168+0000","last_change":"2024-02-05T08:12:29.677640+0000","last_active":"2024-02-05T08:13:14.951168+0000","last_peered":"2024-02-05T08:13:14.951168+0000","last_clean":"2024-02-05T08:13:14.951168+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:13:14.951168+0000","last_undegraded":"2024-02-05T08:13:14.951168+0000","last_fullsized":"2024-02-05T08:13:14.951168+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:32.160333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.966886+0000","last_clean_scrub_stamp":"2024-02-05T08:11:32.160333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":370,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.951433+0000","last_change":"2024-02-05T08:12:29.939886+0000","last_active":"2024-02-05T08:13:14.951433+0000","last_peered":"2024-02-05T08:13:14.951433+0000","last_clean":"2024-02-05T08:13:14.951433+0000","last_became_active":"2024-02-05T08:12:29.675657+0000","last_became_peered":"2024-02-05T08:12:29.675657+0000","last_unstale":"2024-02-05T08:13:14.951433+0000","last_undegraded":"2024-02-05T08:13:14.951433+0000","last_fullsized":"2024-02-05T08:13:14.951433+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:12:29.939772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:29.939772+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:49.888863+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":307,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.001716+0000","last_change":"2024-02-05T08:12:41.737779+0000","last_active":"2024-02-05T08:13:18.001716+0000","last_peered":"2024-02-05T08:13:18.001716+0000","last_clean":"2024-02-05T08:13:18.001716+0000","last_became_active":"2024-02-05T08:12:41.736845+0000","last_became_peered":"2024-02-05T08:12:41.736845+0000","last_unstale":"2024-02-05T08:13:18.001716+0000","last_undegraded":"2024-02-05T08:13:18.001716+0000","last_fullsized":"2024-02-05T08:13:18.001716+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:31.474957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:31.474957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":423,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.963941+0000","last_change":"2024-02-05T08:11:32.363524+0000","last_active":"2024-02-05T08:13:14.963941+0000","last_peered":"2024-02-05T08:13:14.963941+0000","last_clean":"2024-02-05T08:13:14.963941+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:13:14.963941+0000","last_undegraded":"2024-02-05T08:13:14.963941+0000","last_fullsized":"2024-02-05T08:13:14.963941+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:11:32.363440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:32.363440+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117195+0000","last_change":"2024-02-05T08:11:11.422476+0000","last_active":"2024-02-05T08:13:15.117195+0000","last_peered":"2024-02-05T08:13:15.117195+0000","last_clean":"2024-02-05T08:13:15.117195+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:13:15.117195+0000","last_undegraded":"2024-02-05T08:13:15.117195+0000","last_fullsized":"2024-02-05T08:13:15.117195+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:11.422371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:11.422371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"129'15","reported_seq":203,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.999057+0000","last_change":"2024-02-05T08:12:41.781948+0000","last_active":"2024-02-05T08:13:17.999057+0000","last_peered":"2024-02-05T08:13:17.999057+0000","last_clean":"2024-02-05T08:13:17.999057+0000","last_became_active":"2024-02-05T08:12:41.726861+0000","last_became_peered":"2024-02-05T08:12:41.726861+0000","last_unstale":"2024-02-05T08:13:17.999057+0000","last_undegraded":"2024-02-05T08:13:17.999057+0000","last_fullsized":"2024-02-05T08:13:17.999057+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'15","last_scrub_stamp":"2024-02-05T08:11:30.787149+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:11:30.787149+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":302,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.951844+0000","last_change":"2024-02-05T08:12:15.881540+0000","last_active":"2024-02-05T08:13:14.951844+0000","last_peered":"2024-02-05T08:13:14.951844+0000","last_clean":"2024-02-05T08:13:14.951844+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:13:14.951844+0000","last_undegraded":"2024-02-05T08:13:14.951844+0000","last_fullsized":"2024-02-05T08:13:14.951844+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:15.881453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:15.881453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:30.969474+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"194'9","reported_seq":77,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.118174+0000","last_change":"2024-02-05T08:13:09.130164+0000","last_active":"2024-02-05T08:13:15.118174+0000","last_peered":"2024-02-05T08:13:15.118174+0000","last_clean":"2024-02-05T08:13:15.118174+0000","last_became_active":"2024-02-05T08:12:41.728687+0000","last_became_peered":"2024-02-05T08:12:41.728687+0000","last_unstale":"2024-02-05T08:13:15.118174+0000","last_undegraded":"2024-02-05T08:13:15.118174+0000","last_fullsized":"2024-02-05T08:13:15.118174+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'9","last_scrub_stamp":"2024-02-05T08:13:09.130041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:13:09.130041+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:14:23.795101+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":319,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.999430+0000","last_change":"2024-02-05T08:12:41.723729+0000","last_active":"2024-02-05T08:13:17.999430+0000","last_peered":"2024-02-05T08:13:17.999430+0000","last_clean":"2024-02-05T08:13:17.999430+0000","last_became_active":"2024-02-05T08:12:41.723383+0000","last_became_peered":"2024-02-05T08:12:41.723383+0000","last_unstale":"2024-02-05T08:13:17.999430+0000","last_undegraded":"2024-02-05T08:13:17.999430+0000","last_fullsized":"2024-02-05T08:13:17.999430+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:11:24.316208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":347,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.964105+0000","last_change":"2024-02-05T08:12:41.736804+0000","last_active":"2024-02-05T08:13:14.964105+0000","last_peered":"2024-02-05T08:13:14.964105+0000","last_clean":"2024-02-05T08:13:14.964105+0000","last_became_active":"2024-02-05T08:12:41.723526+0000","last_became_peered":"2024-02-05T08:12:41.723526+0000","last_unstale":"2024-02-05T08:13:14.964105+0000","last_undegraded":"2024-02-05T08:13:14.964105+0000","last_fullsized":"2024-02-05T08:13:14.964105+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:19.287569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:19.287569+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":316,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953948+0000","last_change":"2024-02-05T08:12:18.337618+0000","last_active":"2024-02-05T08:13:14.953948+0000","last_peered":"2024-02-05T08:13:14.953948+0000","last_clean":"2024-02-05T08:13:14.953948+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:13:14.953948+0000","last_undegraded":"2024-02-05T08:13:14.953948+0000","last_fullsized":"2024-02-05T08:13:14.953948+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:18.337499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:18.337499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:36.647778+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":35,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.002502+0000","last_change":"2024-02-05T08:12:41.722172+0000","last_active":"2024-02-05T08:13:18.002502+0000","last_peered":"2024-02-05T08:13:18.002502+0000","last_clean":"2024-02-05T08:13:18.002502+0000","last_became_active":"2024-02-05T08:12:41.721852+0000","last_became_peered":"2024-02-05T08:12:41.721852+0000","last_unstale":"2024-02-05T08:13:18.002502+0000","last_undegraded":"2024-02-05T08:13:18.002502+0000","last_fullsized":"2024-02-05T08:13:18.002502+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:15.994621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:10:15.994621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":332,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.116938+0000","last_change":"2024-02-05T08:13:12.164416+0000","last_active":"2024-02-05T08:13:15.116938+0000","last_peered":"2024-02-05T08:13:15.116938+0000","last_clean":"2024-02-05T08:13:15.116938+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:13:15.116938+0000","last_undegraded":"2024-02-05T08:13:15.116938+0000","last_fullsized":"2024-02-05T08:13:15.116938+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:12.164344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:13:12.164344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:14:17.454850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":341,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.002622+0000","last_change":"2024-02-05T08:12:41.770999+0000","last_active":"2024-02-05T08:13:18.002622+0000","last_peered":"2024-02-05T08:13:18.002622+0000","last_clean":"2024-02-05T08:13:18.002622+0000","last_became_active":"2024-02-05T08:12:41.736414+0000","last_became_peered":"2024-02-05T08:12:41.736414+0000","last_unstale":"2024-02-05T08:13:18.002622+0000","last_undegraded":"2024-02-05T08:13:18.002622+0000","last_fullsized":"2024-02-05T08:13:18.002622+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:11:23.324165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:23.324165+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.125178+0000","last_change":"2024-02-05T08:12:29.677397+0000","last_active":"2024-02-05T08:13:15.125178+0000","last_peered":"2024-02-05T08:13:15.125178+0000","last_clean":"2024-02-05T08:13:15.125178+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:13:15.125178+0000","last_undegraded":"2024-02-05T08:13:15.125178+0000","last_fullsized":"2024-02-05T08:13:15.125178+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:18.424341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:18.424341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":317,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.963194+0000","last_change":"2024-02-05T08:12:41.721999+0000","last_active":"2024-02-05T08:13:14.963194+0000","last_peered":"2024-02-05T08:13:14.963194+0000","last_clean":"2024-02-05T08:13:14.963194+0000","last_became_active":"2024-02-05T08:12:41.721486+0000","last_became_peered":"2024-02-05T08:12:41.721486+0000","last_unstale":"2024-02-05T08:13:14.963194+0000","last_undegraded":"2024-02-05T08:13:14.963194+0000","last_fullsized":"2024-02-05T08:13:14.963194+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.242229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.242229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:32.796493+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":43,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.952703+0000","last_change":"2024-02-05T08:12:29.684768+0000","last_active":"2024-02-05T08:13:14.952703+0000","last_peered":"2024-02-05T08:13:14.952703+0000","last_clean":"2024-02-05T08:13:14.952703+0000","last_became_active":"2024-02-05T08:12:29.684444+0000","last_became_peered":"2024-02-05T08:12:29.684444+0000","last_unstale":"2024-02-05T08:13:14.952703+0000","last_undegraded":"2024-02-05T08:13:14.952703+0000","last_fullsized":"2024-02-05T08:13:14.952703+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:01.330449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:01.330449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"116'9","reported_seq":173,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117286+0000","last_change":"2024-02-05T08:13:10.202789+0000","last_active":"2024-02-05T08:13:15.117286+0000","last_peered":"2024-02-05T08:13:15.117286+0000","last_clean":"2024-02-05T08:13:15.117286+0000","last_became_active":"2024-02-05T08:12:41.732369+0000","last_became_peered":"2024-02-05T08:12:41.732369+0000","last_unstale":"2024-02-05T08:13:15.117286+0000","last_undegraded":"2024-02-05T08:13:15.117286+0000","last_fullsized":"2024-02-05T08:13:15.117286+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-02-05T08:13:10.202636+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:13:10.202636+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:14:15.783473+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"204'9083","reported_seq":9260,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.566353+0000","last_change":"2024-02-05T08:12:23.713036+0000","last_active":"2024-02-05T08:13:17.566353+0000","last_peered":"2024-02-05T08:13:17.566353+0000","last_clean":"2024-02-05T08:13:17.566353+0000","last_became_active":"2024-02-05T08:10:54.986264+0000","last_became_peered":"2024-02-05T08:10:54.986264+0000","last_unstale":"2024-02-05T08:13:17.566353+0000","last_undegraded":"2024-02-05T08:13:17.566353+0000","last_fullsized":"2024-02-05T08:13:17.566353+0000","mapping_epoch":157,"log_start":"175'4148","ondisk_log_start":"175'4148","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'6431","last_scrub_stamp":"2024-02-05T08:12:23.712829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:12:23.712829+0000","objects_scrubbed":6374,"log_size":4935,"log_dups_size":0,"ondisk_log_size":4935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:27.342344+0000","scrub_duration":1445,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351076352,"num_objects":5357,"num_object_clones":0,"num_object_copies":16071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9083,"num_write_kb":462080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":319,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.962629+0000","last_change":"2024-02-05T08:12:29.680463+0000","last_active":"2024-02-05T08:13:14.962629+0000","last_peered":"2024-02-05T08:13:14.962629+0000","last_clean":"2024-02-05T08:13:14.962629+0000","last_became_active":"2024-02-05T08:12:29.679849+0000","last_became_peered":"2024-02-05T08:12:29.679849+0000","last_unstale":"2024-02-05T08:13:14.962629+0000","last_undegraded":"2024-02-05T08:13:14.962629+0000","last_fullsized":"2024-02-05T08:13:14.962629+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:13.452127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:13.452127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":332,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117542+0000","last_change":"2024-02-05T08:12:22.085674+0000","last_active":"2024-02-05T08:13:15.117542+0000","last_peered":"2024-02-05T08:13:15.117542+0000","last_clean":"2024-02-05T08:13:15.117542+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:13:15.117542+0000","last_undegraded":"2024-02-05T08:13:15.117542+0000","last_fullsized":"2024-02-05T08:13:15.117542+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:12:22.085596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:22.085596+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:23.913734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":384,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.961439+0000","last_change":"2024-02-05T08:12:29.719785+0000","last_active":"2024-02-05T08:13:14.961439+0000","last_peered":"2024-02-05T08:13:14.961439+0000","last_clean":"2024-02-05T08:13:14.961439+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:13:14.961439+0000","last_undegraded":"2024-02-05T08:13:14.961439+0000","last_fullsized":"2024-02-05T08:13:14.961439+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:12:26.319958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.319958+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:48.768788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"190'10","reported_seq":444,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117134+0000","last_change":"2024-02-05T08:12:29.714750+0000","last_active":"2024-02-05T08:13:15.117134+0000","last_peered":"2024-02-05T08:13:15.117134+0000","last_clean":"2024-02-05T08:13:15.117134+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:13:15.117134+0000","last_undegraded":"2024-02-05T08:13:15.117134+0000","last_fullsized":"2024-02-05T08:13:15.117134+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:11:14.422445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:14.422445+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"204'9211","reported_seq":22490,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.510426+0000","last_change":"2024-02-05T08:13:15.194164+0000","last_active":"2024-02-05T08:13:17.510426+0000","last_peered":"2024-02-05T08:13:17.510426+0000","last_clean":"2024-02-05T08:13:17.510426+0000","last_became_active":"2024-02-05T08:13:15.126729+0000","last_became_peered":"2024-02-05T08:13:15.126729+0000","last_unstale":"2024-02-05T08:13:17.510426+0000","last_undegraded":"2024-02-05T08:13:17.510426+0000","last_fullsized":"2024-02-05T08:13:17.510426+0000","mapping_epoch":203,"log_start":"176'4320","ondisk_log_start":"176'4320","created":157,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4891,"log_dups_size":0,"ondisk_log_size":4891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352911360,"num_objects":5385,"num_object_clones":0,"num_object_copies":16155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9211,"num_write_kb":467072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6578,"num_bytes_recovered":431030272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":32,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117277+0000","last_change":"2024-02-05T08:12:41.723494+0000","last_active":"2024-02-05T08:13:15.117277+0000","last_peered":"2024-02-05T08:13:15.117277+0000","last_clean":"2024-02-05T08:13:15.117277+0000","last_became_active":"2024-02-05T08:12:41.722853+0000","last_became_peered":"2024-02-05T08:12:41.722853+0000","last_unstale":"2024-02-05T08:13:15.117277+0000","last_undegraded":"2024-02-05T08:13:15.117277+0000","last_fullsized":"2024-02-05T08:13:15.117277+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:21.562331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:21.562331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:39.534804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117610+0000","last_change":"2024-02-05T08:11:31.474056+0000","last_active":"2024-02-05T08:13:15.117610+0000","last_peered":"2024-02-05T08:13:15.117610+0000","last_clean":"2024-02-05T08:13:15.117610+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:13:15.117610+0000","last_undegraded":"2024-02-05T08:13:15.117610+0000","last_fullsized":"2024-02-05T08:13:15.117610+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:31.473976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:11:31.473976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":348,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.950964+0000","last_change":"2024-02-05T08:12:41.750244+0000","last_active":"2024-02-05T08:13:14.950964+0000","last_peered":"2024-02-05T08:13:14.950964+0000","last_clean":"2024-02-05T08:13:14.950964+0000","last_became_active":"2024-02-05T08:12:41.728925+0000","last_became_peered":"2024-02-05T08:12:41.728925+0000","last_unstale":"2024-02-05T08:13:14.950964+0000","last_undegraded":"2024-02-05T08:13:14.950964+0000","last_fullsized":"2024-02-05T08:13:14.950964+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:12:19.888170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:19.888170+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:27.404627+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.952917+0000","last_change":"2024-02-05T08:12:29.353477+0000","last_active":"2024-02-05T08:13:14.952917+0000","last_peered":"2024-02-05T08:13:14.952917+0000","last_clean":"2024-02-05T08:13:14.952917+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:13:14.952917+0000","last_undegraded":"2024-02-05T08:13:14.952917+0000","last_fullsized":"2024-02-05T08:13:14.952917+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:29.353397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.368730+0000","last_clean_scrub_stamp":"2024-02-05T08:12:29.353397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:42.610728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"205'9249","reported_seq":21528,"reported_epoch":205,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:13:18.400984+0000","last_change":"2024-02-05T08:12:41.770370+0000","last_active":"2024-02-05T08:13:18.400984+0000","last_peered":"2024-02-05T08:13:18.400984+0000","last_clean":"2024-02-05T08:12:28.993658+0000","last_became_active":"2024-02-05T08:12:30.762738+0000","last_became_peered":"2024-02-05T08:12:30.762738+0000","last_unstale":"2024-02-05T08:13:18.400984+0000","last_undegraded":"2024-02-05T08:13:18.400984+0000","last_fullsized":"2024-02-05T08:13:18.400984+0000","mapping_epoch":188,"log_start":"175'4153","ondisk_log_start":"175'4153","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348323840,"num_objects":5315,"num_object_clones":0,"num_object_copies":15945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":284,"num_objects_unfound":0,"num_objects_dirty":5315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9249,"num_write_kb":466048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6082,"num_bytes_recovered":398589952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,7,2],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":5315}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":314,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.116827+0000","last_change":"2024-02-05T08:12:41.725243+0000","last_active":"2024-02-05T08:13:15.116827+0000","last_peered":"2024-02-05T08:13:15.116827+0000","last_clean":"2024-02-05T08:13:15.116827+0000","last_became_active":"2024-02-05T08:12:41.724830+0000","last_became_peered":"2024-02-05T08:12:41.724830+0000","last_unstale":"2024-02-05T08:13:15.116827+0000","last_undegraded":"2024-02-05T08:13:15.116827+0000","last_fullsized":"2024-02-05T08:13:15.116827+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:21.070242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:21.070242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:42.972126+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":312,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953767+0000","last_change":"2024-02-05T08:11:22.231278+0000","last_active":"2024-02-05T08:13:14.953767+0000","last_peered":"2024-02-05T08:13:14.953767+0000","last_clean":"2024-02-05T08:13:14.953767+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:13:14.953767+0000","last_undegraded":"2024-02-05T08:13:14.953767+0000","last_fullsized":"2024-02-05T08:13:14.953767+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:11:22.231148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":332,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.125464+0000","last_change":"2024-02-05T08:12:29.679032+0000","last_active":"2024-02-05T08:13:15.125464+0000","last_peered":"2024-02-05T08:13:15.125464+0000","last_clean":"2024-02-05T08:13:15.125464+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:13:15.125464+0000","last_undegraded":"2024-02-05T08:13:15.125464+0000","last_fullsized":"2024-02-05T08:13:15.125464+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:27.540472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:27.540472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":63,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953316+0000","last_change":"2024-02-05T08:12:29.716681+0000","last_active":"2024-02-05T08:13:14.953316+0000","last_peered":"2024-02-05T08:13:14.953316+0000","last_clean":"2024-02-05T08:13:14.953316+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:13:14.953316+0000","last_undegraded":"2024-02-05T08:13:14.953316+0000","last_fullsized":"2024-02-05T08:13:14.953316+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:11:19.277464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:19.277464+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"204'8996","reported_seq":21640,"reported_epoch":204,"state":"peering","last_fresh":"2024-02-05T08:13:17.998203+0000","last_change":"2024-02-05T08:13:18.004985+0000","last_active":"2024-02-05T08:13:18.004985+0000","last_peered":"2024-02-05T08:13:17.998203+0000","last_clean":"2024-02-05T08:13:17.998203+0000","last_became_active":"2024-02-05T08:12:42.725152+0000","last_became_peered":"2024-02-05T08:12:42.725152+0000","last_unstale":"2024-02-05T08:13:17.998203+0000","last_undegraded":"2024-02-05T08:13:17.998203+0000","last_fullsized":"2024-02-05T08:13:17.998203+0000","mapping_epoch":205,"log_start":"178'4822","ondisk_log_start":"178'4822","created":157,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348782592,"num_objects":5322,"num_object_clones":0,"num_object_copies":15966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8996,"num_write_kb":458176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6262,"num_bytes_recovered":410386432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,6],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":301,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.011492+0000","last_change":"2024-02-05T08:11:08.235398+0000","last_active":"2024-02-05T08:13:18.011492+0000","last_peered":"2024-02-05T08:13:18.011492+0000","last_clean":"2024-02-05T08:13:18.011492+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:13:18.011492+0000","last_undegraded":"2024-02-05T08:13:18.011492+0000","last_fullsized":"2024-02-05T08:13:18.011492+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:08.235281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:08.235281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":318,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953258+0000","last_change":"2024-02-05T08:12:41.744632+0000","last_active":"2024-02-05T08:13:14.953258+0000","last_peered":"2024-02-05T08:13:14.953258+0000","last_clean":"2024-02-05T08:13:14.953258+0000","last_became_active":"2024-02-05T08:12:41.743888+0000","last_became_peered":"2024-02-05T08:12:41.743888+0000","last_unstale":"2024-02-05T08:13:14.953258+0000","last_undegraded":"2024-02-05T08:13:14.953258+0000","last_fullsized":"2024-02-05T08:13:14.953258+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:15.272287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.272287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":337,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.963335+0000","last_change":"2024-02-05T08:12:41.719156+0000","last_active":"2024-02-05T08:13:14.963335+0000","last_peered":"2024-02-05T08:13:14.963335+0000","last_clean":"2024-02-05T08:13:14.963335+0000","last_became_active":"2024-02-05T08:12:41.718843+0000","last_became_peered":"2024-02-05T08:12:41.718843+0000","last_unstale":"2024-02-05T08:13:14.963335+0000","last_undegraded":"2024-02-05T08:13:14.963335+0000","last_fullsized":"2024-02-05T08:13:14.963335+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:18.291829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:18.291829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:38.961092+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":333,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.999543+0000","last_change":"2024-02-05T08:12:22.599613+0000","last_active":"2024-02-05T08:13:17.999543+0000","last_peered":"2024-02-05T08:13:17.999543+0000","last_clean":"2024-02-05T08:13:17.999543+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:13:17.999543+0000","last_undegraded":"2024-02-05T08:13:17.999543+0000","last_fullsized":"2024-02-05T08:13:17.999543+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:12:22.599493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:22.599493+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:42.403843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"204'9150","reported_seq":10724,"reported_epoch":204,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:13:17.632200+0000","last_change":"2024-02-05T08:13:12.869556+0000","last_active":"2024-02-05T08:13:17.632200+0000","last_peered":"2024-02-05T08:13:17.632200+0000","last_clean":"2024-02-05T08:12:40.696594+0000","last_became_active":"2024-02-05T08:12:42.836170+0000","last_became_peered":"2024-02-05T08:12:42.836170+0000","last_unstale":"2024-02-05T08:13:17.632200+0000","last_undegraded":"2024-02-05T08:13:17.632200+0000","last_fullsized":"2024-02-05T08:13:17.632200+0000","mapping_epoch":194,"log_start":"176'4293","ondisk_log_start":"176'4293","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":2849,"log_size":4857,"log_dups_size":0,"ondisk_log_size":4857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352321536,"num_objects":5376,"num_object_clones":0,"num_object_copies":16128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4685,"num_objects_unfound":0,"num_objects_dirty":5376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9150,"num_write_kb":464832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":714,"num_bytes_recovered":46792704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,3],"avail_no_missing":["5","3","7"],"object_location_counts":[{"shards":"3,5,7","objects":5376}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":32,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.951720+0000","last_change":"2024-02-05T08:12:41.727102+0000","last_active":"2024-02-05T08:13:14.951720+0000","last_peered":"2024-02-05T08:13:14.951720+0000","last_clean":"2024-02-05T08:13:14.951720+0000","last_became_active":"2024-02-05T08:12:41.726776+0000","last_became_peered":"2024-02-05T08:12:41.726776+0000","last_unstale":"2024-02-05T08:13:14.951720+0000","last_undegraded":"2024-02-05T08:13:14.951720+0000","last_fullsized":"2024-02-05T08:13:14.951720+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:32.545208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:32.545208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:54.224769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":352,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.962410+0000","last_change":"2024-02-05T08:12:29.715120+0000","last_active":"2024-02-05T08:13:14.962410+0000","last_peered":"2024-02-05T08:13:14.962410+0000","last_clean":"2024-02-05T08:13:14.962410+0000","last_became_active":"2024-02-05T08:12:29.679395+0000","last_became_peered":"2024-02-05T08:12:29.679395+0000","last_unstale":"2024-02-05T08:13:14.962410+0000","last_undegraded":"2024-02-05T08:13:14.962410+0000","last_fullsized":"2024-02-05T08:13:14.962410+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:12:13.266830+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:12:13.266830+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:16.282151+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":324,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.951197+0000","last_change":"2024-02-05T08:12:29.671185+0000","last_active":"2024-02-05T08:13:14.951197+0000","last_peered":"2024-02-05T08:13:14.951197+0000","last_clean":"2024-02-05T08:13:14.951197+0000","last_became_active":"2024-02-05T08:12:29.670859+0000","last_became_peered":"2024-02-05T08:12:29.670859+0000","last_unstale":"2024-02-05T08:13:14.951197+0000","last_undegraded":"2024-02-05T08:13:14.951197+0000","last_fullsized":"2024-02-05T08:13:14.951197+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:07.502393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:11:07.502393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"194'39","reported_seq":509,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.998966+0000","last_change":"2024-02-05T08:12:41.775200+0000","last_active":"2024-02-05T08:13:17.998966+0000","last_peered":"2024-02-05T08:13:17.998966+0000","last_clean":"2024-02-05T08:13:17.998966+0000","last_became_active":"2024-02-05T08:12:41.719820+0000","last_became_peered":"2024-02-05T08:12:41.719820+0000","last_unstale":"2024-02-05T08:13:17.998966+0000","last_undegraded":"2024-02-05T08:13:17.998966+0000","last_fullsized":"2024-02-05T08:13:17.998966+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:11:34.819564+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:11:34.819564+0000","objects_scrubbed":10,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":306,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.001472+0000","last_change":"2024-02-05T08:12:28.727498+0000","last_active":"2024-02-05T08:13:18.001472+0000","last_peered":"2024-02-05T08:13:18.001472+0000","last_clean":"2024-02-05T08:13:18.001472+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:13:18.001472+0000","last_undegraded":"2024-02-05T08:13:18.001472+0000","last_fullsized":"2024-02-05T08:13:18.001472+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:28.721402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:28.721402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:51.573749+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"204'9073","reported_seq":10636,"reported_epoch":204,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:13:17.577794+0000","last_change":"2024-02-05T08:13:12.849898+0000","last_active":"2024-02-05T08:13:17.577794+0000","last_peered":"2024-02-05T08:13:17.577794+0000","last_clean":"2024-02-05T08:12:40.665520+0000","last_became_active":"2024-02-05T08:12:42.797892+0000","last_became_peered":"2024-02-05T08:12:42.797892+0000","last_unstale":"2024-02-05T08:13:17.577794+0000","last_undegraded":"2024-02-05T08:13:17.577794+0000","last_fullsized":"2024-02-05T08:13:17.577794+0000","mapping_epoch":194,"log_start":"178'4587","ondisk_log_start":"178'4587","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4486,"log_dups_size":0,"ondisk_log_size":4486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352911360,"num_objects":5385,"num_object_clones":0,"num_object_copies":16155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4677,"num_objects_unfound":0,"num_objects_dirty":5385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9073,"num_write_kb":462656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":722,"num_bytes_recovered":47185920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,3],"avail_no_missing":["4","1","3"],"object_location_counts":[{"shards":"1,3,4","objects":5385}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":327,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.125669+0000","last_change":"2024-02-05T08:12:41.733166+0000","last_active":"2024-02-05T08:13:15.125669+0000","last_peered":"2024-02-05T08:13:15.125669+0000","last_clean":"2024-02-05T08:13:15.125669+0000","last_became_active":"2024-02-05T08:12:41.732338+0000","last_became_peered":"2024-02-05T08:12:41.732338+0000","last_unstale":"2024-02-05T08:13:15.125669+0000","last_undegraded":"2024-02-05T08:13:15.125669+0000","last_fullsized":"2024-02-05T08:13:15.125669+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:27.114442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:27.114442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:53.398218+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":342,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.001563+0000","last_change":"2024-02-05T08:12:29.682711+0000","last_active":"2024-02-05T08:13:18.001563+0000","last_peered":"2024-02-05T08:13:18.001563+0000","last_clean":"2024-02-05T08:13:18.001563+0000","last_became_active":"2024-02-05T08:12:29.673554+0000","last_became_peered":"2024-02-05T08:12:29.673554+0000","last_unstale":"2024-02-05T08:13:18.001563+0000","last_undegraded":"2024-02-05T08:13:18.001563+0000","last_fullsized":"2024-02-05T08:13:18.001563+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:11:23.934912+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:11:23.934912+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":344,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.952961+0000","last_change":"2024-02-05T08:12:41.746703+0000","last_active":"2024-02-05T08:13:14.952961+0000","last_peered":"2024-02-05T08:13:14.952961+0000","last_clean":"2024-02-05T08:13:14.952961+0000","last_became_active":"2024-02-05T08:12:41.746289+0000","last_became_peered":"2024-02-05T08:12:41.746289+0000","last_unstale":"2024-02-05T08:13:14.952961+0000","last_undegraded":"2024-02-05T08:13:14.952961+0000","last_fullsized":"2024-02-05T08:13:14.952961+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.404533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.404533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:41.027208+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.954571+0000","last_change":"2024-02-05T08:11:25.289984+0000","last_active":"2024-02-05T08:13:14.954571+0000","last_peered":"2024-02-05T08:13:14.954571+0000","last_clean":"2024-02-05T08:13:14.954571+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:13:14.954571+0000","last_undegraded":"2024-02-05T08:13:14.954571+0000","last_fullsized":"2024-02-05T08:13:14.954571+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:11:25.289752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:25.289752+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":320,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.954118+0000","last_change":"2024-02-05T08:12:25.317266+0000","last_active":"2024-02-05T08:13:14.954118+0000","last_peered":"2024-02-05T08:13:14.954118+0000","last_clean":"2024-02-05T08:13:14.954118+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:13:14.954118+0000","last_undegraded":"2024-02-05T08:13:14.954118+0000","last_fullsized":"2024-02-05T08:13:14.954118+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:25.317043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:25.317043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:46.301280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"205'9056","reported_seq":10587,"reported_epoch":205,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:13:18.036914+0000","last_change":"2024-02-05T08:13:13.693062+0000","last_active":"2024-02-05T08:13:18.036914+0000","last_peered":"2024-02-05T08:13:18.036914+0000","last_clean":"2024-02-05T08:12:29.005556+0000","last_became_active":"2024-02-05T08:12:30.765777+0000","last_became_peered":"2024-02-05T08:12:30.765777+0000","last_unstale":"2024-02-05T08:13:18.036914+0000","last_undegraded":"2024-02-05T08:13:18.036914+0000","last_fullsized":"2024-02-05T08:13:18.036914+0000","mapping_epoch":188,"log_start":"168'2396","ondisk_log_start":"168'2396","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":6660,"log_dups_size":0,"ondisk_log_size":6660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353370112,"num_objects":5392,"num_object_clones":0,"num_object_copies":16176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4703,"num_objects_unfound":0,"num_objects_dirty":5392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9056,"num_write_kb":462336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":46333952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[3,5,0],"avail_no_missing":["3","0","5"],"object_location_counts":[{"shards":"0,3,5","objects":5392}],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":313,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.962898+0000","last_change":"2024-02-05T08:12:41.717519+0000","last_active":"2024-02-05T08:13:14.962898+0000","last_peered":"2024-02-05T08:13:14.962898+0000","last_clean":"2024-02-05T08:13:14.962898+0000","last_became_active":"2024-02-05T08:12:41.717170+0000","last_became_peered":"2024-02-05T08:12:41.717170+0000","last_unstale":"2024-02-05T08:13:14.962898+0000","last_undegraded":"2024-02-05T08:13:14.962898+0000","last_fullsized":"2024-02-05T08:13:14.962898+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:23.303613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:23.303613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:28.548919+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":601,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117081+0000","last_change":"2024-02-05T08:12:32.677767+0000","last_active":"2024-02-05T08:13:15.117081+0000","last_peered":"2024-02-05T08:13:15.117081+0000","last_clean":"2024-02-05T08:13:15.117081+0000","last_became_active":"2024-02-05T08:12:32.677426+0000","last_became_peered":"2024-02-05T08:12:32.677426+0000","last_unstale":"2024-02-05T08:13:15.117081+0000","last_undegraded":"2024-02-05T08:13:15.117081+0000","last_fullsized":"2024-02-05T08:13:15.117081+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:11:05.478284+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:11:05.478284+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":314,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.962401+0000","last_change":"2024-02-05T08:12:41.722635+0000","last_active":"2024-02-05T08:13:14.962401+0000","last_peered":"2024-02-05T08:13:14.962401+0000","last_clean":"2024-02-05T08:13:14.962401+0000","last_became_active":"2024-02-05T08:12:41.722331+0000","last_became_peered":"2024-02-05T08:12:41.722331+0000","last_unstale":"2024-02-05T08:13:14.962401+0000","last_undegraded":"2024-02-05T08:13:14.962401+0000","last_fullsized":"2024-02-05T08:13:14.962401+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:29.759802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:29.759802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"190'18","reported_seq":71,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953941+0000","last_change":"2024-02-05T08:12:29.711770+0000","last_active":"2024-02-05T08:13:14.953941+0000","last_peered":"2024-02-05T08:13:14.953941+0000","last_clean":"2024-02-05T08:13:14.953941+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:13:14.953941+0000","last_undegraded":"2024-02-05T08:13:14.953941+0000","last_fullsized":"2024-02-05T08:13:14.953941+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-02-05T08:12:26.852734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.852734+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:51.240118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":305,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.118219+0000","last_change":"2024-02-05T08:11:04.446955+0000","last_active":"2024-02-05T08:13:15.118219+0000","last_peered":"2024-02-05T08:13:15.118219+0000","last_clean":"2024-02-05T08:13:15.118219+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:13:15.118219+0000","last_undegraded":"2024-02-05T08:13:15.118219+0000","last_fullsized":"2024-02-05T08:13:15.118219+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:04.446828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:04.446828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"205'9275","reported_seq":9382,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.076779+0000","last_change":"2024-02-05T08:10:54.985737+0000","last_active":"2024-02-05T08:13:18.076779+0000","last_peered":"2024-02-05T08:13:18.076779+0000","last_clean":"2024-02-05T08:13:18.076779+0000","last_became_active":"2024-02-05T08:10:54.985443+0000","last_became_peered":"2024-02-05T08:10:54.985443+0000","last_unstale":"2024-02-05T08:13:18.076779+0000","last_undegraded":"2024-02-05T08:13:18.076779+0000","last_fullsized":"2024-02-05T08:13:18.076779+0000","mapping_epoch":157,"log_start":"178'5115","ondisk_log_start":"178'5115","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356712448,"num_objects":5443,"num_object_clones":0,"num_object_copies":16329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9275,"num_write_kb":470976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":317,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.998636+0000","last_change":"2024-02-05T08:12:41.716722+0000","last_active":"2024-02-05T08:13:17.998636+0000","last_peered":"2024-02-05T08:13:17.998636+0000","last_clean":"2024-02-05T08:13:17.998636+0000","last_became_active":"2024-02-05T08:12:41.716394+0000","last_became_peered":"2024-02-05T08:12:41.716394+0000","last_unstale":"2024-02-05T08:13:17.998636+0000","last_undegraded":"2024-02-05T08:13:17.998636+0000","last_fullsized":"2024-02-05T08:13:17.998636+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:26.178219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.178219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:28.307031+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":320,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.011203+0000","last_change":"2024-02-05T08:12:41.742379+0000","last_active":"2024-02-05T08:13:18.011203+0000","last_peered":"2024-02-05T08:13:18.011203+0000","last_clean":"2024-02-05T08:13:18.011203+0000","last_became_active":"2024-02-05T08:12:41.741846+0000","last_became_peered":"2024-02-05T08:12:41.741846+0000","last_unstale":"2024-02-05T08:13:18.011203+0000","last_undegraded":"2024-02-05T08:13:18.011203+0000","last_fullsized":"2024-02-05T08:13:18.011203+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:34.336040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:34.336040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":843,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:16.412249+0000","last_change":"2024-02-05T08:11:20.552800+0000","last_active":"2024-02-05T08:13:16.412249+0000","last_peered":"2024-02-05T08:13:16.412249+0000","last_clean":"2024-02-05T08:13:16.412249+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:13:16.412249+0000","last_undegraded":"2024-02-05T08:13:16.412249+0000","last_fullsized":"2024-02-05T08:13:16.412249+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:11:20.552682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:11:20.552682+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":510,"num_read_kb":505,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":327,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.002339+0000","last_change":"2024-02-05T08:12:16.517917+0000","last_active":"2024-02-05T08:13:18.002339+0000","last_peered":"2024-02-05T08:13:18.002339+0000","last_clean":"2024-02-05T08:13:18.002339+0000","last_became_active":"2024-02-05T08:04:43.549687+0000","last_became_peered":"2024-02-05T08:04:43.549687+0000","last_unstale":"2024-02-05T08:13:18.002339+0000","last_undegraded":"2024-02-05T08:13:18.002339+0000","last_fullsized":"2024-02-05T08:13:18.002339+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:12:16.517774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:16.517774+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:29.388717+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":311,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.963287+0000","last_change":"2024-02-05T08:12:24.353439+0000","last_active":"2024-02-05T08:13:14.963287+0000","last_peered":"2024-02-05T08:13:14.963287+0000","last_clean":"2024-02-05T08:13:14.963287+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:13:14.963287+0000","last_undegraded":"2024-02-05T08:13:14.963287+0000","last_fullsized":"2024-02-05T08:13:14.963287+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:24.353354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:24.353354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:41.429618+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"204'9314","reported_seq":18517,"reported_epoch":204,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:13:17.988040+0000","last_change":"2024-02-05T08:12:42.801000+0000","last_active":"2024-02-05T08:13:17.988040+0000","last_peered":"2024-02-05T08:13:17.988040+0000","last_clean":"2024-02-05T08:12:40.643159+0000","last_became_active":"2024-02-05T08:12:42.786875+0000","last_became_peered":"2024-02-05T08:12:42.786875+0000","last_unstale":"2024-02-05T08:13:17.988040+0000","last_undegraded":"2024-02-05T08:13:17.988040+0000","last_fullsized":"2024-02-05T08:13:17.988040+0000","mapping_epoch":194,"log_start":"176'4300","ondisk_log_start":"176'4300","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5014,"log_dups_size":0,"ondisk_log_size":5014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352845824,"num_objects":5384,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1451,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9314,"num_write_kb":470336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4547,"num_bytes_recovered":297992192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,3],"avail_no_missing":["2","1","3"],"object_location_counts":[{"shards":"1,2,3","objects":5384}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":323,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.002136+0000","last_change":"2024-02-05T08:12:29.673477+0000","last_active":"2024-02-05T08:13:18.002136+0000","last_peered":"2024-02-05T08:13:18.002136+0000","last_clean":"2024-02-05T08:13:18.002136+0000","last_became_active":"2024-02-05T08:12:29.672973+0000","last_became_peered":"2024-02-05T08:12:29.672973+0000","last_unstale":"2024-02-05T08:13:18.002136+0000","last_undegraded":"2024-02-05T08:13:18.002136+0000","last_fullsized":"2024-02-05T08:13:18.002136+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:25.219172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:25.219172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:43.872593+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":323,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.998833+0000","last_change":"2024-02-05T08:11:18.828143+0000","last_active":"2024-02-05T08:13:17.998833+0000","last_peered":"2024-02-05T08:13:17.998833+0000","last_clean":"2024-02-05T08:13:17.998833+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:13:17.998833+0000","last_undegraded":"2024-02-05T08:13:17.998833+0000","last_fullsized":"2024-02-05T08:13:17.998833+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:18.828029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:18.828029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":354,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.999374+0000","last_change":"2024-02-05T08:12:27.724075+0000","last_active":"2024-02-05T08:13:17.999374+0000","last_peered":"2024-02-05T08:13:17.999374+0000","last_clean":"2024-02-05T08:13:17.999374+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:13:17.999374+0000","last_undegraded":"2024-02-05T08:13:17.999374+0000","last_fullsized":"2024-02-05T08:13:17.999374+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:12:27.723924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:27.723924+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:30.337919+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":43,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953218+0000","last_change":"2024-02-05T08:12:29.688850+0000","last_active":"2024-02-05T08:13:14.953218+0000","last_peered":"2024-02-05T08:13:14.953218+0000","last_clean":"2024-02-05T08:13:14.953218+0000","last_became_active":"2024-02-05T08:12:29.688534+0000","last_became_peered":"2024-02-05T08:12:29.688534+0000","last_unstale":"2024-02-05T08:13:14.953218+0000","last_undegraded":"2024-02-05T08:13:14.953218+0000","last_fullsized":"2024-02-05T08:13:14.953218+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:21.296072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:11:21.296072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"205'9316","reported_seq":9423,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.372829+0000","last_change":"2024-02-05T08:10:54.987464+0000","last_active":"2024-02-05T08:13:18.372829+0000","last_peered":"2024-02-05T08:13:18.372829+0000","last_clean":"2024-02-05T08:13:18.372829+0000","last_became_active":"2024-02-05T08:10:54.985442+0000","last_became_peered":"2024-02-05T08:10:54.985442+0000","last_unstale":"2024-02-05T08:13:18.372829+0000","last_undegraded":"2024-02-05T08:13:18.372829+0000","last_fullsized":"2024-02-05T08:13:18.372829+0000","mapping_epoch":157,"log_start":"175'4244","ondisk_log_start":"175'4244","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5072,"log_dups_size":0,"ondisk_log_size":5072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357040128,"num_objects":5448,"num_object_clones":0,"num_object_copies":16344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9316,"num_write_kb":472448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":316,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.000908+0000","last_change":"2024-02-05T08:12:41.736005+0000","last_active":"2024-02-05T08:13:18.000908+0000","last_peered":"2024-02-05T08:13:18.000908+0000","last_clean":"2024-02-05T08:13:18.000908+0000","last_became_active":"2024-02-05T08:12:41.735706+0000","last_became_peered":"2024-02-05T08:12:41.735706+0000","last_unstale":"2024-02-05T08:13:18.000908+0000","last_undegraded":"2024-02-05T08:13:18.000908+0000","last_fullsized":"2024-02-05T08:13:18.000908+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.607056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.607056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:31.817287+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":326,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.963991+0000","last_change":"2024-02-05T08:12:41.730442+0000","last_active":"2024-02-05T08:13:14.963991+0000","last_peered":"2024-02-05T08:13:14.963991+0000","last_clean":"2024-02-05T08:13:14.963991+0000","last_became_active":"2024-02-05T08:12:41.730010+0000","last_became_peered":"2024-02-05T08:12:41.730010+0000","last_unstale":"2024-02-05T08:13:14.963991+0000","last_undegraded":"2024-02-05T08:13:14.963991+0000","last_fullsized":"2024-02-05T08:13:14.963991+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:26.212363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:26.212363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":334,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.999892+0000","last_change":"2024-02-05T08:12:19.056472+0000","last_active":"2024-02-05T08:13:17.999892+0000","last_peered":"2024-02-05T08:13:17.999892+0000","last_clean":"2024-02-05T08:13:17.999892+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:13:17.999892+0000","last_undegraded":"2024-02-05T08:13:17.999892+0000","last_fullsized":"2024-02-05T08:13:17.999892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:12:19.056345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:19.056345+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:20.775637+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"190'10","reported_seq":438,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.999628+0000","last_change":"2024-02-05T08:12:29.716459+0000","last_active":"2024-02-05T08:13:17.999628+0000","last_peered":"2024-02-05T08:13:17.999628+0000","last_clean":"2024-02-05T08:13:17.999628+0000","last_became_active":"2024-02-05T08:12:29.683091+0000","last_became_peered":"2024-02-05T08:12:29.683091+0000","last_unstale":"2024-02-05T08:13:17.999628+0000","last_undegraded":"2024-02-05T08:13:17.999628+0000","last_fullsized":"2024-02-05T08:13:17.999628+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-02-05T08:11:13.252683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:13.252683+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"204'9377","reported_seq":9483,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.635205+0000","last_change":"2024-02-05T08:10:54.986461+0000","last_active":"2024-02-05T08:13:17.635205+0000","last_peered":"2024-02-05T08:13:17.635205+0000","last_clean":"2024-02-05T08:13:17.635205+0000","last_became_active":"2024-02-05T08:10:54.986149+0000","last_became_peered":"2024-02-05T08:10:54.986149+0000","last_unstale":"2024-02-05T08:13:17.635205+0000","last_undegraded":"2024-02-05T08:13:17.635205+0000","last_fullsized":"2024-02-05T08:13:17.635205+0000","mapping_epoch":157,"log_start":"177'4511","ondisk_log_start":"177'4511","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365953048,"num_objects":5585,"num_object_clones":0,"num_object_copies":16755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5585,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9377,"num_write_kb":478721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":313,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117558+0000","last_change":"2024-02-05T08:12:29.678526+0000","last_active":"2024-02-05T08:13:15.117558+0000","last_peered":"2024-02-05T08:13:15.117558+0000","last_clean":"2024-02-05T08:13:15.117558+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:13:15.117558+0000","last_undegraded":"2024-02-05T08:13:15.117558+0000","last_fullsized":"2024-02-05T08:13:15.117558+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:09.466721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:09.466721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":332,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.002437+0000","last_change":"2024-02-05T08:12:21.623410+0000","last_active":"2024-02-05T08:13:18.002437+0000","last_peered":"2024-02-05T08:13:18.002437+0000","last_clean":"2024-02-05T08:13:18.002437+0000","last_became_active":"2024-02-05T08:11:23.200753+0000","last_became_peered":"2024-02-05T08:11:23.200753+0000","last_unstale":"2024-02-05T08:13:18.002437+0000","last_undegraded":"2024-02-05T08:13:18.002437+0000","last_fullsized":"2024-02-05T08:13:18.002437+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:12:21.623291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:21.623291+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:47.267748+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":373,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.950826+0000","last_change":"2024-02-05T08:12:29.712640+0000","last_active":"2024-02-05T08:13:14.950826+0000","last_peered":"2024-02-05T08:13:14.950826+0000","last_clean":"2024-02-05T08:13:14.950826+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:13:14.950826+0000","last_undegraded":"2024-02-05T08:13:14.950826+0000","last_fullsized":"2024-02-05T08:13:14.950826+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-02-05T08:12:27.883928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:27.883928+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:48.176622+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"194'10","reported_seq":362,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.001438+0000","last_change":"2024-02-05T08:12:41.773528+0000","last_active":"2024-02-05T08:13:18.001438+0000","last_peered":"2024-02-05T08:13:18.001438+0000","last_clean":"2024-02-05T08:13:18.001438+0000","last_became_active":"2024-02-05T08:12:41.721686+0000","last_became_peered":"2024-02-05T08:12:41.721686+0000","last_unstale":"2024-02-05T08:13:18.001438+0000","last_undegraded":"2024-02-05T08:13:18.001438+0000","last_fullsized":"2024-02-05T08:13:18.001438+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'9","last_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:11:27.751188+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"205'9131","reported_seq":10783,"reported_epoch":205,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:13:18.398613+0000","last_change":"2024-02-05T08:13:12.845287+0000","last_active":"2024-02-05T08:13:18.398613+0000","last_peered":"2024-02-05T08:13:18.398613+0000","last_clean":"2024-02-05T08:12:40.686058+0000","last_became_active":"2024-02-05T08:12:42.799583+0000","last_became_peered":"2024-02-05T08:12:42.799583+0000","last_unstale":"2024-02-05T08:13:18.398613+0000","last_undegraded":"2024-02-05T08:13:18.398613+0000","last_fullsized":"2024-02-05T08:13:18.398613+0000","mapping_epoch":194,"log_start":"175'4101","ondisk_log_start":"175'4101","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350552064,"num_objects":5349,"num_object_clones":0,"num_object_copies":16047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4612,"num_objects_unfound":0,"num_objects_dirty":5349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9131,"num_write_kb":463360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":766,"num_bytes_recovered":50135040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":5349}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":314,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.950809+0000","last_change":"2024-02-05T08:12:41.731865+0000","last_active":"2024-02-05T08:13:14.950809+0000","last_peered":"2024-02-05T08:13:14.950809+0000","last_clean":"2024-02-05T08:13:14.950809+0000","last_became_active":"2024-02-05T08:12:41.731253+0000","last_became_peered":"2024-02-05T08:12:41.731253+0000","last_unstale":"2024-02-05T08:13:14.950809+0000","last_undegraded":"2024-02-05T08:13:14.950809+0000","last_fullsized":"2024-02-05T08:13:14.950809+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:25.851312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:25.851312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:44.639876+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":44,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117844+0000","last_change":"2024-02-05T08:12:41.761180+0000","last_active":"2024-02-05T08:13:15.117844+0000","last_peered":"2024-02-05T08:13:15.117844+0000","last_clean":"2024-02-05T08:13:15.117844+0000","last_became_active":"2024-02-05T08:12:41.723699+0000","last_became_peered":"2024-02-05T08:12:41.723699+0000","last_unstale":"2024-02-05T08:13:15.117844+0000","last_undegraded":"2024-02-05T08:13:15.117844+0000","last_fullsized":"2024-02-05T08:13:15.117844+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:12:23.575903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:23.575903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:44.725076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"186'26","reported_seq":356,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:15.117183+0000","last_change":"2024-02-05T08:12:41.783220+0000","last_active":"2024-02-05T08:13:15.117183+0000","last_peered":"2024-02-05T08:13:15.117183+0000","last_clean":"2024-02-05T08:13:15.117183+0000","last_became_active":"2024-02-05T08:12:41.728044+0000","last_became_peered":"2024-02-05T08:12:41.728044+0000","last_unstale":"2024-02-05T08:13:15.117183+0000","last_undegraded":"2024-02-05T08:13:15.117183+0000","last_fullsized":"2024-02-05T08:13:15.117183+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'26","last_scrub_stamp":"2024-02-05T08:12:24.579532+0000","last_deep_scrub":"186'26","last_deep_scrub_stamp":"2024-02-05T08:12:24.579532+0000","last_clean_scrub_stamp":"2024-02-05T08:12:24.579532+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:28.490849+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":310,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.950595+0000","last_change":"2024-02-05T08:12:41.720557+0000","last_active":"2024-02-05T08:13:14.950595+0000","last_peered":"2024-02-05T08:13:14.950595+0000","last_clean":"2024-02-05T08:13:14.950595+0000","last_became_active":"2024-02-05T08:12:41.720247+0000","last_became_peered":"2024-02-05T08:12:41.720247+0000","last_unstale":"2024-02-05T08:13:14.950595+0000","last_undegraded":"2024-02-05T08:13:14.950595+0000","last_fullsized":"2024-02-05T08:13:14.950595+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:28.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:28.185891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"205'9202","reported_seq":10279,"reported_epoch":205,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:13:18.091804+0000","last_change":"2024-02-05T08:13:17.974631+0000","last_active":"2024-02-05T08:13:18.091804+0000","last_peered":"2024-02-05T08:13:18.091804+0000","last_clean":"2024-02-05T08:12:29.029449+0000","last_became_active":"2024-02-05T08:12:30.750855+0000","last_became_peered":"2024-02-05T08:12:30.750855+0000","last_unstale":"2024-02-05T08:13:18.091804+0000","last_undegraded":"2024-02-05T08:13:18.091804+0000","last_fullsized":"2024-02-05T08:13:18.091804+0000","mapping_epoch":188,"log_start":"178'5068","ondisk_log_start":"178'5068","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4134,"log_dups_size":0,"ondisk_log_size":4134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354418688,"num_objects":5408,"num_object_clones":0,"num_object_copies":16224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5046,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9202,"num_write_kb":467520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":477,"num_bytes_recovered":31260672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,2],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":5408}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":307,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.963712+0000","last_change":"2024-02-05T08:11:08.491427+0000","last_active":"2024-02-05T08:13:14.963712+0000","last_peered":"2024-02-05T08:13:14.963712+0000","last_clean":"2024-02-05T08:13:14.963712+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:13:14.963712+0000","last_undegraded":"2024-02-05T08:13:14.963712+0000","last_fullsized":"2024-02-05T08:13:14.963712+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:08.491314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:08.491314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":415,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.952069+0000","last_change":"2024-02-05T08:11:30.179068+0000","last_active":"2024-02-05T08:13:14.952069+0000","last_peered":"2024-02-05T08:13:14.952069+0000","last_clean":"2024-02-05T08:13:14.952069+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:13:14.952069+0000","last_undegraded":"2024-02-05T08:13:14.952069+0000","last_fullsized":"2024-02-05T08:13:14.952069+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:11:30.178959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:30.178959+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":34,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.001654+0000","last_change":"2024-02-05T08:12:41.725451+0000","last_active":"2024-02-05T08:13:18.001654+0000","last_peered":"2024-02-05T08:13:18.001654+0000","last_clean":"2024-02-05T08:13:18.001654+0000","last_became_active":"2024-02-05T08:12:41.724969+0000","last_became_peered":"2024-02-05T08:12:41.724969+0000","last_unstale":"2024-02-05T08:13:18.001654+0000","last_undegraded":"2024-02-05T08:13:18.001654+0000","last_fullsized":"2024-02-05T08:13:18.001654+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:03.761546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:11:03.761546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":365,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.002350+0000","last_change":"2024-02-05T08:12:41.775704+0000","last_active":"2024-02-05T08:13:18.002350+0000","last_peered":"2024-02-05T08:13:18.002350+0000","last_clean":"2024-02-05T08:13:18.002350+0000","last_became_active":"2024-02-05T08:12:41.725007+0000","last_became_peered":"2024-02-05T08:12:41.725007+0000","last_unstale":"2024-02-05T08:13:18.002350+0000","last_undegraded":"2024-02-05T08:13:18.002350+0000","last_fullsized":"2024-02-05T08:13:18.002350+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:11:30.843620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:30.843620+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"204'9146","reported_seq":9251,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.568818+0000","last_change":"2024-02-05T08:10:54.985813+0000","last_active":"2024-02-05T08:13:17.568818+0000","last_peered":"2024-02-05T08:13:17.568818+0000","last_clean":"2024-02-05T08:13:17.568818+0000","last_became_active":"2024-02-05T08:10:54.985501+0000","last_became_peered":"2024-02-05T08:10:54.985501+0000","last_unstale":"2024-02-05T08:13:17.568818+0000","last_undegraded":"2024-02-05T08:13:17.568818+0000","last_fullsized":"2024-02-05T08:13:17.568818+0000","mapping_epoch":157,"log_start":"178'4710","ondisk_log_start":"178'4710","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4436,"log_dups_size":0,"ondisk_log_size":4436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351928320,"num_objects":5370,"num_object_clones":0,"num_object_copies":16110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9146,"num_write_kb":464512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":302,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953197+0000","last_change":"2024-02-05T08:12:41.720396+0000","last_active":"2024-02-05T08:13:14.953197+0000","last_peered":"2024-02-05T08:13:14.953197+0000","last_clean":"2024-02-05T08:13:14.953197+0000","last_became_active":"2024-02-05T08:12:41.719835+0000","last_became_peered":"2024-02-05T08:12:41.719835+0000","last_unstale":"2024-02-05T08:13:14.953197+0000","last_undegraded":"2024-02-05T08:13:14.953197+0000","last_fullsized":"2024-02-05T08:13:14.953197+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:26.583813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.583813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:29.636244+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":323,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.963200+0000","last_change":"2024-02-05T08:12:29.686189+0000","last_active":"2024-02-05T08:13:14.963200+0000","last_peered":"2024-02-05T08:13:14.963200+0000","last_clean":"2024-02-05T08:13:14.963200+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:13:14.963200+0000","last_undegraded":"2024-02-05T08:13:14.963200+0000","last_fullsized":"2024-02-05T08:13:14.963200+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:29.241689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:29.241689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":319,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.950689+0000","last_change":"2024-02-05T08:11:31.201157+0000","last_active":"2024-02-05T08:13:14.950689+0000","last_peered":"2024-02-05T08:13:14.950689+0000","last_clean":"2024-02-05T08:13:14.950689+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:13:14.950689+0000","last_undegraded":"2024-02-05T08:13:14.950689+0000","last_fullsized":"2024-02-05T08:13:14.950689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:11:31.201062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":362,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.951851+0000","last_change":"2024-02-05T08:12:29.702808+0000","last_active":"2024-02-05T08:13:14.951851+0000","last_peered":"2024-02-05T08:13:14.951851+0000","last_clean":"2024-02-05T08:13:14.951851+0000","last_became_active":"2024-02-05T08:12:29.666284+0000","last_became_peered":"2024-02-05T08:12:29.666284+0000","last_unstale":"2024-02-05T08:13:14.951851+0000","last_undegraded":"2024-02-05T08:13:14.951851+0000","last_fullsized":"2024-02-05T08:13:14.951851+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:11:10.470749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:10.470749+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"205'9131","reported_seq":9238,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:18.072088+0000","last_change":"2024-02-05T08:10:54.982668+0000","last_active":"2024-02-05T08:13:18.072088+0000","last_peered":"2024-02-05T08:13:18.072088+0000","last_clean":"2024-02-05T08:13:18.072088+0000","last_became_active":"2024-02-05T08:10:54.982372+0000","last_became_peered":"2024-02-05T08:10:54.982372+0000","last_unstale":"2024-02-05T08:13:18.072088+0000","last_undegraded":"2024-02-05T08:13:18.072088+0000","last_fullsized":"2024-02-05T08:13:18.072088+0000","mapping_epoch":157,"log_start":"178'4926","ondisk_log_start":"178'4926","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:10:53.958581+0000","objects_scrubbed":0,"log_size":4205,"log_dups_size":0,"ondisk_log_size":4205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356319232,"num_objects":5437,"num_object_clones":0,"num_object_copies":16311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9131,"num_write_kb":466176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":327,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.954732+0000","last_change":"2024-02-05T08:12:41.755839+0000","last_active":"2024-02-05T08:13:14.954732+0000","last_peered":"2024-02-05T08:13:14.954732+0000","last_clean":"2024-02-05T08:13:14.954732+0000","last_became_active":"2024-02-05T08:12:41.737224+0000","last_became_peered":"2024-02-05T08:12:41.737224+0000","last_unstale":"2024-02-05T08:13:14.954732+0000","last_undegraded":"2024-02-05T08:13:14.954732+0000","last_fullsized":"2024-02-05T08:13:14.954732+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:12:26.409403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.409403+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:47.963356+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":306,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.999141+0000","last_change":"2024-02-05T08:11:11.789034+0000","last_active":"2024-02-05T08:13:17.999141+0000","last_peered":"2024-02-05T08:13:17.999141+0000","last_clean":"2024-02-05T08:13:17.999141+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:13:17.999141+0000","last_undegraded":"2024-02-05T08:13:17.999141+0000","last_fullsized":"2024-02-05T08:13:17.999141+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:11.788902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:11:11.788902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":43,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953525+0000","last_change":"2024-02-05T08:12:29.678813+0000","last_active":"2024-02-05T08:13:14.953525+0000","last_peered":"2024-02-05T08:13:14.953525+0000","last_clean":"2024-02-05T08:13:14.953525+0000","last_became_active":"2024-02-05T08:12:29.678501+0000","last_became_peered":"2024-02-05T08:12:29.678501+0000","last_unstale":"2024-02-05T08:13:14.953525+0000","last_undegraded":"2024-02-05T08:13:14.953525+0000","last_fullsized":"2024-02-05T08:13:14.953525+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:10:25.446724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:10:25.446724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":335,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.951676+0000","last_change":"2024-02-05T08:12:17.870737+0000","last_active":"2024-02-05T08:13:14.951676+0000","last_peered":"2024-02-05T08:13:14.951676+0000","last_clean":"2024-02-05T08:13:14.951676+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:13:14.951676+0000","last_undegraded":"2024-02-05T08:13:14.951676+0000","last_fullsized":"2024-02-05T08:13:14.951676+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-02-05T08:12:17.870617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:17.870617+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:18.886680+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"204'9247","reported_seq":22609,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.632080+0000","last_change":"2024-02-05T08:13:08.671123+0000","last_active":"2024-02-05T08:13:17.632080+0000","last_peered":"2024-02-05T08:13:17.632080+0000","last_clean":"2024-02-05T08:13:17.632080+0000","last_became_active":"2024-02-05T08:13:05.683652+0000","last_became_peered":"2024-02-05T08:13:05.683652+0000","last_unstale":"2024-02-05T08:13:17.632080+0000","last_undegraded":"2024-02-05T08:13:17.632080+0000","last_fullsized":"2024-02-05T08:13:17.632080+0000","mapping_epoch":200,"log_start":"177'4400","ondisk_log_start":"177'4400","created":157,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'8775","last_scrub_stamp":"2024-02-05T08:13:08.670820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:53.958581+0000","last_clean_scrub_stamp":"2024-02-05T08:13:08.670820+0000","objects_scrubbed":5931,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:14:22.484964+0000","scrub_duration":1562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354091008,"num_objects":5403,"num_object_clones":0,"num_object_copies":16209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9247,"num_write_kb":468800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6589,"num_bytes_recovered":431816704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":299,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.952060+0000","last_change":"2024-02-05T08:11:15.360217+0000","last_active":"2024-02-05T08:13:14.952060+0000","last_peered":"2024-02-05T08:13:14.952060+0000","last_clean":"2024-02-05T08:13:14.952060+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:13:14.952060+0000","last_undegraded":"2024-02-05T08:13:14.952060+0000","last_fullsized":"2024-02-05T08:13:14.952060+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:11:15.360090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":433,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.953808+0000","last_change":"2024-02-05T08:12:19.354343+0000","last_active":"2024-02-05T08:13:14.953808+0000","last_peered":"2024-02-05T08:13:14.953808+0000","last_clean":"2024-02-05T08:13:14.953808+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:13:14.953808+0000","last_undegraded":"2024-02-05T08:13:14.953808+0000","last_fullsized":"2024-02-05T08:13:14.953808+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:12:19.351324+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:12:19.351324+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:22.424067+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":34,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-02-05T08:13:17.999460+0000","last_change":"2024-02-05T08:12:41.757481+0000","last_active":"2024-02-05T08:13:17.999460+0000","last_peered":"2024-02-05T08:13:17.999460+0000","last_clean":"2024-02-05T08:13:17.999460+0000","last_became_active":"2024-02-05T08:12:41.757159+0000","last_became_peered":"2024-02-05T08:12:41.757159+0000","last_unstale":"2024-02-05T08:13:17.999460+0000","last_undegraded":"2024-02-05T08:13:17.999460+0000","last_fullsized":"2024-02-05T08:13:17.999460+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:16.521723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:16.521723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:13:23.352820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":376,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-02-05T08:13:14.951310+0000","last_change":"2024-02-05T08:12:41.770037+0000","last_active":"2024-02-05T08:13:14.951310+0000","last_peered":"2024-02-05T08:13:14.951310+0000","last_clean":"2024-02-05T08:13:14.951310+0000","last_became_active":"2024-02-05T08:12:41.727271+0000","last_became_peered":"2024-02-05T08:12:41.727271+0000","last_unstale":"2024-02-05T08:13:14.951310+0000","last_undegraded":"2024-02-05T08:13:14.951310+0000","last_fullsized":"2024-02-05T08:13:14.951310+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:11:14.388825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:14.388825+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5659557912,"num_objects":86359,"num_object_clones":0,"num_object_copies":259077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":25458,"num_objects_unfound":0,"num_objects_dirty":86359,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":146957,"num_write_kb":7466049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33444,"num_bytes_recovered":2191523840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18469433344,"data_stored":18469421128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76863,"ondisk_log_size":76863,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":510,"num_read_kb":505,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":694,"num_read_kb":451,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":213,"num_bytes_recovered":3922,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":12894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":443,"ondisk_log_size":443,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037868,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1917940,"kb_used_data":1742380,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91827212,"statfs":{"total":95995035648,"available":94031065088,"internally_reserved":0,"allocated":1784197120,"data_stored":1783876471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":179763658},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.946,"15min":1.0620000000000001},"min":{"1min":0.22,"5min":0.218,"15min":0.217},"max":{"1min":4.819,"5min":60.127000000000002,"15min":60.127000000000002},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.94699999999999995,"5min":1.9790000000000001,"15min":1.0429999999999999},"min":{"1min":0.248,"5min":0.23300000000000001,"15min":0.22500000000000001},"max":{"1min":4.2300000000000004,"5min":58.283000000000001,"15min":58.283000000000001},"last":0.35999999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":2.7349999999999999,"5min":2.1909999999999998,"15min":1.123},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":32.682000000000002,"5min":38.399999999999999,"15min":38.399999999999999},"last":4.4020000000000001},{"interface":"front","average":{"1min":2.7410000000000001,"5min":1.9219999999999999,"15min":1.02},"min":{"1min":0.312,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":32.412999999999997,"5min":38.195999999999998,"15min":38.195999999999998},"last":0.63400000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.139,"15min":0.76500000000000001},"min":{"1min":0.26800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.2629999999999999,"5min":7.6779999999999999,"15min":7.6779999999999999},"last":2.5720000000000001},{"interface":"front","average":{"1min":1.575,"5min":1.3049999999999999,"15min":0.84499999999999997},"min":{"1min":0.28499999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.8600000000000003,"5min":8.9100000000000001,"15min":8.9100000000000001},"last":2.827}]},{"osd":3,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":2.0009999999999999,"15min":1.101},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":2.7050000000000001,"5min":58.515000000000001,"15min":58.515000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":2.0430000000000001,"15min":1.0780000000000001},"min":{"1min":0.27400000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.6879999999999997,"5min":58.225999999999999,"15min":58.225999999999999},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":0.624,"15min":0.498},"min":{"1min":0.23699999999999999,"5min":0.23100000000000001,"15min":0.22},"max":{"1min":19.838999999999999,"5min":19.838999999999999,"15min":19.838999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.6419999999999999,"5min":0.65400000000000003,"15min":0.50900000000000001},"min":{"1min":0.28999999999999998,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":20.113,"5min":20.113,"15min":20.113},"last":0.375}]},{"osd":5,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.46999999999999997,"15min":0.45400000000000001},"min":{"1min":0.251,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.224,"5min":1.752,"15min":1.752},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.47799999999999998,"15min":0.48699999999999999},"min":{"1min":0.33900000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.80300000000000005,"5min":0.99199999999999999,"15min":0.998},"last":0.46200000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:12:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47399999999999998,"15min":0.496},"min":{"1min":0.32800000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.72299999999999998,"5min":1.0269999999999999,"15min":1.256},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.5,"15min":0.48699999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.433}]}]},{"osd":6,"up_from":51,"seq":219043332210,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1183816,"kb_used_data":1054076,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92561336,"statfs":{"total":95995035648,"available":94782808064,"internally_reserved":0,"allocated":1079373824,"data_stored":1079036097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":132839878},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:13:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.04,"15min":0.80600000000000005},"min":{"1min":0.22900000000000001,"5min":0.22700000000000001,"15min":0.223},"max":{"1min":6.1500000000000004,"5min":12.096,"15min":12.096},"last":0.24099999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.3260000000000001,"15min":0.92800000000000005},"min":{"1min":0.27900000000000003,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":4.0780000000000003,"5min":12.675000000000001,"15min":12.675000000000001},"last":0.57999999999999996}]},{"osd":1,"last update":"Mon Feb 5 08:13:16 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.119,"15min":0.83499999999999996},"min":{"1min":0.24199999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":7.5830000000000002,"5min":7.5830000000000002,"15min":7.5830000000000002},"last":0.307},{"interface":"front","average":{"1min":1.319,"5min":1.194,"15min":0.81000000000000005},"min":{"1min":0.33900000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.7220000000000004,"5min":5.7220000000000004,"15min":5.7220000000000004},"last":0.75600000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:13:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.234,"15min":0.86399999999999999},"min":{"1min":0.31900000000000001,"5min":0.28599999999999998,"15min":0.27200000000000002},"max":{"1min":5.335,"5min":5.9800000000000004,"15min":6.282},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.028,"15min":0.79300000000000004},"min":{"1min":0.30499999999999999,"5min":0.249,"15min":0.24199999999999999},"max":{"1min":2.048,"5min":6.2699999999999996,"15min":6.2699999999999996},"last":0.63500000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:13:16 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":1.2070000000000001,"15min":0.87},"min":{"1min":0.308,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.4449999999999998,"5min":12.32,"15min":12.32},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.069,"15min":0.86299999999999999},"min":{"1min":0.316,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.9370000000000003,"5min":12.146000000000001,"15min":12.762},"last":0.82299999999999995}]},{"osd":4,"last update":"Mon Feb 5 08:13:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.46300000000000002,"15min":0.44400000000000001},"min":{"1min":0.26700000000000002,"5min":0.249,"15min":0.223},"max":{"1min":2.5569999999999999,"5min":2.5569999999999999,"15min":2.5569999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.50900000000000001,"15min":0.46700000000000003},"min":{"1min":0.30599999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":2.7570000000000001},"last":0.84899999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.7429999999999999,"5min":0.97899999999999998,"15min":0.64500000000000002},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":43.643000000000001,"5min":43.643000000000001,"15min":43.643000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":2.6669999999999998,"5min":1.02,"15min":0.66500000000000004},"min":{"1min":0.307,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":43.893000000000001,"5min":43.893000000000001,"15min":43.893000000000001},"last":0.88700000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53500000000000003,"15min":0.52600000000000002},"min":{"1min":0.34799999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":2.5329999999999999,"5min":3.5529999999999999,"15min":3.5529999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.53000000000000003,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.33700000000000002,"15min":0.29299999999999998},"max":{"1min":0.59299999999999997,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.73099999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2737096,"kb_used_data":2426752,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91008056,"statfs":{"total":95995035648,"available":93192249344,"internally_reserved":0,"allocated":2484994048,"data_stored":2484652169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:13:09 2024","interfaces":[{"interface":"back","average":{"1min":2.0510000000000002,"5min":1.8149999999999999,"15min":1.2230000000000001},"min":{"1min":0.28199999999999997,"5min":0.186,"15min":0.14699999999999999},"max":{"1min":4.6369999999999996,"5min":7.7300000000000004,"15min":86.105000000000004},"last":0.40600000000000003},{"interface":"front","average":{"1min":2.0030000000000001,"5min":2.1930000000000001,"15min":1.421},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.16},"max":{"1min":8.0640000000000001,"5min":13.800000000000001,"15min":85.665000000000006},"last":0.63300000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:12:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.6200000000000001,"15min":1.0529999999999999},"min":{"1min":0.219,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":3.9750000000000001,"5min":9.3840000000000003,"15min":49.030999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.431,"5min":1.831,"15min":1.1679999999999999},"min":{"1min":0.23599999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.1459999999999999,"5min":13.707000000000001,"15min":48.984000000000002},"last":0.29199999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:13:01 2024","interfaces":[{"interface":"back","average":{"1min":2.3759999999999999,"5min":1.887,"15min":1.25},"min":{"1min":0.39700000000000002,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.8460000000000001,"5min":12.468999999999999,"15min":49.067999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.355,"5min":1.6259999999999999,"15min":1.206},"min":{"1min":0.36199999999999999,"5min":0.27800000000000002,"15min":0.24199999999999999},"max":{"1min":3.2919999999999998,"5min":9.5120000000000005,"15min":48.734999999999999},"last":0.495}]},{"osd":4,"last update":"Mon Feb 5 08:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":1.087,"15min":0.76300000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.593,"5min":9.4269999999999996,"15min":9.4269999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.224,"5min":1.3420000000000001,"15min":0.83699999999999997},"min":{"1min":0.30199999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.258,"5min":6.3890000000000002,"15min":6.3890000000000002},"last":0.80300000000000005}]},{"osd":5,"last update":"Mon Feb 5 08:12:51 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.173,"15min":0.80800000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.4430000000000001,"5min":9.4559999999999995,"15min":9.4559999999999995},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.95099999999999996,"5min":1.355,"15min":0.878},"min":{"1min":0.47999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.9770000000000001,"5min":13.776,"15min":13.776},"last":0.51500000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:12:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4039999999999999,"5min":1.373,"15min":0.85499999999999998},"min":{"1min":0.38900000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.3449999999999998,"5min":6.6319999999999997,"15min":6.6319999999999997},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":1.2030000000000001,"15min":0.82799999999999996},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.6519999999999999,"5min":9.4800000000000004,"15min":9.4800000000000004},"last":0.81699999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:12:37 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":1.494,"15min":0.94299999999999995},"min":{"1min":0.55500000000000005,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.974,"5min":13.749000000000001,"15min":13.749000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.466,"5min":1.4550000000000001,"15min":0.92300000000000004},"min":{"1min":0.48399999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.585,"5min":6.6029999999999998,"15min":6.6029999999999998},"last":0.74399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738512,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2745348,"kb_used_data":2465656,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":90999804,"statfs":{"total":95995035648,"available":93183799296,"internally_reserved":0,"allocated":2524831744,"data_stored":2524506921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:12:17 2024","interfaces":[{"interface":"back","average":{"1min":3.3100000000000001,"5min":3.5270000000000001,"15min":1.4359999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.152},"max":{"1min":16.228000000000002,"5min":66.772999999999996,"15min":66.772999999999996},"last":3.6389999999999998},{"interface":"front","average":{"1min":2.7719999999999998,"5min":2.5569999999999999,"15min":1.097},"min":{"1min":0.218,"5min":0.218,"15min":0.17199999999999999},"max":{"1min":11.247999999999999,"5min":46.198999999999998,"15min":46.198999999999998},"last":3.3050000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:12:43 2024","interfaces":[{"interface":"back","average":{"1min":2.0270000000000001,"5min":2.3010000000000002,"15min":1.1319999999999999},"min":{"1min":0.23699999999999999,"5min":0.216,"15min":0.216},"max":{"1min":7.8579999999999997,"5min":46.170999999999999,"15min":46.170999999999999},"last":4.8120000000000003},{"interface":"front","average":{"1min":2.5619999999999998,"5min":2.75,"15min":1.302},"min":{"1min":0.20399999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":9.1210000000000004,"5min":46.543999999999997,"15min":46.543999999999997},"last":3.5489999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:13:11 2024","interfaces":[{"interface":"back","average":{"1min":2.2559999999999998,"5min":3.1869999999999998,"15min":1.49},"min":{"1min":0.35499999999999998,"5min":0.26900000000000002,"15min":0.21099999999999999},"max":{"1min":5.9089999999999998,"5min":67.888000000000005,"15min":67.888000000000005},"last":4.391},{"interface":"front","average":{"1min":3.8319999999999999,"5min":4.1189999999999998,"15min":1.825},"min":{"1min":0.439,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":15.494,"5min":66.891999999999996,"15min":66.891999999999996},"last":3.613}]},{"osd":4,"last update":"Mon Feb 5 08:12:34 2024","interfaces":[{"interface":"back","average":{"1min":1.7669999999999999,"5min":1.415,"15min":0.85099999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":7.5709999999999997,"5min":13.295,"15min":13.295},"last":3.677},{"interface":"front","average":{"1min":1.7210000000000001,"5min":1.458,"15min":0.90300000000000002},"min":{"1min":0.29999999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":10.928000000000001,"5min":12.989000000000001,"15min":12.989000000000001},"last":3.3799999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:12:52 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.454,"15min":0.96299999999999997},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":4.2190000000000003,"5min":9.1660000000000004,"15min":13.073},"last":3.492},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.3260000000000001,"15min":0.93200000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.782,"5min":9.5289999999999999,"15min":12.94},"last":4.008}]},{"osd":6,"last update":"Mon Feb 5 08:12:17 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.4650000000000001,"15min":0.90700000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":5.8559999999999999,"5min":12.967000000000001,"15min":12.967000000000001},"last":3.4649999999999999},{"interface":"front","average":{"1min":1.9710000000000001,"5min":1.6040000000000001,"15min":0.95899999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":9.1929999999999996,"5min":13.108000000000001,"15min":13.108000000000001},"last":3.5779999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:12:43 2024","interfaces":[{"interface":"back","average":{"1min":1.9379999999999999,"5min":1.3300000000000001,"15min":0.878},"min":{"1min":0.374,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":7.6020000000000003,"5min":7.6020000000000003,"15min":13.263},"last":4.1340000000000003},{"interface":"front","average":{"1min":2.1869999999999998,"5min":1.4650000000000001,"15min":0.98199999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":6.718,"5min":9.6300000000000008,"15min":13.012},"last":4.7430000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411462,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3478076,"kb_used_data":3172020,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":90267076,"statfs":{"total":95995035648,"available":92433485824,"internally_reserved":0,"allocated":3248148480,"data_stored":3247818511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":313391558},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:12:46 2024","interfaces":[{"interface":"back","average":{"1min":6.1449999999999996,"5min":3.073,"15min":1.3640000000000001},"min":{"1min":0.26200000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":40.354999999999997,"5min":40.354999999999997,"15min":40.354999999999997},"last":3.9470000000000001},{"interface":"front","average":{"1min":6.2599999999999998,"5min":3.206,"15min":1.419},"min":{"1min":0.35099999999999998,"5min":0.24199999999999999,"15min":0.20699999999999999},"max":{"1min":40.566000000000003,"5min":40.566000000000003,"15min":40.566000000000003},"last":2.9580000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:12:46 2024","interfaces":[{"interface":"back","average":{"1min":4.875,"5min":2.6560000000000001,"15min":1.2130000000000001},"min":{"1min":0.20000000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":55.942,"5min":55.942,"15min":55.942},"last":1.1259999999999999},{"interface":"front","average":{"1min":5.5519999999999996,"5min":2.8359999999999999,"15min":1.2709999999999999},"min":{"1min":0.22900000000000001,"5min":0.193,"15min":0.193},"max":{"1min":55.960999999999999,"5min":55.960999999999999,"15min":55.960999999999999},"last":0.96899999999999997}]},{"osd":3,"last update":"Mon Feb 5 08:13:05 2024","interfaces":[{"interface":"back","average":{"1min":2.254,"5min":3.4100000000000001,"15min":1.597},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":6.3559999999999999,"5min":68.108000000000004,"15min":68.108000000000004},"last":0.80400000000000005},{"interface":"front","average":{"1min":2.1600000000000001,"5min":3.173,"15min":1.4690000000000001},"min":{"1min":0.23999999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":6.8600000000000003,"5min":67.945999999999998,"15min":67.945999999999998},"last":3.98}]},{"osd":4,"last update":"Mon Feb 5 08:12:24 2024","interfaces":[{"interface":"back","average":{"1min":2.0150000000000001,"5min":2.0470000000000002,"15min":1.095},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":7.4359999999999999,"5min":31.596,"15min":31.596},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.617,"15min":0.93999999999999995},"min":{"1min":0.33100000000000002,"5min":0.252,"15min":0.252},"max":{"1min":6.149,"5min":31.57,"15min":31.57},"last":1.988}]},{"osd":5,"last update":"Mon Feb 5 08:12:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":2.0150000000000001,"15min":1.1040000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":6.0910000000000002,"5min":58.799999999999997,"15min":58.799999999999997},"last":2.0449999999999999},{"interface":"front","average":{"1min":2.0579999999999998,"5min":2.0920000000000001,"15min":1.139},"min":{"1min":0.34999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":6.593,"5min":58.615000000000002,"15min":58.615000000000002},"last":0.38800000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:12:24 2024","interfaces":[{"interface":"back","average":{"1min":2.0110000000000001,"5min":1.641,"15min":0.98299999999999998},"min":{"1min":0.36299999999999999,"5min":0.308,"15min":0.308},"max":{"1min":7.5099999999999998,"5min":15.859,"15min":15.859},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.2729999999999999,"15min":0.86199999999999999},"min":{"1min":0.41199999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":6.1820000000000004,"5min":6.1820000000000004,"15min":6.1820000000000004},"last":2.02}]},{"osd":7,"last update":"Mon Feb 5 08:12:46 2024","interfaces":[{"interface":"back","average":{"1min":1.8320000000000001,"5min":1.6870000000000001,"15min":0.99299999999999999},"min":{"1min":0.40100000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":6.5490000000000004,"5min":15.986000000000001,"15min":15.986000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":1.9970000000000001,"5min":1.7410000000000001,"15min":1.0529999999999999},"min":{"1min":0.40400000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":7.4779999999999998,"5min":15.951000000000001,"15min":15.951000000000001},"last":0.93600000000000005}]}]},{"osd":3,"up_from":0,"seq":115964117121,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953596,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2584300,"kb_used_data":2332388,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91160852,"statfs":{"total":95995035648,"available":93348712448,"internally_reserved":0,"allocated":2388365312,"data_stored":2388050824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:13:16 2024","interfaces":[{"interface":"back","average":{"1min":2.8420000000000001,"5min":1.621,"15min":1.276},"min":{"1min":0.28499999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":20.510000000000002,"5min":27.966000000000001,"15min":73.078999999999994},"last":0.42199999999999999},{"interface":"front","average":{"1min":2.2650000000000001,"5min":1.456,"15min":1.2130000000000001},"min":{"1min":0.214,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":20.495999999999999,"5min":26.097999999999999,"15min":73.412000000000006},"last":2.218}]},{"osd":1,"last update":"Mon Feb 5 08:13:16 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.131,"15min":0.78100000000000003},"min":{"1min":0.248,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.3899999999999997,"5min":8.3409999999999993,"15min":8.3409999999999993},"last":4.4130000000000003},{"interface":"front","average":{"1min":1.629,"5min":0.97899999999999998,"15min":0.70599999999999996},"min":{"1min":0.27600000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":5.1319999999999997,"5min":5.1319999999999997,"15min":5.1319999999999997},"last":1.2490000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:13:16 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":0.98999999999999999,"15min":1.079},"min":{"1min":0.249,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.3010000000000002,"5min":5.9640000000000004,"15min":96.849999999999994},"last":1.395},{"interface":"front","average":{"1min":1.427,"5min":1.1240000000000001,"15min":1.1259999999999999},"min":{"1min":0.28499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":3.6309999999999998,"5min":7.032,"15min":96.143000000000001},"last":2.258}]},{"osd":3,"last update":"Mon Feb 5 08:13:16 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":2.3210000000000002,"15min":1.4750000000000001},"min":{"1min":0.28599999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.9900000000000002,"5min":57.204000000000001,"15min":73.602999999999994},"last":2.323},{"interface":"front","average":{"1min":1.202,"5min":2.3599999999999999,"15min":1.514},"min":{"1min":0.32600000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.3700000000000001,"5min":57.606999999999999,"15min":74.817999999999998},"last":4.3700000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:12:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.48099999999999998,"15min":0.41899999999999998},"min":{"1min":0.25800000000000001,"5min":0.224,"15min":0.224},"max":{"1min":4.4050000000000002,"5min":4.4050000000000002,"15min":4.4050000000000002},"last":4.4459999999999997},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.51100000000000001,"15min":0.44500000000000001},"min":{"1min":0.27500000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":3.7410000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:13:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.51100000000000001,"15min":0.46400000000000002},"min":{"1min":0.32100000000000001,"5min":0.29599999999999999,"15min":0.29299999999999998},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":2.2890000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.51000000000000001,"15min":0.46200000000000002},"min":{"1min":0.32000000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.792,"5min":1.792,"15min":1.792},"last":1.5569999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:12:37 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.33100000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.88800000000000001,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.5,"15min":0.51000000000000001},"min":{"1min":0.30099999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":1.0349999999999999,"5min":1.147,"15min":1.147},"last":1.591}]}]},{"osd":5,"up_from":42,"seq":180388626550,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3085672,"kb_used_data":2833692,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":90659480,"statfs":{"total":95995035648,"available":92835307520,"internally_reserved":0,"allocated":2901700608,"data_stored":2901395183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:12:48 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.831,"15min":1.409},"min":{"1min":0.252,"5min":0.22800000000000001,"15min":0.19400000000000001},"max":{"1min":3.7370000000000001,"5min":46.962000000000003,"15min":143.06299999999999},"last":0.309},{"interface":"front","average":{"1min":1.4410000000000001,"5min":1.8320000000000001,"15min":1.411},"min":{"1min":0.30299999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":5.4859999999999998,"5min":47.067,"15min":143.09700000000001},"last":0.24299999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.379,"15min":0.84199999999999997},"min":{"1min":0.25,"5min":0.23200000000000001,"15min":0.215},"max":{"1min":2.5099999999999998,"5min":46.200000000000003,"15min":46.200000000000003},"last":0.5},{"interface":"front","average":{"1min":1.903,"5min":1.863,"15min":0.98399999999999999},"min":{"1min":0.44500000000000001,"5min":0.246,"15min":0.22900000000000001},"max":{"1min":5.4809999999999999,"5min":46.343000000000004,"15min":46.343000000000004},"last":0.443}]},{"osd":2,"last update":"Mon Feb 5 08:12:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.256,"15min":0.81000000000000005},"min":{"1min":0.26700000000000002,"5min":0.246,"15min":0.246},"max":{"1min":4.0629999999999997,"5min":10.733000000000001,"15min":10.733000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.389,"15min":0.84099999999999997},"min":{"1min":0.309,"5min":0.28399999999999997,"15min":0.26400000000000001},"max":{"1min":8.0220000000000002,"5min":8.0220000000000002,"15min":8.0220000000000002},"last":1.071}]},{"osd":3,"last update":"Mon Feb 5 08:12:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.9099999999999999,"15min":1.472},"min":{"1min":0.36199999999999999,"5min":0.309,"15min":0.309},"max":{"1min":4.3330000000000002,"5min":46.569000000000003,"15min":143.03399999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.8069999999999999,"5min":1.9750000000000001,"15min":1.4770000000000001},"min":{"1min":0.32600000000000001,"5min":0.28899999999999998,"15min":0.28100000000000003},"max":{"1min":7.5270000000000001,"5min":47.820999999999998,"15min":143.46799999999999},"last":7.0309999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.47199999999999998,"15min":0.42599999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.839,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.45200000000000001,"15min":0.434},"min":{"1min":0.28299999999999997,"5min":0.26900000000000002,"15min":0.26500000000000001},"max":{"1min":0.53100000000000003,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.29499999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.51500000000000001,"15min":0.46899999999999997},"min":{"1min":0.33300000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.55700000000000005,"15min":0.51000000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.29299999999999998},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.33000000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.53200000000000003,"15min":0.49399999999999999},"min":{"1min":0.34699999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.0089999999999999,"5min":1.26,"15min":1.26},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.53600000000000003,"15min":0.502},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.28199999999999997},"max":{"1min":0.82399999999999995,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.38800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2520121344,"data_stored":2520121344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2471755776,"data_stored":2471755776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3240759296,"data_stored":3240755224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2107703296,"data_stored":2107703296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2377908224,"data_stored":2377908224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2887061504,"data_stored":2887057432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1088618496,"data_stored":1088618496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1775505408,"data_stored":1775501336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:13:22.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:22.774 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:22.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:22.875 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:22.919 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:22.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:23.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:23.057 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:23.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:23.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:23.159 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:23.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:23.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:23.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:23.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:23 smithi134 ceph-mon[30086]: pgmap v369: 148 pgs: 1 peering, 7 active+remapped+backfilling, 140 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 523 op/s; 25458/259671 objects misplaced (9.804%); 59 MiB/s, 949 objects/s recovering 2024-02-05T08:13:23.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:13:23.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:13:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:23 smithi134 ceph-mon[39278]: pgmap v369: 148 pgs: 1 peering, 7 active+remapped+backfilling, 140 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 523 op/s; 25458/259671 objects misplaced (9.804%); 59 MiB/s, 949 objects/s recovering 2024-02-05T08:13:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:13:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:13:23.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:23 smithi196 ceph-mon[33018]: pgmap v369: 148 pgs: 1 peering, 7 active+remapped+backfilling, 140 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 523 op/s; 25458/259671 objects misplaced (9.804%); 59 MiB/s, 949 objects/s recovering 2024-02-05T08:13:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:13:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:13:23.334 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:23.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:23.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:23.435 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:23.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:23.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:23.570 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:23.611 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:23.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:23.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:23.768 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:23.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:23.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:23.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:13:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:13:23.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:23.912 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:23.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:24.014 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:24.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:24.055 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:24.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:24.127 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:24.150 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:13:24.150 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:13:24.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:24.187 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:24.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:24.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:24.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:24 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:24.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:24 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:24.278 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:24 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:24.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:24.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:24.339 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:24.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:24.440 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:24.483 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:24.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:24.584 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:24.627 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:24.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:24.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:24.760 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:24.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:24.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:24.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:24.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:24.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:24.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:25.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:25.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:25.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:25.177 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:25.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:25.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:25.279 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:25.312 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:25.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:25.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:25 smithi196 ceph-mon[33018]: pgmap v370: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 902 B/s rd, 0 B/s wr, 921 op/s; 18066/249621 objects misplaced (7.237%); 106 MiB/s, 1.69k objects/s recovering 2024-02-05T08:13:25.316 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:25 smithi134 ceph-mon[30086]: pgmap v370: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 902 B/s rd, 0 B/s wr, 921 op/s; 18066/249621 objects misplaced (7.237%); 106 MiB/s, 1.69k objects/s recovering 2024-02-05T08:13:25.316 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:25 smithi134 ceph-mon[39278]: pgmap v370: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 902 B/s rd, 0 B/s wr, 921 op/s; 18066/249621 objects misplaced (7.237%); 106 MiB/s, 1.69k objects/s recovering 2024-02-05T08:13:25.414 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:25.447 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:25.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:25.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:25.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:13:25.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:25.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:25.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:25.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:25.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:25.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:25.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:25.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:25.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:25.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:25.954 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:26.009 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T08:13:26.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:26.033 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:26.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:26.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:26.196 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:26.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:26.298 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:26.348 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:26.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:26.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:26.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:26.499 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:26.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:26.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:26.601 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:26.605 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:26.632 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:26.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:26.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:26.734 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:26.785 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:26.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:26.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:26.887 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:26.922 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:26.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:27.024 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:27.066 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:27.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:27.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:27.167 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:27.201 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:27.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:27.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:27 smithi134 ceph-mon[30086]: pgmap v371: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 502 op/s; 18066/249621 objects misplaced (7.237%); 62 MiB/s, 988 objects/s recovering 2024-02-05T08:13:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:27 smithi134 ceph-mon[39278]: pgmap v371: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 502 op/s; 18066/249621 objects misplaced (7.237%); 62 MiB/s, 988 objects/s recovering 2024-02-05T08:13:27.303 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:27 smithi196 ceph-mon[33018]: pgmap v371: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 5.1 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 502 op/s; 18066/249621 objects misplaced (7.237%); 62 MiB/s, 988 objects/s recovering 2024-02-05T08:13:27.334 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:27.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:27.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:27.436 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:27.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:27.467 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:27.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:27.568 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:27.600 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:27.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:27.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:27.702 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:27.733 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:27.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:27.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:27.775 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:13:27.775 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:13:27.775 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-02-05T08:13:27.775 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-05T08:13:27.775 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-05T08:13:27.776 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-02-05T08:13:27.818 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:13:27.835 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:27.849 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:13:27.866 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:27.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:27.881 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:13:27.911 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:13:27.912 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:13:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:13:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:13:27.939 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:13:27.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:27.969 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:13:27.996 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:13:27.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:27.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:28.024 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:13:28.026 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:13:28.026 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:13:28.026 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:13:28.026 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:13:28.026 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:13:28.026 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:13:28.026 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:13:28.026 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:13:28.097 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:13:28.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:28.136 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:28.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:28.205 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:28.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:28 smithi134 ceph-mon[30086]: osdmap e209: 8 total, 8 up, 7 in 2024-02-05T08:13:28.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:28 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2231224957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:13:28.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:28 smithi134 ceph-mon[39278]: osdmap e209: 8 total, 8 up, 7 in 2024-02-05T08:13:28.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:28 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2231224957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:13:28.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:28.276 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:28.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:28.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:28.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:28 smithi196 ceph-mon[33018]: osdmap e209: 8 total, 8 up, 7 in 2024-02-05T08:13:28.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:28 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2231224957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:13:28.415 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:28.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:28.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:28.555 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:28.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:28.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:28.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:28.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:28.692 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:28.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:28.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:28.839 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:28.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:28.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:28.941 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:28.979 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:28.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:29.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:29.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:29.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:29.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:29.218 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:29.271 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:29.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:29.372 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:29.397 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:29 smithi134 ceph-mon[30086]: pgmap v373: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 712 op/s; 16251/243324 objects misplaced (6.679%); 66 MiB/s, 1.05k objects/s recovering 2024-02-05T08:13:29.398 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:29 smithi134 ceph-mon[30086]: osdmap e210: 8 total, 8 up, 7 in 2024-02-05T08:13:29.398 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:29 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:13:29.398 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:29 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:13:29.398 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:29 smithi134 ceph-mon[39278]: pgmap v373: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 712 op/s; 16251/243324 objects misplaced (6.679%); 66 MiB/s, 1.05k objects/s recovering 2024-02-05T08:13:29.398 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:29 smithi134 ceph-mon[39278]: osdmap e210: 8 total, 8 up, 7 in 2024-02-05T08:13:29.398 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:29 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:13:29.398 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:29 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:13:29.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:29.422 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:29.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:29.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:29.559 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:29.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:29.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:29 smithi196 ceph-mon[33018]: pgmap v373: 148 pgs: 6 active+remapped+backfilling, 142 active+clean; 4.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 712 op/s; 16251/243324 objects misplaced (6.679%); 66 MiB/s, 1.05k objects/s recovering 2024-02-05T08:13:29.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:29 smithi196 ceph-mon[33018]: osdmap e210: 8 total, 8 up, 7 in 2024-02-05T08:13:29.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:29 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:13:29.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:29 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:13:29.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:29.705 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:29.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:29.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:29.847 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:29.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:29.949 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:29.985 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:29.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:30.087 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:30.127 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:30.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:30.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:30.228 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:30.242 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:30 smithi196 ceph-mon[33018]: osdmap e211: 8 total, 8 up, 7 in 2024-02-05T08:13:30.267 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:30.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:30.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:30.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:30.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:30.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:30.447 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:13:30.447 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-05T08:13:30.447 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:13:30.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:30.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:30 smithi134 ceph-mon[30086]: osdmap e211: 8 total, 8 up, 7 in 2024-02-05T08:13:30.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:30 smithi134 ceph-mon[39278]: osdmap e211: 8 total, 8 up, 7 in 2024-02-05T08:13:30.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:30.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:30.640 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:30.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:30.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:30.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:30.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:30.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:30.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:30.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:30.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:30.937 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:30.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:31.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:31.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:31.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:31.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:31.197 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:31.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:31.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:31.299 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:31.347 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:31.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:31.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:31.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:31.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:31.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:31 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:13:31.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:31 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:13:31.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:31 smithi134 ceph-mon[30086]: pgmap v376: 148 pgs: 1 peering, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 736 op/s; 12703/236382 objects misplaced (5.374%); 47 MiB/s, 746 objects/s recovering 2024-02-05T08:13:31.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:31 smithi134 ceph-mon[30086]: osdmap e212: 8 total, 8 up, 7 in 2024-02-05T08:13:31.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:31 smithi134 ceph-mon[30086]: 5.4 scrub starts 2024-02-05T08:13:31.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:31 smithi134 ceph-mon[30086]: 5.4 scrub ok 2024-02-05T08:13:31.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:31 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:13:31.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:31 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:13:31.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:31 smithi134 ceph-mon[39278]: pgmap v376: 148 pgs: 1 peering, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 736 op/s; 12703/236382 objects misplaced (5.374%); 47 MiB/s, 746 objects/s recovering 2024-02-05T08:13:31.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:31 smithi134 ceph-mon[39278]: osdmap e212: 8 total, 8 up, 7 in 2024-02-05T08:13:31.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:31 smithi134 ceph-mon[39278]: 5.4 scrub starts 2024-02-05T08:13:31.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:31 smithi134 ceph-mon[39278]: 5.4 scrub ok 2024-02-05T08:13:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:31 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:13:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:31 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:13:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:31 smithi196 ceph-mon[33018]: pgmap v376: 148 pgs: 1 peering, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 736 op/s; 12703/236382 objects misplaced (5.374%); 47 MiB/s, 746 objects/s recovering 2024-02-05T08:13:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:31 smithi196 ceph-mon[33018]: osdmap e212: 8 total, 8 up, 7 in 2024-02-05T08:13:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:31 smithi196 ceph-mon[33018]: 5.4 scrub starts 2024-02-05T08:13:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:31 smithi196 ceph-mon[33018]: 5.4 scrub ok 2024-02-05T08:13:31.582 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:31.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:31.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:31.718 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:31.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:31.765 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:31.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:31.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:31.899 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:31.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:31.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:32.001 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:32.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:32.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:32.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:32.165 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:32.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:32.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:32.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:32 smithi134 ceph-mon[30086]: 1.0 scrub starts 2024-02-05T08:13:32.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:32 smithi134 ceph-mon[30086]: 1.0 scrub ok 2024-02-05T08:13:32.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:32 smithi134 ceph-mon[39278]: 1.0 scrub starts 2024-02-05T08:13:32.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:32 smithi134 ceph-mon[39278]: 1.0 scrub ok 2024-02-05T08:13:32.294 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:32.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:32.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:32.396 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:32.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:32.429 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:32.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:32.530 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:32.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:32 smithi196 ceph-mon[33018]: 1.0 scrub starts 2024-02-05T08:13:32.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:32 smithi196 ceph-mon[33018]: 1.0 scrub ok 2024-02-05T08:13:32.581 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:32.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:32.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:32.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:32.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:32.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:32.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:32.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:32.859 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:32.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:32.880 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:13:32.960 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:32.984 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:32.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:32.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:33.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:33.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:33.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:33.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:33.218 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:33.267 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:33.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:33.368 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:33.408 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:33.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:33.509 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:33.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:33 smithi134 ceph-mon[30086]: pgmap v378: 148 pgs: 1 peering, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 957 op/s; 12703/236382 objects misplaced (5.374%); 61 MiB/s, 971 objects/s recovering 2024-02-05T08:13:33.515 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:33 smithi134 ceph-mon[39278]: pgmap v378: 148 pgs: 1 peering, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 957 op/s; 12703/236382 objects misplaced (5.374%); 61 MiB/s, 971 objects/s recovering 2024-02-05T08:13:33.534 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:33.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:33 smithi196 ceph-mon[33018]: pgmap v378: 148 pgs: 1 peering, 5 active+remapped+backfilling, 142 active+clean; 4.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 957 op/s; 12703/236382 objects misplaced (5.374%); 61 MiB/s, 971 objects/s recovering 2024-02-05T08:13:33.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:13:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:13:33.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:33.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:33.663 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:33.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:33.764 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:33.797 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:33.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:33.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:33.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:33.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:34.028 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:34.087 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:34.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:34.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:34.189 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:34.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:34.231 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:34.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:34 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:34 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:34.332 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:34.363 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:34.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:34.466 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:34.502 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:34.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:34 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:34.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:34.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:34.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:34.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:34.737 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:34.774 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:34.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:34.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:34.921 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:34.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:35.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:35.058 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:35.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:35.159 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:35.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:35.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:35 smithi134 ceph-mon[30086]: pgmap v379: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.17k op/s; 8873/222285 objects misplaced (3.992%); 65 MiB/s, 1.03k objects/s recovering 2024-02-05T08:13:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:35 smithi134 ceph-mon[30086]: 6.15 scrub starts 2024-02-05T08:13:35.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:35 smithi134 ceph-mon[30086]: 6.15 scrub ok 2024-02-05T08:13:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:35 smithi134 ceph-mon[39278]: pgmap v379: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.17k op/s; 8873/222285 objects misplaced (3.992%); 65 MiB/s, 1.03k objects/s recovering 2024-02-05T08:13:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:35 smithi134 ceph-mon[39278]: 6.15 scrub starts 2024-02-05T08:13:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:35 smithi134 ceph-mon[39278]: 6.15 scrub ok 2024-02-05T08:13:35.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:35.325 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:35.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:35.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:35.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:35.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:35.460 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:35.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:35.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:35 smithi196 ceph-mon[33018]: pgmap v379: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.17k op/s; 8873/222285 objects misplaced (3.992%); 65 MiB/s, 1.03k objects/s recovering 2024-02-05T08:13:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:35 smithi196 ceph-mon[33018]: 6.15 scrub starts 2024-02-05T08:13:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:35 smithi196 ceph-mon[33018]: 6.15 scrub ok 2024-02-05T08:13:35.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:35.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:35.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:35.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:35.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:35.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:35.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:35.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:35.858 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:35.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:35.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:35.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:35.986 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:35.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:35.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:36.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:36.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:36.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:36.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:36.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:36.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:36.357 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:36.395 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:36.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:36.459 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:13:36.460 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:36 smithi134 ceph-mon[30086]: 6.12 scrub starts 2024-02-05T08:13:36.460 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:36 smithi134 ceph-mon[30086]: 6.12 scrub ok 2024-02-05T08:13:36.460 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/123270343' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:13:36.460 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:36 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:13:36.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:36 smithi134 ceph-mon[39278]: 6.12 scrub starts 2024-02-05T08:13:36.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:36 smithi134 ceph-mon[39278]: 6.12 scrub ok 2024-02-05T08:13:36.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/123270343' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:13:36.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:36 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:13:36.497 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:36.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:36.538 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:36.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:36 smithi196 ceph-mon[33018]: 6.12 scrub starts 2024-02-05T08:13:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:36 smithi196 ceph-mon[33018]: 6.12 scrub ok 2024-02-05T08:13:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/123270343' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:13:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:36 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:13:36.638 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:36.662 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:36.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:36.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:36.764 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:36.807 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:36.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:36.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:36.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:36.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:36.933 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:36.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:37.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:37.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:37.077 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:37.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:37.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:37.216 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:37.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:37.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:37.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:37.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:37.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:37.445 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:37.475 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:37.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:37.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:37 smithi134 ceph-mon[30086]: pgmap v380: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 673 B/s rd, 0 B/s wr, 852 op/s; 8873/222285 objects misplaced (3.992%); 41 MiB/s, 652 objects/s recovering 2024-02-05T08:13:37.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:37 smithi134 ceph-mon[30086]: 2.0 deep-scrub starts 2024-02-05T08:13:37.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:37 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:13:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:37 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:13:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:37 smithi134 ceph-mon[30086]: 2.0 deep-scrub ok 2024-02-05T08:13:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:37 smithi134 ceph-mon[30086]: osdmap e213: 8 total, 8 up, 7 in 2024-02-05T08:13:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:37 smithi134 ceph-mon[39278]: pgmap v380: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 673 B/s rd, 0 B/s wr, 852 op/s; 8873/222285 objects misplaced (3.992%); 41 MiB/s, 652 objects/s recovering 2024-02-05T08:13:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:37 smithi134 ceph-mon[39278]: 2.0 deep-scrub starts 2024-02-05T08:13:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:37 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:13:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:37 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:13:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:37 smithi134 ceph-mon[39278]: 2.0 deep-scrub ok 2024-02-05T08:13:37.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:37 smithi134 ceph-mon[39278]: osdmap e213: 8 total, 8 up, 7 in 2024-02-05T08:13:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:37 smithi196 ceph-mon[33018]: pgmap v380: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 673 B/s rd, 0 B/s wr, 852 op/s; 8873/222285 objects misplaced (3.992%); 41 MiB/s, 652 objects/s recovering 2024-02-05T08:13:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:37 smithi196 ceph-mon[33018]: 2.0 deep-scrub starts 2024-02-05T08:13:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:37 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:13:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:37 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:13:37.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:37 smithi196 ceph-mon[33018]: 2.0 deep-scrub ok 2024-02-05T08:13:37.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:37 smithi196 ceph-mon[33018]: osdmap e213: 8 total, 8 up, 7 in 2024-02-05T08:13:37.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:37.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:37.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:37.716 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:37.761 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:37.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:37.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:37.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:37.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:37.999 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:38.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:13:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:13:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:13:38.040 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:38.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:38.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:38.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:38.179 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:38.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:38.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:38.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:38.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:38.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:38.417 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:38.454 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:38.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:38.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:38 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:13:38.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:38 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:13:38.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:38 smithi134 ceph-mon[30086]: osdmap e214: 8 total, 8 up, 7 in 2024-02-05T08:13:38.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:13:38.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:13:38.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:38 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:13:38.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:38 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:13:38.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:38 smithi134 ceph-mon[39278]: osdmap e214: 8 total, 8 up, 7 in 2024-02-05T08:13:38.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:13:38.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:13:38.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:38.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:38 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:13:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:38 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:13:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:38 smithi196 ceph-mon[33018]: osdmap e214: 8 total, 8 up, 7 in 2024-02-05T08:13:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:13:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:13:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:38.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:38.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:38.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:38.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:38.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:38.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:38.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:38.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:38.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:38.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:38.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:39.000 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:39.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:39.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:39.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:39.125 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:39.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:39.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:39.143 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:13:39.143 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:13:39.143 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-02-05T08:13:39.143 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-05T08:13:39.143 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-05T08:13:39.144 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-02-05T08:13:39.226 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:39.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:39.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:39.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:39.408 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:39.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:39.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:39.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:39 smithi134 ceph-mon[30086]: pgmap v383: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 725 op/s; 8192/219939 objects misplaced (3.725%); 31 MiB/s, 495 objects/s recovering 2024-02-05T08:13:39.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:39 smithi134 ceph-mon[39278]: pgmap v383: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 725 op/s; 8192/219939 objects misplaced (3.725%); 31 MiB/s, 495 objects/s recovering 2024-02-05T08:13:39.545 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:39.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:39.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:39.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:39 smithi196 ceph-mon[33018]: pgmap v383: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 725 op/s; 8192/219939 objects misplaced (3.725%); 31 MiB/s, 495 objects/s recovering 2024-02-05T08:13:39.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:39.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:39.697 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:39.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:39.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:39.835 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:39.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:39.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:39.977 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:39.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:40.078 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:40.115 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:40.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:40.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:40.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:40.240 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:40.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:40.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:40.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:40.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:40.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:40.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:40.478 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:40.518 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:40.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:40.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:40.620 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:40.659 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:40.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:40.760 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:40.803 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:40.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:40.905 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:40.947 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:40.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:41.049 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:41.092 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:41.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:41.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:41.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:41.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:41.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:41.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:41.328 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:41.373 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:41.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:41.452 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:41 smithi134 ceph-mon[30086]: pgmap v384: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 5626/209865 objects misplaced (2.681%); 47 MiB/s, 747 objects/s recovering 2024-02-05T08:13:41.452 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:41 smithi134 ceph-mon[39278]: pgmap v384: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 5626/209865 objects misplaced (2.681%); 47 MiB/s, 747 objects/s recovering 2024-02-05T08:13:41.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:41.528 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:41.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:41.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:41 smithi196 ceph-mon[33018]: pgmap v384: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.10k op/s; 5626/209865 objects misplaced (2.681%); 47 MiB/s, 747 objects/s recovering 2024-02-05T08:13:41.590 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:41.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:41.662 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:41.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:41.673 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:13:41.673 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:13:41.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.673 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:167080 8.3 8:cd65b41f:::benchmark_data_smithi134_115836_object50420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-05T08:13:41.673 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:41.669082+0000", 2024-02-05T08:13:41.673 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0031970309999999999, 2024-02-05T08:13:41.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003230723, 2024-02-05T08:13:41.673 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 167080 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.669082+0000", 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.669082+0000", 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.669084+0000", 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1710000000000001e-06 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.669086+0000", 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8009999999999999e-06 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.669094+0000", 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8339999999999999e-06 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.669100+0000", 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6300000000000003e-06 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.670666+0000", 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001566071 2024-02-05T08:13:41.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.670807+0000", 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014096000000000001 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5,6", 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.670952+0000", 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000144893 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.15087.0:167065 8.3 e214/188)", 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:41.669435+0000", 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0028440010000000001, 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0029675729999999998, 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.669435+0000", 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.669435+0000", 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.669436+0000", 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0199999999999998e-07 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.669439+0000", 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8909999999999999e-06 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.669445+0000", 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2319999999999998e-06 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.669450+0000", 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9660000000000004e-06 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.680 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:167082 8.3 8.de20f5c3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:41.671765+0000", 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.000514487, 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00069424299999999999, 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 167082 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671765+0000", 2024-02-05T08:13:41.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671765+0000", 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671766+0000", 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.15e-06 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671772+0000", 2024-02-05T08:13:41.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3689999999999998e-06 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671782+0000", 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0302999999999999e-05 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671789+0000", 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4370000000000001e-06 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:167065 8.3 8:ce106278:::benchmark_data_smithi134_115836_object50405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:41.650284+0000", 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.021995338999999999, 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.022238944, 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 167065 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.650284+0000", 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.650284+0000", 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.650286+0000", 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5910000000000001e-06 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.650292+0000", 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7910000000000001e-06 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.650306+0000", 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4442e-05 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.650315+0000", 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6540000000000005e-06 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.686 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:13:41.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.653666+0000", 2024-02-05T08:13:41.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033516819999999999 2024-02-05T08:13:41.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:13:41.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.653821+0000", 2024-02-05T08:13:41.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015502100000000001 2024-02-05T08:13:41.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.688 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5,6", 2024-02-05T08:13:41.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.653975+0000", 2024-02-05T08:13:41.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000153776 2024-02-05T08:13:41.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.688 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:13:41.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.657122+0000", 2024-02-05T08:13:41.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031470080000000002 2024-02-05T08:13:41.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.688 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:13:41.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.658636+0000", 2024-02-05T08:13:41.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00151428 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.658954+0000", 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031803499999999999 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:167084 8.a 8:5c4fb78e:::benchmark_data_smithi134_115836_object50424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:41.671802+0000", 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.000477066, 2024-02-05T08:13:41.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00081366900000000005, 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 167084 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671802+0000", 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671802+0000", 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671803+0000", 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.84e-07 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671805+0000", 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.103e-06 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.691 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671811+0000", 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8200000000000002e-06 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671816+0000", 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9269999999999996e-06 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671864+0000", 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8485000000000001e-05 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.671966+0000", 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010166300000000001 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,5", 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672064+0000", 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8423000000000003e-05 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:167088 8.7 8:ee4479b8:::benchmark_data_smithi134_115836_object50428:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-05T08:13:41.693 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:41.672421+0000", 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4294967295.9998584, 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00027062199999999998, 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 167088 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672421+0000", 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.694 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672421+0000", 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672422+0000", 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8700000000000004e-07 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672427+0000", 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5929999999999999e-06 2024-02-05T08:13:41.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672438+0000", 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1556000000000001e-05 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672446+0000", 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7549999999999994e-06 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672560+0000", 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114005 2024-02-05T08:13:41.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672668+0000", 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010788700000000001 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:167090 8.3 8.b6726d23 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:41.672465+0000", 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 4294967295.9998145, 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00029209400000000001, 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:13:41.697 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 167090 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672465+0000", 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672465+0000", 2024-02-05T08:13:41.698 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672466+0000", 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9500000000000003e-07 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672469+0000", 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9809999999999999e-06 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672476+0000", 2024-02-05T08:13:41.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3679999999999996e-06 2024-02-05T08:13:41.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:13:41.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:13:41.700 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:13:41.700 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:41.672482+0000", 2024-02-05T08:13:41.700 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3219999999999998e-06 2024-02-05T08:13:41.700 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.700 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:13:41.700 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.700 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:13:41.700 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:13:41.700 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 7 2024-02-05T08:13:41.700 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:13:41.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:41.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:41.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:41.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:41.935 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:41.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:42.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:42.075 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:42.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:42.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:42.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:42.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:42.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:42.313 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:42.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:42.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:42.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:42.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:42.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:42.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:42.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:42.627 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:42.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:42.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:42.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:42.762 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:42.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:42.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:42.863 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:42.905 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:42.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:43.007 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:43.046 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:43.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:43.148 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:43.187 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:43.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:43.288 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:43.329 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:43.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:43.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:43.431 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:43.469 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:43.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:43.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:43 smithi134 ceph-mon[30086]: pgmap v385: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 517 op/s; 5626/209865 objects misplaced (2.681%); 22 MiB/s, 350 objects/s recovering 2024-02-05T08:13:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:43 smithi134 ceph-mon[39278]: pgmap v385: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 517 op/s; 5626/209865 objects misplaced (2.681%); 22 MiB/s, 350 objects/s recovering 2024-02-05T08:13:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:43 smithi196 ceph-mon[33018]: pgmap v385: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 517 op/s; 5626/209865 objects misplaced (2.681%); 22 MiB/s, 350 objects/s recovering 2024-02-05T08:13:43.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:13:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:13:43.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:43.595 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:43.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:43.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:43.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:43.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:43.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:43.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:43.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:43.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:43.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:44.017 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:44.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:44.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:44.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:44.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:44.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:44.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:44.267 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:44 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:44.294 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:44.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:44.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:44.314 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:13:44.328 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T08:13:44.396 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:44.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:44.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:44.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:44.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:44 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:44.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:44 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:44.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:44.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:44.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:44.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:44.737 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:44.770 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:44.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:44.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:44.872 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:44.908 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:44.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:44.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:45.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:45.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:45.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:45.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:45.136 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:45.164 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:45.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:45.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:45.266 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:45.293 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:45.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:45.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:45.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:45.421 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:45.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:45.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:45.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:45.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:45 smithi134 ceph-mon[30086]: pgmap v386: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 2526/195105 objects misplaced (1.295%); 44 MiB/s, 703 objects/s recovering 2024-02-05T08:13:45.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:45 smithi134 ceph-mon[30086]: osdmap e215: 8 total, 8 up, 7 in 2024-02-05T08:13:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:45 smithi134 ceph-mon[39278]: pgmap v386: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 2526/195105 objects misplaced (1.295%); 44 MiB/s, 703 objects/s recovering 2024-02-05T08:13:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:45 smithi134 ceph-mon[39278]: osdmap e215: 8 total, 8 up, 7 in 2024-02-05T08:13:45.551 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:45.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:45.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:45 smithi196 ceph-mon[33018]: pgmap v386: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 2526/195105 objects misplaced (1.295%); 44 MiB/s, 703 objects/s recovering 2024-02-05T08:13:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:45 smithi196 ceph-mon[33018]: osdmap e215: 8 total, 8 up, 7 in 2024-02-05T08:13:45.652 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:45.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:45.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:45.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:45.765 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:45.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:45.824 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:45.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:45.834 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:45.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:45.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:45.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:45.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:46.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:46.076 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:46.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:46.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:46.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:46.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:46.249 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:46.303 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:46.327 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:46.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:46.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:46.452 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:46.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:46.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:46.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:46.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:46.578 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:46.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:46.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:46.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:46.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:46.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:46.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:46.831 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:46.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:46.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:46.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:46.953 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:46 smithi134 ceph-mon[30086]: pgmap v388: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 2526/195105 objects misplaced (1.295%); 41 MiB/s, 661 objects/s recovering 2024-02-05T08:13:46.953 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:46 smithi134 ceph-mon[30086]: osdmap e216: 8 total, 8 up, 7 in 2024-02-05T08:13:46.953 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:46 smithi134 ceph-mon[39278]: pgmap v388: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 2526/195105 objects misplaced (1.295%); 41 MiB/s, 661 objects/s recovering 2024-02-05T08:13:46.953 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:46 smithi134 ceph-mon[39278]: osdmap e216: 8 total, 8 up, 7 in 2024-02-05T08:13:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:46.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:46.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:47.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:46 smithi196 ceph-mon[33018]: pgmap v388: 148 pgs: 5 active+remapped+backfilling, 143 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 2526/195105 objects misplaced (1.295%); 41 MiB/s, 661 objects/s recovering 2024-02-05T08:13:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:46 smithi196 ceph-mon[33018]: osdmap e216: 8 total, 8 up, 7 in 2024-02-05T08:13:47.080 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:47.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:47.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:47.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:47.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:47.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:47.308 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:47.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:47.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:47.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:47.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:47.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:47.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:47.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:47.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:47.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:47.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:47.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:47.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:47.710 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:47.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:47.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:47.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:47 smithi134 ceph-mon[30086]: osdmap e217: 8 total, 8 up, 7 in 2024-02-05T08:13:47.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:47 smithi134 ceph-mon[39278]: osdmap e217: 8 total, 8 up, 7 in 2024-02-05T08:13:47.812 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:47.863 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:47.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:47.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:47.921 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:13:47.962 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:13:47.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:47.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:48.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:48.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:48.001 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:13:48.002 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:13:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:13:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:13:48.043 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:13:48.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:47 smithi196 ceph-mon[33018]: osdmap e217: 8 total, 8 up, 7 in 2024-02-05T08:13:48.081 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:13:48.101 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:48.122 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:13:48.141 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:48.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:48.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:48.160 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:13:48.199 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:13:48.208 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:13:48.208 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:13:48.208 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:13:48.208 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:13:48.208 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:13:48.208 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:13:48.208 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:13:48.208 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:13:48.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:48.285 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:48.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:48.387 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:48.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:48.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:48.521 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:48.556 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:48.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:48.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:48.657 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:48.680 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:48.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:48.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:48.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:48.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:48.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:48.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:48.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:48.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:49.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:48 smithi134 ceph-mon[30086]: pgmap v391: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 919 op/s; 1907/193344 objects misplaced (0.986%); 36 MiB/s, 571 objects/s recovering 2024-02-05T08:13:49.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:48 smithi134 ceph-mon[30086]: osdmap e218: 8 total, 8 up, 7 in 2024-02-05T08:13:49.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:48 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2144131827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:13:49.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:48 smithi134 ceph-mon[39278]: pgmap v391: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 919 op/s; 1907/193344 objects misplaced (0.986%); 36 MiB/s, 571 objects/s recovering 2024-02-05T08:13:49.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:48 smithi134 ceph-mon[39278]: osdmap e218: 8 total, 8 up, 7 in 2024-02-05T08:13:49.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:48 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2144131827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:13:49.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:48 smithi196 ceph-mon[33018]: pgmap v391: 148 pgs: 3 active+remapped+backfilling, 145 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 919 op/s; 1907/193344 objects misplaced (0.986%); 36 MiB/s, 571 objects/s recovering 2024-02-05T08:13:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:48 smithi196 ceph-mon[33018]: osdmap e218: 8 total, 8 up, 7 in 2024-02-05T08:13:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:48 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2144131827' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:13:49.092 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:49.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:49.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:49.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:49.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:49.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:49.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:49.370 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:49.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:49.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:49.510 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:49.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:49.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:49.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:49.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:49.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:49.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:49.789 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:49.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:49.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:49.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:49.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:49.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:50.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:50.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:50.108 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:50.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:50.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:50.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:50.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:50.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:50.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:50.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:50.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:50.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:50.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:50.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:50.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:50.682 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:50.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:50.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:50.784 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:50.817 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:50.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:50.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:50.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:50.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:50.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:51.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:50 smithi134 ceph-mon[30086]: pgmap v393: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 492 op/s; 31 MiB/s, 495 objects/s recovering 2024-02-05T08:13:51.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:50 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:13:51.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:50 smithi134 ceph-mon[39278]: pgmap v393: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 492 op/s; 31 MiB/s, 495 objects/s recovering 2024-02-05T08:13:51.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:50 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:13:51.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:50 smithi196 ceph-mon[33018]: pgmap v393: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 492 op/s; 31 MiB/s, 495 objects/s recovering 2024-02-05T08:13:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:50 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:13:51.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:51.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:51.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:51.225 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:51.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:51.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:51.326 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:51.363 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:51.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:51.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:51.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:51.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:51.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:51.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:51.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:51.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:51.736 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:51.773 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:51.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:51.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:51.906 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:51.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:51.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:52.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:52.047 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:52.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:52.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:52.178 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:52.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:52.279 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:52.309 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:52.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:52.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:52.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:52.440 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:13:52.440 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:13:52.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:52.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:52.450 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:13:52.450 INFO:teuthology.orchestra.run.smithi134.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-05T08:13:52.450 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:13:52.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:52.583 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:52.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:52.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:52.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:52.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:52.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:52.742 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:13:52.825 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:52.865 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:52.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:52.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:52.967 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:53.003 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:53.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:53.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:52 smithi134 ceph-mon[30086]: pgmap v394: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 409 op/s; 26 MiB/s, 411 objects/s recovering 2024-02-05T08:13:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:52 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1272648426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:13:53.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:52 smithi134 ceph-mon[39278]: pgmap v394: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 409 op/s; 26 MiB/s, 411 objects/s recovering 2024-02-05T08:13:53.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:52 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1272648426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:13:53.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:52 smithi196 ceph-mon[33018]: pgmap v394: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 409 op/s; 26 MiB/s, 411 objects/s recovering 2024-02-05T08:13:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:52 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1272648426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:13:53.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:53.140 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:53.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:53.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:53.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T08:13:53.242 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:53.311 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:53.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:53.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:53.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:53.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:53.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:53.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:53.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:53.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:53.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:53.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:53.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:13:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:13:53.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:53.858 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:53.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:53.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:53.960 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:53.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:53.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:53.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:13:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1272648426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:13:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[30086]: osdmap e219: 8 total, 8 up, 7 in 2024-02-05T08:13:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:13:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:13:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:13:54.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:13:54.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:13:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1272648426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:13:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[39278]: osdmap e219: 8 total, 8 up, 7 in 2024-02-05T08:13:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:13:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:13:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:13:54.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:13:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:53 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:13:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:53 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1272648426' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:13:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:53 smithi196 ceph-mon[33018]: osdmap e219: 8 total, 8 up, 7 in 2024-02-05T08:13:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:13:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:53 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:13:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:13:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:13:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:13:54.099 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:54.138 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:54.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:54.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:54.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:54.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:54.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:54.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:54.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:54.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:54.521 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:54.559 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:54.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:54.659 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:54.701 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:54.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:54.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:54.802 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:54.840 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:54.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:54.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:54.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:54.989 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:54.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:54.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:55.008 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:55.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:55.127 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:55.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:55.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:55.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:55.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:55.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:55.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:55.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:55.408 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:55.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:55.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:55.438 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:55 smithi134 ceph-mon[30086]: pgmap v396: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.32k op/s; 22 MiB/s, 353 objects/s recovering 2024-02-05T08:13:55.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:55.537 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:55.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:55.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:55.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:55.679 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:55.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:55.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:55.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:55 smithi134 ceph-mon[39278]: pgmap v396: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.32k op/s; 22 MiB/s, 353 objects/s recovering 2024-02-05T08:13:55.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:55.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:55 smithi196 ceph-mon[33018]: pgmap v396: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.32k op/s; 22 MiB/s, 353 objects/s recovering 2024-02-05T08:13:55.820 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:55.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:55.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:55.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:55.952 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:55.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:55.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:56.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:56.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:56.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:56.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:56.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:13:56.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:56.237 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:56.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:56.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:56.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:56.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:56.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:56.482 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:56.519 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:56.538 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:56.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:56.559 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:13:56.639 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:56.681 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:56.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:56.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:56.838 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:56.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:56.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:56.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:56.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:56.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:56.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:57.075 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:57.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:57.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:57.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:57.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:57.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:57.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:57.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:13:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:57.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:57.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:13:57.488 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:57.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:57.524 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:57.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:57.625 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:57.659 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:57.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:57.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:57.693 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:57 smithi134 ceph-mon[30086]: pgmap v397: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 15 MiB/s, 233 objects/s recovering 2024-02-05T08:13:57.694 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:57 smithi134 ceph-mon[39278]: pgmap v397: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 15 MiB/s, 233 objects/s recovering 2024-02-05T08:13:57.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:57.798 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:57.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:57.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:57 smithi196 ceph-mon[33018]: pgmap v397: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.08k op/s; 15 MiB/s, 233 objects/s recovering 2024-02-05T08:13:57.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:13:57.938 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:57.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:57.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:13:57.977 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:13:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:13:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:13:58.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:58.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:58.088 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:58.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:58.127 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:13:58.128 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:13:58.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:58.229 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:58.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:58.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:58.373 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:58.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:58.474 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:58.513 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:58.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:58.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:58.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:13:58.654 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:58.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:58.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:13:58.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:58.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:58.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:58.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:58.822 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:13:58.839 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T08:13:58.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:58.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:58.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:58.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:59.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:13:59.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:59.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:13:59.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:59.204 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:59.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:59.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:59.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:13:59.329 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:59.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:13:59.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:13:59.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:59.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:59.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:13:59.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:59.610 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:59.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:59.699 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:13:59 smithi134 ceph-mon[30086]: pgmap v398: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.03k op/s; 12 MiB/s, 186 objects/s recovering 2024-02-05T08:13:59.699 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:13:59 smithi134 ceph-mon[39278]: pgmap v398: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.03k op/s; 12 MiB/s, 186 objects/s recovering 2024-02-05T08:13:59.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:13:59.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:59.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:13:59.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:13:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:13:59 smithi196 ceph-mon[33018]: pgmap v398: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.03k op/s; 12 MiB/s, 186 objects/s recovering 2024-02-05T08:13:59.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:13:59.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:13:59.887 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:13:59.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:13:59.973 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:13:59.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 0 1 2024-02-05T08:13:59.988 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:00.077 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:00.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:00.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:00.180 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:00.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:00.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:00.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:00.303 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:14:00.303 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:14:00.303 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:14:00.303 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:14:00.304 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:14:00.304 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:14:00.304 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:14:00.304 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:14:00.304 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:14:00.305 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:14:00.305 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:14:00.305 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:14:00.305 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:14:00.305 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:14:00.305 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:14:00.306 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:14:00.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:00.356 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:00.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:00.458 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:00.496 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:00.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:00.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:00.633 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:00.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:00.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:00.735 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:00.773 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:00.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:00.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:00.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:00.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:00.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:01.016 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:01.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:00 smithi134 ceph-mon[30086]: pgmap v399: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-02-05T08:14:01.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:00 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1260912072' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:00 smithi134 ceph-mon[39278]: pgmap v399: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-02-05T08:14:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:00 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1260912072' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:01.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:01.054 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:01.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:01.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:00 smithi196 ceph-mon[33018]: pgmap v399: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-02-05T08:14:01.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:00 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1260912072' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:01.154 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:01.200 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:01.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:01.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:01.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:01.339 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:01.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:01.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:01.441 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:01.478 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:01.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:01.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:01.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:01.613 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:01.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:01.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:01.715 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:01.745 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:01.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:01.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:01.759 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:01.807 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:01.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:01.880 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:01.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:01.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:02.020 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:02.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:02.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:02.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:02.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:02.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:02.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:02.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:02.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:02.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:02.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:02.437 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:02.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:02.538 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:02.574 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:02.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:02.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:02.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:02.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:02.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:02.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:02 smithi134 ceph-mon[30086]: pgmap v400: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-02-05T08:14:02.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:02 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:14:02.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:02 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:14:02.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:02 smithi134 ceph-mon[39278]: pgmap v400: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-02-05T08:14:02.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:02 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:14:02.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:02 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:14:02.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:02.852 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:02.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:02.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:02.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:02.990 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:02.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:02.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:03.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:02 smithi196 ceph-mon[33018]: pgmap v400: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-02-05T08:14:03.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:02 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:14:03.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:02 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:14:03.092 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:03.135 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:03.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:03.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:03.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:14:03.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:03.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:03.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:03.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:03.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:03.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:03.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:03.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:03.546 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:03.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:03.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:03.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:03.687 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:03.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:03.720 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:14:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:14:03.737 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:14:03.788 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:03.825 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:03.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:03.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:03.927 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:03.968 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:03.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:03.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:04.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:03 smithi134 ceph-mon[30086]: osdmap e220: 8 total, 8 up, 7 in 2024-02-05T08:14:04.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:03 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2939703256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:14:04.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:03 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:04.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:03 smithi134 ceph-mon[39278]: osdmap e220: 8 total, 8 up, 7 in 2024-02-05T08:14:04.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:03 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2939703256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:14:04.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:03 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:04.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:03 smithi196 ceph-mon[33018]: osdmap e220: 8 total, 8 up, 7 in 2024-02-05T08:14:04.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:03 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2939703256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:14:04.067 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:03 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:04.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:04.104 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:04.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:04.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:04.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:04.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:04.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:04.347 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:04.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:04.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:04.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:04.485 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:04.526 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:04.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:04.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:04.661 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:04.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:04.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:04.763 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:04.803 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:04.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:04.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:04.904 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:04.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:04.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:04.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:05.002 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:04 smithi134 ceph-mon[30086]: pgmap v402: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-02-05T08:14:05.002 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:04 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:14:05.002 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:04 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2939703256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:14:05.002 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:04 smithi134 ceph-mon[30086]: osdmap e221: 8 total, 8 up, 7 in 2024-02-05T08:14:05.002 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:04 smithi134 ceph-mon[30086]: 4.18 scrub starts 2024-02-05T08:14:05.002 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:04 smithi134 ceph-mon[30086]: 4.18 scrub ok 2024-02-05T08:14:05.003 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:04 smithi134 ceph-mon[39278]: pgmap v402: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-02-05T08:14:05.003 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:04 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:14:05.003 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:04 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2939703256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:14:05.003 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:04 smithi134 ceph-mon[39278]: osdmap e221: 8 total, 8 up, 7 in 2024-02-05T08:14:05.003 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:04 smithi134 ceph-mon[39278]: 4.18 scrub starts 2024-02-05T08:14:05.003 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:04 smithi134 ceph-mon[39278]: 4.18 scrub ok 2024-02-05T08:14:05.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:05.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:04 smithi196 ceph-mon[33018]: pgmap v402: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-02-05T08:14:05.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:04 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:14:05.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:04 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2939703256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:14:05.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:04 smithi196 ceph-mon[33018]: osdmap e221: 8 total, 8 up, 7 in 2024-02-05T08:14:05.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:04 smithi196 ceph-mon[33018]: 4.18 scrub starts 2024-02-05T08:14:05.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:04 smithi196 ceph-mon[33018]: 4.18 scrub ok 2024-02-05T08:14:05.082 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:05.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:05.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:05.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:05.234 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:05.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:05.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:05.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:05.380 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:05.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:05.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:05.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:05.518 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:05.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:05.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:05.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:05.658 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:05.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:05.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:05.798 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:05.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:05.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:05.899 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:05.936 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:05.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:05.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:06.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:05 smithi134 ceph-mon[30086]: 8.f scrub starts 2024-02-05T08:14:06.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:05 smithi134 ceph-mon[30086]: 3.3 scrub starts 2024-02-05T08:14:06.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:05 smithi134 ceph-mon[30086]: 3.3 scrub ok 2024-02-05T08:14:06.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:05 smithi134 ceph-mon[39278]: 8.f scrub starts 2024-02-05T08:14:06.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:05 smithi134 ceph-mon[39278]: 3.3 scrub starts 2024-02-05T08:14:06.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:05 smithi134 ceph-mon[39278]: 3.3 scrub ok 2024-02-05T08:14:06.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:06.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:05 smithi196 ceph-mon[33018]: 8.f scrub starts 2024-02-05T08:14:06.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:05 smithi196 ceph-mon[33018]: 3.3 scrub starts 2024-02-05T08:14:06.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:05 smithi196 ceph-mon[33018]: 3.3 scrub ok 2024-02-05T08:14:06.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:06.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:06.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:06.214 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:06.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:06.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:06.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:06.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:06.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:06.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:06.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:06.475 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:06.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:06.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:06.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:06.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:06.718 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:06.747 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:06.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:06.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:06.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:06.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:06.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:06.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:07.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:07.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:07.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:07.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[30086]: pgmap v404: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.50k op/s 2024-02-05T08:14:07.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[30086]: 5.17 scrub starts 2024-02-05T08:14:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[30086]: 5.17 scrub ok 2024-02-05T08:14:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:14:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:14:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[30086]: 8.f scrub ok 2024-02-05T08:14:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[30086]: 5.9 scrub starts 2024-02-05T08:14:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[30086]: 5.9 scrub ok 2024-02-05T08:14:07.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[39278]: pgmap v404: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.50k op/s 2024-02-05T08:14:07.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[39278]: 5.17 scrub starts 2024-02-05T08:14:07.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[39278]: 5.17 scrub ok 2024-02-05T08:14:07.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:14:07.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:14:07.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[39278]: 8.f scrub ok 2024-02-05T08:14:07.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[39278]: 5.9 scrub starts 2024-02-05T08:14:07.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:06 smithi134 ceph-mon[39278]: 5.9 scrub ok 2024-02-05T08:14:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:06 smithi196 ceph-mon[33018]: pgmap v404: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.50k op/s 2024-02-05T08:14:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:06 smithi196 ceph-mon[33018]: 5.17 scrub starts 2024-02-05T08:14:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:06 smithi196 ceph-mon[33018]: 5.17 scrub ok 2024-02-05T08:14:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:06 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:14:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:06 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:14:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:06 smithi196 ceph-mon[33018]: 8.f scrub ok 2024-02-05T08:14:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:06 smithi196 ceph-mon[33018]: 5.9 scrub starts 2024-02-05T08:14:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:06 smithi196 ceph-mon[33018]: 5.9 scrub ok 2024-02-05T08:14:07.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:07.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:07.164 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:07.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:07.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:07.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:07.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:07.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:07.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:07.444 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:07.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:07.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:07.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:07.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:07.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:07.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:07.718 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:07.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:07.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:07.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:07.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:07.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:07.946 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:07.985 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:07.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:08.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:07 smithi134 ceph-mon[30086]: osdmap e222: 8 total, 8 up, 7 in 2024-02-05T08:14:08.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:07 smithi134 ceph-mon[30086]: 8.b scrub starts 2024-02-05T08:14:08.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:07 smithi134 ceph-mon[30086]: 6.d scrub starts 2024-02-05T08:14:08.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:07 smithi134 ceph-mon[30086]: 8.b scrub ok 2024-02-05T08:14:08.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:14:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:14:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:14:08.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:07 smithi134 ceph-mon[39278]: osdmap e222: 8 total, 8 up, 7 in 2024-02-05T08:14:08.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:07 smithi134 ceph-mon[39278]: 8.b scrub starts 2024-02-05T08:14:08.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:07 smithi134 ceph-mon[39278]: 6.d scrub starts 2024-02-05T08:14:08.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:07 smithi134 ceph-mon[39278]: 8.b scrub ok 2024-02-05T08:14:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:07 smithi196 ceph-mon[33018]: osdmap e222: 8 total, 8 up, 7 in 2024-02-05T08:14:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:07 smithi196 ceph-mon[33018]: 8.b scrub starts 2024-02-05T08:14:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:07 smithi196 ceph-mon[33018]: 6.d scrub starts 2024-02-05T08:14:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:07 smithi196 ceph-mon[33018]: 8.b scrub ok 2024-02-05T08:14:08.087 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:08.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:08.129 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:08.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:08.230 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:08.263 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:08.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:08.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:08.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:08.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:08.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:08.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:08.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:08.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:08.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:08.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:08.749 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.0 to 1 (10000) 2024-02-05T08:14:08.769 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:08.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:08.814 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:08.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:08.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:08.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:08.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:08.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:09.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:08 smithi134 ceph-mon[30086]: 6.d scrub ok 2024-02-05T08:14:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:08 smithi134 ceph-mon[30086]: pgmap v406: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-02-05T08:14:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:08 smithi134 ceph-mon[30086]: 8.1 scrub starts 2024-02-05T08:14:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:14:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:08 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/72230139' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:14:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:08 smithi134 ceph-mon[30086]: 8.1 scrub ok 2024-02-05T08:14:09.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:08 smithi134 ceph-mon[39278]: 6.d scrub ok 2024-02-05T08:14:09.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:08 smithi134 ceph-mon[39278]: pgmap v406: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-02-05T08:14:09.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:08 smithi134 ceph-mon[39278]: 8.1 scrub starts 2024-02-05T08:14:09.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:14:09.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:08 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/72230139' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:14:09.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:08 smithi134 ceph-mon[39278]: 8.1 scrub ok 2024-02-05T08:14:09.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:08 smithi196 ceph-mon[33018]: 6.d scrub ok 2024-02-05T08:14:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:08 smithi196 ceph-mon[33018]: pgmap v406: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-02-05T08:14:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:08 smithi196 ceph-mon[33018]: 8.1 scrub starts 2024-02-05T08:14:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:14:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:08 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/72230139' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:14:09.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:08 smithi196 ceph-mon[33018]: 8.1 scrub ok 2024-02-05T08:14:09.089 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:09.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:09.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:09.191 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:09.232 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:09.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:09.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:09.236 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:14:09.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:09.377 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:09.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:09.479 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:09.525 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:09.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:09.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:09.666 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:09.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:09.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:09.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:09.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:09.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:09.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:09.943 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:09.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:10.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:09 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/72230139' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:14:10.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:09 smithi134 ceph-mon[30086]: osdmap e223: 8 total, 8 up, 7 in 2024-02-05T08:14:10.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:09 smithi134 ceph-mon[30086]: 5.13 scrub starts 2024-02-05T08:14:10.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:09 smithi134 ceph-mon[30086]: 5.13 scrub ok 2024-02-05T08:14:10.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:09 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/72230139' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:14:10.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:09 smithi134 ceph-mon[39278]: osdmap e223: 8 total, 8 up, 7 in 2024-02-05T08:14:10.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:09 smithi134 ceph-mon[39278]: 5.13 scrub starts 2024-02-05T08:14:10.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:09 smithi134 ceph-mon[39278]: 5.13 scrub ok 2024-02-05T08:14:10.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:10.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:09 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/72230139' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:14:10.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:09 smithi196 ceph-mon[33018]: osdmap e223: 8 total, 8 up, 7 in 2024-02-05T08:14:10.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:09 smithi196 ceph-mon[33018]: 5.13 scrub starts 2024-02-05T08:14:10.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:09 smithi196 ceph-mon[33018]: 5.13 scrub ok 2024-02-05T08:14:10.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:10.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:10.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:10.169 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:10.193 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:10.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:10.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:10.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:10.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:10.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:10.432 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:10.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:10.469 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:10.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:10.569 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:10.610 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:10.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:10.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:10.712 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:10.756 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:10.757 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:10.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:10.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:10.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[30086]: 3.1e scrub starts 2024-02-05T08:14:10.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[30086]: 3.1e scrub ok 2024-02-05T08:14:10.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[30086]: pgmap v408: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T08:14:10.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[30086]: 4.1d scrub starts 2024-02-05T08:14:10.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[30086]: 4.1d scrub ok 2024-02-05T08:14:10.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[30086]: 4.e scrub starts 2024-02-05T08:14:10.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[30086]: 4.e scrub ok 2024-02-05T08:14:10.770 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[30086]: 3.1f scrub starts 2024-02-05T08:14:10.771 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[30086]: 3.1f scrub ok 2024-02-05T08:14:10.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[39278]: 3.1e scrub starts 2024-02-05T08:14:10.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[39278]: 3.1e scrub ok 2024-02-05T08:14:10.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[39278]: pgmap v408: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T08:14:10.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[39278]: 4.1d scrub starts 2024-02-05T08:14:10.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[39278]: 4.1d scrub ok 2024-02-05T08:14:10.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[39278]: 4.e scrub starts 2024-02-05T08:14:10.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[39278]: 4.e scrub ok 2024-02-05T08:14:10.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[39278]: 3.1f scrub starts 2024-02-05T08:14:10.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:10 smithi134 ceph-mon[39278]: 3.1f scrub ok 2024-02-05T08:14:10.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:10.902 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:10.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:11.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:11.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:11.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:10 smithi196 ceph-mon[33018]: 3.1e scrub starts 2024-02-05T08:14:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:10 smithi196 ceph-mon[33018]: 3.1e scrub ok 2024-02-05T08:14:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:10 smithi196 ceph-mon[33018]: pgmap v408: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T08:14:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:10 smithi196 ceph-mon[33018]: 4.1d scrub starts 2024-02-05T08:14:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:10 smithi196 ceph-mon[33018]: 4.1d scrub ok 2024-02-05T08:14:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:10 smithi196 ceph-mon[33018]: 4.e scrub starts 2024-02-05T08:14:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:10 smithi196 ceph-mon[33018]: 4.e scrub ok 2024-02-05T08:14:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:10 smithi196 ceph-mon[33018]: 3.1f scrub starts 2024-02-05T08:14:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:10 smithi196 ceph-mon[33018]: 3.1f scrub ok 2024-02-05T08:14:11.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:11.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:11.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:11.276 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:11.312 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:11.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:11.414 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:11.449 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:11.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:11.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:11.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:11.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:11.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:11.608 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 1 1 2024-02-05T08:14:11.685 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:11.740 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:11.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:11.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:11.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:11.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:11.900 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:14:11.900 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:14:11.900 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-02-05T08:14:11.900 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:14:11.901 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[30086]: 3.a scrub starts 2024-02-05T08:14:11.901 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[30086]: 3.a scrub ok 2024-02-05T08:14:11.901 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[30086]: 3.13 scrub starts 2024-02-05T08:14:11.901 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[30086]: 3.13 scrub ok 2024-02-05T08:14:11.901 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[30086]: 3.19 scrub starts 2024-02-05T08:14:11.901 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[30086]: 3.19 scrub ok 2024-02-05T08:14:11.901 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[30086]: 4.19 scrub starts 2024-02-05T08:14:11.901 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[30086]: 4.19 scrub ok 2024-02-05T08:14:11.901 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[30086]: 8.e scrub starts 2024-02-05T08:14:11.902 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[39278]: 3.a scrub starts 2024-02-05T08:14:11.902 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[39278]: 3.a scrub ok 2024-02-05T08:14:11.902 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[39278]: 3.13 scrub starts 2024-02-05T08:14:11.902 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[39278]: 3.13 scrub ok 2024-02-05T08:14:11.902 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[39278]: 3.19 scrub starts 2024-02-05T08:14:11.902 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[39278]: 3.19 scrub ok 2024-02-05T08:14:11.902 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[39278]: 4.19 scrub starts 2024-02-05T08:14:11.902 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[39278]: 4.19 scrub ok 2024-02-05T08:14:11.902 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:11 smithi134 ceph-mon[39278]: 8.e scrub starts 2024-02-05T08:14:11.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:12.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:12.014 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:12.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:12.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:11 smithi196 ceph-mon[33018]: 3.a scrub starts 2024-02-05T08:14:12.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:11 smithi196 ceph-mon[33018]: 3.a scrub ok 2024-02-05T08:14:12.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:11 smithi196 ceph-mon[33018]: 3.13 scrub starts 2024-02-05T08:14:12.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:11 smithi196 ceph-mon[33018]: 3.13 scrub ok 2024-02-05T08:14:12.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:11 smithi196 ceph-mon[33018]: 3.19 scrub starts 2024-02-05T08:14:12.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:11 smithi196 ceph-mon[33018]: 3.19 scrub ok 2024-02-05T08:14:12.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:11 smithi196 ceph-mon[33018]: 4.19 scrub starts 2024-02-05T08:14:12.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:11 smithi196 ceph-mon[33018]: 4.19 scrub ok 2024-02-05T08:14:12.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:11 smithi196 ceph-mon[33018]: 8.e scrub starts 2024-02-05T08:14:12.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:12.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:12.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:12.249 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:12.286 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:12.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:12.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:12.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:12.425 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:12.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:12.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:12.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:12.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:12.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:12.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:12.699 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:12.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:12.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:12.824 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:12.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:12.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:12.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:12.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:12.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:13.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[30086]: 8.9 scrub starts 2024-02-05T08:14:13.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[30086]: pgmap v409: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 643 B/s rd, 0 B/s wr, 790 op/s 2024-02-05T08:14:13.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[30086]: 8.e scrub ok 2024-02-05T08:14:13.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:14:13.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:14:13.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[30086]: 8.9 scrub ok 2024-02-05T08:14:13.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[30086]: 8.2 scrub starts 2024-02-05T08:14:13.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[30086]: 8.2 scrub ok 2024-02-05T08:14:13.058 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[39278]: 8.9 scrub starts 2024-02-05T08:14:13.058 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[39278]: pgmap v409: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 643 B/s rd, 0 B/s wr, 790 op/s 2024-02-05T08:14:13.058 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[39278]: 8.e scrub ok 2024-02-05T08:14:13.058 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:14:13.058 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:14:13.058 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[39278]: 8.9 scrub ok 2024-02-05T08:14:13.058 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[39278]: 8.2 scrub starts 2024-02-05T08:14:13.058 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:12 smithi134 ceph-mon[39278]: 8.2 scrub ok 2024-02-05T08:14:13.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:13.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:12 smithi196 ceph-mon[33018]: 8.9 scrub starts 2024-02-05T08:14:13.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:12 smithi196 ceph-mon[33018]: pgmap v409: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 2.2 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 643 B/s rd, 0 B/s wr, 790 op/s 2024-02-05T08:14:13.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:12 smithi196 ceph-mon[33018]: 8.e scrub ok 2024-02-05T08:14:13.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:12 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:14:13.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:12 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:14:13.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:12 smithi196 ceph-mon[33018]: 8.9 scrub ok 2024-02-05T08:14:13.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:12 smithi196 ceph-mon[33018]: 8.2 scrub starts 2024-02-05T08:14:13.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:12 smithi196 ceph-mon[33018]: 8.2 scrub ok 2024-02-05T08:14:13.119 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:13.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:13.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:13.254 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:13.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:13.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:13.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:13.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:13.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:13.494 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:13.527 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:13.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:13.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:13.629 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:13.663 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:13.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:13.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:13.764 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:13.806 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:13.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:13.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:13.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:14:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:14:13.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:13.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:13.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:13.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[30086]: 3.d scrub starts 2024-02-05T08:14:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[30086]: 3.d scrub ok 2024-02-05T08:14:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[30086]: 8.3 scrub starts 2024-02-05T08:14:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:14:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:14:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[30086]: osdmap e224: 8 total, 8 up, 7 in 2024-02-05T08:14:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[30086]: 8.3 scrub ok 2024-02-05T08:14:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[30086]: 8.a scrub starts 2024-02-05T08:14:14.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[39278]: 3.d scrub starts 2024-02-05T08:14:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[39278]: 3.d scrub ok 2024-02-05T08:14:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[39278]: 8.3 scrub starts 2024-02-05T08:14:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:14:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:14:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[39278]: osdmap e224: 8 total, 8 up, 7 in 2024-02-05T08:14:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[39278]: 8.3 scrub ok 2024-02-05T08:14:14.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:13 smithi134 ceph-mon[39278]: 8.a scrub starts 2024-02-05T08:14:14.044 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:14.081 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:14.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:14.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:14.099 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:14.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:14.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:14.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:14.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:13 smithi196 ceph-mon[33018]: 3.d scrub starts 2024-02-05T08:14:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:13 smithi196 ceph-mon[33018]: 3.d scrub ok 2024-02-05T08:14:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:13 smithi196 ceph-mon[33018]: 8.3 scrub starts 2024-02-05T08:14:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:13 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:14:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:13 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:14:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:13 smithi196 ceph-mon[33018]: osdmap e224: 8 total, 8 up, 7 in 2024-02-05T08:14:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:13 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:13 smithi196 ceph-mon[33018]: 8.3 scrub ok 2024-02-05T08:14:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:13 smithi196 ceph-mon[33018]: 8.a scrub starts 2024-02-05T08:14:14.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:14.354 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:14.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:14.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:14.495 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:14.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:14.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:14.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:14.627 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:14.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:14.699 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T08:14:14.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:14.779 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:14.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:14.880 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:14.922 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:14.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:14.947 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:14 smithi134 ceph-mon[30086]: pgmap v411: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 1.8 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.78k op/s 2024-02-05T08:14:14.947 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:14 smithi134 ceph-mon[30086]: 5.18 scrub starts 2024-02-05T08:14:14.947 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:14 smithi134 ceph-mon[30086]: 5.18 scrub ok 2024-02-05T08:14:14.947 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:14 smithi134 ceph-mon[30086]: 8.a scrub ok 2024-02-05T08:14:14.947 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:14 smithi134 ceph-mon[30086]: 6.18 scrub starts 2024-02-05T08:14:14.947 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:14 smithi134 ceph-mon[30086]: 6.18 scrub ok 2024-02-05T08:14:14.947 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:14 smithi134 ceph-mon[39278]: pgmap v411: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 1.8 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.78k op/s 2024-02-05T08:14:14.947 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:14 smithi134 ceph-mon[39278]: 5.18 scrub starts 2024-02-05T08:14:14.947 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:14 smithi134 ceph-mon[39278]: 5.18 scrub ok 2024-02-05T08:14:14.947 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:14 smithi134 ceph-mon[39278]: 8.a scrub ok 2024-02-05T08:14:14.948 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:14 smithi134 ceph-mon[39278]: 6.18 scrub starts 2024-02-05T08:14:14.948 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:14 smithi134 ceph-mon[39278]: 6.18 scrub ok 2024-02-05T08:14:15.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:15.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:15.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:15.164 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:15.199 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:15.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:15.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:15.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:14 smithi196 ceph-mon[33018]: pgmap v411: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 1.8 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.78k op/s 2024-02-05T08:14:15.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:14 smithi196 ceph-mon[33018]: 5.18 scrub starts 2024-02-05T08:14:15.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:14 smithi196 ceph-mon[33018]: 5.18 scrub ok 2024-02-05T08:14:15.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:14 smithi196 ceph-mon[33018]: 8.a scrub ok 2024-02-05T08:14:15.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:14 smithi196 ceph-mon[33018]: 6.18 scrub starts 2024-02-05T08:14:15.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:14 smithi196 ceph-mon[33018]: 6.18 scrub ok 2024-02-05T08:14:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:15.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:15.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:15.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:15.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:15.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:15.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:15.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:15.691 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:15.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:15.792 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:15.825 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:15.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:15.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:15.927 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:15.963 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:15.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:15.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:16.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[30086]: 8.5 scrub starts 2024-02-05T08:14:16.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[30086]: 3.e scrub starts 2024-02-05T08:14:16.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[30086]: 3.e scrub ok 2024-02-05T08:14:16.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[30086]: 8.5 scrub ok 2024-02-05T08:14:16.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[30086]: 4.c scrub starts 2024-02-05T08:14:16.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[30086]: 4.c scrub ok 2024-02-05T08:14:16.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[30086]: 8.c scrub starts 2024-02-05T08:14:16.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[39278]: 8.5 scrub starts 2024-02-05T08:14:16.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[39278]: 3.e scrub starts 2024-02-05T08:14:16.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[39278]: 3.e scrub ok 2024-02-05T08:14:16.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[39278]: 8.5 scrub ok 2024-02-05T08:14:16.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[39278]: 4.c scrub starts 2024-02-05T08:14:16.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[39278]: 4.c scrub ok 2024-02-05T08:14:16.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:15 smithi134 ceph-mon[39278]: 8.c scrub starts 2024-02-05T08:14:16.064 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:16.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:16.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:16.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:16.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:16.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:16.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:16.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:15 smithi196 ceph-mon[33018]: 8.5 scrub starts 2024-02-05T08:14:16.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:15 smithi196 ceph-mon[33018]: 3.e scrub starts 2024-02-05T08:14:16.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:15 smithi196 ceph-mon[33018]: 3.e scrub ok 2024-02-05T08:14:16.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:15 smithi196 ceph-mon[33018]: 8.5 scrub ok 2024-02-05T08:14:16.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:15 smithi196 ceph-mon[33018]: 4.c scrub starts 2024-02-05T08:14:16.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:15 smithi196 ceph-mon[33018]: 4.c scrub ok 2024-02-05T08:14:16.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:15 smithi196 ceph-mon[33018]: 8.c scrub starts 2024-02-05T08:14:16.335 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:16.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:16.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:16.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:16.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:16.494 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:16.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:16.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:16.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:16.636 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:16.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:16.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:16.773 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:16.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:16.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:16.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:16.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:16.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:16.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:17.003 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:16 smithi134 ceph-mon[30086]: pgmap v412: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 1.8 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.34k op/s 2024-02-05T08:14:17.003 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:16 smithi134 ceph-mon[30086]: 8.c scrub ok 2024-02-05T08:14:17.004 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:16 smithi134 ceph-mon[30086]: 6.9 scrub starts 2024-02-05T08:14:17.004 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:16 smithi134 ceph-mon[39278]: pgmap v412: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 1.8 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.34k op/s 2024-02-05T08:14:17.004 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:16 smithi134 ceph-mon[39278]: 8.c scrub ok 2024-02-05T08:14:17.004 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:16 smithi134 ceph-mon[39278]: 6.9 scrub starts 2024-02-05T08:14:17.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:17.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:17.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:17.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:17.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:17.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:17.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:17.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:17.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:16 smithi196 ceph-mon[33018]: pgmap v412: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 1.8 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.34k op/s 2024-02-05T08:14:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:16 smithi196 ceph-mon[33018]: 8.c scrub ok 2024-02-05T08:14:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:16 smithi196 ceph-mon[33018]: 6.9 scrub starts 2024-02-05T08:14:17.318 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:17.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:17.420 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:17.458 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:17.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:17.559 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:17.585 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:17.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:17.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:17.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:17.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:17.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:17.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:17.857 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:17.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:17.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:17.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:17.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:17.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:17.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:18.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:14:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:14:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:14:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:17 smithi134 ceph-mon[30086]: 6.9 scrub ok 2024-02-05T08:14:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:17 smithi134 ceph-mon[30086]: 4.1a scrub starts 2024-02-05T08:14:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:17 smithi134 ceph-mon[30086]: 4.1a scrub ok 2024-02-05T08:14:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:17 smithi134 ceph-mon[30086]: 3.8 scrub starts 2024-02-05T08:14:18.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:17 smithi134 ceph-mon[30086]: 3.8 scrub ok 2024-02-05T08:14:18.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:17 smithi134 ceph-mon[39278]: 6.9 scrub ok 2024-02-05T08:14:18.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:17 smithi134 ceph-mon[39278]: 4.1a scrub starts 2024-02-05T08:14:18.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:17 smithi134 ceph-mon[39278]: 4.1a scrub ok 2024-02-05T08:14:18.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:17 smithi134 ceph-mon[39278]: 3.8 scrub starts 2024-02-05T08:14:18.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:17 smithi134 ceph-mon[39278]: 3.8 scrub ok 2024-02-05T08:14:18.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:18.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:18.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:18.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:18.234 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:18.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:18.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:18.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:17 smithi196 ceph-mon[33018]: 6.9 scrub ok 2024-02-05T08:14:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:17 smithi196 ceph-mon[33018]: 4.1a scrub starts 2024-02-05T08:14:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:17 smithi196 ceph-mon[33018]: 4.1a scrub ok 2024-02-05T08:14:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:17 smithi196 ceph-mon[33018]: 3.8 scrub starts 2024-02-05T08:14:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:17 smithi196 ceph-mon[33018]: 3.8 scrub ok 2024-02-05T08:14:18.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:18.401 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:18.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:18.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:18.539 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:18.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:18.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:18.630 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:14:18.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:18.664 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:14:18.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:18.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:18.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:18.695 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:18.711 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:14:18.748 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:14:18.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:18.794 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:14:18.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:18.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:18.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:18.833 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:14:18.872 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:14:18.911 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:14:18.914 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:14:18.914 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:14:18.914 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:14:18.914 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:14:18.914 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:14:18.914 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:14:18.914 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:14:18.914 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:14:18.919 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:18.966 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:18.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:18.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:19.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:19.107 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:19.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:19.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:19.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:19.242 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:19.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:19.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[30086]: 3.10 scrub starts 2024-02-05T08:14:19.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[30086]: 3.10 scrub ok 2024-02-05T08:14:19.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[30086]: 4.11 scrub starts 2024-02-05T08:14:19.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[30086]: 4.11 scrub ok 2024-02-05T08:14:19.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[30086]: pgmap v413: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 854 op/s 2024-02-05T08:14:19.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[30086]: 8.8 scrub starts 2024-02-05T08:14:19.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3035083733' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:19.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[30086]: 8.7 scrub starts 2024-02-05T08:14:19.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[39278]: 3.10 scrub starts 2024-02-05T08:14:19.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[39278]: 3.10 scrub ok 2024-02-05T08:14:19.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[39278]: 4.11 scrub starts 2024-02-05T08:14:19.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[39278]: 4.11 scrub ok 2024-02-05T08:14:19.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[39278]: pgmap v413: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 854 op/s 2024-02-05T08:14:19.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[39278]: 8.8 scrub starts 2024-02-05T08:14:19.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3035083733' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:19.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:18 smithi134 ceph-mon[39278]: 8.7 scrub starts 2024-02-05T08:14:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:18 smithi196 ceph-mon[33018]: 3.10 scrub starts 2024-02-05T08:14:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:18 smithi196 ceph-mon[33018]: 3.10 scrub ok 2024-02-05T08:14:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:18 smithi196 ceph-mon[33018]: 4.11 scrub starts 2024-02-05T08:14:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:18 smithi196 ceph-mon[33018]: 4.11 scrub ok 2024-02-05T08:14:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:18 smithi196 ceph-mon[33018]: pgmap v413: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 854 op/s 2024-02-05T08:14:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:18 smithi196 ceph-mon[33018]: 8.8 scrub starts 2024-02-05T08:14:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:18 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3035083733' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:18 smithi196 ceph-mon[33018]: 8.7 scrub starts 2024-02-05T08:14:19.344 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:19.381 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:19.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:19.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:19.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:19.524 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:19.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:19.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:19.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:19.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:19.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:19.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:19.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:19.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:19.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:19.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:19.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:19.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:19.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:14:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:14:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[30086]: 8.8 scrub ok 2024-02-05T08:14:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[30086]: 4.1b scrub starts 2024-02-05T08:14:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[30086]: 4.1b scrub ok 2024-02-05T08:14:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[30086]: 8.7 scrub ok 2024-02-05T08:14:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[30086]: 3.4 scrub starts 2024-02-05T08:14:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[30086]: 3.4 scrub ok 2024-02-05T08:14:20.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:14:20.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:14:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[39278]: 8.8 scrub ok 2024-02-05T08:14:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[39278]: 4.1b scrub starts 2024-02-05T08:14:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[39278]: 4.1b scrub ok 2024-02-05T08:14:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[39278]: 8.7 scrub ok 2024-02-05T08:14:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[39278]: 3.4 scrub starts 2024-02-05T08:14:20.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:19 smithi134 ceph-mon[39278]: 3.4 scrub ok 2024-02-05T08:14:20.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:20.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:20.083 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:20.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:20.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:20.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:20.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:20.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:19 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:14:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:19 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:14:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:19 smithi196 ceph-mon[33018]: 8.8 scrub ok 2024-02-05T08:14:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:19 smithi196 ceph-mon[33018]: 4.1b scrub starts 2024-02-05T08:14:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:19 smithi196 ceph-mon[33018]: 4.1b scrub ok 2024-02-05T08:14:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:19 smithi196 ceph-mon[33018]: 8.7 scrub ok 2024-02-05T08:14:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:19 smithi196 ceph-mon[33018]: 3.4 scrub starts 2024-02-05T08:14:20.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:19 smithi196 ceph-mon[33018]: 3.4 scrub ok 2024-02-05T08:14:20.322 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:20.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:20.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:20.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:20.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:20.487 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:20.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:20.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:20.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:20.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:20.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:20.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:20.725 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:20.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:20.764 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:20.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:20.796 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:20.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:20.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:20.913 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:20.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:21.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:21.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:20 smithi134 ceph-mon[30086]: pgmap v414: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-02-05T08:14:21.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:20 smithi134 ceph-mon[30086]: 8.d scrub starts 2024-02-05T08:14:21.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:20 smithi134 ceph-mon[39278]: pgmap v414: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-02-05T08:14:21.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:20 smithi134 ceph-mon[39278]: 8.d scrub starts 2024-02-05T08:14:21.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:21.054 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:21.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:21.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:21.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:21.198 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:21.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:21.299 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:21.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:20 smithi196 ceph-mon[33018]: pgmap v414: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-02-05T08:14:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:20 smithi196 ceph-mon[33018]: 8.d scrub starts 2024-02-05T08:14:21.343 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:21.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:21.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:21.445 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:21.485 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:21.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:21.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:21.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:21.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:21.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:21.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:21.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:21.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:21.865 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:21.903 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:21.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:21.946 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:14:22.005 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:22.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:22.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:22.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:22.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:22.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:22.255 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:14:22.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:21 smithi134 ceph-mon[30086]: 8.d scrub ok 2024-02-05T08:14:22.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:21 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1291908460' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:14:22.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:21 smithi134 ceph-mon[30086]: 4.12 scrub starts 2024-02-05T08:14:22.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:21 smithi134 ceph-mon[30086]: 4.12 scrub ok 2024-02-05T08:14:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:21 smithi134 ceph-mon[39278]: 8.d scrub ok 2024-02-05T08:14:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:21 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1291908460' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:14:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:21 smithi134 ceph-mon[39278]: 4.12 scrub starts 2024-02-05T08:14:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:21 smithi134 ceph-mon[39278]: 4.12 scrub ok 2024-02-05T08:14:22.277 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:21 smithi196 ceph-mon[33018]: 8.d scrub ok 2024-02-05T08:14:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:21 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1291908460' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:14:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:21 smithi196 ceph-mon[33018]: 4.12 scrub starts 2024-02-05T08:14:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:21 smithi196 ceph-mon[33018]: 4.12 scrub ok 2024-02-05T08:14:22.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:22.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:22.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:22.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:22.474 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:22.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:22.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:22.613 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:22.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:22.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:22.715 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:22.750 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:22.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:22.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:22.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:22.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:22.987 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:23.020 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:23.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:23.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:23.158 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:23.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:23.168 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[30086]: 4.7 scrub starts 2024-02-05T08:14:23.168 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[30086]: 4.7 scrub ok 2024-02-05T08:14:23.168 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[30086]: pgmap v415: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-02-05T08:14:23.168 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[30086]: 6.e scrub starts 2024-02-05T08:14:23.168 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[30086]: 6.e scrub ok 2024-02-05T08:14:23.168 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:14:23.168 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:14:23.168 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1291908460' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:14:23.169 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[30086]: osdmap e225: 8 total, 8 up, 7 in 2024-02-05T08:14:23.169 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:14:23.169 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:14:23.169 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[39278]: 4.7 scrub starts 2024-02-05T08:14:23.170 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[39278]: 4.7 scrub ok 2024-02-05T08:14:23.170 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[39278]: pgmap v415: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-02-05T08:14:23.170 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[39278]: 6.e scrub starts 2024-02-05T08:14:23.170 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[39278]: 6.e scrub ok 2024-02-05T08:14:23.170 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:14:23.170 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:14:23.170 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1291908460' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:14:23.170 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[39278]: osdmap e225: 8 total, 8 up, 7 in 2024-02-05T08:14:23.170 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:14:23.170 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:22 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:14:23.260 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:23.299 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:23.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:22 smithi196 ceph-mon[33018]: 4.7 scrub starts 2024-02-05T08:14:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:22 smithi196 ceph-mon[33018]: 4.7 scrub ok 2024-02-05T08:14:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:22 smithi196 ceph-mon[33018]: pgmap v415: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-02-05T08:14:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:22 smithi196 ceph-mon[33018]: 6.e scrub starts 2024-02-05T08:14:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:22 smithi196 ceph-mon[33018]: 6.e scrub ok 2024-02-05T08:14:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:22 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:14:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:22 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:14:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:22 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1291908460' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:14:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:22 smithi196 ceph-mon[33018]: osdmap e225: 8 total, 8 up, 7 in 2024-02-05T08:14:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:14:23.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:22 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:14:23.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:23.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:23.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:23.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:23.573 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:23.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:23.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:23.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:23.712 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:23.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:23.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:23.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:14:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:14:23.849 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:23.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:23.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:23.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:23.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:24.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:24.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:24.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:24.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:24.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:24.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:24.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:24.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:24.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[30086]: 5.2 scrub starts 2024-02-05T08:14:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[30086]: 5.2 scrub ok 2024-02-05T08:14:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[30086]: 3.f scrub starts 2024-02-05T08:14:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[30086]: 3.f scrub ok 2024-02-05T08:14:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-05T08:14:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-05T08:14:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[30086]: 6.10 scrub starts 2024-02-05T08:14:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[30086]: 6.10 scrub ok 2024-02-05T08:14:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:24.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[39278]: 5.2 scrub starts 2024-02-05T08:14:24.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[39278]: 5.2 scrub ok 2024-02-05T08:14:24.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[39278]: 3.f scrub starts 2024-02-05T08:14:24.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[39278]: 3.f scrub ok 2024-02-05T08:14:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-05T08:14:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-05T08:14:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[39278]: 6.10 scrub starts 2024-02-05T08:14:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[39278]: 6.10 scrub ok 2024-02-05T08:14:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:23 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:23 smithi196 ceph-mon[33018]: 5.2 scrub starts 2024-02-05T08:14:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:23 smithi196 ceph-mon[33018]: 5.2 scrub ok 2024-02-05T08:14:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:23 smithi196 ceph-mon[33018]: 3.f scrub starts 2024-02-05T08:14:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:23 smithi196 ceph-mon[33018]: 3.f scrub ok 2024-02-05T08:14:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-02-05T08:14:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-02-05T08:14:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:23 smithi196 ceph-mon[33018]: 6.10 scrub starts 2024-02-05T08:14:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:23 smithi196 ceph-mon[33018]: 6.10 scrub ok 2024-02-05T08:14:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:23 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:24.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:24.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:24.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:24.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:24.493 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:24.527 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:24.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:24.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:24.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:24.668 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:24.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:24.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:24.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:24.808 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:24.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:24.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:24.950 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:24.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:25.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:25.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:25.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:25.189 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:25.230 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:25.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:25.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[30086]: pgmap v417: 148 pgs: 148 active+clean; 1.0 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-02-05T08:14:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-02-05T08:14:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-05T08:14:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[30086]: osdmap e226: 8 total, 8 up, 7 in 2024-02-05T08:14:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:14:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:14:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:14:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:14:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[39278]: pgmap v417: 148 pgs: 148 active+clean; 1.0 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-02-05T08:14:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-02-05T08:14:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-05T08:14:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[39278]: osdmap e226: 8 total, 8 up, 7 in 2024-02-05T08:14:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:14:25.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:14:25.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:14:25.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:24 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:14:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:24 smithi196 ceph-mon[33018]: pgmap v417: 148 pgs: 148 active+clean; 1.0 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.29k op/s 2024-02-05T08:14:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:24 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-02-05T08:14:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:24 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-02-05T08:14:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:24 smithi196 ceph-mon[33018]: osdmap e226: 8 total, 8 up, 7 in 2024-02-05T08:14:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:24 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:14:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:24 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:14:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:24 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:14:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:24 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:14:25.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:25.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:25.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:25.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:25.507 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:25.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:25.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:25.609 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:25.630 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:25.669 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:25.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:25.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:25.805 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:25.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:25.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:25.907 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:25.947 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:25.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:25.998 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.1 to 1 (10000) 2024-02-05T08:14:26.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:26.083 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:26.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:26.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:26.114 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[30086]: 4.0 scrub starts 2024-02-05T08:14:26.114 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[30086]: 4.0 scrub ok 2024-02-05T08:14:26.114 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[30086]: osdmap e227: 8 total, 8 up, 7 in 2024-02-05T08:14:26.114 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[30086]: 3.9 scrub starts 2024-02-05T08:14:26.114 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[30086]: 3.9 scrub ok 2024-02-05T08:14:26.114 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[30086]: 8.6 scrub starts 2024-02-05T08:14:26.114 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1666281964' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:14:26.114 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:14:26.114 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[39278]: 4.0 scrub starts 2024-02-05T08:14:26.115 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[39278]: 4.0 scrub ok 2024-02-05T08:14:26.115 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[39278]: osdmap e227: 8 total, 8 up, 7 in 2024-02-05T08:14:26.115 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[39278]: 3.9 scrub starts 2024-02-05T08:14:26.115 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[39278]: 3.9 scrub ok 2024-02-05T08:14:26.115 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[39278]: 8.6 scrub starts 2024-02-05T08:14:26.115 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1666281964' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:14:26.115 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:25 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:14:26.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:26.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:26.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:25 smithi196 ceph-mon[33018]: 4.0 scrub starts 2024-02-05T08:14:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:25 smithi196 ceph-mon[33018]: 4.0 scrub ok 2024-02-05T08:14:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:25 smithi196 ceph-mon[33018]: osdmap e227: 8 total, 8 up, 7 in 2024-02-05T08:14:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:25 smithi196 ceph-mon[33018]: 3.9 scrub starts 2024-02-05T08:14:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:25 smithi196 ceph-mon[33018]: 3.9 scrub ok 2024-02-05T08:14:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:25 smithi196 ceph-mon[33018]: 8.6 scrub starts 2024-02-05T08:14:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:25 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1666281964' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:14:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:25 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:14:26.319 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:14:26.320 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:14:26.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:26.364 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:26.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:26.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:26.507 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:26.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:26.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:26.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:26.642 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:26.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:26.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:26.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:26.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:26.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:26.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:26.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:26.916 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:26.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:26.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:27.008 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:14:27.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:27.085 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:27.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:27.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:27.223 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:27.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:27.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:27.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:26 smithi134 ceph-mon[30086]: 8.6 scrub ok 2024-02-05T08:14:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:26 smithi134 ceph-mon[30086]: pgmap v420: 148 pgs: 148 active+clean; 1.0 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T08:14:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:26 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:14:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:26 smithi134 ceph-mon[30086]: osdmap e228: 8 total, 8 up, 7 in 2024-02-05T08:14:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:26 smithi134 ceph-mon[39278]: 8.6 scrub ok 2024-02-05T08:14:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:26 smithi134 ceph-mon[39278]: pgmap v420: 148 pgs: 148 active+clean; 1.0 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T08:14:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:26 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:14:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:26 smithi134 ceph-mon[39278]: osdmap e228: 8 total, 8 up, 7 in 2024-02-05T08:14:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:26 smithi196 ceph-mon[33018]: 8.6 scrub ok 2024-02-05T08:14:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:26 smithi196 ceph-mon[33018]: pgmap v420: 148 pgs: 148 active+clean; 1.0 GiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T08:14:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:26 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:14:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:26 smithi196 ceph-mon[33018]: osdmap e228: 8 total, 8 up, 7 in 2024-02-05T08:14:27.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:27.368 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:27.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:27.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:27.470 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:27.506 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:27.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:27.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:27.608 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:27.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:27.643 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:27.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:27.744 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:27.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:27.783 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:27.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:27.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:27.889 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T08:14:27.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:27.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:27.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:28.012 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:14:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:14:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:14:28.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:28.064 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:28.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:28.164 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:28.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:28.203 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:28.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:28.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:28 smithi134 ceph-mon[39278]: 4.1f scrub starts 2024-02-05T08:14:28.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:28 smithi134 ceph-mon[39278]: 4.1f scrub ok 2024-02-05T08:14:28.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:28 smithi134 ceph-mon[39278]: 6.1c scrub starts 2024-02-05T08:14:28.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:28 smithi134 ceph-mon[39278]: 6.1c scrub ok 2024-02-05T08:14:28.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:28 smithi134 ceph-mon[39278]: 5.1 scrub starts 2024-02-05T08:14:28.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:28 smithi134 ceph-mon[39278]: 5.1 scrub ok 2024-02-05T08:14:28.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:28 smithi134 ceph-mon[30086]: 4.1f scrub starts 2024-02-05T08:14:28.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:28 smithi134 ceph-mon[30086]: 4.1f scrub ok 2024-02-05T08:14:28.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:28 smithi134 ceph-mon[30086]: 6.1c scrub starts 2024-02-05T08:14:28.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:28 smithi134 ceph-mon[30086]: 6.1c scrub ok 2024-02-05T08:14:28.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:28 smithi134 ceph-mon[30086]: 5.1 scrub starts 2024-02-05T08:14:28.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:28 smithi134 ceph-mon[30086]: 5.1 scrub ok 2024-02-05T08:14:28.287 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 2 1 2024-02-05T08:14:28.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:28 smithi196 ceph-mon[33018]: 4.1f scrub starts 2024-02-05T08:14:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:28 smithi196 ceph-mon[33018]: 4.1f scrub ok 2024-02-05T08:14:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:28 smithi196 ceph-mon[33018]: 6.1c scrub starts 2024-02-05T08:14:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:28 smithi196 ceph-mon[33018]: 6.1c scrub ok 2024-02-05T08:14:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:28 smithi196 ceph-mon[33018]: 5.1 scrub starts 2024-02-05T08:14:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:28 smithi196 ceph-mon[33018]: 5.1 scrub ok 2024-02-05T08:14:28.380 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:28.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:28.482 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:28.533 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:28.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:28.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:28.669 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:28.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:28.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:28.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:28.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:28.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:28.825 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:28.831 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:28.904 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:28.937 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:28.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:28.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:29.039 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:29.080 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:29.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:29.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:29.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:29.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:29.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:29.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:29.274 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:29 smithi134 ceph-mon[30086]: 3.1d scrub starts 2024-02-05T08:14:29.275 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:29 smithi134 ceph-mon[30086]: 3.1d scrub ok 2024-02-05T08:14:29.275 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:29 smithi134 ceph-mon[30086]: pgmap v422: 148 pgs: 148 active+clean; 943 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-02-05T08:14:29.275 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:29 smithi134 ceph-mon[30086]: 6.a scrub starts 2024-02-05T08:14:29.275 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:29 smithi134 ceph-mon[39278]: 3.1d scrub starts 2024-02-05T08:14:29.275 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:29 smithi134 ceph-mon[39278]: 3.1d scrub ok 2024-02-05T08:14:29.275 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:29 smithi134 ceph-mon[39278]: pgmap v422: 148 pgs: 148 active+clean; 943 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-02-05T08:14:29.275 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:29 smithi134 ceph-mon[39278]: 6.a scrub starts 2024-02-05T08:14:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:29 smithi196 ceph-mon[33018]: 3.1d scrub starts 2024-02-05T08:14:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:29 smithi196 ceph-mon[33018]: 3.1d scrub ok 2024-02-05T08:14:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:29 smithi196 ceph-mon[33018]: pgmap v422: 148 pgs: 148 active+clean; 943 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-02-05T08:14:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:29 smithi196 ceph-mon[33018]: 6.a scrub starts 2024-02-05T08:14:29.319 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:29.356 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:29.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:29.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:29.457 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:29.509 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:29.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:29.611 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:29.651 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:29.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:29.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:29.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:29.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:29.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:29.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:29.890 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:29.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:29.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:29.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:30.031 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:30.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:30.072 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:30.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:30.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:30.211 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:30.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:30.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[30086]: 6.a scrub ok 2024-02-05T08:14:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:14:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:14:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[30086]: 5.f scrub starts 2024-02-05T08:14:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[30086]: 4.16 scrub starts 2024-02-05T08:14:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[30086]: 4.16 scrub ok 2024-02-05T08:14:30.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[30086]: osdmap e229: 8 total, 8 up, 7 in 2024-02-05T08:14:30.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[39278]: 6.a scrub ok 2024-02-05T08:14:30.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:14:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:14:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[39278]: 5.f scrub starts 2024-02-05T08:14:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[39278]: 4.16 scrub starts 2024-02-05T08:14:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[39278]: 4.16 scrub ok 2024-02-05T08:14:30.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:30 smithi134 ceph-mon[39278]: osdmap e229: 8 total, 8 up, 7 in 2024-02-05T08:14:30.313 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:30 smithi196 ceph-mon[33018]: 6.a scrub ok 2024-02-05T08:14:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:30 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:14:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:30 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:14:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:30 smithi196 ceph-mon[33018]: 5.f scrub starts 2024-02-05T08:14:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:30 smithi196 ceph-mon[33018]: 4.16 scrub starts 2024-02-05T08:14:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:30 smithi196 ceph-mon[33018]: 4.16 scrub ok 2024-02-05T08:14:30.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:30 smithi196 ceph-mon[33018]: osdmap e229: 8 total, 8 up, 7 in 2024-02-05T08:14:30.339 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:14:30.339 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:14:30.339 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:14:30.339 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:14:30.339 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:14:30.347 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:14:30.349 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:30.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:30.387 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:14:30.431 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:14:30.436 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:14:30.436 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:14:30.436 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:14:30.436 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:14:30.437 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:14:30.437 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:14:30.437 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:14:30.437 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:14:30.451 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:30.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:30.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:30.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:30.637 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:30.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:30.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:30.738 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:30.772 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:30.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:30.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:30.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:30.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:31.009 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:31.063 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:31.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:31.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:31.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:31.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:31.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[30086]: 5.f scrub ok 2024-02-05T08:14:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[30086]: pgmap v423: 148 pgs: 148 active+clean; 618 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 18 B/s, 1 objects/s recovering 2024-02-05T08:14:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[30086]: 3.14 deep-scrub starts 2024-02-05T08:14:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[30086]: 3.14 deep-scrub ok 2024-02-05T08:14:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1831356717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[30086]: 5.3 scrub starts 2024-02-05T08:14:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[30086]: 5.3 scrub ok 2024-02-05T08:14:31.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[39278]: 5.f scrub ok 2024-02-05T08:14:31.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[39278]: pgmap v423: 148 pgs: 148 active+clean; 618 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 18 B/s, 1 objects/s recovering 2024-02-05T08:14:31.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[39278]: 3.14 deep-scrub starts 2024-02-05T08:14:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[39278]: 3.14 deep-scrub ok 2024-02-05T08:14:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1831356717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[39278]: 5.3 scrub starts 2024-02-05T08:14:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:31 smithi134 ceph-mon[39278]: 5.3 scrub ok 2024-02-05T08:14:31.303 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:31 smithi196 ceph-mon[33018]: 5.f scrub ok 2024-02-05T08:14:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:31 smithi196 ceph-mon[33018]: pgmap v423: 148 pgs: 148 active+clean; 618 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 18 B/s, 1 objects/s recovering 2024-02-05T08:14:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:31 smithi196 ceph-mon[33018]: 3.14 deep-scrub starts 2024-02-05T08:14:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:31 smithi196 ceph-mon[33018]: 3.14 deep-scrub ok 2024-02-05T08:14:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:31 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1831356717' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:31 smithi196 ceph-mon[33018]: 5.3 scrub starts 2024-02-05T08:14:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:31 smithi196 ceph-mon[33018]: 5.3 scrub ok 2024-02-05T08:14:31.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:31.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:31.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:31.443 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:31.478 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:31.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:31.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:31.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:31.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:31.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:31.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:31.715 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:31.753 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:31.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:31.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:31.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:31.889 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:31.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:31.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:31.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:32.023 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:32.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:32.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:32.125 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:32.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:32.175 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:32.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:32.275 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:32.305 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:14:32.306 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:14:32.306 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[30086]: osdmap e230: 8 total, 8 up, 7 in 2024-02-05T08:14:32.306 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[30086]: 5.e scrub starts 2024-02-05T08:14:32.306 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[30086]: 5.e scrub ok 2024-02-05T08:14:32.306 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[30086]: 3.18 scrub starts 2024-02-05T08:14:32.306 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[30086]: 3.18 scrub ok 2024-02-05T08:14:32.308 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:14:32.308 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:14:32.308 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[39278]: osdmap e230: 8 total, 8 up, 7 in 2024-02-05T08:14:32.308 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[39278]: 5.e scrub starts 2024-02-05T08:14:32.308 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[39278]: 5.e scrub ok 2024-02-05T08:14:32.308 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[39278]: 3.18 scrub starts 2024-02-05T08:14:32.308 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:32 smithi134 ceph-mon[39278]: 3.18 scrub ok 2024-02-05T08:14:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:32 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:14:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:32 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:14:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:32 smithi196 ceph-mon[33018]: osdmap e230: 8 total, 8 up, 7 in 2024-02-05T08:14:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:32 smithi196 ceph-mon[33018]: 5.e scrub starts 2024-02-05T08:14:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:32 smithi196 ceph-mon[33018]: 5.e scrub ok 2024-02-05T08:14:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:32 smithi196 ceph-mon[33018]: 3.18 scrub starts 2024-02-05T08:14:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:32 smithi196 ceph-mon[33018]: 3.18 scrub ok 2024-02-05T08:14:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:32.318 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:32.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:32.407 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:14:32.407 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:14:32.408 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:14:32.408 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:14:32.408 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.408 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21740 7.e 7:702731a7:::benchmark_data_smithi134_83016_object21739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:14:32.408 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.327781+0000", 2024-02-05T08:14:32.408 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 452.075922031, 2024-02-05T08:14:32.408 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15303014300000001, 2024-02-05T08:14:32.408 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.408 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.408 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.408 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:14:32.408 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:14:32.408 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21740 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327781+0000", 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327781+0000", 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.409 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327783+0000", 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.688e-06 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327805+0000", 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2300999999999999e-05 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327817+0000", 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2233e-05 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.410 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327825+0000", 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1450000000000008e-06 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.327885+0000", 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9546999999999998e-05 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.328002+0000", 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000117374 2024-02-05T08:14:32.411 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.412 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.412 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:14:32.412 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.328125+0000", 2024-02-05T08:14:32.412 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012256800000000001 2024-02-05T08:14:32.412 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.412 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.412 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.412 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329306+0000", 2024-02-05T08:14:32.412 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001181423 2024-02-05T08:14:32.412 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.412 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.412 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.412 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330032+0000", 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00072612800000000004 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480676+0000", 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15064364599999999 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480734+0000", 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7960000000000001e-05 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.413 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.414 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480811+0000", 2024-02-05T08:14:32.414 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7130000000000002e-05 2024-02-05T08:14:32.414 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.414 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.414 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.414 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.414 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.414 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.14802.0:21738 7.a e104/95 7:5f5817b2:::benchmark_data_smithi134_83016_object21737:head v 104'1353, mlcod=104'1351)", 2024-02-05T08:14:32.414 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.329123+0000", 2024-02-05T08:14:32.414 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 452.07457981800002, 2024-02-05T08:14:32.414 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15121406700000001, 2024-02-05T08:14:32.414 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.414 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.415 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.415 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.415 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.415 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329123+0000", 2024-02-05T08:14:32.415 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.415 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.415 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.415 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.415 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329123+0000", 2024-02-05T08:14:32.415 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.415 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.415 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.415 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.416 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329124+0000", 2024-02-05T08:14:32.416 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2500000000000001e-07 2024-02-05T08:14:32.416 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.416 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.416 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.416 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329156+0000", 2024-02-05T08:14:32.416 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2563e-05 2024-02-05T08:14:32.416 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.416 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.416 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.416 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329166+0000", 2024-02-05T08:14:32.416 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6069999999999997e-06 2024-02-05T08:14:32.416 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329172+0000", 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5339999999999996e-06 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329270+0000", 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7374999999999995e-05 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329289+0000", 2024-02-05T08:14:32.417 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9258000000000001e-05 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330442+0000", 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00115318 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480337+0000", 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14989512499999999 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.418 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:21746 7.8 7:1fd2f339:::benchmark_data_smithi134_83016_object21745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.329211+0000", 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 452.07449234299997, 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15186722999999999, 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 21746 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.419 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329211+0000", 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329211+0000", 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329212+0000", 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1200000000000001e-06 2024-02-05T08:14:32.420 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329233+0000", 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1157000000000001e-05 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329245+0000", 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2095e-05 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329253+0000", 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0059999999999992e-06 2024-02-05T08:14:32.421 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329312+0000", 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8953000000000003e-05 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329460+0000", 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014786199999999999 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,7", 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.329592+0000", 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013251199999999999 2024-02-05T08:14:32.422 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330443+0000", 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00085042699999999995 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480847+0000", 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15040416000000001 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480896+0000", 2024-02-05T08:14:32.423 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9152e-05 2024-02-05T08:14:32.424 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.424 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.424 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.424 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.481044+0000", 2024-02-05T08:14:32.424 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014736299999999999 2024-02-05T08:14:32.424 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.424 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.424 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.424 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.481078+0000", 2024-02-05T08:14:32.424 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4422999999999999e-05 2024-02-05T08:14:32.424 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.424 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.424 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:21746 7.8 e104/95 ondisk, result = 0)", 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.330823+0000", 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 452.07288021599999, 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15025973000000001, 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330823+0000", 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.425 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330823+0000", 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330823+0000", 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7100000000000002e-07 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330827+0000", 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7249999999999999e-06 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.426 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480582+0000", 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14975533799999999 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480591+0000", 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8789999999999992e-06 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480886+0000", 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00029442999999999998 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480892+0000", 2024-02-05T08:14:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1240000000000002e-06 2024-02-05T08:14:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.481083+0000", 2024-02-05T08:14:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019066299999999999 2024-02-05T08:14:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.428 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.14802.0:21746 7.8 e104/95 ondisk, result = 0)", 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:07:00.330832+0000", 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 452.072871147, 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.150026044, 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330832+0000", 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.429 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330832+0000", 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330832+0000", 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3799999999999997e-07 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.330835+0000", 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.971e-06 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.430 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480208+0000", 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.149372539 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480221+0000", 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2695e-05 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480323+0000", 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010194799999999999 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480331+0000", 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3499999999999997e-06 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:07:00.480858+0000", 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052700299999999995 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.432 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106678 7.e 7:7f897189:::benchmark_data_smithi134_83016_object106677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.632287+0000", 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 375.771416096, 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14517614000000001, 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106678 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.433 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632287+0000", 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632287+0000", 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632288+0000", 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3e-06 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.434 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632314+0000", 2024-02-05T08:14:32.435 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5622e-05 2024-02-05T08:14:32.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.435 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.435 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632326+0000", 2024-02-05T08:14:32.435 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:32.436 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1901e-05 2024-02-05T08:14:32.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632333+0000", 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.08e-06 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633044+0000", 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071079200000000002 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633761+0000", 2024-02-05T08:14:32.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071729699999999999 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633976+0000", 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021524099999999999 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650870+0000", 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.016893551 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.438 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650906+0000", 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6427000000000002e-05 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777225+0000", 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.126318758 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777321+0000", 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6436999999999998e-05 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777463+0000", 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000141734 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106682 7.e 7:7cfa27eb:::benchmark_data_smithi134_83016_object106681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.633782+0000", 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 375.76992116700001, 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14385479700000001, 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.440 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106682 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633782+0000", 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633782+0000", 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633783+0000", 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1090000000000001e-06 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633813+0000", 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9856999999999999e-05 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633823+0000", 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0222e-05 2024-02-05T08:14:32.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633830+0000", 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0639999999999996e-06 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642537+0000", 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0087064080000000005 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642683+0000", 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000146527 2024-02-05T08:14:32.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.444 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:14:32.444 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642858+0000", 2024-02-05T08:14:32.444 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017454499999999999 2024-02-05T08:14:32.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.444 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.444 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650945+0000", 2024-02-05T08:14:32.444 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0080870500000000001 2024-02-05T08:14:32.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.444 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.444 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.769282+0000", 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.118337331 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777475+0000", 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0081926900000000007 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777545+0000", 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0735000000000006e-05 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777637+0000", 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1259000000000002e-05 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:131934 8.f 8:fd4b76ad:::benchmark_data_smithi134_115836_object15274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:58.583316+0000", 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 93.820386909000007, 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15811914799999999, 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.446 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 131934 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583316+0000", 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583316+0000", 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583318+0000", 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.593e-06 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583324+0000", 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3620000000000002e-06 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583342+0000", 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7396000000000001e-05 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.448 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583351+0000", 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7429999999999997e-06 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.589777+0000", 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0064258070000000004 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.589907+0000", 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000129913 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.450 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:14:32.450 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.590038+0000", 2024-02-05T08:14:32.450 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013120100000000001 2024-02-05T08:14:32.450 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.450 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.450 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.450 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.595790+0000", 2024-02-05T08:14:32.450 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0057520330000000001 2024-02-05T08:14:32.450 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.450 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.450 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.450 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596232+0000", 2024-02-05T08:14:32.450 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000442075 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600111+0000", 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003878898 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.603530+0000", 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0034189889999999999 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.741275+0000", 2024-02-05T08:14:32.451 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13774504400000001 2024-02-05T08:14:32.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.452 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.452 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.452 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.741435+0000", 2024-02-05T08:14:32.452 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000160094 2024-02-05T08:14:32.452 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.452 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.452 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.452 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.452 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:131938 8.f 8:fc014979:::benchmark_data_smithi134_115836_object15278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:14:32.452 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:58.588880+0000", 2024-02-05T08:14:32.452 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 93.814823059000005, 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15328541100000001, 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 131938 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588880+0000", 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.453 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588880+0000", 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588882+0000", 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6500000000000001e-06 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588888+0000", 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0839999999999997e-06 2024-02-05T08:14:32.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.455 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.455 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588902+0000", 2024-02-05T08:14:32.455 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4208e-05 2024-02-05T08:14:32.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.455 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.455 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588912+0000", 2024-02-05T08:14:32.455 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7839999999999998e-06 2024-02-05T08:14:32.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.455 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.456 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.592812+0000", 2024-02-05T08:14:32.456 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003899881 2024-02-05T08:14:32.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.456 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.456 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.592939+0000", 2024-02-05T08:14:32.456 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012735700000000001 2024-02-05T08:14:32.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.456 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:14:32.456 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.593065+0000", 2024-02-05T08:14:32.456 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000125659 2024-02-05T08:14:32.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.599122+0000", 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0060574239999999996 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600149+0000", 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001027095 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600698+0000", 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00054859500000000003 2024-02-05T08:14:32.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.741935+0000", 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.141237063 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.742052+0000", 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000117544 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.742165+0000", 2024-02-05T08:14:32.458 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011306699999999999 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:131939 8.f 8:fbb4e894:::benchmark_data_smithi134_115836_object15279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:58.588930+0000", 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 93.814772848000004, 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15448906300000001, 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:14:32.459 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 131939 2024-02-05T08:14:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588930+0000", 2024-02-05T08:14:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588930+0000", 2024-02-05T08:14:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588931+0000", 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0800000000000005e-07 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588933+0000", 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5749999999999999e-06 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588941+0000", 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9489999999999996e-06 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588947+0000", 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8749999999999997e-06 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596264+0000", 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0073165060000000004 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596396+0000", 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013257000000000001 2024-02-05T08:14:32.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596529+0000", 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013312400000000001 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600168+0000", 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036390340000000002 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600367+0000", 2024-02-05T08:14:32.463 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019905500000000001 2024-02-05T08:14:32.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.464 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.464 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600806+0000", 2024-02-05T08:14:32.464 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00043826 2024-02-05T08:14:32.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.464 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.464 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.743237+0000", 2024-02-05T08:14:32.464 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14243134299999999 2024-02-05T08:14:32.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.743303+0000", 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6328999999999995e-05 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.743419+0000", 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011593499999999999 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDPGPushReply(8.f 198/194 [PushReplyOp(8:f4899214:::benchmark_data_smithi134_115836_object113206:head)])", 2024-02-05T08:14:32.465 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:59.890203+0000", 2024-02-05T08:14:32.466 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 92.513500538000002, 2024-02-05T08:14:32.466 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14734050700000001, 2024-02-05T08:14:32.466 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.466 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:14:32.466 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.466 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.466 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890203+0000", 2024-02-05T08:14:32.466 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890203+0000", 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890203+0000", 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9400000000000001e-07 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890205+0000", 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.627e-06 2024-02-05T08:14:32.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890211+0000", 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5439999999999998e-06 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890216+0000", 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1460000000000004e-06 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890262+0000", 2024-02-05T08:14:32.468 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5782999999999999e-05 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:00.037543+0000", 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14728111299999999 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:134926 8.f 8:f72340dc:::benchmark_data_smithi134_115836_object18266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.535476+0000", 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 89.868227494999999, 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16460388500000001, 2024-02-05T08:14:32.469 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 134926 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535476+0000", 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.470 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535476+0000", 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535477+0000", 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8160000000000001e-06 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535482+0000", 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9400000000000001e-06 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.471 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535494+0000", 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1657999999999999e-05 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535503+0000", 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7129999999999994e-06 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535674+0000", 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017146300000000001 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.472 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535806+0000", 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000131707 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535947+0000", 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014141300000000001 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.539323+0000", 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033756630000000001 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.539337+0000", 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4443e-05 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.540469+0000", 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011320169999999999 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.699796+0000", 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15932692800000001 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.474 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.699972+0000", 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000175513 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.700079+0000", 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107611 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15087.0:134935 8.d e199/157 8:bc5808d6:::benchmark_data_smithi134_115836_object18275:head v 199'8633, mlcod=199'8631)", 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.544759+0000", 2024-02-05T08:14:32.475 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 89.858943718999996, 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15565003399999999, 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.544759+0000", 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.544759+0000", 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.476 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.477 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.478 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.544760+0000", 2024-02-05T08:14:32.478 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.06e-07 2024-02-05T08:14:32.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.479 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.479 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.544764+0000", 2024-02-05T08:14:32.479 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1629999999999998e-06 2024-02-05T08:14:32.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.479 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.479 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.694991+0000", 2024-02-05T08:14:32.479 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15022724000000001 2024-02-05T08:14:32.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.479 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.694999+0000", 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2339999999999994e-06 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.695129+0000", 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012998799999999999 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.695151+0000", 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1145000000000002e-05 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.480 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.700293+0000", 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0051418979999999998 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.700409+0000", 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011686 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:161420 8.f 8:f12e680b:::benchmark_data_smithi134_115836_object44760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:35.294162+0000", 2024-02-05T08:14:32.481 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 57.109540656999997, 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14680094699999999, 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 161420 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294162+0000", 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294162+0000", 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294163+0000", 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8899999999999999e-07 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294167+0000", 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8290000000000001e-06 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.483 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294181+0000", 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3961000000000001e-05 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294188+0000", 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.939e-06 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.297429+0000", 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032408910000000001 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.297540+0000", 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000110961 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.297646+0000", 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010610100000000001 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.300140+0000", 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0024944670000000002 2024-02-05T08:14:32.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.301421+0000", 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001281157 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.440869+0000", 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.139447667 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.440927+0000", 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7918999999999999e-05 2024-02-05T08:14:32.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.440963+0000", 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6365999999999997e-05 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:162662 8.f 8:f41fb899:::benchmark_data_smithi134_115836_object46002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:36.809228+0000", 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 55.594474857000002, 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16428367499999999, 2024-02-05T08:14:32.487 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.488 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.488 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.488 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:14:32.488 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:14:32.488 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 162662 2024-02-05T08:14:32.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.488 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.488 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809228+0000", 2024-02-05T08:14:32.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809228+0000", 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809230+0000", 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.488e-06 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809239+0000", 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2180000000000005e-06 2024-02-05T08:14:32.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809248+0000", 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4040000000000001e-06 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809255+0000", 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5119999999999997e-06 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809371+0000", 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115905 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809483+0000", 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000112136 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809632+0000", 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000149285 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.815527+0000", 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0058951819999999997 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.819817+0000", 2024-02-05T08:14:32.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042893899999999997 2024-02-05T08:14:32.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973380+0000", 2024-02-05T08:14:32.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15356336900000001 2024-02-05T08:14:32.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973474+0000", 2024-02-05T08:14:32.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3459000000000001e-05 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973512+0000", 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8327000000000001e-05 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:162673 8.f 8:f0c11af9:::benchmark_data_smithi134_115836_object46013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:36.816303+0000", 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 55.587400438000003, 2024-02-05T08:14:32.493 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.157358891, 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 162673 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816303+0000", 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816303+0000", 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816304+0000", 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2449999999999999e-06 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816308+0000", 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2679999999999996e-06 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816318+0000", 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4779999999999997e-06 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816324+0000", 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.725e-06 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816371+0000", 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6983000000000001e-05 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816476+0000", 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104721 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816581+0000", 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010494500000000001 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.823858+0000", 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0072768090000000004 2024-02-05T08:14:32.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.823889+0000", 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1165999999999999e-05 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973555+0000", 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14966643499999999 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973629+0000", 2024-02-05T08:14:32.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3936000000000005e-05 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973662+0000", 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.218e-05 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:162674 8.f 8:f8e2bae8:::benchmark_data_smithi134_115836_object46014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:36.818247+0000", 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 55.585456383, 2024-02-05T08:14:32.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.155548574, 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 162674 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818247+0000", 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818247+0000", 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818248+0000", 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.113e-06 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818251+0000", 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.647e-06 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818262+0000", 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0519e-05 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818269+0000", 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3679999999999996e-06 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.819987+0000", 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017176610000000001 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.820094+0000", 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000106597 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.820209+0000", 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114962 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.823966+0000", 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0037573390000000002 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.825554+0000", 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001587647 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973698+0000", 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14814437499999999 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973765+0000", 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7377000000000002e-05 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973795+0000", 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9969000000000001e-05 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:197385 8.f 8:f14f197d:::benchmark_data_smithi134_115836_object80725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:14:07.714003+0000", 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 24.689699938, 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14987962099999999, 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.505 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 197385 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714003+0000", 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714003+0000", 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714004+0000", 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.173e-06 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714008+0000", 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8130000000000002e-06 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714019+0000", 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1114999999999999e-05 2024-02-05T08:14:32.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714026+0000", 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2690000000000002e-06 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.717045+0000", 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030182009999999999 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.717183+0000", 2024-02-05T08:14:32.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013826499999999999 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.717300+0000", 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000116941 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.724371+0000", 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0070706450000000004 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.726733+0000", 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023622299999999999 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.863757+0000", 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13702392899999999 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.863841+0000", 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4518999999999995e-05 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.863883+0000", 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1520999999999998e-05 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:197386 8.f 8:ffb8808e:::benchmark_data_smithi134_115836_object80726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:14:07.715023+0000", 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 24.688680375000001, 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15136122299999999, 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.511 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 197386 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715023+0000", 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715023+0000", 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715023+0000", 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6700000000000003e-07 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715027+0000", 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7100000000000001e-06 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715038+0000", 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1262e-05 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715046+0000", 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7919999999999993e-06 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.720250+0000", 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0052037560000000004 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.721564+0000", 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001314052 2024-02-05T08:14:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.721676+0000", 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000112118 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.726778+0000", 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0051023309999999999 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.728765+0000", 2024-02-05T08:14:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0019865550000000001 2024-02-05T08:14:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866283+0000", 2024-02-05T08:14:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.137518106 2024-02-05T08:14:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866340+0000", 2024-02-05T08:14:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7144e-05 2024-02-05T08:14:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866384+0000", 2024-02-05T08:14:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3630000000000001e-05 2024-02-05T08:14:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:197388 8.f 8:fbe8cc34:::benchmark_data_smithi134_115836_object80728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-02-05T08:14:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:14:07.718059+0000", 2024-02-05T08:14:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 24.685644138000001, 2024-02-05T08:14:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.148440035, 2024-02-05T08:14:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:14:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:14:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:14:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:14:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 197388 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.718059+0000", 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.718059+0000", 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.718060+0000", 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1739999999999999e-06 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.718071+0000", 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1243999999999999e-05 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.718081+0000", 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.251e-06 2024-02-05T08:14:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.718087+0000", 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7209999999999997e-06 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.724408+0000", 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0063203469999999996 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.724521+0000", 2024-02-05T08:14:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000112916 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.724634+0000", 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000113113 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.728684+0000", 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0040504169999999997 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.730701+0000", 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0020171120000000002 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866426+0000", 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13572458000000001 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866469+0000", 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2790000000000002e-05 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:14:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866499+0000", 2024-02-05T08:14:32.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0369999999999999e-05 2024-02-05T08:14:32.526 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.526 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.526 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.526 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:14:32.526 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:14:32.526 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:14:32.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:32.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:32.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:32.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:32.662 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:32.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:32.665 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:32.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:32.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:32.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:32.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:32.903 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:32.940 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:32.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:33.042 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:33.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:33.081 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:33.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:33.181 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:33.217 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:33.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:33 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:14:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:33 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:14:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:33 smithi196 ceph-mon[33018]: pgmap v426: 148 pgs: 148 active+clean; 618 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 18 B/s, 1 objects/s recovering 2024-02-05T08:14:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:33 smithi196 ceph-mon[33018]: 6.b scrub starts 2024-02-05T08:14:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:33 smithi196 ceph-mon[33018]: 6.b scrub ok 2024-02-05T08:14:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:33 smithi196 ceph-mon[33018]: 4.1c deep-scrub starts 2024-02-05T08:14:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:33 smithi196 ceph-mon[33018]: 4.1c deep-scrub ok 2024-02-05T08:14:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:33 smithi196 ceph-mon[33018]: 5.1e deep-scrub starts 2024-02-05T08:14:33.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:33 smithi196 ceph-mon[33018]: 5.1e deep-scrub ok 2024-02-05T08:14:33.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:33.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:33.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:33.407 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:14:33.407 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:14:33.408 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[30086]: pgmap v426: 148 pgs: 148 active+clean; 618 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 18 B/s, 1 objects/s recovering 2024-02-05T08:14:33.408 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[30086]: 6.b scrub starts 2024-02-05T08:14:33.408 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[30086]: 6.b scrub ok 2024-02-05T08:14:33.408 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[30086]: 4.1c deep-scrub starts 2024-02-05T08:14:33.408 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[30086]: 4.1c deep-scrub ok 2024-02-05T08:14:33.408 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[30086]: 5.1e deep-scrub starts 2024-02-05T08:14:33.408 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[30086]: 5.1e deep-scrub ok 2024-02-05T08:14:33.408 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:14:33.408 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:14:33.408 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[39278]: pgmap v426: 148 pgs: 148 active+clean; 618 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 18 B/s, 1 objects/s recovering 2024-02-05T08:14:33.408 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[39278]: 6.b scrub starts 2024-02-05T08:14:33.409 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[39278]: 6.b scrub ok 2024-02-05T08:14:33.409 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[39278]: 4.1c deep-scrub starts 2024-02-05T08:14:33.409 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[39278]: 4.1c deep-scrub ok 2024-02-05T08:14:33.409 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[39278]: 5.1e deep-scrub starts 2024-02-05T08:14:33.409 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:33 smithi134 ceph-mon[39278]: 5.1e deep-scrub ok 2024-02-05T08:14:33.458 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:33.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:33.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:33.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:33.634 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:33.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:33.735 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:33.775 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:33.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:33.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:14:33.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:14:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:14:33.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:33.927 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:33.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:34.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:34.061 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:34.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:34.129 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:34 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:34.129 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:34 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:34.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:34.204 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:34.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:34.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:34.305 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:34.339 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:34.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:34.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:34.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:34.475 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:34.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:34.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:34 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:34.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:34.620 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:34.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:34.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:34.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:34.761 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:34.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:34.863 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:34.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:34.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:34.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:34.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:35.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:35.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:35.132 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:35.176 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:35.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:35.278 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:35.310 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:35.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:35.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:35.371 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Removed 116657 objects 2024-02-05T08:14:35.371 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Clean up completed and total clean up time :115.896 2024-02-05T08:14:35.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:35.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:35.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:35.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[30086]: 6.1f scrub starts 2024-02-05T08:14:35.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[30086]: 6.1f scrub ok 2024-02-05T08:14:35.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[30086]: pgmap v427: 148 pgs: 148 active+clean; 189 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.78k op/s; 14 B/s, 1 objects/s recovering 2024-02-05T08:14:35.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[30086]: 5.b scrub starts 2024-02-05T08:14:35.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[30086]: 5.b scrub ok 2024-02-05T08:14:35.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[30086]: 4.d scrub starts 2024-02-05T08:14:35.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[30086]: 4.d scrub ok 2024-02-05T08:14:35.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[30086]: osdmap e231: 8 total, 8 up, 7 in 2024-02-05T08:14:35.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2107431472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:14:35.494 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[39278]: 6.1f scrub starts 2024-02-05T08:14:35.494 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[39278]: 6.1f scrub ok 2024-02-05T08:14:35.494 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[39278]: pgmap v427: 148 pgs: 148 active+clean; 189 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.78k op/s; 14 B/s, 1 objects/s recovering 2024-02-05T08:14:35.494 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[39278]: 5.b scrub starts 2024-02-05T08:14:35.495 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[39278]: 5.b scrub ok 2024-02-05T08:14:35.495 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[39278]: 4.d scrub starts 2024-02-05T08:14:35.495 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[39278]: 4.d scrub ok 2024-02-05T08:14:35.495 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[39278]: osdmap e231: 8 total, 8 up, 7 in 2024-02-05T08:14:35.495 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:35 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2107431472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:14:35.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T08:14:35.533 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:35.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:35 smithi196 ceph-mon[33018]: 6.1f scrub starts 2024-02-05T08:14:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:35 smithi196 ceph-mon[33018]: 6.1f scrub ok 2024-02-05T08:14:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:35 smithi196 ceph-mon[33018]: pgmap v427: 148 pgs: 148 active+clean; 189 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.78k op/s; 14 B/s, 1 objects/s recovering 2024-02-05T08:14:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:35 smithi196 ceph-mon[33018]: 5.b scrub starts 2024-02-05T08:14:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:35 smithi196 ceph-mon[33018]: 5.b scrub ok 2024-02-05T08:14:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:35 smithi196 ceph-mon[33018]: 4.d scrub starts 2024-02-05T08:14:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:35 smithi196 ceph-mon[33018]: 4.d scrub ok 2024-02-05T08:14:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:35 smithi196 ceph-mon[33018]: osdmap e231: 8 total, 8 up, 7 in 2024-02-05T08:14:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:35 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2107431472' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:14:35.564 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:35.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:35.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:35.666 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:35.691 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:35.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:35.732 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:14:35.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:35.816 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:35.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:35.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:35.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:35.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:35.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:36.044 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:36.068 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:36.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:36.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:36.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:36 smithi134 ceph-mon[30086]: 4.9 scrub starts 2024-02-05T08:14:36.128 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:36 smithi134 ceph-mon[30086]: 4.9 scrub ok 2024-02-05T08:14:36.128 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:36 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:14:36.128 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2107431472' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:14:36.128 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:36 smithi134 ceph-mon[30086]: osdmap e232: 8 total, 8 up, 7 in 2024-02-05T08:14:36.128 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:36 smithi134 ceph-mon[39278]: 4.9 scrub starts 2024-02-05T08:14:36.128 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:36 smithi134 ceph-mon[39278]: 4.9 scrub ok 2024-02-05T08:14:36.128 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:36 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:14:36.128 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2107431472' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:14:36.128 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:36 smithi134 ceph-mon[39278]: osdmap e232: 8 total, 8 up, 7 in 2024-02-05T08:14:36.170 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:36.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:36.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:36.297 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:36.321 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:36.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:36.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:36.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:36.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:36.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:36.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:36.549 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:36 smithi196 ceph-mon[33018]: 4.9 scrub starts 2024-02-05T08:14:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:36 smithi196 ceph-mon[33018]: 4.9 scrub ok 2024-02-05T08:14:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:36 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:14:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2107431472' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:14:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:36 smithi196 ceph-mon[33018]: osdmap e232: 8 total, 8 up, 7 in 2024-02-05T08:14:36.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:36.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:36.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:36.701 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:36.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:36.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:36.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:36.830 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:36.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:36.931 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:36.955 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:36.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:36.986 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-02-05T08:14:36.987 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-02-05T08:14:37.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:37.085 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:37.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:37.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:37.187 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:37.212 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:37.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:37.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:37.313 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:37.337 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:37.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:37.439 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:37.444 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:37 smithi134 ceph-mon[30086]: 4.15 scrub starts 2024-02-05T08:14:37.445 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:37 smithi134 ceph-mon[30086]: 4.15 scrub ok 2024-02-05T08:14:37.445 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:37 smithi134 ceph-mon[30086]: pgmap v429: 148 pgs: 148 active+clean; 189 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T08:14:37.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:37 smithi134 ceph-mon[39278]: 4.15 scrub starts 2024-02-05T08:14:37.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:37 smithi134 ceph-mon[39278]: 4.15 scrub ok 2024-02-05T08:14:37.445 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:37 smithi134 ceph-mon[39278]: pgmap v429: 148 pgs: 148 active+clean; 189 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T08:14:37.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:37.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:37 smithi196 ceph-mon[33018]: 4.15 scrub starts 2024-02-05T08:14:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:37 smithi196 ceph-mon[33018]: 4.15 scrub ok 2024-02-05T08:14:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:37 smithi196 ceph-mon[33018]: pgmap v429: 148 pgs: 148 active+clean; 189 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T08:14:37.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:37.593 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:37.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:37.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:37.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:37.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:37.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:37.820 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:37.822 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:37.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:37.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:37.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:37.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:37.971 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:37.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:38.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:14:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:14:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:14:38.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:38.089 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.2 to 1 (10000) 2024-02-05T08:14:38.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:38.100 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:38.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:38.201 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:38.224 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:38.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:38.326 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:38.349 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:38.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:38.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:38.451 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:38 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4153772521' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:14:38.451 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:38 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:14:38.451 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:14:38.451 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:14:38.451 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:38 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4153772521' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:14:38.451 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:38 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:14:38.451 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:14:38.451 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:14:38.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:38.457 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:38.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:38.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:38.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:38 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4153772521' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:14:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:38 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:14:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:14:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:14:38.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:38.604 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:38.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:38.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:38.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:38.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:38.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:38.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:14:38.835 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:38.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:38.879 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:38.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:38.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:39.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:39.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:39.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:39.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:39.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:39.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:39.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:39.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:39.259 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:39.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:39.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:39.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:39 smithi134 ceph-mon[30086]: pgmap v431: 148 pgs: 148 active+clean; 132 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-02-05T08:14:39.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:39 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:14:39.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:39 smithi134 ceph-mon[30086]: osdmap e233: 8 total, 8 up, 7 in 2024-02-05T08:14:39.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:39 smithi134 ceph-mon[39278]: pgmap v431: 148 pgs: 148 active+clean; 132 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-02-05T08:14:39.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:39 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:14:39.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:39 smithi134 ceph-mon[39278]: osdmap e233: 8 total, 8 up, 7 in 2024-02-05T08:14:39.361 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:39.384 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:39.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:39.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:39.432 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 4 1 2024-02-05T08:14:39.487 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:39.511 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:39.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:39 smithi196 ceph-mon[33018]: pgmap v431: 148 pgs: 148 active+clean; 132 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-02-05T08:14:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:39 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:14:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:39 smithi196 ceph-mon[33018]: osdmap e233: 8 total, 8 up, 7 in 2024-02-05T08:14:39.612 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:39.640 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:39.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:39.681 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:14:39.700 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:14:39.725 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:14:39.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:39.745 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:14:39.766 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:39.766 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:14:39.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:39.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:39.786 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:14:39.806 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:14:39.827 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:14:39.829 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:14:39.829 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:14:39.829 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:14:39.829 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:14:39.829 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:14:39.829 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:14:39.829 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:14:39.829 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:14:39.868 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:39.894 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:39.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:39.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:39.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:40.021 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:40.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:40.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:40.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:40.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:40.249 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:40.273 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:40.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:40.375 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:40.400 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:40.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:40.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:40.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:40 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:14:40.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:40 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:14:40.500 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:40 smithi134 ceph-mon[30086]: osdmap e234: 8 total, 8 up, 7 in 2024-02-05T08:14:40.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:40 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3151601356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:40.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:40 smithi134 ceph-mon[30086]: osdmap e235: 8 total, 8 up, 7 in 2024-02-05T08:14:40.501 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:40 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:14:40.501 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:40 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:14:40.501 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:40 smithi134 ceph-mon[39278]: osdmap e234: 8 total, 8 up, 7 in 2024-02-05T08:14:40.501 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:40 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3151601356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:40.501 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:40 smithi134 ceph-mon[39278]: osdmap e235: 8 total, 8 up, 7 in 2024-02-05T08:14:40.501 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:40.525 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:40.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:40.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:40 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:14:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:40 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:14:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:40 smithi196 ceph-mon[33018]: osdmap e234: 8 total, 8 up, 7 in 2024-02-05T08:14:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:40 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3151601356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:40 smithi196 ceph-mon[33018]: osdmap e235: 8 total, 8 up, 7 in 2024-02-05T08:14:40.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:40.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:40.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:40.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:40.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:40.785 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:40.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:40.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:40.912 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:40.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:40.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:41.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:41.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:41.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:41.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:41.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:41.167 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:41.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:41.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:41.218 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:41 smithi134 ceph-mon[30086]: pgmap v434: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 606 op/s 2024-02-05T08:14:41.219 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:41 smithi134 ceph-mon[39278]: pgmap v434: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 606 op/s 2024-02-05T08:14:41.269 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:41.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:41.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:41.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:41.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:41.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:41.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:41.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:41.555 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:41.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:41.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:41 smithi196 ceph-mon[33018]: pgmap v434: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 606 op/s 2024-02-05T08:14:41.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:41.686 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:41.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:41.788 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:41.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:41.814 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:41.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:41.816 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:41.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:41.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:41.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:41.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:42.045 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:42.071 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:42.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:42.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:42.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:42.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:42.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:42.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:42.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:42.331 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:42.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:42.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:42.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:42.460 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:42.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:42.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:42.562 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:42.586 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:42.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:42.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:42.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:42.712 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:42.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:42.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:42.813 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:42.841 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:42.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:42.943 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:42.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:14:42.966 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:42.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:42.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:43.069 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:43.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:43.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:43.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:43.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:43.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:43.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:43.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:43.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:43 smithi134 ceph-mon[30086]: pgmap v436: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 857 B/s rd, 0 B/s wr, 506 op/s 2024-02-05T08:14:43.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:43 smithi134 ceph-mon[39278]: pgmap v436: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 857 B/s rd, 0 B/s wr, 506 op/s 2024-02-05T08:14:43.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:43.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:43.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:43.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:43.365 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:43 smithi196 ceph-mon[33018]: pgmap v436: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 857 B/s rd, 0 B/s wr, 506 op/s 2024-02-05T08:14:43.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:43.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:43.485 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:43.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:43.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:43.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:43.615 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:43.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:43.717 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:43.741 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:43.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:43.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:14:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:14:43.842 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:43.866 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:43.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:43.967 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:43.991 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:43.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:43.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:44.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:44.120 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:44.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:44.222 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:44.245 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:44.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:44.271 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:44 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:44.271 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:44 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:44.347 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:44.370 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:44.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:44.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:44.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:44.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:44.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:44.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:44.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:44 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:44.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:44.630 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:44.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:44.732 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:44.756 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:44.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:44.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:44.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:44.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:44.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:44.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:44.988 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:45.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:45.014 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:45.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:45.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:45.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:45.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:45.215 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'unique_pool_1' removed 2024-02-05T08:14:45.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:45.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:45.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:45.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:45.401 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:45.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:45.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:45.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:45.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:45 smithi134 ceph-mon[30086]: pgmap v437: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 351 op/s 2024-02-05T08:14:45.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:45 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2549576462' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:14:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:45 smithi134 ceph-mon[39278]: pgmap v437: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 351 op/s 2024-02-05T08:14:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:45 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2549576462' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:14:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:45.528 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:45.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:45 smithi196 ceph-mon[33018]: pgmap v437: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 351 op/s 2024-02-05T08:14:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:45 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2549576462' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:14:45.629 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:45.654 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:45.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:45.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:45.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:45.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:45.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:45.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:45.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:45.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:45.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:46.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:46.031 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:46.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:46.133 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:46.147 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T08:14:46.147 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T08:14:46.148 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-02-05T08:14:46.148 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-02-05T08:14:46.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:46.158 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:46.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:46.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:46 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2549576462' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:14:46.226 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:46 smithi134 ceph-mon[30086]: osdmap e236: 8 total, 8 up, 7 in 2024-02-05T08:14:46.226 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:46 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2549576462' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:14:46.226 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:46 smithi134 ceph-mon[39278]: osdmap e236: 8 total, 8 up, 7 in 2024-02-05T08:14:46.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:46.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:46.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:46.357 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:14:46.357 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:14:46.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:46.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:46.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:46.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:46.533 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:46.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:46 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2549576462' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:14:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:46 smithi196 ceph-mon[33018]: osdmap e236: 8 total, 8 up, 7 in 2024-02-05T08:14:46.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:46.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:46.659 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:46.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:46.760 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:46.784 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:46.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:46.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:46.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:46.910 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:46.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:46.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:47.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:47.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:47.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:47.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:47.160 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:47.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:47.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:47.286 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:47.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:47.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:47.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:47.407 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:47 smithi134 ceph-mon[30086]: pgmap v439: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 619 B/s rd, 0 op/s 2024-02-05T08:14:47.407 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:47 smithi134 ceph-mon[39278]: pgmap v439: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 619 B/s rd, 0 op/s 2024-02-05T08:14:47.412 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:47.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:47.514 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:47.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:47.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:47 smithi196 ceph-mon[33018]: pgmap v439: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 619 B/s rd, 0 op/s 2024-02-05T08:14:47.640 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:47.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:47.664 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:47.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:47.765 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:47.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:47.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:47.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:47.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:47.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:47.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:47.974 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:14:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:14:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:14:48.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:48.040 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:48.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:48.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:48.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:48.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:48.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:48.290 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:48.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:48.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:48.416 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:48.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:48.517 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:48.542 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:48.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:48.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:48.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:48.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:48.769 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:48.795 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:48.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:48.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:48.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:48.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:48.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:49.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:49.047 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:49.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:49.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:49.173 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:49.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:49.275 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:49.298 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:49.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:49.400 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:49.414 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:49 smithi134 ceph-mon[30086]: pgmap v440: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:14:49.414 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:49 smithi134 ceph-mon[39278]: pgmap v440: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:14:49.424 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:49.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:49.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:49.549 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:49.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:49.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:49.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:49 smithi196 ceph-mon[33018]: pgmap v440: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:14:49.651 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:49.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T08:14:49.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:49.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:49.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:49.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:49.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:49.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:49.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:49.903 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:49.927 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:49.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:49.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:50.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:50.053 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:50.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:50.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:50.155 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:50.179 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:50.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:50.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:50.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:50.305 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:50.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:50.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:50.406 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:50.430 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:50.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:50.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:50.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:50.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:50.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:50.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:50.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:50.680 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:50.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:50.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:50.726 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.4 to 1 (10000) 2024-02-05T08:14:50.782 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:50.805 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:50.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:50.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:50.907 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:50.930 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:50.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:50.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:51.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:50 smithi134 ceph-mon[30086]: pgmap v441: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 922 B/s rd, 0 op/s 2024-02-05T08:14:51.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:50 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/747097212' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:14:51.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:50 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:14:51.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:50 smithi134 ceph-mon[39278]: pgmap v441: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 922 B/s rd, 0 op/s 2024-02-05T08:14:51.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:50 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/747097212' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:14:51.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:50 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:14:51.031 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:51.055 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:51.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:50 smithi196 ceph-mon[33018]: pgmap v441: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 922 B/s rd, 0 op/s 2024-02-05T08:14:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:50 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/747097212' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:14:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:50 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:14:51.156 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:51.179 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:51.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:51.220 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:51.282 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:51.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:51.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:51.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:51.410 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:51.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:51.435 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:51.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:51.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:51.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:51.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:51.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:51.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:51.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:51.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:51.725 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:51 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:14:51.725 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:51 smithi134 ceph-mon[39278]: osdmap e237: 8 total, 8 up, 7 in 2024-02-05T08:14:51.732 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:14:51.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:51.824 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:51.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:51.925 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:51.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:51.950 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:51.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:52.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:51 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2102553637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:14:52.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:51 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:14:52.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:51 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:14:52.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:51 smithi134 ceph-mon[30086]: osdmap e237: 8 total, 8 up, 7 in 2024-02-05T08:14:52.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:51 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2102553637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:14:52.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:51 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:14:52.051 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:52.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:51 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:14:52.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:51 smithi196 ceph-mon[33018]: osdmap e237: 8 total, 8 up, 7 in 2024-02-05T08:14:52.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:51 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2102553637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:14:52.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:51 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:14:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:52.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:52.181 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:52.207 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:52.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:52.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:52.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:52.337 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:52.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:52.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:52.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:52.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:52.570 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:52.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:52.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:52.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:52.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:52.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:52.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:52.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:52.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:52.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:52.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:52.965 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:52 smithi134 ceph-mon[30086]: pgmap v443: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T08:14:52.966 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:52 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:14:52.966 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:52 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:14:52.966 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:52 smithi134 ceph-mon[30086]: osdmap e238: 8 total, 8 up, 7 in 2024-02-05T08:14:52.966 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:52 smithi134 ceph-mon[39278]: pgmap v443: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T08:14:52.966 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:52 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:14:52.966 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:52 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:14:52.966 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:52 smithi134 ceph-mon[39278]: osdmap e238: 8 total, 8 up, 7 in 2024-02-05T08:14:52.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:52.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:52.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:52 smithi196 ceph-mon[33018]: pgmap v443: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T08:14:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:52 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:14:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:52 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:14:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:52 smithi196 ceph-mon[33018]: osdmap e238: 8 total, 8 up, 7 in 2024-02-05T08:14:53.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:53.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:53.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:53.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:53.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:53.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:53.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:53.326 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:53.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:53.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:53.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:53.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:53.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:53.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:53.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:53.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:53.625 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:53.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:53.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:53.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:53.730 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:14:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:14:53.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:53.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:53.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:53.776 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 5 1 2024-02-05T08:14:53.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:53.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:53.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:53.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:53.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:53.991 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:14:53.991 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:53 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:53.991 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:14:53.991 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:14:53.991 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:53 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:53.991 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:14:54.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:54.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:54.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:54.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:14:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:53 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:14:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:14:54.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:54.125 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:54.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:54.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:54.227 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:54.253 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:54.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:54.354 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:54.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:54.379 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:54.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:54.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:54.503 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:54.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:54.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:54.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:54.629 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:54.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:54.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:54.757 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:54.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:54.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:54.858 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:54.883 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:54.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:54.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:54.904 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:54 smithi134 ceph-mon[30086]: pgmap v445: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-02-05T08:14:54.904 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:14:54.904 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:14:54.905 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:14:54.905 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:54 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4174771586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:54.905 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:54 smithi134 ceph-mon[39278]: pgmap v445: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-02-05T08:14:54.905 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:14:54.905 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:14:54.905 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:14:54.905 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:54 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4174771586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:54.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:55.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:55.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:54 smithi196 ceph-mon[33018]: pgmap v445: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 55 KiB/s rd, 0 B/s wr, 90 op/s 2024-02-05T08:14:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:14:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:14:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:14:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:54 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4174771586' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:14:55.112 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:55.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:55.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:55.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:55.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:55.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:55.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:55.299 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:14:55.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:55.386 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:55.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:55.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:55.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:55.513 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:55.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:55.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:55.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:55.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:55.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:55.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:55.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:55.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:55.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:55.890 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:55.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:55.992 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:56.017 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:56.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:56.033 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:14:56.033 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:14:56.034 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:14:56.118 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:56.141 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:56.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:56.226 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:56.243 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:56.271 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:56.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:56.372 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:56.396 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:56.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:56.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:56.498 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:56.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:56.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:56.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:56.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:56.651 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:56.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:56.747 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'unique_pool_2' created 2024-02-05T08:14:56.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:56.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:56.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:56.881 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:56.905 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:56.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:56.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:57.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:57.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:56 smithi134 ceph-mon[30086]: pgmap v446: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-02-05T08:14:57.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:56 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1531943978' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-05T08:14:57.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:56 smithi134 ceph-mon[39278]: pgmap v446: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-02-05T08:14:57.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:56 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1531943978' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-05T08:14:57.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:57.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:57.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:57.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:56 smithi196 ceph-mon[33018]: pgmap v446: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s 2024-02-05T08:14:57.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:56 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1531943978' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-05T08:14:57.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:57.163 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:57.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:57.264 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:57.292 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:57.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:57.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:57.418 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:57.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:57.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:57.420 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-02-05T08:14:57.497 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:57.505 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:14:57.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:57.546 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:57.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:57.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:57.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:57.673 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:57.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:57.774 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:57.799 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:57.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:57.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:57.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:57.924 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:57.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:58.006 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:57 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1531943978' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-05T08:14:58.007 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:57 smithi134 ceph-mon[30086]: osdmap e239: 8 total, 8 up, 7 in 2024-02-05T08:14:58.007 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:14:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:14:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:14:58.007 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:57 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1531943978' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-05T08:14:58.007 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:57 smithi134 ceph-mon[39278]: osdmap e239: 8 total, 8 up, 7 in 2024-02-05T08:14:58.025 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:58.049 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:58.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:57 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1531943978' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-05T08:14:58.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:57 smithi196 ceph-mon[33018]: osdmap e239: 8 total, 8 up, 7 in 2024-02-05T08:14:58.150 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:58.176 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:58.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:58.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:58.277 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:58.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:58.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:58.404 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:58.429 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:58.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:58.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:58.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:58.554 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:58.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:58.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:58.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:14:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:58.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:58.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:14:58.739 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:14:58.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:58.806 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:58.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:58.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:58.907 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:58.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:58.931 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:58.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:59.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:58 smithi134 ceph-mon[30086]: pgmap v448: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-02-05T08:14:59.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:58 smithi134 ceph-mon[30086]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:14:59.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:58 smithi134 ceph-mon[30086]: osdmap e240: 8 total, 8 up, 7 in 2024-02-05T08:14:59.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:58 smithi134 ceph-mon[39278]: pgmap v448: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-02-05T08:14:59.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:58 smithi134 ceph-mon[39278]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:14:59.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:58 smithi134 ceph-mon[39278]: osdmap e240: 8 total, 8 up, 7 in 2024-02-05T08:14:59.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:14:59.060 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:59.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:14:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:58 smithi196 ceph-mon[33018]: pgmap v448: 148 pgs: 5 creating+peering, 11 unknown, 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 63 KiB/s rd, 0 B/s wr, 103 op/s 2024-02-05T08:14:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:58 smithi196 ceph-mon[33018]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:14:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:58 smithi196 ceph-mon[33018]: osdmap e240: 8 total, 8 up, 7 in 2024-02-05T08:14:59.162 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:59.188 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:59.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:14:59.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:14:59.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:59.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:14:59.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:14:59.446 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:59.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:59.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:14:59.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:14:59.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:14:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:59.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:14:59.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:14:59.701 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:59.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:14:59.803 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:14:59.826 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:59.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:59.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:14:59.928 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:14:59.951 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:14:59.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:14:59.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:00.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:14:59 smithi134 ceph-mon[30086]: osdmap e241: 8 total, 8 up, 7 in 2024-02-05T08:15:00.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:14:59 smithi134 ceph-mon[39278]: osdmap e241: 8 total, 8 up, 7 in 2024-02-05T08:15:00.053 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:00.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:14:59 smithi196 ceph-mon[33018]: osdmap e241: 8 total, 8 up, 7 in 2024-02-05T08:15:00.076 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:00.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:00.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:00.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:00.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:00.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:00.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:00.334 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:00.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:00.436 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:00.459 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:00.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:00.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:00.499 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:00.561 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:00.583 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:00.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:00.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:00.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:00.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:00.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:00.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:00.766 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.5 to 1 (10000) 2024-02-05T08:15:00.815 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:00.838 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:00.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:00.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:00.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:00.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:00.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:00.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:01.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:00 smithi134 ceph-mon[30086]: pgmap v451: 148 pgs: 5 creating+peering, 143 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:15:01.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:00 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/526839795' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:15:01.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:00 smithi134 ceph-mon[39278]: pgmap v451: 148 pgs: 5 creating+peering, 143 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:15:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:00 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/526839795' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:15:01.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:00 smithi196 ceph-mon[33018]: pgmap v451: 148 pgs: 5 creating+peering, 143 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:15:01.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:00 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/526839795' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:15:01.069 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:01.093 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:01.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:01.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:01.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:01.223 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:01.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:01.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:01.348 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:01.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:01.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:01.473 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:01.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:01.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:01.575 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:01.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:01.599 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:01.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:01.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:01.724 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:01.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:01.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:01.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:01.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:01.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:01.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:01.979 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:01.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:02.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:01 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/526839795' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:15:02.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:01 smithi134 ceph-mon[30086]: osdmap e242: 8 total, 8 up, 7 in 2024-02-05T08:15:02.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:01 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/526839795' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:15:02.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:01 smithi134 ceph-mon[39278]: osdmap e242: 8 total, 8 up, 7 in 2024-02-05T08:15:02.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:01 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/526839795' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:15:02.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:01 smithi196 ceph-mon[33018]: osdmap e242: 8 total, 8 up, 7 in 2024-02-05T08:15:02.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:02.109 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:02.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:02.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:02.170 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:02.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:02.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:02.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:02.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:02.340 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:02.364 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:02.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:02.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:02.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:02.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:02.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:02.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:02.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:02.619 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:02.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:02.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:02.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:02.750 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:02.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:02.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:02.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:02.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:02.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:02.919 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:15:02.919 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:15:02.919 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-02-05T08:15:02.920 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-02-05T08:15:02.920 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:15:02.920 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:02 smithi134 ceph-mon[30086]: pgmap v453: 148 pgs: 5 creating+peering, 143 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T08:15:02.920 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:02 smithi134 ceph-mon[39278]: pgmap v453: 148 pgs: 5 creating+peering, 143 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T08:15:02.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:03.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:03.007 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:03.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:03.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:02 smithi196 ceph-mon[33018]: pgmap v453: 148 pgs: 5 creating+peering, 143 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T08:15:03.108 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:03.132 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:03.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:03.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:03.256 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:03.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:03.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:03.347 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 7 1 2024-02-05T08:15:03.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:03.413 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:03.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:03.515 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:03.543 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:03.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:03.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:03.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:03.669 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:03.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:03.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:03.780 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:15:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:15:03.795 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:03.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:03.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:03.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:03.923 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:03 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:03.923 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:03 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:03.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:03.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:04.027 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:04.052 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:04.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:03 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:04.154 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:04.179 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:04.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:04.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:04.304 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:04.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:04.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:04.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:04.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:04.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:04.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:04.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:04.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:04.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:04.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:04.665 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:04.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:04.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:04.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:04.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:04.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:04.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:04.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:04.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:04.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:05.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:04 smithi134 ceph-mon[30086]: pgmap v454: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:15:05.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:04 smithi134 ceph-mon[39278]: pgmap v454: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:15:05.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:05.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:04 smithi196 ceph-mon[33018]: pgmap v454: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:15:05.081 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:05.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:05.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:05.183 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:05.208 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:05.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:05.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:05.290 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:05.310 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:05.333 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:05.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:05.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:05.463 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:05.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:05.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:05.591 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:05.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:05.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:05.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:05.714 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T08:15:05.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:05.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:05.798 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:15:05.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:05.847 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:05.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:05.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:05.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:05.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:05.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:06.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:05 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/531882831' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:15:06.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:05 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:15:06.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:05 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/531882831' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:15:06.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:05 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:15:06.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:05 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/531882831' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:15:06.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:05 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:15:06.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:06.102 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:06.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:06.204 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:06.227 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:06.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:06.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:06.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:06.355 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:06.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:06.458 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:06.483 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:06.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:06.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:06.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:06.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:06.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:06.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:06.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:06.735 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:06.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:06.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:06.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:06.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:06.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:06.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:06.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:06.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:06.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:06 smithi134 ceph-mon[30086]: pgmap v455: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 643 B/s rd, 0 op/s 2024-02-05T08:15:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:06 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:15:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:06 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:15:07.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:06 smithi134 ceph-mon[30086]: osdmap e243: 8 total, 8 up, 7 in 2024-02-05T08:15:07.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:06 smithi134 ceph-mon[39278]: pgmap v455: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 643 B/s rd, 0 op/s 2024-02-05T08:15:07.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:06 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:15:07.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:06 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:15:07.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:06 smithi134 ceph-mon[39278]: osdmap e243: 8 total, 8 up, 7 in 2024-02-05T08:15:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:06 smithi196 ceph-mon[33018]: pgmap v455: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 643 B/s rd, 0 op/s 2024-02-05T08:15:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:06 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:15:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:06 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:15:07.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:06 smithi196 ceph-mon[33018]: osdmap e243: 8 total, 8 up, 7 in 2024-02-05T08:15:07.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:07.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:07.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:07.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:07.220 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:07.245 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:07.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:07.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:07.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:07.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:07.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:07.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:07.474 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:07.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:07.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:07.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:07.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:07.617 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:07.625 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:07.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:07.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:07.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:07.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:07.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:07.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:07.854 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:07.878 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:07.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:07.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:07.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:08.003 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:08.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:08.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:08.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:15:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:15:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:15:08.105 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:08.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:08.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:08.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:08.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:08.253 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:08.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:08.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:08.354 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:08.378 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:08.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:08.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:08.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:08.503 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:08.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:08.605 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:08.628 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:08.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:08.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:08.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:08.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:08.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:08.788 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:15:08.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:08.887 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:08.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:08.989 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:09.015 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:09.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:09.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:08 smithi134 ceph-mon[39278]: pgmap v457: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:15:09.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:08 smithi134 ceph-mon[39278]: 6.17 scrub starts 2024-02-05T08:15:09.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:09.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:15:09.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:08 smithi134 ceph-mon[39278]: 6.17 scrub ok 2024-02-05T08:15:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:08 smithi134 ceph-mon[30086]: pgmap v457: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:15:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:08 smithi134 ceph-mon[30086]: 6.17 scrub starts 2024-02-05T08:15:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:15:09.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:08 smithi134 ceph-mon[30086]: 6.17 scrub ok 2024-02-05T08:15:09.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:09.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:09.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:09.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:09.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:09.270 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:09.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:09.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:08 smithi196 ceph-mon[33018]: pgmap v457: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:15:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:08 smithi196 ceph-mon[33018]: 6.17 scrub starts 2024-02-05T08:15:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:15:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:08 smithi196 ceph-mon[33018]: 6.17 scrub ok 2024-02-05T08:15:09.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:09.400 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:09.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:09.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:09.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:09.528 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:09.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:09.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:09.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:09.653 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:09.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:09.755 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:09.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:09.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:09.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:09.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:09.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:09.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:09.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:09.953 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-02-05T08:15:10.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:10.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:10.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:10.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:10.059 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:09 smithi134 ceph-mon[30086]: 5.15 scrub starts 2024-02-05T08:15:10.059 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:09 smithi134 ceph-mon[30086]: 5.15 scrub ok 2024-02-05T08:15:10.059 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:09 smithi134 ceph-mon[30086]: 6.c scrub starts 2024-02-05T08:15:10.059 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:09 smithi134 ceph-mon[30086]: 6.c scrub ok 2024-02-05T08:15:10.059 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:09 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4077759850' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:15:10.059 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:09 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:15:10.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:09 smithi134 ceph-mon[39278]: 5.15 scrub starts 2024-02-05T08:15:10.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:09 smithi134 ceph-mon[39278]: 5.15 scrub ok 2024-02-05T08:15:10.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:09 smithi134 ceph-mon[39278]: 6.c scrub starts 2024-02-05T08:15:10.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:09 smithi134 ceph-mon[39278]: 6.c scrub ok 2024-02-05T08:15:10.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:09 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4077759850' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:15:10.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:09 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:15:10.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:10.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:10.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:10.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:10.185 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:15:10.205 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:15:10.224 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:15:10.243 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:15:10.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:10.263 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:15:10.282 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:15:10.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:10.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:10.302 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:15:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:09 smithi196 ceph-mon[33018]: 5.15 scrub starts 2024-02-05T08:15:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:09 smithi196 ceph-mon[33018]: 5.15 scrub ok 2024-02-05T08:15:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:09 smithi196 ceph-mon[33018]: 6.c scrub starts 2024-02-05T08:15:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:09 smithi196 ceph-mon[33018]: 6.c scrub ok 2024-02-05T08:15:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:09 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4077759850' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:15:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:09 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:15:10.322 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:15:10.323 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:15:10.323 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:15:10.323 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:15:10.323 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:15:10.323 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:15:10.324 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:15:10.324 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:15:10.324 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:15:10.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:10.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:10.412 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:10.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:10.513 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:10.537 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:10.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:10.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:10.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:10.663 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:10.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:10.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:10.764 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:10.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:10.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:10.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:10.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:10.915 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:10.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:10.952 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.7 to 1 (10000) 2024-02-05T08:15:11.016 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:11.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[30086]: pgmap v458: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:15:11.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:15:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[30086]: osdmap e244: 8 total, 8 up, 7 in 2024-02-05T08:15:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3750920148' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:15:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3591192525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[30086]: 3.15 scrub starts 2024-02-05T08:15:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[30086]: 3.15 scrub ok 2024-02-05T08:15:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[39278]: pgmap v458: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:15:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:15:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[39278]: osdmap e244: 8 total, 8 up, 7 in 2024-02-05T08:15:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3750920148' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:15:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3591192525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[39278]: 3.15 scrub starts 2024-02-05T08:15:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:10 smithi134 ceph-mon[39278]: 3.15 scrub ok 2024-02-05T08:15:11.040 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:11.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:11.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:11.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:11.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:11.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:11.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:11.292 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:11.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:10 smithi196 ceph-mon[33018]: pgmap v458: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:15:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:10 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:15:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:10 smithi196 ceph-mon[33018]: osdmap e244: 8 total, 8 up, 7 in 2024-02-05T08:15:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:10 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3750920148' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:15:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:10 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3591192525' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:10 smithi196 ceph-mon[33018]: 3.15 scrub starts 2024-02-05T08:15:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:10 smithi196 ceph-mon[33018]: 3.15 scrub ok 2024-02-05T08:15:11.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:11.419 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:11.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:11.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:11.547 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:11.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:11.641 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:15:11.641 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:15:11.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:11.674 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:11.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:11.776 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:11.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:11.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:11.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:11.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:11.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:11.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:11.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:12.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:12.054 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:12.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:12.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:12.180 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:12.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:12.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:12.258 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:11 smithi134 ceph-mon[30086]: 5.d scrub starts 2024-02-05T08:15:12.258 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:11 smithi134 ceph-mon[30086]: 5.d scrub ok 2024-02-05T08:15:12.258 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:11 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3750920148' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:15:12.258 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:11 smithi134 ceph-mon[30086]: osdmap e245: 8 total, 8 up, 7 in 2024-02-05T08:15:12.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:11 smithi134 ceph-mon[39278]: 5.d scrub starts 2024-02-05T08:15:12.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:11 smithi134 ceph-mon[39278]: 5.d scrub ok 2024-02-05T08:15:12.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:11 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3750920148' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:15:12.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:11 smithi134 ceph-mon[39278]: osdmap e245: 8 total, 8 up, 7 in 2024-02-05T08:15:12.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:12.305 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:12.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:12.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:11 smithi196 ceph-mon[33018]: 5.d scrub starts 2024-02-05T08:15:12.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:11 smithi196 ceph-mon[33018]: 5.d scrub ok 2024-02-05T08:15:12.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:11 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3750920148' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:15:12.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:11 smithi196 ceph-mon[33018]: osdmap e245: 8 total, 8 up, 7 in 2024-02-05T08:15:12.407 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:12.431 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:12.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:12.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:12.557 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:12.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:12.561 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 6 1 2024-02-05T08:15:12.658 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:12.682 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:12.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:12.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:12.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:12.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:12.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:12.948 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:12.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:12.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:13.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:13.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:13.077 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:13.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:13.105 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[30086]: 3.17 scrub starts 2024-02-05T08:15:13.105 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[30086]: 3.17 scrub ok 2024-02-05T08:15:13.106 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[30086]: pgmap v461: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:15:13.106 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[30086]: 5.1d scrub starts 2024-02-05T08:15:13.106 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[30086]: 5.1d scrub ok 2024-02-05T08:15:13.106 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[30086]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:15:13.106 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[30086]: 5.11 scrub starts 2024-02-05T08:15:13.106 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[30086]: 5.11 scrub ok 2024-02-05T08:15:13.106 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[39278]: 3.17 scrub starts 2024-02-05T08:15:13.106 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[39278]: 3.17 scrub ok 2024-02-05T08:15:13.106 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[39278]: pgmap v461: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:15:13.106 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[39278]: 5.1d scrub starts 2024-02-05T08:15:13.107 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[39278]: 5.1d scrub ok 2024-02-05T08:15:13.107 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[39278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:15:13.107 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[39278]: 5.11 scrub starts 2024-02-05T08:15:13.107 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:12 smithi134 ceph-mon[39278]: 5.11 scrub ok 2024-02-05T08:15:13.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:13.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:13.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:13.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:12 smithi196 ceph-mon[33018]: 3.17 scrub starts 2024-02-05T08:15:13.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:12 smithi196 ceph-mon[33018]: 3.17 scrub ok 2024-02-05T08:15:13.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:12 smithi196 ceph-mon[33018]: pgmap v461: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:15:13.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:12 smithi196 ceph-mon[33018]: 5.1d scrub starts 2024-02-05T08:15:13.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:12 smithi196 ceph-mon[33018]: 5.1d scrub ok 2024-02-05T08:15:13.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:12 smithi196 ceph-mon[33018]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:15:13.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:12 smithi196 ceph-mon[33018]: 5.11 scrub starts 2024-02-05T08:15:13.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:12 smithi196 ceph-mon[33018]: 5.11 scrub ok 2024-02-05T08:15:13.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:13.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:13.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:13.432 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:13.457 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:13.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:13.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:13.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:13.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:15:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:15:13.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:13.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:13.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:13.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:13.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:13.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:13.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:13.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:13.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:13.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:13.874 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:13.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T08:15:13.882 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:13.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:13.969 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:13.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:14.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:14.095 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:14.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:14.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:14.123 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:13 smithi134 ceph-mon[30086]: 6.1e scrub starts 2024-02-05T08:15:14.123 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:13 smithi134 ceph-mon[30086]: 6.1e scrub ok 2024-02-05T08:15:14.123 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:13 smithi134 ceph-mon[30086]: 6.8 scrub starts 2024-02-05T08:15:14.123 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:13 smithi134 ceph-mon[30086]: 6.8 scrub ok 2024-02-05T08:15:14.123 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:13 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:14.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:13 smithi134 ceph-mon[39278]: 6.1e scrub starts 2024-02-05T08:15:14.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:13 smithi134 ceph-mon[39278]: 6.1e scrub ok 2024-02-05T08:15:14.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:13 smithi134 ceph-mon[39278]: 6.8 scrub starts 2024-02-05T08:15:14.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:13 smithi134 ceph-mon[39278]: 6.8 scrub ok 2024-02-05T08:15:14.123 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:13 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:14.128 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:15:14.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:15:14.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:14.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:14.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:14.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:14.250 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:13 smithi196 ceph-mon[33018]: 6.1e scrub starts 2024-02-05T08:15:14.250 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:13 smithi196 ceph-mon[33018]: 6.1e scrub ok 2024-02-05T08:15:14.250 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:13 smithi196 ceph-mon[33018]: 6.8 scrub starts 2024-02-05T08:15:14.250 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:13 smithi196 ceph-mon[33018]: 6.8 scrub ok 2024-02-05T08:15:14.250 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:13 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:14.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:14.347 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:14.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:14.425 DEBUG:teuthology.orchestra.run.smithi134:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-02-05T08:15:14.428 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T08:15:14.428 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T08:15:14.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:14.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:14.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:14.476 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:hints = 1 2024-02-05T08:15:14.476 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T08:15:14.476 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_144349 2024-02-05T08:15:14.477 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:15:14.477 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-02-05T08:15:14.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:14.611 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:14.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:14.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:14.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:14.750 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:14.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:14.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:14.893 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:14.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:14.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:14.995 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:15.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:14 smithi134 ceph-mon[30086]: pgmap v462: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:15:15.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:14 smithi134 ceph-mon[30086]: 3.2 scrub starts 2024-02-05T08:15:15.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:14 smithi134 ceph-mon[30086]: 3.2 scrub ok 2024-02-05T08:15:15.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:14 smithi134 ceph-mon[30086]: osdmap e246: 8 total, 8 up, 7 in 2024-02-05T08:15:15.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:14 smithi134 ceph-mon[39278]: pgmap v462: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:15:15.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:14 smithi134 ceph-mon[39278]: 3.2 scrub starts 2024-02-05T08:15:15.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:14 smithi134 ceph-mon[39278]: 3.2 scrub ok 2024-02-05T08:15:15.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:14 smithi134 ceph-mon[39278]: osdmap e246: 8 total, 8 up, 7 in 2024-02-05T08:15:15.032 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:15.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:15.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:15.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:15.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:15.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:15.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:15.274 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:14 smithi196 ceph-mon[33018]: pgmap v462: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:15:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:14 smithi196 ceph-mon[33018]: 3.2 scrub starts 2024-02-05T08:15:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:14 smithi196 ceph-mon[33018]: 3.2 scrub ok 2024-02-05T08:15:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:14 smithi196 ceph-mon[33018]: osdmap e246: 8 total, 8 up, 7 in 2024-02-05T08:15:15.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:15.316 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:15.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:15.417 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:15.455 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:15.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:15.477 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 1 16 3667 3651 228.178 228.188 0.00328377 0.00436958 2024-02-05T08:15:15.557 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:15.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:15.591 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:15.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:15.692 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:15.730 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:15.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:15.832 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:15.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:15.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:15.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:16.011 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:16.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:16.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:16.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:15 smithi134 ceph-mon[30086]: 5.c scrub starts 2024-02-05T08:15:16.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:15 smithi134 ceph-mon[30086]: 5.c scrub ok 2024-02-05T08:15:16.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:15 smithi134 ceph-mon[30086]: 4.5 scrub starts 2024-02-05T08:15:16.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:15 smithi134 ceph-mon[30086]: 4.5 scrub ok 2024-02-05T08:15:16.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:15 smithi134 ceph-mon[30086]: 3.7 scrub starts 2024-02-05T08:15:16.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:15 smithi134 ceph-mon[30086]: 3.7 scrub ok 2024-02-05T08:15:16.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:15 smithi134 ceph-mon[39278]: 5.c scrub starts 2024-02-05T08:15:16.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:15 smithi134 ceph-mon[39278]: 5.c scrub ok 2024-02-05T08:15:16.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:15 smithi134 ceph-mon[39278]: 4.5 scrub starts 2024-02-05T08:15:16.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:15 smithi134 ceph-mon[39278]: 4.5 scrub ok 2024-02-05T08:15:16.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:15 smithi134 ceph-mon[39278]: 3.7 scrub starts 2024-02-05T08:15:16.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:15 smithi134 ceph-mon[39278]: 3.7 scrub ok 2024-02-05T08:15:16.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:15 smithi196 ceph-mon[33018]: 5.c scrub starts 2024-02-05T08:15:16.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:15 smithi196 ceph-mon[33018]: 5.c scrub ok 2024-02-05T08:15:16.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:15 smithi196 ceph-mon[33018]: 4.5 scrub starts 2024-02-05T08:15:16.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:15 smithi196 ceph-mon[33018]: 4.5 scrub ok 2024-02-05T08:15:16.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:15 smithi196 ceph-mon[33018]: 3.7 scrub starts 2024-02-05T08:15:16.042 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:15 smithi196 ceph-mon[33018]: 3.7 scrub ok 2024-02-05T08:15:16.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:16.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:16.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:16.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:16.254 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:16.276 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:15:16.276 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:15:16.296 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:16.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:16.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:16.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:16.434 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:16.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:16.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:16.477 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 2 16 7054 7038 219.923 211.688 0.00239479 0.004535 2024-02-05T08:15:16.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:16.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:16.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:16.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:16.677 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:16.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:16.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:16.816 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:16.854 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:16.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:16.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:16.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:16.993 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:16.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:16.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:17.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T08:15:17.077 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:16 smithi134 ceph-mon[30086]: 3.1 scrub starts 2024-02-05T08:15:17.077 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:16 smithi134 ceph-mon[30086]: 3.1 scrub ok 2024-02-05T08:15:17.078 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:16 smithi134 ceph-mon[30086]: pgmap v464: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:15:17.078 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:16 smithi134 ceph-mon[30086]: 4.f scrub starts 2024-02-05T08:15:17.078 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:16 smithi134 ceph-mon[30086]: 4.f scrub ok 2024-02-05T08:15:17.078 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:16 smithi134 ceph-mon[39278]: 3.1 scrub starts 2024-02-05T08:15:17.078 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:16 smithi134 ceph-mon[39278]: 3.1 scrub ok 2024-02-05T08:15:17.078 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:16 smithi134 ceph-mon[39278]: pgmap v464: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:15:17.078 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:16 smithi134 ceph-mon[39278]: 4.f scrub starts 2024-02-05T08:15:17.078 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:16 smithi134 ceph-mon[39278]: 4.f scrub ok 2024-02-05T08:15:17.095 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:17.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:17.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:17.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:17.233 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:17.269 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:17.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:16 smithi196 ceph-mon[33018]: 3.1 scrub starts 2024-02-05T08:15:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:16 smithi196 ceph-mon[33018]: 3.1 scrub ok 2024-02-05T08:15:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:16 smithi196 ceph-mon[33018]: pgmap v464: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:15:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:16 smithi196 ceph-mon[33018]: 4.f scrub starts 2024-02-05T08:15:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:16 smithi196 ceph-mon[33018]: 4.f scrub ok 2024-02-05T08:15:17.371 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:17.403 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:17.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:17.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:17.457 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:15:17.477 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 3 16 10215 10199 212.464 197.562 0.00336499 0.00469775 2024-02-05T08:15:17.504 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:17.542 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:17.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:17.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:17.644 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:17.684 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:17.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:17.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:17.805 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:17.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:17.831 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:17.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:17.933 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:17.949 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:15:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:15:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:15:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:17.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:17.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:18.076 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:18.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:18.117 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:18.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:18.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:18.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:18.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:18.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:18.272 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:17 smithi134 ceph-mon[30086]: 3.5 scrub starts 2024-02-05T08:15:18.273 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:17 smithi134 ceph-mon[30086]: 3.5 scrub ok 2024-02-05T08:15:18.273 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:17 smithi134 ceph-mon[30086]: 6.1a scrub starts 2024-02-05T08:15:18.273 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:17 smithi134 ceph-mon[30086]: 6.1a scrub ok 2024-02-05T08:15:18.273 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: 3.5 scrub starts 2024-02-05T08:15:18.273 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: 3.5 scrub ok 2024-02-05T08:15:18.273 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: 6.1a scrub starts 2024-02-05T08:15:18.273 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: 6.1a scrub ok 2024-02-05T08:15:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:17 smithi196 ceph-mon[33018]: 3.5 scrub starts 2024-02-05T08:15:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:17 smithi196 ceph-mon[33018]: 3.5 scrub ok 2024-02-05T08:15:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:17 smithi196 ceph-mon[33018]: 6.1a scrub starts 2024-02-05T08:15:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:17 smithi196 ceph-mon[33018]: 6.1a scrub ok 2024-02-05T08:15:18.355 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:18.392 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:18.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:18.477 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 4 16 13031 13015 203.344 176 0.00454822 0.00491069 2024-02-05T08:15:18.493 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:18.530 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:18.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:18.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:18.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:18.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:18.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:18.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:18.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:18.807 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:18.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:18.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:18.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:18.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:19.011 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.6 to 1 (10000) 2024-02-05T08:15:19.020 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:15:19.033 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:15:19.033 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[30086]: pgmap v465: 148 pgs: 148 active+clean; 201 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 26 MiB/s wr, 415 op/s 2024-02-05T08:15:19.034 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:15:19.034 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[30086]: 3.0 scrub starts 2024-02-05T08:15:19.034 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[30086]: 3.0 scrub ok 2024-02-05T08:15:19.034 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3239411078' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:15:19.034 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1050585487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:15:19.034 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:15:19.034 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[30086]: 5.1a scrub starts 2024-02-05T08:15:19.034 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[30086]: 5.1a scrub ok 2024-02-05T08:15:19.035 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:15:19.035 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: pgmap v465: 148 pgs: 148 active+clean; 201 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 26 MiB/s wr, 415 op/s 2024-02-05T08:15:19.035 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:15:19.035 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: 3.0 scrub starts 2024-02-05T08:15:19.035 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: 3.0 scrub ok 2024-02-05T08:15:19.035 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3239411078' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:15:19.035 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1050585487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:15:19.035 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:15:19.035 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: 5.1a scrub starts 2024-02-05T08:15:19.035 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:18 smithi134 ceph-mon[39278]: 5.1a scrub ok 2024-02-05T08:15:19.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:19.089 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:19.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:19.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:19.229 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:19.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:18 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:15:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:18 smithi196 ceph-mon[33018]: pgmap v465: 148 pgs: 148 active+clean; 201 MiB data, 2.9 GiB used, 623 GiB / 626 GiB avail; 1.3 KiB/s rd, 26 MiB/s wr, 415 op/s 2024-02-05T08:15:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:18 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:15:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:18 smithi196 ceph-mon[33018]: 3.0 scrub starts 2024-02-05T08:15:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:18 smithi196 ceph-mon[33018]: 3.0 scrub ok 2024-02-05T08:15:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:18 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3239411078' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:15:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:18 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1050585487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:15:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:18 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:15:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:18 smithi196 ceph-mon[33018]: 5.1a scrub starts 2024-02-05T08:15:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:18 smithi196 ceph-mon[33018]: 5.1a scrub ok 2024-02-05T08:15:19.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:19.366 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:19.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:19.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:19.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:19.478 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 5 16 15741 15725 196.499 169.375 0.00640435 0.00508156 2024-02-05T08:15:19.506 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:19.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:19.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:19.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:19.651 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:19.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:19.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:19.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:19.796 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:19.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:19.898 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:19.941 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:19.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:19.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:20.043 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:20.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:19 smithi134 ceph-mon[30086]: 2.1 scrub starts 2024-02-05T08:15:20.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:19 smithi134 ceph-mon[30086]: 2.1 scrub ok 2024-02-05T08:15:20.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:19 smithi134 ceph-mon[30086]: 2.2 scrub starts 2024-02-05T08:15:20.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:19 smithi134 ceph-mon[30086]: 2.2 scrub ok 2024-02-05T08:15:20.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:19 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:15:20.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:19 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:15:20.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:19 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3239411078' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T08:15:20.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:19 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:15:20.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:19 smithi134 ceph-mon[30086]: osdmap e247: 8 total, 8 up, 7 in 2024-02-05T08:15:20.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:19 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:15:20.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:19 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:15:20.058 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:19 smithi134 ceph-mon[30086]: osdmap e248: 8 total, 8 up, 7 in 2024-02-05T08:15:20.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:20 smithi134 ceph-mon[39278]: 2.1 scrub starts 2024-02-05T08:15:20.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:20 smithi134 ceph-mon[39278]: 2.1 scrub ok 2024-02-05T08:15:20.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:20 smithi134 ceph-mon[39278]: 2.2 scrub starts 2024-02-05T08:15:20.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:20 smithi134 ceph-mon[39278]: 2.2 scrub ok 2024-02-05T08:15:20.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:20 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:15:20.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:20 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:15:20.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:20 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3239411078' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T08:15:20.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:20 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:15:20.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:20 smithi134 ceph-mon[39278]: osdmap e247: 8 total, 8 up, 7 in 2024-02-05T08:15:20.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:20 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:15:20.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:20 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:15:20.059 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:20 smithi134 ceph-mon[39278]: osdmap e248: 8 total, 8 up, 7 in 2024-02-05T08:15:20.083 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:20.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:20.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:20.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:20 smithi196 ceph-mon[33018]: 2.1 scrub starts 2024-02-05T08:15:20.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:20 smithi196 ceph-mon[33018]: 2.1 scrub ok 2024-02-05T08:15:20.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:20 smithi196 ceph-mon[33018]: 2.2 scrub starts 2024-02-05T08:15:20.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:20 smithi196 ceph-mon[33018]: 2.2 scrub ok 2024-02-05T08:15:20.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:20 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:15:20.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:20 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:15:20.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:20 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3239411078' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T08:15:20.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:20 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:15:20.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:20 smithi196 ceph-mon[33018]: osdmap e247: 8 total, 8 up, 7 in 2024-02-05T08:15:20.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:20 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:15:20.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:20 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:15:20.171 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:20 smithi196 ceph-mon[33018]: osdmap e248: 8 total, 8 up, 7 in 2024-02-05T08:15:20.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:20.220 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:20.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:20.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:20.247 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:15:20.248 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:15:20.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:20.359 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:20.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:20.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:20.478 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 6 16 18275 18259 190.144 158.375 0.00661203 0.00525342 2024-02-05T08:15:20.494 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:20.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:20.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:20.634 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:20.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:20.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:20.736 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:20.773 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:20.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:20.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:20.875 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:20.909 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:20.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:20.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T08:15:21.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:21.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:21.055 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:21.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:21.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:21.199 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:21.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:21.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[30086]: pgmap v467: 148 pgs: 148 active+clean; 729 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T08:15:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[30086]: 5.8 deep-scrub starts 2024-02-05T08:15:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[30086]: 5.8 deep-scrub ok 2024-02-05T08:15:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[30086]: 3.1a scrub starts 2024-02-05T08:15:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[30086]: 3.1a scrub ok 2024-02-05T08:15:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[30086]: 6.5 scrub starts 2024-02-05T08:15:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[30086]: 6.5 scrub ok 2024-02-05T08:15:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[39278]: pgmap v467: 148 pgs: 148 active+clean; 729 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T08:15:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[39278]: 5.8 deep-scrub starts 2024-02-05T08:15:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[39278]: 5.8 deep-scrub ok 2024-02-05T08:15:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[39278]: 3.1a scrub starts 2024-02-05T08:15:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[39278]: 3.1a scrub ok 2024-02-05T08:15:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[39278]: 6.5 scrub starts 2024-02-05T08:15:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:21 smithi134 ceph-mon[39278]: 6.5 scrub ok 2024-02-05T08:15:21.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:21 smithi196 ceph-mon[33018]: pgmap v467: 148 pgs: 148 active+clean; 729 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T08:15:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:21 smithi196 ceph-mon[33018]: 5.8 deep-scrub starts 2024-02-05T08:15:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:21 smithi196 ceph-mon[33018]: 5.8 deep-scrub ok 2024-02-05T08:15:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:21 smithi196 ceph-mon[33018]: 3.1a scrub starts 2024-02-05T08:15:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:21 smithi196 ceph-mon[33018]: 3.1a scrub ok 2024-02-05T08:15:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:21 smithi196 ceph-mon[33018]: 6.5 scrub starts 2024-02-05T08:15:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:21 smithi196 ceph-mon[33018]: 6.5 scrub ok 2024-02-05T08:15:21.341 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:15:21.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:21.352 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:21.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:21.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:21.479 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 7 15 20602 20587 183.765 145.5 0.00862327 0.00543622 2024-02-05T08:15:21.490 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:21.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:21.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:21.592 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:21.626 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:21.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:21.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:21.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:21.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:21.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:21.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:21.872 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:21.902 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:21.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:22.004 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-05T08:15:22.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:15:22.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:22.044 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:22.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:22.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:22 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:15:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:22 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:15:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:22 smithi196 ceph-mon[33018]: 3.1c scrub starts 2024-02-05T08:15:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:22 smithi196 ceph-mon[33018]: 3.1c scrub ok 2024-02-05T08:15:22.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:22.204 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:22.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:22.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:22.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:22 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:15:22.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:22 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:15:22.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:22 smithi134 ceph-mon[30086]: 3.1c scrub starts 2024-02-05T08:15:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:22 smithi134 ceph-mon[30086]: 3.1c scrub ok 2024-02-05T08:15:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:22 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:15:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:22 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:15:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:22 smithi134 ceph-mon[39278]: 3.1c scrub starts 2024-02-05T08:15:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:22 smithi134 ceph-mon[39278]: 3.1c scrub ok 2024-02-05T08:15:22.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:22.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:22.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:22.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:22.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:22.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:22.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:22.479 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 8 16 22687 22671 177.076 130.25 0.00535156 0.00564015 2024-02-05T08:15:22.566 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:22.601 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:22.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:22.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:22.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:22.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:22.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:22.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:22.841 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:22.877 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:22.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:22.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:22.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:23.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:23.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:23.116 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:23.153 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:23.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:23.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:23 smithi196 ceph-mon[33018]: pgmap v469: 148 pgs: 148 active+clean; 729 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 732 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-02-05T08:15:23.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:23 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:15:23.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:23 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:15:23.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:23 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:15:23.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:23 smithi196 ceph-mon[33018]: 5.19 scrub starts 2024-02-05T08:15:23.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:23 smithi196 ceph-mon[33018]: 5.19 scrub ok 2024-02-05T08:15:23.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:23 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:15:23.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:23 smithi196 ceph-mon[33018]: osdmap e249: 8 total, 8 up, 7 in 2024-02-05T08:15:23.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:23 smithi196 ceph-mon[33018]: 6.1b deep-scrub starts 2024-02-05T08:15:23.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:23 smithi196 ceph-mon[33018]: 6.1b deep-scrub ok 2024-02-05T08:15:23.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:23.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:15:23.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[30086]: pgmap v469: 148 pgs: 148 active+clean; 729 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 732 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-02-05T08:15:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:15:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:15:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:15:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[30086]: 5.19 scrub starts 2024-02-05T08:15:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[30086]: 5.19 scrub ok 2024-02-05T08:15:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:15:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[30086]: osdmap e249: 8 total, 8 up, 7 in 2024-02-05T08:15:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[30086]: 6.1b deep-scrub starts 2024-02-05T08:15:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[30086]: 6.1b deep-scrub ok 2024-02-05T08:15:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:15:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[39278]: pgmap v469: 148 pgs: 148 active+clean; 729 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 732 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-02-05T08:15:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:15:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:15:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:15:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[39278]: 5.19 scrub starts 2024-02-05T08:15:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[39278]: 5.19 scrub ok 2024-02-05T08:15:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:15:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[39278]: osdmap e249: 8 total, 8 up, 7 in 2024-02-05T08:15:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[39278]: 6.1b deep-scrub starts 2024-02-05T08:15:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[39278]: 6.1b deep-scrub ok 2024-02-05T08:15:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:15:23.292 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:23.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:23.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:23.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:23.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:23.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:23.479 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 9 16 24589 24573 170.608 118.875 0.00249567 0.00579651 2024-02-05T08:15:23.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:15:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:15:23.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:23.550 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:19178 7.1 7:843e4ad0:::benchmark_data_smithi134_83016_object19177:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:06:58.646746+0000", 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 504.898487429, 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.137073784, 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 19178 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.551 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646746+0000", 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646746+0000", 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646747+0000", 2024-02-05T08:15:23.552 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.415e-06 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646769+0000", 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2388999999999999e-05 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646783+0000", 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3696000000000001e-05 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646791+0000", 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1300000000000001e-06 2024-02-05T08:15:23.553 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646834+0000", 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3006999999999997e-05 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.646936+0000", 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101811 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.647049+0000", 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113516 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.554 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.648395+0000", 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001345572 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.648785+0000", 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00039036099999999998 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.783734+0000", 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13494893399999999 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.555 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.783780+0000", 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5284000000000002e-05 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:58.783819+0000", 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9669000000000003e-05 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:20690 7.1 7:85067d19:::benchmark_data_smithi134_83016_object20689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:15:23.556 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:06:59.674363+0000", 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 503.87086964399998, 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14902151899999999, 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 20690 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674363+0000", 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.557 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674363+0000", 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674365+0000", 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.748e-06 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674395+0000", 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9620000000000001e-05 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.558 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674405+0000", 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0220999999999999e-05 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674412+0000", 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5739999999999998e-06 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674509+0000", 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6210000000000002e-05 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.559 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674634+0000", 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000125241 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674750+0000", 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116585 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.675575+0000", 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000824236 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.560 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.676042+0000", 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000466995 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.823321+0000", 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14727914 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.823361+0000", 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0445999999999999e-05 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.561 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.823385+0000", 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3502999999999998e-05 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:99792 7.1 7:8fc2463e:::benchmark_data_smithi134_83016_object99791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:08:08.947855+0000", 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 434.597377967, 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13887300799999999, 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:15:23.562 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 99792 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.947855+0000", 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.947855+0000", 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.563 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.947856+0000", 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.091e-06 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.947913+0000", 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7017e-05 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.947925+0000", 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1654999999999999e-05 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.564 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.947931+0000", 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3770000000000001e-06 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.948005+0000", 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4165999999999994e-05 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.948103+0000", 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8139000000000003e-05 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3", 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.948203+0000", 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9871000000000007e-05 2024-02-05T08:15:23.565 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:08.950138+0000", 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019349269999999999 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:09.077895+0000", 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12775708699999999 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:09.086623+0000", 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0087272589999999994 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.566 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:09.086687+0000", 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4357999999999996e-05 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:08:09.086728+0000", 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1060999999999999e-05 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:25746 8.2 8:4ade417d:::benchmark_data_smithi134_115836_object25745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-02-05T08:15:23.567 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:11:20.413491+0000", 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 243.13174217400001, 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14147770600000001, 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 25746 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413491+0000", 2024-02-05T08:15:23.568 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413491+0000", 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413491+0000", 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0299999999999998e-07 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413556+0000", 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4455000000000005e-05 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.569 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413567+0000", 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0749e-05 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413575+0000", 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9710000000000004e-06 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413621+0000", 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6288000000000001e-05 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.570 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.571 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413721+0000", 2024-02-05T08:15:23.571 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100373 2024-02-05T08:15:23.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.571 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.571 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:23.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.413813+0000", 2024-02-05T08:15:23.571 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1324999999999997e-05 2024-02-05T08:15:23.571 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.571 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.571 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.571 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414683+0000", 2024-02-05T08:15:23.571 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00087058100000000005 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414951+0000", 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026772299999999998 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.554907+0000", 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13995589999999999 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.554944+0000", 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7314000000000002e-05 2024-02-05T08:15:23.572 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.554968+0000", 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4324e-05 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:25750 8.5 8:a65889c9:::benchmark_data_smithi134_115836_object25749:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:11:20.414049+0000", 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 243.13118372700001, 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13532624500000001, 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.573 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 25750 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414049+0000", 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414049+0000", 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.574 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.575 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414050+0000", 2024-02-05T08:15:23.575 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0400000000000005e-07 2024-02-05T08:15:23.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.575 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.575 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.575 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414098+0000", 2024-02-05T08:15:23.575 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8439999999999997e-05 2024-02-05T08:15:23.575 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.575 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.575 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414111+0000", 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.22e-05 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414118+0000", 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0829999999999998e-06 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414159+0000", 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1511999999999998e-05 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.576 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414251+0000", 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1366999999999994e-05 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.414346+0000", 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5629000000000002e-05 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.415357+0000", 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010108809999999999 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.577 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.415535+0000", 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000178004 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.549281+0000", 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13374603199999999 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.549336+0000", 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5210999999999997e-05 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.578 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:11:20.549375+0000", 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9082000000000003e-05 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:131942 8.2 8:40ced295:::benchmark_data_smithi134_115836_object15282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:58.589962+0000", 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 144.955270603, 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.154613257, 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.579 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 131942 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.589962+0000", 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.589962+0000", 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.580 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.589963+0000", 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0550000000000001e-06 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.589968+0000", 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5929999999999999e-06 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.589981+0000", 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2729e-05 2024-02-05T08:15:23.581 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.589988+0000", 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6629999999999993e-06 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.590030+0000", 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2045000000000001e-05 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.590111+0000", 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0148000000000006e-05 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.582 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.590185+0000", 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4167999999999999e-05 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.592612+0000", 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024273319999999999 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.592751+0000", 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013892099999999999 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.583 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.744474+0000", 2024-02-05T08:15:23.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15172269599999999 2024-02-05T08:15:23.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.744535+0000", 2024-02-05T08:15:23.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1130000000000006e-05 2024-02-05T08:15:23.584 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.584 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.584 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.584 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.744576+0000", 2024-02-05T08:15:23.584 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0776999999999999e-05 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:131957 8.2 8:4fdaea78:::benchmark_data_smithi134_115836_object15297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:58.603787+0000", 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 144.941445926, 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.143598265, 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 131957 2024-02-05T08:15:23.585 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.603787+0000", 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.603787+0000", 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.603788+0000", 2024-02-05T08:15:23.586 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0610000000000001e-06 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.603792+0000", 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1389999999999997e-06 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.603802+0000", 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0159e-05 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.603808+0000", 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6180000000000003e-06 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.587 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.588 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:23.589 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:23.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:23.589 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.603868+0000", 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0489999999999999e-05 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.603958+0000", 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9268e-05 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.604034+0000", 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5943000000000001e-05 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.590 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.606140+0000", 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021064410000000001 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.606484+0000", 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00034428699999999998 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.747314+0000", 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14082968300000001 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.591 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.747361+0000", 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7367999999999998e-05 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.747385+0000", 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3808e-05 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:133200 8.b 8:daeb1f29:::benchmark_data_smithi134_115836_object16540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:00.227983+0000", 2024-02-05T08:15:23.592 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 143.317250218, 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.145027513, 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 133200 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.227983+0000", 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.593 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.227983+0000", 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.227984+0000", 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2619999999999999e-06 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.227991+0000", 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.793e-06 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.594 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.278852+0000", 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.050861363999999999 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.278878+0000", 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5619999999999999e-05 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.284653+0000", 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0057753379999999997 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.284727+0000", 2024-02-05T08:15:23.595 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3419999999999998e-05 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2,3", 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.284815+0000", 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8430000000000005e-05 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.288890+0000", 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040753129999999997 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.596 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.288942+0000", 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1280999999999999e-05 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.291912+0000", 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029703329999999999 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.372794+0000", 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.080882308 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.597 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.372866+0000", 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1445999999999993e-05 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:00.373010+0000", 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014460499999999999 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:134718 8.b 8:dc8749ae:::benchmark_data_smithi134_115836_object18058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:02.196552+0000", 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 141.34868078, 2024-02-05T08:15:23.598 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.139401156, 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 134718 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.196552+0000", 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.599 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.196552+0000", 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.196553+0000", 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.322e-06 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.196558+0000", 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6990000000000002e-06 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.600 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.196567+0000", 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3009999999999995e-06 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.196574+0000", 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5030000000000002e-06 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.196629+0000", 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5472e-05 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.601 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.196777+0000", 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014731799999999999 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2,3", 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.196915+0000", 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000138551 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.199363+0000", 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024480560000000001 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.602 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.199795+0000", 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00043115500000000001 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.328270+0000", 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12847539099999999 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.335739+0000", 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0074694419999999997 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.603 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.335816+0000", 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6262999999999998e-05 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.335953+0000", 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013768299999999999 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:134925 8.2 8:40737ef3:::benchmark_data_smithi134_115836_object18265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:02.535531+0000", 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 141.00970229699999, 2024-02-05T08:15:23.604 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.170932836, 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 134925 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535531+0000", 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.605 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535531+0000", 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535565+0000", 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3880000000000001e-05 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535571+0000", 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6390000000000003e-06 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.606 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535587+0000", 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5664000000000001e-05 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535595+0000", 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3690000000000007e-06 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535692+0000", 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7102999999999995e-05 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.607 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535829+0000", 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013684299999999999 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.536032+0000", 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020306 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538954+0000", 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002921313 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.608 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.539112+0000", 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015816499999999999 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.706382+0000", 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16727075999999999 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.706429+0000", 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6851000000000003e-05 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.609 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.706463+0000", 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4189000000000001e-05 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:134931 8.b 8:d4cd9ded:::benchmark_data_smithi134_115836_object18271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:02.538007+0000", 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 141.00722598300001, 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16854570699999999, 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.610 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 134931 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538007+0000", 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538007+0000", 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.611 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538008+0000", 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3880000000000001e-06 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538018+0000", 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5710000000000004e-06 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538033+0000", 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4928e-05 2024-02-05T08:15:23.612 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538043+0000", 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0382e-05 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538111+0000", 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7556000000000002e-05 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538257+0000", 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000146649 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.613 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2,3", 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538415+0000", 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015738199999999999 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.540847+0000", 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024317710000000001 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.541060+0000", 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021372900000000001 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.614 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.541562+0000", 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000502058 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.706359+0000", 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.164796741 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.706428+0000", 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8547999999999995e-05 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.615 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.706553+0000", 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012500400000000001 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:161406 8.2 8:410186ff:::benchmark_data_smithi134_115836_object44746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:35.283606+0000", 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 108.261626509, 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13728402100000001, 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.616 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 161406 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.283606+0000", 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.283606+0000", 2024-02-05T08:15:23.617 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.283608+0000", 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6780000000000001e-06 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.283613+0000", 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.994e-06 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.283627+0000", 2024-02-05T08:15:23.618 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3736e-05 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.283636+0000", 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6810000000000004e-06 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.283695+0000", 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9712e-05 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.619 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.283810+0000", 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114654 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.283921+0000", 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011112600000000001 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.286681+0000", 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027600580000000001 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.620 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.287452+0000", 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00077077299999999999 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.420830+0000", 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.133377679 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.420868+0000", 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8080999999999997e-05 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:35.420890+0000", 2024-02-05T08:15:23.621 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2849000000000001e-05 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:162947 8.b 8:d02f707b:::benchmark_data_smithi134_115836_object46287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:37.217853+0000", 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 106.327379639, 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14594627299999999, 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.622 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 162947 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.217853+0000", 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.217853+0000", 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.217855+0000", 2024-02-05T08:15:23.623 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.455e-06 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.217860+0000", 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4500000000000003e-06 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.217877+0000", 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6498e-05 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.217884+0000", 2024-02-05T08:15:23.624 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5560000000000002e-06 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.221862+0000", 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039781060000000004 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.221976+0000", 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113201 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2,3", 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.222107+0000", 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000131743 2024-02-05T08:15:23.625 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.224873+0000", 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027655790000000002 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.226643+0000", 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001769858 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.231832+0000", 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051890779999999997 2024-02-05T08:15:23.626 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.363612+0000", 2024-02-05T08:15:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13178054 2024-02-05T08:15:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.363681+0000", 2024-02-05T08:15:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8775999999999993e-05 2024-02-05T08:15:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.627 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.363800+0000", 2024-02-05T08:15:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000118433 2024-02-05T08:15:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.628 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:162964 8.b 8:d8f90343:::benchmark_data_smithi134_115836_object46304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:37.231740+0000", 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 106.31349258199999, 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13450068700000001, 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 162964 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.629 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.231740+0000", 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.231740+0000", 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.231742+0000", 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3850000000000001e-06 2024-02-05T08:15:23.630 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.231746+0000", 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4349999999999999e-06 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.231758+0000", 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2078e-05 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.231766+0000", 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7999999999999999e-06 2024-02-05T08:15:23.631 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.232085+0000", 2024-02-05T08:15:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00031916300000000002 2024-02-05T08:15:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.232287+0000", 2024-02-05T08:15:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000201986 2024-02-05T08:15:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.632 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2,3", 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.232427+0000", 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014007399999999999 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.234637+0000", 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022100290000000001 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.234651+0000", 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3551e-05 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.633 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.279647+0000", 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.044996376999999997 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.366032+0000", 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.086385007 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.366092+0000", 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0043000000000001e-05 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.634 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.366241+0000", 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014875899999999999 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:164620 8.b 8:db824b65:::benchmark_data_smithi134_115836_object47960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:39.045586+0000", 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 104.499646564, 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14162455700000001, 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.635 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 164620 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.045586+0000", 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.045586+0000", 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.636 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.045588+0000", 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3680000000000001e-06 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.045593+0000", 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5589999999999996e-06 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.045607+0000", 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3589e-05 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.637 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.045615+0000", 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5690000000000005e-06 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.045719+0000", 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000104029 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.045851+0000", 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000131025 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.638 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2,3", 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.045987+0000", 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000136563 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.049945+0000", 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039576790000000004 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.050057+0000", 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112439 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.639 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.050070+0000", 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3294e-05 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.187096+0000", 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13702529799999999 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.187140+0000", 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4484000000000003e-05 2024-02-05T08:15:23.640 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:39.187211+0000", 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0661000000000007e-05 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170415 8.b 8:d9f796c8:::benchmark_data_smithi134_115836_object53755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.829318+0000", 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 97.715915351000007, 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.91218017299999998, 2024-02-05T08:15:23.641 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170415 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829318+0000", 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829318+0000", 2024-02-05T08:15:23.642 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829319+0000", 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6640000000000001e-06 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829335+0000", 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5282999999999999e-05 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829350+0000", 2024-02-05T08:15:23.643 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5820000000000001e-05 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829374+0000", 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4136999999999999e-05 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829424+0000", 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9091000000000001e-05 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829432+0000", 2024-02-05T08:15:23.644 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6959999999999994e-06 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.734517+0000", 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.90508505800000005 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.734603+0000", 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6011e-05 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.734676+0000", 2024-02-05T08:15:23.645 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2261999999999994e-05 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741372+0000", 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066968749999999997 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741411+0000", 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8059999999999998e-05 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741439+0000", 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8787000000000001e-05 2024-02-05T08:15:23.646 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741474+0000", 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4329999999999998e-05 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741498+0000", 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4099000000000001e-05 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.647 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170468 8.b 8:da1853ce:::benchmark_data_smithi134_115836_object53808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:15:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.873146+0000", 2024-02-05T08:15:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 97.672086481999997, 2024-02-05T08:15:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.86851456500000002, 2024-02-05T08:15:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170468 2024-02-05T08:15:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.648 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873146+0000", 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873146+0000", 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873148+0000", 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1459999999999999e-06 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.649 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873152+0000", 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5040000000000004e-06 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873167+0000", 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4642e-05 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873176+0000", 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3370000000000006e-06 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.650 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873241+0000", 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4597999999999994e-05 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873249+0000", 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3100000000000001e-06 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.737138+0000", 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.86388931499999999 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.651 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.737244+0000", 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105386 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.737361+0000", 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117689 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741555+0000", 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041938380000000001 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.652 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741589+0000", 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3438000000000001e-05 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741615+0000", 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6696e-05 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741643+0000", 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.747e-05 2024-02-05T08:15:23.653 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741661+0000", 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8196e-05 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170493 8.b 8:d1db0339:::benchmark_data_smithi134_115836_object53833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.894206+0000", 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 97.651027116999998, 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.85198245399999994, 2024-02-05T08:15:23.654 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170493 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894206+0000", 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.655 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894206+0000", 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894207+0000", 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.201e-06 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894211+0000", 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0779999999999997e-06 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.656 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894236+0000", 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5241000000000001e-05 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894246+0000", 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2159999999999995e-06 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894315+0000", 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9863000000000005e-05 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.657 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894333+0000", 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7793999999999999e-05 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.739546+0000", 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.84521263000000002 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.739615+0000", 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9233000000000006e-05 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:15:23.658 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.739685+0000", 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0118999999999997e-05 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741667+0000", 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019816230000000001 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746011+0000", 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043444979999999996 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.659 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746094+0000", 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2254000000000002e-05 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746153+0000", 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9231999999999998e-05 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746188+0000", 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5472000000000002e-05 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.660 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170502 8.b 8:d6d92551:::benchmark_data_smithi134_115836_object53842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.904511+0000", 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 97.640722181000001, 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.84494767500000001, 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170502 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.661 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904511+0000", 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904511+0000", 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904512+0000", 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5499999999999996e-07 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.662 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904516+0000", 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9140000000000001e-06 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904527+0000", 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1263999999999999e-05 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904533+0000", 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0279999999999997e-06 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904616+0000", 2024-02-05T08:15:23.663 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2929999999999994e-05 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904645+0000", 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9439000000000001e-05 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741703+0000", 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83705783 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.664 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741809+0000", 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105708 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741919+0000", 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010994599999999999 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746024+0000", 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041055070000000004 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.665 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746054+0000", 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0000000000000001e-05 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.749365+0000", 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003311072 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.749425+0000", 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9728999999999997e-05 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.749458+0000", 2024-02-05T08:15:23.666 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3352999999999997e-05 2024-02-05T08:15:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.667 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.667 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:15:23.667 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:15:23.668 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:15:23.668 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:15:23.668 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:15:23.668 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:15:23.668 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:15:23.668 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:15:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170522 8.b 8:d777389e:::benchmark_data_smithi134_115836_object53862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:15:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.943354+0000", 2024-02-05T08:15:23.668 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 97.601879135999994, 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80622890899999999, 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170522 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943354+0000", 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.669 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943354+0000", 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943355+0000", 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3519999999999999e-06 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943360+0000", 2024-02-05T08:15:23.670 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.237e-06 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943373+0000", 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2850999999999999e-05 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943381+0000", 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4170000000000003e-06 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.671 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943435+0000", 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4799999999999997e-05 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943444+0000", 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3650000000000004e-06 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.743871+0000", 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80042767599999998 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.672 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.743978+0000", 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010660800000000001 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.744084+0000", 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010608 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746203+0000", 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002119147 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.673 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.748861+0000", 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026574300000000001 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.749495+0000", 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00063419099999999999 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.749537+0000", 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2132000000000003e-05 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:23.674 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.749583+0000", 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5623e-05 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:15:23.675 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:15:23.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:23.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:23.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:23.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:23.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:23.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:23.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:23.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:23.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:23.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:24.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:24.136 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:24.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:24.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:24.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:15:24.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:24.272 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:24.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:24 smithi196 ceph-mon[33018]: 3.1b scrub starts 2024-02-05T08:15:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:24 smithi196 ceph-mon[33018]: 3.1b scrub ok 2024-02-05T08:15:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:24 smithi196 ceph-mon[33018]: 5.1b scrub starts 2024-02-05T08:15:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:24 smithi196 ceph-mon[33018]: 5.1b scrub ok 2024-02-05T08:15:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:24 smithi196 ceph-mon[33018]: 6.2 scrub starts 2024-02-05T08:15:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:24 smithi196 ceph-mon[33018]: 6.2 scrub ok 2024-02-05T08:15:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:24 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3533470215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:24 smithi196 ceph-mon[33018]: 4.17 scrub starts 2024-02-05T08:15:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:24 smithi196 ceph-mon[33018]: 4.17 scrub ok 2024-02-05T08:15:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:24 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:24.374 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:24.416 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:24.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:24.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 10 16 26800 26784 167.345 138.188 0.00661916 0.00596972 2024-02-05T08:15:24.489 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[30086]: 3.1b scrub starts 2024-02-05T08:15:24.489 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[30086]: 3.1b scrub ok 2024-02-05T08:15:24.489 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[30086]: 5.1b scrub starts 2024-02-05T08:15:24.489 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[30086]: 5.1b scrub ok 2024-02-05T08:15:24.489 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[30086]: 6.2 scrub starts 2024-02-05T08:15:24.489 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[30086]: 6.2 scrub ok 2024-02-05T08:15:24.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3533470215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:24.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[30086]: 4.17 scrub starts 2024-02-05T08:15:24.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[30086]: 4.17 scrub ok 2024-02-05T08:15:24.490 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:24.490 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[39278]: 3.1b scrub starts 2024-02-05T08:15:24.490 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[39278]: 3.1b scrub ok 2024-02-05T08:15:24.490 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[39278]: 5.1b scrub starts 2024-02-05T08:15:24.490 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[39278]: 5.1b scrub ok 2024-02-05T08:15:24.490 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[39278]: 6.2 scrub starts 2024-02-05T08:15:24.491 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[39278]: 6.2 scrub ok 2024-02-05T08:15:24.491 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3533470215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:24.491 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[39278]: 4.17 scrub starts 2024-02-05T08:15:24.491 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[39278]: 4.17 scrub ok 2024-02-05T08:15:24.491 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:24 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:24.518 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:24.581 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:24.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:24.617 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:24.623 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:24.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:24.722 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:24.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:24.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:24.858 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:24.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:24.960 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:24.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:24.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:24.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:25.100 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:25.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:25.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:25.225 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:25.262 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:25.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:25.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[30086]: 3.12 scrub starts 2024-02-05T08:15:25.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[30086]: 3.12 scrub ok 2024-02-05T08:15:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[30086]: pgmap v471: 148 pgs: 148 active+clean; 1.4 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 213 MiB/s wr, 3.41k op/s 2024-02-05T08:15:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[30086]: 4.10 scrub starts 2024-02-05T08:15:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[30086]: 4.10 scrub ok 2024-02-05T08:15:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[30086]: 4.1 scrub starts 2024-02-05T08:15:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[30086]: 4.1 scrub ok 2024-02-05T08:15:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[39278]: 3.12 scrub starts 2024-02-05T08:15:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[39278]: 3.12 scrub ok 2024-02-05T08:15:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[39278]: pgmap v471: 148 pgs: 148 active+clean; 1.4 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 213 MiB/s wr, 3.41k op/s 2024-02-05T08:15:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[39278]: 4.10 scrub starts 2024-02-05T08:15:25.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[39278]: 4.10 scrub ok 2024-02-05T08:15:25.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[39278]: 4.1 scrub starts 2024-02-05T08:15:25.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:25 smithi134 ceph-mon[39278]: 4.1 scrub ok 2024-02-05T08:15:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:25 smithi196 ceph-mon[33018]: 3.12 scrub starts 2024-02-05T08:15:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:25 smithi196 ceph-mon[33018]: 3.12 scrub ok 2024-02-05T08:15:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:25 smithi196 ceph-mon[33018]: pgmap v471: 148 pgs: 148 active+clean; 1.4 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 213 MiB/s wr, 3.41k op/s 2024-02-05T08:15:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:25 smithi196 ceph-mon[33018]: 4.10 scrub starts 2024-02-05T08:15:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:25 smithi196 ceph-mon[33018]: 4.10 scrub ok 2024-02-05T08:15:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:25 smithi196 ceph-mon[33018]: 4.1 scrub starts 2024-02-05T08:15:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:25 smithi196 ceph-mon[33018]: 4.1 scrub ok 2024-02-05T08:15:25.364 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:25.396 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:25.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:25.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:25.480 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 11 16 28376 28360 161.087 98.5 0.0034708 0.00620173 2024-02-05T08:15:25.498 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:25.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:25.535 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:25.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:25.636 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:25.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:25.672 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:25.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:25.773 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:25.807 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:25.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:25.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:25.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:25.947 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:25.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:26.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:26.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:26.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:26.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:26.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:26.220 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T08:15:26.230 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:26.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:26 smithi196 ceph-mon[33018]: 4.19 scrub starts 2024-02-05T08:15:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:26 smithi196 ceph-mon[33018]: 4.19 scrub ok 2024-02-05T08:15:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:26 smithi196 ceph-mon[33018]: 4.13 deep-scrub starts 2024-02-05T08:15:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:26 smithi196 ceph-mon[33018]: 4.13 deep-scrub ok 2024-02-05T08:15:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:26 smithi196 ceph-mon[33018]: 3.11 deep-scrub starts 2024-02-05T08:15:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:26 smithi196 ceph-mon[33018]: 3.11 deep-scrub ok 2024-02-05T08:15:26.332 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:26.365 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:26.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:26.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:26.480 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 12 16 30298 30282 157.673 120.125 0.0108099 0.00633636 2024-02-05T08:15:26.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:26.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:26.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:26 smithi134 ceph-mon[30086]: 4.19 scrub starts 2024-02-05T08:15:26.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:26 smithi134 ceph-mon[30086]: 4.19 scrub ok 2024-02-05T08:15:26.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:26 smithi134 ceph-mon[30086]: 4.13 deep-scrub starts 2024-02-05T08:15:26.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:26 smithi134 ceph-mon[30086]: 4.13 deep-scrub ok 2024-02-05T08:15:26.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:26 smithi134 ceph-mon[30086]: 3.11 deep-scrub starts 2024-02-05T08:15:26.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:26 smithi134 ceph-mon[30086]: 3.11 deep-scrub ok 2024-02-05T08:15:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:26 smithi134 ceph-mon[39278]: 4.19 scrub starts 2024-02-05T08:15:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:26 smithi134 ceph-mon[39278]: 4.19 scrub ok 2024-02-05T08:15:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:26 smithi134 ceph-mon[39278]: 4.13 deep-scrub starts 2024-02-05T08:15:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:26 smithi134 ceph-mon[39278]: 4.13 deep-scrub ok 2024-02-05T08:15:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:26 smithi134 ceph-mon[39278]: 3.11 deep-scrub starts 2024-02-05T08:15:26.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:26 smithi134 ceph-mon[39278]: 3.11 deep-scrub ok 2024-02-05T08:15:26.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:15:26.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:26.650 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:26.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:26.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:26.659 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:15:26.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:26.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:26.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:26.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:26.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:26.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:26.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:27.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:27.057 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:27.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:27.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:27.159 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:27.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:27.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:27.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[30086]: 6.16 scrub starts 2024-02-05T08:15:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[30086]: 6.16 scrub ok 2024-02-05T08:15:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[30086]: pgmap v472: 148 pgs: 148 active+clean; 1.4 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 764 B/s rd, 112 MiB/s wr, 1.80k op/s 2024-02-05T08:15:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:15:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:15:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[30086]: 5.1c scrub starts 2024-02-05T08:15:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[30086]: 5.1c scrub ok 2024-02-05T08:15:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[39278]: 6.16 scrub starts 2024-02-05T08:15:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[39278]: 6.16 scrub ok 2024-02-05T08:15:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[39278]: pgmap v472: 148 pgs: 148 active+clean; 1.4 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 764 B/s rd, 112 MiB/s wr, 1.80k op/s 2024-02-05T08:15:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:15:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:15:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[39278]: 5.1c scrub starts 2024-02-05T08:15:27.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-mon[39278]: 5.1c scrub ok 2024-02-05T08:15:27.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:27.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:27 smithi196 ceph-mon[33018]: 6.16 scrub starts 2024-02-05T08:15:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:27 smithi196 ceph-mon[33018]: 6.16 scrub ok 2024-02-05T08:15:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:27 smithi196 ceph-mon[33018]: pgmap v472: 148 pgs: 148 active+clean; 1.4 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 764 B/s rd, 112 MiB/s wr, 1.80k op/s 2024-02-05T08:15:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:27 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:15:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:27 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:15:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:27 smithi196 ceph-mon[33018]: 5.1c scrub starts 2024-02-05T08:15:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:27 smithi196 ceph-mon[33018]: 5.1c scrub ok 2024-02-05T08:15:27.318 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:27.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:27.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:27.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:27.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:27.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:27.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:27.480 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 13 16 31959 31943 153.53 103.812 0.00510248 0.0065074 2024-02-05T08:15:27.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:27.587 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:27.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:27.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:27.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:27.713 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:27.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:27.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:27.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:27.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:27.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:27.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:27.953 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:27.983 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:27.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:27.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:28.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:15:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:15:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:15:28.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:28.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:28.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:28.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:28.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:28.259 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:28.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:28.308 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:15:28.309 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:28 smithi134 ceph-mon[30086]: 4.1e scrub starts 2024-02-05T08:15:28.309 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:28 smithi134 ceph-mon[30086]: 4.1e scrub ok 2024-02-05T08:15:28.309 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:28 smithi134 ceph-mon[30086]: 5.14 scrub starts 2024-02-05T08:15:28.309 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:28 smithi134 ceph-mon[30086]: 5.14 scrub ok 2024-02-05T08:15:28.311 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:28 smithi134 ceph-mon[39278]: 4.1e scrub starts 2024-02-05T08:15:28.311 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:28 smithi134 ceph-mon[39278]: 4.1e scrub ok 2024-02-05T08:15:28.311 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:28 smithi134 ceph-mon[39278]: 5.14 scrub starts 2024-02-05T08:15:28.311 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:28 smithi134 ceph-mon[39278]: 5.14 scrub ok 2024-02-05T08:15:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:28 smithi196 ceph-mon[33018]: 4.1e scrub starts 2024-02-05T08:15:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:28 smithi196 ceph-mon[33018]: 4.1e scrub ok 2024-02-05T08:15:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:28 smithi196 ceph-mon[33018]: 5.14 scrub starts 2024-02-05T08:15:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:28 smithi196 ceph-mon[33018]: 5.14 scrub ok 2024-02-05T08:15:28.319 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:15:28.361 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:28.387 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:28.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:28.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:28.480 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 14 16 33497 33481 149.43 96.125 0.00353356 0.00668705 2024-02-05T08:15:28.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:28.528 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:28.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:28.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:28.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:28.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:28.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:28.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:28.766 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:28.795 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:28.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:28.864 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:28.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:28.939 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:28.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:28.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:29.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:29.075 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:29.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:29.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:29.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:29.213 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:29.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:29.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:29 smithi134 ceph-mon[30086]: pgmap v473: 148 pgs: 148 active+clean; 1.5 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-05T08:15:29.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:29 smithi134 ceph-mon[30086]: from='client.25247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:15:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:29 smithi134 ceph-mon[39278]: pgmap v473: 148 pgs: 148 active+clean; 1.5 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-05T08:15:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:29 smithi134 ceph-mon[39278]: from='client.25247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:15:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:29 smithi196 ceph-mon[33018]: pgmap v473: 148 pgs: 148 active+clean; 1.5 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-05T08:15:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:29 smithi196 ceph-mon[33018]: from='client.25247 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:15:29.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:29.338 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:29.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:29.440 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:29.477 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:29.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:29.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:29.480 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 15 16 35171 35155 146.443 104.625 0.00607675 0.00682367 2024-02-05T08:15:29.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:29.613 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:29.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:29.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:29.651 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-02-05T08:15:27.709108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1629231452,"num_objects":25049,"num_object_clones":0,"num_object_copies":75147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25049,"num_whiteouts":0,"num_read":1987,"num_read_kb":1833,"num_write":25752,"num_write_kb":1592925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":2957685,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25815,"ondisk_log_size":25815,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":7025180,"kb_used_data":4746656,"kb_used_omap":10,"kb_used_meta":2278389,"kb_avail":649190884,"statfs":{"total":671965249536,"available":664771465216,"internally_reserved":0,"allocated":4860575744,"data_stored":4857825128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":2333070442},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":864419840,"num_objects":13190,"num_object_clones":0,"num_object_copies":39570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13190,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":13190,"num_write_kb":844160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002333"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":406,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.027131+0000","last_change":"2024-02-05T08:15:16.927873+0000","last_active":"2024-02-05T08:15:22.027131+0000","last_peered":"2024-02-05T08:15:22.027131+0000","last_clean":"2024-02-05T08:15:22.027131+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:15:22.027131+0000","last_undegraded":"2024-02-05T08:15:22.027131+0000","last_fullsized":"2024-02-05T08:15:22.027131+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:16.927723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:16.927723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:39.613027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.027812+0000","last_change":"2024-02-05T08:15:21.824643+0000","last_active":"2024-02-05T08:15:22.027812+0000","last_peered":"2024-02-05T08:15:22.027812+0000","last_clean":"2024-02-05T08:15:22.027812+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:15:22.027812+0000","last_undegraded":"2024-02-05T08:15:22.027812+0000","last_fullsized":"2024-02-05T08:15:22.027812+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:21.824531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:21.824531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:31.555157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":410,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031119+0000","last_change":"2024-02-05T08:14:10.178234+0000","last_active":"2024-02-05T08:15:22.031119+0000","last_peered":"2024-02-05T08:15:22.031119+0000","last_clean":"2024-02-05T08:15:22.031119+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:15:22.031119+0000","last_undegraded":"2024-02-05T08:15:22.031119+0000","last_fullsized":"2024-02-05T08:15:22.031119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:10.178136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:10.178136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"236'19","reported_seq":467,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.026974+0000","last_change":"2024-02-05T08:14:03.970129+0000","last_active":"2024-02-05T08:15:22.026974+0000","last_peered":"2024-02-05T08:15:22.026974+0000","last_clean":"2024-02-05T08:15:22.026974+0000","last_became_active":"2024-02-05T08:12:29.675863+0000","last_became_peered":"2024-02-05T08:12:29.675863+0000","last_unstale":"2024-02-05T08:15:22.026974+0000","last_undegraded":"2024-02-05T08:15:22.026974+0000","last_fullsized":"2024-02-05T08:15:22.026974+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:14:03.970014+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:14:03.970014+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:26.194270+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":401,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.548406+0000","last_change":"2024-02-05T08:15:22.548444+0000","last_active":"2024-02-05T08:15:22.548406+0000","last_peered":"2024-02-05T08:15:22.548406+0000","last_clean":"2024-02-05T08:15:22.548406+0000","last_became_active":"2024-02-05T08:12:41.722414+0000","last_became_peered":"2024-02-05T08:12:41.722414+0000","last_unstale":"2024-02-05T08:15:22.548406+0000","last_undegraded":"2024-02-05T08:15:22.548406+0000","last_fullsized":"2024-02-05T08:15:22.548406+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.548159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:38.794097+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":414,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.027203+0000","last_change":"2024-02-05T08:14:13.968249+0000","last_active":"2024-02-05T08:15:22.027203+0000","last_peered":"2024-02-05T08:15:22.027203+0000","last_clean":"2024-02-05T08:15:22.027203+0000","last_became_active":"2024-02-05T08:12:29.676482+0000","last_became_peered":"2024-02-05T08:12:29.676482+0000","last_unstale":"2024-02-05T08:15:22.027203+0000","last_undegraded":"2024-02-05T08:15:22.027203+0000","last_fullsized":"2024-02-05T08:15:22.027203+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:13.968037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:14:13.968037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.507067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":128,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031490+0000","last_change":"2024-02-05T08:14:09.180369+0000","last_active":"2024-02-05T08:15:22.031490+0000","last_peered":"2024-02-05T08:15:22.031490+0000","last_clean":"2024-02-05T08:15:22.031490+0000","last_became_active":"2024-02-05T08:12:41.727450+0000","last_became_peered":"2024-02-05T08:12:41.727450+0000","last_unstale":"2024-02-05T08:15:22.031490+0000","last_undegraded":"2024-02-05T08:15:22.031490+0000","last_fullsized":"2024-02-05T08:15:22.031490+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:09.180097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:28.007713+0000","last_clean_scrub_stamp":"2024-02-05T08:14:09.180097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:23.976666+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"236'34","reported_seq":309,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.036148+0000","last_change":"2024-02-05T08:14:10.776549+0000","last_active":"2024-02-05T08:15:22.036148+0000","last_peered":"2024-02-05T08:15:22.036148+0000","last_clean":"2024-02-05T08:15:22.036148+0000","last_became_active":"2024-02-05T08:12:41.745917+0000","last_became_peered":"2024-02-05T08:12:41.745917+0000","last_unstale":"2024-02-05T08:15:22.036148+0000","last_undegraded":"2024-02-05T08:15:22.036148+0000","last_fullsized":"2024-02-05T08:15:22.036148+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'22","last_scrub_stamp":"2024-02-05T08:14:10.776137+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:14:10.776137+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":397,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032251+0000","last_change":"2024-02-05T08:14:14.237605+0000","last_active":"2024-02-05T08:15:22.032251+0000","last_peered":"2024-02-05T08:15:22.032251+0000","last_clean":"2024-02-05T08:15:22.032251+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:15:22.032251+0000","last_undegraded":"2024-02-05T08:15:22.032251+0000","last_fullsized":"2024-02-05T08:15:22.032251+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:14.237096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:14.237096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":402,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033646+0000","last_change":"2024-02-05T08:12:15.282974+0000","last_active":"2024-02-05T08:15:22.033646+0000","last_peered":"2024-02-05T08:15:22.033646+0000","last_clean":"2024-02-05T08:15:22.033646+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:15:22.033646+0000","last_undegraded":"2024-02-05T08:15:22.033646+0000","last_fullsized":"2024-02-05T08:15:22.033646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:15.282863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:12:15.282863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":430,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034843+0000","last_change":"2024-02-05T08:14:27.200067+0000","last_active":"2024-02-05T08:15:22.034843+0000","last_peered":"2024-02-05T08:15:22.034843+0000","last_clean":"2024-02-05T08:15:22.034843+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:15:22.034843+0000","last_undegraded":"2024-02-05T08:15:22.034843+0000","last_fullsized":"2024-02-05T08:15:22.034843+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:27.199962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:27.199962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:32.610560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"236'15","reported_seq":453,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.027343+0000","last_change":"2024-02-05T08:14:16.970924+0000","last_active":"2024-02-05T08:15:22.027343+0000","last_peered":"2024-02-05T08:15:22.027343+0000","last_clean":"2024-02-05T08:15:22.027343+0000","last_became_active":"2024-02-05T08:12:41.731452+0000","last_became_peered":"2024-02-05T08:12:41.731452+0000","last_unstale":"2024-02-05T08:15:22.027343+0000","last_undegraded":"2024-02-05T08:15:22.027343+0000","last_fullsized":"2024-02-05T08:15:22.027343+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:14:16.970843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:16.970843+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:32.592772+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":404,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033793+0000","last_change":"2024-02-05T08:12:41.722105+0000","last_active":"2024-02-05T08:15:22.033793+0000","last_peered":"2024-02-05T08:15:22.033793+0000","last_clean":"2024-02-05T08:15:22.033793+0000","last_became_active":"2024-02-05T08:12:41.721611+0000","last_became_peered":"2024-02-05T08:12:41.721611+0000","last_unstale":"2024-02-05T08:15:22.033793+0000","last_undegraded":"2024-02-05T08:15:22.033793+0000","last_fullsized":"2024-02-05T08:15:22.033793+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:14.301656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:14.301656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033349+0000","last_change":"2024-02-05T08:15:18.440781+0000","last_active":"2024-02-05T08:15:22.033349+0000","last_peered":"2024-02-05T08:15:22.033349+0000","last_clean":"2024-02-05T08:15:22.033349+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:15:22.033349+0000","last_undegraded":"2024-02-05T08:15:22.033349+0000","last_fullsized":"2024-02-05T08:15:22.033349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:18.440664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:18.440664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:34.989634+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033718+0000","last_change":"2024-02-05T08:15:20.904492+0000","last_active":"2024-02-05T08:15:22.033718+0000","last_peered":"2024-02-05T08:15:22.033718+0000","last_clean":"2024-02-05T08:15:22.033718+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:15:22.033718+0000","last_undegraded":"2024-02-05T08:15:22.033718+0000","last_fullsized":"2024-02-05T08:15:22.033718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:20.904369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:15:20.904369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:21.716191+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":431,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.028075+0000","last_change":"2024-02-05T08:14:18.923493+0000","last_active":"2024-02-05T08:15:22.028075+0000","last_peered":"2024-02-05T08:15:22.028075+0000","last_clean":"2024-02-05T08:15:22.028075+0000","last_became_active":"2024-02-05T08:12:41.730485+0000","last_became_peered":"2024-02-05T08:12:41.730485+0000","last_unstale":"2024-02-05T08:15:22.028075+0000","last_undegraded":"2024-02-05T08:15:22.028075+0000","last_fullsized":"2024-02-05T08:15:22.028075+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:14:18.923276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:18.923276+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:32.127559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":419,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.028974+0000","last_change":"2024-02-05T08:15:13.039454+0000","last_active":"2024-02-05T08:15:22.028974+0000","last_peered":"2024-02-05T08:15:22.028974+0000","last_clean":"2024-02-05T08:15:22.028974+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:15:22.028974+0000","last_undegraded":"2024-02-05T08:15:22.028974+0000","last_fullsized":"2024-02-05T08:15:22.028974+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:13.039321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:15:13.039321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:40.716512+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":403,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033586+0000","last_change":"2024-02-05T08:15:11.836173+0000","last_active":"2024-02-05T08:15:22.033586+0000","last_peered":"2024-02-05T08:15:22.033586+0000","last_clean":"2024-02-05T08:15:22.033586+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:15:22.033586+0000","last_undegraded":"2024-02-05T08:15:22.033586+0000","last_fullsized":"2024-02-05T08:15:22.033586+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:11.836098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:15:11.836098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:20.798265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":409,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.341564+0000","last_change":"2024-02-05T08:15:22.341597+0000","last_active":"2024-02-05T08:15:22.341564+0000","last_peered":"2024-02-05T08:15:22.341564+0000","last_clean":"2024-02-05T08:15:22.341564+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:15:22.341564+0000","last_undegraded":"2024-02-05T08:15:22.341564+0000","last_fullsized":"2024-02-05T08:15:22.341564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.341491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.341491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:41.569521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"236'34","reported_seq":494,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.036882+0000","last_change":"2024-02-05T08:14:31.867150+0000","last_active":"2024-02-05T08:15:22.036882+0000","last_peered":"2024-02-05T08:15:22.036882+0000","last_clean":"2024-02-05T08:15:22.036882+0000","last_became_active":"2024-02-05T08:12:41.727027+0000","last_became_peered":"2024-02-05T08:12:41.727027+0000","last_unstale":"2024-02-05T08:15:22.036882+0000","last_undegraded":"2024-02-05T08:15:22.036882+0000","last_fullsized":"2024-02-05T08:15:22.036882+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'22","last_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.867008+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:52.313837+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":399,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031808+0000","last_change":"2024-02-05T08:14:33.244556+0000","last_active":"2024-02-05T08:15:22.031808+0000","last_peered":"2024-02-05T08:15:22.031808+0000","last_clean":"2024-02-05T08:15:22.031808+0000","last_became_active":"2024-02-05T08:12:41.713564+0000","last_became_peered":"2024-02-05T08:12:41.713564+0000","last_unstale":"2024-02-05T08:15:22.031808+0000","last_undegraded":"2024-02-05T08:15:22.031808+0000","last_fullsized":"2024-02-05T08:15:22.031808+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:33.244471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:33.244471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:39.761831+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":416,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:25.922654+0000","last_change":"2024-02-05T08:15:25.922691+0000","last_active":"2024-02-05T08:15:25.922654+0000","last_peered":"2024-02-05T08:15:25.922654+0000","last_clean":"2024-02-05T08:15:25.922654+0000","last_became_active":"2024-02-05T08:12:41.735240+0000","last_became_peered":"2024-02-05T08:12:41.735240+0000","last_unstale":"2024-02-05T08:15:25.922654+0000","last_undegraded":"2024-02-05T08:15:25.922654+0000","last_fullsized":"2024-02-05T08:15:25.922654+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.922139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.922139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:46.247310+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":410,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.027450+0000","last_change":"2024-02-05T08:15:19.927561+0000","last_active":"2024-02-05T08:15:22.027450+0000","last_peered":"2024-02-05T08:15:22.027450+0000","last_clean":"2024-02-05T08:15:22.027450+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:15:22.027450+0000","last_undegraded":"2024-02-05T08:15:22.027450+0000","last_fullsized":"2024-02-05T08:15:22.027450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:19.927454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:19.927454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:22.894101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"236'23","reported_seq":230,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.028394+0000","last_change":"2024-02-05T08:14:09.718995+0000","last_active":"2024-02-05T08:15:22.028394+0000","last_peered":"2024-02-05T08:15:22.028394+0000","last_clean":"2024-02-05T08:15:22.028394+0000","last_became_active":"2024-02-05T08:12:41.716794+0000","last_became_peered":"2024-02-05T08:12:41.716794+0000","last_unstale":"2024-02-05T08:15:22.028394+0000","last_undegraded":"2024-02-05T08:15:22.028394+0000","last_fullsized":"2024-02-05T08:15:22.028394+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:14:09.718864+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:14:09.718864+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":417,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033940+0000","last_change":"2024-02-05T08:14:26.616349+0000","last_active":"2024-02-05T08:15:22.033940+0000","last_peered":"2024-02-05T08:15:22.033940+0000","last_clean":"2024-02-05T08:15:22.033940+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:15:22.033940+0000","last_undegraded":"2024-02-05T08:15:22.033940+0000","last_fullsized":"2024-02-05T08:15:22.033940+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:14:26.616223+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:14:26.616223+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:34.032614+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":180,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032154+0000","last_change":"2024-02-05T08:14:18.174978+0000","last_active":"2024-02-05T08:15:22.032154+0000","last_peered":"2024-02-05T08:15:22.032154+0000","last_clean":"2024-02-05T08:15:22.032154+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:15:22.032154+0000","last_undegraded":"2024-02-05T08:15:22.032154+0000","last_fullsized":"2024-02-05T08:15:22.032154+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:14:18.174860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:18.174860+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:33.966481+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":422,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.027727+0000","last_change":"2024-02-05T08:14:10.699857+0000","last_active":"2024-02-05T08:15:22.027727+0000","last_peered":"2024-02-05T08:15:22.027727+0000","last_clean":"2024-02-05T08:15:22.027727+0000","last_became_active":"2024-02-05T08:12:41.719010+0000","last_became_peered":"2024-02-05T08:12:41.719010+0000","last_unstale":"2024-02-05T08:15:22.027727+0000","last_undegraded":"2024-02-05T08:15:22.027727+0000","last_fullsized":"2024-02-05T08:15:22.027727+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:14:10.699732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:10.699732+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:31.297479+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"236'20","reported_seq":482,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031420+0000","last_change":"2024-02-05T08:12:41.775561+0000","last_active":"2024-02-05T08:15:22.031420+0000","last_peered":"2024-02-05T08:15:22.031420+0000","last_clean":"2024-02-05T08:15:22.031420+0000","last_became_active":"2024-02-05T08:12:41.757718+0000","last_became_peered":"2024-02-05T08:12:41.757718+0000","last_unstale":"2024-02-05T08:15:22.031420+0000","last_undegraded":"2024-02-05T08:15:22.031420+0000","last_fullsized":"2024-02-05T08:15:22.031420+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:12:18.538594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:18.538594+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":399,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033718+0000","last_change":"2024-02-05T08:11:33.834287+0000","last_active":"2024-02-05T08:15:22.033718+0000","last_peered":"2024-02-05T08:15:22.033718+0000","last_clean":"2024-02-05T08:15:22.033718+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:15:22.033718+0000","last_undegraded":"2024-02-05T08:15:22.033718+0000","last_fullsized":"2024-02-05T08:15:22.033718+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:33.834034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:33.834034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":402,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032158+0000","last_change":"2024-02-05T08:14:32.232117+0000","last_active":"2024-02-05T08:15:22.032158+0000","last_peered":"2024-02-05T08:15:22.032158+0000","last_clean":"2024-02-05T08:15:22.032158+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:15:22.032158+0000","last_undegraded":"2024-02-05T08:15:22.032158+0000","last_fullsized":"2024-02-05T08:15:22.032158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:32.231861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:32.231861+0000","last_clean_scrub_stamp":"2024-02-05T08:14:32.231861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.000116+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":130,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032118+0000","last_change":"2024-02-05T08:14:31.199352+0000","last_active":"2024-02-05T08:15:22.032118+0000","last_peered":"2024-02-05T08:15:22.032118+0000","last_clean":"2024-02-05T08:15:22.032118+0000","last_became_active":"2024-02-05T08:12:41.727406+0000","last_became_peered":"2024-02-05T08:12:41.727406+0000","last_unstale":"2024-02-05T08:15:22.032118+0000","last_undegraded":"2024-02-05T08:15:22.032118+0000","last_fullsized":"2024-02-05T08:15:22.032118+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:31.199238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.199238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:31.570968+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"236'19","reported_seq":164,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031892+0000","last_change":"2024-02-05T08:14:26.230319+0000","last_active":"2024-02-05T08:15:22.031892+0000","last_peered":"2024-02-05T08:15:22.031892+0000","last_clean":"2024-02-05T08:15:22.031892+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:15:22.031892+0000","last_undegraded":"2024-02-05T08:15:22.031892+0000","last_fullsized":"2024-02-05T08:15:22.031892+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:14:26.230191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:26.230191+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:44.242536+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":399,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033614+0000","last_change":"2024-02-05T08:13:35.143784+0000","last_active":"2024-02-05T08:15:22.033614+0000","last_peered":"2024-02-05T08:15:22.033614+0000","last_clean":"2024-02-05T08:15:22.033614+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:15:22.033614+0000","last_undegraded":"2024-02-05T08:15:22.033614+0000","last_fullsized":"2024-02-05T08:15:22.033614+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:35.143602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:13:35.143602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":140,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.030545+0000","last_change":"2024-02-05T08:15:12.041843+0000","last_active":"2024-02-05T08:15:22.030545+0000","last_peered":"2024-02-05T08:15:22.030545+0000","last_clean":"2024-02-05T08:15:22.030545+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:15:22.030545+0000","last_undegraded":"2024-02-05T08:15:22.030545+0000","last_fullsized":"2024-02-05T08:15:22.030545+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:12.041759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:15:12.041759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:41.328162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":415,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031577+0000","last_change":"2024-02-05T08:15:11.364628+0000","last_active":"2024-02-05T08:15:22.031577+0000","last_peered":"2024-02-05T08:15:22.031577+0000","last_clean":"2024-02-05T08:15:22.031577+0000","last_became_active":"2024-02-05T08:12:41.736991+0000","last_became_peered":"2024-02-05T08:12:41.736991+0000","last_unstale":"2024-02-05T08:15:22.031577+0000","last_undegraded":"2024-02-05T08:15:22.031577+0000","last_fullsized":"2024-02-05T08:15:22.031577+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:11.364552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:11.364552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:18.417672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":160,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.029632+0000","last_change":"2024-02-05T08:12:41.743649+0000","last_active":"2024-02-05T08:15:22.029632+0000","last_peered":"2024-02-05T08:15:22.029632+0000","last_clean":"2024-02-05T08:15:22.029632+0000","last_became_active":"2024-02-05T08:12:41.717459+0000","last_became_peered":"2024-02-05T08:12:41.717459+0000","last_unstale":"2024-02-05T08:15:22.029632+0000","last_undegraded":"2024-02-05T08:15:22.029632+0000","last_fullsized":"2024-02-05T08:15:22.029632+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-02-05T08:11:32.785856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:11:32.785856+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":392,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032487+0000","last_change":"2024-02-05T08:12:41.738206+0000","last_active":"2024-02-05T08:15:22.032487+0000","last_peered":"2024-02-05T08:15:22.032487+0000","last_clean":"2024-02-05T08:15:22.032487+0000","last_became_active":"2024-02-05T08:12:41.737803+0000","last_became_peered":"2024-02-05T08:12:41.737803+0000","last_unstale":"2024-02-05T08:15:22.032487+0000","last_undegraded":"2024-02-05T08:15:22.032487+0000","last_fullsized":"2024-02-05T08:15:22.032487+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:33.610823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:33.610823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":423,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033056+0000","last_change":"2024-02-05T08:13:19.251251+0000","last_active":"2024-02-05T08:15:22.033056+0000","last_peered":"2024-02-05T08:15:22.033056+0000","last_clean":"2024-02-05T08:15:22.033056+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:15:22.033056+0000","last_undegraded":"2024-02-05T08:15:22.033056+0000","last_fullsized":"2024-02-05T08:15:22.033056+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:19.251134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:13:19.251134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034258+0000","last_change":"2024-02-05T08:13:10.189057+0000","last_active":"2024-02-05T08:15:22.034258+0000","last_peered":"2024-02-05T08:15:22.034258+0000","last_clean":"2024-02-05T08:15:22.034258+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:15:22.034258+0000","last_undegraded":"2024-02-05T08:15:22.034258+0000","last_fullsized":"2024-02-05T08:15:22.034258+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:10.188977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:13:10.188977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"236'24","reported_seq":367,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034170+0000","last_change":"2024-02-05T08:14:17.606299+0000","last_active":"2024-02-05T08:15:22.034170+0000","last_peered":"2024-02-05T08:15:22.034170+0000","last_clean":"2024-02-05T08:15:22.034170+0000","last_became_active":"2024-02-05T08:12:41.718651+0000","last_became_peered":"2024-02-05T08:12:41.718651+0000","last_unstale":"2024-02-05T08:15:22.034170+0000","last_undegraded":"2024-02-05T08:15:22.034170+0000","last_fullsized":"2024-02-05T08:15:22.034170+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'16","last_scrub_stamp":"2024-02-05T08:14:17.606161+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:14:17.606161+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:34.534470+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":397,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031768+0000","last_change":"2024-02-05T08:14:23.281718+0000","last_active":"2024-02-05T08:15:22.031768+0000","last_peered":"2024-02-05T08:15:22.031768+0000","last_clean":"2024-02-05T08:15:22.031768+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:15:22.031768+0000","last_undegraded":"2024-02-05T08:15:22.031768+0000","last_fullsized":"2024-02-05T08:15:22.031768+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:23.281493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:23.281493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:49.551355+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":131,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032568+0000","last_change":"2024-02-05T08:14:09.229078+0000","last_active":"2024-02-05T08:15:22.032568+0000","last_peered":"2024-02-05T08:15:22.032568+0000","last_clean":"2024-02-05T08:15:22.032568+0000","last_became_active":"2024-02-05T08:12:41.757901+0000","last_became_peered":"2024-02-05T08:12:41.757901+0000","last_unstale":"2024-02-05T08:15:22.032568+0000","last_undegraded":"2024-02-05T08:15:22.032568+0000","last_fullsized":"2024-02-05T08:15:22.032568+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:09.228973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:09.228973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":425,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034496+0000","last_change":"2024-02-05T08:15:10.525698+0000","last_active":"2024-02-05T08:15:22.034496+0000","last_peered":"2024-02-05T08:15:22.034496+0000","last_clean":"2024-02-05T08:15:22.034496+0000","last_became_active":"2024-02-05T08:12:41.734072+0000","last_became_peered":"2024-02-05T08:12:41.734072+0000","last_unstale":"2024-02-05T08:15:22.034496+0000","last_undegraded":"2024-02-05T08:15:22.034496+0000","last_fullsized":"2024-02-05T08:15:22.034496+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:10.525625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:10.525625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:26.616052+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"236'15","reported_seq":74,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034651+0000","last_change":"2024-02-05T08:14:24.999924+0000","last_active":"2024-02-05T08:15:22.034651+0000","last_peered":"2024-02-05T08:15:22.034651+0000","last_clean":"2024-02-05T08:15:22.034651+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:15:22.034651+0000","last_undegraded":"2024-02-05T08:15:22.034651+0000","last_fullsized":"2024-02-05T08:15:22.034651+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:14:21.660270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:21.660270+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:26.793166+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":122,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032872+0000","last_change":"2024-02-05T08:12:41.742137+0000","last_active":"2024-02-05T08:15:22.032872+0000","last_peered":"2024-02-05T08:15:22.032872+0000","last_clean":"2024-02-05T08:15:22.032872+0000","last_became_active":"2024-02-05T08:12:41.737246+0000","last_became_peered":"2024-02-05T08:12:41.737246+0000","last_unstale":"2024-02-05T08:15:22.032872+0000","last_undegraded":"2024-02-05T08:15:22.032872+0000","last_fullsized":"2024-02-05T08:15:22.032872+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.507800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.507800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":409,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.036438+0000","last_change":"2024-02-05T08:12:41.717743+0000","last_active":"2024-02-05T08:15:22.036438+0000","last_peered":"2024-02-05T08:15:22.036438+0000","last_clean":"2024-02-05T08:15:22.036438+0000","last_became_active":"2024-02-05T08:12:41.717285+0000","last_became_peered":"2024-02-05T08:12:41.717285+0000","last_unstale":"2024-02-05T08:15:22.036438+0000","last_undegraded":"2024-02-05T08:15:22.036438+0000","last_fullsized":"2024-02-05T08:15:22.036438+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:12.952558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:12.952558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":430,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.027447+0000","last_change":"2024-02-05T08:14:30.022191+0000","last_active":"2024-02-05T08:15:22.027447+0000","last_peered":"2024-02-05T08:15:22.027447+0000","last_clean":"2024-02-05T08:15:22.027447+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:15:22.027447+0000","last_undegraded":"2024-02-05T08:15:22.027447+0000","last_fullsized":"2024-02-05T08:15:22.027447+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:14:30.022077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:32.501112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"249'24","reported_seq":486,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:24.970874+0000","last_change":"2024-02-05T08:15:24.970945+0000","last_active":"2024-02-05T08:15:24.970874+0000","last_peered":"2024-02-05T08:15:24.970874+0000","last_clean":"2024-02-05T08:15:24.970874+0000","last_became_active":"2024-02-05T08:12:29.675657+0000","last_became_peered":"2024-02-05T08:12:29.675657+0000","last_unstale":"2024-02-05T08:15:24.970874+0000","last_undegraded":"2024-02-05T08:15:24.970874+0000","last_fullsized":"2024-02-05T08:15:24.970874+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:15:24.970769+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:33.443975+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":395,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031911+0000","last_change":"2024-02-05T08:12:41.737779+0000","last_active":"2024-02-05T08:15:22.031911+0000","last_peered":"2024-02-05T08:15:22.031911+0000","last_clean":"2024-02-05T08:15:22.031911+0000","last_became_active":"2024-02-05T08:12:41.736845+0000","last_became_peered":"2024-02-05T08:12:41.736845+0000","last_unstale":"2024-02-05T08:15:22.031911+0000","last_undegraded":"2024-02-05T08:15:22.031911+0000","last_fullsized":"2024-02-05T08:15:22.031911+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:31.474957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:31.474957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":547,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034925+0000","last_change":"2024-02-05T08:15:08.943424+0000","last_active":"2024-02-05T08:15:22.034925+0000","last_peered":"2024-02-05T08:15:22.034925+0000","last_clean":"2024-02-05T08:15:22.034925+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:15:22.034925+0000","last_undegraded":"2024-02-05T08:15:22.034925+0000","last_fullsized":"2024-02-05T08:15:22.034925+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:15:08.943297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:08.943297+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:10.217549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":419,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033404+0000","last_change":"2024-02-05T08:14:10.570218+0000","last_active":"2024-02-05T08:15:22.033404+0000","last_peered":"2024-02-05T08:15:22.033404+0000","last_clean":"2024-02-05T08:15:22.033404+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:15:22.033404+0000","last_undegraded":"2024-02-05T08:15:22.033404+0000","last_fullsized":"2024-02-05T08:15:22.033404+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:10.570113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:10.570113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":307,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.029468+0000","last_change":"2024-02-05T08:12:41.781948+0000","last_active":"2024-02-05T08:15:22.029468+0000","last_peered":"2024-02-05T08:15:22.029468+0000","last_clean":"2024-02-05T08:15:22.029468+0000","last_became_active":"2024-02-05T08:12:41.726861+0000","last_became_peered":"2024-02-05T08:12:41.726861+0000","last_unstale":"2024-02-05T08:15:22.029468+0000","last_undegraded":"2024-02-05T08:15:22.029468+0000","last_fullsized":"2024-02-05T08:15:22.029468+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'15","last_scrub_stamp":"2024-02-05T08:11:30.787149+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:11:30.787149+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":401,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.029005+0000","last_change":"2024-02-05T08:15:07.991636+0000","last_active":"2024-02-05T08:15:22.029005+0000","last_peered":"2024-02-05T08:15:22.029005+0000","last_clean":"2024-02-05T08:15:22.029005+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:15:22.029005+0000","last_undegraded":"2024-02-05T08:15:22.029005+0000","last_fullsized":"2024-02-05T08:15:22.029005+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:07.991536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:07.991536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:09.960940+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"194'9","reported_seq":193,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033842+0000","last_change":"2024-02-05T08:13:09.130164+0000","last_active":"2024-02-05T08:15:22.033842+0000","last_peered":"2024-02-05T08:15:22.033842+0000","last_clean":"2024-02-05T08:15:22.033842+0000","last_became_active":"2024-02-05T08:12:41.728687+0000","last_became_peered":"2024-02-05T08:12:41.728687+0000","last_unstale":"2024-02-05T08:15:22.033842+0000","last_undegraded":"2024-02-05T08:15:22.033842+0000","last_fullsized":"2024-02-05T08:15:22.033842+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'9","last_scrub_stamp":"2024-02-05T08:13:09.130041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:13:09.130041+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":415,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:23.494377+0000","last_change":"2024-02-05T08:15:23.494414+0000","last_active":"2024-02-05T08:15:23.494377+0000","last_peered":"2024-02-05T08:15:23.494377+0000","last_clean":"2024-02-05T08:15:23.494377+0000","last_became_active":"2024-02-05T08:12:41.723383+0000","last_became_peered":"2024-02-05T08:12:41.723383+0000","last_unstale":"2024-02-05T08:15:23.494377+0000","last_undegraded":"2024-02-05T08:15:23.494377+0000","last_fullsized":"2024-02-05T08:15:23.494377+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:23.494299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:15:23.494299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:33.179614+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":455,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.035162+0000","last_change":"2024-02-05T08:14:35.244918+0000","last_active":"2024-02-05T08:15:22.035162+0000","last_peered":"2024-02-05T08:15:22.035162+0000","last_clean":"2024-02-05T08:15:22.035162+0000","last_became_active":"2024-02-05T08:12:41.723526+0000","last_became_peered":"2024-02-05T08:12:41.723526+0000","last_unstale":"2024-02-05T08:15:22.035162+0000","last_undegraded":"2024-02-05T08:15:22.035162+0000","last_fullsized":"2024-02-05T08:15:22.035162+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:14:35.244359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:35.244359+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:45.975379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":414,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.037192+0000","last_change":"2024-02-05T08:15:17.707705+0000","last_active":"2024-02-05T08:15:22.037192+0000","last_peered":"2024-02-05T08:15:22.037192+0000","last_clean":"2024-02-05T08:15:22.037192+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:15:22.037192+0000","last_undegraded":"2024-02-05T08:15:22.037192+0000","last_fullsized":"2024-02-05T08:15:22.037192+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:17.707565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:17.707565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:29.022820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":132,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033833+0000","last_change":"2024-02-05T08:14:05.758633+0000","last_active":"2024-02-05T08:15:22.033833+0000","last_peered":"2024-02-05T08:15:22.033833+0000","last_clean":"2024-02-05T08:15:22.033833+0000","last_became_active":"2024-02-05T08:12:41.721852+0000","last_became_peered":"2024-02-05T08:12:41.721852+0000","last_unstale":"2024-02-05T08:15:22.033833+0000","last_undegraded":"2024-02-05T08:15:22.033833+0000","last_fullsized":"2024-02-05T08:15:22.033833+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:05.758519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:05.758519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:30.913765+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":423,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034015+0000","last_change":"2024-02-05T08:13:12.164416+0000","last_active":"2024-02-05T08:15:22.034015+0000","last_peered":"2024-02-05T08:15:22.034015+0000","last_clean":"2024-02-05T08:15:22.034015+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:15:22.034015+0000","last_undegraded":"2024-02-05T08:15:22.034015+0000","last_fullsized":"2024-02-05T08:15:22.034015+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:12.164344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:13:12.164344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"236'15","reported_seq":446,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032003+0000","last_change":"2024-02-05T08:14:29.189057+0000","last_active":"2024-02-05T08:15:22.032003+0000","last_peered":"2024-02-05T08:15:22.032003+0000","last_clean":"2024-02-05T08:15:22.032003+0000","last_became_active":"2024-02-05T08:12:41.736414+0000","last_became_peered":"2024-02-05T08:12:41.736414+0000","last_unstale":"2024-02-05T08:15:22.032003+0000","last_undegraded":"2024-02-05T08:15:22.032003+0000","last_fullsized":"2024-02-05T08:15:22.032003+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:14:29.188820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:29.188820+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:49.923284+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":419,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034403+0000","last_change":"2024-02-05T08:13:34.175520+0000","last_active":"2024-02-05T08:15:22.034403+0000","last_peered":"2024-02-05T08:15:22.034403+0000","last_clean":"2024-02-05T08:15:22.034403+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:15:22.034403+0000","last_undegraded":"2024-02-05T08:15:22.034403+0000","last_fullsized":"2024-02-05T08:15:22.034403+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:34.174842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:13:34.174842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":408,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034606+0000","last_change":"2024-02-05T08:12:41.721999+0000","last_active":"2024-02-05T08:15:22.034606+0000","last_peered":"2024-02-05T08:15:22.034606+0000","last_clean":"2024-02-05T08:15:22.034606+0000","last_became_active":"2024-02-05T08:12:41.721486+0000","last_became_peered":"2024-02-05T08:12:41.721486+0000","last_unstale":"2024-02-05T08:15:22.034606+0000","last_undegraded":"2024-02-05T08:15:22.034606+0000","last_fullsized":"2024-02-05T08:15:22.034606+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.242229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.242229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":140,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.030433+0000","last_change":"2024-02-05T08:14:17.208565+0000","last_active":"2024-02-05T08:15:22.030433+0000","last_peered":"2024-02-05T08:15:22.030433+0000","last_clean":"2024-02-05T08:15:22.030433+0000","last_became_active":"2024-02-05T08:12:29.684444+0000","last_became_peered":"2024-02-05T08:12:29.684444+0000","last_unstale":"2024-02-05T08:15:22.030433+0000","last_undegraded":"2024-02-05T08:15:22.030433+0000","last_fullsized":"2024-02-05T08:15:22.030433+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:17.208445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:17.208445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:29.451142+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'15","reported_seq":273,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034506+0000","last_change":"2024-02-05T08:13:10.202789+0000","last_active":"2024-02-05T08:15:22.034506+0000","last_peered":"2024-02-05T08:15:22.034506+0000","last_clean":"2024-02-05T08:15:22.034506+0000","last_became_active":"2024-02-05T08:12:41.732369+0000","last_became_peered":"2024-02-05T08:12:41.732369+0000","last_unstale":"2024-02-05T08:15:22.034506+0000","last_undegraded":"2024-02-05T08:15:22.034506+0000","last_fullsized":"2024-02-05T08:15:22.034506+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'9","last_scrub_stamp":"2024-02-05T08:13:10.202636+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:13:10.202636+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"249'1511","reported_seq":1541,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:23.492368+0000","last_change":"2024-02-05T08:14:57.766906+0000","last_active":"2024-02-05T08:15:23.492368+0000","last_peered":"2024-02-05T08:15:23.492368+0000","last_clean":"2024-02-05T08:15:23.492368+0000","last_became_active":"2024-02-05T08:14:57.766609+0000","last_became_peered":"2024-02-05T08:14:57.766609+0000","last_unstale":"2024-02-05T08:15:23.492368+0000","last_undegraded":"2024-02-05T08:15:23.492368+0000","last_fullsized":"2024-02-05T08:15:23.492368+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1511,"log_dups_size":0,"ondisk_log_size":1511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:57.478302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99024896,"num_objects":1511,"num_object_clones":0,"num_object_copies":4533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1511,"num_write_kb":96704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.035031+0000","last_change":"2024-02-05T08:14:28.178983+0000","last_active":"2024-02-05T08:15:22.035031+0000","last_peered":"2024-02-05T08:15:22.035031+0000","last_clean":"2024-02-05T08:15:22.035031+0000","last_became_active":"2024-02-05T08:12:29.679849+0000","last_became_peered":"2024-02-05T08:12:29.679849+0000","last_unstale":"2024-02-05T08:15:22.035031+0000","last_undegraded":"2024-02-05T08:15:22.035031+0000","last_fullsized":"2024-02-05T08:15:22.035031+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:28.178770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:28.178770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:43.924283+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":431,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034413+0000","last_change":"2024-02-05T08:14:22.627522+0000","last_active":"2024-02-05T08:15:22.034413+0000","last_peered":"2024-02-05T08:15:22.034413+0000","last_clean":"2024-02-05T08:15:22.034413+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:15:22.034413+0000","last_undegraded":"2024-02-05T08:15:22.034413+0000","last_fullsized":"2024-02-05T08:15:22.034413+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:14:22.627289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:22.627289+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:23.801525+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"236'27","reported_seq":493,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033952+0000","last_change":"2024-02-05T08:12:29.719785+0000","last_active":"2024-02-05T08:15:22.033952+0000","last_peered":"2024-02-05T08:15:22.033952+0000","last_clean":"2024-02-05T08:15:22.033952+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:15:22.033952+0000","last_undegraded":"2024-02-05T08:15:22.033952+0000","last_fullsized":"2024-02-05T08:15:22.033952+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:12:26.319958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.319958+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"190'10","reported_seq":568,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034270+0000","last_change":"2024-02-05T08:14:06.549828+0000","last_active":"2024-02-05T08:15:22.034270+0000","last_peered":"2024-02-05T08:15:22.034270+0000","last_clean":"2024-02-05T08:15:22.034270+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:15:22.034270+0000","last_undegraded":"2024-02-05T08:15:22.034270+0000","last_fullsized":"2024-02-05T08:15:22.034270+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:14:06.549691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:06.549691+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:31.450792+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"249'1571","reported_seq":1601,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:23.252286+0000","last_change":"2024-02-05T08:14:57.758422+0000","last_active":"2024-02-05T08:15:23.252286+0000","last_peered":"2024-02-05T08:15:23.252286+0000","last_clean":"2024-02-05T08:15:23.252286+0000","last_became_active":"2024-02-05T08:14:57.758070+0000","last_became_peered":"2024-02-05T08:14:57.758070+0000","last_unstale":"2024-02-05T08:15:23.252286+0000","last_undegraded":"2024-02-05T08:15:23.252286+0000","last_fullsized":"2024-02-05T08:15:23.252286+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1571,"log_dups_size":0,"ondisk_log_size":1571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:22.195699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102957056,"num_objects":1571,"num_object_clones":0,"num_object_copies":4713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1571,"num_write_kb":100544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":131,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033488+0000","last_change":"2024-02-05T08:14:31.779172+0000","last_active":"2024-02-05T08:15:22.033488+0000","last_peered":"2024-02-05T08:15:22.033488+0000","last_clean":"2024-02-05T08:15:22.033488+0000","last_became_active":"2024-02-05T08:12:41.722853+0000","last_became_peered":"2024-02-05T08:12:41.722853+0000","last_unstale":"2024-02-05T08:15:22.033488+0000","last_undegraded":"2024-02-05T08:15:22.033488+0000","last_fullsized":"2024-02-05T08:15:22.033488+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:31.779046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.779046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:44.022928+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":419,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034228+0000","last_change":"2024-02-05T08:14:14.597997+0000","last_active":"2024-02-05T08:15:22.034228+0000","last_peered":"2024-02-05T08:15:22.034228+0000","last_clean":"2024-02-05T08:15:22.034228+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:15:22.034228+0000","last_undegraded":"2024-02-05T08:15:22.034228+0000","last_fullsized":"2024-02-05T08:15:22.034228+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:14.597867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:14:14.597867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:35.633041+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":456,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.028302+0000","last_change":"2024-02-05T08:14:35.038653+0000","last_active":"2024-02-05T08:15:22.028302+0000","last_peered":"2024-02-05T08:15:22.028302+0000","last_clean":"2024-02-05T08:15:22.028302+0000","last_became_active":"2024-02-05T08:12:41.728925+0000","last_became_peered":"2024-02-05T08:12:41.728925+0000","last_unstale":"2024-02-05T08:15:22.028302+0000","last_undegraded":"2024-02-05T08:15:22.028302+0000","last_fullsized":"2024-02-05T08:15:22.028302+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:14:35.038563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:35.038563+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.757754+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.035901+0000","last_change":"2024-02-05T08:15:19.765825+0000","last_active":"2024-02-05T08:15:22.035901+0000","last_peered":"2024-02-05T08:15:22.035901+0000","last_clean":"2024-02-05T08:15:22.035901+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:15:22.035901+0000","last_undegraded":"2024-02-05T08:15:22.035901+0000","last_fullsized":"2024-02-05T08:15:22.035901+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:15:19.765714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:25.591609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"249'1479","reported_seq":1509,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.809402+0000","last_change":"2024-02-05T08:14:57.763466+0000","last_active":"2024-02-05T08:15:22.809402+0000","last_peered":"2024-02-05T08:15:22.809402+0000","last_clean":"2024-02-05T08:15:22.809402+0000","last_became_active":"2024-02-05T08:14:57.763165+0000","last_became_peered":"2024-02-05T08:14:57.763165+0000","last_unstale":"2024-02-05T08:15:22.809402+0000","last_undegraded":"2024-02-05T08:15:22.809402+0000","last_fullsized":"2024-02-05T08:15:22.809402+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1479,"log_dups_size":0,"ondisk_log_size":1479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:25.940494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96927744,"num_objects":1479,"num_object_clones":0,"num_object_copies":4437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1479,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1479,"num_write_kb":94656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":413,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032773+0000","last_change":"2024-02-05T08:15:13.442673+0000","last_active":"2024-02-05T08:15:22.032773+0000","last_peered":"2024-02-05T08:15:22.032773+0000","last_clean":"2024-02-05T08:15:22.032773+0000","last_became_active":"2024-02-05T08:12:41.724830+0000","last_became_peered":"2024-02-05T08:12:41.724830+0000","last_unstale":"2024-02-05T08:15:22.032773+0000","last_undegraded":"2024-02-05T08:15:22.032773+0000","last_fullsized":"2024-02-05T08:15:22.032773+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:13.442573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:13.442573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:28.200585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":410,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.036657+0000","last_change":"2024-02-05T08:14:33.899458+0000","last_active":"2024-02-05T08:15:22.036657+0000","last_peered":"2024-02-05T08:15:22.036657+0000","last_clean":"2024-02-05T08:15:22.036657+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:15:22.036657+0000","last_undegraded":"2024-02-05T08:15:22.036657+0000","last_fullsized":"2024-02-05T08:15:22.036657+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:33.896717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:14:33.896717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:34.161273+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":432,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034694+0000","last_change":"2024-02-05T08:14:12.608866+0000","last_active":"2024-02-05T08:15:22.034694+0000","last_peered":"2024-02-05T08:15:22.034694+0000","last_clean":"2024-02-05T08:15:22.034694+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:15:22.034694+0000","last_undegraded":"2024-02-05T08:15:22.034694+0000","last_fullsized":"2024-02-05T08:15:22.034694+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:12.608763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:12.608763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"236'35","reported_seq":184,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031692+0000","last_change":"2024-02-05T08:14:31.191982+0000","last_active":"2024-02-05T08:15:22.031692+0000","last_peered":"2024-02-05T08:15:22.031692+0000","last_clean":"2024-02-05T08:15:22.031692+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:15:22.031692+0000","last_undegraded":"2024-02-05T08:15:22.031692+0000","last_fullsized":"2024-02-05T08:15:22.031692+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:14:31.191827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.191827+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.315305+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"249'1443","reported_seq":1473,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.799803+0000","last_change":"2024-02-05T08:14:57.761472+0000","last_active":"2024-02-05T08:15:22.799803+0000","last_peered":"2024-02-05T08:15:22.799803+0000","last_clean":"2024-02-05T08:15:22.799803+0000","last_became_active":"2024-02-05T08:14:57.761156+0000","last_became_peered":"2024-02-05T08:14:57.761156+0000","last_unstale":"2024-02-05T08:15:22.799803+0000","last_undegraded":"2024-02-05T08:15:22.799803+0000","last_fullsized":"2024-02-05T08:15:22.799803+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1443,"log_dups_size":0,"ondisk_log_size":1443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:13.756234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94568448,"num_objects":1443,"num_object_clones":0,"num_object_copies":4329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1443,"num_write_kb":92352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":397,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033088+0000","last_change":"2024-02-05T08:14:16.274043+0000","last_active":"2024-02-05T08:15:22.033088+0000","last_peered":"2024-02-05T08:15:22.033088+0000","last_clean":"2024-02-05T08:15:22.033088+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:15:22.033088+0000","last_undegraded":"2024-02-05T08:15:22.033088+0000","last_fullsized":"2024-02-05T08:15:22.033088+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:16.273936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:16.273936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:22.675528+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":416,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.036189+0000","last_change":"2024-02-05T08:14:11.787222+0000","last_active":"2024-02-05T08:15:22.036189+0000","last_peered":"2024-02-05T08:15:22.036189+0000","last_clean":"2024-02-05T08:15:22.036189+0000","last_became_active":"2024-02-05T08:12:41.743888+0000","last_became_peered":"2024-02-05T08:12:41.743888+0000","last_unstale":"2024-02-05T08:15:22.036189+0000","last_undegraded":"2024-02-05T08:15:22.036189+0000","last_fullsized":"2024-02-05T08:15:22.036189+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:11.787102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:11.787102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:25.049773+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":428,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034100+0000","last_change":"2024-02-05T08:12:41.719156+0000","last_active":"2024-02-05T08:15:22.034100+0000","last_peered":"2024-02-05T08:15:22.034100+0000","last_clean":"2024-02-05T08:15:22.034100+0000","last_became_active":"2024-02-05T08:12:41.718843+0000","last_became_peered":"2024-02-05T08:12:41.718843+0000","last_unstale":"2024-02-05T08:15:22.034100+0000","last_undegraded":"2024-02-05T08:15:22.034100+0000","last_fullsized":"2024-02-05T08:15:22.034100+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:18.291829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:18.291829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"236'15","reported_seq":430,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032908+0000","last_change":"2024-02-05T08:12:22.599613+0000","last_active":"2024-02-05T08:15:22.032908+0000","last_peered":"2024-02-05T08:15:22.032908+0000","last_clean":"2024-02-05T08:15:22.032908+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:15:22.032908+0000","last_undegraded":"2024-02-05T08:15:22.032908+0000","last_fullsized":"2024-02-05T08:15:22.032908+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:12:22.599493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:22.599493+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"249'1581","reported_seq":1610,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:23.492325+0000","last_change":"2024-02-05T08:14:57.766128+0000","last_active":"2024-02-05T08:15:23.492325+0000","last_peered":"2024-02-05T08:15:23.492325+0000","last_clean":"2024-02-05T08:15:23.492325+0000","last_became_active":"2024-02-05T08:14:57.765694+0000","last_became_peered":"2024-02-05T08:14:57.765694+0000","last_unstale":"2024-02-05T08:15:23.492325+0000","last_undegraded":"2024-02-05T08:15:23.492325+0000","last_fullsized":"2024-02-05T08:15:23.492325+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1581,"log_dups_size":0,"ondisk_log_size":1581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:25.701081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103612416,"num_objects":1581,"num_object_clones":0,"num_object_copies":4743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1581,"num_write_kb":101184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":123,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.027024+0000","last_change":"2024-02-05T08:12:41.727102+0000","last_active":"2024-02-05T08:15:22.027024+0000","last_peered":"2024-02-05T08:15:22.027024+0000","last_clean":"2024-02-05T08:15:22.027024+0000","last_became_active":"2024-02-05T08:12:41.726776+0000","last_became_peered":"2024-02-05T08:12:41.726776+0000","last_unstale":"2024-02-05T08:15:22.027024+0000","last_undegraded":"2024-02-05T08:15:22.027024+0000","last_fullsized":"2024-02-05T08:15:22.027024+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:32.545208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:32.545208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":451,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034788+0000","last_change":"2024-02-05T08:15:18.892281+0000","last_active":"2024-02-05T08:15:22.034788+0000","last_peered":"2024-02-05T08:15:22.034788+0000","last_clean":"2024-02-05T08:15:22.034788+0000","last_became_active":"2024-02-05T08:12:29.679395+0000","last_became_peered":"2024-02-05T08:12:29.679395+0000","last_unstale":"2024-02-05T08:15:22.034788+0000","last_undegraded":"2024-02-05T08:15:22.034788+0000","last_fullsized":"2024-02-05T08:15:22.034788+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:15:18.892157+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:15:18.892157+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:21.437539+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":423,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.027781+0000","last_change":"2024-02-05T08:14:04.926625+0000","last_active":"2024-02-05T08:15:22.027781+0000","last_peered":"2024-02-05T08:15:22.027781+0000","last_clean":"2024-02-05T08:15:22.027781+0000","last_became_active":"2024-02-05T08:12:29.670859+0000","last_became_peered":"2024-02-05T08:12:29.670859+0000","last_unstale":"2024-02-05T08:15:22.027781+0000","last_undegraded":"2024-02-05T08:15:22.027781+0000","last_fullsized":"2024-02-05T08:15:22.027781+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:04.926518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:14:04.926518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:34.690352+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"236'51","reported_seq":641,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.029370+0000","last_change":"2024-02-05T08:12:41.775200+0000","last_active":"2024-02-05T08:15:22.029370+0000","last_peered":"2024-02-05T08:15:22.029370+0000","last_clean":"2024-02-05T08:15:22.029370+0000","last_became_active":"2024-02-05T08:12:41.719820+0000","last_became_peered":"2024-02-05T08:12:41.719820+0000","last_unstale":"2024-02-05T08:15:22.029370+0000","last_undegraded":"2024-02-05T08:15:22.029370+0000","last_fullsized":"2024-02-05T08:15:22.029370+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:11:34.819564+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:11:34.819564+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":394,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033852+0000","last_change":"2024-02-05T08:12:28.727498+0000","last_active":"2024-02-05T08:15:22.033852+0000","last_peered":"2024-02-05T08:15:22.033852+0000","last_clean":"2024-02-05T08:15:22.033852+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:15:22.033852+0000","last_undegraded":"2024-02-05T08:15:22.033852+0000","last_fullsized":"2024-02-05T08:15:22.033852+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:28.721402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:28.721402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"249'1469","reported_seq":1499,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.815710+0000","last_change":"2024-02-05T08:14:57.763579+0000","last_active":"2024-02-05T08:15:22.815710+0000","last_peered":"2024-02-05T08:15:22.815710+0000","last_clean":"2024-02-05T08:15:22.815710+0000","last_became_active":"2024-02-05T08:14:57.763184+0000","last_became_peered":"2024-02-05T08:14:57.763184+0000","last_unstale":"2024-02-05T08:15:22.815710+0000","last_undegraded":"2024-02-05T08:15:22.815710+0000","last_fullsized":"2024-02-05T08:15:22.815710+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1469,"log_dups_size":0,"ondisk_log_size":1469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:00.113741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96272384,"num_objects":1469,"num_object_clones":0,"num_object_copies":4407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1469,"num_write_kb":94016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":426,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034829+0000","last_change":"2024-02-05T08:15:20.467544+0000","last_active":"2024-02-05T08:15:22.034829+0000","last_peered":"2024-02-05T08:15:22.034829+0000","last_clean":"2024-02-05T08:15:22.034829+0000","last_became_active":"2024-02-05T08:12:41.732338+0000","last_became_peered":"2024-02-05T08:12:41.732338+0000","last_unstale":"2024-02-05T08:15:22.034829+0000","last_undegraded":"2024-02-05T08:15:22.034829+0000","last_fullsized":"2024-02-05T08:15:22.034829+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:20.467407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:20.467407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:45.580190+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":439,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.028853+0000","last_change":"2024-02-05T08:15:17.824514+0000","last_active":"2024-02-05T08:15:22.028853+0000","last_peered":"2024-02-05T08:15:22.028853+0000","last_clean":"2024-02-05T08:15:22.028853+0000","last_became_active":"2024-02-05T08:12:29.673554+0000","last_became_peered":"2024-02-05T08:12:29.673554+0000","last_unstale":"2024-02-05T08:15:22.028853+0000","last_undegraded":"2024-02-05T08:15:22.028853+0000","last_fullsized":"2024-02-05T08:15:22.028853+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:15:17.824391+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:15:17.824391+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:31.698204+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":442,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.036057+0000","last_change":"2024-02-05T08:15:18.722410+0000","last_active":"2024-02-05T08:15:22.036057+0000","last_peered":"2024-02-05T08:15:22.036057+0000","last_clean":"2024-02-05T08:15:22.036057+0000","last_became_active":"2024-02-05T08:12:41.746289+0000","last_became_peered":"2024-02-05T08:12:41.746289+0000","last_unstale":"2024-02-05T08:15:22.036057+0000","last_undegraded":"2024-02-05T08:15:22.036057+0000","last_fullsized":"2024-02-05T08:15:22.036057+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:18.722163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:15:18.722163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:42.755882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'17","reported_seq":424,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.037858+0000","last_change":"2024-02-05T08:14:21.682688+0000","last_active":"2024-02-05T08:15:22.037858+0000","last_peered":"2024-02-05T08:15:22.037858+0000","last_clean":"2024-02-05T08:15:22.037858+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:15:22.037858+0000","last_undegraded":"2024-02-05T08:15:22.037858+0000","last_fullsized":"2024-02-05T08:15:22.037858+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:14:21.682517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:21.682517+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:47.773747+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.036890+0000","last_change":"2024-02-05T08:15:20.774445+0000","last_active":"2024-02-05T08:15:22.036890+0000","last_peered":"2024-02-05T08:15:22.036890+0000","last_clean":"2024-02-05T08:15:22.036890+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:15:22.036890+0000","last_undegraded":"2024-02-05T08:15:22.036890+0000","last_fullsized":"2024-02-05T08:15:22.036890+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:20.774326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:20.774326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:27.506687+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"249'1456","reported_seq":1486,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:23.359481+0000","last_change":"2024-02-05T08:14:57.762460+0000","last_active":"2024-02-05T08:15:23.359481+0000","last_peered":"2024-02-05T08:15:23.359481+0000","last_clean":"2024-02-05T08:15:23.359481+0000","last_became_active":"2024-02-05T08:14:57.762164+0000","last_became_peered":"2024-02-05T08:14:57.762164+0000","last_unstale":"2024-02-05T08:15:23.359481+0000","last_undegraded":"2024-02-05T08:15:23.359481+0000","last_fullsized":"2024-02-05T08:15:23.359481+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1456,"log_dups_size":0,"ondisk_log_size":1456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:12.058644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95420416,"num_objects":1456,"num_object_clones":0,"num_object_copies":4368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1456,"num_write_kb":93184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":404,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.035195+0000","last_change":"2024-02-05T08:12:41.717519+0000","last_active":"2024-02-05T08:15:22.035195+0000","last_peered":"2024-02-05T08:15:22.035195+0000","last_clean":"2024-02-05T08:15:22.035195+0000","last_became_active":"2024-02-05T08:12:41.717170+0000","last_became_peered":"2024-02-05T08:12:41.717170+0000","last_unstale":"2024-02-05T08:15:22.035195+0000","last_undegraded":"2024-02-05T08:15:22.035195+0000","last_fullsized":"2024-02-05T08:15:22.035195+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:23.303613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:23.303613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":700,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032882+0000","last_change":"2024-02-05T08:13:31.220426+0000","last_active":"2024-02-05T08:15:22.032882+0000","last_peered":"2024-02-05T08:15:22.032882+0000","last_clean":"2024-02-05T08:15:22.032882+0000","last_became_active":"2024-02-05T08:12:32.677426+0000","last_became_peered":"2024-02-05T08:12:32.677426+0000","last_unstale":"2024-02-05T08:15:22.032882+0000","last_undegraded":"2024-02-05T08:15:22.032882+0000","last_fullsized":"2024-02-05T08:15:22.032882+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:13:31.220332+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:13:31.220332+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":413,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.035103+0000","last_change":"2024-02-05T08:15:13.853710+0000","last_active":"2024-02-05T08:15:22.035103+0000","last_peered":"2024-02-05T08:15:22.035103+0000","last_clean":"2024-02-05T08:15:22.035103+0000","last_became_active":"2024-02-05T08:12:41.722331+0000","last_became_peered":"2024-02-05T08:12:41.722331+0000","last_unstale":"2024-02-05T08:15:22.035103+0000","last_undegraded":"2024-02-05T08:15:22.035103+0000","last_fullsized":"2024-02-05T08:15:22.035103+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:13.853624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:13.853624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:35.667987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"236'30","reported_seq":212,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031139+0000","last_change":"2024-02-05T08:15:14.981439+0000","last_active":"2024-02-05T08:15:22.031139+0000","last_peered":"2024-02-05T08:15:22.031139+0000","last_clean":"2024-02-05T08:15:22.031139+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:15:22.031139+0000","last_undegraded":"2024-02-05T08:15:22.031139+0000","last_fullsized":"2024-02-05T08:15:22.031139+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:15:14.981271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:14.981271+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:36.831841+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":404,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034039+0000","last_change":"2024-02-05T08:13:30.200414+0000","last_active":"2024-02-05T08:15:22.034039+0000","last_peered":"2024-02-05T08:15:22.034039+0000","last_clean":"2024-02-05T08:15:22.034039+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:15:22.034039+0000","last_undegraded":"2024-02-05T08:15:22.034039+0000","last_fullsized":"2024-02-05T08:15:22.034039+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:30.200341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:13:30.200341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"249'1613","reported_seq":1643,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:23.499221+0000","last_change":"2024-02-05T08:14:57.761686+0000","last_active":"2024-02-05T08:15:23.499221+0000","last_peered":"2024-02-05T08:15:23.499221+0000","last_clean":"2024-02-05T08:15:23.499221+0000","last_became_active":"2024-02-05T08:14:57.761382+0000","last_became_peered":"2024-02-05T08:14:57.761382+0000","last_unstale":"2024-02-05T08:15:23.499221+0000","last_undegraded":"2024-02-05T08:15:23.499221+0000","last_fullsized":"2024-02-05T08:15:23.499221+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1613,"log_dups_size":0,"ondisk_log_size":1613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:57.838632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105709568,"num_objects":1613,"num_object_clones":0,"num_object_copies":4839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1613,"num_write_kb":103232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":406,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.029045+0000","last_change":"2024-02-05T08:12:41.716722+0000","last_active":"2024-02-05T08:15:22.029045+0000","last_peered":"2024-02-05T08:15:22.029045+0000","last_clean":"2024-02-05T08:15:22.029045+0000","last_became_active":"2024-02-05T08:12:41.716394+0000","last_became_peered":"2024-02-05T08:12:41.716394+0000","last_unstale":"2024-02-05T08:15:22.029045+0000","last_undegraded":"2024-02-05T08:15:22.029045+0000","last_fullsized":"2024-02-05T08:15:22.029045+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:26.178219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.178219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":416,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032785+0000","last_change":"2024-02-05T08:15:15.359343+0000","last_active":"2024-02-05T08:15:22.032785+0000","last_peered":"2024-02-05T08:15:22.032785+0000","last_clean":"2024-02-05T08:15:22.032785+0000","last_became_active":"2024-02-05T08:12:41.741846+0000","last_became_peered":"2024-02-05T08:12:41.741846+0000","last_unstale":"2024-02-05T08:15:22.032785+0000","last_undegraded":"2024-02-05T08:15:22.032785+0000","last_fullsized":"2024-02-05T08:15:22.032785+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:15.359190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:15.359190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:41.785491+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":1068,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032891+0000","last_change":"2024-02-05T08:13:36.138202+0000","last_active":"2024-02-05T08:15:22.032891+0000","last_peered":"2024-02-05T08:15:22.032891+0000","last_clean":"2024-02-05T08:15:22.032891+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:15:22.032891+0000","last_undegraded":"2024-02-05T08:15:22.032891+0000","last_fullsized":"2024-02-05T08:15:22.032891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:13:36.138048+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":635,"num_read_kb":630,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":453,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031027+0000","last_change":"2024-02-05T08:14:25.014118+0000","last_active":"2024-02-05T08:15:22.031027+0000","last_peered":"2024-02-05T08:15:22.031027+0000","last_clean":"2024-02-05T08:15:22.031027+0000","last_became_active":"2024-02-05T08:14:24.992820+0000","last_became_peered":"2024-02-05T08:14:24.992820+0000","last_unstale":"2024-02-05T08:15:22.031027+0000","last_undegraded":"2024-02-05T08:15:22.031027+0000","last_fullsized":"2024-02-05T08:15:22.031027+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:12:16.517774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:16.517774+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":402,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.035408+0000","last_change":"2024-02-05T08:12:24.353439+0000","last_active":"2024-02-05T08:15:22.035408+0000","last_peered":"2024-02-05T08:15:22.035408+0000","last_clean":"2024-02-05T08:15:22.035408+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:15:22.035408+0000","last_undegraded":"2024-02-05T08:15:22.035408+0000","last_fullsized":"2024-02-05T08:15:22.035408+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:24.353354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:24.353354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"249'1412","reported_seq":1441,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.628420+0000","last_change":"2024-02-05T08:14:57.764321+0000","last_active":"2024-02-05T08:15:22.628420+0000","last_peered":"2024-02-05T08:15:22.628420+0000","last_clean":"2024-02-05T08:15:22.628420+0000","last_became_active":"2024-02-05T08:14:57.764021+0000","last_became_peered":"2024-02-05T08:14:57.764021+0000","last_unstale":"2024-02-05T08:15:22.628420+0000","last_undegraded":"2024-02-05T08:15:22.628420+0000","last_fullsized":"2024-02-05T08:15:22.628420+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1412,"log_dups_size":0,"ondisk_log_size":1412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:57.697074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92536832,"num_objects":1412,"num_object_clones":0,"num_object_copies":4236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1412,"num_write_kb":90368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":412,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.027887+0000","last_change":"2024-02-05T08:12:29.673477+0000","last_active":"2024-02-05T08:15:22.027887+0000","last_peered":"2024-02-05T08:15:22.027887+0000","last_clean":"2024-02-05T08:15:22.027887+0000","last_became_active":"2024-02-05T08:12:29.672973+0000","last_became_peered":"2024-02-05T08:12:29.672973+0000","last_unstale":"2024-02-05T08:15:22.027887+0000","last_undegraded":"2024-02-05T08:15:22.027887+0000","last_fullsized":"2024-02-05T08:15:22.027887+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:25.219172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:25.219172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":420,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.029229+0000","last_change":"2024-02-05T08:14:19.636646+0000","last_active":"2024-02-05T08:15:22.029229+0000","last_peered":"2024-02-05T08:15:22.029229+0000","last_clean":"2024-02-05T08:15:22.029229+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:15:22.029229+0000","last_undegraded":"2024-02-05T08:15:22.029229+0000","last_fullsized":"2024-02-05T08:15:22.029229+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:19.636535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:19.636535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:43.736754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":457,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032865+0000","last_change":"2024-02-05T08:12:27.724075+0000","last_active":"2024-02-05T08:15:22.032865+0000","last_peered":"2024-02-05T08:15:22.032865+0000","last_clean":"2024-02-05T08:15:22.032865+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:15:22.032865+0000","last_undegraded":"2024-02-05T08:15:22.032865+0000","last_fullsized":"2024-02-05T08:15:22.032865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:12:27.723924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:27.723924+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":140,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031611+0000","last_change":"2024-02-05T08:14:22.219846+0000","last_active":"2024-02-05T08:15:22.031611+0000","last_peered":"2024-02-05T08:15:22.031611+0000","last_clean":"2024-02-05T08:15:22.031611+0000","last_became_active":"2024-02-05T08:12:29.688534+0000","last_became_peered":"2024-02-05T08:12:29.688534+0000","last_unstale":"2024-02-05T08:15:22.031611+0000","last_undegraded":"2024-02-05T08:15:22.031611+0000","last_fullsized":"2024-02-05T08:15:22.031611+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:22.219740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:14:22.219740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:33.510414+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"249'1427","reported_seq":1457,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.811777+0000","last_change":"2024-02-05T08:14:57.764336+0000","last_active":"2024-02-05T08:15:22.811777+0000","last_peered":"2024-02-05T08:15:22.811777+0000","last_clean":"2024-02-05T08:15:22.811777+0000","last_became_active":"2024-02-05T08:14:57.764038+0000","last_became_peered":"2024-02-05T08:14:57.764038+0000","last_unstale":"2024-02-05T08:15:22.811777+0000","last_undegraded":"2024-02-05T08:15:22.811777+0000","last_fullsized":"2024-02-05T08:15:22.811777+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1427,"log_dups_size":0,"ondisk_log_size":1427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:06.265859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93519872,"num_objects":1427,"num_object_clones":0,"num_object_copies":4281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1427,"num_write_kb":91328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":404,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031866+0000","last_change":"2024-02-05T08:12:41.736005+0000","last_active":"2024-02-05T08:15:22.031866+0000","last_peered":"2024-02-05T08:15:22.031866+0000","last_clean":"2024-02-05T08:15:22.031866+0000","last_became_active":"2024-02-05T08:12:41.735706+0000","last_became_peered":"2024-02-05T08:12:41.735706+0000","last_unstale":"2024-02-05T08:15:22.031866+0000","last_undegraded":"2024-02-05T08:15:22.031866+0000","last_fullsized":"2024-02-05T08:15:22.031866+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.607056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.607056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":425,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.035020+0000","last_change":"2024-02-05T08:15:16.880868+0000","last_active":"2024-02-05T08:15:22.035020+0000","last_peered":"2024-02-05T08:15:22.035020+0000","last_clean":"2024-02-05T08:15:22.035020+0000","last_became_active":"2024-02-05T08:12:41.730010+0000","last_became_peered":"2024-02-05T08:12:41.730010+0000","last_unstale":"2024-02-05T08:15:22.035020+0000","last_undegraded":"2024-02-05T08:15:22.035020+0000","last_fullsized":"2024-02-05T08:15:22.035020+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:16.880446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:16.880446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:46.844168+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'20","reported_seq":438,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.036517+0000","last_change":"2024-02-05T08:12:19.056472+0000","last_active":"2024-02-05T08:15:22.036517+0000","last_peered":"2024-02-05T08:15:22.036517+0000","last_clean":"2024-02-05T08:15:22.036517+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:15:22.036517+0000","last_undegraded":"2024-02-05T08:15:22.036517+0000","last_fullsized":"2024-02-05T08:15:22.036517+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:12:19.056345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:19.056345+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"190'10","reported_seq":559,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032993+0000","last_change":"2024-02-05T08:14:30.163779+0000","last_active":"2024-02-05T08:15:22.032993+0000","last_peered":"2024-02-05T08:15:22.032993+0000","last_clean":"2024-02-05T08:15:22.032993+0000","last_became_active":"2024-02-05T08:12:29.683091+0000","last_became_peered":"2024-02-05T08:12:29.683091+0000","last_unstale":"2024-02-05T08:15:22.032993+0000","last_undegraded":"2024-02-05T08:15:22.032993+0000","last_fullsized":"2024-02-05T08:15:22.032993+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:14:30.163629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:30.163629+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:46.283057+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"249'1976","reported_seq":2006,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:27.581177+0000","last_change":"2024-02-05T08:14:57.761716+0000","last_active":"2024-02-05T08:15:27.581177+0000","last_peered":"2024-02-05T08:15:27.581177+0000","last_clean":"2024-02-05T08:15:27.581177+0000","last_became_active":"2024-02-05T08:14:57.761443+0000","last_became_peered":"2024-02-05T08:14:57.761443+0000","last_unstale":"2024-02-05T08:15:27.581177+0000","last_undegraded":"2024-02-05T08:15:27.581177+0000","last_fullsized":"2024-02-05T08:15:27.581177+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1976,"log_dups_size":0,"ondisk_log_size":1976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:23.258466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129499136,"num_objects":1976,"num_object_clones":0,"num_object_copies":5928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1976,"num_write_kb":126464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":412,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033777+0000","last_change":"2024-02-05T08:13:18.250752+0000","last_active":"2024-02-05T08:15:22.033777+0000","last_peered":"2024-02-05T08:15:22.033777+0000","last_clean":"2024-02-05T08:15:22.033777+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:15:22.033777+0000","last_undegraded":"2024-02-05T08:15:22.033777+0000","last_fullsized":"2024-02-05T08:15:22.033777+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:18.250590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:13:18.250590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":420,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031130+0000","last_change":"2024-02-05T08:12:21.623410+0000","last_active":"2024-02-05T08:15:22.031130+0000","last_peered":"2024-02-05T08:15:22.031130+0000","last_clean":"2024-02-05T08:15:22.031130+0000","last_became_active":"2024-02-05T08:11:23.200753+0000","last_became_peered":"2024-02-05T08:11:23.200753+0000","last_unstale":"2024-02-05T08:15:22.031130+0000","last_undegraded":"2024-02-05T08:15:22.031130+0000","last_fullsized":"2024-02-05T08:15:22.031130+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:12:21.623291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:21.623291+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"236'28","reported_seq":493,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:24.002177+0000","last_change":"2024-02-05T08:15:24.002222+0000","last_active":"2024-02-05T08:15:24.002177+0000","last_peered":"2024-02-05T08:15:24.002177+0000","last_clean":"2024-02-05T08:15:24.002177+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:15:24.002177+0000","last_undegraded":"2024-02-05T08:15:24.002177+0000","last_fullsized":"2024-02-05T08:15:24.002177+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:15:24.002106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:24.002106+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:36.950373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"194'10","reported_seq":476,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.028004+0000","last_change":"2024-02-05T08:12:41.773528+0000","last_active":"2024-02-05T08:15:22.028004+0000","last_peered":"2024-02-05T08:15:22.028004+0000","last_clean":"2024-02-05T08:15:22.028004+0000","last_became_active":"2024-02-05T08:12:41.721686+0000","last_became_peered":"2024-02-05T08:12:41.721686+0000","last_unstale":"2024-02-05T08:15:22.028004+0000","last_undegraded":"2024-02-05T08:15:22.028004+0000","last_fullsized":"2024-02-05T08:15:22.028004+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'9","last_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:11:27.751188+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"249'1415","reported_seq":1444,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.616786+0000","last_change":"2024-02-05T08:14:57.765949+0000","last_active":"2024-02-05T08:15:22.616786+0000","last_peered":"2024-02-05T08:15:22.616786+0000","last_clean":"2024-02-05T08:15:22.616786+0000","last_became_active":"2024-02-05T08:14:57.765609+0000","last_became_peered":"2024-02-05T08:14:57.765609+0000","last_unstale":"2024-02-05T08:15:22.616786+0000","last_undegraded":"2024-02-05T08:15:22.616786+0000","last_fullsized":"2024-02-05T08:15:22.616786+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1415,"log_dups_size":0,"ondisk_log_size":1415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:07.832990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92733440,"num_objects":1415,"num_object_clones":0,"num_object_copies":4245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1415,"num_write_kb":90560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":413,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.953568+0000","last_change":"2024-02-05T08:15:22.953610+0000","last_active":"2024-02-05T08:15:22.953568+0000","last_peered":"2024-02-05T08:15:22.953568+0000","last_clean":"2024-02-05T08:15:22.953568+0000","last_became_active":"2024-02-05T08:12:41.731253+0000","last_became_peered":"2024-02-05T08:12:41.731253+0000","last_unstale":"2024-02-05T08:15:22.953568+0000","last_undegraded":"2024-02-05T08:15:22.953568+0000","last_fullsized":"2024-02-05T08:15:22.953568+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.953481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.953481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:26.478801+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":143,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034209+0000","last_change":"2024-02-05T08:15:15.390114+0000","last_active":"2024-02-05T08:15:22.034209+0000","last_peered":"2024-02-05T08:15:22.034209+0000","last_clean":"2024-02-05T08:15:22.034209+0000","last_became_active":"2024-02-05T08:12:41.723699+0000","last_became_peered":"2024-02-05T08:12:41.723699+0000","last_unstale":"2024-02-05T08:15:22.034209+0000","last_undegraded":"2024-02-05T08:15:22.034209+0000","last_fullsized":"2024-02-05T08:15:22.034209+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:15:15.389990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:15.389990+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:24.889149+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"236'38","reported_seq":473,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033023+0000","last_change":"2024-02-05T08:14:24.593255+0000","last_active":"2024-02-05T08:15:22.033023+0000","last_peered":"2024-02-05T08:15:22.033023+0000","last_clean":"2024-02-05T08:15:22.033023+0000","last_became_active":"2024-02-05T08:12:41.728044+0000","last_became_peered":"2024-02-05T08:12:41.728044+0000","last_unstale":"2024-02-05T08:15:22.033023+0000","last_undegraded":"2024-02-05T08:15:22.033023+0000","last_fullsized":"2024-02-05T08:15:22.033023+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'26","last_scrub_stamp":"2024-02-05T08:14:24.591818+0000","last_deep_scrub":"186'26","last_deep_scrub_stamp":"2024-02-05T08:12:24.579532+0000","last_clean_scrub_stamp":"2024-02-05T08:14:24.591818+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:26.931473+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":409,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.027671+0000","last_change":"2024-02-05T08:14:26.979859+0000","last_active":"2024-02-05T08:15:22.027671+0000","last_peered":"2024-02-05T08:15:22.027671+0000","last_clean":"2024-02-05T08:15:22.027671+0000","last_became_active":"2024-02-05T08:12:41.720247+0000","last_became_peered":"2024-02-05T08:12:41.720247+0000","last_unstale":"2024-02-05T08:15:22.027671+0000","last_undegraded":"2024-02-05T08:15:22.027671+0000","last_fullsized":"2024-02-05T08:15:22.027671+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:26.979755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:26.979755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:42.561175+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"249'1508","reported_seq":1538,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:23.253228+0000","last_change":"2024-02-05T08:14:57.764494+0000","last_active":"2024-02-05T08:15:23.253228+0000","last_peered":"2024-02-05T08:15:23.253228+0000","last_clean":"2024-02-05T08:15:23.253228+0000","last_became_active":"2024-02-05T08:14:57.764174+0000","last_became_peered":"2024-02-05T08:14:57.764174+0000","last_unstale":"2024-02-05T08:15:23.253228+0000","last_undegraded":"2024-02-05T08:15:23.253228+0000","last_fullsized":"2024-02-05T08:15:23.253228+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1508,"log_dups_size":0,"ondisk_log_size":1508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:20.765265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98828288,"num_objects":1508,"num_object_clones":0,"num_object_copies":4524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1508,"num_write_kb":96512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":406,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.034758+0000","last_change":"2024-02-05T08:14:07.163475+0000","last_active":"2024-02-05T08:15:22.034758+0000","last_peered":"2024-02-05T08:15:22.034758+0000","last_clean":"2024-02-05T08:15:22.034758+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:15:22.034758+0000","last_undegraded":"2024-02-05T08:15:22.034758+0000","last_fullsized":"2024-02-05T08:15:22.034758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:07.163372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:07.163372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:34.754956+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":540,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.029088+0000","last_change":"2024-02-05T08:14:31.071675+0000","last_active":"2024-02-05T08:15:22.029088+0000","last_peered":"2024-02-05T08:15:22.029088+0000","last_clean":"2024-02-05T08:15:22.029088+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:15:22.029088+0000","last_undegraded":"2024-02-05T08:15:22.029088+0000","last_fullsized":"2024-02-05T08:15:22.029088+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:14:31.071536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.071536+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:40.656118+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":131,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.028950+0000","last_change":"2024-02-05T08:14:17.648700+0000","last_active":"2024-02-05T08:15:22.028950+0000","last_peered":"2024-02-05T08:15:22.028950+0000","last_clean":"2024-02-05T08:15:22.028950+0000","last_became_active":"2024-02-05T08:12:41.724969+0000","last_became_peered":"2024-02-05T08:12:41.724969+0000","last_unstale":"2024-02-05T08:15:22.028950+0000","last_undegraded":"2024-02-05T08:15:22.028950+0000","last_fullsized":"2024-02-05T08:15:22.028950+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:17.648596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:17.648596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"236'27","reported_seq":480,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.033680+0000","last_change":"2024-02-05T08:15:15.777707+0000","last_active":"2024-02-05T08:15:22.033680+0000","last_peered":"2024-02-05T08:15:22.033680+0000","last_clean":"2024-02-05T08:15:22.033680+0000","last_became_active":"2024-02-05T08:12:41.725007+0000","last_became_peered":"2024-02-05T08:12:41.725007+0000","last_unstale":"2024-02-05T08:15:22.033680+0000","last_undegraded":"2024-02-05T08:15:22.033680+0000","last_fullsized":"2024-02-05T08:15:22.033680+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:15:15.777496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:15.777496+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:43.525899+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"249'1421","reported_seq":1450,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.619914+0000","last_change":"2024-02-05T08:14:57.765880+0000","last_active":"2024-02-05T08:15:22.619914+0000","last_peered":"2024-02-05T08:15:22.619914+0000","last_clean":"2024-02-05T08:15:22.619914+0000","last_became_active":"2024-02-05T08:14:57.765255+0000","last_became_peered":"2024-02-05T08:14:57.765255+0000","last_unstale":"2024-02-05T08:15:22.619914+0000","last_undegraded":"2024-02-05T08:15:22.619914+0000","last_fullsized":"2024-02-05T08:15:22.619914+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1421,"log_dups_size":0,"ondisk_log_size":1421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:10.520554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93126656,"num_objects":1421,"num_object_clones":0,"num_object_copies":4263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1421,"num_write_kb":90944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":399,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.030606+0000","last_change":"2024-02-05T08:15:09.025688+0000","last_active":"2024-02-05T08:15:22.030606+0000","last_peered":"2024-02-05T08:15:22.030606+0000","last_clean":"2024-02-05T08:15:22.030606+0000","last_became_active":"2024-02-05T08:12:41.719835+0000","last_became_peered":"2024-02-05T08:12:41.719835+0000","last_unstale":"2024-02-05T08:15:22.030606+0000","last_undegraded":"2024-02-05T08:15:22.030606+0000","last_fullsized":"2024-02-05T08:15:22.030606+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:09.025580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:15:09.025580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:34.218391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":422,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.035376+0000","last_change":"2024-02-05T08:14:29.177432+0000","last_active":"2024-02-05T08:15:22.035376+0000","last_peered":"2024-02-05T08:15:22.035376+0000","last_clean":"2024-02-05T08:15:22.035376+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:15:22.035376+0000","last_undegraded":"2024-02-05T08:15:22.035376+0000","last_fullsized":"2024-02-05T08:15:22.035376+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:29.177314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:29.177314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:35.329432+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.027812+0000","last_change":"2024-02-05T08:14:24.985198+0000","last_active":"2024-02-05T08:15:22.027812+0000","last_peered":"2024-02-05T08:15:22.027812+0000","last_clean":"2024-02-05T08:15:22.027812+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:15:22.027812+0000","last_undegraded":"2024-02-05T08:15:22.027812+0000","last_fullsized":"2024-02-05T08:15:22.027812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:24.985069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:14:24.985069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:50.692856+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"236'19","reported_seq":473,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.028886+0000","last_change":"2024-02-05T08:14:09.985107+0000","last_active":"2024-02-05T08:15:22.028886+0000","last_peered":"2024-02-05T08:15:22.028886+0000","last_clean":"2024-02-05T08:15:22.028886+0000","last_became_active":"2024-02-05T08:12:29.666284+0000","last_became_peered":"2024-02-05T08:12:29.666284+0000","last_unstale":"2024-02-05T08:15:22.028886+0000","last_undegraded":"2024-02-05T08:15:22.028886+0000","last_fullsized":"2024-02-05T08:15:22.028886+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:14:09.984989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:09.984989+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.008972+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"249'1514","reported_seq":1544,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:23.258046+0000","last_change":"2024-02-05T08:14:57.765243+0000","last_active":"2024-02-05T08:15:23.258046+0000","last_peered":"2024-02-05T08:15:23.258046+0000","last_clean":"2024-02-05T08:15:23.258046+0000","last_became_active":"2024-02-05T08:14:57.764870+0000","last_became_peered":"2024-02-05T08:14:57.764870+0000","last_unstale":"2024-02-05T08:15:23.258046+0000","last_undegraded":"2024-02-05T08:15:23.258046+0000","last_fullsized":"2024-02-05T08:15:23.258046+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":1514,"log_dups_size":0,"ondisk_log_size":1514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:09.829403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99221504,"num_objects":1514,"num_object_clones":0,"num_object_copies":4542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1514,"num_write_kb":96896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":425,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.038022+0000","last_change":"2024-02-05T08:15:21.884006+0000","last_active":"2024-02-05T08:15:22.038022+0000","last_peered":"2024-02-05T08:15:22.038022+0000","last_clean":"2024-02-05T08:15:22.038022+0000","last_became_active":"2024-02-05T08:12:41.737224+0000","last_became_peered":"2024-02-05T08:12:41.737224+0000","last_unstale":"2024-02-05T08:15:22.038022+0000","last_undegraded":"2024-02-05T08:15:22.038022+0000","last_fullsized":"2024-02-05T08:15:22.038022+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:15:21.883859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:21.883859+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:32.454548+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":403,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.029559+0000","last_change":"2024-02-05T08:15:14.746706+0000","last_active":"2024-02-05T08:15:22.029559+0000","last_peered":"2024-02-05T08:15:22.029559+0000","last_clean":"2024-02-05T08:15:22.029559+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:15:22.029559+0000","last_undegraded":"2024-02-05T08:15:22.029559+0000","last_fullsized":"2024-02-05T08:15:22.029559+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:14.746593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:14.746593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:42.161153+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":140,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.031781+0000","last_change":"2024-02-05T08:14:10.208350+0000","last_active":"2024-02-05T08:15:22.031781+0000","last_peered":"2024-02-05T08:15:22.031781+0000","last_clean":"2024-02-05T08:15:22.031781+0000","last_became_active":"2024-02-05T08:12:29.678501+0000","last_became_peered":"2024-02-05T08:12:29.678501+0000","last_unstale":"2024-02-05T08:15:22.031781+0000","last_undegraded":"2024-02-05T08:15:22.031781+0000","last_fullsized":"2024-02-05T08:15:22.031781+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:10.208236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:14:10.208236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:30.477431+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"236'31","reported_seq":455,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.028558+0000","last_change":"2024-02-05T08:14:34.016671+0000","last_active":"2024-02-05T08:15:22.028558+0000","last_peered":"2024-02-05T08:15:22.028558+0000","last_clean":"2024-02-05T08:15:22.028558+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:15:22.028558+0000","last_undegraded":"2024-02-05T08:15:22.028558+0000","last_fullsized":"2024-02-05T08:15:22.028558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-02-05T08:14:34.016456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:34.016456+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.345431+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"249'2055","reported_seq":2084,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:27.559766+0000","last_change":"2024-02-05T08:14:57.762954+0000","last_active":"2024-02-05T08:15:27.559766+0000","last_peered":"2024-02-05T08:15:27.559766+0000","last_clean":"2024-02-05T08:15:27.559766+0000","last_became_active":"2024-02-05T08:14:57.762652+0000","last_became_peered":"2024-02-05T08:14:57.762652+0000","last_unstale":"2024-02-05T08:15:27.559766+0000","last_undegraded":"2024-02-05T08:15:27.559766+0000","last_fullsized":"2024-02-05T08:15:27.559766+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2055,"log_dups_size":0,"ondisk_log_size":2055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:07.296366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134676480,"num_objects":2055,"num_object_clones":0,"num_object_copies":6165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2055,"num_write_kb":131520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":398,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.029067+0000","last_change":"2024-02-05T08:14:21.925615+0000","last_active":"2024-02-05T08:15:22.029067+0000","last_peered":"2024-02-05T08:15:22.029067+0000","last_clean":"2024-02-05T08:15:22.029067+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:15:22.029067+0000","last_undegraded":"2024-02-05T08:15:22.029067+0000","last_fullsized":"2024-02-05T08:15:22.029067+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:21.925502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:14:21.925502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:47.212458+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":556,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.035846+0000","last_change":"2024-02-05T08:15:10.756204+0000","last_active":"2024-02-05T08:15:22.035846+0000","last_peered":"2024-02-05T08:15:22.035846+0000","last_clean":"2024-02-05T08:15:22.035846+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:15:22.035846+0000","last_undegraded":"2024-02-05T08:15:22.035846+0000","last_fullsized":"2024-02-05T08:15:22.035846+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:15:10.756097+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:15:10.756097+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:20.981357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":122,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.032818+0000","last_change":"2024-02-05T08:12:41.757481+0000","last_active":"2024-02-05T08:15:22.032818+0000","last_peered":"2024-02-05T08:15:22.032818+0000","last_clean":"2024-02-05T08:15:22.032818+0000","last_became_active":"2024-02-05T08:12:41.757159+0000","last_became_peered":"2024-02-05T08:12:41.757159+0000","last_unstale":"2024-02-05T08:15:22.032818+0000","last_undegraded":"2024-02-05T08:15:22.032818+0000","last_fullsized":"2024-02-05T08:15:22.032818+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:16.521723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:16.521723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"236'20","reported_seq":490,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-02-05T08:15:22.027701+0000","last_change":"2024-02-05T08:14:14.999089+0000","last_active":"2024-02-05T08:15:22.027701+0000","last_peered":"2024-02-05T08:15:22.027701+0000","last_clean":"2024-02-05T08:15:22.027701+0000","last_became_active":"2024-02-05T08:12:41.727271+0000","last_became_peered":"2024-02-05T08:12:41.727271+0000","last_unstale":"2024-02-05T08:15:22.027701+0000","last_undegraded":"2024-02-05T08:15:22.027701+0000","last_fullsized":"2024-02-05T08:15:22.027701+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:14:14.998858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:14.998858+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1628635136,"num_objects":24851,"num_object_clones":0,"num_object_copies":74553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24851,"num_write_kb":1590464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4902289408,"data_stored":4902289408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24851,"ondisk_log_size":24851,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":635,"num_read_kb":630,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1123,"num_read_kb":880,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":4032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":12894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":741,"ondisk_log_size":741,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037893,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1257856,"kb_used_data":991992,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92487296,"statfs":{"total":95995035648,"available":94706991104,"internally_reserved":0,"allocated":1015799808,"data_stored":1015412401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":272169418},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":1.881,"15min":1.3009999999999999},"min":{"1min":0.254,"5min":0.22,"15min":0.217},"max":{"1min":3.972,"5min":52.488999999999997,"15min":60.127000000000002},"last":1.329},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.8660000000000001,"15min":1.2989999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":5.25,"5min":52.335000000000001,"15min":58.283000000000001},"last":1.6699999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.8180000000000001,"15min":1.2350000000000001},"min":{"1min":0.27600000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.343,"5min":38.399999999999999,"15min":38.399999999999999},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.7609999999999999,"15min":1.113},"min":{"1min":0.28699999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.9449999999999998,"5min":38.195999999999998,"15min":38.195999999999998},"last":1.5960000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1890000000000001,"15min":0.85099999999999998},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":3.8889999999999998,"5min":6.2480000000000002,"15min":7.6779999999999999},"last":1.385},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.119,"15min":0.91900000000000004},"min":{"1min":0.35799999999999998,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":2.8010000000000002,"5min":6.399,"15min":8.9100000000000001},"last":0.40500000000000003}]},{"osd":3,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.98899999999999999,"15min":1.139},"min":{"1min":0.307,"5min":0.249,"15min":0.249},"max":{"1min":2.8279999999999998,"5min":4.6630000000000003,"15min":58.515000000000001},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.1200000000000001,"15min":1.123},"min":{"1min":0.30199999999999999,"5min":0.27400000000000002,"15min":0.27000000000000002},"max":{"1min":5.391,"5min":6.202,"15min":58.225999999999999},"last":1.6160000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.292,"15min":0.70999999999999996},"min":{"1min":0.32800000000000001,"5min":0.23300000000000001,"15min":0.22},"max":{"1min":3.9550000000000001,"5min":49.601999999999997,"15min":49.601999999999997},"last":1.3560000000000001},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.339,"15min":0.72999999999999998},"min":{"1min":0.32800000000000001,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":3.6349999999999998,"5min":49.795999999999999,"15min":49.795999999999999},"last":0.438}]},{"osd":5,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.2669999999999999,"15min":0.71499999999999997},"min":{"1min":0.307,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.298,"5min":49.914000000000001,"15min":49.914000000000001},"last":1.7250000000000001},{"interface":"front","average":{"1min":0.96299999999999997,"5min":1.151,"15min":0.70399999999999996},"min":{"1min":0.372,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.7320000000000002,"5min":52.256,"15min":52.256},"last":1.407}]},{"osd":6,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":0.74099999999999999,"15min":0.57899999999999996},"min":{"1min":0.39100000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":8.0050000000000008,"5min":8.0050000000000008,"15min":8.0050000000000008},"last":1.758},{"interface":"front","average":{"1min":1.23,"5min":0.69199999999999995,"15min":0.54700000000000004},"min":{"1min":0.34200000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":7.2850000000000001,"5min":7.2850000000000001,"15min":7.2850000000000001},"last":0.51800000000000002}]}]},{"osd":6,"up_from":51,"seq":219043332236,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":617876,"kb_used_data":384012,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":93127276,"statfs":{"total":95995035648,"available":95362330624,"internally_reserved":0,"allocated":393228288,"data_stored":392819731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":239466950},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":2.488,"15min":1.3879999999999999},"min":{"1min":0.28499999999999998,"5min":0.22900000000000001,"15min":0.223},"max":{"1min":3.8849999999999998,"5min":102.819,"15min":102.819},"last":1.9930000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":2.4860000000000002,"15min":1.4610000000000001},"min":{"1min":0.28899999999999998,"5min":0.214,"15min":0.214},"max":{"1min":4.5389999999999997,"5min":102.75,"15min":102.75},"last":0.27200000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.5660000000000001,"15min":1.0780000000000001},"min":{"1min":0.28199999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.98,"5min":23.773,"15min":23.773},"last":2.0499999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.4039999999999999,"15min":1.004},"min":{"1min":0.36699999999999999,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":1.843,"5min":23.719000000000001,"15min":23.719000000000001},"last":0.51100000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":1.234,"15min":0.96799999999999997},"min":{"1min":0.28699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.931,"5min":22.503,"15min":22.503},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.202,"15min":0.90100000000000002},"min":{"1min":0.36399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.4510000000000001,"5min":22.466000000000001,"15min":22.466000000000001},"last":0.52300000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":1.8939999999999999,"15min":1.26},"min":{"1min":0.36499999999999999,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":3.3100000000000001,"5min":102.438,"15min":102.438},"last":2.085},{"interface":"front","average":{"1min":0.67900000000000005,"5min":1.8220000000000001,"15min":1.242},"min":{"1min":0.38700000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.621,"5min":102.288,"15min":102.288},"last":0.434}]},{"osd":4,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.68400000000000005,"15min":0.51400000000000001},"min":{"1min":0.309,"5min":0.26100000000000001,"15min":0.223},"max":{"1min":3.3679999999999999,"5min":9.8970000000000002,"15min":9.8970000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.79400000000000004,"15min":0.56000000000000005},"min":{"1min":0.311,"5min":0.30199999999999999,"15min":0.26600000000000001},"max":{"1min":2.9300000000000002,"5min":4.9589999999999996,"15min":4.9589999999999996},"last":2.117}]},{"osd":5,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.2430000000000001,"15min":0.72699999999999998},"min":{"1min":0.32400000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.3250000000000002,"5min":43.643000000000001,"15min":43.643000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":1.048,"5min":1.2629999999999999,"15min":0.745},"min":{"1min":0.40799999999999997,"5min":0.307,"15min":0.30399999999999999},"max":{"1min":3.7549999999999999,"5min":43.893000000000001,"15min":43.893000000000001},"last":2.1930000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:14:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.70599999999999996,"15min":0.57399999999999995},"min":{"1min":0.372,"5min":0.32000000000000001,"15min":0.29999999999999999},"max":{"1min":3.3980000000000001,"5min":5.0380000000000003,"15min":5.0380000000000003},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.69199999999999995,"15min":0.58199999999999996},"min":{"1min":0.38900000000000001,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":3.0430000000000001,"5min":4.851,"15min":4.851},"last":2.278}]}]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1220756,"kb_used_data":838348,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92524396,"statfs":{"total":95995035648,"available":94744981504,"internally_reserved":0,"allocated":858468352,"data_stored":858059629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.46,"15min":1.3779999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.14699999999999999},"max":{"1min":3.1040000000000001,"5min":21.120999999999999,"15min":86.105000000000004},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.081,"5min":1.8300000000000001,"15min":1.623},"min":{"1min":0.23400000000000001,"5min":0.20699999999999999,"15min":0.16},"max":{"1min":6.1710000000000003,"5min":21.172999999999998,"15min":85.665000000000006},"last":1.089}]},{"osd":2,"last update":"Mon Feb 5 08:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.4550000000000001,"15min":1.268},"min":{"1min":0.221,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":3.544,"5min":19.016999999999999,"15min":49.030999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.901,"15min":1.4790000000000001},"min":{"1min":0.21099999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.9800000000000004,"5min":20.434000000000001,"15min":48.984000000000002},"last":1.054}]},{"osd":3,"last update":"Mon Feb 5 08:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.623,"15min":1.4330000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20599999999999999},"max":{"1min":3.141,"5min":31.896000000000001,"15min":49.067999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.3440000000000001,"15min":1.373},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.24199999999999999},"max":{"1min":3.286,"5min":32.018999999999998,"15min":48.734999999999999},"last":0.61499999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:14:30 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3320000000000001,"15min":0.997},"min":{"1min":0.33100000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.5920000000000001,"5min":25.952000000000002,"15min":25.952000000000002},"last":2.1829999999999998},{"interface":"front","average":{"1min":1.627,"5min":1.726,"15min":1.1160000000000001},"min":{"1min":0.52400000000000002,"5min":0.30199999999999999,"15min":0.27100000000000002},"max":{"1min":4.5220000000000002,"5min":25.922000000000001,"15min":25.922000000000001},"last":0.78300000000000003}]},{"osd":5,"last update":"Mon Feb 5 08:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.6890000000000001,"15min":1.1319999999999999},"min":{"1min":0.41199999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.4660000000000002,"5min":54.552999999999997,"15min":54.552999999999997},"last":1.8280000000000001},{"interface":"front","average":{"1min":1.3320000000000001,"5min":2.0139999999999998,"15min":1.2430000000000001},"min":{"1min":0.40000000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":6.0259999999999998,"5min":55.811,"15min":55.811},"last":1.206}]},{"osd":6,"last update":"Mon Feb 5 08:14:26 2024","interfaces":[{"interface":"back","average":{"1min":3.4409999999999998,"5min":1.758,"15min":1.1120000000000001},"min":{"1min":0.38700000000000001,"5min":0.36799999999999999,"15min":0.33100000000000002},"max":{"1min":54.384,"5min":54.384,"15min":54.384},"last":0.754},{"interface":"front","average":{"1min":3.2839999999999998,"5min":1.601,"15min":1.0940000000000001},"min":{"1min":0.41399999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":55.210999999999999,"5min":55.210999999999999,"15min":55.210999999999999},"last":1.7929999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.3799999999999999,"15min":1.0940000000000001},"min":{"1min":0.42899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.0579999999999998,"5min":7.9390000000000001,"15min":13.749000000000001},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.3660000000000001,"15min":1.0569999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":4.6180000000000003,"5min":7.9640000000000004,"15min":7.9640000000000004},"last":0.80600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738537,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1054664,"kb_used_data":702780,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92690488,"statfs":{"total":95995035648,"available":94915059712,"internally_reserved":0,"allocated":719646720,"data_stored":719254960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":2.0449999999999999,"15min":1.6830000000000001},"min":{"1min":0.188,"5min":0.186,"15min":0.152},"max":{"1min":2.5619999999999998,"5min":32.066000000000003,"15min":66.772999999999996},"last":0.17499999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":1.8360000000000001,"15min":1.427},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17199999999999999},"max":{"1min":2.4209999999999998,"5min":15.327999999999999,"15min":46.198999999999998},"last":1.173}]},{"osd":2,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.901,"15min":1.4099999999999999},"min":{"1min":0.30399999999999999,"5min":0.216,"15min":0.216},"max":{"1min":8.75,"5min":12.074,"15min":46.170999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.998,"5min":2.1949999999999998,"15min":1.5620000000000001},"min":{"1min":0.219,"5min":0.20399999999999999,"15min":0.19400000000000001},"max":{"1min":3.04,"5min":15.569000000000001,"15min":46.543999999999997},"last":0.34399999999999997}]},{"osd":3,"last update":"Mon Feb 5 08:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.9750000000000001,"15min":1.6180000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.21099999999999999},"max":{"1min":6.5940000000000003,"5min":13.013,"15min":67.888000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":2.5649999999999999,"15min":1.9179999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.22600000000000001},"max":{"1min":1.6180000000000001,"5min":37.262999999999998,"15min":66.891999999999996},"last":0.29299999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:14:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.6870000000000001,"15min":1.073},"min":{"1min":0.27700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":3.3119999999999998,"5min":15.159000000000001,"15min":15.159000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.462,"15min":1.0309999999999999},"min":{"1min":0.35699999999999998,"5min":0.28999999999999998,"15min":0.26900000000000002},"max":{"1min":6.3179999999999996,"5min":10.928000000000001,"15min":12.989000000000001},"last":0.96799999999999997}]},{"osd":5,"last update":"Mon Feb 5 08:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.6120000000000001,"15min":1.1200000000000001},"min":{"1min":0.39100000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.1589999999999998,"5min":15.689,"15min":15.689},"last":0.377},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.3720000000000001,"15min":1.0569999999999999},"min":{"1min":0.438,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":12.94},"last":1.002}]},{"osd":6,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.2529999999999999,"15min":1.0489999999999999},"min":{"1min":0.49399999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.8220000000000001,"5min":7.8220000000000001,"15min":12.967000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.4350000000000001,"15min":1.133},"min":{"1min":0.47199999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.6240000000000001,"5min":15.721,"15min":15.721},"last":0.432}]},{"osd":7,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.637,"15min":1.083},"min":{"1min":0.34399999999999997,"5min":0.33600000000000002,"15min":0.32900000000000001},"max":{"1min":4.2199999999999998,"5min":15.211,"15min":15.211},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.508,"15min":1.1200000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":7.8650000000000002,"5min":7.8650000000000002,"15min":13.012},"last":1.034}]}]},{"osd":2,"up_from":18,"seq":77309411487,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":843400,"kb_used_data":465152,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92901752,"statfs":{"total":95995035648,"available":95131394048,"internally_reserved":0,"allocated":476315648,"data_stored":475918921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":387316166},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":2.5800000000000001,"15min":1.542},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.19700000000000001},"max":{"1min":3.8540000000000001,"5min":40.354999999999997,"15min":40.354999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.581,"5min":2.7930000000000001,"15min":1.6699999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":5.5650000000000004,"5min":40.566000000000003,"15min":40.566000000000003},"last":0.84999999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":2.2610000000000001,"15min":1.3480000000000001},"min":{"1min":0.25900000000000001,"5min":0.19400000000000001,"15min":0.17999999999999999},"max":{"1min":2.843,"5min":55.942,"15min":55.942},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.2689999999999999,"5min":2.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.29299999999999998,"5min":0.22900000000000001,"15min":0.193},"max":{"1min":4.7530000000000001,"5min":55.960999999999999,"15min":55.960999999999999},"last":1.27}]},{"osd":3,"last update":"Mon Feb 5 08:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.6200000000000001,"15min":1.6919999999999999},"min":{"1min":0.26100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.605,"5min":8.6590000000000007,"15min":68.108000000000004},"last":1.9590000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":1.5,"15min":1.54},"min":{"1min":0.29199999999999998,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":1.8029999999999999,"5min":7.8879999999999999,"15min":67.945999999999998},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:14:29 2024","interfaces":[{"interface":"back","average":{"1min":1.9039999999999999,"5min":1.9339999999999999,"15min":1.236},"min":{"1min":0.34100000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.9370000000000003,"5min":31.596,"15min":31.596},"last":1.119},{"interface":"front","average":{"1min":1.9950000000000001,"5min":1.7769999999999999,"15min":1.1000000000000001},"min":{"1min":0.40100000000000002,"5min":0.27200000000000002,"15min":0.252},"max":{"1min":4.5540000000000003,"5min":31.57,"15min":31.57},"last":1.079}]},{"osd":5,"last update":"Mon Feb 5 08:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.9450000000000001,"15min":1.2330000000000001},"min":{"1min":0.35099999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":3.7970000000000002,"5min":58.799999999999997,"15min":58.799999999999997},"last":0.78600000000000003},{"interface":"front","average":{"1min":1.196,"5min":2.0419999999999998,"15min":1.2170000000000001},"min":{"1min":0.30399999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.593,"5min":58.615000000000002,"15min":58.615000000000002},"last":0.752}]},{"osd":6,"last update":"Mon Feb 5 08:14:29 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.4390000000000001,"15min":1.1080000000000001},"min":{"1min":0.441,"5min":0.32500000000000001,"15min":0.308},"max":{"1min":4.7770000000000001,"5min":7.5099999999999998,"15min":15.859},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.3149999999999999,"15min":0.98499999999999999},"min":{"1min":0.35899999999999999,"5min":0.34100000000000003,"15min":0.32900000000000001},"max":{"1min":3.7610000000000001,"5min":6.1820000000000004,"15min":6.1820000000000004},"last":0.52600000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3959999999999999,"15min":1.091},"min":{"1min":0.33100000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.516,"5min":7.3600000000000003,"15min":15.986000000000001},"last":0.498},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.5429999999999999,"15min":1.171},"min":{"1min":0.41699999999999998,"5min":0.38,"15min":0.35299999999999998},"max":{"1min":4.6749999999999998,"5min":7.4779999999999998,"15min":15.951000000000001},"last":1.379}]}]},{"osd":3,"up_from":0,"seq":115964117146,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953622,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":848444,"kb_used_data":502132,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92896708,"statfs":{"total":95995035648,"available":95126228992,"internally_reserved":0,"allocated":514183168,"data_stored":513797850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":1.6819999999999999,"15min":1.4890000000000001},"min":{"1min":0.26800000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.4769999999999999,"5min":39.552,"15min":73.078999999999994},"last":1.3919999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.6990000000000001,"15min":1.444},"min":{"1min":0.29999999999999999,"5min":0.214,"15min":0.21099999999999999},"max":{"1min":3.0489999999999999,"5min":39.350999999999999,"15min":73.412000000000006},"last":0.40100000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":2.536,"15min":1.351},"min":{"1min":0.33700000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.6419999999999999,"5min":125.054,"15min":125.054},"last":1.478},{"interface":"front","average":{"1min":0.872,"5min":2.5910000000000002,"15min":1.2929999999999999},"min":{"1min":0.308,"5min":0.249,"15min":0.23499999999999999},"max":{"1min":5.1260000000000003,"5min":125.08799999999999,"15min":125.08799999999999},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.995,"15min":1.1379999999999999},"min":{"1min":0.28199999999999997,"5min":0.247,"15min":0.24199999999999999},"max":{"1min":1.9750000000000001,"5min":5.649,"15min":96.849999999999994},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.0700000000000001,"15min":1.1970000000000001},"min":{"1min":0.41799999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.79,"5min":4.1299999999999999,"15min":96.143000000000001},"last":0.65300000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.9950000000000001,"15min":1.698},"min":{"1min":0.376,"5min":0.28599999999999998,"15min":0.24399999999999999},"max":{"1min":2.1160000000000001,"5min":57.204000000000001,"15min":73.602999999999994},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":2.0510000000000002,"15min":1.7210000000000001},"min":{"1min":0.36899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76400000000000001,"5min":57.606999999999999,"15min":74.817999999999998},"last":1.4099999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:15:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7690000000000001,"5min":1.0780000000000001,"15min":0.61699999999999999},"min":{"1min":0.39900000000000002,"5min":0.224,"15min":0.224},"max":{"1min":34.167000000000002,"5min":34.167000000000002,"15min":34.167000000000002},"last":1.371},{"interface":"front","average":{"1min":3.1789999999999998,"5min":1.252,"15min":0.69599999999999995},"min":{"1min":0.29599999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":36.051000000000002,"5min":36.051000000000002,"15min":36.051000000000002},"last":0.96799999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.72699999999999998,"15min":0.53700000000000003},"min":{"1min":0.37,"5min":0.29599999999999999,"15min":0.29299999999999998},"max":{"1min":2.0800000000000001,"5min":3.9220000000000002,"15min":3.9220000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64500000000000002,"15min":0.51000000000000001},"min":{"1min":0.47099999999999997,"5min":0.32000000000000001,"15min":0.29499999999999998},"max":{"1min":0.83799999999999997,"5min":3.306,"15min":3.306},"last":1.2130000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:14:38 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":0.75700000000000001,"15min":0.56699999999999995},"min":{"1min":0.314,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.4859999999999998,"5min":4.4859999999999998,"15min":4.4859999999999998},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.70499999999999996,"15min":0.56999999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.7450000000000001,"5min":5.7450000000000001,"15min":5.7450000000000001},"last":1.5289999999999999}]}]},{"osd":5,"up_from":42,"seq":180388626575,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1182184,"kb_used_data":862240,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92562968,"statfs":{"total":95995035648,"available":94784479232,"internally_reserved":0,"allocated":882933760,"data_stored":882561636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:14:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.786,"15min":1.5309999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.19400000000000001},"max":{"1min":3.794,"5min":46.962000000000003,"15min":143.06299999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.73,"15min":1.516},"min":{"1min":0.27200000000000002,"5min":0.23699999999999999,"15min":0.20499999999999999},"max":{"1min":4.7560000000000002,"5min":47.067,"15min":143.09700000000001},"last":1.581}]},{"osd":1,"last update":"Mon Feb 5 08:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.5469999999999999,"15min":0.93899999999999995},"min":{"1min":0.27600000000000002,"5min":0.25,"15min":0.215},"max":{"1min":4.2519999999999998,"5min":46.200000000000003,"15min":46.200000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.159,"5min":1.8720000000000001,"15min":1.0569999999999999},"min":{"1min":0.27400000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.8760000000000003,"5min":46.343000000000004,"15min":46.343000000000004},"last":1.556}]},{"osd":2,"last update":"Mon Feb 5 08:14:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.262,"15min":0.91300000000000003},"min":{"1min":0.379,"5min":0.246,"15min":0.246},"max":{"1min":4.2439999999999998,"5min":10.733000000000001,"15min":10.733000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.222,"5min":1.24,"15min":0.93999999999999995},"min":{"1min":0.29099999999999998,"5min":0.28399999999999997,"15min":0.26400000000000001},"max":{"1min":4.3970000000000002,"5min":8.0220000000000002,"15min":8.0220000000000002},"last":0.53700000000000003}]},{"osd":3,"last update":"Mon Feb 5 08:14:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.635,"15min":1.5249999999999999},"min":{"1min":0.371,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.6560000000000001,"5min":46.569000000000003,"15min":143.03399999999999},"last":1.482},{"interface":"front","average":{"1min":1.252,"5min":2.052,"15min":1.6160000000000001},"min":{"1min":0.34100000000000003,"5min":0.32600000000000001,"15min":0.28100000000000003},"max":{"1min":4.3070000000000004,"5min":47.820999999999998,"15min":143.46799999999999},"last":0.61299999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:14:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":0.79900000000000004,"15min":0.54300000000000004},"min":{"1min":0.24399999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.8300000000000001,"5min":4.8689999999999998,"15min":4.8689999999999998},"last":1.609},{"interface":"front","average":{"1min":1.024,"5min":0.65700000000000003,"15min":0.50800000000000001},"min":{"1min":0.34399999999999997,"5min":0.28299999999999997,"15min":0.26500000000000001},"max":{"1min":3.7149999999999999,"5min":3.7149999999999999,"15min":3.7149999999999999},"last":0.39100000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:15:17 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.73799999999999999,"15min":0.54400000000000004},"min":{"1min":0.314,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":3.5859999999999999,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":1.518},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.76200000000000001,"15min":0.57999999999999996},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":3.6859999999999999,"5min":6.6180000000000003,"15min":6.6180000000000003},"last":1.5449999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:14:50 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.76100000000000001,"15min":0.57099999999999995},"min":{"1min":0.38300000000000001,"5min":0.34399999999999997,"15min":0.25900000000000001},"max":{"1min":3.8290000000000002,"5min":4.3300000000000001,"15min":4.3300000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.79500000000000004,"15min":0.59399999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.28199999999999997},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":2.169}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":607780864,"data_stored":607780864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":893779968,"data_stored":893779968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":482803712,"data_stored":482803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":529858560,"data_stored":529858560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":956301312,"data_stored":956301312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":401997824,"data_stored":401997824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1029767168,"data_stored":1029767168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:15:29.658 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:15:29.715 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:29.740 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:29.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:29.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:29.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:29.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:29.980 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:30.006 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:30.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:30.064 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:15:30.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:30.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:30.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:30.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:30.263 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:30.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:30 smithi196 ceph-mon[33018]: 5.0 scrub starts 2024-02-05T08:15:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:30 smithi196 ceph-mon[33018]: 5.0 scrub ok 2024-02-05T08:15:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:30 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3644119152' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:15:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:30 smithi196 ceph-mon[33018]: 6.3 scrub starts 2024-02-05T08:15:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:30 smithi196 ceph-mon[33018]: 6.3 scrub ok 2024-02-05T08:15:30.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:30.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:30.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:30.410 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:30 smithi134 ceph-mon[30086]: 5.0 scrub starts 2024-02-05T08:15:30.410 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:30 smithi134 ceph-mon[30086]: 5.0 scrub ok 2024-02-05T08:15:30.410 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:30 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3644119152' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:15:30.410 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:30 smithi134 ceph-mon[30086]: 6.3 scrub starts 2024-02-05T08:15:30.410 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:30 smithi134 ceph-mon[30086]: 6.3 scrub ok 2024-02-05T08:15:30.411 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:30 smithi134 ceph-mon[39278]: 5.0 scrub starts 2024-02-05T08:15:30.411 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:30 smithi134 ceph-mon[39278]: 5.0 scrub ok 2024-02-05T08:15:30.411 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:30 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3644119152' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:15:30.411 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:30 smithi134 ceph-mon[39278]: 6.3 scrub starts 2024-02-05T08:15:30.411 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:30 smithi134 ceph-mon[39278]: 6.3 scrub ok 2024-02-05T08:15:30.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 16 16 36067 36051 140.791 56 0.00755095 0.00709426 2024-02-05T08:15:30.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:30.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:30.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:30.635 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:30.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:30.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:30.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:30.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:30.797 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:30.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:30.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:30.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:30.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:30.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:31.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:31.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:31.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:31.126 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[30086]: 3.b scrub starts 2024-02-05T08:15:31.126 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[30086]: 3.b scrub ok 2024-02-05T08:15:31.126 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[30086]: pgmap v474: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-02-05T08:15:31.126 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[30086]: 6.6 scrub starts 2024-02-05T08:15:31.126 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[30086]: 3.1e deep-scrub starts 2024-02-05T08:15:31.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[30086]: 3.1e deep-scrub ok 2024-02-05T08:15:31.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[30086]: 6.6 scrub ok 2024-02-05T08:15:31.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:15:31.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3644119152' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:15:31.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[30086]: osdmap e250: 8 total, 8 up, 7 in 2024-02-05T08:15:31.127 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[39278]: 3.b scrub starts 2024-02-05T08:15:31.127 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[39278]: 3.b scrub ok 2024-02-05T08:15:31.127 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[39278]: pgmap v474: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-02-05T08:15:31.127 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[39278]: 6.6 scrub starts 2024-02-05T08:15:31.127 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[39278]: 3.1e deep-scrub starts 2024-02-05T08:15:31.127 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[39278]: 3.1e deep-scrub ok 2024-02-05T08:15:31.127 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[39278]: 6.6 scrub ok 2024-02-05T08:15:31.128 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:15:31.128 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3644119152' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:15:31.128 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:31 smithi134 ceph-mon[39278]: osdmap e250: 8 total, 8 up, 7 in 2024-02-05T08:15:31.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:31.199 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:31.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:31.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:31.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:31 smithi196 ceph-mon[33018]: 3.b scrub starts 2024-02-05T08:15:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:31 smithi196 ceph-mon[33018]: 3.b scrub ok 2024-02-05T08:15:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:31 smithi196 ceph-mon[33018]: pgmap v474: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-02-05T08:15:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:31 smithi196 ceph-mon[33018]: 6.6 scrub starts 2024-02-05T08:15:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:31 smithi196 ceph-mon[33018]: 3.1e deep-scrub starts 2024-02-05T08:15:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:31 smithi196 ceph-mon[33018]: 3.1e deep-scrub ok 2024-02-05T08:15:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:31 smithi196 ceph-mon[33018]: 6.6 scrub ok 2024-02-05T08:15:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:31 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:15:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:31 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3644119152' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:15:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:31 smithi196 ceph-mon[33018]: osdmap e250: 8 total, 8 up, 7 in 2024-02-05T08:15:31.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:31.333 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:31.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:31.434 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:31.470 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:31.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:31.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:31.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 17 16 37725 37709 138.604 103.625 0.00730654 0.00720849 2024-02-05T08:15:31.572 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:31.610 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:31.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:31.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:31.724 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:31.748 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:31.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:31.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:31.883 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:31.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:31.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:31.985 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:32.024 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:32.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:32.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:32.071 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:15:32.110 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:15:32.126 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:32.145 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:15:32.165 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:32.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:32.184 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:15:32.195 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:32 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3055491534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:32.223 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:15:32.262 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:15:32.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:32.308 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:15:32.311 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:32.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:32.346 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:15:32.346 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:32 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3055491534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:32.347 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:32 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3055491534' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:32.349 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:15:32.349 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:15:32.349 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:15:32.349 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:15:32.349 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:15:32.349 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:15:32.349 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:15:32.349 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:15:32.413 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:32.450 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:32.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:32.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:32.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 18 16 39597 39581 137.404 117 0.004443 0.00727292 2024-02-05T08:15:32.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:32.588 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:32.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:32.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:32.690 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:32.724 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:32.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:32.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:32.726 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:15:32.726 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:15:32.826 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:32.861 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:32.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:32.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:33.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:33.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:33.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:33.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:33.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:33.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:33.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:33.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:33.278 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:33.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:33 smithi196 ceph-mon[33018]: pgmap v476: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-02-05T08:15:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:33 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:15:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:33 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:15:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:33 smithi196 ceph-mon[33018]: osdmap e251: 8 total, 8 up, 7 in 2024-02-05T08:15:33.380 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:33.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T08:15:33.412 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:33.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:33.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:33.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 19 16 41447 41431 136.257 115.625 0.0106172 0.00733405 2024-02-05T08:15:33.512 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:15:33.514 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:33.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:33 smithi134 ceph-mon[30086]: pgmap v476: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-02-05T08:15:33.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:33 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:15:33.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:33 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:15:33.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:33 smithi134 ceph-mon[30086]: osdmap e251: 8 total, 8 up, 7 in 2024-02-05T08:15:33.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:33 smithi134 ceph-mon[39278]: pgmap v476: 148 pgs: 148 active+clean; 2.0 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 136 MiB/s wr, 2.18k op/s 2024-02-05T08:15:33.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:33 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:15:33.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:33 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:15:33.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:33 smithi134 ceph-mon[39278]: osdmap e251: 8 total, 8 up, 7 in 2024-02-05T08:15:33.552 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:33.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:33.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:33.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:33.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:33.694 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:33.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:33.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:33.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:15:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:15:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:33.844 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:33.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:33.845 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:15:33.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:33.989 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:33.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:34.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:34.126 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:34.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:34.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:34.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:34.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:34.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:34 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:15:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:34 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:15:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:34 smithi196 ceph-mon[33018]: osdmap e252: 8 total, 8 up, 7 in 2024-02-05T08:15:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:34 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:34.368 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:34.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:34.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:34.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:15:34.482272+0000 min lat: 0.00130673 max lat: 0.198515 avg lat: 0.00741532 2024-02-05T08:15:34.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:15:34.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 20 16 43149 43133 134.763 106.375 0.00892549 0.00741532 2024-02-05T08:15:34.508 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:34.522 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:34 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:15:34.522 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:34 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:15:34.522 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:34 smithi134 ceph-mon[39278]: osdmap e252: 8 total, 8 up, 7 in 2024-02-05T08:15:34.522 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:34 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:34.522 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:34 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:15:34.522 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:34 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:15:34.522 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:34 smithi134 ceph-mon[30086]: osdmap e252: 8 total, 8 up, 7 in 2024-02-05T08:15:34.523 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:34 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:34.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:34.550 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:34.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:34.579 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:34.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:34.694 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:34.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:34.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:34.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:34.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:34.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:34.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:34.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:34.981 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:34.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:35.082 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:35.127 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:35.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:35.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:35.229 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:35.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:35 smithi196 ceph-mon[33018]: pgmap v479: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 164 MiB/s wr, 2.62k op/s 2024-02-05T08:15:35.230 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:35 smithi196 ceph-mon[33018]: osdmap e253: 8 total, 8 up, 7 in 2024-02-05T08:15:35.270 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:35.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:35.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:35.393 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:35 smithi134 ceph-mon[30086]: pgmap v479: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 164 MiB/s wr, 2.62k op/s 2024-02-05T08:15:35.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:35 smithi134 ceph-mon[30086]: osdmap e253: 8 total, 8 up, 7 in 2024-02-05T08:15:35.394 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:35 smithi134 ceph-mon[39278]: pgmap v479: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.5 KiB/s rd, 164 MiB/s wr, 2.62k op/s 2024-02-05T08:15:35.394 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:35 smithi134 ceph-mon[39278]: osdmap e253: 8 total, 8 up, 7 in 2024-02-05T08:15:35.411 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:35.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:35.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:35.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 21 16 44771 44755 133.173 101.375 0.00557324 0.00750451 2024-02-05T08:15:35.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:35.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:35.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:35.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:35.648 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:15:35.651 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:35.654 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:15:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:35.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:35.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:35.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:35.812 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:15:35.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:35.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:35.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:35.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:35.972 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:35.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:35.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:36.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:36.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:36.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:36.224 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:36.273 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:36.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:36.374 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:36.416 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:36.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:36.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:36.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 22 16 46225 46209 131.25 90.875 0.00924686 0.00761384 2024-02-05T08:15:36.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:36.552 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:36.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:36.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:36.693 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:36.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:36.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:36.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:36.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:36.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:36.904 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:15:36.904 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:15:36.931 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:36.935 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:36.988 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:36.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:37.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:37.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:37.126 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:37.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:37.226 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:37.263 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:37.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:37.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:37.284 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:37 smithi196 ceph-mon[33018]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:15:37.285 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:37 smithi196 ceph-mon[33018]: pgmap v481: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 724 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T08:15:37.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:37.404 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:37.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:37.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 23 16 47780 47764 129.769 97.1875 0.00799229 0.00770043 2024-02-05T08:15:37.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:37.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:37 smithi134 ceph-mon[30086]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:15:37.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:37 smithi134 ceph-mon[30086]: pgmap v481: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 724 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T08:15:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:37 smithi134 ceph-mon[39278]: from='client.25259 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:15:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:37 smithi134 ceph-mon[39278]: pgmap v481: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 724 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T08:15:37.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:37.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:37.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T08:15:37.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:37.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:37.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:37.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:37.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:37.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:37.828 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:37.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:37.839 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-02-05T08:15:34.728675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2661292380,"num_objects":40797,"num_object_clones":0,"num_object_copies":122391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40797,"num_whiteouts":0,"num_read":1996,"num_read_kb":1842,"num_write":41500,"num_write_kb":2600797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":2957685,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41563,"ondisk_log_size":41563,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":10053704,"kb_used_data":7722492,"kb_used_omap":10,"kb_used_meta":2331125,"kb_avail":646162360,"statfs":{"total":671965249536,"available":661670256640,"internally_reserved":0,"allocated":7907831808,"data_stored":7905028257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":2387072106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,7,4,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":512098304,"num_objects":7814,"num_object_clones":0,"num_object_copies":23442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7814,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":7814,"num_write_kb":500096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.019097"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":410,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.079045+0000","last_change":"2024-02-05T08:15:16.927873+0000","last_active":"2024-02-05T08:15:32.079045+0000","last_peered":"2024-02-05T08:15:32.079045+0000","last_clean":"2024-02-05T08:15:32.079045+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:15:32.079045+0000","last_undegraded":"2024-02-05T08:15:32.079045+0000","last_fullsized":"2024-02-05T08:15:32.079045+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:16.927723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:16.927723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:39.613027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":417,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.080140+0000","last_change":"2024-02-05T08:15:21.824643+0000","last_active":"2024-02-05T08:15:33.080140+0000","last_peered":"2024-02-05T08:15:33.080140+0000","last_clean":"2024-02-05T08:15:33.080140+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:15:33.080140+0000","last_undegraded":"2024-02-05T08:15:33.080140+0000","last_fullsized":"2024-02-05T08:15:33.080140+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:21.824531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:21.824531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:31.555157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":416,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.086648+0000","last_change":"2024-02-05T08:14:10.178234+0000","last_active":"2024-02-05T08:15:33.086648+0000","last_peered":"2024-02-05T08:15:33.086648+0000","last_clean":"2024-02-05T08:15:33.086648+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:15:33.086648+0000","last_undegraded":"2024-02-05T08:15:33.086648+0000","last_fullsized":"2024-02-05T08:15:33.086648+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:10.178136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:10.178136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.461584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"236'19","reported_seq":471,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.077468+0000","last_change":"2024-02-05T08:14:03.970129+0000","last_active":"2024-02-05T08:15:32.077468+0000","last_peered":"2024-02-05T08:15:32.077468+0000","last_clean":"2024-02-05T08:15:32.077468+0000","last_became_active":"2024-02-05T08:12:29.675863+0000","last_became_peered":"2024-02-05T08:12:29.675863+0000","last_unstale":"2024-02-05T08:15:32.077468+0000","last_undegraded":"2024-02-05T08:15:32.077468+0000","last_fullsized":"2024-02-05T08:15:32.077468+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:14:03.970014+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:14:03.970014+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":405,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086555+0000","last_change":"2024-02-05T08:15:22.548444+0000","last_active":"2024-02-05T08:15:32.086555+0000","last_peered":"2024-02-05T08:15:32.086555+0000","last_clean":"2024-02-05T08:15:32.086555+0000","last_became_active":"2024-02-05T08:12:41.722414+0000","last_became_peered":"2024-02-05T08:12:41.722414+0000","last_unstale":"2024-02-05T08:15:32.086555+0000","last_undegraded":"2024-02-05T08:15:32.086555+0000","last_fullsized":"2024-02-05T08:15:32.086555+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.548159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:38.794097+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":418,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.077674+0000","last_change":"2024-02-05T08:14:13.968249+0000","last_active":"2024-02-05T08:15:32.077674+0000","last_peered":"2024-02-05T08:15:32.077674+0000","last_clean":"2024-02-05T08:15:32.077674+0000","last_became_active":"2024-02-05T08:12:29.676482+0000","last_became_peered":"2024-02-05T08:12:29.676482+0000","last_unstale":"2024-02-05T08:15:32.077674+0000","last_undegraded":"2024-02-05T08:15:32.077674+0000","last_fullsized":"2024-02-05T08:15:32.077674+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:13.968037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:14:13.968037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.507067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":140,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.081768+0000","last_change":"2024-02-05T08:15:29.985455+0000","last_active":"2024-02-05T08:15:32.081768+0000","last_peered":"2024-02-05T08:15:32.081768+0000","last_clean":"2024-02-05T08:15:32.081768+0000","last_became_active":"2024-02-05T08:12:41.727450+0000","last_became_peered":"2024-02-05T08:12:41.727450+0000","last_unstale":"2024-02-05T08:15:32.081768+0000","last_undegraded":"2024-02-05T08:15:32.081768+0000","last_fullsized":"2024-02-05T08:15:32.081768+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:15:29.985333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:55.457550+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"236'34","reported_seq":321,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.079696+0000","last_change":"2024-02-05T08:15:24.771294+0000","last_active":"2024-02-05T08:15:32.079696+0000","last_peered":"2024-02-05T08:15:32.079696+0000","last_clean":"2024-02-05T08:15:32.079696+0000","last_became_active":"2024-02-05T08:12:41.745917+0000","last_became_peered":"2024-02-05T08:12:41.745917+0000","last_unstale":"2024-02-05T08:15:32.079696+0000","last_undegraded":"2024-02-05T08:15:32.079696+0000","last_fullsized":"2024-02-05T08:15:32.079696+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:15:24.771146+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:15:24.771146+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:25.971833+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":403,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.086283+0000","last_change":"2024-02-05T08:14:14.237605+0000","last_active":"2024-02-05T08:15:33.086283+0000","last_peered":"2024-02-05T08:15:33.086283+0000","last_clean":"2024-02-05T08:15:33.086283+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:15:33.086283+0000","last_undegraded":"2024-02-05T08:15:33.086283+0000","last_fullsized":"2024-02-05T08:15:33.086283+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:14.237096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:14.237096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.461558+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":414,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.084610+0000","last_change":"2024-02-05T08:15:22.941148+0000","last_active":"2024-02-05T08:15:32.084610+0000","last_peered":"2024-02-05T08:15:32.084610+0000","last_clean":"2024-02-05T08:15:32.084610+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:15:32.084610+0000","last_undegraded":"2024-02-05T08:15:32.084610+0000","last_fullsized":"2024-02-05T08:15:32.084610+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.940396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.940396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:47.298258+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":434,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.085302+0000","last_change":"2024-02-05T08:14:27.200067+0000","last_active":"2024-02-05T08:15:32.085302+0000","last_peered":"2024-02-05T08:15:32.085302+0000","last_clean":"2024-02-05T08:15:32.085302+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:15:32.085302+0000","last_undegraded":"2024-02-05T08:15:32.085302+0000","last_fullsized":"2024-02-05T08:15:32.085302+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:27.199962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:27.199962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:32.610560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"236'15","reported_seq":457,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.079255+0000","last_change":"2024-02-05T08:14:16.970924+0000","last_active":"2024-02-05T08:15:32.079255+0000","last_peered":"2024-02-05T08:15:32.079255+0000","last_clean":"2024-02-05T08:15:32.079255+0000","last_became_active":"2024-02-05T08:12:41.731452+0000","last_became_peered":"2024-02-05T08:12:41.731452+0000","last_unstale":"2024-02-05T08:15:32.079255+0000","last_undegraded":"2024-02-05T08:15:32.079255+0000","last_fullsized":"2024-02-05T08:15:32.079255+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:14:16.970843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:16.970843+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:32.592772+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":408,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.084557+0000","last_change":"2024-02-05T08:12:41.722105+0000","last_active":"2024-02-05T08:15:32.084557+0000","last_peered":"2024-02-05T08:15:32.084557+0000","last_clean":"2024-02-05T08:15:32.084557+0000","last_became_active":"2024-02-05T08:12:41.721611+0000","last_became_peered":"2024-02-05T08:12:41.721611+0000","last_unstale":"2024-02-05T08:15:32.084557+0000","last_undegraded":"2024-02-05T08:15:32.084557+0000","last_fullsized":"2024-02-05T08:15:32.084557+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:14.301656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:14.301656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.073722+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":415,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.085747+0000","last_change":"2024-02-05T08:15:18.440781+0000","last_active":"2024-02-05T08:15:32.085747+0000","last_peered":"2024-02-05T08:15:32.085747+0000","last_clean":"2024-02-05T08:15:32.085747+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:15:32.085747+0000","last_undegraded":"2024-02-05T08:15:32.085747+0000","last_fullsized":"2024-02-05T08:15:32.085747+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:18.440664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:18.440664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:34.989634+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":422,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.084122+0000","last_change":"2024-02-05T08:15:20.904492+0000","last_active":"2024-02-05T08:15:32.084122+0000","last_peered":"2024-02-05T08:15:32.084122+0000","last_clean":"2024-02-05T08:15:32.084122+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:15:32.084122+0000","last_undegraded":"2024-02-05T08:15:32.084122+0000","last_fullsized":"2024-02-05T08:15:32.084122+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:20.904369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:15:20.904369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:21.716191+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":435,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078011+0000","last_change":"2024-02-05T08:14:18.923493+0000","last_active":"2024-02-05T08:15:32.078011+0000","last_peered":"2024-02-05T08:15:32.078011+0000","last_clean":"2024-02-05T08:15:32.078011+0000","last_became_active":"2024-02-05T08:12:41.730485+0000","last_became_peered":"2024-02-05T08:12:41.730485+0000","last_unstale":"2024-02-05T08:15:32.078011+0000","last_undegraded":"2024-02-05T08:15:32.078011+0000","last_fullsized":"2024-02-05T08:15:32.078011+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:14:18.923276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:18.923276+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:32.127559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":423,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078283+0000","last_change":"2024-02-05T08:15:13.039454+0000","last_active":"2024-02-05T08:15:32.078283+0000","last_peered":"2024-02-05T08:15:32.078283+0000","last_clean":"2024-02-05T08:15:32.078283+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:15:32.078283+0000","last_undegraded":"2024-02-05T08:15:32.078283+0000","last_fullsized":"2024-02-05T08:15:32.078283+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:13.039321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:15:13.039321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:40.716512+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":409,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.080678+0000","last_change":"2024-02-05T08:15:11.836173+0000","last_active":"2024-02-05T08:15:33.080678+0000","last_peered":"2024-02-05T08:15:33.080678+0000","last_clean":"2024-02-05T08:15:33.080678+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:15:33.080678+0000","last_undegraded":"2024-02-05T08:15:33.080678+0000","last_fullsized":"2024-02-05T08:15:33.080678+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:11.836098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:15:11.836098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:20.798265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":415,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087808+0000","last_change":"2024-02-05T08:15:22.341597+0000","last_active":"2024-02-05T08:15:33.087808+0000","last_peered":"2024-02-05T08:15:33.087808+0000","last_clean":"2024-02-05T08:15:33.087808+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:15:33.087808+0000","last_undegraded":"2024-02-05T08:15:33.087808+0000","last_fullsized":"2024-02-05T08:15:33.087808+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.341491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.341491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:41.569521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"236'34","reported_seq":498,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.079927+0000","last_change":"2024-02-05T08:14:31.867150+0000","last_active":"2024-02-05T08:15:32.079927+0000","last_peered":"2024-02-05T08:15:32.079927+0000","last_clean":"2024-02-05T08:15:32.079927+0000","last_became_active":"2024-02-05T08:12:41.727027+0000","last_became_peered":"2024-02-05T08:12:41.727027+0000","last_unstale":"2024-02-05T08:15:32.079927+0000","last_undegraded":"2024-02-05T08:15:32.079927+0000","last_fullsized":"2024-02-05T08:15:32.079927+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'22","last_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.867008+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:52.313837+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":403,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.081851+0000","last_change":"2024-02-05T08:14:33.244556+0000","last_active":"2024-02-05T08:15:32.081851+0000","last_peered":"2024-02-05T08:15:32.081851+0000","last_clean":"2024-02-05T08:15:32.081851+0000","last_became_active":"2024-02-05T08:12:41.713564+0000","last_became_peered":"2024-02-05T08:12:41.713564+0000","last_unstale":"2024-02-05T08:15:32.081851+0000","last_undegraded":"2024-02-05T08:15:32.081851+0000","last_fullsized":"2024-02-05T08:15:32.081851+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:33.244471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:33.244471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:39.761831+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":420,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078698+0000","last_change":"2024-02-05T08:15:25.922691+0000","last_active":"2024-02-05T08:15:32.078698+0000","last_peered":"2024-02-05T08:15:32.078698+0000","last_clean":"2024-02-05T08:15:32.078698+0000","last_became_active":"2024-02-05T08:12:41.735240+0000","last_became_peered":"2024-02-05T08:12:41.735240+0000","last_unstale":"2024-02-05T08:15:32.078698+0000","last_undegraded":"2024-02-05T08:15:32.078698+0000","last_fullsized":"2024-02-05T08:15:32.078698+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.922139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.922139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:46.247310+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":414,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.079343+0000","last_change":"2024-02-05T08:15:19.927561+0000","last_active":"2024-02-05T08:15:32.079343+0000","last_peered":"2024-02-05T08:15:32.079343+0000","last_clean":"2024-02-05T08:15:32.079343+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:15:32.079343+0000","last_undegraded":"2024-02-05T08:15:32.079343+0000","last_fullsized":"2024-02-05T08:15:32.079343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:19.927454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:19.927454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:22.894101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"236'23","reported_seq":236,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.080565+0000","last_change":"2024-02-05T08:14:09.718995+0000","last_active":"2024-02-05T08:15:33.080565+0000","last_peered":"2024-02-05T08:15:33.080565+0000","last_clean":"2024-02-05T08:15:33.080565+0000","last_became_active":"2024-02-05T08:12:41.716794+0000","last_became_peered":"2024-02-05T08:12:41.716794+0000","last_unstale":"2024-02-05T08:15:33.080565+0000","last_undegraded":"2024-02-05T08:15:33.080565+0000","last_fullsized":"2024-02-05T08:15:33.080565+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:14:09.718864+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:14:09.718864+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":421,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086956+0000","last_change":"2024-02-05T08:14:26.616349+0000","last_active":"2024-02-05T08:15:32.086956+0000","last_peered":"2024-02-05T08:15:32.086956+0000","last_clean":"2024-02-05T08:15:32.086956+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:15:32.086956+0000","last_undegraded":"2024-02-05T08:15:32.086956+0000","last_fullsized":"2024-02-05T08:15:32.086956+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:14:26.616223+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:14:26.616223+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:34.032614+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":185,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.081949+0000","last_change":"2024-02-05T08:14:18.174978+0000","last_active":"2024-02-05T08:15:32.081949+0000","last_peered":"2024-02-05T08:15:32.081949+0000","last_clean":"2024-02-05T08:15:32.081949+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:15:32.081949+0000","last_undegraded":"2024-02-05T08:15:32.081949+0000","last_fullsized":"2024-02-05T08:15:32.081949+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:14:18.174860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:18.174860+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:33.966481+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":428,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.080024+0000","last_change":"2024-02-05T08:14:10.699857+0000","last_active":"2024-02-05T08:15:33.080024+0000","last_peered":"2024-02-05T08:15:33.080024+0000","last_clean":"2024-02-05T08:15:33.080024+0000","last_became_active":"2024-02-05T08:12:41.719010+0000","last_became_peered":"2024-02-05T08:12:41.719010+0000","last_unstale":"2024-02-05T08:15:33.080024+0000","last_undegraded":"2024-02-05T08:15:33.080024+0000","last_fullsized":"2024-02-05T08:15:33.080024+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:14:10.699732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:10.699732+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"236'20","reported_seq":496,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087279+0000","last_change":"2024-02-05T08:15:26.443674+0000","last_active":"2024-02-05T08:15:33.087279+0000","last_peered":"2024-02-05T08:15:33.087279+0000","last_clean":"2024-02-05T08:15:33.087279+0000","last_became_active":"2024-02-05T08:12:41.757718+0000","last_became_peered":"2024-02-05T08:12:41.757718+0000","last_unstale":"2024-02-05T08:15:33.087279+0000","last_undegraded":"2024-02-05T08:15:33.087279+0000","last_fullsized":"2024-02-05T08:15:33.087279+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:15:26.443406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:26.443406+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:30.531093+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":405,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.080764+0000","last_change":"2024-02-05T08:11:33.834287+0000","last_active":"2024-02-05T08:15:33.080764+0000","last_peered":"2024-02-05T08:15:33.080764+0000","last_clean":"2024-02-05T08:15:33.080764+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:15:33.080764+0000","last_undegraded":"2024-02-05T08:15:33.080764+0000","last_fullsized":"2024-02-05T08:15:33.080764+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:33.834034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:33.834034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":408,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087513+0000","last_change":"2024-02-05T08:14:32.232117+0000","last_active":"2024-02-05T08:15:33.087513+0000","last_peered":"2024-02-05T08:15:33.087513+0000","last_clean":"2024-02-05T08:15:33.087513+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:15:33.087513+0000","last_undegraded":"2024-02-05T08:15:33.087513+0000","last_fullsized":"2024-02-05T08:15:33.087513+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:32.231861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:32.231861+0000","last_clean_scrub_stamp":"2024-02-05T08:14:32.231861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.000116+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":136,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.086199+0000","last_change":"2024-02-05T08:14:31.199352+0000","last_active":"2024-02-05T08:15:33.086199+0000","last_peered":"2024-02-05T08:15:33.086199+0000","last_clean":"2024-02-05T08:15:33.086199+0000","last_became_active":"2024-02-05T08:12:41.727406+0000","last_became_peered":"2024-02-05T08:12:41.727406+0000","last_unstale":"2024-02-05T08:15:33.086199+0000","last_undegraded":"2024-02-05T08:15:33.086199+0000","last_fullsized":"2024-02-05T08:15:33.086199+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:31.199238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.199238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"236'19","reported_seq":168,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.081750+0000","last_change":"2024-02-05T08:14:26.230319+0000","last_active":"2024-02-05T08:15:32.081750+0000","last_peered":"2024-02-05T08:15:32.081750+0000","last_clean":"2024-02-05T08:15:32.081750+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:15:32.081750+0000","last_undegraded":"2024-02-05T08:15:32.081750+0000","last_fullsized":"2024-02-05T08:15:32.081750+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:14:26.230191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:26.230191+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:44.242536+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":403,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.085894+0000","last_change":"2024-02-05T08:13:35.143784+0000","last_active":"2024-02-05T08:15:32.085894+0000","last_peered":"2024-02-05T08:15:32.085894+0000","last_clean":"2024-02-05T08:15:32.085894+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:15:32.085894+0000","last_undegraded":"2024-02-05T08:15:32.085894+0000","last_fullsized":"2024-02-05T08:15:32.085894+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:35.143602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:13:35.143602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.476338+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":144,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.081245+0000","last_change":"2024-02-05T08:15:12.041843+0000","last_active":"2024-02-05T08:15:32.081245+0000","last_peered":"2024-02-05T08:15:32.081245+0000","last_clean":"2024-02-05T08:15:32.081245+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:15:32.081245+0000","last_undegraded":"2024-02-05T08:15:32.081245+0000","last_fullsized":"2024-02-05T08:15:32.081245+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:12.041759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:15:12.041759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:41.328162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":421,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.086844+0000","last_change":"2024-02-05T08:15:11.364628+0000","last_active":"2024-02-05T08:15:33.086844+0000","last_peered":"2024-02-05T08:15:33.086844+0000","last_clean":"2024-02-05T08:15:33.086844+0000","last_became_active":"2024-02-05T08:12:41.736991+0000","last_became_peered":"2024-02-05T08:12:41.736991+0000","last_unstale":"2024-02-05T08:15:33.086844+0000","last_undegraded":"2024-02-05T08:15:33.086844+0000","last_fullsized":"2024-02-05T08:15:33.086844+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:11.364552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:11.364552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:18.417672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":174,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.080455+0000","last_change":"2024-02-05T08:15:23.861142+0000","last_active":"2024-02-05T08:15:33.080455+0000","last_peered":"2024-02-05T08:15:33.080455+0000","last_clean":"2024-02-05T08:15:33.080455+0000","last_became_active":"2024-02-05T08:12:41.717459+0000","last_became_peered":"2024-02-05T08:12:41.717459+0000","last_unstale":"2024-02-05T08:15:33.080455+0000","last_undegraded":"2024-02-05T08:15:33.080455+0000","last_fullsized":"2024-02-05T08:15:33.080455+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:15:23.861011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:23.861011+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:40.974821+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":398,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087646+0000","last_change":"2024-02-05T08:12:41.738206+0000","last_active":"2024-02-05T08:15:33.087646+0000","last_peered":"2024-02-05T08:15:33.087646+0000","last_clean":"2024-02-05T08:15:33.087646+0000","last_became_active":"2024-02-05T08:12:41.737803+0000","last_became_peered":"2024-02-05T08:12:41.737803+0000","last_unstale":"2024-02-05T08:15:33.087646+0000","last_undegraded":"2024-02-05T08:15:33.087646+0000","last_fullsized":"2024-02-05T08:15:33.087646+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:33.610823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:33.610823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.461425+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":427,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086565+0000","last_change":"2024-02-05T08:13:19.251251+0000","last_active":"2024-02-05T08:15:32.086565+0000","last_peered":"2024-02-05T08:15:32.086565+0000","last_clean":"2024-02-05T08:15:32.086565+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:15:32.086565+0000","last_undegraded":"2024-02-05T08:15:32.086565+0000","last_fullsized":"2024-02-05T08:15:32.086565+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:19.251134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:13:19.251134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.476261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":422,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.084869+0000","last_change":"2024-02-05T08:13:10.189057+0000","last_active":"2024-02-05T08:15:32.084869+0000","last_peered":"2024-02-05T08:15:32.084869+0000","last_clean":"2024-02-05T08:15:32.084869+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:15:32.084869+0000","last_undegraded":"2024-02-05T08:15:32.084869+0000","last_fullsized":"2024-02-05T08:15:32.084869+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:10.188977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:13:10.188977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.073852+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"236'24","reported_seq":371,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.087037+0000","last_change":"2024-02-05T08:14:17.606299+0000","last_active":"2024-02-05T08:15:32.087037+0000","last_peered":"2024-02-05T08:15:32.087037+0000","last_clean":"2024-02-05T08:15:32.087037+0000","last_became_active":"2024-02-05T08:12:41.718651+0000","last_became_peered":"2024-02-05T08:12:41.718651+0000","last_unstale":"2024-02-05T08:15:32.087037+0000","last_undegraded":"2024-02-05T08:15:32.087037+0000","last_fullsized":"2024-02-05T08:15:32.087037+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'16","last_scrub_stamp":"2024-02-05T08:14:17.606161+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:14:17.606161+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:34.534470+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":403,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.085824+0000","last_change":"2024-02-05T08:14:23.281718+0000","last_active":"2024-02-05T08:15:33.085824+0000","last_peered":"2024-02-05T08:15:33.085824+0000","last_clean":"2024-02-05T08:15:33.085824+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:15:33.085824+0000","last_undegraded":"2024-02-05T08:15:33.085824+0000","last_fullsized":"2024-02-05T08:15:33.085824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:23.281493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:23.281493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:49.551355+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":137,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087727+0000","last_change":"2024-02-05T08:14:09.229078+0000","last_active":"2024-02-05T08:15:33.087727+0000","last_peered":"2024-02-05T08:15:33.087727+0000","last_clean":"2024-02-05T08:15:33.087727+0000","last_became_active":"2024-02-05T08:12:41.757901+0000","last_became_peered":"2024-02-05T08:12:41.757901+0000","last_unstale":"2024-02-05T08:15:33.087727+0000","last_undegraded":"2024-02-05T08:15:33.087727+0000","last_fullsized":"2024-02-05T08:15:33.087727+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:09.228973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:09.228973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.461532+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":429,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.088313+0000","last_change":"2024-02-05T08:15:10.525698+0000","last_active":"2024-02-05T08:15:32.088313+0000","last_peered":"2024-02-05T08:15:32.088313+0000","last_clean":"2024-02-05T08:15:32.088313+0000","last_became_active":"2024-02-05T08:12:41.734072+0000","last_became_peered":"2024-02-05T08:12:41.734072+0000","last_unstale":"2024-02-05T08:15:32.088313+0000","last_undegraded":"2024-02-05T08:15:32.088313+0000","last_fullsized":"2024-02-05T08:15:32.088313+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:10.525625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:10.525625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:26.616052+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"236'15","reported_seq":78,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.084545+0000","last_change":"2024-02-05T08:14:24.999924+0000","last_active":"2024-02-05T08:15:32.084545+0000","last_peered":"2024-02-05T08:15:32.084545+0000","last_clean":"2024-02-05T08:15:32.084545+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:15:32.084545+0000","last_undegraded":"2024-02-05T08:15:32.084545+0000","last_fullsized":"2024-02-05T08:15:32.084545+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:14:21.660270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:21.660270+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.073882+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":128,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087233+0000","last_change":"2024-02-05T08:12:41.742137+0000","last_active":"2024-02-05T08:15:33.087233+0000","last_peered":"2024-02-05T08:15:33.087233+0000","last_clean":"2024-02-05T08:15:33.087233+0000","last_became_active":"2024-02-05T08:12:41.737246+0000","last_became_peered":"2024-02-05T08:12:41.737246+0000","last_unstale":"2024-02-05T08:15:33.087233+0000","last_undegraded":"2024-02-05T08:15:33.087233+0000","last_fullsized":"2024-02-05T08:15:33.087233+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.507800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.507800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.461309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":415,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.080281+0000","last_change":"2024-02-05T08:12:41.717743+0000","last_active":"2024-02-05T08:15:33.080281+0000","last_peered":"2024-02-05T08:15:33.080281+0000","last_clean":"2024-02-05T08:15:33.080281+0000","last_became_active":"2024-02-05T08:12:41.717285+0000","last_became_peered":"2024-02-05T08:12:41.717285+0000","last_unstale":"2024-02-05T08:15:33.080281+0000","last_undegraded":"2024-02-05T08:15:33.080281+0000","last_fullsized":"2024-02-05T08:15:33.080281+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:12.952558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:12.952558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":434,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078545+0000","last_change":"2024-02-05T08:14:30.022191+0000","last_active":"2024-02-05T08:15:32.078545+0000","last_peered":"2024-02-05T08:15:32.078545+0000","last_clean":"2024-02-05T08:15:32.078545+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:15:32.078545+0000","last_undegraded":"2024-02-05T08:15:32.078545+0000","last_fullsized":"2024-02-05T08:15:32.078545+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:14:30.022077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:32.501112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"249'24","reported_seq":490,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078143+0000","last_change":"2024-02-05T08:15:24.970945+0000","last_active":"2024-02-05T08:15:32.078143+0000","last_peered":"2024-02-05T08:15:32.078143+0000","last_clean":"2024-02-05T08:15:32.078143+0000","last_became_active":"2024-02-05T08:12:29.675657+0000","last_became_peered":"2024-02-05T08:12:29.675657+0000","last_unstale":"2024-02-05T08:15:32.078143+0000","last_undegraded":"2024-02-05T08:15:32.078143+0000","last_fullsized":"2024-02-05T08:15:32.078143+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:15:24.970769+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:33.443975+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":409,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.086483+0000","last_change":"2024-02-05T08:15:25.472795+0000","last_active":"2024-02-05T08:15:33.086483+0000","last_peered":"2024-02-05T08:15:33.086483+0000","last_clean":"2024-02-05T08:15:33.086483+0000","last_became_active":"2024-02-05T08:12:41.736845+0000","last_became_peered":"2024-02-05T08:12:41.736845+0000","last_unstale":"2024-02-05T08:15:33.086483+0000","last_undegraded":"2024-02-05T08:15:33.086483+0000","last_fullsized":"2024-02-05T08:15:33.086483+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.472655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.472655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:46.290853+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":553,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.085383+0000","last_change":"2024-02-05T08:15:08.943424+0000","last_active":"2024-02-05T08:15:32.085383+0000","last_peered":"2024-02-05T08:15:32.085383+0000","last_clean":"2024-02-05T08:15:32.085383+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:15:32.085383+0000","last_undegraded":"2024-02-05T08:15:32.085383+0000","last_fullsized":"2024-02-05T08:15:32.085383+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:15:08.943297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:08.943297+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:10.217549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":423,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086639+0000","last_change":"2024-02-05T08:14:10.570218+0000","last_active":"2024-02-05T08:15:32.086639+0000","last_peered":"2024-02-05T08:15:32.086639+0000","last_clean":"2024-02-05T08:15:32.086639+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:15:32.086639+0000","last_undegraded":"2024-02-05T08:15:32.086639+0000","last_fullsized":"2024-02-05T08:15:32.086639+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:10.570113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:10.570113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.476457+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":313,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.081936+0000","last_change":"2024-02-05T08:12:41.781948+0000","last_active":"2024-02-05T08:15:33.081936+0000","last_peered":"2024-02-05T08:15:33.081936+0000","last_clean":"2024-02-05T08:15:33.081936+0000","last_became_active":"2024-02-05T08:12:41.726861+0000","last_became_peered":"2024-02-05T08:12:41.726861+0000","last_unstale":"2024-02-05T08:15:33.081936+0000","last_undegraded":"2024-02-05T08:15:33.081936+0000","last_fullsized":"2024-02-05T08:15:33.081936+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'15","last_scrub_stamp":"2024-02-05T08:11:30.787149+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:11:30.787149+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":405,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078780+0000","last_change":"2024-02-05T08:15:07.991636+0000","last_active":"2024-02-05T08:15:32.078780+0000","last_peered":"2024-02-05T08:15:32.078780+0000","last_clean":"2024-02-05T08:15:32.078780+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:15:32.078780+0000","last_undegraded":"2024-02-05T08:15:32.078780+0000","last_fullsized":"2024-02-05T08:15:32.078780+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:07.991536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:07.991536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:09.960940+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"194'9","reported_seq":207,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086850+0000","last_change":"2024-02-05T08:15:27.464031+0000","last_active":"2024-02-05T08:15:32.086850+0000","last_peered":"2024-02-05T08:15:32.086850+0000","last_clean":"2024-02-05T08:15:32.086850+0000","last_became_active":"2024-02-05T08:12:41.728687+0000","last_became_peered":"2024-02-05T08:12:41.728687+0000","last_unstale":"2024-02-05T08:15:32.086850+0000","last_undegraded":"2024-02-05T08:15:32.086850+0000","last_fullsized":"2024-02-05T08:15:32.086850+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'9","last_scrub_stamp":"2024-02-05T08:15:27.463918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:27.463918+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:31.994300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":421,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.086317+0000","last_change":"2024-02-05T08:15:23.494414+0000","last_active":"2024-02-05T08:15:33.086317+0000","last_peered":"2024-02-05T08:15:33.086317+0000","last_clean":"2024-02-05T08:15:33.086317+0000","last_became_active":"2024-02-05T08:12:41.723383+0000","last_became_peered":"2024-02-05T08:12:41.723383+0000","last_unstale":"2024-02-05T08:15:33.086317+0000","last_undegraded":"2024-02-05T08:15:33.086317+0000","last_fullsized":"2024-02-05T08:15:33.086317+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:23.494299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:15:23.494299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:33.179614+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":459,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.085574+0000","last_change":"2024-02-05T08:14:35.244918+0000","last_active":"2024-02-05T08:15:32.085574+0000","last_peered":"2024-02-05T08:15:32.085574+0000","last_clean":"2024-02-05T08:15:32.085574+0000","last_became_active":"2024-02-05T08:12:41.723526+0000","last_became_peered":"2024-02-05T08:12:41.723526+0000","last_unstale":"2024-02-05T08:15:32.085574+0000","last_undegraded":"2024-02-05T08:15:32.085574+0000","last_fullsized":"2024-02-05T08:15:32.085574+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:14:35.244359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:35.244359+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:45.975379+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":418,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.080370+0000","last_change":"2024-02-05T08:15:17.707705+0000","last_active":"2024-02-05T08:15:32.080370+0000","last_peered":"2024-02-05T08:15:32.080370+0000","last_clean":"2024-02-05T08:15:32.080370+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:15:32.080370+0000","last_undegraded":"2024-02-05T08:15:32.080370+0000","last_fullsized":"2024-02-05T08:15:32.080370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:17.707565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:17.707565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:29.022820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":138,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.081241+0000","last_change":"2024-02-05T08:14:05.758633+0000","last_active":"2024-02-05T08:15:33.081241+0000","last_peered":"2024-02-05T08:15:33.081241+0000","last_clean":"2024-02-05T08:15:33.081241+0000","last_became_active":"2024-02-05T08:12:41.721852+0000","last_became_peered":"2024-02-05T08:12:41.721852+0000","last_unstale":"2024-02-05T08:15:33.081241+0000","last_undegraded":"2024-02-05T08:15:33.081241+0000","last_fullsized":"2024-02-05T08:15:33.081241+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:05.758519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:05.758519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":435,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086652+0000","last_change":"2024-02-05T08:15:25.470332+0000","last_active":"2024-02-05T08:15:32.086652+0000","last_peered":"2024-02-05T08:15:32.086652+0000","last_clean":"2024-02-05T08:15:32.086652+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:15:32.086652+0000","last_undegraded":"2024-02-05T08:15:32.086652+0000","last_fullsized":"2024-02-05T08:15:32.086652+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.470117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:40.542227+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"236'15","reported_seq":452,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087375+0000","last_change":"2024-02-05T08:14:29.189057+0000","last_active":"2024-02-05T08:15:33.087375+0000","last_peered":"2024-02-05T08:15:33.087375+0000","last_clean":"2024-02-05T08:15:33.087375+0000","last_became_active":"2024-02-05T08:12:41.736414+0000","last_became_peered":"2024-02-05T08:12:41.736414+0000","last_unstale":"2024-02-05T08:15:33.087375+0000","last_undegraded":"2024-02-05T08:15:33.087375+0000","last_fullsized":"2024-02-05T08:15:33.087375+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:14:29.188820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:29.188820+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:49.923284+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":423,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.088209+0000","last_change":"2024-02-05T08:13:34.175520+0000","last_active":"2024-02-05T08:15:32.088209+0000","last_peered":"2024-02-05T08:15:32.088209+0000","last_clean":"2024-02-05T08:15:32.088209+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:15:32.088209+0000","last_undegraded":"2024-02-05T08:15:32.088209+0000","last_fullsized":"2024-02-05T08:15:32.088209+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:34.174842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:13:34.174842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.476412+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":412,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.085151+0000","last_change":"2024-02-05T08:12:41.721999+0000","last_active":"2024-02-05T08:15:32.085151+0000","last_peered":"2024-02-05T08:15:32.085151+0000","last_clean":"2024-02-05T08:15:32.085151+0000","last_became_active":"2024-02-05T08:12:41.721486+0000","last_became_peered":"2024-02-05T08:12:41.721486+0000","last_unstale":"2024-02-05T08:15:32.085151+0000","last_undegraded":"2024-02-05T08:15:32.085151+0000","last_fullsized":"2024-02-05T08:15:32.085151+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.242229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.242229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.073697+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":144,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.082663+0000","last_change":"2024-02-05T08:14:17.208565+0000","last_active":"2024-02-05T08:15:32.082663+0000","last_peered":"2024-02-05T08:15:32.082663+0000","last_clean":"2024-02-05T08:15:32.082663+0000","last_became_active":"2024-02-05T08:12:29.684444+0000","last_became_peered":"2024-02-05T08:12:29.684444+0000","last_unstale":"2024-02-05T08:15:32.082663+0000","last_undegraded":"2024-02-05T08:15:32.082663+0000","last_fullsized":"2024-02-05T08:15:32.082663+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:17.208445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:17.208445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:35.997803+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'15","reported_seq":285,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086094+0000","last_change":"2024-02-05T08:15:23.404290+0000","last_active":"2024-02-05T08:15:32.086094+0000","last_peered":"2024-02-05T08:15:32.086094+0000","last_clean":"2024-02-05T08:15:32.086094+0000","last_became_active":"2024-02-05T08:12:41.732369+0000","last_became_peered":"2024-02-05T08:12:41.732369+0000","last_unstale":"2024-02-05T08:15:32.086094+0000","last_undegraded":"2024-02-05T08:15:32.086094+0000","last_fullsized":"2024-02-05T08:15:32.086094+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:15:23.404196+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:15:23.404196+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:25.374530+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"252'2592","reported_seq":2628,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.470903+0000","last_change":"2024-02-05T08:14:57.766906+0000","last_active":"2024-02-05T08:15:33.470903+0000","last_peered":"2024-02-05T08:15:33.470903+0000","last_clean":"2024-02-05T08:15:33.470903+0000","last_became_active":"2024-02-05T08:14:57.766609+0000","last_became_peered":"2024-02-05T08:14:57.766609+0000","last_unstale":"2024-02-05T08:15:33.470903+0000","last_undegraded":"2024-02-05T08:15:33.470903+0000","last_fullsized":"2024-02-05T08:15:33.470903+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2592,"log_dups_size":0,"ondisk_log_size":2592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:57.478302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169869312,"num_objects":2592,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2592,"num_write_kb":165888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":422,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.084848+0000","last_change":"2024-02-05T08:14:28.178983+0000","last_active":"2024-02-05T08:15:32.084848+0000","last_peered":"2024-02-05T08:15:32.084848+0000","last_clean":"2024-02-05T08:15:32.084848+0000","last_became_active":"2024-02-05T08:12:29.679849+0000","last_became_peered":"2024-02-05T08:12:29.679849+0000","last_unstale":"2024-02-05T08:15:32.084848+0000","last_undegraded":"2024-02-05T08:15:32.084848+0000","last_fullsized":"2024-02-05T08:15:32.084848+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:28.178770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:28.178770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:43.924283+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":435,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.085994+0000","last_change":"2024-02-05T08:14:22.627522+0000","last_active":"2024-02-05T08:15:32.085994+0000","last_peered":"2024-02-05T08:15:32.085994+0000","last_clean":"2024-02-05T08:15:32.085994+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:15:32.085994+0000","last_undegraded":"2024-02-05T08:15:32.085994+0000","last_fullsized":"2024-02-05T08:15:32.085994+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:14:22.627289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:22.627289+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.476482+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"236'27","reported_seq":497,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.084136+0000","last_change":"2024-02-05T08:12:29.719785+0000","last_active":"2024-02-05T08:15:32.084136+0000","last_peered":"2024-02-05T08:15:32.084136+0000","last_clean":"2024-02-05T08:15:32.084136+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:15:32.084136+0000","last_undegraded":"2024-02-05T08:15:32.084136+0000","last_fullsized":"2024-02-05T08:15:32.084136+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:12:26.319958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.319958+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.073801+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"190'10","reported_seq":574,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.087139+0000","last_change":"2024-02-05T08:14:06.549828+0000","last_active":"2024-02-05T08:15:32.087139+0000","last_peered":"2024-02-05T08:15:32.087139+0000","last_clean":"2024-02-05T08:15:32.087139+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:15:32.087139+0000","last_undegraded":"2024-02-05T08:15:32.087139+0000","last_fullsized":"2024-02-05T08:15:32.087139+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:14:06.549691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:06.549691+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.476518+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"252'2611","reported_seq":2647,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.255157+0000","last_change":"2024-02-05T08:14:57.758422+0000","last_active":"2024-02-05T08:15:33.255157+0000","last_peered":"2024-02-05T08:15:33.255157+0000","last_clean":"2024-02-05T08:15:33.255157+0000","last_became_active":"2024-02-05T08:14:57.758070+0000","last_became_peered":"2024-02-05T08:14:57.758070+0000","last_unstale":"2024-02-05T08:15:33.255157+0000","last_undegraded":"2024-02-05T08:15:33.255157+0000","last_fullsized":"2024-02-05T08:15:33.255157+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2611,"log_dups_size":0,"ondisk_log_size":2611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:22.195699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171114496,"num_objects":2611,"num_object_clones":0,"num_object_copies":7833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2611,"num_write_kb":167104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":135,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086712+0000","last_change":"2024-02-05T08:14:31.779172+0000","last_active":"2024-02-05T08:15:32.086712+0000","last_peered":"2024-02-05T08:15:32.086712+0000","last_clean":"2024-02-05T08:15:32.086712+0000","last_became_active":"2024-02-05T08:12:41.722853+0000","last_became_peered":"2024-02-05T08:12:41.722853+0000","last_unstale":"2024-02-05T08:15:32.086712+0000","last_undegraded":"2024-02-05T08:15:32.086712+0000","last_fullsized":"2024-02-05T08:15:32.086712+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:31.779046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.779046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:44.022928+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":423,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086212+0000","last_change":"2024-02-05T08:14:14.597997+0000","last_active":"2024-02-05T08:15:32.086212+0000","last_peered":"2024-02-05T08:15:32.086212+0000","last_clean":"2024-02-05T08:15:32.086212+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:15:32.086212+0000","last_undegraded":"2024-02-05T08:15:32.086212+0000","last_fullsized":"2024-02-05T08:15:32.086212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:14.597867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:14:14.597867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:35.633041+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":460,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078932+0000","last_change":"2024-02-05T08:14:35.038653+0000","last_active":"2024-02-05T08:15:32.078932+0000","last_peered":"2024-02-05T08:15:32.078932+0000","last_clean":"2024-02-05T08:15:32.078932+0000","last_became_active":"2024-02-05T08:12:41.728925+0000","last_became_peered":"2024-02-05T08:12:41.728925+0000","last_unstale":"2024-02-05T08:15:32.078932+0000","last_undegraded":"2024-02-05T08:15:32.078932+0000","last_fullsized":"2024-02-05T08:15:32.078932+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:14:35.038563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:35.038563+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.757754+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":422,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.077506+0000","last_change":"2024-02-05T08:15:19.765825+0000","last_active":"2024-02-05T08:15:32.077506+0000","last_peered":"2024-02-05T08:15:32.077506+0000","last_clean":"2024-02-05T08:15:32.077506+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:15:32.077506+0000","last_undegraded":"2024-02-05T08:15:32.077506+0000","last_fullsized":"2024-02-05T08:15:32.077506+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:15:19.765714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:25.591609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"251'2493","reported_seq":2527,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.818758+0000","last_change":"2024-02-05T08:14:57.763466+0000","last_active":"2024-02-05T08:15:32.818758+0000","last_peered":"2024-02-05T08:15:32.818758+0000","last_clean":"2024-02-05T08:15:32.818758+0000","last_became_active":"2024-02-05T08:14:57.763165+0000","last_became_peered":"2024-02-05T08:14:57.763165+0000","last_unstale":"2024-02-05T08:15:32.818758+0000","last_undegraded":"2024-02-05T08:15:32.818758+0000","last_fullsized":"2024-02-05T08:15:32.818758+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2493,"log_dups_size":0,"ondisk_log_size":2493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:25.940494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163381248,"num_objects":2493,"num_object_clones":0,"num_object_copies":7479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2493,"num_write_kb":159552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":417,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.085955+0000","last_change":"2024-02-05T08:15:13.442673+0000","last_active":"2024-02-05T08:15:32.085955+0000","last_peered":"2024-02-05T08:15:32.085955+0000","last_clean":"2024-02-05T08:15:32.085955+0000","last_became_active":"2024-02-05T08:12:41.724830+0000","last_became_peered":"2024-02-05T08:12:41.724830+0000","last_unstale":"2024-02-05T08:15:32.085955+0000","last_undegraded":"2024-02-05T08:15:32.085955+0000","last_fullsized":"2024-02-05T08:15:32.085955+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:13.442573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:13.442573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:28.200585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":414,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.079886+0000","last_change":"2024-02-05T08:14:33.899458+0000","last_active":"2024-02-05T08:15:32.079886+0000","last_peered":"2024-02-05T08:15:32.079886+0000","last_clean":"2024-02-05T08:15:32.079886+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:15:32.079886+0000","last_undegraded":"2024-02-05T08:15:32.079886+0000","last_fullsized":"2024-02-05T08:15:32.079886+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:33.896717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:14:33.896717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:34.161273+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":436,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.088523+0000","last_change":"2024-02-05T08:14:12.608866+0000","last_active":"2024-02-05T08:15:32.088523+0000","last_peered":"2024-02-05T08:15:32.088523+0000","last_clean":"2024-02-05T08:15:32.088523+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:15:32.088523+0000","last_undegraded":"2024-02-05T08:15:32.088523+0000","last_fullsized":"2024-02-05T08:15:32.088523+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:12.608763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:12.608763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.476430+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"236'35","reported_seq":188,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.081150+0000","last_change":"2024-02-05T08:14:31.191982+0000","last_active":"2024-02-05T08:15:32.081150+0000","last_peered":"2024-02-05T08:15:32.081150+0000","last_clean":"2024-02-05T08:15:32.081150+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:15:32.081150+0000","last_undegraded":"2024-02-05T08:15:32.081150+0000","last_fullsized":"2024-02-05T08:15:32.081150+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:14:31.191827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.191827+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.315305+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"251'2478","reported_seq":2512,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.802378+0000","last_change":"2024-02-05T08:14:57.761472+0000","last_active":"2024-02-05T08:15:32.802378+0000","last_peered":"2024-02-05T08:15:32.802378+0000","last_clean":"2024-02-05T08:15:32.802378+0000","last_became_active":"2024-02-05T08:14:57.761156+0000","last_became_peered":"2024-02-05T08:14:57.761156+0000","last_unstale":"2024-02-05T08:15:32.802378+0000","last_undegraded":"2024-02-05T08:15:32.802378+0000","last_fullsized":"2024-02-05T08:15:32.802378+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2478,"log_dups_size":0,"ondisk_log_size":2478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:13.756234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162398208,"num_objects":2478,"num_object_clones":0,"num_object_copies":7434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2478,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":403,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087417+0000","last_change":"2024-02-05T08:14:16.274043+0000","last_active":"2024-02-05T08:15:33.087417+0000","last_peered":"2024-02-05T08:15:33.087417+0000","last_clean":"2024-02-05T08:15:33.087417+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:15:33.087417+0000","last_undegraded":"2024-02-05T08:15:33.087417+0000","last_fullsized":"2024-02-05T08:15:33.087417+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:16.273936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:16.273936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.461610+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":428,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.080247+0000","last_change":"2024-02-05T08:15:25.793425+0000","last_active":"2024-02-05T08:15:32.080247+0000","last_peered":"2024-02-05T08:15:32.080247+0000","last_clean":"2024-02-05T08:15:32.080247+0000","last_became_active":"2024-02-05T08:12:41.743888+0000","last_became_peered":"2024-02-05T08:12:41.743888+0000","last_unstale":"2024-02-05T08:15:32.080247+0000","last_undegraded":"2024-02-05T08:15:32.080247+0000","last_fullsized":"2024-02-05T08:15:32.080247+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.793224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.793224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:34.555688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":432,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.084211+0000","last_change":"2024-02-05T08:12:41.719156+0000","last_active":"2024-02-05T08:15:32.084211+0000","last_peered":"2024-02-05T08:15:32.084211+0000","last_clean":"2024-02-05T08:15:32.084211+0000","last_became_active":"2024-02-05T08:12:41.718843+0000","last_became_peered":"2024-02-05T08:12:41.718843+0000","last_unstale":"2024-02-05T08:15:32.084211+0000","last_undegraded":"2024-02-05T08:15:32.084211+0000","last_fullsized":"2024-02-05T08:15:32.084211+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:18.291829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:18.291829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.073748+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"236'15","reported_seq":436,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087936+0000","last_change":"2024-02-05T08:12:22.599613+0000","last_active":"2024-02-05T08:15:33.087936+0000","last_peered":"2024-02-05T08:15:33.087936+0000","last_clean":"2024-02-05T08:15:33.087936+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:15:33.087936+0000","last_undegraded":"2024-02-05T08:15:33.087936+0000","last_fullsized":"2024-02-05T08:15:33.087936+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:12:22.599493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:22.599493+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.461454+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"252'2678","reported_seq":2714,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.506044+0000","last_change":"2024-02-05T08:14:57.766128+0000","last_active":"2024-02-05T08:15:33.506044+0000","last_peered":"2024-02-05T08:15:33.506044+0000","last_clean":"2024-02-05T08:15:33.506044+0000","last_became_active":"2024-02-05T08:14:57.765694+0000","last_became_peered":"2024-02-05T08:14:57.765694+0000","last_unstale":"2024-02-05T08:15:33.506044+0000","last_undegraded":"2024-02-05T08:15:33.506044+0000","last_fullsized":"2024-02-05T08:15:33.506044+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2678,"log_dups_size":0,"ondisk_log_size":2678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:25.701081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175505408,"num_objects":2678,"num_object_clones":0,"num_object_copies":8034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2678,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2678,"num_write_kb":171392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":135,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078010+0000","last_change":"2024-02-05T08:15:29.993055+0000","last_active":"2024-02-05T08:15:32.078010+0000","last_peered":"2024-02-05T08:15:32.078010+0000","last_clean":"2024-02-05T08:15:32.078010+0000","last_became_active":"2024-02-05T08:12:41.726776+0000","last_became_peered":"2024-02-05T08:12:41.726776+0000","last_unstale":"2024-02-05T08:15:32.078010+0000","last_undegraded":"2024-02-05T08:15:32.078010+0000","last_fullsized":"2024-02-05T08:15:32.078010+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:29.992957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:29.992957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:51.419597+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":455,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.084724+0000","last_change":"2024-02-05T08:15:18.892281+0000","last_active":"2024-02-05T08:15:32.084724+0000","last_peered":"2024-02-05T08:15:32.084724+0000","last_clean":"2024-02-05T08:15:32.084724+0000","last_became_active":"2024-02-05T08:12:29.679395+0000","last_became_peered":"2024-02-05T08:12:29.679395+0000","last_unstale":"2024-02-05T08:15:32.084724+0000","last_undegraded":"2024-02-05T08:15:32.084724+0000","last_fullsized":"2024-02-05T08:15:32.084724+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:15:18.892157+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:15:18.892157+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:21.437539+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":427,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.077833+0000","last_change":"2024-02-05T08:14:04.926625+0000","last_active":"2024-02-05T08:15:32.077833+0000","last_peered":"2024-02-05T08:15:32.077833+0000","last_clean":"2024-02-05T08:15:32.077833+0000","last_became_active":"2024-02-05T08:12:29.670859+0000","last_became_peered":"2024-02-05T08:12:29.670859+0000","last_unstale":"2024-02-05T08:15:32.077833+0000","last_undegraded":"2024-02-05T08:15:32.077833+0000","last_fullsized":"2024-02-05T08:15:32.077833+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:04.926518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:14:04.926518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:34.690352+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"236'51","reported_seq":649,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.081847+0000","last_change":"2024-02-05T08:12:41.775200+0000","last_active":"2024-02-05T08:15:33.081847+0000","last_peered":"2024-02-05T08:15:33.081847+0000","last_clean":"2024-02-05T08:15:33.081847+0000","last_became_active":"2024-02-05T08:12:41.719820+0000","last_became_peered":"2024-02-05T08:12:41.719820+0000","last_unstale":"2024-02-05T08:15:33.081847+0000","last_undegraded":"2024-02-05T08:15:33.081847+0000","last_fullsized":"2024-02-05T08:15:33.081847+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:11:34.819564+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:11:34.819564+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":400,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087011+0000","last_change":"2024-02-05T08:12:28.727498+0000","last_active":"2024-02-05T08:15:33.087011+0000","last_peered":"2024-02-05T08:15:33.087011+0000","last_clean":"2024-02-05T08:15:33.087011+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:15:33.087011+0000","last_undegraded":"2024-02-05T08:15:33.087011+0000","last_fullsized":"2024-02-05T08:15:33.087011+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:28.721402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:28.721402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.461507+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"251'2496","reported_seq":2530,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.815186+0000","last_change":"2024-02-05T08:14:57.763579+0000","last_active":"2024-02-05T08:15:32.815186+0000","last_peered":"2024-02-05T08:15:32.815186+0000","last_clean":"2024-02-05T08:15:32.815186+0000","last_became_active":"2024-02-05T08:14:57.763184+0000","last_became_peered":"2024-02-05T08:14:57.763184+0000","last_unstale":"2024-02-05T08:15:32.815186+0000","last_undegraded":"2024-02-05T08:15:32.815186+0000","last_fullsized":"2024-02-05T08:15:32.815186+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2496,"log_dups_size":0,"ondisk_log_size":2496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:00.113741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163577856,"num_objects":2496,"num_object_clones":0,"num_object_copies":7488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2496,"num_write_kb":159744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":430,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.088641+0000","last_change":"2024-02-05T08:15:20.467544+0000","last_active":"2024-02-05T08:15:32.088641+0000","last_peered":"2024-02-05T08:15:32.088641+0000","last_clean":"2024-02-05T08:15:32.088641+0000","last_became_active":"2024-02-05T08:12:41.732338+0000","last_became_peered":"2024-02-05T08:12:41.732338+0000","last_unstale":"2024-02-05T08:15:32.088641+0000","last_undegraded":"2024-02-05T08:15:32.088641+0000","last_fullsized":"2024-02-05T08:15:32.088641+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:20.467407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:20.467407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:45.580190+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":445,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.080328+0000","last_change":"2024-02-05T08:15:17.824514+0000","last_active":"2024-02-05T08:15:33.080328+0000","last_peered":"2024-02-05T08:15:33.080328+0000","last_clean":"2024-02-05T08:15:33.080328+0000","last_became_active":"2024-02-05T08:12:29.673554+0000","last_became_peered":"2024-02-05T08:12:29.673554+0000","last_unstale":"2024-02-05T08:15:33.080328+0000","last_undegraded":"2024-02-05T08:15:33.080328+0000","last_fullsized":"2024-02-05T08:15:33.080328+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:15:17.824391+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:15:17.824391+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:31.698204+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":446,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.079604+0000","last_change":"2024-02-05T08:15:18.722410+0000","last_active":"2024-02-05T08:15:32.079604+0000","last_peered":"2024-02-05T08:15:32.079604+0000","last_clean":"2024-02-05T08:15:32.079604+0000","last_became_active":"2024-02-05T08:12:41.746289+0000","last_became_peered":"2024-02-05T08:12:41.746289+0000","last_unstale":"2024-02-05T08:15:32.079604+0000","last_undegraded":"2024-02-05T08:15:32.079604+0000","last_fullsized":"2024-02-05T08:15:32.079604+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:18.722163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:15:18.722163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:42.755882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'17","reported_seq":428,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.080615+0000","last_change":"2024-02-05T08:14:21.682688+0000","last_active":"2024-02-05T08:15:32.080615+0000","last_peered":"2024-02-05T08:15:32.080615+0000","last_clean":"2024-02-05T08:15:32.080615+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:15:32.080615+0000","last_undegraded":"2024-02-05T08:15:32.080615+0000","last_fullsized":"2024-02-05T08:15:32.080615+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:14:21.682517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:21.682517+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:47.773747+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":423,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.080607+0000","last_change":"2024-02-05T08:15:20.774445+0000","last_active":"2024-02-05T08:15:32.080607+0000","last_peered":"2024-02-05T08:15:32.080607+0000","last_clean":"2024-02-05T08:15:32.080607+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:15:32.080607+0000","last_undegraded":"2024-02-05T08:15:32.080607+0000","last_fullsized":"2024-02-05T08:15:32.080607+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:20.774326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:20.774326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:27.506687+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"252'2480","reported_seq":2516,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.484931+0000","last_change":"2024-02-05T08:14:57.762460+0000","last_active":"2024-02-05T08:15:33.484931+0000","last_peered":"2024-02-05T08:15:33.484931+0000","last_clean":"2024-02-05T08:15:33.484931+0000","last_became_active":"2024-02-05T08:14:57.762164+0000","last_became_peered":"2024-02-05T08:14:57.762164+0000","last_unstale":"2024-02-05T08:15:33.484931+0000","last_undegraded":"2024-02-05T08:15:33.484931+0000","last_fullsized":"2024-02-05T08:15:33.484931+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2480,"log_dups_size":0,"ondisk_log_size":2480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:12.058644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162529280,"num_objects":2480,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2480,"num_write_kb":158720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":408,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086009+0000","last_change":"2024-02-05T08:12:41.717519+0000","last_active":"2024-02-05T08:15:32.086009+0000","last_peered":"2024-02-05T08:15:32.086009+0000","last_clean":"2024-02-05T08:15:32.086009+0000","last_became_active":"2024-02-05T08:12:41.717170+0000","last_became_peered":"2024-02-05T08:12:41.717170+0000","last_unstale":"2024-02-05T08:15:32.086009+0000","last_undegraded":"2024-02-05T08:15:32.086009+0000","last_fullsized":"2024-02-05T08:15:32.086009+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:23.303613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:23.303613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.073669+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":704,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086248+0000","last_change":"2024-02-05T08:13:31.220426+0000","last_active":"2024-02-05T08:15:32.086248+0000","last_peered":"2024-02-05T08:15:32.086248+0000","last_clean":"2024-02-05T08:15:32.086248+0000","last_became_active":"2024-02-05T08:12:32.677426+0000","last_became_peered":"2024-02-05T08:12:32.677426+0000","last_unstale":"2024-02-05T08:15:32.086248+0000","last_undegraded":"2024-02-05T08:15:32.086248+0000","last_fullsized":"2024-02-05T08:15:32.086248+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:13:31.220332+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:13:31.220332+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.476298+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":417,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.084661+0000","last_change":"2024-02-05T08:15:13.853710+0000","last_active":"2024-02-05T08:15:32.084661+0000","last_peered":"2024-02-05T08:15:32.084661+0000","last_clean":"2024-02-05T08:15:32.084661+0000","last_became_active":"2024-02-05T08:12:41.722331+0000","last_became_peered":"2024-02-05T08:12:41.722331+0000","last_unstale":"2024-02-05T08:15:32.084661+0000","last_undegraded":"2024-02-05T08:15:32.084661+0000","last_fullsized":"2024-02-05T08:15:32.084661+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:13.853624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:13.853624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:35.667987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"236'30","reported_seq":217,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.081471+0000","last_change":"2024-02-05T08:15:14.981439+0000","last_active":"2024-02-05T08:15:32.081471+0000","last_peered":"2024-02-05T08:15:32.081471+0000","last_clean":"2024-02-05T08:15:32.081471+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:15:32.081471+0000","last_undegraded":"2024-02-05T08:15:32.081471+0000","last_fullsized":"2024-02-05T08:15:32.081471+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:15:14.981271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:14.981271+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:36.831841+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":408,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.087073+0000","last_change":"2024-02-05T08:13:30.200414+0000","last_active":"2024-02-05T08:15:32.087073+0000","last_peered":"2024-02-05T08:15:32.087073+0000","last_clean":"2024-02-05T08:15:32.087073+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:15:32.087073+0000","last_undegraded":"2024-02-05T08:15:32.087073+0000","last_fullsized":"2024-02-05T08:15:32.087073+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:30.200341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:13:30.200341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.476223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"252'2712","reported_seq":2748,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.497515+0000","last_change":"2024-02-05T08:14:57.761686+0000","last_active":"2024-02-05T08:15:33.497515+0000","last_peered":"2024-02-05T08:15:33.497515+0000","last_clean":"2024-02-05T08:15:33.497515+0000","last_became_active":"2024-02-05T08:14:57.761382+0000","last_became_peered":"2024-02-05T08:14:57.761382+0000","last_unstale":"2024-02-05T08:15:33.497515+0000","last_undegraded":"2024-02-05T08:15:33.497515+0000","last_fullsized":"2024-02-05T08:15:33.497515+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2712,"log_dups_size":0,"ondisk_log_size":2712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:57.838632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177733632,"num_objects":2712,"num_object_clones":0,"num_object_copies":8136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2712,"num_write_kb":173568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":412,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.081520+0000","last_change":"2024-02-05T08:12:41.716722+0000","last_active":"2024-02-05T08:15:33.081520+0000","last_peered":"2024-02-05T08:15:33.081520+0000","last_clean":"2024-02-05T08:15:33.081520+0000","last_became_active":"2024-02-05T08:12:41.716394+0000","last_became_peered":"2024-02-05T08:12:41.716394+0000","last_unstale":"2024-02-05T08:15:33.081520+0000","last_undegraded":"2024-02-05T08:15:33.081520+0000","last_fullsized":"2024-02-05T08:15:33.081520+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:26.178219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.178219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":422,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087118+0000","last_change":"2024-02-05T08:15:15.359343+0000","last_active":"2024-02-05T08:15:33.087118+0000","last_peered":"2024-02-05T08:15:33.087118+0000","last_clean":"2024-02-05T08:15:33.087118+0000","last_became_active":"2024-02-05T08:12:41.741846+0000","last_became_peered":"2024-02-05T08:12:41.741846+0000","last_unstale":"2024-02-05T08:15:33.087118+0000","last_undegraded":"2024-02-05T08:15:33.087118+0000","last_fullsized":"2024-02-05T08:15:33.087118+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:15.359190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:15.359190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:41.785491+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":1081,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086052+0000","last_change":"2024-02-05T08:13:36.138202+0000","last_active":"2024-02-05T08:15:32.086052+0000","last_peered":"2024-02-05T08:15:32.086052+0000","last_clean":"2024-02-05T08:15:32.086052+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:15:32.086052+0000","last_undegraded":"2024-02-05T08:15:32.086052+0000","last_fullsized":"2024-02-05T08:15:32.086052+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:13:36.138048+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.476384+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":644,"num_read_kb":639,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":460,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087130+0000","last_change":"2024-02-05T08:14:25.014118+0000","last_active":"2024-02-05T08:15:33.087130+0000","last_peered":"2024-02-05T08:15:33.087130+0000","last_clean":"2024-02-05T08:15:33.087130+0000","last_became_active":"2024-02-05T08:14:24.992820+0000","last_became_peered":"2024-02-05T08:14:24.992820+0000","last_unstale":"2024-02-05T08:15:33.087130+0000","last_undegraded":"2024-02-05T08:15:33.087130+0000","last_fullsized":"2024-02-05T08:15:33.087130+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:12:16.517774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:16.517774+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.461399+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":407,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086429+0000","last_change":"2024-02-05T08:12:24.353439+0000","last_active":"2024-02-05T08:15:32.086429+0000","last_peered":"2024-02-05T08:15:32.086429+0000","last_clean":"2024-02-05T08:15:32.086429+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:15:32.086429+0000","last_undegraded":"2024-02-05T08:15:32.086429+0000","last_fullsized":"2024-02-05T08:15:32.086429+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:24.353354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:24.353354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.073773+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"251'2477","reported_seq":2511,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.706966+0000","last_change":"2024-02-05T08:14:57.764321+0000","last_active":"2024-02-05T08:15:32.706966+0000","last_peered":"2024-02-05T08:15:32.706966+0000","last_clean":"2024-02-05T08:15:32.706966+0000","last_became_active":"2024-02-05T08:14:57.764021+0000","last_became_peered":"2024-02-05T08:14:57.764021+0000","last_unstale":"2024-02-05T08:15:32.706966+0000","last_undegraded":"2024-02-05T08:15:32.706966+0000","last_fullsized":"2024-02-05T08:15:32.706966+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2477,"log_dups_size":0,"ondisk_log_size":2477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:57.697074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162332672,"num_objects":2477,"num_object_clones":0,"num_object_copies":7431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2477,"num_write_kb":158528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":418,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.080235+0000","last_change":"2024-02-05T08:12:29.673477+0000","last_active":"2024-02-05T08:15:33.080235+0000","last_peered":"2024-02-05T08:15:33.080235+0000","last_clean":"2024-02-05T08:15:33.080235+0000","last_became_active":"2024-02-05T08:12:29.672973+0000","last_became_peered":"2024-02-05T08:12:29.672973+0000","last_unstale":"2024-02-05T08:15:33.080235+0000","last_undegraded":"2024-02-05T08:15:33.080235+0000","last_fullsized":"2024-02-05T08:15:33.080235+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:25.219172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:25.219172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":426,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.081712+0000","last_change":"2024-02-05T08:14:19.636646+0000","last_active":"2024-02-05T08:15:33.081712+0000","last_peered":"2024-02-05T08:15:33.081712+0000","last_clean":"2024-02-05T08:15:33.081712+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:15:33.081712+0000","last_undegraded":"2024-02-05T08:15:33.081712+0000","last_fullsized":"2024-02-05T08:15:33.081712+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:19.636535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:19.636535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:43.736754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":463,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087803+0000","last_change":"2024-02-05T08:12:27.724075+0000","last_active":"2024-02-05T08:15:33.087803+0000","last_peered":"2024-02-05T08:15:33.087803+0000","last_clean":"2024-02-05T08:15:33.087803+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:15:33.087803+0000","last_undegraded":"2024-02-05T08:15:33.087803+0000","last_fullsized":"2024-02-05T08:15:33.087803+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:12:27.723924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:27.723924+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.461343+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":144,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.081057+0000","last_change":"2024-02-05T08:14:22.219846+0000","last_active":"2024-02-05T08:15:32.081057+0000","last_peered":"2024-02-05T08:15:32.081057+0000","last_clean":"2024-02-05T08:15:32.081057+0000","last_became_active":"2024-02-05T08:12:29.688534+0000","last_became_peered":"2024-02-05T08:12:29.688534+0000","last_unstale":"2024-02-05T08:15:32.081057+0000","last_undegraded":"2024-02-05T08:15:32.081057+0000","last_fullsized":"2024-02-05T08:15:32.081057+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:22.219740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:14:22.219740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:33.510414+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"251'2444","reported_seq":2478,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.803690+0000","last_change":"2024-02-05T08:14:57.764336+0000","last_active":"2024-02-05T08:15:32.803690+0000","last_peered":"2024-02-05T08:15:32.803690+0000","last_clean":"2024-02-05T08:15:32.803690+0000","last_became_active":"2024-02-05T08:14:57.764038+0000","last_became_peered":"2024-02-05T08:14:57.764038+0000","last_unstale":"2024-02-05T08:15:32.803690+0000","last_undegraded":"2024-02-05T08:15:32.803690+0000","last_fullsized":"2024-02-05T08:15:32.803690+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2444,"log_dups_size":0,"ondisk_log_size":2444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:06.265859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160169984,"num_objects":2444,"num_object_clones":0,"num_object_copies":7332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2444,"num_write_kb":156416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":410,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.085936+0000","last_change":"2024-02-05T08:12:41.736005+0000","last_active":"2024-02-05T08:15:33.085936+0000","last_peered":"2024-02-05T08:15:33.085936+0000","last_clean":"2024-02-05T08:15:33.085936+0000","last_became_active":"2024-02-05T08:12:41.735706+0000","last_became_peered":"2024-02-05T08:12:41.735706+0000","last_unstale":"2024-02-05T08:15:33.085936+0000","last_undegraded":"2024-02-05T08:15:33.085936+0000","last_fullsized":"2024-02-05T08:15:33.085936+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.607056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.607056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.461369+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":429,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.085460+0000","last_change":"2024-02-05T08:15:16.880868+0000","last_active":"2024-02-05T08:15:32.085460+0000","last_peered":"2024-02-05T08:15:32.085460+0000","last_clean":"2024-02-05T08:15:32.085460+0000","last_became_active":"2024-02-05T08:12:41.730010+0000","last_became_peered":"2024-02-05T08:12:41.730010+0000","last_unstale":"2024-02-05T08:15:32.085460+0000","last_undegraded":"2024-02-05T08:15:32.085460+0000","last_fullsized":"2024-02-05T08:15:32.085460+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:16.880446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:16.880446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:46.844168+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'20","reported_seq":444,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.080315+0000","last_change":"2024-02-05T08:12:19.056472+0000","last_active":"2024-02-05T08:15:33.080315+0000","last_peered":"2024-02-05T08:15:33.080315+0000","last_clean":"2024-02-05T08:15:33.080315+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:15:33.080315+0000","last_undegraded":"2024-02-05T08:15:33.080315+0000","last_fullsized":"2024-02-05T08:15:33.080315+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:12:19.056345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:19.056345+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"190'10","reported_seq":567,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087971+0000","last_change":"2024-02-05T08:14:30.163779+0000","last_active":"2024-02-05T08:15:33.087971+0000","last_peered":"2024-02-05T08:15:33.087971+0000","last_clean":"2024-02-05T08:15:33.087971+0000","last_became_active":"2024-02-05T08:12:29.683091+0000","last_became_peered":"2024-02-05T08:12:29.683091+0000","last_unstale":"2024-02-05T08:15:33.087971+0000","last_undegraded":"2024-02-05T08:15:33.087971+0000","last_fullsized":"2024-02-05T08:15:33.087971+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:14:30.163629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:30.163629+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:46.283057+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"251'2486","reported_seq":2520,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.655675+0000","last_change":"2024-02-05T08:14:57.761716+0000","last_active":"2024-02-05T08:15:32.655675+0000","last_peered":"2024-02-05T08:15:32.655675+0000","last_clean":"2024-02-05T08:15:32.655675+0000","last_became_active":"2024-02-05T08:14:57.761443+0000","last_became_peered":"2024-02-05T08:14:57.761443+0000","last_unstale":"2024-02-05T08:15:32.655675+0000","last_undegraded":"2024-02-05T08:15:32.655675+0000","last_fullsized":"2024-02-05T08:15:32.655675+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2486,"log_dups_size":0,"ondisk_log_size":2486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:23.258466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162922496,"num_objects":2486,"num_object_clones":0,"num_object_copies":7458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":159104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":424,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086877+0000","last_change":"2024-02-05T08:15:29.563981+0000","last_active":"2024-02-05T08:15:32.086877+0000","last_peered":"2024-02-05T08:15:32.086877+0000","last_clean":"2024-02-05T08:15:32.086877+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:15:32.086877+0000","last_undegraded":"2024-02-05T08:15:32.086877+0000","last_fullsized":"2024-02-05T08:15:32.086877+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:29.563876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:29.563876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:57.970791+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":427,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087227+0000","last_change":"2024-02-05T08:12:21.623410+0000","last_active":"2024-02-05T08:15:33.087227+0000","last_peered":"2024-02-05T08:15:33.087227+0000","last_clean":"2024-02-05T08:15:33.087227+0000","last_became_active":"2024-02-05T08:11:23.200753+0000","last_became_peered":"2024-02-05T08:11:23.200753+0000","last_unstale":"2024-02-05T08:15:33.087227+0000","last_undegraded":"2024-02-05T08:15:33.087227+0000","last_fullsized":"2024-02-05T08:15:33.087227+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:12:21.623291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:21.623291+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.461482+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"236'28","reported_seq":497,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078626+0000","last_change":"2024-02-05T08:15:24.002222+0000","last_active":"2024-02-05T08:15:32.078626+0000","last_peered":"2024-02-05T08:15:32.078626+0000","last_clean":"2024-02-05T08:15:32.078626+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:15:32.078626+0000","last_undegraded":"2024-02-05T08:15:32.078626+0000","last_fullsized":"2024-02-05T08:15:32.078626+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:15:24.002106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:24.002106+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:36.950373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"194'10","reported_seq":492,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.080141+0000","last_change":"2024-02-05T08:15:28.826050+0000","last_active":"2024-02-05T08:15:33.080141+0000","last_peered":"2024-02-05T08:15:33.080141+0000","last_clean":"2024-02-05T08:15:33.080141+0000","last_became_active":"2024-02-05T08:12:41.721686+0000","last_became_peered":"2024-02-05T08:12:41.721686+0000","last_unstale":"2024-02-05T08:15:33.080141+0000","last_undegraded":"2024-02-05T08:15:33.080141+0000","last_fullsized":"2024-02-05T08:15:33.080141+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'10","last_scrub_stamp":"2024-02-05T08:15:28.825911+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:15:28.825911+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:45.870601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"251'2495","reported_seq":2527,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.717571+0000","last_change":"2024-02-05T08:14:57.765949+0000","last_active":"2024-02-05T08:15:32.717571+0000","last_peered":"2024-02-05T08:15:32.717571+0000","last_clean":"2024-02-05T08:15:32.717571+0000","last_became_active":"2024-02-05T08:14:57.765609+0000","last_became_peered":"2024-02-05T08:14:57.765609+0000","last_unstale":"2024-02-05T08:15:32.717571+0000","last_undegraded":"2024-02-05T08:15:32.717571+0000","last_fullsized":"2024-02-05T08:15:32.717571+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2495,"log_dups_size":0,"ondisk_log_size":2495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:07.832990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163512320,"num_objects":2495,"num_object_clones":0,"num_object_copies":7485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":159680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":417,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.079129+0000","last_change":"2024-02-05T08:15:22.953610+0000","last_active":"2024-02-05T08:15:32.079129+0000","last_peered":"2024-02-05T08:15:32.079129+0000","last_clean":"2024-02-05T08:15:32.079129+0000","last_became_active":"2024-02-05T08:12:41.731253+0000","last_became_peered":"2024-02-05T08:12:41.731253+0000","last_unstale":"2024-02-05T08:15:32.079129+0000","last_undegraded":"2024-02-05T08:15:32.079129+0000","last_fullsized":"2024-02-05T08:15:32.079129+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.953481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.953481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:26.478801+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":147,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.085772+0000","last_change":"2024-02-05T08:15:15.390114+0000","last_active":"2024-02-05T08:15:32.085772+0000","last_peered":"2024-02-05T08:15:32.085772+0000","last_clean":"2024-02-05T08:15:32.085772+0000","last_became_active":"2024-02-05T08:12:41.723699+0000","last_became_peered":"2024-02-05T08:12:41.723699+0000","last_unstale":"2024-02-05T08:15:32.085772+0000","last_undegraded":"2024-02-05T08:15:32.085772+0000","last_fullsized":"2024-02-05T08:15:32.085772+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:15:15.389990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:15.389990+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:24.889149+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"236'38","reported_seq":477,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086471+0000","last_change":"2024-02-05T08:14:24.593255+0000","last_active":"2024-02-05T08:15:32.086471+0000","last_peered":"2024-02-05T08:15:32.086471+0000","last_clean":"2024-02-05T08:15:32.086471+0000","last_became_active":"2024-02-05T08:12:41.728044+0000","last_became_peered":"2024-02-05T08:12:41.728044+0000","last_unstale":"2024-02-05T08:15:32.086471+0000","last_undegraded":"2024-02-05T08:15:32.086471+0000","last_fullsized":"2024-02-05T08:15:32.086471+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'26","last_scrub_stamp":"2024-02-05T08:14:24.591818+0000","last_deep_scrub":"186'26","last_deep_scrub_stamp":"2024-02-05T08:12:24.579532+0000","last_clean_scrub_stamp":"2024-02-05T08:14:24.591818+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:36.476500+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":413,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078391+0000","last_change":"2024-02-05T08:14:26.979859+0000","last_active":"2024-02-05T08:15:32.078391+0000","last_peered":"2024-02-05T08:15:32.078391+0000","last_clean":"2024-02-05T08:15:32.078391+0000","last_became_active":"2024-02-05T08:12:41.720247+0000","last_became_peered":"2024-02-05T08:12:41.720247+0000","last_unstale":"2024-02-05T08:15:32.078391+0000","last_undegraded":"2024-02-05T08:15:32.078391+0000","last_fullsized":"2024-02-05T08:15:32.078391+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:26.979755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:26.979755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:42.561175+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"252'2573","reported_seq":2609,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.258974+0000","last_change":"2024-02-05T08:14:57.764494+0000","last_active":"2024-02-05T08:15:33.258974+0000","last_peered":"2024-02-05T08:15:33.258974+0000","last_clean":"2024-02-05T08:15:33.258974+0000","last_became_active":"2024-02-05T08:14:57.764174+0000","last_became_peered":"2024-02-05T08:14:57.764174+0000","last_unstale":"2024-02-05T08:15:33.258974+0000","last_undegraded":"2024-02-05T08:15:33.258974+0000","last_fullsized":"2024-02-05T08:15:33.258974+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2573,"log_dups_size":0,"ondisk_log_size":2573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:20.765265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168624128,"num_objects":2573,"num_object_clones":0,"num_object_copies":7719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2573,"num_write_kb":164672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":410,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.085222+0000","last_change":"2024-02-05T08:14:07.163475+0000","last_active":"2024-02-05T08:15:32.085222+0000","last_peered":"2024-02-05T08:15:32.085222+0000","last_clean":"2024-02-05T08:15:32.085222+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:15:32.085222+0000","last_undegraded":"2024-02-05T08:15:32.085222+0000","last_fullsized":"2024-02-05T08:15:32.085222+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:07.163372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:07.163372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:34.754956+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":545,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078287+0000","last_change":"2024-02-05T08:14:31.071675+0000","last_active":"2024-02-05T08:15:32.078287+0000","last_peered":"2024-02-05T08:15:32.078287+0000","last_clean":"2024-02-05T08:15:32.078287+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:15:32.078287+0000","last_undegraded":"2024-02-05T08:15:32.078287+0000","last_fullsized":"2024-02-05T08:15:32.078287+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:14:31.071536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.071536+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:40.656118+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":137,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.080454+0000","last_change":"2024-02-05T08:14:17.648700+0000","last_active":"2024-02-05T08:15:33.080454+0000","last_peered":"2024-02-05T08:15:33.080454+0000","last_clean":"2024-02-05T08:15:33.080454+0000","last_became_active":"2024-02-05T08:12:41.724969+0000","last_became_peered":"2024-02-05T08:12:41.724969+0000","last_unstale":"2024-02-05T08:15:33.080454+0000","last_undegraded":"2024-02-05T08:15:33.080454+0000","last_fullsized":"2024-02-05T08:15:33.080454+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:17.648596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:17.648596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"236'27","reported_seq":486,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.081185+0000","last_change":"2024-02-05T08:15:15.777707+0000","last_active":"2024-02-05T08:15:33.081185+0000","last_peered":"2024-02-05T08:15:33.081185+0000","last_clean":"2024-02-05T08:15:33.081185+0000","last_became_active":"2024-02-05T08:12:41.725007+0000","last_became_peered":"2024-02-05T08:12:41.725007+0000","last_unstale":"2024-02-05T08:15:33.081185+0000","last_undegraded":"2024-02-05T08:15:33.081185+0000","last_fullsized":"2024-02-05T08:15:33.081185+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:15:15.777496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:15.777496+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:43.525899+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"251'2503","reported_seq":2537,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.695715+0000","last_change":"2024-02-05T08:14:57.765880+0000","last_active":"2024-02-05T08:15:32.695715+0000","last_peered":"2024-02-05T08:15:32.695715+0000","last_clean":"2024-02-05T08:15:32.695715+0000","last_became_active":"2024-02-05T08:14:57.765255+0000","last_became_peered":"2024-02-05T08:14:57.765255+0000","last_unstale":"2024-02-05T08:15:32.695715+0000","last_undegraded":"2024-02-05T08:15:32.695715+0000","last_fullsized":"2024-02-05T08:15:32.695715+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2503,"log_dups_size":0,"ondisk_log_size":2503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:10.520554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164036608,"num_objects":2503,"num_object_clones":0,"num_object_copies":7509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2503,"num_write_kb":160192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":403,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.081241+0000","last_change":"2024-02-05T08:15:09.025688+0000","last_active":"2024-02-05T08:15:32.081241+0000","last_peered":"2024-02-05T08:15:32.081241+0000","last_clean":"2024-02-05T08:15:32.081241+0000","last_became_active":"2024-02-05T08:12:41.719835+0000","last_became_peered":"2024-02-05T08:12:41.719835+0000","last_unstale":"2024-02-05T08:15:32.081241+0000","last_undegraded":"2024-02-05T08:15:32.081241+0000","last_fullsized":"2024-02-05T08:15:32.081241+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:09.025580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:15:09.025580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:34.218391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":426,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.086306+0000","last_change":"2024-02-05T08:14:29.177432+0000","last_active":"2024-02-05T08:15:32.086306+0000","last_peered":"2024-02-05T08:15:32.086306+0000","last_clean":"2024-02-05T08:15:32.086306+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:15:32.086306+0000","last_undegraded":"2024-02-05T08:15:32.086306+0000","last_fullsized":"2024-02-05T08:15:32.086306+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:29.177314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:29.177314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:35.329432+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":422,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078427+0000","last_change":"2024-02-05T08:14:24.985198+0000","last_active":"2024-02-05T08:15:32.078427+0000","last_peered":"2024-02-05T08:15:32.078427+0000","last_clean":"2024-02-05T08:15:32.078427+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:15:32.078427+0000","last_undegraded":"2024-02-05T08:15:32.078427+0000","last_fullsized":"2024-02-05T08:15:32.078427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:24.985069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:14:24.985069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:50.692856+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"236'19","reported_seq":477,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078141+0000","last_change":"2024-02-05T08:14:09.985107+0000","last_active":"2024-02-05T08:15:32.078141+0000","last_peered":"2024-02-05T08:15:32.078141+0000","last_clean":"2024-02-05T08:15:32.078141+0000","last_became_active":"2024-02-05T08:12:29.666284+0000","last_became_peered":"2024-02-05T08:12:29.666284+0000","last_unstale":"2024-02-05T08:15:32.078141+0000","last_undegraded":"2024-02-05T08:15:32.078141+0000","last_fullsized":"2024-02-05T08:15:32.078141+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:14:09.984989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:09.984989+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:37.008972+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"252'2545","reported_seq":2581,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.241226+0000","last_change":"2024-02-05T08:14:57.765243+0000","last_active":"2024-02-05T08:15:33.241226+0000","last_peered":"2024-02-05T08:15:33.241226+0000","last_clean":"2024-02-05T08:15:33.241226+0000","last_became_active":"2024-02-05T08:14:57.764870+0000","last_became_peered":"2024-02-05T08:14:57.764870+0000","last_unstale":"2024-02-05T08:15:33.241226+0000","last_undegraded":"2024-02-05T08:15:33.241226+0000","last_fullsized":"2024-02-05T08:15:33.241226+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2545,"log_dups_size":0,"ondisk_log_size":2545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:09.829403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166789120,"num_objects":2545,"num_object_clones":0,"num_object_copies":7635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2545,"num_write_kb":162880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":429,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.080724+0000","last_change":"2024-02-05T08:15:21.884006+0000","last_active":"2024-02-05T08:15:32.080724+0000","last_peered":"2024-02-05T08:15:32.080724+0000","last_clean":"2024-02-05T08:15:32.080724+0000","last_became_active":"2024-02-05T08:12:41.737224+0000","last_became_peered":"2024-02-05T08:12:41.737224+0000","last_unstale":"2024-02-05T08:15:32.080724+0000","last_undegraded":"2024-02-05T08:15:32.080724+0000","last_fullsized":"2024-02-05T08:15:32.080724+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:15:21.883859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:21.883859+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:32.454548+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":409,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.082023+0000","last_change":"2024-02-05T08:15:14.746706+0000","last_active":"2024-02-05T08:15:33.082023+0000","last_peered":"2024-02-05T08:15:33.082023+0000","last_clean":"2024-02-05T08:15:33.082023+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:15:33.082023+0000","last_undegraded":"2024-02-05T08:15:33.082023+0000","last_fullsized":"2024-02-05T08:15:33.082023+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:14.746593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:14.746593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:42.161153+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":144,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.081240+0000","last_change":"2024-02-05T08:14:10.208350+0000","last_active":"2024-02-05T08:15:32.081240+0000","last_peered":"2024-02-05T08:15:32.081240+0000","last_clean":"2024-02-05T08:15:32.081240+0000","last_became_active":"2024-02-05T08:12:29.678501+0000","last_became_peered":"2024-02-05T08:12:29.678501+0000","last_unstale":"2024-02-05T08:15:32.081240+0000","last_undegraded":"2024-02-05T08:15:32.081240+0000","last_fullsized":"2024-02-05T08:15:32.081240+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:10.208236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:14:10.208236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:35.997831+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"236'31","reported_seq":459,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078532+0000","last_change":"2024-02-05T08:14:34.016671+0000","last_active":"2024-02-05T08:15:32.078532+0000","last_peered":"2024-02-05T08:15:32.078532+0000","last_clean":"2024-02-05T08:15:32.078532+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:15:32.078532+0000","last_undegraded":"2024-02-05T08:15:32.078532+0000","last_fullsized":"2024-02-05T08:15:32.078532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-02-05T08:14:34.016456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:34.016456+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.345431+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"251'2536","reported_seq":2569,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.564610+0000","last_change":"2024-02-05T08:14:57.762954+0000","last_active":"2024-02-05T08:15:32.564610+0000","last_peered":"2024-02-05T08:15:32.564610+0000","last_clean":"2024-02-05T08:15:32.564610+0000","last_became_active":"2024-02-05T08:14:57.762652+0000","last_became_peered":"2024-02-05T08:14:57.762652+0000","last_unstale":"2024-02-05T08:15:32.564610+0000","last_undegraded":"2024-02-05T08:15:32.564610+0000","last_fullsized":"2024-02-05T08:15:32.564610+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":2536,"log_dups_size":0,"ondisk_log_size":2536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:07.296366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166199296,"num_objects":2536,"num_object_clones":0,"num_object_copies":7608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2536,"num_write_kb":162304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":402,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078389+0000","last_change":"2024-02-05T08:14:21.925615+0000","last_active":"2024-02-05T08:15:32.078389+0000","last_peered":"2024-02-05T08:15:32.078389+0000","last_clean":"2024-02-05T08:15:32.078389+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:15:32.078389+0000","last_undegraded":"2024-02-05T08:15:32.078389+0000","last_fullsized":"2024-02-05T08:15:32.078389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:21.925502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:14:21.925502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:47.212458+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":562,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.079725+0000","last_change":"2024-02-05T08:15:10.756204+0000","last_active":"2024-02-05T08:15:32.079725+0000","last_peered":"2024-02-05T08:15:32.079725+0000","last_clean":"2024-02-05T08:15:32.079725+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:15:32.079725+0000","last_undegraded":"2024-02-05T08:15:32.079725+0000","last_fullsized":"2024-02-05T08:15:32.079725+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:15:10.756097+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:15:10.756097+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:20.981357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":136,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-02-05T08:15:33.087894+0000","last_change":"2024-02-05T08:15:29.437631+0000","last_active":"2024-02-05T08:15:33.087894+0000","last_peered":"2024-02-05T08:15:33.087894+0000","last_clean":"2024-02-05T08:15:33.087894+0000","last_became_active":"2024-02-05T08:12:41.757159+0000","last_became_peered":"2024-02-05T08:12:41.757159+0000","last_unstale":"2024-02-05T08:15:33.087894+0000","last_undegraded":"2024-02-05T08:15:33.087894+0000","last_fullsized":"2024-02-05T08:15:33.087894+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:29.437487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:29.437487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:40.329392+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"236'20","reported_seq":494,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-02-05T08:15:32.078678+0000","last_change":"2024-02-05T08:14:14.999089+0000","last_active":"2024-02-05T08:15:32.078678+0000","last_peered":"2024-02-05T08:15:32.078678+0000","last_clean":"2024-02-05T08:15:32.078678+0000","last_became_active":"2024-02-05T08:12:41.727271+0000","last_became_peered":"2024-02-05T08:12:41.727271+0000","last_unstale":"2024-02-05T08:15:32.078678+0000","last_undegraded":"2024-02-05T08:15:32.078678+0000","last_fullsized":"2024-02-05T08:15:32.078678+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:14:14.998858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:14.998858+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2660696064,"num_objects":40599,"num_object_clones":0,"num_object_copies":121797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40599,"num_write_kb":2598336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7959281664,"data_stored":7959281664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40599,"ondisk_log_size":40599,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":644,"num_read_kb":639,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1123,"num_read_kb":880,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":4032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":12894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":741,"ondisk_log_size":741,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037895,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2009372,"kb_used_data":1739408,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91735780,"statfs":{"total":95995035648,"available":93937438720,"internally_reserved":0,"allocated":1781153792,"data_stored":1780758245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":276429258},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":1.881,"15min":1.3009999999999999},"min":{"1min":0.254,"5min":0.22,"15min":0.217},"max":{"1min":3.972,"5min":52.488999999999997,"15min":60.127000000000002},"last":0.498},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.8660000000000001,"15min":1.2989999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":5.25,"5min":52.335000000000001,"15min":58.283000000000001},"last":0.80700000000000005}]},{"osd":1,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.8180000000000001,"15min":1.2350000000000001},"min":{"1min":0.27600000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.343,"5min":38.399999999999999,"15min":38.399999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.7609999999999999,"15min":1.113},"min":{"1min":0.28699999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.9449999999999998,"5min":38.195999999999998,"15min":38.195999999999998},"last":0.61799999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.1890000000000001,"15min":0.85099999999999998},"min":{"1min":0.36299999999999999,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":3.8889999999999998,"5min":6.2480000000000002,"15min":7.6779999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.119,"15min":0.91900000000000004},"min":{"1min":0.35799999999999998,"5min":0.253,"15min":0.24299999999999999},"max":{"1min":2.8010000000000002,"5min":6.399,"15min":8.9100000000000001},"last":0.96399999999999997}]},{"osd":3,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.98899999999999999,"15min":1.139},"min":{"1min":0.307,"5min":0.249,"15min":0.249},"max":{"1min":2.8279999999999998,"5min":4.6630000000000003,"15min":58.515000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.1200000000000001,"15min":1.123},"min":{"1min":0.30199999999999999,"5min":0.27400000000000002,"15min":0.27000000000000002},"max":{"1min":5.391,"5min":6.202,"15min":58.225999999999999},"last":0.92800000000000005}]},{"osd":4,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.292,"15min":0.70999999999999996},"min":{"1min":0.32800000000000001,"5min":0.23300000000000001,"15min":0.22},"max":{"1min":3.9550000000000001,"5min":49.601999999999997,"15min":49.601999999999997},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.97499999999999998,"5min":1.339,"15min":0.72999999999999998},"min":{"1min":0.32800000000000001,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":3.6349999999999998,"5min":49.795999999999999,"15min":49.795999999999999},"last":1.3149999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.2669999999999999,"15min":0.71499999999999997},"min":{"1min":0.307,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":5.298,"5min":49.914000000000001,"15min":49.914000000000001},"last":0.871},{"interface":"front","average":{"1min":0.96299999999999997,"5min":1.151,"15min":0.70399999999999996},"min":{"1min":0.372,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.7320000000000002,"5min":52.256,"15min":52.256},"last":0.748}]},{"osd":6,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":0.74099999999999999,"15min":0.57899999999999996},"min":{"1min":0.39100000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":8.0050000000000008,"5min":8.0050000000000008,"15min":8.0050000000000008},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.23,"5min":0.69199999999999995,"15min":0.54700000000000004},"min":{"1min":0.34200000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":7.2850000000000001,"5min":7.2850000000000001,"15min":7.2850000000000001},"last":1.282}]}]},{"osd":6,"up_from":51,"seq":219043332237,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":712804,"kb_used_data":474776,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93032348,"statfs":{"total":95995035648,"available":95265124352,"internally_reserved":0,"allocated":486170624,"data_stored":485757972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":243726790},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":2.488,"15min":1.3879999999999999},"min":{"1min":0.28499999999999998,"5min":0.22900000000000001,"15min":0.223},"max":{"1min":3.8849999999999998,"5min":102.819,"15min":102.819},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.85499999999999998,"5min":2.4860000000000002,"15min":1.4610000000000001},"min":{"1min":0.28899999999999998,"5min":0.214,"15min":0.214},"max":{"1min":4.5389999999999997,"5min":102.75,"15min":102.75},"last":0.38900000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.5660000000000001,"15min":1.0780000000000001},"min":{"1min":0.28199999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.98,"5min":23.773,"15min":23.773},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.4039999999999999,"15min":1.004},"min":{"1min":0.36699999999999999,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":1.843,"5min":23.719000000000001,"15min":23.719000000000001},"last":0.64900000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":1.234,"15min":0.96799999999999997},"min":{"1min":0.28699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.931,"5min":22.503,"15min":22.503},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.202,"15min":0.90100000000000002},"min":{"1min":0.36399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.4510000000000001,"5min":22.466000000000001,"15min":22.466000000000001},"last":0.47699999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":1.8939999999999999,"15min":1.26},"min":{"1min":0.36499999999999999,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":3.3100000000000001,"5min":102.438,"15min":102.438},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.67900000000000005,"5min":1.8220000000000001,"15min":1.242},"min":{"1min":0.38700000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.621,"5min":102.288,"15min":102.288},"last":0.61499999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.68400000000000005,"15min":0.51400000000000001},"min":{"1min":0.309,"5min":0.26100000000000001,"15min":0.223},"max":{"1min":3.3679999999999999,"5min":9.8970000000000002,"15min":9.8970000000000002},"last":1.7350000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.79400000000000004,"15min":0.56000000000000005},"min":{"1min":0.311,"5min":0.30199999999999999,"15min":0.26600000000000001},"max":{"1min":2.9300000000000002,"5min":4.9589999999999996,"15min":4.9589999999999996},"last":1.2450000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.2430000000000001,"15min":0.72699999999999998},"min":{"1min":0.32400000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.3250000000000002,"5min":43.643000000000001,"15min":43.643000000000001},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.048,"5min":1.2629999999999999,"15min":0.745},"min":{"1min":0.40799999999999997,"5min":0.307,"15min":0.30399999999999999},"max":{"1min":3.7549999999999999,"5min":43.893000000000001,"15min":43.893000000000001},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:14:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.70599999999999996,"15min":0.57399999999999995},"min":{"1min":0.372,"5min":0.32000000000000001,"15min":0.29999999999999999},"max":{"1min":3.3980000000000001,"5min":5.0380000000000003,"15min":5.0380000000000003},"last":1.7669999999999999},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.69199999999999995,"15min":0.58199999999999996},"min":{"1min":0.38900000000000001,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":3.0430000000000001,"5min":4.851,"15min":4.851},"last":0.76800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575014,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1827696,"kb_used_data":1445284,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91917456,"statfs":{"total":95995035648,"available":94123474944,"internally_reserved":0,"allocated":1479970816,"data_stored":1479553953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.46,"15min":1.3779999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.14699999999999999},"max":{"1min":3.1040000000000001,"5min":21.120999999999999,"15min":86.105000000000004},"last":0.23499999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.8300000000000001,"15min":1.623},"min":{"1min":0.23400000000000001,"5min":0.20699999999999999,"15min":0.16},"max":{"1min":6.1710000000000003,"5min":21.172999999999998,"15min":85.665000000000006},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon Feb 5 08:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.4550000000000001,"15min":1.268},"min":{"1min":0.221,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":3.544,"5min":19.016999999999999,"15min":49.030999999999999},"last":0.221},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.901,"15min":1.4790000000000001},"min":{"1min":0.21099999999999999,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":5.9800000000000004,"5min":20.434000000000001,"15min":48.984000000000002},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.623,"15min":1.4330000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20599999999999999},"max":{"1min":3.141,"5min":31.896000000000001,"15min":49.067999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.3440000000000001,"15min":1.373},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.24199999999999999},"max":{"1min":3.286,"5min":32.018999999999998,"15min":48.734999999999999},"last":0.27100000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:15:31 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":1.3779999999999999,"15min":1.0189999999999999},"min":{"1min":0.42799999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.5510000000000002,"5min":25.952000000000002,"15min":25.952000000000002},"last":1.5209999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":1.625,"15min":1.127},"min":{"1min":0.45900000000000002,"5min":0.30199999999999999,"15min":0.27100000000000002},"max":{"1min":1.5049999999999999,"5min":25.922000000000001,"15min":25.922000000000001},"last":1.5049999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:14:51 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.6890000000000001,"15min":1.1319999999999999},"min":{"1min":0.41199999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.4660000000000002,"5min":54.552999999999997,"15min":54.552999999999997},"last":1.024},{"interface":"front","average":{"1min":1.3320000000000001,"5min":2.0139999999999998,"15min":1.2430000000000001},"min":{"1min":0.40000000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":6.0259999999999998,"5min":55.811,"15min":55.811},"last":0.50600000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":1.5620000000000001,"15min":1.1180000000000001},"min":{"1min":0.42899999999999999,"5min":0.38,"15min":0.33100000000000002},"max":{"1min":1.256,"5min":54.384,"15min":54.384},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.5469999999999999,"15min":1.125},"min":{"1min":0.47499999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.339,"5min":55.210999999999999,"15min":55.210999999999999},"last":0.42499999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.3799999999999999,"15min":1.0940000000000001},"min":{"1min":0.42899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.0579999999999998,"5min":7.9390000000000001,"15min":13.749000000000001},"last":1.119},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.3660000000000001,"15min":1.0569999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":4.6180000000000003,"5min":7.9640000000000004,"15min":7.9640000000000004},"last":0.64500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738539,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1472556,"kb_used_data":1094304,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92272596,"statfs":{"total":95995035648,"available":94487138304,"internally_reserved":0,"allocated":1120567296,"data_stored":1120163351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":2.0449999999999999,"15min":1.6830000000000001},"min":{"1min":0.188,"5min":0.186,"15min":0.152},"max":{"1min":2.5619999999999998,"5min":32.066000000000003,"15min":66.772999999999996},"last":7.9580000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":1.8360000000000001,"15min":1.427},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17199999999999999},"max":{"1min":2.4209999999999998,"5min":15.327999999999999,"15min":46.198999999999998},"last":6.569}]},{"osd":2,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.901,"15min":1.4099999999999999},"min":{"1min":0.30399999999999999,"5min":0.216,"15min":0.216},"max":{"1min":8.75,"5min":12.074,"15min":46.170999999999999},"last":6.7050000000000001},{"interface":"front","average":{"1min":0.998,"5min":2.1949999999999998,"15min":1.5620000000000001},"min":{"1min":0.219,"5min":0.20399999999999999,"15min":0.19400000000000001},"max":{"1min":3.04,"5min":15.569000000000001,"15min":46.543999999999997},"last":3.1360000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.9750000000000001,"15min":1.6180000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.21099999999999999},"max":{"1min":6.5940000000000003,"5min":13.013,"15min":67.888000000000005},"last":6.7439999999999998},{"interface":"front","average":{"1min":0.57999999999999996,"5min":2.5649999999999999,"15min":1.9179999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.22600000000000001},"max":{"1min":1.6180000000000001,"5min":37.262999999999998,"15min":66.891999999999996},"last":3.1709999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:14:38 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.6870000000000001,"15min":1.073},"min":{"1min":0.27700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":3.3119999999999998,"5min":15.159000000000001,"15min":15.159000000000001},"last":6.407},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.462,"15min":1.0309999999999999},"min":{"1min":0.35699999999999998,"5min":0.28999999999999998,"15min":0.26900000000000002},"max":{"1min":6.3179999999999996,"5min":10.928000000000001,"15min":12.989000000000001},"last":6.915}]},{"osd":5,"last update":"Mon Feb 5 08:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.6120000000000001,"15min":1.1200000000000001},"min":{"1min":0.39100000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.1589999999999998,"5min":15.689,"15min":15.689},"last":4.101},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.3720000000000001,"15min":1.0569999999999999},"min":{"1min":0.438,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":12.94},"last":6.944}]},{"osd":6,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.2529999999999999,"15min":1.0489999999999999},"min":{"1min":0.49399999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.8220000000000001,"5min":7.8220000000000001,"15min":12.967000000000001},"last":6.9749999999999996},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.4350000000000001,"15min":1.133},"min":{"1min":0.47199999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.6240000000000001,"5min":15.721,"15min":15.721},"last":3.23}]},{"osd":7,"last update":"Mon Feb 5 08:14:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.637,"15min":1.083},"min":{"1min":0.34399999999999997,"5min":0.33600000000000002,"15min":0.32900000000000001},"max":{"1min":4.2199999999999998,"5min":15.211,"15min":15.211},"last":6.4480000000000004},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.508,"15min":1.1200000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":7.8650000000000002,"5min":7.8650000000000002,"15min":13.012},"last":6.8789999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411489,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1177188,"kb_used_data":794776,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92567964,"statfs":{"total":95995035648,"available":94789595136,"internally_reserved":0,"allocated":813850624,"data_stored":813445757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":391576006},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":2.5800000000000001,"15min":1.542},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.19700000000000001},"max":{"1min":3.8540000000000001,"5min":40.354999999999997,"15min":40.354999999999997},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.581,"5min":2.7930000000000001,"15min":1.6699999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":5.5650000000000004,"5min":40.566000000000003,"15min":40.566000000000003},"last":1.978}]},{"osd":1,"last update":"Mon Feb 5 08:14:48 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":2.2610000000000001,"15min":1.3480000000000001},"min":{"1min":0.25900000000000001,"5min":0.19400000000000001,"15min":0.17999999999999999},"max":{"1min":2.843,"5min":55.942,"15min":55.942},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":2.4079999999999999,"15min":1.4079999999999999},"min":{"1min":0.29299999999999998,"5min":0.22900000000000001,"15min":0.193},"max":{"1min":4.7530000000000001,"5min":55.960999999999999,"15min":55.960999999999999},"last":1.6870000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.6200000000000001,"15min":1.6919999999999999},"min":{"1min":0.26100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.605,"5min":8.6590000000000007,"15min":68.108000000000004},"last":1.496},{"interface":"front","average":{"1min":0.65300000000000002,"5min":1.5,"15min":1.54},"min":{"1min":0.29199999999999998,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":1.8029999999999999,"5min":7.8879999999999999,"15min":67.945999999999998},"last":0.83999999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:14:29 2024","interfaces":[{"interface":"back","average":{"1min":1.9039999999999999,"5min":1.9339999999999999,"15min":1.236},"min":{"1min":0.34100000000000003,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":4.9370000000000003,"5min":31.596,"15min":31.596},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.9950000000000001,"5min":1.7769999999999999,"15min":1.1000000000000001},"min":{"1min":0.40100000000000002,"5min":0.27200000000000002,"15min":0.252},"max":{"1min":4.5540000000000003,"5min":31.57,"15min":31.57},"last":0.748}]},{"osd":5,"last update":"Mon Feb 5 08:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.9450000000000001,"15min":1.2330000000000001},"min":{"1min":0.35099999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":3.7970000000000002,"5min":58.799999999999997,"15min":58.799999999999997},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.196,"5min":2.0419999999999998,"15min":1.2170000000000001},"min":{"1min":0.30399999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.593,"5min":58.615000000000002,"15min":58.615000000000002},"last":2.5499999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:14:29 2024","interfaces":[{"interface":"back","average":{"1min":1.607,"5min":1.4390000000000001,"15min":1.1080000000000001},"min":{"1min":0.441,"5min":0.32500000000000001,"15min":0.308},"max":{"1min":4.7770000000000001,"5min":7.5099999999999998,"15min":15.859},"last":1.653},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.3149999999999999,"15min":0.98499999999999999},"min":{"1min":0.35899999999999999,"5min":0.34100000000000003,"15min":0.32900000000000001},"max":{"1min":3.7610000000000001,"5min":6.1820000000000004,"15min":6.1820000000000004},"last":0.78500000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:14:48 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.3959999999999999,"15min":1.091},"min":{"1min":0.33100000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.516,"5min":7.3600000000000003,"15min":15.986000000000001},"last":2.5089999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.5429999999999999,"15min":1.171},"min":{"1min":0.41699999999999998,"5min":0.38,"15min":0.35299999999999998},"max":{"1min":4.6749999999999998,"5min":7.4779999999999998,"15min":15.951000000000001},"last":1.6279999999999999}]}]},{"osd":3,"up_from":0,"seq":115964117148,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953623,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":960328,"kb_used_data":622336,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92784824,"statfs":{"total":95995035648,"available":95011659776,"internally_reserved":0,"allocated":637272064,"data_stored":636882651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":1.6819999999999999,"15min":1.4890000000000001},"min":{"1min":0.26800000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.4769999999999999,"5min":39.552,"15min":73.078999999999994},"last":1.3919999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.6990000000000001,"15min":1.444},"min":{"1min":0.29999999999999999,"5min":0.214,"15min":0.21099999999999999},"max":{"1min":3.0489999999999999,"5min":39.350999999999999,"15min":73.412000000000006},"last":0.40100000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":2.536,"15min":1.351},"min":{"1min":0.33700000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.6419999999999999,"5min":125.054,"15min":125.054},"last":1.478},{"interface":"front","average":{"1min":0.872,"5min":2.5910000000000002,"15min":1.2929999999999999},"min":{"1min":0.308,"5min":0.249,"15min":0.23499999999999999},"max":{"1min":5.1260000000000003,"5min":125.08799999999999,"15min":125.08799999999999},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.995,"15min":1.1379999999999999},"min":{"1min":0.28199999999999997,"5min":0.247,"15min":0.24199999999999999},"max":{"1min":1.9750000000000001,"5min":5.649,"15min":96.849999999999994},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.0700000000000001,"15min":1.1970000000000001},"min":{"1min":0.41799999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.79,"5min":4.1299999999999999,"15min":96.143000000000001},"last":0.65300000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.9950000000000001,"15min":1.698},"min":{"1min":0.376,"5min":0.28599999999999998,"15min":0.24399999999999999},"max":{"1min":2.1160000000000001,"5min":57.204000000000001,"15min":73.602999999999994},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":2.0510000000000002,"15min":1.7210000000000001},"min":{"1min":0.36899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76400000000000001,"5min":57.606999999999999,"15min":74.817999999999998},"last":1.4099999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:15:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7690000000000001,"5min":1.0780000000000001,"15min":0.61699999999999999},"min":{"1min":0.39900000000000002,"5min":0.224,"15min":0.224},"max":{"1min":34.167000000000002,"5min":34.167000000000002,"15min":34.167000000000002},"last":1.371},{"interface":"front","average":{"1min":3.1789999999999998,"5min":1.252,"15min":0.69599999999999995},"min":{"1min":0.29599999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":36.051000000000002,"5min":36.051000000000002,"15min":36.051000000000002},"last":0.96799999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.72699999999999998,"15min":0.53700000000000003},"min":{"1min":0.37,"5min":0.29599999999999999,"15min":0.29299999999999998},"max":{"1min":2.0800000000000001,"5min":3.9220000000000002,"15min":3.9220000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64500000000000002,"15min":0.51000000000000001},"min":{"1min":0.47099999999999997,"5min":0.32000000000000001,"15min":0.29499999999999998},"max":{"1min":0.83799999999999997,"5min":3.306,"15min":3.306},"last":1.2130000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:14:38 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":0.75700000000000001,"15min":0.56699999999999995},"min":{"1min":0.314,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.4859999999999998,"5min":4.4859999999999998,"15min":4.4859999999999998},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.70499999999999996,"15min":0.56999999999999995},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.7450000000000001,"5min":5.7450000000000001,"15min":5.7450000000000001},"last":1.5289999999999999}]}]},{"osd":5,"up_from":42,"seq":180388626577,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1893760,"kb_used_data":1551608,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":91851392,"statfs":{"total":95995035648,"available":94055825408,"internally_reserved":0,"allocated":1588846592,"data_stored":1588466328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:14:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.786,"15min":1.5309999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.19400000000000001},"max":{"1min":3.794,"5min":46.962000000000003,"15min":143.06299999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.73,"15min":1.516},"min":{"1min":0.27200000000000002,"5min":0.23699999999999999,"15min":0.20499999999999999},"max":{"1min":4.7560000000000002,"5min":47.067,"15min":143.09700000000001},"last":0.56999999999999995}]},{"osd":1,"last update":"Mon Feb 5 08:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":1.5469999999999999,"15min":0.93899999999999995},"min":{"1min":0.27600000000000002,"5min":0.25,"15min":0.215},"max":{"1min":4.2519999999999998,"5min":46.200000000000003,"15min":46.200000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.159,"5min":1.8720000000000001,"15min":1.0569999999999999},"min":{"1min":0.27400000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.8760000000000003,"5min":46.343000000000004,"15min":46.343000000000004},"last":0.97799999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:14:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.262,"15min":0.91300000000000003},"min":{"1min":0.379,"5min":0.246,"15min":0.246},"max":{"1min":4.2439999999999998,"5min":10.733000000000001,"15min":10.733000000000001},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.222,"5min":1.24,"15min":0.93999999999999995},"min":{"1min":0.29099999999999998,"5min":0.28399999999999997,"15min":0.26400000000000001},"max":{"1min":4.3970000000000002,"5min":8.0220000000000002,"15min":8.0220000000000002},"last":0.27600000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:14:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.635,"15min":1.5249999999999999},"min":{"1min":0.371,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.6560000000000001,"5min":46.569000000000003,"15min":143.03399999999999},"last":0.315},{"interface":"front","average":{"1min":1.252,"5min":2.052,"15min":1.6160000000000001},"min":{"1min":0.34100000000000003,"5min":0.32600000000000001,"15min":0.28100000000000003},"max":{"1min":4.3070000000000004,"5min":47.820999999999998,"15min":143.46799999999999},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Feb 5 08:14:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":0.79900000000000004,"15min":0.54300000000000004},"min":{"1min":0.24399999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.8300000000000001,"5min":4.8689999999999998,"15min":4.8689999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.024,"5min":0.65700000000000003,"15min":0.50800000000000001},"min":{"1min":0.34399999999999997,"5min":0.28299999999999997,"15min":0.26500000000000001},"max":{"1min":3.7149999999999999,"5min":3.7149999999999999,"15min":3.7149999999999999},"last":0.23999999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:15:17 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.73799999999999999,"15min":0.54400000000000004},"min":{"1min":0.314,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":3.5859999999999999,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":0.38},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.76200000000000001,"15min":0.57999999999999996},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":3.6859999999999999,"5min":6.6180000000000003,"15min":6.6180000000000003},"last":0.40400000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:14:50 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":0.76100000000000001,"15min":0.57099999999999995},"min":{"1min":0.38300000000000001,"5min":0.34399999999999997,"15min":0.25900000000000001},"max":{"1min":3.8290000000000002,"5min":4.3300000000000001,"15min":4.3300000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.79500000000000004,"15min":0.59399999999999997},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.28199999999999997},"max":{"1min":5.2240000000000002,"5min":5.2240000000000002,"15min":5.2240000000000002},"last":0.53600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1028587520,"data_stored":1028587520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1505558528,"data_stored":1505558528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":823132160,"data_stored":823132160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":658440192,"data_stored":658440192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1644560384,"data_stored":1644560384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":497221632,"data_stored":497221632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1801781248,"data_stored":1801781248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:15:37.879 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-05T08:15:37.879 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:15:37.879 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-05T08:15:37.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T08:15:37.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:37.982 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:37.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:38.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:15:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:15:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:15:38.075 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:15:38.083 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:38.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:38.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:38.133 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:15:38.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:38.271 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:38.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:38.331 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:38 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3234763719' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:15:38.331 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:38.331 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:15:38.331 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:38 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3234763719' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:15:38.332 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:38.332 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:15:38.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:38.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:38 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3234763719' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:15:38.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:38.399 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:15:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:38.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:38.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:38.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 24 16 49337 49321 128.416 97.3125 0.00622586 0.00778123 2024-02-05T08:15:38.518 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:38.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:38.554 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:38.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:38.655 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:38.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:38.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:38.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:38.796 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:38.833 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:38.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:38.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:38.935 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:38.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:38.964 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:38.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:39.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:39.100 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:39.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:39.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:39.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:39 smithi134 ceph-mon[30086]: pgmap v482: 148 pgs: 148 active+clean; 2.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 682 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-05T08:15:39.179 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:39 smithi134 ceph-mon[39278]: pgmap v482: 148 pgs: 148 active+clean; 2.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 682 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-05T08:15:39.202 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:39.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:39.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:39.341 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:39.374 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:39.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:39.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:39.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:39.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 25 16 50970 50954 127.362 102.062 0.00575655 0.00784704 2024-02-05T08:15:39.511 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:39.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:39.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:39 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:15:39.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:39 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3234763719' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:15:39.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:39 smithi134 ceph-mon[30086]: osdmap e254: 8 total, 8 up, 7 in 2024-02-05T08:15:39.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:39 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:15:39.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:39 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3234763719' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:15:39.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:39 smithi134 ceph-mon[39278]: osdmap e254: 8 total, 8 up, 7 in 2024-02-05T08:15:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:39 smithi196 ceph-mon[33018]: pgmap v482: 148 pgs: 148 active+clean; 2.5 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 682 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-05T08:15:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:39 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:15:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:39 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3234763719' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:15:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:39 smithi196 ceph-mon[33018]: osdmap e254: 8 total, 8 up, 7 in 2024-02-05T08:15:39.613 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:39.659 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:39.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:39.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:39.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:39.796 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:39.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:39.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:39.898 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:39.933 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:39.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:40.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:40.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:40.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:40.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:40.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:40.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:40.213 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:40.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:40.224 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:40.313 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:40.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:40.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:40.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:40.368 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:15:40.368 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:15:40.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:40.481 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 26 16 52524 52508 126.199 97.125 0.00862026 0.00791947 2024-02-05T08:15:40.502 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:40.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:40.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:40.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:40.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:40.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:40.742 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:40 smithi134 ceph-mon[30086]: pgmap v484: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 771 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-02-05T08:15:40.742 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:40 smithi134 ceph-mon[30086]: osdmap e255: 8 total, 8 up, 7 in 2024-02-05T08:15:40.742 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:40 smithi134 ceph-mon[39278]: pgmap v484: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 771 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-02-05T08:15:40.742 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:40 smithi134 ceph-mon[39278]: osdmap e255: 8 total, 8 up, 7 in 2024-02-05T08:15:40.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:40.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:40.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:40.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:40.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:40.924 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:40.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:41.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:40 smithi196 ceph-mon[33018]: pgmap v484: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 771 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-02-05T08:15:41.006 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:40 smithi196 ceph-mon[33018]: osdmap e255: 8 total, 8 up, 7 in 2024-02-05T08:15:41.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:41.055 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T08:15:41.068 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:41.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:41.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:41.169 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:41.205 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:41.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:41.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:41.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:41.352 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:41.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:41.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:41.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:41.481 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:15:41.482 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 27 16 54045 54029 125.045 95.0625 0.0106885 0.00799207 2024-02-05T08:15:41.489 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:41.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:41.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:41.519 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:15:41.554 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:15:41.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:15:41.587 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:15:41.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:41.624 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:15:41.636 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:41.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:41.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:41.665 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:15:41.702 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:15:41.738 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:41.748 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:15:41.788 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:41.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:41.789 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:15:41.792 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:15:41.792 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:15:41.792 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:15:41.792 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:15:41.792 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:15:41.792 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:15:41.792 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:15:41.792 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:15:41.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:41.931 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:41.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:41.961 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:41 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1115658872' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:41.962 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:41 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1115658872' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:41.966 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:41 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1115658872' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:42.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:42.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:42.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:42.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:42.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:42.209 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:42.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:42.310 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:42.351 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:42.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:42.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:42.482 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 28 16 55555 55539 123.95 94.375 0.00701729 0.00806246 2024-02-05T08:15:42.490 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:42.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:42.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:42.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:42.632 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:42.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:42.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:42.734 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:42.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:42.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:42.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:42.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:42.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:42.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:43.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:43.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:42 smithi134 ceph-mon[30086]: pgmap v486: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 733 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T08:15:43.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:42 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:15:43.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:42 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:15:43.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:42 smithi134 ceph-mon[39278]: pgmap v486: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 733 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T08:15:43.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:42 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:15:43.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:42 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:15:43.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:43.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:42 smithi196 ceph-mon[33018]: pgmap v486: 148 pgs: 148 active+clean; 3.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 733 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T08:15:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:42 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:15:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:42 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:15:43.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:43.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:43.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:43.197 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:43.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:43.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:43.299 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:43.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:43.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:43.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:43.431 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:43.461 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:43.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:43.482 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 29 16 56893 56877 122.559 83.625 0.0131557 0.00815484 2024-02-05T08:15:43.535 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:15:43.535 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:15:43.535 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T08:15:43.535 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:15:43.563 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:43.591 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:43.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:43.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:43.693 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:43.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:43.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:43.744 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:15:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:15:43.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:43.858 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:43.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:43.960 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:44.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:44.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:44.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:44.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:43 smithi134 ceph-mon[30086]: osdmap e256: 8 total, 8 up, 7 in 2024-02-05T08:15:44.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:43 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:15:44.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:43 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:15:44.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:43 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:43 smithi134 ceph-mon[39278]: osdmap e256: 8 total, 8 up, 7 in 2024-02-05T08:15:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:43 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:15:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:43 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:15:44.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:43 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:43 smithi196 ceph-mon[33018]: osdmap e256: 8 total, 8 up, 7 in 2024-02-05T08:15:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:43 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:15:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:43 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:15:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:43 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:44.106 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:44.135 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:44.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:44.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:44.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:44.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T08:15:44.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:44.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:44.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:44.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:44.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:44.483 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 30 16 58121 58105 121.032 76.75 0.0155605 0.00825734 2024-02-05T08:15:44.516 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:44.546 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:44.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:44.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:44.648 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:44.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:44.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:44.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:44.714 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:15:44.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:44.808 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:44.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:44.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:44.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:15:44.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:44.947 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:44.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:45.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:44 smithi134 ceph-mon[30086]: pgmap v488: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 153 MiB/s wr, 2.45k op/s 2024-02-05T08:15:45.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:44 smithi134 ceph-mon[30086]: osdmap e257: 8 total, 8 up, 7 in 2024-02-05T08:15:45.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:44 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:15:45.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:44 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:15:45.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:44 smithi134 ceph-mon[39278]: pgmap v488: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 153 MiB/s wr, 2.45k op/s 2024-02-05T08:15:45.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:44 smithi134 ceph-mon[39278]: osdmap e257: 8 total, 8 up, 7 in 2024-02-05T08:15:45.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:44 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:15:45.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:44 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:15:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:44 smithi196 ceph-mon[33018]: pgmap v488: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 153 MiB/s wr, 2.45k op/s 2024-02-05T08:15:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:44 smithi196 ceph-mon[33018]: osdmap e257: 8 total, 8 up, 7 in 2024-02-05T08:15:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:44 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:15:45.029 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:44 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:15:45.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:45.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:45.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:45.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:45.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:45.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:45.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:45.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:45.363 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:45.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:45.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:45.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:45.482 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 31 16 59223 59207 119.348 68.875 0.012397 0.00837341 2024-02-05T08:15:45.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:45.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:45.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:45.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:45.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:45.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:45.779 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:45.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:45.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:45.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:45.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:46.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:46.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:45 smithi196 ceph-mon[33018]: osdmap e258: 8 total, 8 up, 7 in 2024-02-05T08:15:46.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:45 smithi134 ceph-mon[30086]: osdmap e258: 8 total, 8 up, 7 in 2024-02-05T08:15:46.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:45 smithi134 ceph-mon[39278]: osdmap e258: 8 total, 8 up, 7 in 2024-02-05T08:15:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:46.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:46.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:46.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:46.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:46.211 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:46.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:46.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:46.337 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:46.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:46.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:46.438 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:46.479 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:46.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:46.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:46.482 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 32 16 60554 60538 118.218 83.1875 0.0149998 0.00845283 2024-02-05T08:15:46.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:46.621 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:46.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:46.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:46.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:46.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:46.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:46.864 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:46.901 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:46.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:47.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:47.034 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:47.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:47.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:47.044 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:15:47.044 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:15:47.044 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T08:15:47.044 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T08:15:47.044 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:15:47.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:46 smithi196 ceph-mon[33018]: pgmap v491: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 855 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T08:15:47.106 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:46 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/492770289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:15:47.136 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:47.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:47.170 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:47.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:47.239 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:46 smithi134 ceph-mon[30086]: pgmap v491: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 855 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T08:15:47.239 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:46 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/492770289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:15:47.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:46 smithi134 ceph-mon[39278]: pgmap v491: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 855 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T08:15:47.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:46 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/492770289' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:15:47.240 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:15:47.240 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:15:47.240 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 258, 2024-02-05T08:15:47.240 INFO:teuthology.orchestra.run.smithi134.stdout: "fsid": "43e13aa4-c3fc-11ee-95b6-87774f69a715", 2024-02-05T08:15:47.240 INFO:teuthology.orchestra.run.smithi134.stdout: "created": "2024-02-05T07:59:09.321262+0000", 2024-02-05T08:15:47.240 INFO:teuthology.orchestra.run.smithi134.stdout: "modified": "2024-02-05T08:15:44.767439+0000", 2024-02-05T08:15:47.240 INFO:teuthology.orchestra.run.smithi134.stdout: "last_up_change": "2024-02-05T08:04:26.849889+0000", 2024-02-05T08:15:47.240 INFO:teuthology.orchestra.run.smithi134.stdout: "last_in_change": "2024-02-05T08:12:40.680828+0000", 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_num": 5806080, 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_set": [ 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "nodeep-scrub", 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "noscrub", 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "pglog_hardlimit", 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "purged_snapdirs", 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_deletes", 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "sortbitwise" 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_version": 18, 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_snapshot": "", 2024-02-05T08:15:47.241 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_max": 9, 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "max_osd": 8, 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "require_min_compat_client": "luminous", 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "min_compat_client": "luminous", 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "require_osd_release": "squid", 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "allow_crimson": false, 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "pools": [ 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 1, 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": ".mgr", 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:02:21.034012+0000", 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:15:47.242 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 1, 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 1, 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 1, 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 1, 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 1, 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:15:47.243 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "172", 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:15:47.244 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:15:47.245 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "options": { 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_max": 32, 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_min": 1 2024-02-05T08:15:47.246 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr": {} 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 7.8899998664855957, 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 7.8899998664855957, 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.37999999523162842, 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 3, 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 3, 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.247 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 2, 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "datapool", 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:39.400746+0000", 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 8193, 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:15:47.248 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 3, 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 3, 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 3, 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 3, 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 3, 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.249 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "70", 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 2, 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 70, 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:15:47.250 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:15:47.251 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd": {} 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.6700000762939453, 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.6700000762939453, 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.75, 2024-02-05T08:15:47.252 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 2, 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 2, 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 3, 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": ".rgw.root", 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:40.005769+0000", 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:15:47.253 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:15:47.254 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "68", 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:15:47.255 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:15:47.256 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:15:47.257 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.9900000095367432, 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.9900000095367432, 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.75, 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.75, 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.258 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 4, 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.log", 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:41.851891+0000", 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:15:47.259 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.260 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "70", 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:15:47.261 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:15:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.2400000095367432, 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.2400000095367432, 2024-02-05T08:15:47.263 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.9700000286102295, 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.9700000286102295, 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 5, 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.control", 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:43.793804+0000", 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:15:47.264 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:15:47.265 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "72", 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:15:47.266 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:15:47.267 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:15:47.268 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:15:47.269 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:15:47.269 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:15:47.269 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:15:47.269 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:15:47.269 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:15:47.269 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:15:47.269 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.269 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:15:47.269 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:15:47.269 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.4900000095367432, 2024-02-05T08:15:47.269 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.4900000095367432, 2024-02-05T08:15:47.269 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:15:47.269 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.309999942779541, 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.309999942779541, 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 6, 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.meta", 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:45.853999+0000", 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:15:47.270 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:15:47.271 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "74", 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:15:47.272 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:15:47.273 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "options": { 2024-02-05T08:15:47.274 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_bias": 4 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.9900000095367432, 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.9900000095367432, 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.75, 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.75, 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:15:47.275 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 9, 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "unique_pool_2", 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:14:56.360721+0000", 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:15:47.276 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 16, 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 16, 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 16, 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 16, 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 16, 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:15:47.277 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "244", 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:15:47.278 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:15:47.279 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "rados": {} 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:15:47.280 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.9900000095367432, 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.9900000095367432, 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.75, 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.75, 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: "osds": [ 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 0, 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "d90a5ea0-9a19-452e-9564-ca7b7a7c45e4", 2024-02-05T08:15:47.281 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 8, 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 239, 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6802", 2024-02-05T08:15:47.282 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6803", 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6804", 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.283 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6805", 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6808", 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.284 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6809", 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6806", 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.285 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6807", 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6803/2540271081", 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6805/2540271081", 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6809/2540271081", 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6807/2540271081", 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 1, 2024-02-05T08:15:47.286 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4", 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 13, 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 239, 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6810", 2024-02-05T08:15:47.287 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:15:47.288 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.288 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.288 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.288 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6811", 2024-02-05T08:15:47.288 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:15:47.288 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.288 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.288 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.288 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:15:47.288 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.288 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.288 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6812", 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6813", 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.289 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6816", 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6817", 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6814", 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:15:47.290 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6815", 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6811/2032802960", 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6813/2032802960", 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6817/2032802960", 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6815/2032802960", 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.291 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 2, 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708", 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 18, 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 209, 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:15:47.292 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6818", 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6819", 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.293 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6820", 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6821", 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.294 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6824", 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6825", 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.295 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6822", 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6823", 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6819/153827987", 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6821/153827987", 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6825/153827987", 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6823/153827987", 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:15:47.296 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 3, 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "6774e330-3202-4671-894e-172cc2f6189d", 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 0, 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 0, 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 27, 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 214, 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:15:47.297 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6826", 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6827", 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.298 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6828", 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6829", 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.299 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6832", 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6833", 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.300 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6830", 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6831", 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.301 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6827/1527507056", 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6829/1527507056", 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6833/1527507056", 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6831/1527507056", 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 4, 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693", 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:15:47.302 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 32, 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 239, 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6800", 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.303 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6801", 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6802", 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.304 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6803", 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6806", 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6807", 2024-02-05T08:15:47.305 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6804", 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6805", 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.306 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6801/3005156808", 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6803/3005156808", 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3005156808", 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3005156808", 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 5, 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955", 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:15:47.307 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 42, 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 239, 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6808", 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.308 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6809", 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6810", 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.309 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6811", 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6814", 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.310 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6815", 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6812", 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.311 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6813", 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6809/1603224792", 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6811/1603224792", 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6815/1603224792", 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6813/1603224792", 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.312 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 6, 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a", 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 51, 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 239, 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.313 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6816", 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6817", 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6818", 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:15:47.314 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6819", 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6822", 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6823", 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6820", 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6821", 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6817/3727413143", 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6819/3727413143", 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6823/3727413143", 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6821/3727413143", 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.317 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 7, 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057", 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 60, 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 239, 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.318 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6824", 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6825", 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.319 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6826", 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6827", 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.320 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6830", 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6831", 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:15:47.321 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6828", 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6829", 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6825/4206571010", 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6827/4206571010", 2024-02-05T08:15:47.322 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6831/4206571010", 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6829/4206571010", 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_xinfo": [ 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 0, 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:15:47.323 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:14:38.529311+0000", 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 1, 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:15:43.296990+0000", 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:15:47.324 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 2, 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:15:42.004559+0000", 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 3, 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:15:47.325 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 65536, 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:15:31.868009+0000", 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 4, 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:15:44.416760+0000", 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:15:47.326 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 5, 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:15:32.979597+0000", 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 6, 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:15:47.327 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:15:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:15:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:15:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:15:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:15:21.714789+0000", 2024-02-05T08:15:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:15:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 7, 2024-02-05T08:15:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:15:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:15:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:15:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:15:47.328 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:15:19.312082+0000", 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap": [ 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "3.1e", 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: "osds": [ 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: 3, 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: 4, 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: 1 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:15:47.329 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap_items": [ 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "3.6", 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 4, 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.6", 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:15:47.330 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 5 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.7", 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.a", 2024-02-05T08:15:47.331 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.12", 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 5 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.332 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.1f", 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap_primaries": [], 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_temp": [], 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_temp": [], 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: "blocklist": { 2024-02-05T08:15:47.333 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/788812699": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/1046500326": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/464208089": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/301941854": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/3607168099": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/673876301": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/4225724311": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/1525062017": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/3607168099": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/673876301": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3200884805": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/2952444615": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3325835814": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3161126612": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/69097091": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/1525062017": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:15:47.334 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/1645475987": "2024-02-06T08:05:21.465844+0000" 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: "range_blocklist": {}, 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profiles": { 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: "default": { 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: "crush-failure-domain": "osd", 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: "k": "2", 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: "m": "1", 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: "plugin": "jerasure", 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: "technique": "reed_sol_van" 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps_queue": [], 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: "new_removed_snaps": [], 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: "new_purged_snaps": [], 2024-02-05T08:15:47.335 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_node_flags": {}, 2024-02-05T08:15:47.336 INFO:teuthology.orchestra.run.smithi134.stdout: "device_class_flags": {}, 2024-02-05T08:15:47.336 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode": { 2024-02-05T08:15:47.336 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode_enabled": false, 2024-02-05T08:15:47.336 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_bucket_count": 0, 2024-02-05T08:15:47.336 INFO:teuthology.orchestra.run.smithi134.stdout: "degraded_stretch_mode": 0, 2024-02-05T08:15:47.336 INFO:teuthology.orchestra.run.smithi134.stdout: "recovering_stretch_mode": 0, 2024-02-05T08:15:47.336 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode_bucket": 0 2024-02-05T08:15:47.336 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:15:47.336 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:15:47.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:47.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:47.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:47.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:47.482 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 33 16 61963 61947 117.304 88.0625 0.0108123 0.00852 2024-02-05T08:15:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:47.520 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:47.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:47.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:47.659 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:47.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:47.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:47.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:47.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:47.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T08:15:47.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:47.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:47.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:47.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:48.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:15:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:15:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:15:48.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:48.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:48.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:48.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:48.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:48.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:48.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:48.302 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:15:48.368 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:48.385 INFO:tasks.thrashosds.thrasher:j is {'epoch': 258, 'fsid': '43e13aa4-c3fc-11ee-95b6-87774f69a715', 'created': '2024-02-05T07:59:09.321262+0000', 'modified': '2024-02-05T08:15:44.767439+0000', 'last_up_change': '2024-02-05T08:04:26.849889+0000', 'last_in_change': '2024-02-05T08:12:40.680828+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T08:02:21.034012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '172', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T08:04:39.400746+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T08:04:40.005769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T08:04:41.851891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T08:04:43.793804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T08:04:45.853999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-02-05T08:14:56.360721+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '244', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'd90a5ea0-9a19-452e-9564-ca7b7a7c45e4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 239, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6802', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 2540271081}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6804', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 2540271081}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6808', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2540271081}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6806', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 2540271081}]}, 'public_addr': '172.21.15.134:6803/2540271081', 'cluster_addr': '172.21.15.134:6805/2540271081', 'heartbeat_back_addr': '172.21.15.134:6809/2540271081', 'heartbeat_front_addr': '172.21.15.134:6807/2540271081', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 239, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6810', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2032802960}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6812', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 2032802960}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6816', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6817', 'nonce': 2032802960}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6814', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 2032802960}]}, 'public_addr': '172.21.15.134:6811/2032802960', 'cluster_addr': '172.21.15.134:6813/2032802960', 'heartbeat_back_addr': '172.21.15.134:6817/2032802960', 'heartbeat_front_addr': '172.21.15.134:6815/2032802960', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '078e578b-855b-49ff-83bd-0548a9c5a708', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 209, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6818', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6819', 'nonce': 153827987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6820', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6821', 'nonce': 153827987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 153827987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6822', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6823', 'nonce': 153827987}]}, 'public_addr': '172.21.15.134:6819/153827987', 'cluster_addr': '172.21.15.134:6821/153827987', 'heartbeat_back_addr': '172.21.15.134:6825/153827987', 'heartbeat_front_addr': '172.21.15.134:6823/153827987', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6774e330-3202-4671-894e-172cc2f6189d', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 1527507056}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6828', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6829', 'nonce': 1527507056}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6832', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6833', 'nonce': 1527507056}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6830', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6831', 'nonce': 1527507056}]}, 'public_addr': '172.21.15.134:6827/1527507056', 'cluster_addr': '172.21.15.134:6829/1527507056', 'heartbeat_back_addr': '172.21.15.134:6833/1527507056', 'heartbeat_front_addr': '172.21.15.134:6831/1527507056', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '52f70cfa-ab0e-4751-a5f8-1045c61c1693', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 239, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3005156808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3005156808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3005156808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3005156808}]}, 'public_addr': '172.21.15.196:6801/3005156808', 'cluster_addr': '172.21.15.196:6803/3005156808', 'heartbeat_back_addr': '172.21.15.196:6807/3005156808', 'heartbeat_front_addr': '172.21.15.196:6805/3005156808', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b4b21769-d40f-43fd-b7e9-2477b74d3955', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 239, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 1603224792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 1603224792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 1603224792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 1603224792}]}, 'public_addr': '172.21.15.196:6809/1603224792', 'cluster_addr': '172.21.15.196:6811/1603224792', 'heartbeat_back_addr': '172.21.15.196:6815/1603224792', 'heartbeat_front_addr': '172.21.15.196:6813/1603224792', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'af1cced4-0015-47f5-b105-27641bfe4f4a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 239, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 3727413143}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 3727413143}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 3727413143}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 3727413143}]}, 'public_addr': '172.21.15.196:6817/3727413143', 'cluster_addr': '172.21.15.196:6819/3727413143', 'heartbeat_back_addr': '172.21.15.196:6823/3727413143', 'heartbeat_front_addr': '172.21.15.196:6821/3727413143', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c54beeae-9f7c-4723-a369-13bb531b4057', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 239, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 4206571010}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 4206571010}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 4206571010}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 4206571010}]}, 'public_addr': '172.21.15.196:6825/4206571010', 'cluster_addr': '172.21.15.196:6827/4206571010', 'heartbeat_back_addr': '172.21.15.196:6831/4206571010', 'heartbeat_front_addr': '172.21.15.196:6829/4206571010', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:14:38.529311+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:15:43.296990+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:15:42.004559+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T08:15:31.868009+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:15:44.416760+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:15:32.979597+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:15:21.714789+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:15:19.312082+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.1e', 'osds': [3, 4, 1]}], 'pg_upmap_items': [{'pgid': '3.6', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.134:0/788812699': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/1046500326': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/464208089': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/301941854': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:6800/3607168099': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6800/673876301': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/4225724311': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6800/1525062017': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:6801/3607168099': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6801/673876301': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/3200884805': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/2952444615': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:0/3325835814': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/3161126612': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:0/69097091': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:6801/1525062017': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/1645475987': '2024-02-06T08:05:21.465844+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T08:15:48.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:15:48.416 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:48.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:48.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:48.444 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:48.483 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 34 16 63298 63282 116.305 83.4375 0.00431372 0.00859321 2024-02-05T08:15:48.513 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:48.518 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:48.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:48.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:48.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:48.747 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:48.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:48.849 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:48.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:48.887 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:48.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:48.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:49.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:49.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:48 smithi196 ceph-mon[33018]: pgmap v492: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-02-05T08:15:49.128 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:49.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:49.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:49.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:49.268 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:49.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:48 smithi134 ceph-mon[30086]: pgmap v492: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-02-05T08:15:49.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:48 smithi134 ceph-mon[39278]: pgmap v492: 148 pgs: 148 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-02-05T08:15:49.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:49.310 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:49.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:49.411 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:49.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:49.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:49.483 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 35 16 64658 64642 115.411 85 0.00452818 0.00865385 2024-02-05T08:15:49.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:49.581 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:49.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:49.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:49.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:49.750 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:49.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:49.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:49.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:49.893 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:49.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:49.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:50.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:50.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:50.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:50.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:50.167 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:50.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:50.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:50.269 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:50.300 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:50.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:50.357 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:15:50.357 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:15:50.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:15:50.357 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:15:50.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.357 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:108961 8.9 8.107267e9 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:15:50.357 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:29.041581+0000", 2024-02-05T08:15:50.357 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 201.311018563, 2024-02-05T08:15:50.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62651763699999996, 2024-02-05T08:15:50.357 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108961 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041581+0000", 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041581+0000", 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041582+0000", 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5570000000000001e-06 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041733+0000", 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015074700000000001 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041745+0000", 2024-02-05T08:15:50.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1508999999999999e-05 2024-02-05T08:15:50.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041755+0000", 2024-02-05T08:15:50.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0533000000000001e-05 2024-02-05T08:15:50.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041803+0000", 2024-02-05T08:15:50.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8022999999999997e-05 2024-02-05T08:15:50.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041811+0000", 2024-02-05T08:15:50.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0460000000000006e-06 2024-02-05T08:15:50.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668084+0000", 2024-02-05T08:15:50.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62627265600000004 2024-02-05T08:15:50.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668098+0000", 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4566000000000001e-05 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:108978 8.9 8.d5065aa9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:29.058870+0000", 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 201.293729337, 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.60925642300000005, 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:15:50.362 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.363 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.363 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.363 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108978 2024-02-05T08:15:50.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.363 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058870+0000", 2024-02-05T08:15:50.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058870+0000", 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058872+0000", 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.655e-06 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058938+0000", 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6568000000000004e-05 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058949+0000", 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.058e-05 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058957+0000", 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8099999999999998e-06 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058997+0000", 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0797999999999998e-05 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059004+0000", 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7510000000000002e-06 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668120+0000", 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.60911546800000005 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668126+0000", 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.793e-06 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.366 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:108980 8.5 8.acde60b5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:29.059192+0000", 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 201.29340698799999, 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.61158693799999997, 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108980 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059192+0000", 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059192+0000", 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059193+0000", 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4900000000000005e-07 2024-02-05T08:15:50.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059259+0000", 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5699999999999998e-05 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059272+0000", 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3658e-05 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059281+0000", 2024-02-05T08:15:50.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9469999999999992e-06 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059332+0000", 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0247999999999997e-05 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059338+0000", 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4659999999999996e-06 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.670768+0000", 2024-02-05T08:15:50.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.61143003200000001 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.670779+0000", 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1138000000000001e-05 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:109000 8.9 8.6538e8d9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:29.089503+0000", 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 201.26309624199999, 2024-02-05T08:15:50.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.57864895999999999, 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 109000 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089503+0000", 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089503+0000", 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089505+0000", 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.559e-06 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089552+0000", 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7407000000000002e-05 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089568+0000", 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5722999999999998e-05 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089576+0000", 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1759999999999993e-06 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089638+0000", 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2442999999999998e-05 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089646+0000", 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2629999999999997e-06 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668145+0000", 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.57849936599999996 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668152+0000", 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0230000000000004e-06 2024-02-05T08:15:50.375 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170181 8.9 8:97b09017:::benchmark_data_smithi134_115836_object53521:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.702283+0000", 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.65031624, 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.090149636, 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.376 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170181 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.702283+0000", 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.702283+0000", 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.702285+0000", 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5990000000000001e-06 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.702290+0000", 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7989999999999999e-06 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.702308+0000", 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7918e-05 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.702318+0000", 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0064999999999999e-05 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.707786+0000", 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054679469999999999 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.707795+0000", 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6729999999999998e-06 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.716814+0000", 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0090188639999999997 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.716824+0000", 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9709999999999999e-06 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.757276+0000", 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.040452063 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for missing object", 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.757390+0000", 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114508 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792380+0000", 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034989287000000001 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792433+0000", 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2942000000000002e-05 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.381 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170200 8.9 8:9eea0c4e:::benchmark_data_smithi134_115836_object53540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.725435+0000", 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.62716387, 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0773159670000001, 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170200 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725435+0000", 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725435+0000", 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725437+0000", 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.463e-06 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725443+0000", 2024-02-05T08:15:50.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5990000000000001e-06 2024-02-05T08:15:50.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725454+0000", 2024-02-05T08:15:50.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1479e-05 2024-02-05T08:15:50.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725462+0000", 2024-02-05T08:15:50.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9170000000000006e-06 2024-02-05T08:15:50.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725517+0000", 2024-02-05T08:15:50.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.533e-05 2024-02-05T08:15:50.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725526+0000", 2024-02-05T08:15:50.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.738e-06 2024-02-05T08:15:50.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.757616+0000", 2024-02-05T08:15:50.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.032089603 2024-02-05T08:15:50.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.757697+0000", 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1055000000000002e-05 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.758135+0000", 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00043868999999999999 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.779652+0000", 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021517172000000001 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.786094+0000", 2024-02-05T08:15:50.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0064418239999999996 2024-02-05T08:15:50.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.802651+0000", 2024-02-05T08:15:50.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016556479999999998 2024-02-05T08:15:50.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.802688+0000", 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7465000000000001e-05 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.802751+0000", 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3151999999999994e-05 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170221 8.9 8:92d7f30d:::benchmark_data_smithi134_115836_object53561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.739479+0000", 2024-02-05T08:15:50.389 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.61312072200001, 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0641018680000001, 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170221 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739479+0000", 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739479+0000", 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739480+0000", 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.584e-06 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739486+0000", 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6099999999999997e-06 2024-02-05T08:15:50.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739497+0000", 2024-02-05T08:15:50.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1277000000000001e-05 2024-02-05T08:15:50.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739507+0000", 2024-02-05T08:15:50.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7459999999999995e-06 2024-02-05T08:15:50.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739593+0000", 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6298999999999995e-05 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739603+0000", 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.011e-05 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.773653+0000", 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0340492779999999 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.773740+0000", 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7072999999999998e-05 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.773820+0000", 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0129000000000005e-05 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.779658+0000", 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058380710000000002 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.788894+0000", 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0092360129999999995 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803498+0000", 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014604114 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803528+0000", 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0465999999999999e-05 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803580+0000", 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2098000000000002e-05 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170223 8.9 8:95eaee90:::benchmark_data_smithi134_115836_object53563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.739633+0000", 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.612966046, 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.064076367, 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.396 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170223 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739633+0000", 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739633+0000", 2024-02-05T08:15:50.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739634+0000", 2024-02-05T08:15:50.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0999999999999998e-07 2024-02-05T08:15:50.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739640+0000", 2024-02-05T08:15:50.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3749999999999999e-06 2024-02-05T08:15:50.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739650+0000", 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6670000000000008e-06 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739657+0000", 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0990000000000001e-06 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739725+0000", 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7936999999999997e-05 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739737+0000", 2024-02-05T08:15:50.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1558e-05 2024-02-05T08:15:50.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.775454+0000", 2024-02-05T08:15:50.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.035717086 2024-02-05T08:15:50.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.775553+0000", 2024-02-05T08:15:50.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9755999999999998e-05 2024-02-05T08:15:50.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.775642+0000", 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8261000000000001e-05 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.782170+0000", 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065284360000000003 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.788944+0000", 2024-02-05T08:15:50.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067739209999999996 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803613+0000", 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014668936 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803651+0000", 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7505000000000001e-05 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803710+0000", 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9120000000000003e-05 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170234 8.9 8:9450d93c:::benchmark_data_smithi134_115836_object53574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.755021+0000", 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.59757787700001, 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.048796149, 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.403 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.404 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.404 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170234 2024-02-05T08:15:50.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.404 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755021+0000", 2024-02-05T08:15:50.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755021+0000", 2024-02-05T08:15:50.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755024+0000", 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1169999999999998e-06 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755029+0000", 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8119999999999996e-06 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755042+0000", 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2541000000000001e-05 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755051+0000", 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2250000000000006e-06 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755134+0000", 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3196999999999996e-05 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755145+0000", 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0212e-05 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.776979+0000", 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.021834707 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.777109+0000", 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000129868 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.777181+0000", 2024-02-05T08:15:50.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1717000000000005e-05 2024-02-05T08:15:50.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.782174+0000", 2024-02-05T08:15:50.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049935209999999999 2024-02-05T08:15:50.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.791943+0000", 2024-02-05T08:15:50.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0097690020000000006 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803736+0000", 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011792810000000001 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803766+0000", 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.006e-05 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803818+0000", 2024-02-05T08:15:50.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1360000000000003e-05 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170243 8.9 8:90429ea4:::benchmark_data_smithi134_115836_object53583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.761267+0000", 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.591331954, 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0487509880000001, 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.410 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170243 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761267+0000", 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761267+0000", 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761269+0000", 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4559999999999999e-06 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761274+0000", 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7090000000000001e-06 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761286+0000", 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2884999999999999e-05 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761294+0000", 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7209999999999994e-06 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761361+0000", 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6507000000000005e-05 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761371+0000", 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0225e-05 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.778407+0000", 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0170360919999999 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.778486+0000", 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8974000000000007e-05 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.778554+0000", 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8254000000000001e-05 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.784715+0000", 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006161265 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792019+0000", 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073035729999999998 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.809736+0000", 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017717096000000002 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.809843+0000", 2024-02-05T08:15:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010713399999999999 2024-02-05T08:15:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.810018+0000", 2024-02-05T08:15:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017509700000000001 2024-02-05T08:15:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.416 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170265 8.9 8:93c691bd:::benchmark_data_smithi134_115836_object53605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.786248+0000", 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.56635145, 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0239562790000001, 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170265 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786248+0000", 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786248+0000", 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786249+0000", 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.051e-06 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786254+0000", 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4500000000000003e-06 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786270+0000", 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5780000000000001e-05 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786278+0000", 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2539999999999992e-06 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786346+0000", 2024-02-05T08:15:50.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7643000000000003e-05 2024-02-05T08:15:50.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786355+0000", 2024-02-05T08:15:50.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0690000000000008e-06 2024-02-05T08:15:50.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.779698+0000", 2024-02-05T08:15:50.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99334322799999997 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.779812+0000", 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113348 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.779943+0000", 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013156600000000001 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.786038+0000", 2024-02-05T08:15:50.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060945799999999996 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792480+0000", 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0064420859999999996 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.810091+0000", 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017610614 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.810128+0000", 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7329999999999997e-05 2024-02-05T08:15:50.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.810204+0000", 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6279999999999995e-05 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170309 8.9 8:96bff997:::benchmark_data_smithi134_115836_object53649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.842207+0000", 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.51039224599999, 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99189979299999997, 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.423 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170309 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842207+0000", 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842207+0000", 2024-02-05T08:15:50.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.425 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842209+0000", 2024-02-05T08:15:50.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.764e-06 2024-02-05T08:15:50.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.425 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:50.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842215+0000", 2024-02-05T08:15:50.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0630000000000002e-06 2024-02-05T08:15:50.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842231+0000", 2024-02-05T08:15:50.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5920999999999999e-05 2024-02-05T08:15:50.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842240+0000", 2024-02-05T08:15:50.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4919999999999999e-06 2024-02-05T08:15:50.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842307+0000", 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6278000000000005e-05 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842318+0000", 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1601e-05 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.781005+0000", 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93868725799999997 2024-02-05T08:15:50.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.781107+0000", 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101585 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.781179+0000", 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1582999999999995e-05 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.787700+0000", 2024-02-05T08:15:50.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065215459999999996 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792531+0000", 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048308650000000002 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834010+0000", 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.041478810999999997 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834066+0000", 2024-02-05T08:15:50.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6192e-05 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834107+0000", 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0834000000000002e-05 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170330 8.9 8:92c65df7:::benchmark_data_smithi134_115836_object53670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.865357+0000", 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.487242708, 2024-02-05T08:15:50.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96889826400000001, 2024-02-05T08:15:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170330 2024-02-05T08:15:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865357+0000", 2024-02-05T08:15:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865357+0000", 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865358+0000", 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4839999999999999e-06 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865364+0000", 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1020000000000002e-06 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865383+0000", 2024-02-05T08:15:50.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8620000000000001e-05 2024-02-05T08:15:50.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865392+0000", 2024-02-05T08:15:50.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8689999999999993e-06 2024-02-05T08:15:50.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865477+0000", 2024-02-05T08:15:50.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5654000000000004e-05 2024-02-05T08:15:50.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865490+0000", 2024-02-05T08:15:50.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2308e-05 2024-02-05T08:15:50.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.782216+0000", 2024-02-05T08:15:50.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.91672598400000005 2024-02-05T08:15:50.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.782347+0000", 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013109100000000001 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.782515+0000", 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000168593 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.787704+0000", 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051884519999999996 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.794555+0000", 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068516999999999996 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834163+0000", 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039607246999999998 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834220+0000", 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6884e-05 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834255+0000", 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5275999999999999e-05 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170336 8.9 8:9638bc28:::benchmark_data_smithi134_115836_object53676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.869806+0000", 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.482793528, 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96456107099999999, 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.438 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170336 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869806+0000", 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869806+0000", 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869807+0000", 2024-02-05T08:15:50.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3e-06 2024-02-05T08:15:50.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869811+0000", 2024-02-05T08:15:50.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.216e-06 2024-02-05T08:15:50.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869830+0000", 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8218000000000001e-05 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869843+0000", 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2977e-05 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869900+0000", 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7697e-05 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869909+0000", 2024-02-05T08:15:50.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7849999999999997e-06 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.783591+0000", 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.91368222399999999 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.783670+0000", 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8839999999999997e-05 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.783743+0000", 2024-02-05T08:15:50.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2649000000000002e-05 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.787707+0000", 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039647909999999996 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.794616+0000", 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0069084580000000001 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834295+0000", 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039678791999999997 2024-02-05T08:15:50.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834332+0000", 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7585e-05 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834367+0000", 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4539000000000003e-05 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170349 8.9 8:9486e005:::benchmark_data_smithi134_115836_object53689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.884848+0000", 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.46775176, 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94965263499999997, 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170349 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884848+0000", 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884848+0000", 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884849+0000", 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.079e-06 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.446 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884853+0000", 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0400000000000003e-06 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884863+0000", 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0521e-05 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884870+0000", 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0509999999999999e-06 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884913+0000", 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2555000000000002e-05 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884919+0000", 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1510000000000001e-06 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.784743+0000", 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.89982445499999997 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.784894+0000", 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015011199999999999 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.785027+0000", 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013361800000000001 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.788953+0000", 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039257370000000003 2024-02-05T08:15:50.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.796649+0000", 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.007695744 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834414+0000", 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037765493999999997 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834466+0000", 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1848000000000002e-05 2024-02-05T08:15:50.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834500+0000", 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4230000000000003e-05 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170351 8.9 8:968af758:::benchmark_data_smithi134_115836_object53691:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.889654+0000", 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.46294558, 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94761108199999999, 2024-02-05T08:15:50.451 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170351 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889654+0000", 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889654+0000", 2024-02-05T08:15:50.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889655+0000", 2024-02-05T08:15:50.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.35e-06 2024-02-05T08:15:50.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889661+0000", 2024-02-05T08:15:50.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5949999999999998e-06 2024-02-05T08:15:50.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889673+0000", 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2588e-05 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889683+0000", 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8260000000000005e-06 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889744+0000", 2024-02-05T08:15:50.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0723000000000001e-05 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889753+0000", 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3270000000000007e-06 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.786146+0000", 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.89639291600000004 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.786261+0000", 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114831 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.786400+0000", 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013884300000000001 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.791892+0000", 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054924190000000001 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.796690+0000", 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047979820000000001 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837182+0000", 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040491744000000003 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837229+0000", 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7046999999999999e-05 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837265+0000", 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5890999999999999e-05 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170364 8.9 8:96172a8c:::benchmark_data_smithi134_115836_object53704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.908364+0000", 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.44423531, 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92901532799999997, 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.458 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170364 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908364+0000", 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908364+0000", 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908365+0000", 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.034e-06 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908369+0000", 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8430000000000003e-06 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908379+0000", 2024-02-05T08:15:50.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0528e-05 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908385+0000", 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0480000000000004e-06 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908435+0000", 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9166999999999999e-05 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908441+0000", 2024-02-05T08:15:50.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5409999999999997e-06 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.787740+0000", 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87929909299999998 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.787810+0000", 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9851999999999995e-05 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.787877+0000", 2024-02-05T08:15:50.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6863e-05 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.791898+0000", 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040212030000000001 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.796750+0000", 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048520569999999999 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837303+0000", 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040552610000000003 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837350+0000", 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7002000000000002e-05 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837379+0000", 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9487000000000001e-05 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.464 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170377 8.9 8:9cb6fb33:::benchmark_data_smithi134_115836_object53717:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.927537+0000", 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.42506229200001, 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.90994256200000001, 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170377 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927537+0000", 2024-02-05T08:15:50.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.466 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927537+0000", 2024-02-05T08:15:50.466 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927538+0000", 2024-02-05T08:15:50.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0860000000000001e-06 2024-02-05T08:15:50.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927542+0000", 2024-02-05T08:15:50.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0319999999999997e-06 2024-02-05T08:15:50.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927554+0000", 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1647999999999999e-05 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927560+0000", 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5810000000000002e-06 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927622+0000", 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1507000000000006e-05 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927632+0000", 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.045e-05 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.788974+0000", 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.86134118900000001 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.789086+0000", 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011277699999999999 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.789219+0000", 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013251800000000001 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.791958+0000", 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027390470000000001 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.796792+0000", 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048341469999999996 2024-02-05T08:15:50.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837412+0000", 2024-02-05T08:15:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040620235999999997 2024-02-05T08:15:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837451+0000", 2024-02-05T08:15:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8872000000000003e-05 2024-02-05T08:15:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837480+0000", 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8472000000000001e-05 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170391 8.9 8:9bfec3b7:::benchmark_data_smithi134_115836_object53731:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.958818+0000", 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.393781581, 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87876214900000005, 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.473 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.474 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170391 2024-02-05T08:15:50.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.474 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958818+0000", 2024-02-05T08:15:50.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958818+0000", 2024-02-05T08:15:50.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958819+0000", 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2240000000000001e-06 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958823+0000", 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1659999999999996e-06 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958835+0000", 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1590000000000001e-05 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958841+0000", 2024-02-05T08:15:50.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3879999999999997e-06 2024-02-05T08:15:50.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958894+0000", 2024-02-05T08:15:50.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3151e-05 2024-02-05T08:15:50.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958901+0000", 2024-02-05T08:15:50.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8719999999999996e-06 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.790448+0000", 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.831546813 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.790573+0000", 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000124673 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.790704+0000", 2024-02-05T08:15:50.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013115100000000001 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792495+0000", 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001791614 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.796825+0000", 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043295790000000001 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837511+0000", 2024-02-05T08:15:50.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040686310000000003 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837550+0000", 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8303999999999998e-05 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837580+0000", 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0314000000000001e-05 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170400 8.9 8:9216392e:::benchmark_data_smithi134_115836_object53740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.988572+0000", 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 125.364026939, 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.84910828500000002, 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170400 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988572+0000", 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988572+0000", 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988574+0000", 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.761e-06 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988580+0000", 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6659999999999996e-06 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988595+0000", 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5441e-05 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988604+0000", 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7250000000000003e-06 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988667+0000", 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3484000000000004e-05 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988679+0000", 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1958e-05 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792565+0000", 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80388560200000003 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792682+0000", 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117037 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792813+0000", 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013053500000000001 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.794221+0000", 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014081300000000001 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.797909+0000", 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036880709999999998 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837613+0000", 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039704383000000003 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837652+0000", 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8931999999999999e-05 2024-02-05T08:15:50.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:15:50.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:15:50.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:15:50.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837681+0000", 2024-02-05T08:15:50.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8560000000000001e-05 2024-02-05T08:15:50.486 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.486 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.486 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.486 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:15:50.486 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:15:50.486 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:15:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:50.487 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:50.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:50.488 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 36 16 65992 65976 114.521 83.375 0.00897111 0.00872671 2024-02-05T08:15:50.588 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:50.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:50.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:50.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:50.761 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:50.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:50.863 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:50.893 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:50.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:50.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:50.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:51.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:50 smithi134 ceph-mon[30086]: pgmap v493: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 735 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-02-05T08:15:51.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:50 smithi134 ceph-mon[30086]: osdmap e259: 8 total, 8 up, 7 in 2024-02-05T08:15:51.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:50 smithi134 ceph-mon[39278]: pgmap v493: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 735 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-02-05T08:15:51.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:50 smithi134 ceph-mon[39278]: osdmap e259: 8 total, 8 up, 7 in 2024-02-05T08:15:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:51.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:51.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:50 smithi196 ceph-mon[33018]: pgmap v493: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 735 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-02-05T08:15:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:50 smithi196 ceph-mon[33018]: osdmap e259: 8 total, 8 up, 7 in 2024-02-05T08:15:51.141 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:51.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:51.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:51.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:51.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:51.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:51.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:51.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:51.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:51.476 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T08:15:51.484 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 37 16 67347 67331 113.712 84.6875 0.00914185 0.0087897 2024-02-05T08:15:51.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:51.624 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:51.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:51.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:51.725 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:51.770 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:51.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:51.786 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:51.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:51.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:51.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:51.949 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:15:52.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:52.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:52.078 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:52.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:52.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:52.211 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:52.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:52.313 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:52.348 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:52.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:52.421 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:15:52.448 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:15:52.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:52.474 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:15:52.484 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 38 16 68522 68506 112.653 73.4375 0.0165294 0.00887233 2024-02-05T08:15:52.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:52.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:52.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:52.507 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:15:52.550 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:15:52.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:52.603 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:15:52.630 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:52.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:52.642 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:15:52.693 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:15:52.699 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:15:52.699 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:15:52.699 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:15:52.699 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:15:52.699 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:15:52.699 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:15:52.699 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:15:52.699 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:15:52.732 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:52.773 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:52.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:52.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:52 smithi134 ceph-mon[30086]: pgmap v495: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 644 B/s rd, 50 MiB/s wr, 806 op/s 2024-02-05T08:15:52.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:52 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3913390620' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:15:52.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:52 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1849568674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:52.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:52 smithi134 ceph-mon[39278]: pgmap v495: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 644 B/s rd, 50 MiB/s wr, 806 op/s 2024-02-05T08:15:52.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:52 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3913390620' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:15:52.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:52 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1849568674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:52.778 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:15:52.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:52.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:52.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:53.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:53.055 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:53.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:53.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:52 smithi196 ceph-mon[33018]: pgmap v495: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 644 B/s rd, 50 MiB/s wr, 806 op/s 2024-02-05T08:15:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:52 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3913390620' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:15:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:52 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1849568674' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:15:53.157 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:53.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:53.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:53.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:53.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:53.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:53.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:53.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:53.463 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:53.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:53.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:53.484 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 39 15 69796 69781 111.807 79.6875 0.0102411 0.00893875 2024-02-05T08:15:53.565 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:53.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:53.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:53.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:53.706 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:53.742 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:15:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:15:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:53.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:53.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:53.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:53.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:53.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:53.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:54.005 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:15:54.005 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3913390620' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:15:54.005 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[30086]: osdmap e260: 8 total, 8 up, 7 in 2024-02-05T08:15:54.005 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:15:54.006 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:15:54.006 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:15:54.006 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:54.006 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:15:54.006 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3913390620' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:15:54.006 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[39278]: osdmap e260: 8 total, 8 up, 7 in 2024-02-05T08:15:54.006 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:15:54.006 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:15:54.006 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:15:54.006 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:53 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:54.007 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:54.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:53 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:15:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:53 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3913390620' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:15:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:53 smithi196 ceph-mon[33018]: osdmap e260: 8 total, 8 up, 7 in 2024-02-05T08:15:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:53 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:15:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:53 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:15:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:15:54.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:53 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:15:54.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:54.149 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:54.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:54.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:54.285 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:54.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:54.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:54.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:54.423 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:54.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:54.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:54.484 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:15:54.485708+0000 min lat: 0.00130673 max lat: 0.198515 avg lat: 0.0090208 2024-02-05T08:15:54.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:15:54.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 40 15 70938 70923 110.796 71.375 0.0146302 0.0090208 2024-02-05T08:15:54.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:54.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:54.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:54.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:54.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:54.716 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:54.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:54.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:54.818 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:54.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:54.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:54.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:55.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:55.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:55.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:54 smithi134 ceph-mon[30086]: pgmap v497: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-02-05T08:15:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:54 smithi134 ceph-mon[30086]: 4.1c scrub starts 2024-02-05T08:15:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:54 smithi134 ceph-mon[30086]: 4.1c scrub ok 2024-02-05T08:15:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:54 smithi134 ceph-mon[30086]: osdmap e261: 8 total, 8 up, 7 in 2024-02-05T08:15:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:15:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:54 smithi134 ceph-mon[39278]: pgmap v497: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-02-05T08:15:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:54 smithi134 ceph-mon[39278]: 4.1c scrub starts 2024-02-05T08:15:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:54 smithi134 ceph-mon[39278]: 4.1c scrub ok 2024-02-05T08:15:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:54 smithi134 ceph-mon[39278]: osdmap e261: 8 total, 8 up, 7 in 2024-02-05T08:15:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:15:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:54 smithi196 ceph-mon[33018]: pgmap v497: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.62k op/s 2024-02-05T08:15:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:54 smithi196 ceph-mon[33018]: 4.1c scrub starts 2024-02-05T08:15:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:54 smithi196 ceph-mon[33018]: 4.1c scrub ok 2024-02-05T08:15:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:54 smithi196 ceph-mon[33018]: osdmap e261: 8 total, 8 up, 7 in 2024-02-05T08:15:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:15:55.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:55.173 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:55.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:55.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:55.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:55.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:55.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:55.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:55.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:55.459 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:55.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:55.484 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 41 16 72094 72078 109.855 72.1875 0.0143821 0.00909802 2024-02-05T08:15:55.560 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:55.603 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:55.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:55.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:55.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:55.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:55.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:55.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:55.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:55.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:55.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:56.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:56.031 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:15:56.053 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:56.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:56.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:55 smithi196 ceph-mon[33018]: 6.11 scrub starts 2024-02-05T08:15:56.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:55 smithi196 ceph-mon[33018]: 6.11 scrub ok 2024-02-05T08:15:56.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:55 smithi196 ceph-mon[33018]: 6.7 scrub starts 2024-02-05T08:15:56.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:55 smithi196 ceph-mon[33018]: 6.7 scrub ok 2024-02-05T08:15:56.069 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:55 smithi134 ceph-mon[30086]: 6.11 scrub starts 2024-02-05T08:15:56.069 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:55 smithi134 ceph-mon[30086]: 6.11 scrub ok 2024-02-05T08:15:56.069 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:55 smithi134 ceph-mon[30086]: 6.7 scrub starts 2024-02-05T08:15:56.069 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:55 smithi134 ceph-mon[30086]: 6.7 scrub ok 2024-02-05T08:15:56.070 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:55 smithi134 ceph-mon[39278]: 6.11 scrub starts 2024-02-05T08:15:56.070 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:55 smithi134 ceph-mon[39278]: 6.11 scrub ok 2024-02-05T08:15:56.070 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:55 smithi134 ceph-mon[39278]: 6.7 scrub starts 2024-02-05T08:15:56.070 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:55 smithi134 ceph-mon[39278]: 6.7 scrub ok 2024-02-05T08:15:56.155 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:56.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:56.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:56.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:56.307 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:56.348 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:56.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:56.444 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:56.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:56.484 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 42 16 73269 73253 108.987 73.4375 0.0263356 0.00917052 2024-02-05T08:15:56.495 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:56.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:56.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:56.559 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:15:56.562 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:15:56.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:56.603 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:15:56.603 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:15:56.631 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:56.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:56.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:56.686 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:15:56.732 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:56.789 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:56.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:56.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:56.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:56.932 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:56.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:57.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:57.071 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:57.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:57.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:57.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:57.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:57.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:57.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:57.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: pgmap v499: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T08:15:57.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: Adjusting osd_memory_target on smithi134 to 3248M 2024-02-05T08:15:57.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:57.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: 5.16 scrub starts 2024-02-05T08:15:57.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: 5.16 scrub ok 2024-02-05T08:15:57.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.237 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.238 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: Adjusting osd_memory_target on smithi196 to 3248M 2024-02-05T08:15:57.238 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:57.238 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:15:57.238 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:15:57.238 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:57.238 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[30086]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:15:57.238 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: pgmap v499: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T08:15:57.238 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.238 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.238 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: Adjusting osd_memory_target on smithi134 to 3248M 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: 5.16 scrub starts 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: 5.16 scrub ok 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: Adjusting osd_memory_target on smithi196 to 3248M 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:15:57.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:57.240 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:56 smithi134 ceph-mon[39278]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:15:57.303 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: pgmap v499: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T08:15:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: Adjusting osd_memory_target on smithi134 to 3248M 2024-02-05T08:15:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: 5.16 scrub starts 2024-02-05T08:15:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: 5.16 scrub ok 2024-02-05T08:15:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T08:15:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: Adjusting osd_memory_target on smithi196 to 3248M 2024-02-05T08:15:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:15:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:15:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:15:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:56 smithi196 ceph-mon[33018]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:15:57.331 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:57.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:57.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:57.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:57.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:57.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:57.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 43 16 74501 74485 108.243 77 0.00746463 0.0092334 2024-02-05T08:15:57.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:15:57.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:57.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:57.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:15:57.712 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:57.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T08:15:57.744 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:57.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:57.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:57.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:57.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:57.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:57.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:58.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:57 smithi134 ceph-mon[30086]: 3.10 scrub starts 2024-02-05T08:15:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:57 smithi134 ceph-mon[30086]: 3.10 scrub ok 2024-02-05T08:15:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:57 smithi134 ceph-mon[30086]: 4.3 scrub starts 2024-02-05T08:15:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:57 smithi134 ceph-mon[30086]: 4.3 scrub ok 2024-02-05T08:15:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:57 smithi134 ceph-mon[30086]: 6.19 scrub starts 2024-02-05T08:15:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:57 smithi134 ceph-mon[30086]: 6.19 scrub ok 2024-02-05T08:15:58.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:15:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:15:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:15:58.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:57 smithi134 ceph-mon[39278]: 3.10 scrub starts 2024-02-05T08:15:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:57 smithi134 ceph-mon[39278]: 3.10 scrub ok 2024-02-05T08:15:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:57 smithi134 ceph-mon[39278]: 4.3 scrub starts 2024-02-05T08:15:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:57 smithi134 ceph-mon[39278]: 4.3 scrub ok 2024-02-05T08:15:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:57 smithi134 ceph-mon[39278]: 6.19 scrub starts 2024-02-05T08:15:58.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:57 smithi134 ceph-mon[39278]: 6.19 scrub ok 2024-02-05T08:15:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:58.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:58.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:58.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:57 smithi196 ceph-mon[33018]: 3.10 scrub starts 2024-02-05T08:15:58.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:57 smithi196 ceph-mon[33018]: 3.10 scrub ok 2024-02-05T08:15:58.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:57 smithi196 ceph-mon[33018]: 4.3 scrub starts 2024-02-05T08:15:58.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:57 smithi196 ceph-mon[33018]: 4.3 scrub ok 2024-02-05T08:15:58.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:57 smithi196 ceph-mon[33018]: 6.19 scrub starts 2024-02-05T08:15:58.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:57 smithi196 ceph-mon[33018]: 6.19 scrub ok 2024-02-05T08:15:58.110 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:15:58.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:58.197 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:58.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:58.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:15:58.335 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:58.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:15:58.437 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:58.478 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:58.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:58.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 44 16 75628 75612 107.384 70.4375 0.00913401 0.00930596 2024-02-05T08:15:58.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:58.607 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:58.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:58.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:58.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:58.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:58.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:58.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:58.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:58.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:58.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:59.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:59.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:59.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:59.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[30086]: 3.a scrub starts 2024-02-05T08:15:59.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[30086]: 3.a scrub ok 2024-02-05T08:15:59.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[30086]: pgmap v500: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 640 B/s rd, 57 MiB/s wr, 914 op/s 2024-02-05T08:15:59.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[30086]: 5.b scrub starts 2024-02-05T08:15:59.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[30086]: 4.14 scrub starts 2024-02-05T08:15:59.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[30086]: 5.b scrub ok 2024-02-05T08:15:59.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[30086]: 4.14 scrub ok 2024-02-05T08:15:59.120 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[39278]: 3.a scrub starts 2024-02-05T08:15:59.120 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[39278]: 3.a scrub ok 2024-02-05T08:15:59.120 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[39278]: pgmap v500: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 640 B/s rd, 57 MiB/s wr, 914 op/s 2024-02-05T08:15:59.120 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[39278]: 5.b scrub starts 2024-02-05T08:15:59.120 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[39278]: 4.14 scrub starts 2024-02-05T08:15:59.120 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[39278]: 5.b scrub ok 2024-02-05T08:15:59.120 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:58 smithi134 ceph-mon[39278]: 4.14 scrub ok 2024-02-05T08:15:59.121 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2024-02-05T08:15:55.717238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4539226460,"num_objects":69452,"num_object_clones":0,"num_object_copies":208356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69452,"num_whiteouts":0,"num_read":2016,"num_read_kb":1862,"num_write":70155,"num_write_kb":4434717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":2957685,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69518,"ondisk_log_size":69518,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15640144,"kb_used_data":13199244,"kb_used_omap":10,"kb_used_meta":2440821,"kb_avail":640575920,"statfs":{"total":671965249536,"available":655949742080,"internally_reserved":0,"allocated":13516025856,"data_stored":13512978105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":2499400810},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,3,12,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":428212224,"num_objects":6534,"num_object_clones":0,"num_object_copies":19602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6534,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6534,"num_write_kb":418176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001612"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":426,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.741524+0000","last_change":"2024-02-05T08:15:16.927873+0000","last_active":"2024-02-05T08:15:49.741524+0000","last_peered":"2024-02-05T08:15:49.741524+0000","last_clean":"2024-02-05T08:15:49.741524+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:15:49.741524+0000","last_undegraded":"2024-02-05T08:15:49.741524+0000","last_fullsized":"2024-02-05T08:15:49.741524+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:16.927723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:16.927723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:39.613027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":433,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.763915+0000","last_change":"2024-02-05T08:15:21.824643+0000","last_active":"2024-02-05T08:15:52.763915+0000","last_peered":"2024-02-05T08:15:52.763915+0000","last_clean":"2024-02-05T08:15:52.763915+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:15:52.763915+0000","last_undegraded":"2024-02-05T08:15:52.763915+0000","last_fullsized":"2024-02-05T08:15:52.763915+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:21.824531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:21.824531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:31.555157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":432,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.768895+0000","last_change":"2024-02-05T08:14:10.178234+0000","last_active":"2024-02-05T08:15:52.768895+0000","last_peered":"2024-02-05T08:15:52.768895+0000","last_clean":"2024-02-05T08:15:52.768895+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:15:52.768895+0000","last_undegraded":"2024-02-05T08:15:52.768895+0000","last_fullsized":"2024-02-05T08:15:52.768895+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:10.178136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:10.178136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"236'19","reported_seq":487,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.741245+0000","last_change":"2024-02-05T08:14:03.970129+0000","last_active":"2024-02-05T08:15:49.741245+0000","last_peered":"2024-02-05T08:15:49.741245+0000","last_clean":"2024-02-05T08:15:49.741245+0000","last_became_active":"2024-02-05T08:12:29.675863+0000","last_became_peered":"2024-02-05T08:12:29.675863+0000","last_unstale":"2024-02-05T08:15:49.741245+0000","last_undegraded":"2024-02-05T08:15:49.741245+0000","last_fullsized":"2024-02-05T08:15:49.741245+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'13","last_scrub_stamp":"2024-02-05T08:14:03.970014+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:14:03.970014+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":421,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.750389+0000","last_change":"2024-02-05T08:15:22.548444+0000","last_active":"2024-02-05T08:15:49.750389+0000","last_peered":"2024-02-05T08:15:49.750389+0000","last_clean":"2024-02-05T08:15:49.750389+0000","last_became_active":"2024-02-05T08:12:41.722414+0000","last_became_peered":"2024-02-05T08:12:41.722414+0000","last_unstale":"2024-02-05T08:15:49.750389+0000","last_undegraded":"2024-02-05T08:15:49.750389+0000","last_fullsized":"2024-02-05T08:15:49.750389+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.548159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:38.794097+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":434,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.741446+0000","last_change":"2024-02-05T08:14:13.968249+0000","last_active":"2024-02-05T08:15:49.741446+0000","last_peered":"2024-02-05T08:15:49.741446+0000","last_clean":"2024-02-05T08:15:49.741446+0000","last_became_active":"2024-02-05T08:12:29.676482+0000","last_became_peered":"2024-02-05T08:12:29.676482+0000","last_unstale":"2024-02-05T08:15:49.741446+0000","last_undegraded":"2024-02-05T08:15:49.741446+0000","last_fullsized":"2024-02-05T08:15:49.741446+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:13.968037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:14:13.968037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":156,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.740250+0000","last_change":"2024-02-05T08:15:29.985455+0000","last_active":"2024-02-05T08:15:49.740250+0000","last_peered":"2024-02-05T08:15:49.740250+0000","last_clean":"2024-02-05T08:15:49.740250+0000","last_became_active":"2024-02-05T08:12:41.727450+0000","last_became_peered":"2024-02-05T08:12:41.727450+0000","last_unstale":"2024-02-05T08:15:49.740250+0000","last_undegraded":"2024-02-05T08:15:49.740250+0000","last_fullsized":"2024-02-05T08:15:49.740250+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:15:29.985333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:55.457550+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"236'34","reported_seq":339,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.767299+0000","last_change":"2024-02-05T08:15:24.771294+0000","last_active":"2024-02-05T08:15:52.767299+0000","last_peered":"2024-02-05T08:15:52.767299+0000","last_clean":"2024-02-05T08:15:52.767299+0000","last_became_active":"2024-02-05T08:12:41.745917+0000","last_became_peered":"2024-02-05T08:12:41.745917+0000","last_unstale":"2024-02-05T08:15:52.767299+0000","last_undegraded":"2024-02-05T08:15:52.767299+0000","last_fullsized":"2024-02-05T08:15:52.767299+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:15:24.771146+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:15:24.771146+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:25.971833+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":419,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766480+0000","last_change":"2024-02-05T08:14:14.237605+0000","last_active":"2024-02-05T08:15:52.766480+0000","last_peered":"2024-02-05T08:15:52.766480+0000","last_clean":"2024-02-05T08:15:52.766480+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:15:52.766480+0000","last_undegraded":"2024-02-05T08:15:52.766480+0000","last_fullsized":"2024-02-05T08:15:52.766480+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:14.237096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:14.237096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666853+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":432,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.769850+0000","last_change":"2024-02-05T08:15:22.941148+0000","last_active":"2024-02-05T08:15:52.769850+0000","last_peered":"2024-02-05T08:15:52.769850+0000","last_clean":"2024-02-05T08:15:52.769850+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:15:52.769850+0000","last_undegraded":"2024-02-05T08:15:52.769850+0000","last_fullsized":"2024-02-05T08:15:52.769850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.940396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.940396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:47.298258+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":452,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.767907+0000","last_change":"2024-02-05T08:14:27.200067+0000","last_active":"2024-02-05T08:15:52.767907+0000","last_peered":"2024-02-05T08:15:52.767907+0000","last_clean":"2024-02-05T08:15:52.767907+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:15:52.767907+0000","last_undegraded":"2024-02-05T08:15:52.767907+0000","last_fullsized":"2024-02-05T08:15:52.767907+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:27.199962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:27.199962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.005928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"236'15","reported_seq":473,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.741764+0000","last_change":"2024-02-05T08:14:16.970924+0000","last_active":"2024-02-05T08:15:49.741764+0000","last_peered":"2024-02-05T08:15:49.741764+0000","last_clean":"2024-02-05T08:15:49.741764+0000","last_became_active":"2024-02-05T08:12:41.731452+0000","last_became_peered":"2024-02-05T08:12:41.731452+0000","last_unstale":"2024-02-05T08:15:49.741764+0000","last_undegraded":"2024-02-05T08:15:49.741764+0000","last_fullsized":"2024-02-05T08:15:49.741764+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:14:16.970843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:16.970843+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":426,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766739+0000","last_change":"2024-02-05T08:12:41.722105+0000","last_active":"2024-02-05T08:15:52.766739+0000","last_peered":"2024-02-05T08:15:52.766739+0000","last_clean":"2024-02-05T08:15:52.766739+0000","last_became_active":"2024-02-05T08:12:41.721611+0000","last_became_peered":"2024-02-05T08:12:41.721611+0000","last_unstale":"2024-02-05T08:15:52.766739+0000","last_undegraded":"2024-02-05T08:15:52.766739+0000","last_fullsized":"2024-02-05T08:15:52.766739+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:14.301656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:14.301656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.005726+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":431,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.746735+0000","last_change":"2024-02-05T08:15:18.440781+0000","last_active":"2024-02-05T08:15:49.746735+0000","last_peered":"2024-02-05T08:15:49.746735+0000","last_clean":"2024-02-05T08:15:49.746735+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:15:49.746735+0000","last_undegraded":"2024-02-05T08:15:49.746735+0000","last_fullsized":"2024-02-05T08:15:49.746735+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:18.440664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:18.440664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:34.989634+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":440,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.767250+0000","last_change":"2024-02-05T08:15:20.904492+0000","last_active":"2024-02-05T08:15:52.767250+0000","last_peered":"2024-02-05T08:15:52.767250+0000","last_clean":"2024-02-05T08:15:52.767250+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:15:52.767250+0000","last_undegraded":"2024-02-05T08:15:52.767250+0000","last_fullsized":"2024-02-05T08:15:52.767250+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:20.904369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:15:20.904369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:21.716191+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":451,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.741792+0000","last_change":"2024-02-05T08:14:18.923493+0000","last_active":"2024-02-05T08:15:49.741792+0000","last_peered":"2024-02-05T08:15:49.741792+0000","last_clean":"2024-02-05T08:15:49.741792+0000","last_became_active":"2024-02-05T08:12:41.730485+0000","last_became_peered":"2024-02-05T08:12:41.730485+0000","last_unstale":"2024-02-05T08:15:49.741792+0000","last_undegraded":"2024-02-05T08:15:49.741792+0000","last_fullsized":"2024-02-05T08:15:49.741792+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-02-05T08:14:18.923276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:18.923276+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":439,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.742988+0000","last_change":"2024-02-05T08:15:13.039454+0000","last_active":"2024-02-05T08:15:49.742988+0000","last_peered":"2024-02-05T08:15:49.742988+0000","last_clean":"2024-02-05T08:15:49.742988+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:15:49.742988+0000","last_undegraded":"2024-02-05T08:15:49.742988+0000","last_fullsized":"2024-02-05T08:15:49.742988+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:13.039321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:15:13.039321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:40.716512+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":425,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.765501+0000","last_change":"2024-02-05T08:15:11.836173+0000","last_active":"2024-02-05T08:15:52.765501+0000","last_peered":"2024-02-05T08:15:52.765501+0000","last_clean":"2024-02-05T08:15:52.765501+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:15:52.765501+0000","last_undegraded":"2024-02-05T08:15:52.765501+0000","last_fullsized":"2024-02-05T08:15:52.765501+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:11.836098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:15:11.836098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:20.798265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":431,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766647+0000","last_change":"2024-02-05T08:15:22.341597+0000","last_active":"2024-02-05T08:15:52.766647+0000","last_peered":"2024-02-05T08:15:52.766647+0000","last_clean":"2024-02-05T08:15:52.766647+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:15:52.766647+0000","last_undegraded":"2024-02-05T08:15:52.766647+0000","last_fullsized":"2024-02-05T08:15:52.766647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.341491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.341491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:41.569521+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"236'34","reported_seq":516,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766287+0000","last_change":"2024-02-05T08:14:31.867150+0000","last_active":"2024-02-05T08:15:52.766287+0000","last_peered":"2024-02-05T08:15:52.766287+0000","last_clean":"2024-02-05T08:15:52.766287+0000","last_became_active":"2024-02-05T08:12:41.727027+0000","last_became_peered":"2024-02-05T08:12:41.727027+0000","last_unstale":"2024-02-05T08:15:52.766287+0000","last_undegraded":"2024-02-05T08:15:52.766287+0000","last_fullsized":"2024-02-05T08:15:52.766287+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'22","last_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.867008+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":419,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.739144+0000","last_change":"2024-02-05T08:14:33.244556+0000","last_active":"2024-02-05T08:15:49.739144+0000","last_peered":"2024-02-05T08:15:49.739144+0000","last_clean":"2024-02-05T08:15:49.739144+0000","last_became_active":"2024-02-05T08:12:41.713564+0000","last_became_peered":"2024-02-05T08:12:41.713564+0000","last_unstale":"2024-02-05T08:15:49.739144+0000","last_undegraded":"2024-02-05T08:15:49.739144+0000","last_fullsized":"2024-02-05T08:15:49.739144+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:33.244471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:33.244471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":436,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.742502+0000","last_change":"2024-02-05T08:15:25.922691+0000","last_active":"2024-02-05T08:15:49.742502+0000","last_peered":"2024-02-05T08:15:49.742502+0000","last_clean":"2024-02-05T08:15:49.742502+0000","last_became_active":"2024-02-05T08:12:41.735240+0000","last_became_peered":"2024-02-05T08:12:41.735240+0000","last_unstale":"2024-02-05T08:15:49.742502+0000","last_undegraded":"2024-02-05T08:15:49.742502+0000","last_fullsized":"2024-02-05T08:15:49.742502+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.922139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.922139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:46.247310+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":430,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.741847+0000","last_change":"2024-02-05T08:15:19.927561+0000","last_active":"2024-02-05T08:15:49.741847+0000","last_peered":"2024-02-05T08:15:49.741847+0000","last_clean":"2024-02-05T08:15:49.741847+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:15:49.741847+0000","last_undegraded":"2024-02-05T08:15:49.741847+0000","last_fullsized":"2024-02-05T08:15:49.741847+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:19.927454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:19.927454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:22.894101+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"236'23","reported_seq":252,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.764710+0000","last_change":"2024-02-05T08:14:09.718995+0000","last_active":"2024-02-05T08:15:52.764710+0000","last_peered":"2024-02-05T08:15:52.764710+0000","last_clean":"2024-02-05T08:15:52.764710+0000","last_became_active":"2024-02-05T08:12:41.716794+0000","last_became_peered":"2024-02-05T08:12:41.716794+0000","last_unstale":"2024-02-05T08:15:52.764710+0000","last_undegraded":"2024-02-05T08:15:52.764710+0000","last_fullsized":"2024-02-05T08:15:52.764710+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'17","last_scrub_stamp":"2024-02-05T08:14:09.718864+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:14:09.718864+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.849507+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":437,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.747933+0000","last_change":"2024-02-05T08:14:26.616349+0000","last_active":"2024-02-05T08:15:49.747933+0000","last_peered":"2024-02-05T08:15:49.747933+0000","last_clean":"2024-02-05T08:15:49.747933+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:15:49.747933+0000","last_undegraded":"2024-02-05T08:15:49.747933+0000","last_fullsized":"2024-02-05T08:15:49.747933+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:14:26.616223+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:14:26.616223+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276509+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":205,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.740186+0000","last_change":"2024-02-05T08:14:18.174978+0000","last_active":"2024-02-05T08:15:49.740186+0000","last_peered":"2024-02-05T08:15:49.740186+0000","last_clean":"2024-02-05T08:15:49.740186+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:15:49.740186+0000","last_undegraded":"2024-02-05T08:15:49.740186+0000","last_fullsized":"2024-02-05T08:15:49.740186+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:14:18.174860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:18.174860+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:55.170762+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":444,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.763825+0000","last_change":"2024-02-05T08:14:10.699857+0000","last_active":"2024-02-05T08:15:52.763825+0000","last_peered":"2024-02-05T08:15:52.763825+0000","last_clean":"2024-02-05T08:15:52.763825+0000","last_became_active":"2024-02-05T08:12:41.719010+0000","last_became_peered":"2024-02-05T08:12:41.719010+0000","last_unstale":"2024-02-05T08:15:52.763825+0000","last_undegraded":"2024-02-05T08:15:52.763825+0000","last_fullsized":"2024-02-05T08:15:52.763825+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:14:10.699732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:10.699732+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.849540+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"236'20","reported_seq":512,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.769139+0000","last_change":"2024-02-05T08:15:26.443674+0000","last_active":"2024-02-05T08:15:52.769139+0000","last_peered":"2024-02-05T08:15:52.769139+0000","last_clean":"2024-02-05T08:15:52.769139+0000","last_became_active":"2024-02-05T08:12:41.757718+0000","last_became_peered":"2024-02-05T08:12:41.757718+0000","last_unstale":"2024-02-05T08:15:52.769139+0000","last_undegraded":"2024-02-05T08:15:52.769139+0000","last_fullsized":"2024-02-05T08:15:52.769139+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:15:26.443406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:26.443406+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:30.531093+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":421,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.765561+0000","last_change":"2024-02-05T08:11:33.834287+0000","last_active":"2024-02-05T08:15:52.765561+0000","last_peered":"2024-02-05T08:15:52.765561+0000","last_clean":"2024-02-05T08:15:52.765561+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:15:52.765561+0000","last_undegraded":"2024-02-05T08:15:52.765561+0000","last_fullsized":"2024-02-05T08:15:52.765561+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:11:33.834034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:11:33.834034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.849406+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":424,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.769676+0000","last_change":"2024-02-05T08:14:32.232117+0000","last_active":"2024-02-05T08:15:52.769676+0000","last_peered":"2024-02-05T08:15:52.769676+0000","last_clean":"2024-02-05T08:15:52.769676+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:15:52.769676+0000","last_undegraded":"2024-02-05T08:15:52.769676+0000","last_fullsized":"2024-02-05T08:15:52.769676+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:32.231861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:32.231861+0000","last_clean_scrub_stamp":"2024-02-05T08:14:32.231861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666966+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":152,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766409+0000","last_change":"2024-02-05T08:14:31.199352+0000","last_active":"2024-02-05T08:15:52.766409+0000","last_peered":"2024-02-05T08:15:52.766409+0000","last_clean":"2024-02-05T08:15:52.766409+0000","last_became_active":"2024-02-05T08:12:41.727406+0000","last_became_peered":"2024-02-05T08:12:41.727406+0000","last_unstale":"2024-02-05T08:15:52.766409+0000","last_undegraded":"2024-02-05T08:15:52.766409+0000","last_fullsized":"2024-02-05T08:15:52.766409+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:31.199238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.199238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666939+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"236'19","reported_seq":184,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.739246+0000","last_change":"2024-02-05T08:14:26.230319+0000","last_active":"2024-02-05T08:15:49.739246+0000","last_peered":"2024-02-05T08:15:49.739246+0000","last_clean":"2024-02-05T08:15:49.739246+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:15:49.739246+0000","last_undegraded":"2024-02-05T08:15:49.739246+0000","last_fullsized":"2024-02-05T08:15:49.739246+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:14:26.230191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:26.230191+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":419,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.746874+0000","last_change":"2024-02-05T08:13:35.143784+0000","last_active":"2024-02-05T08:15:49.746874+0000","last_peered":"2024-02-05T08:15:49.746874+0000","last_clean":"2024-02-05T08:15:49.746874+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:15:49.746874+0000","last_undegraded":"2024-02-05T08:15:49.746874+0000","last_fullsized":"2024-02-05T08:15:49.746874+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:35.143602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:13:35.143602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276193+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":160,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.739281+0000","last_change":"2024-02-05T08:15:12.041843+0000","last_active":"2024-02-05T08:15:49.739281+0000","last_peered":"2024-02-05T08:15:49.739281+0000","last_clean":"2024-02-05T08:15:49.739281+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:15:49.739281+0000","last_undegraded":"2024-02-05T08:15:49.739281+0000","last_fullsized":"2024-02-05T08:15:49.739281+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:12.041759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:15:12.041759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:41.328162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":437,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.769288+0000","last_change":"2024-02-05T08:15:11.364628+0000","last_active":"2024-02-05T08:15:52.769288+0000","last_peered":"2024-02-05T08:15:52.769288+0000","last_clean":"2024-02-05T08:15:52.769288+0000","last_became_active":"2024-02-05T08:12:41.736991+0000","last_became_peered":"2024-02-05T08:12:41.736991+0000","last_unstale":"2024-02-05T08:15:52.769288+0000","last_undegraded":"2024-02-05T08:15:52.769288+0000","last_fullsized":"2024-02-05T08:15:52.769288+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:11.364552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:11.364552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:18.417672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":190,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.765019+0000","last_change":"2024-02-05T08:15:23.861142+0000","last_active":"2024-02-05T08:15:52.765019+0000","last_peered":"2024-02-05T08:15:52.765019+0000","last_clean":"2024-02-05T08:15:52.765019+0000","last_became_active":"2024-02-05T08:12:41.717459+0000","last_became_peered":"2024-02-05T08:12:41.717459+0000","last_unstale":"2024-02-05T08:15:52.765019+0000","last_undegraded":"2024-02-05T08:15:52.765019+0000","last_fullsized":"2024-02-05T08:15:52.765019+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:15:23.861011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:23.861011+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:40.974821+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":414,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766314+0000","last_change":"2024-02-05T08:12:41.738206+0000","last_active":"2024-02-05T08:15:52.766314+0000","last_peered":"2024-02-05T08:15:52.766314+0000","last_clean":"2024-02-05T08:15:52.766314+0000","last_became_active":"2024-02-05T08:12:41.737803+0000","last_became_peered":"2024-02-05T08:12:41.737803+0000","last_unstale":"2024-02-05T08:15:52.766314+0000","last_undegraded":"2024-02-05T08:15:52.766314+0000","last_fullsized":"2024-02-05T08:15:52.766314+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:33.610823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:33.610823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666706+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":443,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.746254+0000","last_change":"2024-02-05T08:13:19.251251+0000","last_active":"2024-02-05T08:15:49.746254+0000","last_peered":"2024-02-05T08:15:49.746254+0000","last_clean":"2024-02-05T08:15:49.746254+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:15:49.746254+0000","last_undegraded":"2024-02-05T08:15:49.746254+0000","last_fullsized":"2024-02-05T08:15:49.746254+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:19.251134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:13:19.251134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":440,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.768703+0000","last_change":"2024-02-05T08:13:10.189057+0000","last_active":"2024-02-05T08:15:52.768703+0000","last_peered":"2024-02-05T08:15:52.768703+0000","last_clean":"2024-02-05T08:15:52.768703+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:15:52.768703+0000","last_undegraded":"2024-02-05T08:15:52.768703+0000","last_fullsized":"2024-02-05T08:15:52.768703+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:10.188977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:13:10.188977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.005865+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"236'24","reported_seq":387,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.747793+0000","last_change":"2024-02-05T08:14:17.606299+0000","last_active":"2024-02-05T08:15:49.747793+0000","last_peered":"2024-02-05T08:15:49.747793+0000","last_clean":"2024-02-05T08:15:49.747793+0000","last_became_active":"2024-02-05T08:12:41.718651+0000","last_became_peered":"2024-02-05T08:12:41.718651+0000","last_unstale":"2024-02-05T08:15:49.747793+0000","last_undegraded":"2024-02-05T08:15:49.747793+0000","last_fullsized":"2024-02-05T08:15:49.747793+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'16","last_scrub_stamp":"2024-02-05T08:14:17.606161+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:14:17.606161+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276543+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":419,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766087+0000","last_change":"2024-02-05T08:14:23.281718+0000","last_active":"2024-02-05T08:15:52.766087+0000","last_peered":"2024-02-05T08:15:52.766087+0000","last_clean":"2024-02-05T08:15:52.766087+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:15:52.766087+0000","last_undegraded":"2024-02-05T08:15:52.766087+0000","last_fullsized":"2024-02-05T08:15:52.766087+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:23.281493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:23.281493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.646764+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":153,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766458+0000","last_change":"2024-02-05T08:14:09.229078+0000","last_active":"2024-02-05T08:15:52.766458+0000","last_peered":"2024-02-05T08:15:52.766458+0000","last_clean":"2024-02-05T08:15:52.766458+0000","last_became_active":"2024-02-05T08:12:41.757901+0000","last_became_peered":"2024-02-05T08:12:41.757901+0000","last_unstale":"2024-02-05T08:15:52.766458+0000","last_undegraded":"2024-02-05T08:15:52.766458+0000","last_fullsized":"2024-02-05T08:15:52.766458+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:09.228973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:09.228973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666825+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":445,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.751213+0000","last_change":"2024-02-05T08:15:10.525698+0000","last_active":"2024-02-05T08:15:49.751213+0000","last_peered":"2024-02-05T08:15:49.751213+0000","last_clean":"2024-02-05T08:15:49.751213+0000","last_became_active":"2024-02-05T08:12:41.734072+0000","last_became_peered":"2024-02-05T08:12:41.734072+0000","last_unstale":"2024-02-05T08:15:49.751213+0000","last_undegraded":"2024-02-05T08:15:49.751213+0000","last_fullsized":"2024-02-05T08:15:49.751213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:10.525625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:10.525625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:26.616052+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"236'15","reported_seq":96,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766753+0000","last_change":"2024-02-05T08:14:24.999924+0000","last_active":"2024-02-05T08:15:52.766753+0000","last_peered":"2024-02-05T08:15:52.766753+0000","last_clean":"2024-02-05T08:15:52.766753+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:15:52.766753+0000","last_undegraded":"2024-02-05T08:15:52.766753+0000","last_fullsized":"2024-02-05T08:15:52.766753+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:14:21.660270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:21.660270+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.005899+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":144,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.765940+0000","last_change":"2024-02-05T08:12:41.742137+0000","last_active":"2024-02-05T08:15:52.765940+0000","last_peered":"2024-02-05T08:15:52.765940+0000","last_clean":"2024-02-05T08:15:52.765940+0000","last_became_active":"2024-02-05T08:12:41.737246+0000","last_became_peered":"2024-02-05T08:12:41.737246+0000","last_unstale":"2024-02-05T08:15:52.765940+0000","last_undegraded":"2024-02-05T08:15:52.765940+0000","last_fullsized":"2024-02-05T08:15:52.765940+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.507800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.507800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":431,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.764299+0000","last_change":"2024-02-05T08:12:41.717743+0000","last_active":"2024-02-05T08:15:52.764299+0000","last_peered":"2024-02-05T08:15:52.764299+0000","last_clean":"2024-02-05T08:15:52.764299+0000","last_became_active":"2024-02-05T08:12:41.717285+0000","last_became_peered":"2024-02-05T08:12:41.717285+0000","last_unstale":"2024-02-05T08:15:52.764299+0000","last_undegraded":"2024-02-05T08:15:52.764299+0000","last_fullsized":"2024-02-05T08:15:52.764299+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:12.952558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:12.952558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.849456+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":450,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.742262+0000","last_change":"2024-02-05T08:14:30.022191+0000","last_active":"2024-02-05T08:15:49.742262+0000","last_peered":"2024-02-05T08:15:49.742262+0000","last_clean":"2024-02-05T08:15:49.742262+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:15:49.742262+0000","last_undegraded":"2024-02-05T08:15:49.742262+0000","last_fullsized":"2024-02-05T08:15:49.742262+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:14:30.022077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"249'24","reported_seq":506,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.741932+0000","last_change":"2024-02-05T08:15:24.970945+0000","last_active":"2024-02-05T08:15:49.741932+0000","last_peered":"2024-02-05T08:15:49.741932+0000","last_clean":"2024-02-05T08:15:49.741932+0000","last_became_active":"2024-02-05T08:12:29.675657+0000","last_became_peered":"2024-02-05T08:12:29.675657+0000","last_unstale":"2024-02-05T08:15:49.741932+0000","last_undegraded":"2024-02-05T08:15:49.741932+0000","last_fullsized":"2024-02-05T08:15:49.741932+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:15:24.970769+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:33.443975+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":425,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.768837+0000","last_change":"2024-02-05T08:15:25.472795+0000","last_active":"2024-02-05T08:15:52.768837+0000","last_peered":"2024-02-05T08:15:52.768837+0000","last_clean":"2024-02-05T08:15:52.768837+0000","last_became_active":"2024-02-05T08:12:41.736845+0000","last_became_peered":"2024-02-05T08:12:41.736845+0000","last_unstale":"2024-02-05T08:15:52.768837+0000","last_undegraded":"2024-02-05T08:15:52.768837+0000","last_fullsized":"2024-02-05T08:15:52.768837+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.472655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.472655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:46.290853+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":575,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.768002+0000","last_change":"2024-02-05T08:15:08.943424+0000","last_active":"2024-02-05T08:15:52.768002+0000","last_peered":"2024-02-05T08:15:52.768002+0000","last_clean":"2024-02-05T08:15:52.768002+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:15:52.768002+0000","last_undegraded":"2024-02-05T08:15:52.768002+0000","last_fullsized":"2024-02-05T08:15:52.768002+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:15:08.943297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:08.943297+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:10.217549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":439,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.750478+0000","last_change":"2024-02-05T08:14:10.570218+0000","last_active":"2024-02-05T08:15:49.750478+0000","last_peered":"2024-02-05T08:15:49.750478+0000","last_clean":"2024-02-05T08:15:49.750478+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:15:49.750478+0000","last_undegraded":"2024-02-05T08:15:49.750478+0000","last_fullsized":"2024-02-05T08:15:49.750478+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:10.570113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:10.570113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276330+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":329,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.764758+0000","last_change":"2024-02-05T08:12:41.781948+0000","last_active":"2024-02-05T08:15:52.764758+0000","last_peered":"2024-02-05T08:15:52.764758+0000","last_clean":"2024-02-05T08:15:52.764758+0000","last_became_active":"2024-02-05T08:12:41.726861+0000","last_became_peered":"2024-02-05T08:12:41.726861+0000","last_unstale":"2024-02-05T08:15:52.764758+0000","last_undegraded":"2024-02-05T08:15:52.764758+0000","last_fullsized":"2024-02-05T08:15:52.764758+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'15","last_scrub_stamp":"2024-02-05T08:11:30.787149+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:11:30.787149+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.849359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":421,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.741414+0000","last_change":"2024-02-05T08:15:07.991636+0000","last_active":"2024-02-05T08:15:49.741414+0000","last_peered":"2024-02-05T08:15:49.741414+0000","last_clean":"2024-02-05T08:15:49.741414+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:15:49.741414+0000","last_undegraded":"2024-02-05T08:15:49.741414+0000","last_fullsized":"2024-02-05T08:15:49.741414+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:07.991536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:07.991536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:09.960940+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"194'9","reported_seq":227,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.747879+0000","last_change":"2024-02-05T08:15:27.464031+0000","last_active":"2024-02-05T08:15:49.747879+0000","last_peered":"2024-02-05T08:15:49.747879+0000","last_clean":"2024-02-05T08:15:49.747879+0000","last_became_active":"2024-02-05T08:12:41.728687+0000","last_became_peered":"2024-02-05T08:12:41.728687+0000","last_unstale":"2024-02-05T08:15:49.747879+0000","last_undegraded":"2024-02-05T08:15:49.747879+0000","last_fullsized":"2024-02-05T08:15:49.747879+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'9","last_scrub_stamp":"2024-02-05T08:15:27.463918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:27.463918+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:31.994300+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":437,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.765452+0000","last_change":"2024-02-05T08:15:23.494414+0000","last_active":"2024-02-05T08:15:52.765452+0000","last_peered":"2024-02-05T08:15:52.765452+0000","last_clean":"2024-02-05T08:15:52.765452+0000","last_became_active":"2024-02-05T08:12:41.723383+0000","last_became_peered":"2024-02-05T08:12:41.723383+0000","last_unstale":"2024-02-05T08:15:52.765452+0000","last_undegraded":"2024-02-05T08:15:52.765452+0000","last_fullsized":"2024-02-05T08:15:52.765452+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:23.494299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:15:23.494299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:33.179614+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":477,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.768208+0000","last_change":"2024-02-05T08:14:35.244918+0000","last_active":"2024-02-05T08:15:52.768208+0000","last_peered":"2024-02-05T08:15:52.768208+0000","last_clean":"2024-02-05T08:15:52.768208+0000","last_became_active":"2024-02-05T08:12:41.723526+0000","last_became_peered":"2024-02-05T08:12:41.723526+0000","last_unstale":"2024-02-05T08:15:52.768208+0000","last_undegraded":"2024-02-05T08:15:52.768208+0000","last_fullsized":"2024-02-05T08:15:52.768208+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:14:35.244359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:35.244359+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.006145+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":436,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766597+0000","last_change":"2024-02-05T08:15:17.707705+0000","last_active":"2024-02-05T08:15:52.766597+0000","last_peered":"2024-02-05T08:15:52.766597+0000","last_clean":"2024-02-05T08:15:52.766597+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:15:52.766597+0000","last_undegraded":"2024-02-05T08:15:52.766597+0000","last_fullsized":"2024-02-05T08:15:52.766597+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:17.707565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:17.707565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:29.022820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":154,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.770014+0000","last_change":"2024-02-05T08:14:05.758633+0000","last_active":"2024-02-05T08:15:52.770014+0000","last_peered":"2024-02-05T08:15:52.770014+0000","last_clean":"2024-02-05T08:15:52.770014+0000","last_became_active":"2024-02-05T08:12:41.721852+0000","last_became_peered":"2024-02-05T08:12:41.721852+0000","last_unstale":"2024-02-05T08:15:52.770014+0000","last_undegraded":"2024-02-05T08:15:52.770014+0000","last_fullsized":"2024-02-05T08:15:52.770014+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:05.758519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:05.758519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.849522+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":451,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.747582+0000","last_change":"2024-02-05T08:15:25.470332+0000","last_active":"2024-02-05T08:15:49.747582+0000","last_peered":"2024-02-05T08:15:49.747582+0000","last_clean":"2024-02-05T08:15:49.747582+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:15:49.747582+0000","last_undegraded":"2024-02-05T08:15:49.747582+0000","last_fullsized":"2024-02-05T08:15:49.747582+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.470117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:40.542227+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"236'15","reported_seq":468,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.769497+0000","last_change":"2024-02-05T08:14:29.189057+0000","last_active":"2024-02-05T08:15:52.769497+0000","last_peered":"2024-02-05T08:15:52.769497+0000","last_clean":"2024-02-05T08:15:52.769497+0000","last_became_active":"2024-02-05T08:12:41.736414+0000","last_became_peered":"2024-02-05T08:12:41.736414+0000","last_unstale":"2024-02-05T08:15:52.769497+0000","last_undegraded":"2024-02-05T08:15:52.769497+0000","last_fullsized":"2024-02-05T08:15:52.769497+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:14:29.188820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:29.188820+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:56.706856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":439,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.747445+0000","last_change":"2024-02-05T08:13:34.175520+0000","last_active":"2024-02-05T08:15:49.747445+0000","last_peered":"2024-02-05T08:15:49.747445+0000","last_clean":"2024-02-05T08:15:49.747445+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:15:49.747445+0000","last_undegraded":"2024-02-05T08:15:49.747445+0000","last_fullsized":"2024-02-05T08:15:49.747445+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:34.174842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:13:34.174842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276251+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":430,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.768127+0000","last_change":"2024-02-05T08:12:41.721999+0000","last_active":"2024-02-05T08:15:52.768127+0000","last_peered":"2024-02-05T08:15:52.768127+0000","last_clean":"2024-02-05T08:15:52.768127+0000","last_became_active":"2024-02-05T08:12:41.721486+0000","last_became_peered":"2024-02-05T08:12:41.721486+0000","last_unstale":"2024-02-05T08:15:52.768127+0000","last_undegraded":"2024-02-05T08:15:52.768127+0000","last_fullsized":"2024-02-05T08:15:52.768127+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.242229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.242229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.005687+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":160,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.738291+0000","last_change":"2024-02-05T08:14:17.208565+0000","last_active":"2024-02-05T08:15:49.738291+0000","last_peered":"2024-02-05T08:15:49.738291+0000","last_clean":"2024-02-05T08:15:49.738291+0000","last_became_active":"2024-02-05T08:12:29.684444+0000","last_became_peered":"2024-02-05T08:12:29.684444+0000","last_unstale":"2024-02-05T08:15:49.738291+0000","last_undegraded":"2024-02-05T08:15:49.738291+0000","last_fullsized":"2024-02-05T08:15:49.738291+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:17.208445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:17.208445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'15","reported_seq":301,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.746947+0000","last_change":"2024-02-05T08:15:23.404290+0000","last_active":"2024-02-05T08:15:49.746947+0000","last_peered":"2024-02-05T08:15:49.746947+0000","last_clean":"2024-02-05T08:15:49.746947+0000","last_became_active":"2024-02-05T08:12:41.732369+0000","last_became_peered":"2024-02-05T08:12:41.732369+0000","last_unstale":"2024-02-05T08:15:49.746947+0000","last_undegraded":"2024-02-05T08:15:49.746947+0000","last_fullsized":"2024-02-05T08:15:49.746947+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:15:23.404196+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:15:23.404196+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:25.374530+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"260'4327","reported_seq":4379,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:53.519695+0000","last_change":"2024-02-05T08:14:57.766906+0000","last_active":"2024-02-05T08:15:53.519695+0000","last_peered":"2024-02-05T08:15:53.519695+0000","last_clean":"2024-02-05T08:15:53.519695+0000","last_became_active":"2024-02-05T08:14:57.766609+0000","last_became_peered":"2024-02-05T08:14:57.766609+0000","last_unstale":"2024-02-05T08:15:53.519695+0000","last_undegraded":"2024-02-05T08:15:53.519695+0000","last_fullsized":"2024-02-05T08:15:53.519695+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4327,"log_dups_size":0,"ondisk_log_size":4327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:57.478302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283574272,"num_objects":4327,"num_object_clones":0,"num_object_copies":12981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4327,"num_write_kb":276928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":441,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.767127+0000","last_change":"2024-02-05T08:14:28.178983+0000","last_active":"2024-02-05T08:15:52.767127+0000","last_peered":"2024-02-05T08:15:52.767127+0000","last_clean":"2024-02-05T08:15:52.767127+0000","last_became_active":"2024-02-05T08:12:29.679849+0000","last_became_peered":"2024-02-05T08:12:29.679849+0000","last_unstale":"2024-02-05T08:15:52.767127+0000","last_undegraded":"2024-02-05T08:15:52.767127+0000","last_fullsized":"2024-02-05T08:15:52.767127+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:28.178770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:28.178770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.006108+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":451,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.746857+0000","last_change":"2024-02-05T08:14:22.627522+0000","last_active":"2024-02-05T08:15:49.746857+0000","last_peered":"2024-02-05T08:15:49.746857+0000","last_clean":"2024-02-05T08:15:49.746857+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:15:49.746857+0000","last_undegraded":"2024-02-05T08:15:49.746857+0000","last_fullsized":"2024-02-05T08:15:49.746857+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:14:22.627289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:22.627289+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276399+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"236'27","reported_seq":516,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.767476+0000","last_change":"2024-02-05T08:12:29.719785+0000","last_active":"2024-02-05T08:15:52.767476+0000","last_peered":"2024-02-05T08:15:52.767476+0000","last_clean":"2024-02-05T08:15:52.767476+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:15:52.767476+0000","last_undegraded":"2024-02-05T08:15:52.767476+0000","last_fullsized":"2024-02-05T08:15:52.767476+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-02-05T08:12:26.319958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.319958+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.005821+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"190'10","reported_seq":595,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.747935+0000","last_change":"2024-02-05T08:14:06.549828+0000","last_active":"2024-02-05T08:15:49.747935+0000","last_peered":"2024-02-05T08:15:49.747935+0000","last_clean":"2024-02-05T08:15:49.747935+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:15:49.747935+0000","last_undegraded":"2024-02-05T08:15:49.747935+0000","last_fullsized":"2024-02-05T08:15:49.747935+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:14:06.549691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:06.549691+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276474+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"260'4464","reported_seq":4516,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:53.260551+0000","last_change":"2024-02-05T08:14:57.758422+0000","last_active":"2024-02-05T08:15:53.260551+0000","last_peered":"2024-02-05T08:15:53.260551+0000","last_clean":"2024-02-05T08:15:53.260551+0000","last_became_active":"2024-02-05T08:14:57.758070+0000","last_became_peered":"2024-02-05T08:14:57.758070+0000","last_unstale":"2024-02-05T08:15:53.260551+0000","last_undegraded":"2024-02-05T08:15:53.260551+0000","last_fullsized":"2024-02-05T08:15:53.260551+0000","mapping_epoch":239,"log_start":"246'300","ondisk_log_start":"246'300","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4164,"log_dups_size":0,"ondisk_log_size":4164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:22.195699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292552704,"num_objects":4464,"num_object_clones":0,"num_object_copies":13392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4464,"num_write_kb":285696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":151,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.750552+0000","last_change":"2024-02-05T08:14:31.779172+0000","last_active":"2024-02-05T08:15:49.750552+0000","last_peered":"2024-02-05T08:15:49.750552+0000","last_clean":"2024-02-05T08:15:49.750552+0000","last_became_active":"2024-02-05T08:12:41.722853+0000","last_became_peered":"2024-02-05T08:12:41.722853+0000","last_unstale":"2024-02-05T08:15:49.750552+0000","last_undegraded":"2024-02-05T08:15:49.750552+0000","last_fullsized":"2024-02-05T08:15:49.750552+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:31.779046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.779046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276604+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":439,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.749074+0000","last_change":"2024-02-05T08:14:14.597997+0000","last_active":"2024-02-05T08:15:49.749074+0000","last_peered":"2024-02-05T08:15:49.749074+0000","last_clean":"2024-02-05T08:15:49.749074+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:15:49.749074+0000","last_undegraded":"2024-02-05T08:15:49.749074+0000","last_fullsized":"2024-02-05T08:15:49.749074+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:14.597867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:14:14.597867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276575+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":477,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.742904+0000","last_change":"2024-02-05T08:14:35.038653+0000","last_active":"2024-02-05T08:15:49.742904+0000","last_peered":"2024-02-05T08:15:49.742904+0000","last_clean":"2024-02-05T08:15:49.742904+0000","last_became_active":"2024-02-05T08:12:41.728925+0000","last_became_peered":"2024-02-05T08:12:41.728925+0000","last_unstale":"2024-02-05T08:15:49.742904+0000","last_undegraded":"2024-02-05T08:15:49.742904+0000","last_fullsized":"2024-02-05T08:15:49.742904+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:14:35.038563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:35.038563+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":441,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.765890+0000","last_change":"2024-02-05T08:15:19.765825+0000","last_active":"2024-02-05T08:15:52.765890+0000","last_peered":"2024-02-05T08:15:52.765890+0000","last_clean":"2024-02-05T08:15:52.765890+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:15:52.765890+0000","last_undegraded":"2024-02-05T08:15:52.765890+0000","last_fullsized":"2024-02-05T08:15:52.765890+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:15:19.765714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:25.591609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"260'4226","reported_seq":4278,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.843748+0000","last_change":"2024-02-05T08:14:57.763466+0000","last_active":"2024-02-05T08:15:52.843748+0000","last_peered":"2024-02-05T08:15:52.843748+0000","last_clean":"2024-02-05T08:15:52.843748+0000","last_became_active":"2024-02-05T08:14:57.763165+0000","last_became_peered":"2024-02-05T08:14:57.763165+0000","last_unstale":"2024-02-05T08:15:52.843748+0000","last_undegraded":"2024-02-05T08:15:52.843748+0000","last_fullsized":"2024-02-05T08:15:52.843748+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4226,"log_dups_size":0,"ondisk_log_size":4226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:25.940494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276955136,"num_objects":4226,"num_object_clones":0,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4226,"num_write_kb":270464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":434,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.745773+0000","last_change":"2024-02-05T08:15:13.442673+0000","last_active":"2024-02-05T08:15:49.745773+0000","last_peered":"2024-02-05T08:15:49.745773+0000","last_clean":"2024-02-05T08:15:49.745773+0000","last_became_active":"2024-02-05T08:12:41.724830+0000","last_became_peered":"2024-02-05T08:12:41.724830+0000","last_unstale":"2024-02-05T08:15:49.745773+0000","last_undegraded":"2024-02-05T08:15:49.745773+0000","last_fullsized":"2024-02-05T08:15:49.745773+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:13.442573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:13.442573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:28.200585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":432,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.767497+0000","last_change":"2024-02-05T08:14:33.899458+0000","last_active":"2024-02-05T08:15:52.767497+0000","last_peered":"2024-02-05T08:15:52.767497+0000","last_clean":"2024-02-05T08:15:52.767497+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:15:52.767497+0000","last_undegraded":"2024-02-05T08:15:52.767497+0000","last_fullsized":"2024-02-05T08:15:52.767497+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:33.896717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:14:33.896717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.799123+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":452,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.750959+0000","last_change":"2024-02-05T08:14:12.608866+0000","last_active":"2024-02-05T08:15:49.750959+0000","last_peered":"2024-02-05T08:15:49.750959+0000","last_clean":"2024-02-05T08:15:49.750959+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:15:49.750959+0000","last_undegraded":"2024-02-05T08:15:49.750959+0000","last_fullsized":"2024-02-05T08:15:49.750959+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:12.608763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:12.608763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276277+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"236'35","reported_seq":204,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.749476+0000","last_change":"2024-02-05T08:14:31.191982+0000","last_active":"2024-02-05T08:15:49.749476+0000","last_peered":"2024-02-05T08:15:49.749476+0000","last_clean":"2024-02-05T08:15:49.749476+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:15:49.749476+0000","last_undegraded":"2024-02-05T08:15:49.749476+0000","last_fullsized":"2024-02-05T08:15:49.749476+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:14:31.191827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.191827+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.315305+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"260'4303","reported_seq":4355,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.836828+0000","last_change":"2024-02-05T08:14:57.761472+0000","last_active":"2024-02-05T08:15:52.836828+0000","last_peered":"2024-02-05T08:15:52.836828+0000","last_clean":"2024-02-05T08:15:52.836828+0000","last_became_active":"2024-02-05T08:14:57.761156+0000","last_became_peered":"2024-02-05T08:14:57.761156+0000","last_unstale":"2024-02-05T08:15:52.836828+0000","last_undegraded":"2024-02-05T08:15:52.836828+0000","last_fullsized":"2024-02-05T08:15:52.836828+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4303,"log_dups_size":0,"ondisk_log_size":4303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:13.756234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282001408,"num_objects":4303,"num_object_clones":0,"num_object_copies":12909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4303,"num_write_kb":275392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":420,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766141+0000","last_change":"2024-02-05T08:14:16.274043+0000","last_active":"2024-02-05T08:15:52.766141+0000","last_peered":"2024-02-05T08:15:52.766141+0000","last_clean":"2024-02-05T08:15:52.766141+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:15:52.766141+0000","last_undegraded":"2024-02-05T08:15:52.766141+0000","last_fullsized":"2024-02-05T08:15:52.766141+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:16.273936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:16.273936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666909+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":447,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.765819+0000","last_change":"2024-02-05T08:15:25.793425+0000","last_active":"2024-02-05T08:15:52.765819+0000","last_peered":"2024-02-05T08:15:52.765819+0000","last_clean":"2024-02-05T08:15:52.765819+0000","last_became_active":"2024-02-05T08:12:41.743888+0000","last_became_peered":"2024-02-05T08:12:41.743888+0000","last_unstale":"2024-02-05T08:15:52.765819+0000","last_undegraded":"2024-02-05T08:15:52.765819+0000","last_fullsized":"2024-02-05T08:15:52.765819+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.793224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.793224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:34.555688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":450,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.767334+0000","last_change":"2024-02-05T08:12:41.719156+0000","last_active":"2024-02-05T08:15:52.767334+0000","last_peered":"2024-02-05T08:15:52.767334+0000","last_clean":"2024-02-05T08:15:52.767334+0000","last_became_active":"2024-02-05T08:12:41.718843+0000","last_became_peered":"2024-02-05T08:12:41.718843+0000","last_unstale":"2024-02-05T08:15:52.767334+0000","last_undegraded":"2024-02-05T08:15:52.767334+0000","last_fullsized":"2024-02-05T08:15:52.767334+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:18.291829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:18.291829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.005756+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"236'15","reported_seq":452,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.769416+0000","last_change":"2024-02-05T08:12:22.599613+0000","last_active":"2024-02-05T08:15:52.769416+0000","last_peered":"2024-02-05T08:15:52.769416+0000","last_clean":"2024-02-05T08:15:52.769416+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:15:52.769416+0000","last_undegraded":"2024-02-05T08:15:52.769416+0000","last_fullsized":"2024-02-05T08:15:52.769416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-02-05T08:12:22.599493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:22.599493+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666738+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"260'4452","reported_seq":4504,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:53.510834+0000","last_change":"2024-02-05T08:14:57.766128+0000","last_active":"2024-02-05T08:15:53.510834+0000","last_peered":"2024-02-05T08:15:53.510834+0000","last_clean":"2024-02-05T08:15:53.510834+0000","last_became_active":"2024-02-05T08:14:57.765694+0000","last_became_peered":"2024-02-05T08:14:57.765694+0000","last_unstale":"2024-02-05T08:15:53.510834+0000","last_undegraded":"2024-02-05T08:15:53.510834+0000","last_fullsized":"2024-02-05T08:15:53.510834+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4452,"log_dups_size":0,"ondisk_log_size":4452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:25.701081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291766272,"num_objects":4452,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4452,"num_write_kb":284928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":151,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.742032+0000","last_change":"2024-02-05T08:15:29.993055+0000","last_active":"2024-02-05T08:15:49.742032+0000","last_peered":"2024-02-05T08:15:49.742032+0000","last_clean":"2024-02-05T08:15:49.742032+0000","last_became_active":"2024-02-05T08:12:41.726776+0000","last_became_peered":"2024-02-05T08:12:41.726776+0000","last_unstale":"2024-02-05T08:15:49.742032+0000","last_undegraded":"2024-02-05T08:15:49.742032+0000","last_fullsized":"2024-02-05T08:15:49.742032+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:29.992957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:29.992957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:51.419597+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":473,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766984+0000","last_change":"2024-02-05T08:15:18.892281+0000","last_active":"2024-02-05T08:15:52.766984+0000","last_peered":"2024-02-05T08:15:52.766984+0000","last_clean":"2024-02-05T08:15:52.766984+0000","last_became_active":"2024-02-05T08:12:29.679395+0000","last_became_peered":"2024-02-05T08:12:29.679395+0000","last_unstale":"2024-02-05T08:15:52.766984+0000","last_undegraded":"2024-02-05T08:15:52.766984+0000","last_fullsized":"2024-02-05T08:15:52.766984+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:15:18.892157+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:15:18.892157+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:21.437539+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":443,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.741590+0000","last_change":"2024-02-05T08:14:04.926625+0000","last_active":"2024-02-05T08:15:49.741590+0000","last_peered":"2024-02-05T08:15:49.741590+0000","last_clean":"2024-02-05T08:15:49.741590+0000","last_became_active":"2024-02-05T08:12:29.670859+0000","last_became_peered":"2024-02-05T08:12:29.670859+0000","last_unstale":"2024-02-05T08:15:49.741590+0000","last_undegraded":"2024-02-05T08:15:49.741590+0000","last_fullsized":"2024-02-05T08:15:49.741590+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:04.926518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:14:04.926518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"236'51","reported_seq":669,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.764658+0000","last_change":"2024-02-05T08:12:41.775200+0000","last_active":"2024-02-05T08:15:52.764658+0000","last_peered":"2024-02-05T08:15:52.764658+0000","last_clean":"2024-02-05T08:15:52.764658+0000","last_became_active":"2024-02-05T08:12:41.719820+0000","last_became_peered":"2024-02-05T08:12:41.719820+0000","last_unstale":"2024-02-05T08:15:52.764658+0000","last_undegraded":"2024-02-05T08:15:52.764658+0000","last_fullsized":"2024-02-05T08:15:52.764658+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'38","last_scrub_stamp":"2024-02-05T08:11:34.819564+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:11:34.819564+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.849389+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":416,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.770300+0000","last_change":"2024-02-05T08:12:28.727498+0000","last_active":"2024-02-05T08:15:52.770300+0000","last_peered":"2024-02-05T08:15:52.770300+0000","last_clean":"2024-02-05T08:15:52.770300+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:15:52.770300+0000","last_undegraded":"2024-02-05T08:15:52.770300+0000","last_fullsized":"2024-02-05T08:15:52.770300+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:28.721402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:28.721402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666797+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"260'4306","reported_seq":4358,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.844944+0000","last_change":"2024-02-05T08:14:57.763579+0000","last_active":"2024-02-05T08:15:52.844944+0000","last_peered":"2024-02-05T08:15:52.844944+0000","last_clean":"2024-02-05T08:15:52.844944+0000","last_became_active":"2024-02-05T08:14:57.763184+0000","last_became_peered":"2024-02-05T08:14:57.763184+0000","last_unstale":"2024-02-05T08:15:52.844944+0000","last_undegraded":"2024-02-05T08:15:52.844944+0000","last_fullsized":"2024-02-05T08:15:52.844944+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:00.113741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282198016,"num_objects":4306,"num_object_clones":0,"num_object_copies":12918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4306,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4306,"num_write_kb":275584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":446,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.751093+0000","last_change":"2024-02-05T08:15:20.467544+0000","last_active":"2024-02-05T08:15:49.751093+0000","last_peered":"2024-02-05T08:15:49.751093+0000","last_clean":"2024-02-05T08:15:49.751093+0000","last_became_active":"2024-02-05T08:12:41.732338+0000","last_became_peered":"2024-02-05T08:12:41.732338+0000","last_unstale":"2024-02-05T08:15:49.751093+0000","last_undegraded":"2024-02-05T08:15:49.751093+0000","last_fullsized":"2024-02-05T08:15:49.751093+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:20.467407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:20.467407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:45.580190+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":461,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.764923+0000","last_change":"2024-02-05T08:15:17.824514+0000","last_active":"2024-02-05T08:15:52.764923+0000","last_peered":"2024-02-05T08:15:52.764923+0000","last_clean":"2024-02-05T08:15:52.764923+0000","last_became_active":"2024-02-05T08:12:29.673554+0000","last_became_peered":"2024-02-05T08:12:29.673554+0000","last_unstale":"2024-02-05T08:15:52.764923+0000","last_undegraded":"2024-02-05T08:15:52.764923+0000","last_fullsized":"2024-02-05T08:15:52.764923+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:15:17.824391+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:15:17.824391+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:31.698204+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":464,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.767223+0000","last_change":"2024-02-05T08:15:18.722410+0000","last_active":"2024-02-05T08:15:52.767223+0000","last_peered":"2024-02-05T08:15:52.767223+0000","last_clean":"2024-02-05T08:15:52.767223+0000","last_became_active":"2024-02-05T08:12:41.746289+0000","last_became_peered":"2024-02-05T08:12:41.746289+0000","last_unstale":"2024-02-05T08:15:52.767223+0000","last_undegraded":"2024-02-05T08:15:52.767223+0000","last_fullsized":"2024-02-05T08:15:52.767223+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:18.722163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:15:18.722163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:42.755882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'17","reported_seq":447,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.768328+0000","last_change":"2024-02-05T08:14:21.682688+0000","last_active":"2024-02-05T08:15:52.768328+0000","last_peered":"2024-02-05T08:15:52.768328+0000","last_clean":"2024-02-05T08:15:52.768328+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:15:52.768328+0000","last_undegraded":"2024-02-05T08:15:52.768328+0000","last_fullsized":"2024-02-05T08:15:52.768328+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-02-05T08:14:21.682517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:21.682517+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.799157+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":441,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.768128+0000","last_change":"2024-02-05T08:15:20.774445+0000","last_active":"2024-02-05T08:15:52.768128+0000","last_peered":"2024-02-05T08:15:52.768128+0000","last_clean":"2024-02-05T08:15:52.768128+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:15:52.768128+0000","last_undegraded":"2024-02-05T08:15:52.768128+0000","last_fullsized":"2024-02-05T08:15:52.768128+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:20.774326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:20.774326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:27.506687+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"260'4280","reported_seq":4332,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:53.528241+0000","last_change":"2024-02-05T08:14:57.762460+0000","last_active":"2024-02-05T08:15:53.528241+0000","last_peered":"2024-02-05T08:15:53.528241+0000","last_clean":"2024-02-05T08:15:53.528241+0000","last_became_active":"2024-02-05T08:14:57.762164+0000","last_became_peered":"2024-02-05T08:14:57.762164+0000","last_unstale":"2024-02-05T08:15:53.528241+0000","last_undegraded":"2024-02-05T08:15:53.528241+0000","last_fullsized":"2024-02-05T08:15:53.528241+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4280,"log_dups_size":0,"ondisk_log_size":4280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:12.058644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280494080,"num_objects":4280,"num_object_clones":0,"num_object_copies":12840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4280,"num_write_kb":273920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":427,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.768939+0000","last_change":"2024-02-05T08:12:41.717519+0000","last_active":"2024-02-05T08:15:52.768939+0000","last_peered":"2024-02-05T08:15:52.768939+0000","last_clean":"2024-02-05T08:15:52.768939+0000","last_became_active":"2024-02-05T08:12:41.717170+0000","last_became_peered":"2024-02-05T08:12:41.717170+0000","last_unstale":"2024-02-05T08:15:52.768939+0000","last_undegraded":"2024-02-05T08:15:52.768939+0000","last_fullsized":"2024-02-05T08:15:52.768939+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:23.303613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:23.303613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.005486+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":720,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.745993+0000","last_change":"2024-02-05T08:13:31.220426+0000","last_active":"2024-02-05T08:15:49.745993+0000","last_peered":"2024-02-05T08:15:49.745993+0000","last_clean":"2024-02-05T08:15:49.745993+0000","last_became_active":"2024-02-05T08:12:32.677426+0000","last_became_peered":"2024-02-05T08:12:32.677426+0000","last_unstale":"2024-02-05T08:15:49.745993+0000","last_undegraded":"2024-02-05T08:15:49.745993+0000","last_fullsized":"2024-02-05T08:15:49.745993+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:13:31.220332+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:13:31.220332+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276167+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":435,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766902+0000","last_change":"2024-02-05T08:15:13.853710+0000","last_active":"2024-02-05T08:15:52.766902+0000","last_peered":"2024-02-05T08:15:52.766902+0000","last_clean":"2024-02-05T08:15:52.766902+0000","last_became_active":"2024-02-05T08:12:41.722331+0000","last_became_peered":"2024-02-05T08:12:41.722331+0000","last_unstale":"2024-02-05T08:15:52.766902+0000","last_undegraded":"2024-02-05T08:15:52.766902+0000","last_fullsized":"2024-02-05T08:15:52.766902+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:13.853624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:13.853624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:35.667987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"236'30","reported_seq":237,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.739659+0000","last_change":"2024-02-05T08:15:14.981439+0000","last_active":"2024-02-05T08:15:49.739659+0000","last_peered":"2024-02-05T08:15:49.739659+0000","last_clean":"2024-02-05T08:15:49.739659+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:15:49.739659+0000","last_undegraded":"2024-02-05T08:15:49.739659+0000","last_fullsized":"2024-02-05T08:15:49.739659+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:15:14.981271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:14.981271+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:36.831841+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":424,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.748026+0000","last_change":"2024-02-05T08:13:30.200414+0000","last_active":"2024-02-05T08:15:49.748026+0000","last_peered":"2024-02-05T08:15:49.748026+0000","last_clean":"2024-02-05T08:15:49.748026+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:15:49.748026+0000","last_undegraded":"2024-02-05T08:15:49.748026+0000","last_fullsized":"2024-02-05T08:15:49.748026+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:13:30.200341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:13:30.200341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"260'4492","reported_seq":4544,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:53.501628+0000","last_change":"2024-02-05T08:14:57.761686+0000","last_active":"2024-02-05T08:15:53.501628+0000","last_peered":"2024-02-05T08:15:53.501628+0000","last_clean":"2024-02-05T08:15:53.501628+0000","last_became_active":"2024-02-05T08:14:57.761382+0000","last_became_peered":"2024-02-05T08:14:57.761382+0000","last_unstale":"2024-02-05T08:15:53.501628+0000","last_undegraded":"2024-02-05T08:15:53.501628+0000","last_fullsized":"2024-02-05T08:15:53.501628+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4492,"log_dups_size":0,"ondisk_log_size":4492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:57.838632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294387712,"num_objects":4492,"num_object_clones":0,"num_object_copies":13476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4492,"num_write_kb":287488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":428,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.764210+0000","last_change":"2024-02-05T08:12:41.716722+0000","last_active":"2024-02-05T08:15:52.764210+0000","last_peered":"2024-02-05T08:15:52.764210+0000","last_clean":"2024-02-05T08:15:52.764210+0000","last_became_active":"2024-02-05T08:12:41.716394+0000","last_became_peered":"2024-02-05T08:12:41.716394+0000","last_unstale":"2024-02-05T08:15:52.764210+0000","last_undegraded":"2024-02-05T08:15:52.764210+0000","last_fullsized":"2024-02-05T08:15:52.764210+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:26.178219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:26.178219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.849439+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":438,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.765851+0000","last_change":"2024-02-05T08:15:15.359343+0000","last_active":"2024-02-05T08:15:52.765851+0000","last_peered":"2024-02-05T08:15:52.765851+0000","last_clean":"2024-02-05T08:15:52.765851+0000","last_became_active":"2024-02-05T08:12:41.741846+0000","last_became_peered":"2024-02-05T08:12:41.741846+0000","last_unstale":"2024-02-05T08:15:52.765851+0000","last_undegraded":"2024-02-05T08:15:52.765851+0000","last_fullsized":"2024-02-05T08:15:52.765851+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:15.359190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:15.359190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:41.785491+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":1117,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:50.822697+0000","last_change":"2024-02-05T08:13:36.138202+0000","last_active":"2024-02-05T08:15:50.822697+0000","last_peered":"2024-02-05T08:15:50.822697+0000","last_clean":"2024-02-05T08:15:50.822697+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:15:50.822697+0000","last_undegraded":"2024-02-05T08:15:50.822697+0000","last_fullsized":"2024-02-05T08:15:50.822697+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:13:36.138048+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276223+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":664,"num_read_kb":659,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":476,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.768958+0000","last_change":"2024-02-05T08:14:25.014118+0000","last_active":"2024-02-05T08:15:52.768958+0000","last_peered":"2024-02-05T08:15:52.768958+0000","last_clean":"2024-02-05T08:15:52.768958+0000","last_became_active":"2024-02-05T08:14:24.992820+0000","last_became_peered":"2024-02-05T08:14:24.992820+0000","last_unstale":"2024-02-05T08:15:52.768958+0000","last_undegraded":"2024-02-05T08:15:52.768958+0000","last_fullsized":"2024-02-05T08:15:52.768958+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-02-05T08:12:16.517774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:16.517774+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666678+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":425,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.769264+0000","last_change":"2024-02-05T08:12:24.353439+0000","last_active":"2024-02-05T08:15:52.769264+0000","last_peered":"2024-02-05T08:15:52.769264+0000","last_clean":"2024-02-05T08:15:52.769264+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:15:52.769264+0000","last_undegraded":"2024-02-05T08:15:52.769264+0000","last_fullsized":"2024-02-05T08:15:52.769264+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:24.353354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:12:24.353354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.005787+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"259'4298","reported_seq":4348,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.714615+0000","last_change":"2024-02-05T08:14:57.764321+0000","last_active":"2024-02-05T08:15:52.714615+0000","last_peered":"2024-02-05T08:15:52.714615+0000","last_clean":"2024-02-05T08:15:52.714615+0000","last_became_active":"2024-02-05T08:14:57.764021+0000","last_became_peered":"2024-02-05T08:14:57.764021+0000","last_unstale":"2024-02-05T08:15:52.714615+0000","last_undegraded":"2024-02-05T08:15:52.714615+0000","last_fullsized":"2024-02-05T08:15:52.714615+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:57.697074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281673728,"num_objects":4298,"num_object_clones":0,"num_object_copies":12894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4298,"num_write_kb":275072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":434,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.763996+0000","last_change":"2024-02-05T08:12:29.673477+0000","last_active":"2024-02-05T08:15:52.763996+0000","last_peered":"2024-02-05T08:15:52.763996+0000","last_clean":"2024-02-05T08:15:52.763996+0000","last_became_active":"2024-02-05T08:12:29.672973+0000","last_became_peered":"2024-02-05T08:12:29.672973+0000","last_unstale":"2024-02-05T08:15:52.763996+0000","last_undegraded":"2024-02-05T08:15:52.763996+0000","last_fullsized":"2024-02-05T08:15:52.763996+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:25.219172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:25.219172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.849472+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":442,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.764466+0000","last_change":"2024-02-05T08:14:19.636646+0000","last_active":"2024-02-05T08:15:52.764466+0000","last_peered":"2024-02-05T08:15:52.764466+0000","last_clean":"2024-02-05T08:15:52.764466+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:15:52.764466+0000","last_undegraded":"2024-02-05T08:15:52.764466+0000","last_fullsized":"2024-02-05T08:15:52.764466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:19.636535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:19.636535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":479,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.769242+0000","last_change":"2024-02-05T08:12:27.724075+0000","last_active":"2024-02-05T08:15:52.769242+0000","last_peered":"2024-02-05T08:15:52.769242+0000","last_clean":"2024-02-05T08:15:52.769242+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:15:52.769242+0000","last_undegraded":"2024-02-05T08:15:52.769242+0000","last_fullsized":"2024-02-05T08:15:52.769242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-02-05T08:12:27.723924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:27.723924+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666618+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":160,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.749379+0000","last_change":"2024-02-05T08:14:22.219846+0000","last_active":"2024-02-05T08:15:49.749379+0000","last_peered":"2024-02-05T08:15:49.749379+0000","last_clean":"2024-02-05T08:15:49.749379+0000","last_became_active":"2024-02-05T08:12:29.688534+0000","last_became_peered":"2024-02-05T08:12:29.688534+0000","last_unstale":"2024-02-05T08:15:49.749379+0000","last_undegraded":"2024-02-05T08:15:49.749379+0000","last_fullsized":"2024-02-05T08:15:49.749379+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:22.219740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:14:22.219740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"260'4226","reported_seq":4277,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.843915+0000","last_change":"2024-02-05T08:14:57.764336+0000","last_active":"2024-02-05T08:15:52.843915+0000","last_peered":"2024-02-05T08:15:52.843915+0000","last_clean":"2024-02-05T08:15:52.843915+0000","last_became_active":"2024-02-05T08:14:57.764038+0000","last_became_peered":"2024-02-05T08:14:57.764038+0000","last_unstale":"2024-02-05T08:15:52.843915+0000","last_undegraded":"2024-02-05T08:15:52.843915+0000","last_fullsized":"2024-02-05T08:15:52.843915+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4226,"log_dups_size":0,"ondisk_log_size":4226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:06.265859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276955136,"num_objects":4226,"num_object_clones":0,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4226,"num_write_kb":270464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":426,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766174+0000","last_change":"2024-02-05T08:12:41.736005+0000","last_active":"2024-02-05T08:15:52.766174+0000","last_peered":"2024-02-05T08:15:52.766174+0000","last_clean":"2024-02-05T08:15:52.766174+0000","last_became_active":"2024-02-05T08:12:41.735706+0000","last_became_peered":"2024-02-05T08:12:41.735706+0000","last_unstale":"2024-02-05T08:15:52.766174+0000","last_undegraded":"2024-02-05T08:15:52.766174+0000","last_fullsized":"2024-02-05T08:15:52.766174+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:12:20.607056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:12:20.607056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666646+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":447,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.768084+0000","last_change":"2024-02-05T08:15:16.880868+0000","last_active":"2024-02-05T08:15:52.768084+0000","last_peered":"2024-02-05T08:15:52.768084+0000","last_clean":"2024-02-05T08:15:52.768084+0000","last_became_active":"2024-02-05T08:12:41.730010+0000","last_became_peered":"2024-02-05T08:12:41.730010+0000","last_unstale":"2024-02-05T08:15:52.768084+0000","last_undegraded":"2024-02-05T08:15:52.768084+0000","last_fullsized":"2024-02-05T08:15:52.768084+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:16.880446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:16.880446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:46.844168+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'20","reported_seq":460,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.764997+0000","last_change":"2024-02-05T08:12:19.056472+0000","last_active":"2024-02-05T08:15:52.764997+0000","last_peered":"2024-02-05T08:15:52.764997+0000","last_clean":"2024-02-05T08:15:52.764997+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:15:52.764997+0000","last_undegraded":"2024-02-05T08:15:52.764997+0000","last_fullsized":"2024-02-05T08:15:52.764997+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:12:19.056345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:12:19.056345+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.849424+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"190'10","reported_seq":587,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.769507+0000","last_change":"2024-02-05T08:14:30.163779+0000","last_active":"2024-02-05T08:15:52.769507+0000","last_peered":"2024-02-05T08:15:52.769507+0000","last_clean":"2024-02-05T08:15:52.769507+0000","last_became_active":"2024-02-05T08:12:29.683091+0000","last_became_peered":"2024-02-05T08:12:29.683091+0000","last_unstale":"2024-02-05T08:15:52.769507+0000","last_undegraded":"2024-02-05T08:15:52.769507+0000","last_fullsized":"2024-02-05T08:15:52.769507+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:14:30.163629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:30.163629+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666999+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"259'4227","reported_seq":4275,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.660820+0000","last_change":"2024-02-05T08:14:57.761716+0000","last_active":"2024-02-05T08:15:52.660820+0000","last_peered":"2024-02-05T08:15:52.660820+0000","last_clean":"2024-02-05T08:15:52.660820+0000","last_became_active":"2024-02-05T08:14:57.761443+0000","last_became_peered":"2024-02-05T08:14:57.761443+0000","last_unstale":"2024-02-05T08:15:52.660820+0000","last_undegraded":"2024-02-05T08:15:52.660820+0000","last_fullsized":"2024-02-05T08:15:52.660820+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4227,"log_dups_size":0,"ondisk_log_size":4227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:23.258466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277020672,"num_objects":4227,"num_object_clones":0,"num_object_copies":12681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4227,"num_write_kb":270528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":440,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.750795+0000","last_change":"2024-02-05T08:15:29.563981+0000","last_active":"2024-02-05T08:15:49.750795+0000","last_peered":"2024-02-05T08:15:49.750795+0000","last_clean":"2024-02-05T08:15:49.750795+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:15:49.750795+0000","last_undegraded":"2024-02-05T08:15:49.750795+0000","last_fullsized":"2024-02-05T08:15:49.750795+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:29.563876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:29.563876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:57.970791+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":443,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.769052+0000","last_change":"2024-02-05T08:12:21.623410+0000","last_active":"2024-02-05T08:15:52.769052+0000","last_peered":"2024-02-05T08:15:52.769052+0000","last_clean":"2024-02-05T08:15:52.769052+0000","last_became_active":"2024-02-05T08:11:23.200753+0000","last_became_peered":"2024-02-05T08:11:23.200753+0000","last_unstale":"2024-02-05T08:15:52.769052+0000","last_undegraded":"2024-02-05T08:15:52.769052+0000","last_fullsized":"2024-02-05T08:15:52.769052+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:12:21.623291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:12:21.623291+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.666770+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"236'28","reported_seq":513,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.742732+0000","last_change":"2024-02-05T08:15:24.002222+0000","last_active":"2024-02-05T08:15:49.742732+0000","last_peered":"2024-02-05T08:15:49.742732+0000","last_clean":"2024-02-05T08:15:49.742732+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:15:49.742732+0000","last_undegraded":"2024-02-05T08:15:49.742732+0000","last_fullsized":"2024-02-05T08:15:49.742732+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:15:24.002106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:24.002106+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:36.950373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"194'10","reported_seq":512,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.764698+0000","last_change":"2024-02-05T08:15:28.826050+0000","last_active":"2024-02-05T08:15:52.764698+0000","last_peered":"2024-02-05T08:15:52.764698+0000","last_clean":"2024-02-05T08:15:52.764698+0000","last_became_active":"2024-02-05T08:12:41.721686+0000","last_became_peered":"2024-02-05T08:12:41.721686+0000","last_unstale":"2024-02-05T08:15:52.764698+0000","last_undegraded":"2024-02-05T08:15:52.764698+0000","last_fullsized":"2024-02-05T08:15:52.764698+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'10","last_scrub_stamp":"2024-02-05T08:15:28.825911+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:15:28.825911+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:45.870601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"259'4294","reported_seq":4344,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.727704+0000","last_change":"2024-02-05T08:14:57.765949+0000","last_active":"2024-02-05T08:15:52.727704+0000","last_peered":"2024-02-05T08:15:52.727704+0000","last_clean":"2024-02-05T08:15:52.727704+0000","last_became_active":"2024-02-05T08:14:57.765609+0000","last_became_peered":"2024-02-05T08:14:57.765609+0000","last_unstale":"2024-02-05T08:15:52.727704+0000","last_undegraded":"2024-02-05T08:15:52.727704+0000","last_fullsized":"2024-02-05T08:15:52.727704+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4294,"log_dups_size":0,"ondisk_log_size":4294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:07.832990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281411584,"num_objects":4294,"num_object_clones":0,"num_object_copies":12882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4294,"num_write_kb":274816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":433,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.741625+0000","last_change":"2024-02-05T08:15:22.953610+0000","last_active":"2024-02-05T08:15:49.741625+0000","last_peered":"2024-02-05T08:15:49.741625+0000","last_clean":"2024-02-05T08:15:49.741625+0000","last_became_active":"2024-02-05T08:12:41.731253+0000","last_became_peered":"2024-02-05T08:12:41.731253+0000","last_unstale":"2024-02-05T08:15:49.741625+0000","last_undegraded":"2024-02-05T08:15:49.741625+0000","last_fullsized":"2024-02-05T08:15:49.741625+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.953481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.953481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:26.478801+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":163,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.746660+0000","last_change":"2024-02-05T08:15:15.390114+0000","last_active":"2024-02-05T08:15:49.746660+0000","last_peered":"2024-02-05T08:15:49.746660+0000","last_clean":"2024-02-05T08:15:49.746660+0000","last_became_active":"2024-02-05T08:12:41.723699+0000","last_became_peered":"2024-02-05T08:12:41.723699+0000","last_unstale":"2024-02-05T08:15:49.746660+0000","last_undegraded":"2024-02-05T08:15:49.746660+0000","last_fullsized":"2024-02-05T08:15:49.746660+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:15:15.389990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:15.389990+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:24.889149+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"236'38","reported_seq":493,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.746174+0000","last_change":"2024-02-05T08:14:24.593255+0000","last_active":"2024-02-05T08:15:49.746174+0000","last_peered":"2024-02-05T08:15:49.746174+0000","last_clean":"2024-02-05T08:15:49.746174+0000","last_became_active":"2024-02-05T08:12:41.728044+0000","last_became_peered":"2024-02-05T08:12:41.728044+0000","last_unstale":"2024-02-05T08:15:49.746174+0000","last_undegraded":"2024-02-05T08:15:49.746174+0000","last_fullsized":"2024-02-05T08:15:49.746174+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'26","last_scrub_stamp":"2024-02-05T08:14:24.591818+0000","last_deep_scrub":"186'26","last_deep_scrub_stamp":"2024-02-05T08:12:24.579532+0000","last_clean_scrub_stamp":"2024-02-05T08:14:24.591818+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.276438+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":429,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.742532+0000","last_change":"2024-02-05T08:14:26.979859+0000","last_active":"2024-02-05T08:15:49.742532+0000","last_peered":"2024-02-05T08:15:49.742532+0000","last_clean":"2024-02-05T08:15:49.742532+0000","last_became_active":"2024-02-05T08:12:41.720247+0000","last_became_peered":"2024-02-05T08:12:41.720247+0000","last_unstale":"2024-02-05T08:15:49.742532+0000","last_undegraded":"2024-02-05T08:15:49.742532+0000","last_fullsized":"2024-02-05T08:15:49.742532+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:26.979755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:26.979755+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"260'4310","reported_seq":4362,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:53.268089+0000","last_change":"2024-02-05T08:14:57.764494+0000","last_active":"2024-02-05T08:15:53.268089+0000","last_peered":"2024-02-05T08:15:53.268089+0000","last_clean":"2024-02-05T08:15:53.268089+0000","last_became_active":"2024-02-05T08:14:57.764174+0000","last_became_peered":"2024-02-05T08:14:57.764174+0000","last_unstale":"2024-02-05T08:15:53.268089+0000","last_undegraded":"2024-02-05T08:15:53.268089+0000","last_fullsized":"2024-02-05T08:15:53.268089+0000","mapping_epoch":239,"log_start":"246'200","ondisk_log_start":"246'200","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4110,"log_dups_size":0,"ondisk_log_size":4110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:20.765265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282460160,"num_objects":4310,"num_object_clones":0,"num_object_copies":12930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4310,"num_write_kb":275840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":428,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.767750+0000","last_change":"2024-02-05T08:14:07.163475+0000","last_active":"2024-02-05T08:15:52.767750+0000","last_peered":"2024-02-05T08:15:52.767750+0000","last_clean":"2024-02-05T08:15:52.767750+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:15:52.767750+0000","last_undegraded":"2024-02-05T08:15:52.767750+0000","last_fullsized":"2024-02-05T08:15:52.767750+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:07.163372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:14:07.163372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.005957+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":565,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.743120+0000","last_change":"2024-02-05T08:14:31.071675+0000","last_active":"2024-02-05T08:15:49.743120+0000","last_peered":"2024-02-05T08:15:49.743120+0000","last_clean":"2024-02-05T08:15:49.743120+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:15:49.743120+0000","last_undegraded":"2024-02-05T08:15:49.743120+0000","last_fullsized":"2024-02-05T08:15:49.743120+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:14:31.071536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:31.071536+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:54.946542+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":153,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.765110+0000","last_change":"2024-02-05T08:14:17.648700+0000","last_active":"2024-02-05T08:15:52.765110+0000","last_peered":"2024-02-05T08:15:52.765110+0000","last_clean":"2024-02-05T08:15:52.765110+0000","last_became_active":"2024-02-05T08:12:41.724969+0000","last_became_peered":"2024-02-05T08:12:41.724969+0000","last_unstale":"2024-02-05T08:15:52.765110+0000","last_undegraded":"2024-02-05T08:15:52.765110+0000","last_fullsized":"2024-02-05T08:15:52.765110+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:17.648596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:14:17.648596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.849488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"236'27","reported_seq":502,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.769858+0000","last_change":"2024-02-05T08:15:15.777707+0000","last_active":"2024-02-05T08:15:52.769858+0000","last_peered":"2024-02-05T08:15:52.769858+0000","last_clean":"2024-02-05T08:15:52.769858+0000","last_became_active":"2024-02-05T08:12:41.725007+0000","last_became_peered":"2024-02-05T08:12:41.725007+0000","last_unstale":"2024-02-05T08:15:52.769858+0000","last_undegraded":"2024-02-05T08:15:52.769858+0000","last_fullsized":"2024-02-05T08:15:52.769858+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:15:15.777496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:15.777496+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:43.525899+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"259'4400","reported_seq":4450,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.742516+0000","last_change":"2024-02-05T08:14:57.765880+0000","last_active":"2024-02-05T08:15:52.742516+0000","last_peered":"2024-02-05T08:15:52.742516+0000","last_clean":"2024-02-05T08:15:52.742516+0000","last_became_active":"2024-02-05T08:14:57.765255+0000","last_became_peered":"2024-02-05T08:14:57.765255+0000","last_unstale":"2024-02-05T08:15:52.742516+0000","last_undegraded":"2024-02-05T08:15:52.742516+0000","last_fullsized":"2024-02-05T08:15:52.742516+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4400,"log_dups_size":0,"ondisk_log_size":4400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:10.520554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288358400,"num_objects":4400,"num_object_clones":0,"num_object_copies":13200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4400,"num_write_kb":281600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":419,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.739158+0000","last_change":"2024-02-05T08:15:09.025688+0000","last_active":"2024-02-05T08:15:49.739158+0000","last_peered":"2024-02-05T08:15:49.739158+0000","last_clean":"2024-02-05T08:15:49.739158+0000","last_became_active":"2024-02-05T08:12:41.719835+0000","last_became_peered":"2024-02-05T08:12:41.719835+0000","last_unstale":"2024-02-05T08:15:49.739158+0000","last_undegraded":"2024-02-05T08:15:49.739158+0000","last_fullsized":"2024-02-05T08:15:49.739158+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:09.025580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:15:09.025580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:34.218391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":444,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.769221+0000","last_change":"2024-02-05T08:14:29.177432+0000","last_active":"2024-02-05T08:15:52.769221+0000","last_peered":"2024-02-05T08:15:52.769221+0000","last_clean":"2024-02-05T08:15:52.769221+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:15:52.769221+0000","last_undegraded":"2024-02-05T08:15:52.769221+0000","last_fullsized":"2024-02-05T08:15:52.769221+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:29.177314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:14:29.177314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.005985+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":439,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.742606+0000","last_change":"2024-02-05T08:14:24.985198+0000","last_active":"2024-02-05T08:15:49.742606+0000","last_peered":"2024-02-05T08:15:49.742606+0000","last_clean":"2024-02-05T08:15:49.742606+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:15:49.742606+0000","last_undegraded":"2024-02-05T08:15:49.742606+0000","last_fullsized":"2024-02-05T08:15:49.742606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:24.985069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:14:24.985069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:50.692856+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"236'19","reported_seq":493,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.742190+0000","last_change":"2024-02-05T08:14:09.985107+0000","last_active":"2024-02-05T08:15:49.742190+0000","last_peered":"2024-02-05T08:15:49.742190+0000","last_clean":"2024-02-05T08:15:49.742190+0000","last_became_active":"2024-02-05T08:12:29.666284+0000","last_became_peered":"2024-02-05T08:12:29.666284+0000","last_unstale":"2024-02-05T08:15:49.742190+0000","last_undegraded":"2024-02-05T08:15:49.742190+0000","last_fullsized":"2024-02-05T08:15:49.742190+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-02-05T08:14:09.984989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:09.984989+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"260'4349","reported_seq":4401,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:53.232980+0000","last_change":"2024-02-05T08:14:57.765243+0000","last_active":"2024-02-05T08:15:53.232980+0000","last_peered":"2024-02-05T08:15:53.232980+0000","last_clean":"2024-02-05T08:15:53.232980+0000","last_became_active":"2024-02-05T08:14:57.764870+0000","last_became_peered":"2024-02-05T08:14:57.764870+0000","last_unstale":"2024-02-05T08:15:53.232980+0000","last_undegraded":"2024-02-05T08:15:53.232980+0000","last_fullsized":"2024-02-05T08:15:53.232980+0000","mapping_epoch":239,"log_start":"246'200","ondisk_log_start":"246'200","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:09.829403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285016064,"num_objects":4349,"num_object_clones":0,"num_object_copies":13047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4349,"num_write_kb":278336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":447,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.768424+0000","last_change":"2024-02-05T08:15:21.884006+0000","last_active":"2024-02-05T08:15:52.768424+0000","last_peered":"2024-02-05T08:15:52.768424+0000","last_clean":"2024-02-05T08:15:52.768424+0000","last_became_active":"2024-02-05T08:12:41.737224+0000","last_became_peered":"2024-02-05T08:12:41.737224+0000","last_unstale":"2024-02-05T08:15:52.768424+0000","last_undegraded":"2024-02-05T08:15:52.768424+0000","last_fullsized":"2024-02-05T08:15:52.768424+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:15:21.883859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:21.883859+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:32.454548+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":425,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.764849+0000","last_change":"2024-02-05T08:15:14.746706+0000","last_active":"2024-02-05T08:15:52.764849+0000","last_peered":"2024-02-05T08:15:52.764849+0000","last_clean":"2024-02-05T08:15:52.764849+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:15:52.764849+0000","last_undegraded":"2024-02-05T08:15:52.764849+0000","last_fullsized":"2024-02-05T08:15:52.764849+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:14.746593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:14.746593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:42.161153+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":160,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.749578+0000","last_change":"2024-02-05T08:14:10.208350+0000","last_active":"2024-02-05T08:15:49.749578+0000","last_peered":"2024-02-05T08:15:49.749578+0000","last_clean":"2024-02-05T08:15:49.749578+0000","last_became_active":"2024-02-05T08:12:29.678501+0000","last_became_peered":"2024-02-05T08:12:29.678501+0000","last_unstale":"2024-02-05T08:15:49.749578+0000","last_undegraded":"2024-02-05T08:15:49.749578+0000","last_fullsized":"2024-02-05T08:15:49.749578+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:10.208236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:14:10.208236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"236'31","reported_seq":475,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.741752+0000","last_change":"2024-02-05T08:14:34.016671+0000","last_active":"2024-02-05T08:15:49.741752+0000","last_peered":"2024-02-05T08:15:49.741752+0000","last_clean":"2024-02-05T08:15:49.741752+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:15:49.741752+0000","last_undegraded":"2024-02-05T08:15:49.741752+0000","last_fullsized":"2024-02-05T08:15:49.741752+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-02-05T08:14:34.016456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:34.016456+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:15:53.345431+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"259'4300","reported_seq":4348,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.576027+0000","last_change":"2024-02-05T08:14:57.762954+0000","last_active":"2024-02-05T08:15:52.576027+0000","last_peered":"2024-02-05T08:15:52.576027+0000","last_clean":"2024-02-05T08:15:52.576027+0000","last_became_active":"2024-02-05T08:14:57.762652+0000","last_became_peered":"2024-02-05T08:14:57.762652+0000","last_unstale":"2024-02-05T08:15:52.576027+0000","last_undegraded":"2024-02-05T08:15:52.576027+0000","last_fullsized":"2024-02-05T08:15:52.576027+0000","mapping_epoch":239,"log_start":"0'0","ondisk_log_start":"0'0","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4300,"log_dups_size":0,"ondisk_log_size":4300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:07.296366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281804800,"num_objects":4300,"num_object_clones":0,"num_object_copies":12900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4300,"num_write_kb":275200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":418,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.743169+0000","last_change":"2024-02-05T08:14:21.925615+0000","last_active":"2024-02-05T08:15:49.743169+0000","last_peered":"2024-02-05T08:15:49.743169+0000","last_clean":"2024-02-05T08:15:49.743169+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:15:49.743169+0000","last_undegraded":"2024-02-05T08:15:49.743169+0000","last_fullsized":"2024-02-05T08:15:49.743169+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:21.925502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:14:21.925502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":584,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.766805+0000","last_change":"2024-02-05T08:15:10.756204+0000","last_active":"2024-02-05T08:15:52.766805+0000","last_peered":"2024-02-05T08:15:52.766805+0000","last_clean":"2024-02-05T08:15:52.766805+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:15:52.766805+0000","last_undegraded":"2024-02-05T08:15:52.766805+0000","last_fullsized":"2024-02-05T08:15:52.766805+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:15:10.756097+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:15:10.756097+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:20.981357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":152,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-02-05T08:15:52.769334+0000","last_change":"2024-02-05T08:15:29.437631+0000","last_active":"2024-02-05T08:15:52.769334+0000","last_peered":"2024-02-05T08:15:52.769334+0000","last_clean":"2024-02-05T08:15:52.769334+0000","last_became_active":"2024-02-05T08:12:41.757159+0000","last_became_peered":"2024-02-05T08:12:41.757159+0000","last_unstale":"2024-02-05T08:15:52.769334+0000","last_undegraded":"2024-02-05T08:15:52.769334+0000","last_fullsized":"2024-02-05T08:15:52.769334+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:29.437487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:29.437487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:16:40.329392+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"236'20","reported_seq":510,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-02-05T08:15:49.742403+0000","last_change":"2024-02-05T08:14:14.999089+0000","last_active":"2024-02-05T08:15:49.742403+0000","last_peered":"2024-02-05T08:15:49.742403+0000","last_clean":"2024-02-05T08:15:49.742403+0000","last_became_active":"2024-02-05T08:12:41.727271+0000","last_became_peered":"2024-02-05T08:12:41.727271+0000","last_unstale":"2024-02-05T08:15:49.742403+0000","last_undegraded":"2024-02-05T08:15:49.742403+0000","last_fullsized":"2024-02-05T08:15:49.742403+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-02-05T08:14:14.998858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:14:14.998858+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4538630144,"num_objects":69254,"num_object_clones":0,"num_object_copies":207762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69254,"num_write_kb":4432256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13598392320,"data_stored":13598392320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68554,"ondisk_log_size":68554,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":664,"num_read_kb":659,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1123,"num_read_kb":880,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":4032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":12894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":741,"ondisk_log_size":741,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037899,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3297596,"kb_used_data":2995696,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":90447556,"statfs":{"total":95995035648,"available":92618297344,"internally_reserved":0,"allocated":3067592704,"data_stored":3067164597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":309131722},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":1.837,"15min":1.3240000000000001},"min":{"1min":0.372,"5min":0.22,"15min":0.217},"max":{"1min":3.129,"5min":52.488999999999997,"15min":60.127000000000002},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.363,"5min":1.95,"15min":1.355},"min":{"1min":0.41699999999999998,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":5.0759999999999996,"5min":52.335000000000001,"15min":58.283000000000001},"last":3.1139999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:15:50 2024","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.9199999999999999,"15min":1.296},"min":{"1min":0.38400000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.1609999999999996,"5min":38.399999999999999,"15min":38.399999999999999},"last":3.1379999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.806,"15min":1.151},"min":{"1min":0.32000000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.3739999999999997,"5min":38.195999999999998,"15min":38.195999999999998},"last":0.76400000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.995,"15min":0.871},"min":{"1min":0.41499999999999998,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":3.1749999999999998,"5min":5.0590000000000002,"15min":7.6779999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.095,"15min":0.94999999999999996},"min":{"1min":0.318,"5min":0.28499999999999998,"15min":0.24299999999999999},"max":{"1min":4.8360000000000003,"5min":5.8600000000000003,"15min":8.9100000000000001},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:15:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":0.96599999999999997,"15min":1.175},"min":{"1min":0.36499999999999999,"5min":0.249,"15min":0.249},"max":{"1min":4.8840000000000003,"5min":4.8840000000000003,"15min":58.515000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.357,"5min":1.0700000000000001,"15min":1.1779999999999999},"min":{"1min":0.44600000000000001,"5min":0.27400000000000002,"15min":0.27000000000000002},"max":{"1min":5.1040000000000001,"5min":5.6879999999999997,"15min":58.225999999999999},"last":3.0569999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:15:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.4470000000000001,"15min":0.75900000000000001},"min":{"1min":0.28100000000000003,"5min":0.23300000000000001,"15min":0.22},"max":{"1min":4.617,"5min":49.601999999999997,"15min":49.601999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.4690000000000001,"15min":0.77100000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.249},"max":{"1min":5.2169999999999996,"5min":49.795999999999999,"15min":49.795999999999999},"last":0.57899999999999996}]},{"osd":5,"last update":"Mon Feb 5 08:15:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7490000000000001,"5min":1.534,"15min":0.80200000000000005},"min":{"1min":0.30099999999999999,"5min":0.251,"15min":0.24299999999999999},"max":{"1min":8.0709999999999997,"5min":49.914000000000001,"15min":49.914000000000001},"last":3.089},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3320000000000001,"15min":0.76100000000000001},"min":{"1min":0.33100000000000002,"5min":0.314,"15min":0.27000000000000002},"max":{"1min":4.9710000000000001,"5min":52.256,"15min":52.256},"last":1.04}]},{"osd":6,"last update":"Mon Feb 5 08:15:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":0.94799999999999995,"15min":0.64300000000000002},"min":{"1min":0.34799999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":5.1299999999999999,"5min":8.0050000000000008,"15min":8.0050000000000008},"last":3.0169999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":0.82899999999999996,"15min":0.59299999999999997},"min":{"1min":0.41599999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":4.9569999999999999,"5min":7.2850000000000001,"15min":7.2850000000000001},"last":2.0859999999999999}]}]},{"osd":6,"up_from":51,"seq":219043332241,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1071372,"kb_used_data":827780,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":92673780,"statfs":{"total":95995035648,"available":94897950720,"internally_reserved":0,"allocated":847646720,"data_stored":847197463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":249428422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":2.488,"15min":1.3879999999999999},"min":{"1min":0.28499999999999998,"5min":0.22900000000000001,"15min":0.223},"max":{"1min":3.8849999999999998,"5min":102.819,"15min":102.819},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.85499999999999998,"5min":2.4860000000000002,"15min":1.4610000000000001},"min":{"1min":0.28899999999999998,"5min":0.214,"15min":0.214},"max":{"1min":4.5389999999999997,"5min":102.75,"15min":102.75},"last":0.46000000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.5660000000000001,"15min":1.0780000000000001},"min":{"1min":0.28199999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.98,"5min":23.773,"15min":23.773},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.4039999999999999,"15min":1.004},"min":{"1min":0.36699999999999999,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":1.843,"5min":23.719000000000001,"15min":23.719000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":1.234,"15min":0.96799999999999997},"min":{"1min":0.28699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.931,"5min":22.503,"15min":22.503},"last":0.38},{"interface":"front","average":{"1min":0.72499999999999998,"5min":1.202,"15min":0.90100000000000002},"min":{"1min":0.36399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.4510000000000001,"5min":22.466000000000001,"15min":22.466000000000001},"last":0.39800000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":1.8939999999999999,"15min":1.26},"min":{"1min":0.36499999999999999,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":3.3100000000000001,"5min":102.438,"15min":102.438},"last":0.495},{"interface":"front","average":{"1min":0.67900000000000005,"5min":1.8220000000000001,"15min":1.242},"min":{"1min":0.38700000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.621,"5min":102.288,"15min":102.288},"last":0.443}]},{"osd":4,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.68400000000000005,"15min":0.51400000000000001},"min":{"1min":0.309,"5min":0.26100000000000001,"15min":0.223},"max":{"1min":3.3679999999999999,"5min":9.8970000000000002,"15min":9.8970000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.79400000000000004,"15min":0.56000000000000005},"min":{"1min":0.311,"5min":0.30199999999999999,"15min":0.26600000000000001},"max":{"1min":2.9300000000000002,"5min":4.9589999999999996,"15min":4.9589999999999996},"last":0.78200000000000003}]},{"osd":5,"last update":"Mon Feb 5 08:15:20 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.2430000000000001,"15min":0.72699999999999998},"min":{"1min":0.32400000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":3.3250000000000002,"5min":43.643000000000001,"15min":43.643000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.048,"5min":1.2629999999999999,"15min":0.745},"min":{"1min":0.40799999999999997,"5min":0.307,"15min":0.30399999999999999},"max":{"1min":3.7549999999999999,"5min":43.893000000000001,"15min":43.893000000000001},"last":0.81999999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:14:55 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.70599999999999996,"15min":0.57399999999999995},"min":{"1min":0.372,"5min":0.32000000000000001,"15min":0.29999999999999999},"max":{"1min":3.3980000000000001,"5min":5.0380000000000003,"15min":5.0380000000000003},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.88200000000000001,"5min":0.69199999999999995,"15min":0.58199999999999996},"min":{"1min":0.38900000000000001,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":3.0430000000000001,"5min":4.851,"15min":4.851},"last":3.4580000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2881948,"kb_used_data":2489808,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":90863204,"statfs":{"total":95995035648,"available":93043920896,"internally_reserved":0,"allocated":2549563392,"data_stored":2549109874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":1.46,"15min":1.3779999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.14699999999999999},"max":{"1min":3.1040000000000001,"5min":21.120999999999999,"15min":86.105000000000004},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.8300000000000001,"15min":1.623},"min":{"1min":0.23400000000000001,"5min":0.20699999999999999,"15min":0.16},"max":{"1min":6.1710000000000003,"5min":21.172999999999998,"15min":85.665000000000006},"last":0.62}]},{"osd":2,"last update":"Mon Feb 5 08:15:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.4119999999999999,"15min":1.278},"min":{"1min":0.221,"5min":0.191,"15min":0.17799999999999999},"max":{"1min":1.4099999999999999,"5min":19.016999999999999,"15min":49.030999999999999},"last":0.748},{"interface":"front","average":{"1min":0.57499999999999996,"5min":1.7490000000000001,"15min":1.4910000000000001},"min":{"1min":0.28899999999999998,"5min":0.21099999999999999,"15min":0.19900000000000001},"max":{"1min":1.417,"5min":20.434000000000001,"15min":48.984000000000002},"last":0.441}]},{"osd":3,"last update":"Mon Feb 5 08:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.623,"15min":1.4330000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20599999999999999},"max":{"1min":3.141,"5min":31.896000000000001,"15min":49.067999999999998},"last":1.0860000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.3440000000000001,"15min":1.373},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.24199999999999999},"max":{"1min":3.286,"5min":32.018999999999998,"15min":48.734999999999999},"last":0.79400000000000004}]},{"osd":4,"last update":"Mon Feb 5 08:15:31 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":1.3779999999999999,"15min":1.0189999999999999},"min":{"1min":0.42799999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.5510000000000002,"5min":25.952000000000002,"15min":25.952000000000002},"last":3.2200000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":1.625,"15min":1.127},"min":{"1min":0.45900000000000002,"5min":0.30199999999999999,"15min":0.27100000000000002},"max":{"1min":1.5049999999999999,"5min":25.922000000000001,"15min":25.922000000000001},"last":0.48399999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:15:52 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.827,"15min":1.179},"min":{"1min":0.55000000000000004,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.419,"5min":54.552999999999997,"15min":54.552999999999997},"last":9.6820000000000004},{"interface":"front","average":{"1min":1.238,"5min":2.0950000000000002,"15min":1.288},"min":{"1min":0.376,"5min":0.32500000000000001,"15min":0.28999999999999998},"max":{"1min":3.5459999999999998,"5min":55.811,"15min":55.811},"last":5.3659999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:15:30 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":1.5620000000000001,"15min":1.1180000000000001},"min":{"1min":0.42899999999999999,"5min":0.38,"15min":0.33100000000000002},"max":{"1min":1.256,"5min":54.384,"15min":54.384},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.5469999999999999,"15min":1.125},"min":{"1min":0.47499999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.339,"5min":55.210999999999999,"15min":55.210999999999999},"last":1.0629999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:15:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.401,"15min":1.1279999999999999},"min":{"1min":0.438,"5min":0.39500000000000002,"15min":0.28799999999999998},"max":{"1min":3.9809999999999999,"5min":7.9390000000000001,"15min":13.749000000000001},"last":2.3879999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.387,"15min":1.095},"min":{"1min":0.48999999999999999,"5min":0.379,"15min":0.379},"max":{"1min":3.8149999999999999,"5min":7.9640000000000004,"15min":7.9640000000000004},"last":1.9139999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738543,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2115324,"kb_used_data":1714868,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91629828,"statfs":{"total":95995035648,"available":93828943872,"internally_reserved":0,"allocated":1756024832,"data_stored":1755592372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":2.0449999999999999,"15min":1.6830000000000001},"min":{"1min":0.188,"5min":0.186,"15min":0.152},"max":{"1min":2.5619999999999998,"5min":32.066000000000003,"15min":66.772999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":1.8360000000000001,"15min":1.427},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.17199999999999999},"max":{"1min":2.4209999999999998,"5min":15.327999999999999,"15min":46.198999999999998},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Feb 5 08:15:49 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":1.768,"15min":1.4410000000000001},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.216},"max":{"1min":6.7050000000000001,"5min":12.074,"15min":46.170999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":1.8149999999999999,"15min":1.583},"min":{"1min":0.23499999999999999,"5min":0.20399999999999999,"15min":0.19400000000000001},"max":{"1min":3.1360000000000001,"5min":15.569000000000001,"15min":46.543999999999997},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:15:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.9750000000000001,"15min":1.6180000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.21099999999999999},"max":{"1min":6.5940000000000003,"5min":13.013,"15min":67.888000000000005},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":2.5649999999999999,"15min":1.9179999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.22600000000000001},"max":{"1min":1.6180000000000001,"5min":37.262999999999998,"15min":66.891999999999996},"last":0.58899999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.627,"15min":1.1000000000000001},"min":{"1min":0.38200000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":6.407,"5min":15.159000000000001,"15min":15.159000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.3500000000000001,"15min":1.0629999999999999},"min":{"1min":0.46100000000000002,"5min":0.28999999999999998,"15min":0.26900000000000002},"max":{"1min":6.915,"5min":10.928000000000001,"15min":12.989000000000001},"last":0.67200000000000004}]},{"osd":5,"last update":"Mon Feb 5 08:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.6120000000000001,"15min":1.1200000000000001},"min":{"1min":0.39100000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.1589999999999998,"5min":15.689,"15min":15.689},"last":0.998},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.3720000000000001,"15min":1.0569999999999999},"min":{"1min":0.438,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":12.94},"last":1.006}]},{"osd":6,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.2529999999999999,"15min":1.0489999999999999},"min":{"1min":0.49399999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":7.8220000000000001,"5min":7.8220000000000001,"15min":12.967000000000001},"last":1.9059999999999999},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.4350000000000001,"15min":1.133},"min":{"1min":0.47199999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.6240000000000001,"5min":15.721,"15min":15.721},"last":1.833}]},{"osd":7,"last update":"Mon Feb 5 08:15:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.6970000000000001,"15min":1.139},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":6.4480000000000004,"5min":15.211,"15min":15.211},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.512,"5min":1.5800000000000001,"15min":1.181},"min":{"1min":0.47299999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":6.8789999999999996,"5min":7.8650000000000002,"15min":13.012},"last":1.3360000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411493,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1770448,"kb_used_data":1365828,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":91974704,"statfs":{"total":95995035648,"available":94182096896,"internally_reserved":0,"allocated":1398607872,"data_stored":1398166350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":414316998},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:15:49 2024","interfaces":[{"interface":"back","average":{"1min":5.9050000000000002,"5min":3.2429999999999999,"15min":1.911},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.19700000000000001},"max":{"1min":107.155,"5min":107.155,"15min":107.155},"last":0.33700000000000002},{"interface":"front","average":{"1min":5.952,"5min":3.6219999999999999,"15min":2.0419999999999998},"min":{"1min":0.23100000000000001,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":107.221,"5min":107.221,"15min":107.221},"last":2.1819999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:15:49 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.9219999999999999,"15min":1.3620000000000001},"min":{"1min":0.25600000000000001,"5min":0.19400000000000001,"15min":0.17999999999999999},"max":{"1min":1.405,"5min":55.942,"15min":55.942},"last":2.5720000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":2.1030000000000002,"15min":1.423},"min":{"1min":0.23400000000000001,"5min":0.22900000000000001,"15min":0.193},"max":{"1min":1.6870000000000001,"5min":55.960999999999999,"15min":55.960999999999999},"last":1.907}]},{"osd":3,"last update":"Mon Feb 5 08:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.6200000000000001,"15min":1.6919999999999999},"min":{"1min":0.26100000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.605,"5min":8.6590000000000007,"15min":68.108000000000004},"last":1.8680000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":1.5,"15min":1.54},"min":{"1min":0.29199999999999998,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":1.8029999999999999,"5min":7.8879999999999999,"15min":67.945999999999998},"last":0.63900000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:15:33 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":1.851,"15min":1.254},"min":{"1min":0.35199999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.4820000000000002,"5min":31.596,"15min":31.596},"last":3.5009999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":1.6990000000000001,"15min":1.113},"min":{"1min":0.33900000000000002,"5min":0.33100000000000002,"15min":0.252},"max":{"1min":3.5489999999999999,"5min":31.57,"15min":31.57},"last":0.60099999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:14:56 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.9450000000000001,"15min":1.2330000000000001},"min":{"1min":0.35099999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":3.7970000000000002,"5min":58.799999999999997,"15min":58.799999999999997},"last":2.3969999999999998},{"interface":"front","average":{"1min":1.196,"5min":2.0419999999999998,"15min":1.2170000000000001},"min":{"1min":0.30399999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.593,"5min":58.615000000000002,"15min":58.615000000000002},"last":0.49099999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:15:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":1.365,"15min":1.1200000000000001},"min":{"1min":0.45900000000000002,"5min":0.36299999999999999,"15min":0.308},"max":{"1min":2.0739999999999998,"5min":7.5099999999999998,"15min":15.859},"last":3.323},{"interface":"front","average":{"1min":0.67200000000000004,"5min":1.1919999999999999,"15min":0.98699999999999999},"min":{"1min":0.40799999999999997,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":1.9399999999999999,"5min":6.1820000000000004,"15min":6.1820000000000004},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Feb 5 08:15:49 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":1.339,"15min":1.099},"min":{"1min":0.40999999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.5089999999999999,"5min":7.3600000000000003,"15min":15.986000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.464,"15min":1.1899999999999999},"min":{"1min":0.47599999999999998,"5min":0.38,"15min":0.35299999999999998},"max":{"1min":2.3599999999999999,"5min":7.4779999999999998,"15min":15.951000000000001},"last":3.5369999999999999}]}]},{"osd":3,"up_from":0,"seq":115964117152,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953627,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1436724,"kb_used_data":1090412,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92308428,"statfs":{"total":95995035648,"available":94523830272,"internally_reserved":0,"allocated":1116581888,"data_stored":1116155870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":1.6819999999999999,"15min":1.4890000000000001},"min":{"1min":0.26800000000000002,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.4769999999999999,"5min":39.552,"15min":73.078999999999994},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.6990000000000001,"15min":1.444},"min":{"1min":0.29999999999999999,"5min":0.214,"15min":0.21099999999999999},"max":{"1min":3.0489999999999999,"5min":39.350999999999999,"15min":73.412000000000006},"last":0.61399999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":2.536,"15min":1.351},"min":{"1min":0.33700000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":1.6419999999999999,"5min":125.054,"15min":125.054},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.872,"5min":2.5910000000000002,"15min":1.2929999999999999},"min":{"1min":0.308,"5min":0.249,"15min":0.23499999999999999},"max":{"1min":5.1260000000000003,"5min":125.08799999999999,"15min":125.08799999999999},"last":0.27400000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.995,"15min":1.1379999999999999},"min":{"1min":0.28199999999999997,"5min":0.247,"15min":0.24199999999999999},"max":{"1min":1.9750000000000001,"5min":5.649,"15min":96.849999999999994},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":1.0700000000000001,"15min":1.1970000000000001},"min":{"1min":0.41799999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.79,"5min":4.1299999999999999,"15min":96.143000000000001},"last":0.68400000000000005}]},{"osd":3,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.9950000000000001,"15min":1.698},"min":{"1min":0.376,"5min":0.28599999999999998,"15min":0.24399999999999999},"max":{"1min":2.1160000000000001,"5min":57.204000000000001,"15min":73.602999999999994},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":2.0510000000000002,"15min":1.7210000000000001},"min":{"1min":0.36899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.76400000000000001,"5min":57.606999999999999,"15min":74.817999999999998},"last":1.0609999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:15:02 2024","interfaces":[{"interface":"back","average":{"1min":2.7690000000000001,"5min":1.0780000000000001,"15min":0.61699999999999999},"min":{"1min":0.39900000000000002,"5min":0.224,"15min":0.224},"max":{"1min":34.167000000000002,"5min":34.167000000000002,"15min":34.167000000000002},"last":1.909},{"interface":"front","average":{"1min":3.1789999999999998,"5min":1.252,"15min":0.69599999999999995},"min":{"1min":0.29599999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":36.051000000000002,"5min":36.051000000000002,"15min":36.051000000000002},"last":0.71399999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.72699999999999998,"15min":0.53700000000000003},"min":{"1min":0.37,"5min":0.29599999999999999,"15min":0.29299999999999998},"max":{"1min":2.0800000000000001,"5min":3.9220000000000002,"15min":3.9220000000000002},"last":2.5139999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.64500000000000002,"15min":0.51000000000000001},"min":{"1min":0.47099999999999997,"5min":0.32000000000000001,"15min":0.29499999999999998},"max":{"1min":0.83799999999999997,"5min":3.306,"15min":3.306},"last":0.999}]},{"osd":7,"last update":"Mon Feb 5 08:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.83799999999999997,"15min":0.59399999999999997},"min":{"1min":0.38600000000000001,"5min":0.314,"15min":0.27000000000000002},"max":{"1min":2.633,"5min":4.4859999999999998,"15min":4.4859999999999998},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.081,"5min":0.81299999999999994,"15min":0.60799999999999998},"min":{"1min":0.41599999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.5519999999999996,"5min":5.7450000000000001,"15min":5.7450000000000001},"last":1.0329999999999999}]}]},{"osd":5,"up_from":42,"seq":180388626581,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3066732,"kb_used_data":2714852,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":90678420,"statfs":{"total":95995035648,"available":92854702080,"internally_reserved":0,"allocated":2780008448,"data_stored":2779591579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:15:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.8360000000000001,"15min":1.5920000000000001},"min":{"1min":0.23499999999999999,"5min":0.222,"15min":0.19400000000000001},"max":{"1min":4.7830000000000004,"5min":46.962000000000003,"15min":143.06299999999999},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.7130000000000001,"15min":1.546},"min":{"1min":0.27900000000000003,"5min":0.24299999999999999,"15min":0.20499999999999999},"max":{"1min":4.6180000000000003,"5min":47.067,"15min":143.09700000000001},"last":1.482}]},{"osd":1,"last update":"Mon Feb 5 08:15:51 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.661,"15min":1.002},"min":{"1min":0.441,"5min":0.25,"15min":0.215},"max":{"1min":4.7450000000000001,"5min":46.200000000000003,"15min":46.200000000000003},"last":1.6060000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.7589999999999999,"15min":1.087},"min":{"1min":0.246,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.6589999999999998,"5min":46.343000000000004,"15min":46.343000000000004},"last":1.522}]},{"osd":2,"last update":"Mon Feb 5 08:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.179,"15min":0.93799999999999994},"min":{"1min":0.28999999999999998,"5min":0.246,"15min":0.246},"max":{"1min":4.0750000000000002,"5min":4.7729999999999997,"15min":10.733000000000001},"last":3.5099999999999998},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.327,"15min":1},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26400000000000001},"max":{"1min":5.351,"5min":8.0220000000000002,"15min":8.0220000000000002},"last":0.26700000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:15:51 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":1.597,"15min":1.556},"min":{"1min":0.315,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.1029999999999998,"5min":46.569000000000003,"15min":143.03399999999999},"last":3.548},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.986,"15min":1.679},"min":{"1min":0.376,"5min":0.32600000000000001,"15min":0.28100000000000003},"max":{"1min":5.3840000000000003,"5min":47.820999999999998,"15min":143.46799999999999},"last":1.669}]},{"osd":4,"last update":"Mon Feb 5 08:15:51 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":0.97799999999999998,"15min":0.60699999999999998},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.3769999999999998,"5min":5.3769999999999998,"15min":5.3769999999999998},"last":2.9609999999999999},{"interface":"front","average":{"1min":1.073,"5min":0.77700000000000002,"15min":0.55200000000000005},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":4.0090000000000003,"5min":4.0090000000000003,"15min":4.0090000000000003},"last":3.5899999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:15:17 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.73799999999999999,"15min":0.54400000000000004},"min":{"1min":0.314,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":3.5859999999999999,"5min":3.5859999999999999,"15min":3.5859999999999999},"last":3.6200000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.76200000000000001,"15min":0.57999999999999996},"min":{"1min":0.33800000000000002,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":3.6859999999999999,"5min":6.6180000000000003,"15min":6.6180000000000003},"last":3.4689999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:15:51 2024","interfaces":[{"interface":"back","average":{"1min":2.2240000000000002,"5min":1.1000000000000001,"15min":0.68700000000000006},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.25900000000000001},"max":{"1min":9.6940000000000008,"5min":9.6940000000000008,"15min":9.6940000000000008},"last":1.639},{"interface":"front","average":{"1min":1.2509999999999999,"5min":0.94199999999999995,"15min":0.64600000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.28199999999999997},"max":{"1min":8.0289999999999999,"5min":8.0289999999999999,"15min":8.0289999999999999},"last":2.9929999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1725038592,"data_stored":1725038592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2568552448,"data_stored":2568552448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1404960768,"data_stored":1404960768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1130233856,"data_stored":1130233856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2824601600,"data_stored":2824601600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":852951040,"data_stored":852951040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3092054016,"data_stored":3092054016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:15:59.128 INFO:tasks.thrashosds.thrasher:Setting 6.6 to [4, 1, 3] 2024-02-05T08:15:59.129 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.6', '4', '1', '3'] 2024-02-05T08:15:59.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pg-upmap 6.6 4 1 3 2024-02-05T08:15:59.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:59.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:59.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:59.161 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:15:59.239 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:15:59.282 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:59.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:59.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:15:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:58 smithi196 ceph-mon[33018]: 3.a scrub starts 2024-02-05T08:15:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:58 smithi196 ceph-mon[33018]: 3.a scrub ok 2024-02-05T08:15:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:58 smithi196 ceph-mon[33018]: pgmap v500: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 640 B/s rd, 57 MiB/s wr, 914 op/s 2024-02-05T08:15:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:58 smithi196 ceph-mon[33018]: 5.b scrub starts 2024-02-05T08:15:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:58 smithi196 ceph-mon[33018]: 4.14 scrub starts 2024-02-05T08:15:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:58 smithi196 ceph-mon[33018]: 5.b scrub ok 2024-02-05T08:15:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:58 smithi196 ceph-mon[33018]: 4.14 scrub ok 2024-02-05T08:15:59.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:15:59.428 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:59.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:59.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:15:59.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 45 16 76656 76640 106.426 64.25 0.0113188 0.00939121 2024-02-05T08:15:59.529 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:15:59.554 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:59.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:59.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:15:59.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:15:59.691 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:59.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:59.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:15:59.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:15:59.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:59.833 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:15:59.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:15:59.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:15:59.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:15:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:15:59.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:15:59.997 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:59 smithi134 ceph-mon[30086]: 5.2 scrub starts 2024-02-05T08:15:59.997 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:59 smithi134 ceph-mon[30086]: 5.2 scrub ok 2024-02-05T08:15:59.997 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:59 smithi134 ceph-mon[30086]: 4.7 scrub starts 2024-02-05T08:15:59.997 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:59 smithi134 ceph-mon[30086]: 4.7 scrub ok 2024-02-05T08:15:59.997 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:59 smithi134 ceph-mon[30086]: 3.c scrub starts 2024-02-05T08:15:59.998 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:15:59 smithi134 ceph-mon[30086]: 3.c scrub ok 2024-02-05T08:15:59.998 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:59 smithi134 ceph-mon[39278]: 5.2 scrub starts 2024-02-05T08:15:59.998 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:59 smithi134 ceph-mon[39278]: 5.2 scrub ok 2024-02-05T08:15:59.998 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:59 smithi134 ceph-mon[39278]: 4.7 scrub starts 2024-02-05T08:15:59.998 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:59 smithi134 ceph-mon[39278]: 4.7 scrub ok 2024-02-05T08:15:59.998 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:59 smithi134 ceph-mon[39278]: 3.c scrub starts 2024-02-05T08:15:59.998 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:15:59 smithi134 ceph-mon[39278]: 3.c scrub ok 2024-02-05T08:16:00.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:00.109 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:00.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:00.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:00.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:00.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:00.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:59 smithi196 ceph-mon[33018]: 5.2 scrub starts 2024-02-05T08:16:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:59 smithi196 ceph-mon[33018]: 5.2 scrub ok 2024-02-05T08:16:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:59 smithi196 ceph-mon[33018]: 4.7 scrub starts 2024-02-05T08:16:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:59 smithi196 ceph-mon[33018]: 4.7 scrub ok 2024-02-05T08:16:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:59 smithi196 ceph-mon[33018]: 3.c scrub starts 2024-02-05T08:16:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:15:59 smithi196 ceph-mon[33018]: 3.c scrub ok 2024-02-05T08:16:00.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:00.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:00.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:00.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:00.478 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:00.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 46 16 77872 77856 105.764 76 0.017106 0.00944996 2024-02-05T08:16:00.511 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:00.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:00.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:00.648 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:00.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:00.750 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:00.782 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:00.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:00.883 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:00.915 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:00.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:00.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:01.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:01.051 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:01.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:01.126 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:00 smithi134 ceph-mon[30086]: pgmap v501: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-02-05T08:16:01.126 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:00 smithi134 ceph-mon[30086]: 4.4 scrub starts 2024-02-05T08:16:01.126 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:00 smithi134 ceph-mon[30086]: 4.4 scrub ok 2024-02-05T08:16:01.126 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:00 smithi134 ceph-mon[39278]: pgmap v501: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-02-05T08:16:01.126 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:00 smithi134 ceph-mon[39278]: 4.4 scrub starts 2024-02-05T08:16:01.126 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:00 smithi134 ceph-mon[39278]: 4.4 scrub ok 2024-02-05T08:16:01.153 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:01.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:01.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:01.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:01.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:00 smithi196 ceph-mon[33018]: pgmap v501: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-02-05T08:16:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:00 smithi196 ceph-mon[33018]: 4.4 scrub starts 2024-02-05T08:16:01.238 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:00 smithi196 ceph-mon[33018]: 4.4 scrub ok 2024-02-05T08:16:01.287 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:01.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:01.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:01.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:01.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:01.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:01.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:01.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:01.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 47 16 79130 79114 105.187 78.625 0.0062336 0.00950174 2024-02-05T08:16:01.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:01.615 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:01.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:01.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:01.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:01.762 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:01.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:01.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:01.863 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:01.882 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:16:01.924 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:01.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:01.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:02.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:02.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:01 smithi134 ceph-mon[30086]: 6.0 scrub starts 2024-02-05T08:16:02.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:01 smithi134 ceph-mon[30086]: 6.1d scrub starts 2024-02-05T08:16:02.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:01 smithi134 ceph-mon[30086]: 6.0 scrub ok 2024-02-05T08:16:02.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:01 smithi134 ceph-mon[30086]: 6.1d scrub ok 2024-02-05T08:16:02.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:01 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/149701355' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:16:02.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:01 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:16:02.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:01 smithi134 ceph-mon[39278]: 6.0 scrub starts 2024-02-05T08:16:02.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:01 smithi134 ceph-mon[39278]: 6.1d scrub starts 2024-02-05T08:16:02.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:01 smithi134 ceph-mon[39278]: 6.0 scrub ok 2024-02-05T08:16:02.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:01 smithi134 ceph-mon[39278]: 6.1d scrub ok 2024-02-05T08:16:02.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:01 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/149701355' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:16:02.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:01 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:16:02.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:02.062 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:02.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:02.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:01 smithi196 ceph-mon[33018]: 6.0 scrub starts 2024-02-05T08:16:02.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:01 smithi196 ceph-mon[33018]: 6.1d scrub starts 2024-02-05T08:16:02.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:01 smithi196 ceph-mon[33018]: 6.0 scrub ok 2024-02-05T08:16:02.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:01 smithi196 ceph-mon[33018]: 6.1d scrub ok 2024-02-05T08:16:02.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:01 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/149701355' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:16:02.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:01 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:16:02.163 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:02.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:02.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:02.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:02.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:02.339 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:16:02.339 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:16:02.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:02.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:02.442 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:02.482 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:02.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:02.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:02.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 48 16 80304 80288 104.524 73.375 0.00738535 0.0095618 2024-02-05T08:16:02.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:02.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:02.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:02.697 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:02.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:02.763 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:16:02.764 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:02.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:02.794 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:16:02.827 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:16:02.859 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:16:02.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:02.902 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:02.902 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:16:02.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:02.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:02.939 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:16:02.974 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:16:03.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:03.018 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:16:03.021 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:16:03.021 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:16:03.021 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:16:03.021 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:16:03.022 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:16:03.022 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:16:03.022 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:16:03.022 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:16:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:03.045 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:03.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:03.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:02 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:16:03.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:02 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:16:03.054 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:02 smithi196 ceph-mon[33018]: pgmap v502: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 571 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-05T08:16:03.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:02 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:16:03.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:02 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:16:03.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:02 smithi196 ceph-mon[33018]: 4.6 scrub starts 2024-02-05T08:16:03.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:02 smithi196 ceph-mon[33018]: 4.c scrub starts 2024-02-05T08:16:03.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:02 smithi196 ceph-mon[33018]: 4.6 scrub ok 2024-02-05T08:16:03.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:02 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:16:03.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:02 smithi196 ceph-mon[33018]: 4.c scrub ok 2024-02-05T08:16:03.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:02 smithi196 ceph-mon[33018]: osdmap e262: 8 total, 8 up, 7 in 2024-02-05T08:16:03.055 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:02 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1977008007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:03.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T08:16:03.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:03.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:03.198 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:03.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:03.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:16:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:16:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[30086]: pgmap v502: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 571 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-05T08:16:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:16:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:16:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[30086]: 4.6 scrub starts 2024-02-05T08:16:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[30086]: 4.c scrub starts 2024-02-05T08:16:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[30086]: 4.6 scrub ok 2024-02-05T08:16:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:16:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[30086]: 4.c scrub ok 2024-02-05T08:16:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[30086]: osdmap e262: 8 total, 8 up, 7 in 2024-02-05T08:16:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1977008007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:03.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:16:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:16:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[39278]: pgmap v502: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 571 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-05T08:16:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:16:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:16:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[39278]: 4.6 scrub starts 2024-02-05T08:16:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[39278]: 4.c scrub starts 2024-02-05T08:16:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[39278]: 4.6 scrub ok 2024-02-05T08:16:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:16:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[39278]: 4.c scrub ok 2024-02-05T08:16:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[39278]: osdmap e262: 8 total, 8 up, 7 in 2024-02-05T08:16:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:02 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1977008007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:03.299 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:03.333 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:03.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:03.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:03.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:03.471 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:03.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:03.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 49 16 81496 81480 103.911 74.5 0.00388001 0.0096186 2024-02-05T08:16:03.544 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:16:03.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:03.602 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:03.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:03.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:03.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:03.741 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:03.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:03.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:03.764 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:16:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:16:03.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:03.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:03.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:03.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:03.986 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:04.018 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:04.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:03 smithi196 ceph-mon[33018]: 6.1f scrub starts 2024-02-05T08:16:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:03 smithi196 ceph-mon[33018]: 6.1f scrub ok 2024-02-05T08:16:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:03 smithi196 ceph-mon[33018]: 4.2 scrub starts 2024-02-05T08:16:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:03 smithi196 ceph-mon[33018]: 5.7 deep-scrub starts 2024-02-05T08:16:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:03 smithi196 ceph-mon[33018]: 4.2 scrub ok 2024-02-05T08:16:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:03 smithi196 ceph-mon[33018]: 5.7 deep-scrub ok 2024-02-05T08:16:04.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:03 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:04.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:04.157 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:04.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:04.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[30086]: 6.1f scrub starts 2024-02-05T08:16:04.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[30086]: 6.1f scrub ok 2024-02-05T08:16:04.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[30086]: 4.2 scrub starts 2024-02-05T08:16:04.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[30086]: 5.7 deep-scrub starts 2024-02-05T08:16:04.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[30086]: 4.2 scrub ok 2024-02-05T08:16:04.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[30086]: 5.7 deep-scrub ok 2024-02-05T08:16:04.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:04.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[39278]: 6.1f scrub starts 2024-02-05T08:16:04.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[39278]: 6.1f scrub ok 2024-02-05T08:16:04.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[39278]: 4.2 scrub starts 2024-02-05T08:16:04.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[39278]: 5.7 deep-scrub starts 2024-02-05T08:16:04.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[39278]: 4.2 scrub ok 2024-02-05T08:16:04.243 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[39278]: 5.7 deep-scrub ok 2024-02-05T08:16:04.243 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:03 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:04.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:04.304 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:04.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:04.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:04.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:04.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:04.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:04.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 50 16 82710 82694 103.35 75.875 0.00884916 0.0096714 2024-02-05T08:16:04.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:04.585 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:04.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:04.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:04.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:04.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:04.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:04.867 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:04.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:04.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:04.969 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:04.999 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:05.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:05.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:05.101 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:05.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:05.144 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:05.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:05.190 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:04 smithi196 ceph-mon[33018]: pgmap v504: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-05T08:16:05.190 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:04 smithi196 ceph-mon[33018]: 4.18 scrub starts 2024-02-05T08:16:05.190 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:04 smithi196 ceph-mon[33018]: 4.18 scrub ok 2024-02-05T08:16:05.190 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:04 smithi196 ceph-mon[33018]: 6.13 deep-scrub starts 2024-02-05T08:16:05.190 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:04 smithi196 ceph-mon[33018]: 6.4 scrub starts 2024-02-05T08:16:05.190 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:04 smithi196 ceph-mon[33018]: 6.4 scrub ok 2024-02-05T08:16:05.190 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:04 smithi196 ceph-mon[33018]: 6.13 deep-scrub ok 2024-02-05T08:16:05.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[30086]: pgmap v504: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-05T08:16:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[30086]: 4.18 scrub starts 2024-02-05T08:16:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[30086]: 4.18 scrub ok 2024-02-05T08:16:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[30086]: 6.13 deep-scrub starts 2024-02-05T08:16:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[30086]: 6.4 scrub starts 2024-02-05T08:16:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[30086]: 6.4 scrub ok 2024-02-05T08:16:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[30086]: 6.13 deep-scrub ok 2024-02-05T08:16:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[39278]: pgmap v504: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.18k op/s 2024-02-05T08:16:05.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[39278]: 4.18 scrub starts 2024-02-05T08:16:05.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[39278]: 4.18 scrub ok 2024-02-05T08:16:05.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[39278]: 6.13 deep-scrub starts 2024-02-05T08:16:05.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[39278]: 6.4 scrub starts 2024-02-05T08:16:05.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[39278]: 6.4 scrub ok 2024-02-05T08:16:05.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:04 smithi134 ceph-mon[39278]: 6.13 deep-scrub ok 2024-02-05T08:16:05.286 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:05.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:05.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:05.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:05.428 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:05.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:05.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 51 16 83937 83921 102.827 76.6875 0.0211998 0.00971906 2024-02-05T08:16:05.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:05.573 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:05.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:05.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:05.696 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:16:05.696 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:16:05.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:16:05.696 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:16:05.696 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.696 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.14802.0:20686 7.5 7:ae49ae8c:::benchmark_data_smithi134_83016_object20685:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T08:16:05.696 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:06:59.672494+0000", 2024-02-05T08:16:05.696 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 546.01884757599998, 2024-02-05T08:16:05.696 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.150571976, 2024-02-05T08:16:05.696 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.696 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.14802", 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 20686 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.672494+0000", 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.697 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.672494+0000", 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.672495+0000", 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.15e-06 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.672549+0000", 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3859000000000001e-05 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.698 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.672560+0000", 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0689e-05 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.672567+0000", 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0369999999999997e-06 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.672616+0000", 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9359e-05 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.672746+0000", 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000129734 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,2", 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.672896+0000", 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015028900000000001 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.673637+0000", 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00074056199999999995 2024-02-05T08:16:05.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.674644+0000", 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001007567 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.822989+0000", 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14834499100000001 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.823039+0000", 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9443000000000001e-05 2024-02-05T08:16:05.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:06:59.823066+0000", 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7296000000000001e-05 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:27898 9.e 9:752e73bf:::benchmark_data_smithi134_144349_object27897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.030994+0000", 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 40.660347563999998, 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17265594500000001, 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.702 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 27898 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.030994+0000", 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.030994+0000", 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.030996+0000", 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.643e-06 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031038+0000", 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2579e-05 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031051+0000", 2024-02-05T08:16:05.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2975e-05 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031057+0000", 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8409999999999996e-06 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031115+0000", 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7326e-05 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031214+0000", 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9677000000000001e-05 2024-02-05T08:16:05.705 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031358+0000", 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000143454 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032987+0000", 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001629098 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033183+0000", 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019666400000000001 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.203561+0000", 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.170377946 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.203615+0000", 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3207000000000001e-05 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.203650+0000", 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5534999999999998e-05 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.707 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.708 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.708 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:27895 9.5 e249/239 9:a349bc50:::benchmark_data_smithi134_144349_object27894:head v 249'1711, mlcod=249'1707)", 2024-02-05T08:16:05.708 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.032312+0000", 2024-02-05T08:16:05.708 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 40.659029402000002, 2024-02-05T08:16:05.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16973406999999999, 2024-02-05T08:16:05.708 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.708 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.708 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032312+0000", 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032312+0000", 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032313+0000", 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5400000000000002e-07 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032354+0000", 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1622999999999998e-05 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032365+0000", 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0076e-05 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032371+0000", 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2360000000000001e-06 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032515+0000", 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000144141 2024-02-05T08:16:05.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032533+0000", 2024-02-05T08:16:05.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8264e-05 2024-02-05T08:16:05.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033555+0000", 2024-02-05T08:16:05.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001022226 2024-02-05T08:16:05.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202046+0000", 2024-02-05T08:16:05.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16849105 2024-02-05T08:16:05.712 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.712 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.712 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.712 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:27899 9.9 e249/239 9:9d77ddc3:::benchmark_data_smithi134_144349_object27898:head v 249'1791, mlcod=249'1789)", 2024-02-05T08:16:05.712 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.032390+0000", 2024-02-05T08:16:05.712 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 40.658951895000001, 2024-02-05T08:16:05.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.169799125, 2024-02-05T08:16:05.712 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.712 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032390+0000", 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032390+0000", 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032390+0000", 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0600000000000001e-07 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032423+0000", 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2428999999999997e-05 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032430+0000", 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6629999999999993e-06 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032436+0000", 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.02e-06 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032599+0000", 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016209300000000001 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032618+0000", 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9505e-05 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202151+0000", 2024-02-05T08:16:05.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.169532718 2024-02-05T08:16:05.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202189+0000", 2024-02-05T08:16:05.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8291000000000003e-05 2024-02-05T08:16:05.716 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.716 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.716 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.716 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:27902 9.d 9:b50a7d0d:::benchmark_data_smithi134_144349_object27901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-02-05T08:16:05.716 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.032502+0000", 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 40.658839575000002, 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17749103499999999, 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 27902 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032502+0000", 2024-02-05T08:16:05.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032502+0000", 2024-02-05T08:16:05.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032503+0000", 2024-02-05T08:16:05.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0999999999999997e-07 2024-02-05T08:16:05.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032531+0000", 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7786000000000001e-05 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032541+0000", 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0200999999999999e-05 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032548+0000", 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9439999999999999e-06 2024-02-05T08:16:05.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032710+0000", 2024-02-05T08:16:05.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016212600000000001 2024-02-05T08:16:05.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032854+0000", 2024-02-05T08:16:05.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014396499999999999 2024-02-05T08:16:05.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032987+0000", 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000132447 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202061+0000", 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16907446300000001 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202865+0000", 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080343400000000001 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.721 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.209910+0000", 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0070456900000000003 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.209960+0000", 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0050999999999999e-05 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.209993+0000", 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3018000000000001e-05 2024-02-05T08:16:05.722 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.723 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.723 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.723 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.723 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.723 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:27907 9.e 9:74a14f65:::benchmark_data_smithi134_144349_object27906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-02-05T08:16:05.723 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.033274+0000", 2024-02-05T08:16:05.724 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 40.658067991000003, 2024-02-05T08:16:05.724 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17237598000000001, 2024-02-05T08:16:05.724 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.724 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.724 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.724 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.724 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.724 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 27907 2024-02-05T08:16:05.724 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.724 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.724 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.724 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.724 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033274+0000", 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033274+0000", 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033275+0000", 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8199999999999998e-07 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.725 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033344+0000", 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9425999999999996e-05 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033355+0000", 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0975000000000001e-05 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033362+0000", 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0539999999999997e-06 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033400+0000", 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7889000000000003e-05 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.201908+0000", 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.168507921 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202081+0000", 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017288399999999999 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.727 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.204290+0000", 2024-02-05T08:16:05.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022091649999999999 2024-02-05T08:16:05.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.204499+0000", 2024-02-05T08:16:05.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000208996 2024-02-05T08:16:05.728 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.728 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.728 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.728 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205559+0000", 2024-02-05T08:16:05.728 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010599380000000001 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205615+0000", 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5642e-05 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205650+0000", 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5207999999999999e-05 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.729 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:27903 9.8 e249/239 9:1ece6453:::benchmark_data_smithi134_144349_object27902:head v 249'1662, mlcod=249'1659)", 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.033685+0000", 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 40.657657014999998, 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17151197500000001, 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033685+0000", 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033685+0000", 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.730 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033685+0000", 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4999999999999998e-07 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033745+0000", 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9487000000000002e-05 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.201990+0000", 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16824555199999999 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.731 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202001+0000", 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0828e-05 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202060+0000", 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8529999999999997e-05 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202082+0000", 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2340999999999998e-05 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205157+0000", 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030755219999999998 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205197+0000", 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9264999999999998e-05 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:35467 9.2 e249/239 9:4b974889:::benchmark_data_smithi134_144349_object35466:head v 249'2223, mlcod=249'2220)", 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:29.630509+0000", 2024-02-05T08:16:05.733 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 36.060832799000003, 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.154489985, 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.630509+0000", 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.630509+0000", 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.734 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.630509+0000", 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9699999999999996e-07 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.630570+0000", 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0089e-05 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.765054+0000", 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.134484564 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.735 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.765061+0000", 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8900000000000001e-06 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.765153+0000", 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2193000000000003e-05 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.765228+0000", 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4952999999999999e-05 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.736 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.784968+0000", 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019740159 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.784999+0000", 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0639999999999998e-05 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35875 9.e 9:7b26ffef:::benchmark_data_smithi134_144349_object35874:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.104276+0000", 2024-02-05T08:16:05.737 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 35.587066256, 2024-02-05T08:16:05.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19477491999999999, 2024-02-05T08:16:05.738 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.738 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.738 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.738 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.738 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.738 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35875 2024-02-05T08:16:05.738 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.738 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.738 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.738 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.738 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104276+0000", 2024-02-05T08:16:05.738 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104276+0000", 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104277+0000", 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0690000000000001e-06 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104323+0000", 2024-02-05T08:16:05.739 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5961999999999998e-05 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104342+0000", 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9038e-05 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104350+0000", 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3189999999999995e-06 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104571+0000", 2024-02-05T08:16:05.740 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022128600000000001 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104577+0000", 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4469999999999997e-06 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.112613+0000", 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0080362550000000008 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.112736+0000", 2024-02-05T08:16:05.741 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000123087 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.112906+0000", 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000169772 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.117780+0000", 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048745769999999997 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.117817+0000", 2024-02-05T08:16:05.742 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6451e-05 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.298971+0000", 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18115403999999999 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299019+0000", 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8164000000000002e-05 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299050+0000", 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1452999999999999e-05 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.743 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35882 9.e 9:79e542b7:::benchmark_data_smithi134_144349_object35881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.111606+0000", 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 35.579736130999997, 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18759089300000001, 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35882 2024-02-05T08:16:05.744 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111606+0000", 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111606+0000", 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111607+0000", 2024-02-05T08:16:05.745 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4810000000000001e-06 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111642+0000", 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4699000000000001e-05 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111652+0000", 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0574000000000001e-05 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.746 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111660+0000", 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2350000000000002e-06 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.114371+0000", 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027112450000000001 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.114476+0000", 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105139 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:16:05.747 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.114563+0000", 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7253e-05 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.117828+0000", 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032642650000000001 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.118007+0000", 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017978199999999999 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.748 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.749 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299094+0000", 2024-02-05T08:16:05.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:05.750 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:05.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:05.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18108685299999999 2024-02-05T08:16:05.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299148+0000", 2024-02-05T08:16:05.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3956999999999999e-05 2024-02-05T08:16:05.751 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.751 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.751 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.751 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299197+0000", 2024-02-05T08:16:05.751 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8409999999999999e-05 2024-02-05T08:16:05.751 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.751 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35887 9.e 9:724dd1b7:::benchmark_data_smithi134_144349_object35886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.113626+0000", 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 35.577716242999998, 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18572786199999999, 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35887 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.752 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113626+0000", 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113626+0000", 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113627+0000", 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2300000000000001e-06 2024-02-05T08:16:05.753 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113653+0000", 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6208e-05 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113664+0000", 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0523000000000001e-05 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113671+0000", 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4510000000000003e-06 2024-02-05T08:16:05.754 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.116160+0000", 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024895170000000001 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.116289+0000", 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000128355 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.116439+0000", 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015028799999999999 2024-02-05T08:16:05.755 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.119562+0000", 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0031226969999999998 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.119795+0000", 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000233551 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299259+0000", 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17946410800000001 2024-02-05T08:16:05.756 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299310+0000", 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0439000000000003e-05 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299353+0000", 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3495000000000003e-05 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.757 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35906 9.e 9:7cbe34f1:::benchmark_data_smithi134_144349_object35905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.125518+0000", 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 35.565824173000003, 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17395559999999999, 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35906 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125518+0000", 2024-02-05T08:16:05.758 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125518+0000", 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125519+0000", 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3960000000000001e-06 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.759 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125570+0000", 2024-02-05T08:16:05.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1084000000000001e-05 2024-02-05T08:16:05.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125586+0000", 2024-02-05T08:16:05.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5903999999999999e-05 2024-02-05T08:16:05.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.760 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.760 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125594+0000", 2024-02-05T08:16:05.760 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9899999999999997e-06 2024-02-05T08:16:05.760 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.760 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125651+0000", 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7179000000000003e-05 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125776+0000", 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012447200000000001 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125916+0000", 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000140013 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.761 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.127316+0000", 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014006859999999999 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.127928+0000", 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00061120399999999996 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299393+0000", 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17146588300000001 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.762 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299440+0000", 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6684999999999998e-05 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299473+0000", 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3104e-05 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.763 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35911 9.e 9:7aaf23ea:::benchmark_data_smithi134_144349_object35910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.129684+0000", 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 35.561658219999998, 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16995434200000001, 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35911 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129684+0000", 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129684+0000", 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129685+0000", 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8589999999999999e-06 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129749+0000", 2024-02-05T08:16:05.765 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3529999999999997e-05 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129764+0000", 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4793999999999999e-05 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129772+0000", 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2109999999999998e-06 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.766 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129822+0000", 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9974e-05 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129921+0000", 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8887000000000004e-05 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.130017+0000", 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6364000000000001e-05 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.131511+0000", 2024-02-05T08:16:05.767 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014936540000000001 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.132132+0000", 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00062069600000000001 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299515+0000", 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16738375999999999 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.768 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299604+0000", 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8363999999999996e-05 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299638+0000", 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4248999999999997e-05 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35921 9.e 9:776a85fb:::benchmark_data_smithi134_144349_object35920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.138253+0000", 2024-02-05T08:16:05.769 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 35.553088303000003, 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.161497483, 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35921 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138253+0000", 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.770 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138253+0000", 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138255+0000", 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.576e-06 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138299+0000", 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4409999999999997e-05 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138310+0000", 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0794e-05 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138318+0000", 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7500000000000003e-06 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138376+0000", 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8372999999999998e-05 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138498+0000", 2024-02-05T08:16:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012176900000000001 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138626+0000", 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012799 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.139830+0000", 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001204199 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140009+0000", 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017819299999999999 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299676+0000", 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.159667061 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299722+0000", 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6164e-05 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299751+0000", 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9204e-05 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.774 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35922 9.e 9:77e64ba8:::benchmark_data_smithi134_144349_object35921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.140433+0000", 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 35.550908829000001, 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15941660299999999, 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35922 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.775 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140433+0000", 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140433+0000", 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140434+0000", 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3099999999999996e-07 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140488+0000", 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4243000000000002e-05 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140498+0000", 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0015e-05 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140503+0000", 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1390000000000003e-06 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140550+0000", 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6978999999999999e-05 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140649+0000", 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9208000000000003e-05 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140739+0000", 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9967000000000001e-05 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.141731+0000", 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00099119300000000002 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.141745+0000", 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4671000000000001e-05 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299783+0000", 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15803773400000001 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299822+0000", 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9428999999999998e-05 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299850+0000", 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7093999999999998e-05 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58423 9.d 9:b730c3b2:::benchmark_data_smithi134_144349_object58422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.759126+0000", 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 20.932215755000001, 2024-02-05T08:16:05.780 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14701034599999999, 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58423 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759126+0000", 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759126+0000", 2024-02-05T08:16:05.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759128+0000", 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.601e-06 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759180+0000", 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2172999999999997e-05 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759194+0000", 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4131000000000001e-05 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759368+0000", 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017358299999999999 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759575+0000", 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0002073 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763082+0000", 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003507058 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.766845+0000", 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003762933 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776323+0000", 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094780709999999994 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776344+0000", 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0774000000000001e-05 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.906047+0000", 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.129703771 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.906098+0000", 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0495999999999999e-05 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.906136+0000", 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8454999999999997e-05 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.785 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58429 9.e 9:74deef27:::benchmark_data_smithi134_144349_object58428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.763196+0000", 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 20.928145679, 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14953992599999999, 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58429 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763196+0000", 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763196+0000", 2024-02-05T08:16:05.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763198+0000", 2024-02-05T08:16:05.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5960000000000001e-06 2024-02-05T08:16:05.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763237+0000", 2024-02-05T08:16:05.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9320999999999998e-05 2024-02-05T08:16:05.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763247+0000", 2024-02-05T08:16:05.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6909999999999992e-06 2024-02-05T08:16:05.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763254+0000", 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8580000000000002e-06 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.766437+0000", 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003183909 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.766559+0000", 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121561 2024-02-05T08:16:05.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.766696+0000", 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013721 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.778872+0000", 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012176200999999999 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.784715+0000", 2024-02-05T08:16:05.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058425609999999996 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912622+0000", 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.127907411 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912685+0000", 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2501000000000003e-05 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912736+0000", 2024-02-05T08:16:05.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1106000000000002e-05 2024-02-05T08:16:05.792 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.792 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.792 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.792 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.792 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58435 9.e 9:7bef458b:::benchmark_data_smithi134_144349_object58434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:16:05.792 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.765269+0000", 2024-02-05T08:16:05.792 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 20.926072452, 2024-02-05T08:16:05.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14758520999999999, 2024-02-05T08:16:05.792 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.792 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.792 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58435 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765269+0000", 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765269+0000", 2024-02-05T08:16:05.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765271+0000", 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.638e-06 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765301+0000", 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0369e-05 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765313+0000", 2024-02-05T08:16:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1481000000000001e-05 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765321+0000", 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6720000000000004e-06 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776013+0000", 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010692657 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776149+0000", 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000135909 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776445+0000", 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00029550300000000002 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.778879+0000", 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024342790000000001 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.784792+0000", 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059131610000000001 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912788+0000", 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12799622299999999 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912832+0000", 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3402000000000002e-05 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912855+0000", 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2915999999999999e-05 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58444 9.3 9:c881298d:::benchmark_data_smithi134_144349_object58443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.777171+0000", 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 20.914170933000001, 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14967462100000001, 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.798 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58444 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777171+0000", 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777171+0000", 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777172+0000", 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.155e-06 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777202+0000", 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0267e-05 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777211+0000", 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9730000000000003e-06 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777218+0000", 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2940000000000002e-06 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.779461+0000", 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002243423 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.779586+0000", 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000125325 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.779714+0000", 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012767599999999999 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.785089+0000", 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053749059999999996 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.787028+0000", 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019386290000000001 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.925411+0000", 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13838302199999999 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.926806+0000", 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013959249999999999 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.926845+0000", 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9026000000000002e-05 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58447 9.3 9:ce75229c:::benchmark_data_smithi134_144349_object58446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.780114+0000", 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 20.911227804999999, 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14685973299999999, 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58447 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.780114+0000", 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.780114+0000", 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.780115+0000", 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5009999999999999e-06 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:16:05.805 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.782413+0000", 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022973640000000001 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.782431+0000", 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7779999999999999e-05 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.782440+0000", 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0759999999999992e-06 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.782633+0000", 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000193487 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.782778+0000", 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014436599999999999 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.782922+0000", 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014404700000000001 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.785359+0000", 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002437874 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.787065+0000", 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017058349999999999 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.926892+0000", 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.139826541 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.926939+0000", 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7029e-05 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.926974+0000", 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4832999999999997e-05 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:16:05.809 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:16:05.851 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:05.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:05.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:05.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:06.022 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:06.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:06.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:06.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:05 smithi196 ceph-mon[33018]: 4.b deep-scrub starts 2024-02-05T08:16:06.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:05 smithi196 ceph-mon[33018]: 4.b deep-scrub ok 2024-02-05T08:16:06.124 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:06.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:06.165 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:06.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:06.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:06.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:05 smithi134 ceph-mon[30086]: 4.b deep-scrub starts 2024-02-05T08:16:06.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:05 smithi134 ceph-mon[30086]: 4.b deep-scrub ok 2024-02-05T08:16:06.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:05 smithi134 ceph-mon[39278]: 4.b deep-scrub starts 2024-02-05T08:16:06.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:05 smithi134 ceph-mon[39278]: 4.b deep-scrub ok 2024-02-05T08:16:06.291 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:16:06.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:06.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:06.403 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:06.446 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:06.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:06.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 52 16 85083 85067 102.227 71.625 0.0337087 0.00977714 2024-02-05T08:16:06.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:06.590 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:06.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:06.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:06.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:06.722 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:06.730 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:06.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:06.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:06.832 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:06.869 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:06.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:06.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T08:16:06.911 INFO:teuthology.orchestra.run.smithi134.stderr:set 6.6 pg_upmap mapping to [4,1,3] 2024-02-05T08:16:06.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:07.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:07.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:07.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:07.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:07.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:07.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:07.233 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:07.274 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:07.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:07.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:06 smithi134 ceph-mon[30086]: pgmap v505: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-05T08:16:07.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:06 smithi134 ceph-mon[30086]: 4.1b scrub starts 2024-02-05T08:16:07.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:06 smithi134 ceph-mon[30086]: 4.1b scrub ok 2024-02-05T08:16:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:06 smithi134 ceph-mon[30086]: 5.4 deep-scrub starts 2024-02-05T08:16:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:06 smithi134 ceph-mon[30086]: 5.4 deep-scrub ok 2024-02-05T08:16:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:06 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3792877404' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [4, 1, 3]}]: dispatch 2024-02-05T08:16:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:06 smithi134 ceph-mon[39278]: pgmap v505: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-05T08:16:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:06 smithi134 ceph-mon[39278]: 4.1b scrub starts 2024-02-05T08:16:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:06 smithi134 ceph-mon[39278]: 4.1b scrub ok 2024-02-05T08:16:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:06 smithi134 ceph-mon[39278]: 5.4 deep-scrub starts 2024-02-05T08:16:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:06 smithi134 ceph-mon[39278]: 5.4 deep-scrub ok 2024-02-05T08:16:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:06 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3792877404' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [4, 1, 3]}]: dispatch 2024-02-05T08:16:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:06 smithi196 ceph-mon[33018]: pgmap v505: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-02-05T08:16:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:06 smithi196 ceph-mon[33018]: 4.1b scrub starts 2024-02-05T08:16:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:06 smithi196 ceph-mon[33018]: 4.1b scrub ok 2024-02-05T08:16:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:06 smithi196 ceph-mon[33018]: 5.4 deep-scrub starts 2024-02-05T08:16:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:06 smithi196 ceph-mon[33018]: 5.4 deep-scrub ok 2024-02-05T08:16:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:06 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3792877404' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [4, 1, 3]}]: dispatch 2024-02-05T08:16:07.375 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:16:07.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:07.421 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:07.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:07.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:07.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 53 16 86217 86201 101.636 70.875 0.0105775 0.00983442 2024-02-05T08:16:07.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:07.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:07.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:07.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:07.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:07.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:07.792 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:07.850 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:07.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:07.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:07.952 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:07.987 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:07.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:07.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:08.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[30086]: 3.14 scrub starts 2024-02-05T08:16:08.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[30086]: 3.14 scrub ok 2024-02-05T08:16:08.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[30086]: 4.8 deep-scrub starts 2024-02-05T08:16:08.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[30086]: 4.8 deep-scrub ok 2024-02-05T08:16:08.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3792877404' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [4, 1, 3]}]': finished 2024-02-05T08:16:08.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[30086]: osdmap e263: 8 total, 8 up, 7 in 2024-02-05T08:16:08.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[30086]: 5.12 scrub starts 2024-02-05T08:16:08.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[30086]: 5.12 scrub ok 2024-02-05T08:16:08.028 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:16:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:16:08.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[39278]: 3.14 scrub starts 2024-02-05T08:16:08.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[39278]: 3.14 scrub ok 2024-02-05T08:16:08.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[39278]: 4.8 deep-scrub starts 2024-02-05T08:16:08.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[39278]: 4.8 deep-scrub ok 2024-02-05T08:16:08.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3792877404' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [4, 1, 3]}]': finished 2024-02-05T08:16:08.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[39278]: osdmap e263: 8 total, 8 up, 7 in 2024-02-05T08:16:08.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[39278]: 5.12 scrub starts 2024-02-05T08:16:08.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:07 smithi134 ceph-mon[39278]: 5.12 scrub ok 2024-02-05T08:16:08.089 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:08.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:08.125 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:08.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:08.226 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:08.262 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:08.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:07 smithi196 ceph-mon[33018]: 3.14 scrub starts 2024-02-05T08:16:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:07 smithi196 ceph-mon[33018]: 3.14 scrub ok 2024-02-05T08:16:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:07 smithi196 ceph-mon[33018]: 4.8 deep-scrub starts 2024-02-05T08:16:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:07 smithi196 ceph-mon[33018]: 4.8 deep-scrub ok 2024-02-05T08:16:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:07 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3792877404' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.6", "id": [4, 1, 3]}]': finished 2024-02-05T08:16:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:07 smithi196 ceph-mon[33018]: osdmap e263: 8 total, 8 up, 7 in 2024-02-05T08:16:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:07 smithi196 ceph-mon[33018]: 5.12 scrub starts 2024-02-05T08:16:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:07 smithi196 ceph-mon[33018]: 5.12 scrub ok 2024-02-05T08:16:08.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:08.400 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:08.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:08.487 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 54 16 87491 87475 101.228 79.625 0.00425651 0.00987443 2024-02-05T08:16:08.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:08.534 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:08.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:08.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:08.678 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:08.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:08.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:08.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:08.820 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:08.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:08.922 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:08.958 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:08.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:09.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:09.095 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:09.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:09.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:09.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:09.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:09.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:09.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:09.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[30086]: pgmap v507: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 826 op/s 2024-02-05T08:16:09.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[30086]: 4.1a deep-scrub starts 2024-02-05T08:16:09.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[30086]: 4.1a deep-scrub ok 2024-02-05T08:16:09.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[30086]: 3.16 deep-scrub starts 2024-02-05T08:16:09.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[30086]: 3.16 deep-scrub ok 2024-02-05T08:16:09.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[30086]: 6.1 scrub starts 2024-02-05T08:16:09.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[30086]: 6.1 scrub ok 2024-02-05T08:16:09.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:16:09.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:16:09.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[39278]: pgmap v507: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 826 op/s 2024-02-05T08:16:09.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[39278]: 4.1a deep-scrub starts 2024-02-05T08:16:09.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[39278]: 4.1a deep-scrub ok 2024-02-05T08:16:09.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[39278]: 3.16 deep-scrub starts 2024-02-05T08:16:09.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[39278]: 3.16 deep-scrub ok 2024-02-05T08:16:09.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[39278]: 6.1 scrub starts 2024-02-05T08:16:09.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[39278]: 6.1 scrub ok 2024-02-05T08:16:09.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:16:09.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:09 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:16:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:09 smithi196 ceph-mon[33018]: pgmap v507: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 826 op/s 2024-02-05T08:16:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:09 smithi196 ceph-mon[33018]: 4.1a deep-scrub starts 2024-02-05T08:16:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:09 smithi196 ceph-mon[33018]: 4.1a deep-scrub ok 2024-02-05T08:16:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:09 smithi196 ceph-mon[33018]: 3.16 deep-scrub starts 2024-02-05T08:16:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:09 smithi196 ceph-mon[33018]: 3.16 deep-scrub ok 2024-02-05T08:16:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:09 smithi196 ceph-mon[33018]: 6.1 scrub starts 2024-02-05T08:16:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:09 smithi196 ceph-mon[33018]: 6.1 scrub ok 2024-02-05T08:16:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:09 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:16:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:09 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:16:09.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:09.370 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:09.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:09.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:09.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:09.485 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 55 16 88789 88773 100.862 81.125 0.0106926 0.00990952 2024-02-05T08:16:09.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:09.516 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:09.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:09.617 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:09.653 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:09.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:09.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:09.711 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:09.715 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:09.756 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:09.812 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:09.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:09.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:09.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:09.950 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:09.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:09.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:10.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:10.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:10.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:10.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:10.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:10.223 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:10.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:10.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:10.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:10 smithi134 ceph-mon[30086]: 4.1f scrub starts 2024-02-05T08:16:10.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:10 smithi134 ceph-mon[30086]: 4.1f scrub ok 2024-02-05T08:16:10.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:10 smithi134 ceph-mon[30086]: 3.6 scrub starts 2024-02-05T08:16:10.241 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:10 smithi134 ceph-mon[30086]: 3.6 scrub ok 2024-02-05T08:16:10.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:10 smithi134 ceph-mon[39278]: 4.1f scrub starts 2024-02-05T08:16:10.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:10 smithi134 ceph-mon[39278]: 4.1f scrub ok 2024-02-05T08:16:10.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:10 smithi134 ceph-mon[39278]: 3.6 scrub starts 2024-02-05T08:16:10.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:10 smithi134 ceph-mon[39278]: 3.6 scrub ok 2024-02-05T08:16:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:10 smithi196 ceph-mon[33018]: 4.1f scrub starts 2024-02-05T08:16:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:10 smithi196 ceph-mon[33018]: 4.1f scrub ok 2024-02-05T08:16:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:10 smithi196 ceph-mon[33018]: 3.6 scrub starts 2024-02-05T08:16:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:10 smithi196 ceph-mon[33018]: 3.6 scrub ok 2024-02-05T08:16:10.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:10.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:10.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:10.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:10.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:10.489 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 56 16 90016 90000 100.427 76.6875 0.00880087 0.00995265 2024-02-05T08:16:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:10.519 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:10.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:10.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:10.665 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:10.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:10.768 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:10.800 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:10.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:10.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:10.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:10.924 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:10.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:11.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:11.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:11.065 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:11.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:11.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:11.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:11.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:11.221 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:11 smithi134 ceph-mon[30086]: pgmap v508: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-05T08:16:11.221 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:11 smithi134 ceph-mon[30086]: 5.5 scrub starts 2024-02-05T08:16:11.222 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:11 smithi134 ceph-mon[30086]: 5.5 scrub ok 2024-02-05T08:16:11.222 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:11 smithi134 ceph-mon[30086]: 4.12 deep-scrub starts 2024-02-05T08:16:11.222 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:11 smithi134 ceph-mon[30086]: 4.12 deep-scrub ok 2024-02-05T08:16:11.222 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:11 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4148030583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:11.222 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:11 smithi134 ceph-mon[39278]: pgmap v508: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-05T08:16:11.222 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:11 smithi134 ceph-mon[39278]: 5.5 scrub starts 2024-02-05T08:16:11.222 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:11 smithi134 ceph-mon[39278]: 5.5 scrub ok 2024-02-05T08:16:11.222 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:11 smithi134 ceph-mon[39278]: 4.12 deep-scrub starts 2024-02-05T08:16:11.223 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:11 smithi134 ceph-mon[39278]: 4.12 deep-scrub ok 2024-02-05T08:16:11.223 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:11 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4148030583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:11.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:11 smithi196 ceph-mon[33018]: pgmap v508: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-02-05T08:16:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:11 smithi196 ceph-mon[33018]: 5.5 scrub starts 2024-02-05T08:16:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:11 smithi196 ceph-mon[33018]: 5.5 scrub ok 2024-02-05T08:16:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:11 smithi196 ceph-mon[33018]: 4.12 deep-scrub starts 2024-02-05T08:16:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:11 smithi196 ceph-mon[33018]: 4.12 deep-scrub ok 2024-02-05T08:16:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:11 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4148030583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:11.338 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:11.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:11.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:11.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:11.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:11.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:11.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:11.488 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 57 16 91190 91174 99.952 73.375 0.013995 0.0100007 2024-02-05T08:16:11.532 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:16:11.532 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:16:11.532 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:16:11.532 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:16:11.532 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:16:11.532 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:16:11.532 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:16:11.532 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:16:11.533 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:16:11.533 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:16:11.533 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:16:11.533 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:16:11.533 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:16:11.533 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:16:11.533 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:16:11.533 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:16:11.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:11.611 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:11.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:11.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:11.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:11.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:11.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:11.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:11.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:11.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:11.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:11.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:11.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:12.021 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:12.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:12.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:12.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:12.157 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:12.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:12.258 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:12.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:12 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:16:12.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:12 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:16:12.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:12 smithi134 ceph-mon[30086]: 3.8 scrub starts 2024-02-05T08:16:12.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:12 smithi134 ceph-mon[30086]: 3.8 scrub ok 2024-02-05T08:16:12.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:12 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:16:12.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:12 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:16:12.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:12 smithi134 ceph-mon[39278]: 3.8 scrub starts 2024-02-05T08:16:12.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:12 smithi134 ceph-mon[39278]: 3.8 scrub ok 2024-02-05T08:16:12.290 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:12.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:12.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:12.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:12 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:16:12.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:12 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:16:12.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:12 smithi196 ceph-mon[33018]: 3.8 scrub starts 2024-02-05T08:16:12.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:12 smithi196 ceph-mon[33018]: 3.8 scrub ok 2024-02-05T08:16:12.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:12.428 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:12.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:12.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:12.488 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 58 16 92467 92451 99.6047 79.8125 0.0160654 0.0100347 2024-02-05T08:16:12.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:12.567 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:12.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:12.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:12.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:12.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:12.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:12.816 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:12.858 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:12.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:12.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:12.994 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:12.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:12.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:13.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:13.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:13.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:13.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:13.272 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:13.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:13 smithi196 ceph-mon[33018]: pgmap v509: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-02-05T08:16:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:13 smithi196 ceph-mon[33018]: 3.1d deep-scrub starts 2024-02-05T08:16:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:13 smithi196 ceph-mon[33018]: 3.3 scrub starts 2024-02-05T08:16:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:13 smithi196 ceph-mon[33018]: 3.1d deep-scrub ok 2024-02-05T08:16:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:13 smithi196 ceph-mon[33018]: 3.3 scrub ok 2024-02-05T08:16:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:13 smithi196 ceph-mon[33018]: 4.1d scrub starts 2024-02-05T08:16:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:13 smithi196 ceph-mon[33018]: 4.1d scrub ok 2024-02-05T08:16:13.374 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:13.398 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[30086]: pgmap v509: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-02-05T08:16:13.398 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[30086]: 3.1d deep-scrub starts 2024-02-05T08:16:13.398 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[30086]: 3.3 scrub starts 2024-02-05T08:16:13.398 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[30086]: 3.1d deep-scrub ok 2024-02-05T08:16:13.398 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[30086]: 3.3 scrub ok 2024-02-05T08:16:13.398 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[30086]: 4.1d scrub starts 2024-02-05T08:16:13.398 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[30086]: 4.1d scrub ok 2024-02-05T08:16:13.398 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[39278]: pgmap v509: 148 pgs: 148 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-02-05T08:16:13.398 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[39278]: 3.1d deep-scrub starts 2024-02-05T08:16:13.398 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[39278]: 3.3 scrub starts 2024-02-05T08:16:13.398 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[39278]: 3.1d deep-scrub ok 2024-02-05T08:16:13.398 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[39278]: 3.3 scrub ok 2024-02-05T08:16:13.399 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[39278]: 4.1d scrub starts 2024-02-05T08:16:13.399 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:13 smithi134 ceph-mon[39278]: 4.1d scrub ok 2024-02-05T08:16:13.402 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:13.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:13.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:13.488 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 59 16 93699 93683 99.2215 77 0.025066 0.0100737 2024-02-05T08:16:13.505 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:13.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:13.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:13.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:13.661 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:16:13.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 0 1 2024-02-05T08:16:13.684 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:13.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:13.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:13.786 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:13.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:16:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:16:13.825 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:13.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:13.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:13.926 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:13.967 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:13.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:13.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:14.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:14.094 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:14.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:14.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:14.238 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:14.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:14.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:14 smithi196 ceph-mon[33018]: 6.d scrub starts 2024-02-05T08:16:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:14 smithi196 ceph-mon[33018]: 4.e scrub starts 2024-02-05T08:16:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:14 smithi196 ceph-mon[33018]: 4.e scrub ok 2024-02-05T08:16:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:14 smithi196 ceph-mon[33018]: 5.13 scrub starts 2024-02-05T08:16:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:14 smithi196 ceph-mon[33018]: 5.13 scrub ok 2024-02-05T08:16:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:14 smithi196 ceph-mon[33018]: 6.d scrub ok 2024-02-05T08:16:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:14 smithi196 ceph-mon[33018]: 5.10 scrub starts 2024-02-05T08:16:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:14 smithi196 ceph-mon[33018]: 5.10 scrub ok 2024-02-05T08:16:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:14 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:14.316 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[30086]: 6.d scrub starts 2024-02-05T08:16:14.316 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[30086]: 4.e scrub starts 2024-02-05T08:16:14.316 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[30086]: 4.e scrub ok 2024-02-05T08:16:14.316 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[30086]: 5.13 scrub starts 2024-02-05T08:16:14.316 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[30086]: 5.13 scrub ok 2024-02-05T08:16:14.316 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[30086]: 6.d scrub ok 2024-02-05T08:16:14.316 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[30086]: 5.10 scrub starts 2024-02-05T08:16:14.316 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[30086]: 5.10 scrub ok 2024-02-05T08:16:14.316 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:14.316 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[39278]: 6.d scrub starts 2024-02-05T08:16:14.316 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[39278]: 4.e scrub starts 2024-02-05T08:16:14.317 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[39278]: 4.e scrub ok 2024-02-05T08:16:14.317 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[39278]: 5.13 scrub starts 2024-02-05T08:16:14.317 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[39278]: 5.13 scrub ok 2024-02-05T08:16:14.317 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[39278]: 6.d scrub ok 2024-02-05T08:16:14.317 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[39278]: 5.10 scrub starts 2024-02-05T08:16:14.317 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[39278]: 5.10 scrub ok 2024-02-05T08:16:14.317 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:14 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:14.340 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:14.369 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:14.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:14.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:14.488 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:16:14.489532+0000 min lat: 0.00130673 max lat: 0.198515 avg lat: 0.0101383 2024-02-05T08:16:14.488 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:16:14.489 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 60 16 94678 94662 98.5876 61.1875 0.012477 0.0101383 2024-02-05T08:16:14.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:14.521 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:14.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:14.622 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:14.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:14.658 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:14.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:14.684 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:14.760 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:14.797 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:14.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:14.801 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:16:14.899 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:14.935 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:14.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:15.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:15.072 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:15.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:15.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:15.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:15.211 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:15.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:15.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:15.339 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:16:15.339 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:16:15.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:15.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:15.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:15.355 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[30086]: 9.1 scrub starts 2024-02-05T08:16:15.355 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[30086]: pgmap v510: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:16:15.355 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[30086]: 5.18 scrub starts 2024-02-05T08:16:15.355 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[30086]: 5.18 scrub ok 2024-02-05T08:16:15.355 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[30086]: 5.17 scrub starts 2024-02-05T08:16:15.355 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[30086]: 5.17 scrub ok 2024-02-05T08:16:15.355 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/854832889' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:16:15.355 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:16:15.355 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[39278]: 9.1 scrub starts 2024-02-05T08:16:15.355 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[39278]: pgmap v510: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:16:15.355 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[39278]: 5.18 scrub starts 2024-02-05T08:16:15.356 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[39278]: 5.18 scrub ok 2024-02-05T08:16:15.356 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[39278]: 5.17 scrub starts 2024-02-05T08:16:15.356 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[39278]: 5.17 scrub ok 2024-02-05T08:16:15.356 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/854832889' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:16:15.356 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:15 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:16:15.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:15.488 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 61 16 95749 95733 98.0686 66.9375 0.0135045 0.0101927 2024-02-05T08:16:15.495 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:15.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:15 smithi196 ceph-mon[33018]: 9.1 scrub starts 2024-02-05T08:16:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:15 smithi196 ceph-mon[33018]: pgmap v510: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:16:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:15 smithi196 ceph-mon[33018]: 5.18 scrub starts 2024-02-05T08:16:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:15 smithi196 ceph-mon[33018]: 5.18 scrub ok 2024-02-05T08:16:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:15 smithi196 ceph-mon[33018]: 5.17 scrub starts 2024-02-05T08:16:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:15 smithi196 ceph-mon[33018]: 5.17 scrub ok 2024-02-05T08:16:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:15 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/854832889' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:16:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:15 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:16:15.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:15.630 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:15.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:15.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:15.731 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:15.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:15.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:15.774 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:16:15.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:15.912 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:15.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:16.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:16.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:16.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:16.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:16.183 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:16.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:16.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T08:16:16.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:16.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:16.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:16.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:16.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[30086]: 4.9 scrub starts 2024-02-05T08:16:16.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[30086]: 4.9 scrub ok 2024-02-05T08:16:16.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[30086]: 6.18 scrub starts 2024-02-05T08:16:16.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[30086]: 6.18 scrub ok 2024-02-05T08:16:16.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[30086]: 1.0 scrub starts 2024-02-05T08:16:16.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[30086]: 1.0 scrub ok 2024-02-05T08:16:16.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:16:16.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:16:16.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[30086]: osdmap e264: 8 total, 8 up, 7 in 2024-02-05T08:16:16.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[39278]: 4.9 scrub starts 2024-02-05T08:16:16.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[39278]: 4.9 scrub ok 2024-02-05T08:16:16.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[39278]: 6.18 scrub starts 2024-02-05T08:16:16.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[39278]: 6.18 scrub ok 2024-02-05T08:16:16.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[39278]: 1.0 scrub starts 2024-02-05T08:16:16.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[39278]: 1.0 scrub ok 2024-02-05T08:16:16.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:16:16.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:16:16.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:16 smithi134 ceph-mon[39278]: osdmap e264: 8 total, 8 up, 7 in 2024-02-05T08:16:16.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:16.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:16.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:16.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:16.488 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 62 16 96981 96965 97.7287 77 0.0167901 0.0102283 2024-02-05T08:16:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:16 smithi196 ceph-mon[33018]: 4.9 scrub starts 2024-02-05T08:16:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:16 smithi196 ceph-mon[33018]: 4.9 scrub ok 2024-02-05T08:16:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:16 smithi196 ceph-mon[33018]: 6.18 scrub starts 2024-02-05T08:16:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:16 smithi196 ceph-mon[33018]: 6.18 scrub ok 2024-02-05T08:16:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:16 smithi196 ceph-mon[33018]: 1.0 scrub starts 2024-02-05T08:16:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:16 smithi196 ceph-mon[33018]: 1.0 scrub ok 2024-02-05T08:16:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:16 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:16:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:16 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:16:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:16 smithi196 ceph-mon[33018]: osdmap e264: 8 total, 8 up, 7 in 2024-02-05T08:16:16.569 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:16.608 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:16.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:16.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:16.710 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:16.742 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:16.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:16.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:16.844 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:16.882 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:16.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:16.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:16.983 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:17.018 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:17.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:17.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:17.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:17.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:17.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:17.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:17.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:17 smithi134 ceph-mon[30086]: 9.1 scrub ok 2024-02-05T08:16:17.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:17 smithi134 ceph-mon[30086]: pgmap v511: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:16:17.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:17 smithi134 ceph-mon[39278]: 9.1 scrub ok 2024-02-05T08:16:17.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:17 smithi134 ceph-mon[39278]: pgmap v511: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:16:17.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:17.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:17.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:17.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:17.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:17.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:17.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:17.488 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 63 16 98252 98236 97.4383 79.4375 0.00829909 0.0102584 2024-02-05T08:16:17.546 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:17.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:17 smithi196 ceph-mon[33018]: 9.1 scrub ok 2024-02-05T08:16:17.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:17 smithi196 ceph-mon[33018]: pgmap v511: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:16:17.582 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:17.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:17.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:17.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:17.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:17.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:17.827 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:17.867 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:17.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:17.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:18.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:18.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:18.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:16:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:16:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:16:18.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:18.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:18.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:18.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:18.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:18.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:18.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:18.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:18.386 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:18.420 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:18.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:18.489 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 64 16 99505 99489 97.1394 78.3125 0.0131106 0.0102901 2024-02-05T08:16:18.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:18.555 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:18.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:18.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:18.660 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:18.696 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:18.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:18.797 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:18.840 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:18.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:18.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:18.943 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:18.986 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:18.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:18.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:19.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:19.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:19.124 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:19.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:19.225 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:19.235 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:19.329 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:19.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:19.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:19 smithi134 ceph-mon[30086]: pgmap v513: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-02-05T08:16:19.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:19 smithi134 ceph-mon[39278]: pgmap v513: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-02-05T08:16:19.430 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:19.465 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:19.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:19.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:19.489 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 65 16 100784 100768 96.8747 79.9375 0.0088596 0.0103179 2024-02-05T08:16:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:19 smithi196 ceph-mon[33018]: pgmap v513: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-02-05T08:16:19.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:19.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:19.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:19.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:19.706 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:19.741 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:19.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:19.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:19.878 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:19.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:19.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:20.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:20.013 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:20.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:20.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:20.152 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:20.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:20.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:20.289 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:20.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:20.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:20.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:20.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:20.489 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 66 16 102012 101996 96.5697 76.75 0.00808561 0.0103506 2024-02-05T08:16:20.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:20.567 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:20.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:20.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:20.704 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:20.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:20.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:20.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:20.819 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:16:20.842 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:20.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:20.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:20.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:20.984 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:20.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:21.085 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:21.136 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:21 smithi134 ceph-mon[39278]: pgmap v514: 148 pgs: 148 active+clean; 6.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-02-05T08:16:21.136 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:21 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1664716730' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:16:21.136 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:21 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:16:21.139 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:21.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:21.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:21.171 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.0 to 1 (10000) 2024-02-05T08:16:21.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:21.276 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:21.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:21.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:21.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:21.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:21.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:21.489 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 67 16 103226 103210 96.2607 75.875 0.0126962 0.0103841 2024-02-05T08:16:21.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:21 smithi134 ceph-mon[30086]: pgmap v514: 148 pgs: 148 active+clean; 6.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-02-05T08:16:21.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:21 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1664716730' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:16:21.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:21 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:16:21.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:21.560 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:21.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:21.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:21 smithi196 ceph-mon[33018]: pgmap v514: 148 pgs: 148 active+clean; 6.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-02-05T08:16:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:21 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1664716730' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:16:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:21 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:16:21.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:21.698 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:21.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:21.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:21.833 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:21.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:21.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:21.935 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:21.970 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:21.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:22.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:22.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:22.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:22.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:22.208 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:22.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:22.243 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:22.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:22.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:22 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:16:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:22 smithi134 ceph-mon[30086]: osdmap e265: 8 total, 8 up, 7 in 2024-02-05T08:16:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:22 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:16:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:22 smithi134 ceph-mon[39278]: osdmap e265: 8 total, 8 up, 7 in 2024-02-05T08:16:22.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:22.380 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:22.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:22.408 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 1 1 2024-02-05T08:16:22.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:22.489 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 68 16 104525 104509 96.0389 81.1875 0.0133989 0.0104076 2024-02-05T08:16:22.522 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:22.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:22.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:22 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:16:22.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:22 smithi196 ceph-mon[33018]: osdmap e265: 8 total, 8 up, 7 in 2024-02-05T08:16:22.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:22.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:22.657 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:22.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:22.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:22.805 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:22.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:22.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:22.907 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:22.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:22.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:22.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:23.043 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:23.078 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:23.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:23.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:23.180 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:23.214 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:23.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:23.316 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:23.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:23.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:23.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:23.489 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 69 16 105727 105711 95.7358 75.125 0.00644159 0.0104409 2024-02-05T08:16:23.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:23.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:23.523 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:23 smithi196 ceph-mon[33018]: pgmap v516: 148 pgs: 148 active+clean; 6.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 46 MiB/s wr, 730 op/s 2024-02-05T08:16:23.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:16:23.524 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:16:23.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:23 smithi134 ceph-mon[30086]: pgmap v516: 148 pgs: 148 active+clean; 6.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 46 MiB/s wr, 730 op/s 2024-02-05T08:16:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:16:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:16:23.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:23 smithi134 ceph-mon[39278]: pgmap v516: 148 pgs: 148 active+clean; 6.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 46 MiB/s wr, 730 op/s 2024-02-05T08:16:23.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:16:23.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:16:23.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:23.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:23.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:23.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:23.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:23.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:23.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:23.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:23.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:16:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:16:23.881 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:23.920 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:23.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:24.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:24.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:24.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:24.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:24.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:24.197 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:24.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:24.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:24 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:24 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:24.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:24.341 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:24.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:24.442 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:24.476 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:24 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:24.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:24.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:24.489 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 70 16 106972 106956 95.4797 77.8125 0.00780128 0.0104692 2024-02-05T08:16:24.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:24.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:24.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:24.724 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:24.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:24.766 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:24.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:24.814 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:16:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=cleanup t=2024-02-05T08:16:24.476698994Z level=info msg="Completed cleanup jobs" duration=3.652663ms 2024-02-05T08:16:24.860 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:16:24.867 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:24.869 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:24.905 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:16:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:24.924 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:24.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:24.950 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:16:24.976 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:16:24.976 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:16:24.976 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-02-05T08:16:24.976 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-02-05T08:16:24.976 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:16:24.990 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:16:25.024 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:25.027 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:16:25.061 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:25.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:25.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:25.063 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:16:25.100 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:16:25.137 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:16:25.139 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:16:25.139 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:16:25.139 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:16:25.139 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:16:25.140 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:16:25.140 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:16:25.140 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:16:25.140 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:16:25.163 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:25.201 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:25.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:25.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:25.303 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:25.345 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:25.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:25.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:25.430 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:25.447 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:25.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:25.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:25.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:25.492 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 71 16 108189 108173 95.2031 76.0625 0.0119828 0.0104991 2024-02-05T08:16:25.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:25 smithi134 ceph-mon[30086]: pgmap v517: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-02-05T08:16:25.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:25 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1608406000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:25.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:25 smithi134 ceph-mon[39278]: pgmap v517: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-02-05T08:16:25.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:25 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1608406000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:25.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:25 smithi196 ceph-mon[33018]: pgmap v517: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-02-05T08:16:25.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:25 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1608406000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:25.589 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:25.628 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:25.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:25.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:25.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:25.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:25.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:25.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:25.865 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:25.898 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:25.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:25.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:26.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:26.042 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:26.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:26.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:26.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:26.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:26.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:26.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:26 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:16:26.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:26 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:16:26.226 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:26 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:16:26.226 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:26 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:16:26.285 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:26.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:26.320 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:26.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:26.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:26.459 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:26.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:26.491 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 72 16 109457 109441 94.9814 79.25 0.0109564 0.0105239 2024-02-05T08:16:26.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:26.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:26 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:16:26.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:26 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:16:26.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:26.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:26.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:26.736 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:26.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:26.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:26.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:26.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:26.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:26.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:27.007 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:27.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:27.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:27.109 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:27.145 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:27.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:27.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:27.281 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:27.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:27.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:27.416 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:27.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:27.479 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:27 smithi134 ceph-mon[30086]: pgmap v518: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-02-05T08:16:27.479 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:27 smithi134 ceph-mon[30086]: osdmap e266: 8 total, 8 up, 7 in 2024-02-05T08:16:27.479 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:27 smithi134 ceph-mon[39278]: pgmap v518: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-02-05T08:16:27.479 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:27 smithi134 ceph-mon[39278]: osdmap e266: 8 total, 8 up, 7 in 2024-02-05T08:16:27.491 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 73 16 110744 110728 94.7821 80.4375 0.00659756 0.0105459 2024-02-05T08:16:27.518 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:27.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T08:16:27.551 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:27.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:27.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:27 smithi196 ceph-mon[33018]: pgmap v518: 148 pgs: 148 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 76 MiB/s wr, 1.21k op/s 2024-02-05T08:16:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:27 smithi196 ceph-mon[33018]: osdmap e266: 8 total, 8 up, 7 in 2024-02-05T08:16:27.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:27.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:27.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:27.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:27.789 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:27.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:27.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:27.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:27.971 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:27.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:28.010 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:16:28.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:16:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:16:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:16:28.072 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:28.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:28.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:28.239 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:28.280 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:28.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:28.381 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:28.420 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:28.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:28.492 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 74 16 111997 111981 94.5595 78.3125 0.0262291 0.0105707 2024-02-05T08:16:28.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:28.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:28.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:28.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:28.699 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:28.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:28.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:28.839 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:28.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:28.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:28.972 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:28.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:28.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:29.074 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:29.113 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:29.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:29.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:29.215 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:29.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:29.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:29.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:29.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:29.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:29.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:29.492 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 75 16 113270 113254 94.3595 79.5625 0.00913317 0.0105934 2024-02-05T08:16:29.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:29 smithi134 ceph-mon[30086]: pgmap v520: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 55 MiB/s wr, 877 op/s 2024-02-05T08:16:29.494 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:29 smithi134 ceph-mon[39278]: pgmap v520: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 55 MiB/s wr, 877 op/s 2024-02-05T08:16:29.494 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:29.505 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:29.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:29.552 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:29.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:29.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:29 smithi196 ceph-mon[33018]: pgmap v520: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 639 B/s rd, 55 MiB/s wr, 877 op/s 2024-02-05T08:16:29.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:29.705 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:29.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:29.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:29.839 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:29.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:29.941 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:29.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:29.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:29.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:30.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:30.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:30.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:30.222 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:30.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:30.263 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:30.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:30.364 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:30.402 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:30.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:30.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:30.492 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 76 16 114506 114490 94.1343 77.25 0.0134871 0.0106184 2024-02-05T08:16:30.505 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:30.545 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:30.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:30.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:30.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:30.684 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:30.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:30.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:30.732 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:30.779 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.1 to 1 (10000) 2024-02-05T08:16:30.786 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:30.787 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:16:30.842 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:30.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:30.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:30.982 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:30.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:31.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:30 smithi134 ceph-mon[30086]: pgmap v521: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-02-05T08:16:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:30 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/468825765' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:16:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:30 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:16:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:30 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1201438144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:16:31.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:30 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:16:31.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:30 smithi134 ceph-mon[39278]: pgmap v521: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-02-05T08:16:31.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:30 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/468825765' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:16:31.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:30 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:16:31.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:30 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1201438144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:16:31.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:30 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:16:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:30 smithi196 ceph-mon[33018]: pgmap v521: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-02-05T08:16:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:30 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/468825765' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:16:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:30 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:16:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:30 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1201438144' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:16:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:30 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:16:31.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:31.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:31.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:31.226 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:31.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:31.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:31.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:31.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:31.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:31.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:31.492 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 77 16 115760 115744 93.9296 78.375 0.00528262 0.0106417 2024-02-05T08:16:31.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:31.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:31.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:31.639 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:31.676 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:31.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:31.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:31.778 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:31.818 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:31.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:31.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:31.919 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:31.956 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:31.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:31.963 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:31 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:16:31.963 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:31 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:16:31.963 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:31 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:16:31.963 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:31 smithi134 ceph-mon[30086]: osdmap e267: 8 total, 8 up, 7 in 2024-02-05T08:16:31.964 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:31 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:16:31.964 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:31 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:16:31.964 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:31 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:16:31.964 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:31 smithi134 ceph-mon[39278]: osdmap e267: 8 total, 8 up, 7 in 2024-02-05T08:16:32.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:32.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:31 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:16:32.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:31 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:16:32.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:31 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:16:32.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:31 smithi196 ceph-mon[33018]: osdmap e267: 8 total, 8 up, 7 in 2024-02-05T08:16:32.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:32.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:32.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:32.198 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:32.237 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:32.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:32.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:32.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:32.376 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:32.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:32.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:32.478 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:32.492 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 78 16 117069 117053 93.7741 81.8125 0.00940077 0.0106596 2024-02-05T08:16:32.514 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:32.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:32.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:32.616 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:32.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:32.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:32.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:32.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:32.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:32.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:32.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:32.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:32.936 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:32.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:32.977 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 2 1 2024-02-05T08:16:33.012 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:32 smithi134 ceph-mon[30086]: pgmap v523: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 49 MiB/s wr, 780 op/s 2024-02-05T08:16:33.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:32 smithi134 ceph-mon[39278]: pgmap v523: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 49 MiB/s wr, 780 op/s 2024-02-05T08:16:33.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:32 smithi196 ceph-mon[33018]: pgmap v523: 148 pgs: 148 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 49 MiB/s wr, 780 op/s 2024-02-05T08:16:33.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:33.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:33.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:33.175 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:33.211 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:33.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:33.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:33.313 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:33.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:33.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:33.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:33.486 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:33.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:33.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:33.492 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 79 16 118292 118276 93.5545 76.4375 0.0191743 0.0106845 2024-02-05T08:16:33.589 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:33.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:33.632 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:33.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:33.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:33.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:33.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:33 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:33.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:16:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:16:33.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:33.910 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:33.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:33.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:33.940 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:33 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:33.941 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:33 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:34.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:34.052 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:34.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:34.153 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:34.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:34.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:34.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:34.333 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:34.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:34.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:34.483 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:34.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:34.493 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:16:34.494379+0000 min lat: 0.00130673 max lat: 0.198515 avg lat: 0.0107074 2024-02-05T08:16:34.493 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:16:34.493 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 80 16 119537 119521 93.3569 77.8125 0.0127382 0.0107074 2024-02-05T08:16:34.585 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:34.623 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:34.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:34.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:34.762 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:34.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:34.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:34.864 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:34.906 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:34.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:34.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:35.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:35.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:35.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:34 smithi196 ceph-mon[33018]: pgmap v524: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-02-05T08:16:35.084 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:34 smithi134 ceph-mon[39278]: pgmap v524: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-02-05T08:16:35.086 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:34 smithi134 ceph-mon[30086]: pgmap v524: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.56k op/s 2024-02-05T08:16:35.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:35.186 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:35.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:35.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:35.287 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:35.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:35.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:35.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:35.426 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:35.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:35.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:35.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:16:35.494 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 81 16 120774 120758 93.1581 77.3125 0.0113472 0.0107299 2024-02-05T08:16:35.567 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:35.623 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:35.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:35.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:35.731 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:16:35.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:35.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:35.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:35.770 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:16:35.770 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:16:35.770 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:16:35.770 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:16:35.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.770 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108936 8.6 8:6ce9955c:::benchmark_data_smithi134_115836_object108935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-05T08:16:35.770 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:28.999547+0000", 2024-02-05T08:16:35.771 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 246.59808339, 2024-02-05T08:16:35.771 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.767948115, 2024-02-05T08:16:35.771 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.771 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:16:35.771 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.771 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.771 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.771 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108936 2024-02-05T08:16:35.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.771 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.771 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999547+0000", 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999547+0000", 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999548+0000", 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.195e-06 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999572+0000", 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3781000000000002e-05 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999583+0000", 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1143e-05 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999591+0000", 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1140000000000006e-06 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.773 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.005098+0000", 2024-02-05T08:16:35.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0055070800000000001 2024-02-05T08:16:35.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.774 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.005332+0000", 2024-02-05T08:16:35.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023392899999999999 2024-02-05T08:16:35.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.774 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,7", 2024-02-05T08:16:35.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.005485+0000", 2024-02-05T08:16:35.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015303800000000001 2024-02-05T08:16:35.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.015634+0000", 2024-02-05T08:16:35.775 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010149004 2024-02-05T08:16:35.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.015665+0000", 2024-02-05T08:16:35.775 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0966000000000001e-05 2024-02-05T08:16:35.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.776 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.041481+0000", 2024-02-05T08:16:35.776 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.025815647000000001 2024-02-05T08:16:35.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.776 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.041495+0000", 2024-02-05T08:16:35.776 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4079e-05 2024-02-05T08:16:35.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.776 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666718+0000", 2024-02-05T08:16:35.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.62522305499999997 2024-02-05T08:16:35.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666730+0000", 2024-02-05T08:16:35.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2157000000000001e-05 2024-02-05T08:16:35.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.767398+0000", 2024-02-05T08:16:35.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1006681920000001 2024-02-05T08:16:35.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.767495+0000", 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6735000000000001e-05 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108967 8.6 8:6b4bec94:::benchmark_data_smithi134_115836_object108966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.046490+0000", 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 246.551140221, 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.731470512, 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.778 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108967 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046490+0000", 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.779 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046490+0000", 2024-02-05T08:16:35.780 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.780 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.780 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046492+0000", 2024-02-05T08:16:35.780 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7060000000000001e-06 2024-02-05T08:16:35.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.780 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.780 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046518+0000", 2024-02-05T08:16:35.780 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5791000000000001e-05 2024-02-05T08:16:35.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046527+0000", 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3980000000000005e-06 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046537+0000", 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0380999999999999e-05 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.053715+0000", 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0071773959999999999 2024-02-05T08:16:35.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.053726+0000", 2024-02-05T08:16:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1117999999999999e-05 2024-02-05T08:16:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666815+0000", 2024-02-05T08:16:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.61308943100000002 2024-02-05T08:16:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.782 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666824+0000", 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0059999999999998e-06 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.767788+0000", 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1009632439999999 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.767916+0000", 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012850399999999999 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.768062+0000", 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014636200000000001 2024-02-05T08:16:35.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777461+0000", 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0093981900000000007 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777525+0000", 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4302999999999997e-05 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777717+0000", 2024-02-05T08:16:35.784 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019194500000000001 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777765+0000", 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8143999999999998e-05 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777864+0000", 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9068999999999996e-05 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777961+0000", 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6523999999999999e-05 2024-02-05T08:16:35.785 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108968 8.6 8:680b4c21:::benchmark_data_smithi134_115836_object108967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.048233+0000", 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 246.54939771799999, 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7300381819999999, 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108968 2024-02-05T08:16:35.786 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048233+0000", 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048233+0000", 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048235+0000", 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.018e-06 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048260+0000", 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5247000000000001e-05 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048272+0000", 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2483000000000001e-05 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048345+0000", 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2766000000000001e-05 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.053751+0000", 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0054057139999999998 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.053759+0000", 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0479999999999999e-06 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666852+0000", 2024-02-05T08:16:35.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.61309340899999998 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666861+0000", 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5750000000000001e-06 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.770884+0000", 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1040229189999999 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.771021+0000", 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000137353 2024-02-05T08:16:35.790 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.771230+0000", 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020888999999999999 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777723+0000", 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00649338 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778025+0000", 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030212499999999998 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.791 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778073+0000", 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7438000000000002e-05 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778116+0000", 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3133000000000002e-05 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778154+0000", 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7744999999999999e-05 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.792 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778271+0000", 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000116939 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108988 8.6 8:6273284f:::benchmark_data_smithi134_115836_object108987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.068824+0000", 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 246.52880588900001, 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7214295480000001, 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.793 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108988 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068824+0000", 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068824+0000", 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068826+0000", 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.35e-06 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068848+0000", 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2093e-05 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068861+0000", 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2918000000000001e-05 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068871+0000", 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.009e-05 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068950+0000", 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8747999999999999e-05 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068962+0000", 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2023e-05 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666889+0000", 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.59792759500000003 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666898+0000", 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4479999999999994e-06 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778301+0000", 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.111403232 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778649+0000", 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00034766799999999998 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778800+0000", 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000151101 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788785+0000", 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0099855810000000003 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.798 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788832+0000", 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6394000000000003e-05 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788885+0000", 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3412999999999998e-05 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788973+0000", 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7843000000000006e-05 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.799 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.789172+0000", 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019889199999999999 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790254+0000", 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010821590000000001 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108994 8.6 8:69b35254:::benchmark_data_smithi134_115836_object108993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:16:35.800 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.072494+0000", 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 246.52513657599999, 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7180740800000001, 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108994 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072494+0000", 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072494+0000", 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072496+0000", 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.815e-06 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072522+0000", 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6954e-05 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.802 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072539+0000", 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6602000000000001e-05 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072548+0000", 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3070000000000008e-06 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072616+0000", 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7382e-05 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072628+0000", 2024-02-05T08:16:35.804 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1772e-05 2024-02-05T08:16:35.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.804 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.804 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.804 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666926+0000", 2024-02-05T08:16:35.804 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.59429891000000001 2024-02-05T08:16:35.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.804 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.804 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.804 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666935+0000", 2024-02-05T08:16:35.804 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0989999999999999e-06 2024-02-05T08:16:35.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.804 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.782050+0000", 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1151151159999999 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.782294+0000", 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000243849 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.782456+0000", 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016257700000000001 2024-02-05T08:16:35.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788847+0000", 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.006391112 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788929+0000", 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1520000000000006e-05 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790307+0000", 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0013783059999999999 2024-02-05T08:16:35.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.807 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.807 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790406+0000", 2024-02-05T08:16:35.807 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9007999999999998e-05 2024-02-05T08:16:35.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.807 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.807 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790467+0000", 2024-02-05T08:16:35.807 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1057999999999997e-05 2024-02-05T08:16:35.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.807 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.807 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790568+0000", 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010069300000000001 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108995 8.6 8:6351071d:::benchmark_data_smithi134_115836_object108994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.072933+0000", 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 246.52469763900001, 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7266677289999999, 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.808 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.809 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.809 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108995 2024-02-05T08:16:35.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.809 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.809 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.809 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072933+0000", 2024-02-05T08:16:35.809 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.809 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.809 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072933+0000", 2024-02-05T08:16:35.809 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072934+0000", 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0410000000000001e-06 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072956+0000", 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2197000000000001e-05 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.810 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072971+0000", 2024-02-05T08:16:35.811 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5152e-05 2024-02-05T08:16:35.811 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.811 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.811 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072979+0000", 2024-02-05T08:16:35.811 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9689999999999994e-06 2024-02-05T08:16:35.811 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.811 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.811 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.073032+0000", 2024-02-05T08:16:35.811 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3375000000000003e-05 2024-02-05T08:16:35.811 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.073041+0000", 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0480000000000004e-06 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666963+0000", 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.59392174399999997 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666972+0000", 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4500000000000004e-06 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.785402+0000", 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1184303840000001 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.785534+0000", 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013186 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.785689+0000", 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000155487 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.813 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790323+0000", 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0046338509999999996 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790357+0000", 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3657e-05 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790612+0000", 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00025531600000000001 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.814 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.799422+0000", 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0088102289999999993 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.799489+0000", 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6593000000000004e-05 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.799600+0000", 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111376 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.815 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.816 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.816 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108980 8.5 8:ad067b35:::benchmark_data_smithi134_115836_object108979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e188)", 2024-02-05T08:16:35.816 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.663590+0000", 2024-02-05T08:16:35.816 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 245.93404056200001, 2024-02-05T08:16:35.816 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1061829890000001, 2024-02-05T08:16:35.816 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.816 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.816 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.816 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.816 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.816 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108980 2024-02-05T08:16:35.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.663590+0000", 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.663590+0000", 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.817 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.663591+0000", 2024-02-05T08:16:35.818 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7680000000000001e-06 2024-02-05T08:16:35.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.818 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.818 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.818 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.663619+0000", 2024-02-05T08:16:35.818 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7832999999999999e-05 2024-02-05T08:16:35.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.818 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.818 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.818 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.665311+0000", 2024-02-05T08:16:35.818 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0016921149999999999 2024-02-05T08:16:35.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.818 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.819 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.819 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.665334+0000", 2024-02-05T08:16:35.819 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2626e-05 2024-02-05T08:16:35.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.819 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.819 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666524+0000", 2024-02-05T08:16:35.819 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011904350000000001 2024-02-05T08:16:35.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.819 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.819 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666534+0000", 2024-02-05T08:16:35.819 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3009999999999995e-06 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.761805+0000", 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.095271168 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.763439+0000", 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001634251 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,6,7", 2024-02-05T08:16:35.820 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.763616+0000", 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000176711 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769220+0000", 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0056040200000000004 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769258+0000", 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8101e-05 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.821 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769293+0000", 2024-02-05T08:16:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4687999999999997e-05 2024-02-05T08:16:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769610+0000", 2024-02-05T08:16:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031678300000000001 2024-02-05T08:16:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769668+0000", 2024-02-05T08:16:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8451e-05 2024-02-05T08:16:35.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769773+0000", 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104738 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:134904 8.e 8:7d534b82:::benchmark_data_smithi134_115836_object18244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.506668+0000", 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 213.090962109, 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.19134764500000001, 2024-02-05T08:16:35.823 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.824 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.824 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.824 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.824 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.824 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 134904 2024-02-05T08:16:35.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.824 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.824 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.824 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506668+0000", 2024-02-05T08:16:35.824 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506668+0000", 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506669+0000", 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4900000000000004e-07 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506674+0000", 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0529999999999997e-06 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.826 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.826 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506688+0000", 2024-02-05T08:16:35.826 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4287999999999999e-05 2024-02-05T08:16:35.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.826 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.826 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.826 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506696+0000", 2024-02-05T08:16:35.826 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1709999999999996e-06 2024-02-05T08:16:35.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.826 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.826 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.826 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.516764+0000", 2024-02-05T08:16:35.826 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010068344999999999 2024-02-05T08:16:35.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.827 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.827 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.827 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.516879+0000", 2024-02-05T08:16:35.827 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114751 2024-02-05T08:16:35.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.827 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.827 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-02-05T08:16:35.827 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.516995+0000", 2024-02-05T08:16:35.827 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115844 2024-02-05T08:16:35.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.827 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.827 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.828 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.521656+0000", 2024-02-05T08:16:35.828 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0046613660000000001 2024-02-05T08:16:35.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.828 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.828 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.828 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.528669+0000", 2024-02-05T08:16:35.828 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0070134309999999997 2024-02-05T08:16:35.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.828 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.828 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.828 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.697929+0000", 2024-02-05T08:16:35.828 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16926000699999999 2024-02-05T08:16:35.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.829 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.829 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.829 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.697981+0000", 2024-02-05T08:16:35.829 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1521999999999999e-05 2024-02-05T08:16:35.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.829 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.829 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.829 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.698016+0000", 2024-02-05T08:16:35.829 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4918e-05 2024-02-05T08:16:35.829 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.829 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.829 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:134932 8.e 8:747554f1:::benchmark_data_smithi134_115836_object18272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.538807+0000", 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 213.05882317000001, 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.18031301799999999, 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 134932 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.830 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538807+0000", 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538807+0000", 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538808+0000", 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.33e-06 2024-02-05T08:16:35.831 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538813+0000", 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9429999999999999e-06 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.694877+0000", 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15606357700000001 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.694895+0000", 2024-02-05T08:16:35.832 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8104000000000001e-05 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.698658+0000", 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0037627429999999998 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.698821+0000", 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016332799999999999 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.698943+0000", 2024-02-05T08:16:35.833 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012196 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.708664+0000", 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0097209140000000006 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.708709+0000", 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5241999999999998e-05 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.834 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.718868+0000", 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010159062999999999 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.719076+0000", 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020797 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.719120+0000", 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3844000000000002e-05 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.835 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.836 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.836 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.836 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15087.0:162972 8.8 e214/194 8:14305b79:::benchmark_data_smithi134_115836_object46312:head v 214'10137, mlcod=214'10135)", 2024-02-05T08:16:35.836 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:37.286547+0000", 2024-02-05T08:16:35.836 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 178.31108349799999, 2024-02-05T08:16:35.836 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.187515187, 2024-02-05T08:16:35.836 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.836 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.836 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.836 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.836 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.836 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286547+0000", 2024-02-05T08:16:35.837 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.837 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.837 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.837 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286547+0000", 2024-02-05T08:16:35.837 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.837 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.837 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.837 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286547+0000", 2024-02-05T08:16:35.837 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8800000000000002e-07 2024-02-05T08:16:35.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.837 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286553+0000", 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9819999999999996e-06 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286560+0000", 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1380000000000001e-06 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286567+0000", 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3600000000000001e-06 2024-02-05T08:16:35.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286636+0000", 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8868999999999994e-05 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286659+0000", 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.304e-05 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.474023+0000", 2024-02-05T08:16:35.839 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.187364261 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.474062+0000", 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8949000000000003e-05 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDPGPush(8.8 214/194 [PushOp(8:1b8b759a:::benchmark_data_smithi134_115836_object58675:head, version: 173'3646, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:1b8b759a:::benchmark_data_smithi134_115836_object58675:head@173'3646, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:37.291477+0000", 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 178.306152932, 2024-02-05T08:16:35.840 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.18357304299999999, 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.291477+0000", 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.291477+0000", 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.841 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.291478+0000", 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0800000000000004e-07 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.291523+0000", 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5318999999999998e-05 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.464807+0000", 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.17328333600000001 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.842 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.464836+0000", 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9229000000000001e-05 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.474831+0000", 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0099948740000000008 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.474844+0000", 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3338e-05 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.475050+0000", 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020613900000000001 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170535 8.c 8:30bf855a:::benchmark_data_smithi134_115836_object53875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.749814+0000", 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 168.847816056, 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.99481441999999998, 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170535 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749814+0000", 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749814+0000", 2024-02-05T08:16:35.845 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749816+0000", 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.547e-06 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749821+0000", 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8679999999999998e-06 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749834+0000", 2024-02-05T08:16:35.846 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3169999999999999e-05 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749842+0000", 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3229999999999998e-06 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749901+0000", 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8464000000000001e-05 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.847 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749909+0000", 2024-02-05T08:16:35.848 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8550000000000008e-06 2024-02-05T08:16:35.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.848 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.848 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.848 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.733527+0000", 2024-02-05T08:16:35.848 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.98361745899999997 2024-02-05T08:16:35.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.848 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.848 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.848 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.733637+0000", 2024-02-05T08:16:35.848 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109769 2024-02-05T08:16:35.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.733763+0000", 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000126191 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744320+0000", 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010556640000000001 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744459+0000", 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013946999999999999 2024-02-05T08:16:35.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.850 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.850 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.850 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744506+0000", 2024-02-05T08:16:35.850 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7512000000000002e-05 2024-02-05T08:16:35.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.850 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.850 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.850 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744579+0000", 2024-02-05T08:16:35.850 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2857999999999999e-05 2024-02-05T08:16:35.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.850 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.850 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.850 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744629+0000", 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9293999999999999e-05 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170561 8.c 8:320abcbc:::benchmark_data_smithi134_115836_object53901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.770164+0000", 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 168.827465964, 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.97474936999999995, 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.851 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170561 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770164+0000", 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770164+0000", 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.852 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770166+0000", 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6750000000000001e-06 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770187+0000", 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1461999999999999e-05 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770200+0000", 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2884000000000001e-05 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.853 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770208+0000", 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6839999999999996e-06 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770266+0000", 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8081000000000002e-05 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770275+0000", 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4579999999999993e-06 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.736290+0000", 2024-02-05T08:16:35.854 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.96601568100000001 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.736375+0000", 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5116000000000003e-05 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.736487+0000", 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111376 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744402+0000", 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0079155340000000001 2024-02-05T08:16:35.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744684+0000", 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00028183599999999999 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744761+0000", 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7120999999999996e-05 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744873+0000", 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011210299999999999 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.856 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744914+0000", 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0358999999999998e-05 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170591 8.c 8:3e90cc49:::benchmark_data_smithi134_115836_object53931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.795150+0000", 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 168.80248041199999, 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.949924875, 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.857 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170591 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795150+0000", 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795150+0000", 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795151+0000", 2024-02-05T08:16:35.858 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2049999999999999e-06 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795156+0000", 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.634e-06 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795168+0000", 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2119999999999999e-05 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795181+0000", 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3655999999999999e-05 2024-02-05T08:16:35.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795293+0000", 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011196600000000001 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795312+0000", 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8717000000000001e-05 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.738715+0000", 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94340282600000003 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.860 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.738798+0000", 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3123999999999999e-05 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.738868+0000", 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9904e-05 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744471+0000", 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0056029870000000002 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.861 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744956+0000", 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00048495400000000001 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744999+0000", 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3164999999999998e-05 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745044+0000", 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4780999999999997e-05 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.862 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745075+0000", 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0836000000000001e-05 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170592 8.c 8:3f22b68d:::benchmark_data_smithi134_115836_object53932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.795212+0000", 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 168.802417891, 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95563532299999998, 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.863 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170592 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795212+0000", 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795212+0000", 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795213+0000", 2024-02-05T08:16:35.864 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3200000000000005e-07 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795216+0000", 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7470000000000001e-06 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795223+0000", 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9240000000000001e-06 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795228+0000", 2024-02-05T08:16:35.865 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7729999999999996e-06 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795334+0000", 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000105321 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795339+0000", 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7440000000000004e-06 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.866 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.741396+0000", 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94605645199999999 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.741504+0000", 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107666 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.741620+0000", 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011629900000000001 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744703+0000", 2024-02-05T08:16:35.867 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030836259999999999 2024-02-05T08:16:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750711+0000", 2024-02-05T08:16:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0060076019999999999 2024-02-05T08:16:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750749+0000", 2024-02-05T08:16:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8243e-05 2024-02-05T08:16:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.869 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.869 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750814+0000", 2024-02-05T08:16:35.869 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4344999999999995e-05 2024-02-05T08:16:35.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.869 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.869 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.869 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750848+0000", 2024-02-05T08:16:35.869 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4048999999999999e-05 2024-02-05T08:16:35.869 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.869 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.869 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.870 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.870 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170598 8.c 8:396b7832:::benchmark_data_smithi134_115836_object53938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:16:35.870 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.801793+0000", 2024-02-05T08:16:35.870 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 168.795836968, 2024-02-05T08:16:35.870 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95466330200000005, 2024-02-05T08:16:35.870 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.870 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.870 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.870 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.870 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.870 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170598 2024-02-05T08:16:35.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.871 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.871 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.871 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.871 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801793+0000", 2024-02-05T08:16:35.871 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.871 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.871 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.871 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801793+0000", 2024-02-05T08:16:35.871 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.871 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.871 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.872 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801794+0000", 2024-02-05T08:16:35.872 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0699999999999999e-06 2024-02-05T08:16:35.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.872 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.872 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.872 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801798+0000", 2024-02-05T08:16:35.872 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9999999999999998e-06 2024-02-05T08:16:35.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.872 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.872 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.872 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801812+0000", 2024-02-05T08:16:35.872 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3257e-05 2024-02-05T08:16:35.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801820+0000", 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5380000000000002e-06 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801881+0000", 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0494000000000001e-05 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801890+0000", 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5859999999999994e-06 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.873 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.874 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.874 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745103+0000", 2024-02-05T08:16:35.874 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94321233000000004 2024-02-05T08:16:35.874 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.874 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.874 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.874 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745242+0000", 2024-02-05T08:16:35.874 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000139357 2024-02-05T08:16:35.874 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.874 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:35.875 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:16:35.875 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745347+0000", 2024-02-05T08:16:35.875 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000105308 2024-02-05T08:16:35.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.875 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.875 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750885+0000", 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0055373130000000003 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756344+0000", 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.005459518 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756359+0000", 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5268999999999999e-05 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.876 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756414+0000", 2024-02-05T08:16:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4469999999999999e-05 2024-02-05T08:16:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756457+0000", 2024-02-05T08:16:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2792e-05 2024-02-05T08:16:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170600 8.c 8:3e055f0a:::benchmark_data_smithi134_115836_object53940:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:16:35.877 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.801861+0000", 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 168.79576977100001, 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95753421999999999, 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170600 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801861+0000", 2024-02-05T08:16:35.878 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801861+0000", 2024-02-05T08:16:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801861+0000", 2024-02-05T08:16:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7099999999999997e-07 2024-02-05T08:16:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.879 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801863+0000", 2024-02-05T08:16:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4109999999999999e-06 2024-02-05T08:16:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801869+0000", 2024-02-05T08:16:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.789e-06 2024-02-05T08:16:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801874+0000", 2024-02-05T08:16:35.880 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3310000000000003e-06 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801917+0000", 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2317000000000002e-05 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801924+0000", 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4080000000000001e-06 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.881 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.748074+0000", 2024-02-05T08:16:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94614989999999999 2024-02-05T08:16:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.748198+0000", 2024-02-05T08:16:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012395200000000001 2024-02-05T08:16:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:16:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.748296+0000", 2024-02-05T08:16:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8288000000000005e-05 2024-02-05T08:16:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.882 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.883 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.883 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756466+0000", 2024-02-05T08:16:35.883 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0081697049999999993 2024-02-05T08:16:35.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.883 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.883 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.883 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756527+0000", 2024-02-05T08:16:35.883 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1192000000000006e-05 2024-02-05T08:16:35.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.883 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.883 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.883 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759307+0000", 2024-02-05T08:16:35.883 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027796460000000002 2024-02-05T08:16:35.884 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.884 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.884 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.884 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759358+0000", 2024-02-05T08:16:35.884 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0918000000000003e-05 2024-02-05T08:16:35.884 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.884 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.884 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.884 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759395+0000", 2024-02-05T08:16:35.884 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6992000000000001e-05 2024-02-05T08:16:35.884 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.884 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.884 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170603 8.c 8:335f5460:::benchmark_data_smithi134_115836_object53943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.805707+0000", 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 168.79192310900001, 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95838726600000002, 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170603 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.885 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.886 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.886 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.886 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805707+0000", 2024-02-05T08:16:35.886 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.886 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.886 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.886 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805707+0000", 2024-02-05T08:16:35.886 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.886 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.886 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.886 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805709+0000", 2024-02-05T08:16:35.887 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.415e-06 2024-02-05T08:16:35.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.887 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.887 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.887 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805713+0000", 2024-02-05T08:16:35.887 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6020000000000002e-06 2024-02-05T08:16:35.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.887 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.887 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.887 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805725+0000", 2024-02-05T08:16:35.887 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1729000000000001e-05 2024-02-05T08:16:35.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.887 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805732+0000", 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4839999999999998e-06 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805784+0000", 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1134000000000002e-05 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805792+0000", 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6440000000000006e-06 2024-02-05T08:16:35.888 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750912+0000", 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94512030199999997 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.751005+0000", 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2283999999999999e-05 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.751107+0000", 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000101828 2024-02-05T08:16:35.889 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756471+0000", 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0053649029999999999 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759438+0000", 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002966266 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764000+0000", 2024-02-05T08:16:35.890 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0045619750000000002 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764055+0000", 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5158000000000003e-05 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764094+0000", 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9542000000000002e-05 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.891 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170608 8.c 8:33618a8a:::benchmark_data_smithi134_115836_object53948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.807038+0000", 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 168.790591879, 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95716985099999996, 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170608 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.892 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.893 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807038+0000", 2024-02-05T08:16:35.893 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.893 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.893 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.893 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807038+0000", 2024-02-05T08:16:35.893 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.893 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.893 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.893 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807039+0000", 2024-02-05T08:16:35.893 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4e-07 2024-02-05T08:16:35.893 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807044+0000", 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9660000000000004e-06 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807056+0000", 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1437000000000001e-05 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807064+0000", 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2779999999999993e-06 2024-02-05T08:16:35.894 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807115+0000", 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0725e-05 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807122+0000", 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4449999999999999e-06 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.753804+0000", 2024-02-05T08:16:35.895 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94668188499999995 2024-02-05T08:16:35.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.896 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.896 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.896 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.753906+0000", 2024-02-05T08:16:35.896 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000102166 2024-02-05T08:16:35.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.896 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.896 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:16:35.896 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.753985+0000", 2024-02-05T08:16:35.896 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8432999999999999e-05 2024-02-05T08:16:35.896 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.896 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.896 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756543+0000", 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0025579399999999999 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759479+0000", 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0029361249999999999 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764128+0000", 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0046493369999999999 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764168+0000", 2024-02-05T08:16:35.897 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9558999999999999e-05 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764208+0000", 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0714999999999999e-05 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170625 8.c 8:322ad108:::benchmark_data_smithi134_115836_object53965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.834138+0000", 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 168.76349266, 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.93039029699999998, 2024-02-05T08:16:35.898 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170625 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834138+0000", 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:16:35.899 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834138+0000", 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834138+0000", 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8100000000000001e-07 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834143+0000", 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0929999999999996e-06 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.900 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834158+0000", 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5057e-05 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834166+0000", 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2390000000000002e-06 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834213+0000", 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7029e-05 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.901 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834219+0000", 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3319999999999997e-06 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756569+0000", 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.92234994100000001 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756684+0000", 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115053 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.902 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.903 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:16:35.903 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756836+0000", 2024-02-05T08:16:35.903 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015174599999999999 2024-02-05T08:16:35.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.903 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.903 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:16:35.903 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759502+0000", 2024-02-05T08:16:35.903 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0026656649999999998 2024-02-05T08:16:35.903 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.903 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.903 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.903 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759930+0000", 2024-02-05T08:16:35.904 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00042810999999999999 2024-02-05T08:16:35.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.904 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.904 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:16:35.904 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764311+0000", 2024-02-05T08:16:35.904 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0043816990000000002 2024-02-05T08:16:35.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.904 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.904 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:16:35.904 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764492+0000", 2024-02-05T08:16:35.904 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018032500000000001 2024-02-05T08:16:35.904 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:16:35.905 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:16:35.905 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:16:35.905 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764528+0000", 2024-02-05T08:16:35.905 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6127000000000002e-05 2024-02-05T08:16:35.905 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.905 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.905 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.905 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:16:35.905 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:16:35.905 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:16:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:16:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:16:35.906 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:16:35.906 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:35 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1618329679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:35.907 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:16:35.907 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:35 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1618329679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:35.926 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:16:35.942 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:35.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:35.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:35.963 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:16:36.009 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:16:36.012 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:16:36.012 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:16:36.012 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:16:36.013 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:16:36.013 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:16:36.013 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:16:36.013 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:16:36.013 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:16:36.044 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:35 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1618329679' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:36.082 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:36.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:36.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:36.214 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:36.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:36.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:36.316 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:36.350 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:36.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:36.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:36.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:36.486 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:36.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:36.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:36.494 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 82 16 121993 121977 92.9511 76.1875 0.0161101 0.010754 2024-02-05T08:16:36.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:36.625 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:36.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:36.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:36.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:36.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:36.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:36.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:36.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:36.904 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:36.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:37.005 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:37.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:36 smithi134 ceph-mon[30086]: pgmap v525: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T08:16:37.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:36 smithi134 ceph-mon[39278]: pgmap v525: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T08:16:37.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:37.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:36 smithi196 ceph-mon[33018]: pgmap v525: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T08:16:37.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:37.186 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:37.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:37.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:37.288 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:37.330 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:37.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:37.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:37.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:37.467 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:37.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:37.494 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 83 16 123250 123234 92.7777 78.5625 0.0059966 0.0107743 2024-02-05T08:16:37.567 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:37.598 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:37.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:37.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:37.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:37.741 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:37.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:37.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:37.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:37.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:37.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:37.895 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:37.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T08:16:37.985 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:38.019 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:38.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:38.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:16:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:16:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:16:38.122 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:38.162 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:38.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:38.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:38.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:38.295 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:38.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:38.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:38.431 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:38.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:38.494 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 84 16 124511 124495 92.6112 78.8125 0.0145781 0.0107933 2024-02-05T08:16:38.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:38.567 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:38.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:38.639 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:38.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:38.709 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:38.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:38.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:38.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:38.847 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:38.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:38.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:38.984 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:38.989 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:38.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:38.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:39.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:38 smithi134 ceph-mon[30086]: pgmap v526: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-02-05T08:16:39.029 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:16:39.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:38 smithi134 ceph-mon[39278]: pgmap v526: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-02-05T08:16:39.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:16:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:38 smithi196 ceph-mon[33018]: pgmap v526: 148 pgs: 148 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-02-05T08:16:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:16:39.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:39.125 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:39.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:39.227 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:39.261 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:39.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:39.271 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:16:39.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:39.407 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:39.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:39.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:39.494 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 85 16 125763 125747 92.4422 78.25 0.0134954 0.0108133 2024-02-05T08:16:39.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:39.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:39.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:39.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:39.649 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:39.688 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:39.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:39.790 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:39.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:39.829 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:39.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:39.930 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:39.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:39.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:40.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:39 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:16:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:39 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:16:40.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:39 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:16:40.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:39 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:16:40.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:40.093 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:40.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:40.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:40.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:40.218 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:40.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:40.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:39 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:16:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:39 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:16:40.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:40.357 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:40.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:40.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:40.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:40.492 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:40.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:40.494 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 86 16 126829 126813 92.1419 66.625 0.0234178 0.0108483 2024-02-05T08:16:40.594 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:40.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:40.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:40.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:40.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:40.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:40.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:40.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:40.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:40.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:40.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:40.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:40.997 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:41.031 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:41.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:41.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:41.133 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:41.165 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:41.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:41.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:41.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:41.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:40 smithi134 ceph-mon[30086]: pgmap v527: 148 pgs: 148 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T08:16:41.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:40 smithi134 ceph-mon[30086]: osdmap e268: 8 total, 8 up, 7 in 2024-02-05T08:16:41.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:40 smithi134 ceph-mon[39278]: pgmap v527: 148 pgs: 148 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T08:16:41.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:40 smithi134 ceph-mon[39278]: osdmap e268: 8 total, 8 up, 7 in 2024-02-05T08:16:41.301 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:41.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:41.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:41.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:40 smithi196 ceph-mon[33018]: pgmap v527: 148 pgs: 148 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T08:16:41.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:40 smithi196 ceph-mon[33018]: osdmap e268: 8 total, 8 up, 7 in 2024-02-05T08:16:41.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:41.433 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:41.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:41.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:41.494 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 87 16 127978 127962 91.9082 71.8125 0.00428692 0.0108748 2024-02-05T08:16:41.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:41.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:41.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:41.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:41.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:41.708 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:41.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:41.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:41.810 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:41.845 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:41.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:41.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:41.977 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:41.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:41.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:42.079 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:42.113 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:42.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:42.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:42.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:42.248 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:42.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:42.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:42.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:42.385 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:42.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:42.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:42.486 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:42.494 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 88 15 129209 129194 91.7388 77 0.0161532 0.0108963 2024-02-05T08:16:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:42.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:42.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:42.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:42.664 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:42.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:42.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:42.767 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:42.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:42.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:42.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:42.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:42.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:42.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:43.023 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:42 smithi134 ceph-mon[30086]: pgmap v529: 148 pgs: 148 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T08:16:43.023 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:42 smithi134 ceph-mon[39278]: pgmap v529: 148 pgs: 148 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T08:16:43.053 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:43.088 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:43.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:43.115 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:42 smithi196 ceph-mon[33018]: pgmap v529: 148 pgs: 148 active+clean; 7.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T08:16:43.190 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:43.231 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:43.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:43.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:43.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:43.373 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:43.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:43.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:43.475 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:43.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 89 16 130483 130467 91.6019 79.5625 0.00698673 0.0109128 2024-02-05T08:16:43.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:43.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:43.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:43.614 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:43.652 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:43.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:43.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:43.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:43.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:43.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:43.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:16:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:16:43.843 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.2 to 1 (10000) 2024-02-05T08:16:43.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:43.930 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:43.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:43.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:44.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:44.064 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:44.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:44.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:44.124 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:43 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:44.124 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:43 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3686997490' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:16:44.124 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:43 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:44.124 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:43 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3686997490' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:16:44.167 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:44.200 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:44.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:44.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:44.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:43 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:44.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:43 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3686997490' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:16:44.337 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:44.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:44.439 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:44.484 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:44.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:44.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Total time run: 90.0114 2024-02-05T08:16:44.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Total writes made: 131737 2024-02-05T08:16:44.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Write size: 65536 2024-02-05T08:16:44.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Object size: 65536 2024-02-05T08:16:44.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Bandwidth (MB/sec): 91.4724 2024-02-05T08:16:44.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev Bandwidth: 31.718 2024-02-05T08:16:44.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max bandwidth (MB/sec): 228.188 2024-02-05T08:16:44.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min bandwidth (MB/sec): 56 2024-02-05T08:16:44.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Average IOPS: 1463 2024-02-05T08:16:44.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev IOPS: 507.488 2024-02-05T08:16:44.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max IOPS: 3651 2024-02-05T08:16:44.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min IOPS: 896 2024-02-05T08:16:44.495 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Average Latency(s): 0.0109282 2024-02-05T08:16:44.496 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev Latency(s): 0.0083058 2024-02-05T08:16:44.496 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max latency(s): 0.198515 2024-02-05T08:16:44.496 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min latency(s): 0.00130673 2024-02-05T08:16:44.499 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T08:16:44.586 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:44.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:44.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:44.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:44.723 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:44.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:44.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:44.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:44.858 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:44.906 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:44.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:44.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:45.012 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:44 smithi134 ceph-mon[30086]: pgmap v530: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:16:45.012 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:44 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3686997490' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:16:45.012 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:44 smithi134 ceph-mon[30086]: osdmap e269: 8 total, 8 up, 7 in 2024-02-05T08:16:45.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:44 smithi134 ceph-mon[39278]: pgmap v530: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:16:45.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:44 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3686997490' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:16:45.012 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:44 smithi134 ceph-mon[39278]: osdmap e269: 8 total, 8 up, 7 in 2024-02-05T08:16:45.055 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:45.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:45.104 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:45.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:45.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:45.215 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:16:45.215 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:16:45.249 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:45.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:45.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:44 smithi196 ceph-mon[33018]: pgmap v530: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:16:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:44 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3686997490' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:16:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:44 smithi196 ceph-mon[33018]: osdmap e269: 8 total, 8 up, 7 in 2024-02-05T08:16:45.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:45.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:45.388 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:45.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:45.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:45.522 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:45.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:45.624 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:45.659 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:45.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:45.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:45.795 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:45.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:45.896 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:45.898 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:16:45.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:45.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:46.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:45 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:16:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:45 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:16:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:45 smithi134 ceph-mon[30086]: osdmap e270: 8 total, 8 up, 7 in 2024-02-05T08:16:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:45 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/441305803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:16:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:45 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:16:46.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:45 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:16:46.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:45 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:16:46.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:45 smithi134 ceph-mon[39278]: osdmap e270: 8 total, 8 up, 7 in 2024-02-05T08:16:46.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:45 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/441305803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:16:46.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:45 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:16:46.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:46.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:46.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:46.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:46.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:46.207 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:46.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:46.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:46.301 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T08:16:46.309 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:45 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:16:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:45 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:16:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:45 smithi196 ceph-mon[33018]: osdmap e270: 8 total, 8 up, 7 in 2024-02-05T08:16:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:45 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/441305803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:16:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:45 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:16:46.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:46.380 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:46.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:46.481 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:46.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:46.520 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:46.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:46.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:46.638 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:46.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:46.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:46.682 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 4 1 2024-02-05T08:16:46.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:46.795 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:46.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:46.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:46.878 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:46 smithi134 ceph-mon[30086]: pgmap v533: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 853 B/s rd, 62 MiB/s wr, 998 op/s 2024-02-05T08:16:46.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:46.934 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:46.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:47.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:47.066 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:47.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:47.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:47.168 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:47.203 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:47.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:47.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:47.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:46 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:16:47.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:46 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:16:47.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:46 smithi134 ceph-mon[30086]: osdmap e271: 8 total, 8 up, 7 in 2024-02-05T08:16:47.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:46 smithi134 ceph-mon[39278]: pgmap v533: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 853 B/s rd, 62 MiB/s wr, 998 op/s 2024-02-05T08:16:47.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:46 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:16:47.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:46 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:16:47.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:46 smithi134 ceph-mon[39278]: osdmap e271: 8 total, 8 up, 7 in 2024-02-05T08:16:47.305 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:46 smithi196 ceph-mon[33018]: pgmap v533: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 853 B/s rd, 62 MiB/s wr, 998 op/s 2024-02-05T08:16:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:46 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:16:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:46 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:16:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:46 smithi196 ceph-mon[33018]: osdmap e271: 8 total, 8 up, 7 in 2024-02-05T08:16:47.336 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:47.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:47.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:47.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:47.478 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:47.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:47.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:47.621 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:47.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:47.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:47.722 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:47.758 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:47.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:47.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:47.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:47.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:47.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:47.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:48.018 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:47 smithi134 ceph-mon[30086]: 5.1c scrub starts 2024-02-05T08:16:48.018 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:47 smithi134 ceph-mon[30086]: 5.1c scrub ok 2024-02-05T08:16:48.018 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:47 smithi134 ceph-mon[30086]: 3.1f scrub starts 2024-02-05T08:16:48.018 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:47 smithi134 ceph-mon[30086]: 3.1f scrub ok 2024-02-05T08:16:48.019 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:47 smithi134 ceph-mon[39278]: 5.1c scrub starts 2024-02-05T08:16:48.019 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:47 smithi134 ceph-mon[39278]: 5.1c scrub ok 2024-02-05T08:16:48.019 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:47 smithi134 ceph-mon[39278]: 3.1f scrub starts 2024-02-05T08:16:48.019 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:47 smithi134 ceph-mon[39278]: 3.1f scrub ok 2024-02-05T08:16:48.019 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:16:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:16:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:16:48.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:48.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:48.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:48.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:48.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:48.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:48.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:48.292 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:48.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:48.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:47 smithi196 ceph-mon[33018]: 5.1c scrub starts 2024-02-05T08:16:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:47 smithi196 ceph-mon[33018]: 5.1c scrub ok 2024-02-05T08:16:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:47 smithi196 ceph-mon[33018]: 3.1f scrub starts 2024-02-05T08:16:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:47 smithi196 ceph-mon[33018]: 3.1f scrub ok 2024-02-05T08:16:48.393 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:48.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:48.429 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:48.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:48.530 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:48.558 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:48.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:48.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:48.691 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:48.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:48.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:48.792 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:48.816 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:48.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:48.918 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:48.959 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:48.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:48.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:49.061 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:49.103 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:49.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:49.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:49.204 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:49.227 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:49.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:49.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:49.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:48 smithi134 ceph-mon[30086]: pgmap v535: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.07k op/s 2024-02-05T08:16:49.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:48 smithi134 ceph-mon[39278]: pgmap v535: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.07k op/s 2024-02-05T08:16:49.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:48 smithi196 ceph-mon[33018]: pgmap v535: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.07k op/s 2024-02-05T08:16:49.329 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:49.362 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:49.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:49.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:49.464 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:49.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:49.503 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:49.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:49.525 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:49.605 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:49.639 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:49.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:49.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:49.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:49.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:49.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:49.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:49.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:49.918 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:49.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:49.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:49.990 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:50.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:49 smithi134 ceph-mon[30086]: 2.2 scrub starts 2024-02-05T08:16:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:49 smithi134 ceph-mon[30086]: 2.2 scrub ok 2024-02-05T08:16:50.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:49 smithi134 ceph-mon[30086]: 3.19 scrub starts 2024-02-05T08:16:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:49 smithi134 ceph-mon[30086]: 3.19 scrub ok 2024-02-05T08:16:50.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:49 smithi134 ceph-mon[30086]: 5.e scrub starts 2024-02-05T08:16:50.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:49 smithi134 ceph-mon[39278]: 2.2 scrub starts 2024-02-05T08:16:50.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:49 smithi134 ceph-mon[39278]: 2.2 scrub ok 2024-02-05T08:16:50.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:49 smithi134 ceph-mon[39278]: 3.19 scrub starts 2024-02-05T08:16:50.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:49 smithi134 ceph-mon[39278]: 3.19 scrub ok 2024-02-05T08:16:50.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:49 smithi134 ceph-mon[39278]: 5.e scrub starts 2024-02-05T08:16:50.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:50.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:50.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:50.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:50.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:50.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:50.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:50.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:49 smithi196 ceph-mon[33018]: 2.2 scrub starts 2024-02-05T08:16:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:49 smithi196 ceph-mon[33018]: 2.2 scrub ok 2024-02-05T08:16:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:49 smithi196 ceph-mon[33018]: 3.19 scrub starts 2024-02-05T08:16:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:49 smithi196 ceph-mon[33018]: 3.19 scrub ok 2024-02-05T08:16:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:49 smithi196 ceph-mon[33018]: 5.e scrub starts 2024-02-05T08:16:50.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:50.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:50.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:50.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:50.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:50.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:50.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:50.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:50.610 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:50.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:50.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:50.744 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:50.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:50.846 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:50.883 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:50.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:50.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:50.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:50.989 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:16:51.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:51.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:51.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:51.122 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:51.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:51.161 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:51.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:51.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:50 smithi134 ceph-mon[30086]: 5.f scrub starts 2024-02-05T08:16:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:50 smithi134 ceph-mon[30086]: 5.f scrub ok 2024-02-05T08:16:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:50 smithi134 ceph-mon[30086]: 5.e scrub ok 2024-02-05T08:16:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:50 smithi134 ceph-mon[30086]: pgmap v536: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 18 MiB/s wr, 1.09k op/s 2024-02-05T08:16:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:50 smithi134 ceph-mon[30086]: 6.9 scrub starts 2024-02-05T08:16:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:50 smithi134 ceph-mon[30086]: 6.9 scrub ok 2024-02-05T08:16:51.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:50 smithi134 ceph-mon[39278]: 5.f scrub starts 2024-02-05T08:16:51.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:50 smithi134 ceph-mon[39278]: 5.f scrub ok 2024-02-05T08:16:51.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:50 smithi134 ceph-mon[39278]: 5.e scrub ok 2024-02-05T08:16:51.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:50 smithi134 ceph-mon[39278]: pgmap v536: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 18 MiB/s wr, 1.09k op/s 2024-02-05T08:16:51.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:50 smithi134 ceph-mon[39278]: 6.9 scrub starts 2024-02-05T08:16:51.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:50 smithi134 ceph-mon[39278]: 6.9 scrub ok 2024-02-05T08:16:51.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:51.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:51.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:50 smithi196 ceph-mon[33018]: 5.f scrub starts 2024-02-05T08:16:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:50 smithi196 ceph-mon[33018]: 5.f scrub ok 2024-02-05T08:16:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:50 smithi196 ceph-mon[33018]: 5.e scrub ok 2024-02-05T08:16:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:50 smithi196 ceph-mon[33018]: pgmap v536: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 18 MiB/s wr, 1.09k op/s 2024-02-05T08:16:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:50 smithi196 ceph-mon[33018]: 6.9 scrub starts 2024-02-05T08:16:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:50 smithi196 ceph-mon[33018]: 6.9 scrub ok 2024-02-05T08:16:51.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:51.435 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:51.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:51.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:51.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:51.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:51.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:51.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:51.679 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:51.712 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:51.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:51.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:51.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:51.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:51.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:51.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:51.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:51.992 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:51.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:52.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:51 smithi134 ceph-mon[30086]: 6.c scrub starts 2024-02-05T08:16:52.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:51 smithi134 ceph-mon[30086]: 6.c scrub ok 2024-02-05T08:16:52.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:51 smithi134 ceph-mon[30086]: 6.a scrub starts 2024-02-05T08:16:52.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:51 smithi134 ceph-mon[30086]: 6.a scrub ok 2024-02-05T08:16:52.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:51 smithi134 ceph-mon[30086]: 3.4 scrub starts 2024-02-05T08:16:52.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:51 smithi134 ceph-mon[30086]: 3.4 scrub ok 2024-02-05T08:16:52.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:51 smithi134 ceph-mon[39278]: 6.c scrub starts 2024-02-05T08:16:52.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:51 smithi134 ceph-mon[39278]: 6.c scrub ok 2024-02-05T08:16:52.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:51 smithi134 ceph-mon[39278]: 6.a scrub starts 2024-02-05T08:16:52.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:51 smithi134 ceph-mon[39278]: 6.a scrub ok 2024-02-05T08:16:52.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:51 smithi134 ceph-mon[39278]: 3.4 scrub starts 2024-02-05T08:16:52.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:51 smithi134 ceph-mon[39278]: 3.4 scrub ok 2024-02-05T08:16:52.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:52.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:52.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:52.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:52.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:52.270 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:52.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:52.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:51 smithi196 ceph-mon[33018]: 6.c scrub starts 2024-02-05T08:16:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:51 smithi196 ceph-mon[33018]: 6.c scrub ok 2024-02-05T08:16:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:51 smithi196 ceph-mon[33018]: 6.a scrub starts 2024-02-05T08:16:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:51 smithi196 ceph-mon[33018]: 6.a scrub ok 2024-02-05T08:16:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:51 smithi196 ceph-mon[33018]: 3.4 scrub starts 2024-02-05T08:16:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:51 smithi196 ceph-mon[33018]: 3.4 scrub ok 2024-02-05T08:16:52.370 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:16:52.372 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:52.410 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:16:52.412 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:52.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:52.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:52.449 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:16:52.486 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:16:52.514 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:52.528 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:16:52.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:52.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:52.566 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:16:52.605 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:16:52.645 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:16:52.647 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:16:52.647 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:16:52.647 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:16:52.647 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:16:52.648 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:16:52.648 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:16:52.648 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:16:52.648 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:16:52.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:52.685 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:52.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:52.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:52.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:52.824 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:52.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:52.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:52.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:52.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:52.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:53.067 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:53.116 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:53.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:53.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:53.252 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:53.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:53.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:53.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[30086]: pgmap v537: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 778 B/s rd, 14 MiB/s wr, 832 op/s 2024-02-05T08:16:53.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[30086]: 5.1 scrub starts 2024-02-05T08:16:53.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[30086]: 5.1 scrub ok 2024-02-05T08:16:53.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[30086]: 9.0 scrub starts 2024-02-05T08:16:53.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[30086]: 6.12 scrub starts 2024-02-05T08:16:53.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[30086]: 6.12 scrub ok 2024-02-05T08:16:53.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1975817104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:53.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[39278]: pgmap v537: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 778 B/s rd, 14 MiB/s wr, 832 op/s 2024-02-05T08:16:53.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[39278]: 5.1 scrub starts 2024-02-05T08:16:53.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[39278]: 5.1 scrub ok 2024-02-05T08:16:53.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[39278]: 9.0 scrub starts 2024-02-05T08:16:53.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[39278]: 6.12 scrub starts 2024-02-05T08:16:53.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[39278]: 6.12 scrub ok 2024-02-05T08:16:53.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:52 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1975817104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:52 smithi196 ceph-mon[33018]: pgmap v537: 148 pgs: 148 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 778 B/s rd, 14 MiB/s wr, 832 op/s 2024-02-05T08:16:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:52 smithi196 ceph-mon[33018]: 5.1 scrub starts 2024-02-05T08:16:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:52 smithi196 ceph-mon[33018]: 5.1 scrub ok 2024-02-05T08:16:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:52 smithi196 ceph-mon[33018]: 9.0 scrub starts 2024-02-05T08:16:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:52 smithi196 ceph-mon[33018]: 6.12 scrub starts 2024-02-05T08:16:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:52 smithi196 ceph-mon[33018]: 6.12 scrub ok 2024-02-05T08:16:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:52 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1975817104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:16:53.355 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:53.390 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:53.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:53.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:53.491 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:53.526 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:53.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:53.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:53.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:53.661 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:53.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:53.763 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:53.798 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:53.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:53.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:53.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:16:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:16:53.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:53.949 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:53.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:53.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:54.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:54.095 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:54.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:54.196 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:54.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:54.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:54.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:54.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[30086]: 3.18 scrub starts 2024-02-05T08:16:54.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[30086]: 3.18 scrub ok 2024-02-05T08:16:54.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[30086]: 6.e scrub starts 2024-02-05T08:16:54.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[30086]: 6.e scrub ok 2024-02-05T08:16:54.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[30086]: 9.2 scrub starts 2024-02-05T08:16:54.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[30086]: 2.0 scrub starts 2024-02-05T08:16:54.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[30086]: 2.0 scrub ok 2024-02-05T08:16:54.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:16:54.240 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[30086]: 5.d scrub starts 2024-02-05T08:16:54.241 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[30086]: 5.d scrub ok 2024-02-05T08:16:54.241 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:54.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[39278]: 3.18 scrub starts 2024-02-05T08:16:54.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[39278]: 3.18 scrub ok 2024-02-05T08:16:54.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[39278]: 6.e scrub starts 2024-02-05T08:16:54.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[39278]: 6.e scrub ok 2024-02-05T08:16:54.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[39278]: 9.2 scrub starts 2024-02-05T08:16:54.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[39278]: 2.0 scrub starts 2024-02-05T08:16:54.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[39278]: 2.0 scrub ok 2024-02-05T08:16:54.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:16:54.241 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[39278]: 5.d scrub starts 2024-02-05T08:16:54.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[39278]: 5.d scrub ok 2024-02-05T08:16:54.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:53 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:53 smithi196 ceph-mon[33018]: 3.18 scrub starts 2024-02-05T08:16:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:53 smithi196 ceph-mon[33018]: 3.18 scrub ok 2024-02-05T08:16:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:53 smithi196 ceph-mon[33018]: 6.e scrub starts 2024-02-05T08:16:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:53 smithi196 ceph-mon[33018]: 6.e scrub ok 2024-02-05T08:16:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:53 smithi196 ceph-mon[33018]: 9.2 scrub starts 2024-02-05T08:16:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:53 smithi196 ceph-mon[33018]: 2.0 scrub starts 2024-02-05T08:16:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:53 smithi196 ceph-mon[33018]: 2.0 scrub ok 2024-02-05T08:16:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:16:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:53 smithi196 ceph-mon[33018]: 5.d scrub starts 2024-02-05T08:16:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:53 smithi196 ceph-mon[33018]: 5.d scrub ok 2024-02-05T08:16:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:53 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:16:54.336 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:54.370 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:54.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:54.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:54.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:54.510 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:54.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:54.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:54.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:54.650 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:54.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:54.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:54.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:54.792 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:54.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:54.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:54.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:54.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:55.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:55.068 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:55.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:55.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:55.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[30086]: 4.15 scrub starts 2024-02-05T08:16:55.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[30086]: 4.15 scrub ok 2024-02-05T08:16:55.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[30086]: pgmap v538: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-02-05T08:16:55.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[30086]: 9.0 scrub ok 2024-02-05T08:16:55.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[30086]: 5.1d scrub starts 2024-02-05T08:16:55.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[30086]: 5.1d scrub ok 2024-02-05T08:16:55.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[30086]: 6.15 scrub starts 2024-02-05T08:16:55.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[30086]: 6.15 scrub ok 2024-02-05T08:16:55.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[30086]: 5.8 scrub starts 2024-02-05T08:16:55.152 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[30086]: 5.8 scrub ok 2024-02-05T08:16:55.152 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[39278]: 4.15 scrub starts 2024-02-05T08:16:55.152 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[39278]: 4.15 scrub ok 2024-02-05T08:16:55.152 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[39278]: pgmap v538: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-02-05T08:16:55.152 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[39278]: 9.0 scrub ok 2024-02-05T08:16:55.152 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[39278]: 5.1d scrub starts 2024-02-05T08:16:55.152 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[39278]: 5.1d scrub ok 2024-02-05T08:16:55.153 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[39278]: 6.15 scrub starts 2024-02-05T08:16:55.153 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[39278]: 6.15 scrub ok 2024-02-05T08:16:55.153 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[39278]: 5.8 scrub starts 2024-02-05T08:16:55.153 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:54 smithi134 ceph-mon[39278]: 5.8 scrub ok 2024-02-05T08:16:55.158 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:55.171 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:55.212 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:55.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:55.263 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:16:55.264 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:16:55.264 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-02-05T08:16:55.264 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-02-05T08:16:55.264 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:16:55.314 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:54 smithi196 ceph-mon[33018]: 4.15 scrub starts 2024-02-05T08:16:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:54 smithi196 ceph-mon[33018]: 4.15 scrub ok 2024-02-05T08:16:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:54 smithi196 ceph-mon[33018]: pgmap v538: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 1.46k op/s 2024-02-05T08:16:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:54 smithi196 ceph-mon[33018]: 9.0 scrub ok 2024-02-05T08:16:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:54 smithi196 ceph-mon[33018]: 5.1d scrub starts 2024-02-05T08:16:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:54 smithi196 ceph-mon[33018]: 5.1d scrub ok 2024-02-05T08:16:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:54 smithi196 ceph-mon[33018]: 6.15 scrub starts 2024-02-05T08:16:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:54 smithi196 ceph-mon[33018]: 6.15 scrub ok 2024-02-05T08:16:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:54 smithi196 ceph-mon[33018]: 5.8 scrub starts 2024-02-05T08:16:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:54 smithi196 ceph-mon[33018]: 5.8 scrub ok 2024-02-05T08:16:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:55.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:55.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:55.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:55.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:55.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:55.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:55.585 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:16:55.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:55.664 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:55.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:55.766 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:55.800 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:55.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:55.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:55.934 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:55.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:56.024 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:55 smithi134 ceph-mon[30086]: 4.5 scrub starts 2024-02-05T08:16:56.024 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:55 smithi134 ceph-mon[30086]: 4.5 scrub ok 2024-02-05T08:16:56.024 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:55 smithi134 ceph-mon[30086]: 9.2 scrub ok 2024-02-05T08:16:56.025 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:55 smithi134 ceph-mon[39278]: 4.5 scrub starts 2024-02-05T08:16:56.025 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:55 smithi134 ceph-mon[39278]: 4.5 scrub ok 2024-02-05T08:16:56.025 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:55 smithi134 ceph-mon[39278]: 9.2 scrub ok 2024-02-05T08:16:56.036 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:56.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:56.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:56.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:56.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:56.228 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:56.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:55 smithi196 ceph-mon[33018]: 4.5 scrub starts 2024-02-05T08:16:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:55 smithi196 ceph-mon[33018]: 4.5 scrub ok 2024-02-05T08:16:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:55 smithi196 ceph-mon[33018]: 9.2 scrub ok 2024-02-05T08:16:56.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:56.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:56.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:56.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:56.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:56.503 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:56.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:56.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:56.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:56.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:56.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:56.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:56.755 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:56.786 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:56.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:56.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:56.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:56.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:56.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:56.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:56.947 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.4 to 1 (10000) 2024-02-05T08:16:57.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:57.057 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:57.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:57.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:57.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:57.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:57.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:57.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:57.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[30086]: 5.11 scrub starts 2024-02-05T08:16:57.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[30086]: 5.11 scrub ok 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[30086]: 9.a scrub starts 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[30086]: pgmap v539: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[30086]: 3.9 scrub starts 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[30086]: 3.9 scrub ok 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1173189978' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[39278]: 5.11 scrub starts 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[39278]: 5.11 scrub ok 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[39278]: 9.a scrub starts 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[39278]: pgmap v539: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[39278]: 3.9 scrub starts 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[39278]: 3.9 scrub ok 2024-02-05T08:16:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:16:57.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1173189978' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:16:57.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:56 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:16:57.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:56 smithi196 ceph-mon[33018]: 5.11 scrub starts 2024-02-05T08:16:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:56 smithi196 ceph-mon[33018]: 5.11 scrub ok 2024-02-05T08:16:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:56 smithi196 ceph-mon[33018]: 9.a scrub starts 2024-02-05T08:16:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:56 smithi196 ceph-mon[33018]: pgmap v539: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 11 MiB/s wr, 1.30k op/s 2024-02-05T08:16:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:56 smithi196 ceph-mon[33018]: 3.9 scrub starts 2024-02-05T08:16:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:56 smithi196 ceph-mon[33018]: 3.9 scrub ok 2024-02-05T08:16:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:16:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:56 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1173189978' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:16:57.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:56 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:16:57.330 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:57.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:57.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:57.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:57.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:57.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:57.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:57.566 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:57.600 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:57.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:57.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:57.748 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:57.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:57.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:57.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:57.887 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:57.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:57.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T08:16:57.989 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:58.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:57 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:16:58.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:57 smithi134 ceph-mon[39278]: osdmap e272: 8 total, 8 up, 7 in 2024-02-05T08:16:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:57 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:16:58.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:57 smithi134 ceph-mon[30086]: osdmap e272: 8 total, 8 up, 7 in 2024-02-05T08:16:58.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:16:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:16:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:16:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:58.030 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:58.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:58.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:16:58.188 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:58.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:16:58.291 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:57 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:16:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:57 smithi196 ceph-mon[33018]: osdmap e272: 8 total, 8 up, 7 in 2024-02-05T08:16:58.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:58.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:58.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:58.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:58.456 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:58.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:58.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:58.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:16:58.590 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:58.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:58.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:16:58.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:16:58.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:58.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:58.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:16:58.772 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 5 1 2024-02-05T08:16:58.782 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:16:58.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:58.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:58.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:58.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:58.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:16:59.007 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:59.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:16:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[30086]: 3.1e scrub starts 2024-02-05T08:16:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[30086]: 3.1e scrub ok 2024-02-05T08:16:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[30086]: pgmap v541: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 9.9 MiB/s wr, 1.29k op/s 2024-02-05T08:16:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[30086]: 5.1e scrub starts 2024-02-05T08:16:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[30086]: 5.1e scrub ok 2024-02-05T08:16:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[30086]: 3.d scrub starts 2024-02-05T08:16:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[30086]: 3.d scrub ok 2024-02-05T08:16:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:16:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:16:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:16:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[30086]: 9.a scrub ok 2024-02-05T08:16:59.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[39278]: 3.1e scrub starts 2024-02-05T08:16:59.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[39278]: 3.1e scrub ok 2024-02-05T08:16:59.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[39278]: pgmap v541: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 9.9 MiB/s wr, 1.29k op/s 2024-02-05T08:16:59.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[39278]: 5.1e scrub starts 2024-02-05T08:16:59.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[39278]: 5.1e scrub ok 2024-02-05T08:16:59.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[39278]: 3.d scrub starts 2024-02-05T08:16:59.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[39278]: 3.d scrub ok 2024-02-05T08:16:59.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:16:59.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:16:59.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:16:59.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:58 smithi134 ceph-mon[39278]: 9.a scrub ok 2024-02-05T08:16:59.109 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:59.153 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:59.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:59.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:16:59.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:16:59.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:59.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:16:59.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:58 smithi196 ceph-mon[33018]: 3.1e scrub starts 2024-02-05T08:16:59.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:58 smithi196 ceph-mon[33018]: 3.1e scrub ok 2024-02-05T08:16:59.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:58 smithi196 ceph-mon[33018]: pgmap v541: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 9.9 MiB/s wr, 1.29k op/s 2024-02-05T08:16:59.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:58 smithi196 ceph-mon[33018]: 5.1e scrub starts 2024-02-05T08:16:59.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:58 smithi196 ceph-mon[33018]: 5.1e scrub ok 2024-02-05T08:16:59.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:58 smithi196 ceph-mon[33018]: 3.d scrub starts 2024-02-05T08:16:59.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:58 smithi196 ceph-mon[33018]: 3.d scrub ok 2024-02-05T08:16:59.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:58 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:16:59.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:58 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:16:59.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:58 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:16:59.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:58 smithi196 ceph-mon[33018]: 9.a scrub ok 2024-02-05T08:16:59.397 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:59.435 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:59.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:59.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:16:59.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:59.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:16:59.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:59.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:59.720 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:59.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:16:59.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:16:59.858 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:59.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:59.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:16:59.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:16:59.994 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:16:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:16:59.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:00.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:00.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:00.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:00.233 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:00.270 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:00.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:00.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:59 smithi134 ceph-mon[30086]: 4.d scrub starts 2024-02-05T08:17:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:59 smithi134 ceph-mon[30086]: 4.d scrub ok 2024-02-05T08:17:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:59 smithi134 ceph-mon[30086]: 9.4 scrub starts 2024-02-05T08:17:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:59 smithi134 ceph-mon[30086]: 3.13 scrub starts 2024-02-05T08:17:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:59 smithi134 ceph-mon[30086]: 3.13 scrub ok 2024-02-05T08:17:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:16:59 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/178776546' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:17:00.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:59 smithi134 ceph-mon[39278]: 4.d scrub starts 2024-02-05T08:17:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:59 smithi134 ceph-mon[39278]: 4.d scrub ok 2024-02-05T08:17:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:59 smithi134 ceph-mon[39278]: 9.4 scrub starts 2024-02-05T08:17:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:59 smithi134 ceph-mon[39278]: 3.13 scrub starts 2024-02-05T08:17:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:59 smithi134 ceph-mon[39278]: 3.13 scrub ok 2024-02-05T08:17:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:16:59 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/178776546' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:17:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:59 smithi196 ceph-mon[33018]: 4.d scrub starts 2024-02-05T08:17:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:59 smithi196 ceph-mon[33018]: 4.d scrub ok 2024-02-05T08:17:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:59 smithi196 ceph-mon[33018]: 9.4 scrub starts 2024-02-05T08:17:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:59 smithi196 ceph-mon[33018]: 3.13 scrub starts 2024-02-05T08:17:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:59 smithi196 ceph-mon[33018]: 3.13 scrub ok 2024-02-05T08:17:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:16:59 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/178776546' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:17:00.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:00.408 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:00.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:00.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:00.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:00.549 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:00.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:00.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:00.651 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:00.769 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:17:00.983 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:00.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:00.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:01.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[30086]: 9.f scrub starts 2024-02-05T08:17:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[30086]: pgmap v542: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T08:17:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[30086]: 5.19 scrub starts 2024-02-05T08:17:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[30086]: 5.19 scrub ok 2024-02-05T08:17:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:17:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:17:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/178776546' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:17:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[30086]: osdmap e273: 8 total, 8 up, 7 in 2024-02-05T08:17:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[39278]: 9.f scrub starts 2024-02-05T08:17:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[39278]: pgmap v542: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T08:17:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[39278]: 5.19 scrub starts 2024-02-05T08:17:01.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[39278]: 5.19 scrub ok 2024-02-05T08:17:01.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:17:01.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:17:01.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/178776546' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:17:01.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:00 smithi134 ceph-mon[39278]: osdmap e273: 8 total, 8 up, 7 in 2024-02-05T08:17:01.085 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:01.119 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:01.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:01.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:01.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:01.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:01.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:00 smithi196 ceph-mon[33018]: 9.f scrub starts 2024-02-05T08:17:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:00 smithi196 ceph-mon[33018]: pgmap v542: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T08:17:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:00 smithi196 ceph-mon[33018]: 5.19 scrub starts 2024-02-05T08:17:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:00 smithi196 ceph-mon[33018]: 5.19 scrub ok 2024-02-05T08:17:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:00 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:17:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:00 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:17:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:00 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/178776546' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:17:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:00 smithi196 ceph-mon[33018]: osdmap e273: 8 total, 8 up, 7 in 2024-02-05T08:17:01.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:01.393 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:01.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:01.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:01.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:01.512 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:01.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:01.549 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:01.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:01.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:01.681 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:01.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:01.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:01.813 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:01.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:01.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:01.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:01.948 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:01.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:01.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:02.050 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:02.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:02.090 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:02.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:02.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:02.229 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:02.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:02.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[30086]: 5.15 scrub starts 2024-02-05T08:17:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[30086]: 5.15 scrub ok 2024-02-05T08:17:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[30086]: 9.4 scrub ok 2024-02-05T08:17:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[30086]: 3.f scrub starts 2024-02-05T08:17:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[30086]: 3.f scrub ok 2024-02-05T08:17:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[30086]: 4.19 scrub starts 2024-02-05T08:17:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[30086]: 4.19 scrub ok 2024-02-05T08:17:02.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[39278]: 5.15 scrub starts 2024-02-05T08:17:02.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[39278]: 5.15 scrub ok 2024-02-05T08:17:02.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[39278]: 9.4 scrub ok 2024-02-05T08:17:02.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[39278]: 3.f scrub starts 2024-02-05T08:17:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[39278]: 3.f scrub ok 2024-02-05T08:17:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[39278]: 4.19 scrub starts 2024-02-05T08:17:02.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:01 smithi134 ceph-mon[39278]: 4.19 scrub ok 2024-02-05T08:17:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:01 smithi196 ceph-mon[33018]: 5.15 scrub starts 2024-02-05T08:17:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:01 smithi196 ceph-mon[33018]: 5.15 scrub ok 2024-02-05T08:17:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:01 smithi196 ceph-mon[33018]: 9.4 scrub ok 2024-02-05T08:17:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:01 smithi196 ceph-mon[33018]: 3.f scrub starts 2024-02-05T08:17:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:01 smithi196 ceph-mon[33018]: 3.f scrub ok 2024-02-05T08:17:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:01 smithi196 ceph-mon[33018]: 4.19 scrub starts 2024-02-05T08:17:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:01 smithi196 ceph-mon[33018]: 4.19 scrub ok 2024-02-05T08:17:02.332 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:02.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:02.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:02.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:02.474 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:02.496 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:02.511 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:02.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:02.613 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:02.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:02.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:02.751 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:02.783 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:02.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:02.884 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:02.914 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:02.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:03.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:03.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:03.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:03.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:03.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:03.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:03.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[30086]: pgmap v544: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 759 op/s 2024-02-05T08:17:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[30086]: 9.f scrub ok 2024-02-05T08:17:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[30086]: 6.17 scrub starts 2024-02-05T08:17:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:17:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:17:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[30086]: 6.17 scrub ok 2024-02-05T08:17:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[30086]: osdmap e274: 8 total, 8 up, 7 in 2024-02-05T08:17:03.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[39278]: pgmap v544: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 759 op/s 2024-02-05T08:17:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[39278]: 9.f scrub ok 2024-02-05T08:17:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[39278]: 6.17 scrub starts 2024-02-05T08:17:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:17:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:17:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[39278]: 6.17 scrub ok 2024-02-05T08:17:03.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:02 smithi134 ceph-mon[39278]: osdmap e274: 8 total, 8 up, 7 in 2024-02-05T08:17:03.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:02 smithi196 ceph-mon[33018]: pgmap v544: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 759 op/s 2024-02-05T08:17:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:02 smithi196 ceph-mon[33018]: 9.f scrub ok 2024-02-05T08:17:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:02 smithi196 ceph-mon[33018]: 6.17 scrub starts 2024-02-05T08:17:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:02 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:17:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:02 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:17:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:02 smithi196 ceph-mon[33018]: 6.17 scrub ok 2024-02-05T08:17:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:02 smithi196 ceph-mon[33018]: osdmap e274: 8 total, 8 up, 7 in 2024-02-05T08:17:03.325 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:03.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:03.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:03.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:03.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:03.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:03.558 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:03.600 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:03.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:03.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:03.702 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:03.738 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:03.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:03.840 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:03.874 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:03.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:03.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:04.000 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:17:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:17:04.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:04 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:17:04.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:04 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:17:04.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:04 smithi196 ceph-mon[33018]: osdmap e275: 8 total, 8 up, 7 in 2024-02-05T08:17:04.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:04 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2428339698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:17:04.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:04 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:17:04.001 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:04 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:17:04.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:04 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:04.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:04.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:04.084 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:17:04.084 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:17:04.084 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:17:04.084 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:17:04.084 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:17:04.084 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:17:04.084 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:17:04.084 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:17:04.084 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:17:04.085 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:17:04.085 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:17:04.085 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:17:04.085 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:17:04.085 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:17:04.085 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:17:04.085 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:17:04.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:04.151 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:04.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:04.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:04.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:17:04.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:17:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[30086]: osdmap e275: 8 total, 8 up, 7 in 2024-02-05T08:17:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2428339698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:17:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:17:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:17:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:17:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:17:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[39278]: osdmap e275: 8 total, 8 up, 7 in 2024-02-05T08:17:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2428339698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:17:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:17:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:17:04.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:04 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:04.335 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:04.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:04.437 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:04.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:04.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:04.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:04.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:17:04.572 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:04.625 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:04.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:04.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:04.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:04.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:04.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:04.855 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:04.892 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:04.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:04.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:05.017 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:05.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:05.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:05.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:05.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:05.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:05.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:05 smithi134 ceph-mon[30086]: 9.6 scrub starts 2024-02-05T08:17:05.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:05 smithi134 ceph-mon[30086]: pgmap v547: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.00k op/s 2024-02-05T08:17:05.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:05 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:17:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:05 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:17:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:05 smithi134 ceph-mon[39278]: 9.6 scrub starts 2024-02-05T08:17:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:05 smithi134 ceph-mon[39278]: pgmap v547: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.00k op/s 2024-02-05T08:17:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:05 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:17:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:05 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:17:05.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:05.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:05.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:05 smithi196 ceph-mon[33018]: 9.6 scrub starts 2024-02-05T08:17:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:05 smithi196 ceph-mon[33018]: pgmap v547: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.00k op/s 2024-02-05T08:17:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:05 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:17:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:05 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:17:05.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:05.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:05.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:05.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:05.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:05.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:05.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:05.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:05.700 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:05.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:05.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:05.802 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:05.837 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:05.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:05.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:05.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:05.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:05.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:06.080 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:06.112 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:06.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:06.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:06.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:06.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:06.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:06.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:06 smithi134 ceph-mon[30086]: osdmap e276: 8 total, 8 up, 7 in 2024-02-05T08:17:06.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:06 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:17:06.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:06 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:17:06.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:06 smithi134 ceph-mon[39278]: osdmap e276: 8 total, 8 up, 7 in 2024-02-05T08:17:06.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:06 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:17:06.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:06 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:17:06.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:06.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:06.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:06.498 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:06.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:06.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:06.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:06 smithi196 ceph-mon[33018]: osdmap e276: 8 total, 8 up, 7 in 2024-02-05T08:17:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:06 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:17:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:06 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:17:06.639 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:06.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:06.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:06.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:06.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:07.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:07.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:07.109 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:07.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:07.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:07.244 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:07.276 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:07 smithi134 ceph-mon[30086]: pgmap v549: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T08:17:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:07 smithi134 ceph-mon[30086]: 4.0 deep-scrub starts 2024-02-05T08:17:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:07 smithi134 ceph-mon[30086]: 4.0 deep-scrub ok 2024-02-05T08:17:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:07 smithi134 ceph-mon[30086]: 9.6 scrub ok 2024-02-05T08:17:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:07 smithi134 ceph-mon[39278]: pgmap v549: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T08:17:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:07 smithi134 ceph-mon[39278]: 4.0 deep-scrub starts 2024-02-05T08:17:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:07 smithi134 ceph-mon[39278]: 4.0 deep-scrub ok 2024-02-05T08:17:07.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:07 smithi134 ceph-mon[39278]: 9.6 scrub ok 2024-02-05T08:17:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:07.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:07.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:07.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:07.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:07.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:07.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:07.560 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:07.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:07.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:07 smithi196 ceph-mon[33018]: pgmap v549: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T08:17:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:07 smithi196 ceph-mon[33018]: 4.0 deep-scrub starts 2024-02-05T08:17:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:07 smithi196 ceph-mon[33018]: 4.0 deep-scrub ok 2024-02-05T08:17:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:07 smithi196 ceph-mon[33018]: 9.6 scrub ok 2024-02-05T08:17:07.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:07.692 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:07.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:07.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:07.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:07.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:07.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:07.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:07.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:07.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:08.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:17:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:17:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:17:08.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:08.130 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:08.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:08.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:08.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:08.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:08.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:08.347 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[30086]: 3.0 scrub starts 2024-02-05T08:17:08.347 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[30086]: 3.0 scrub ok 2024-02-05T08:17:08.347 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:17:08.347 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:17:08.348 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[30086]: osdmap e277: 8 total, 8 up, 7 in 2024-02-05T08:17:08.348 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[30086]: 3.1a deep-scrub starts 2024-02-05T08:17:08.348 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[30086]: 3.1a deep-scrub ok 2024-02-05T08:17:08.348 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[39278]: 3.0 scrub starts 2024-02-05T08:17:08.348 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[39278]: 3.0 scrub ok 2024-02-05T08:17:08.348 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:17:08.348 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:17:08.348 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[39278]: osdmap e277: 8 total, 8 up, 7 in 2024-02-05T08:17:08.348 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[39278]: 3.1a deep-scrub starts 2024-02-05T08:17:08.348 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:08 smithi134 ceph-mon[39278]: 3.1a deep-scrub ok 2024-02-05T08:17:08.369 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:08.408 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:08.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:08.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:08.543 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:08.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:08 smithi196 ceph-mon[33018]: 3.0 scrub starts 2024-02-05T08:17:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:08 smithi196 ceph-mon[33018]: 3.0 scrub ok 2024-02-05T08:17:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:08 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:17:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:08 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:17:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:08 smithi196 ceph-mon[33018]: osdmap e277: 8 total, 8 up, 7 in 2024-02-05T08:17:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:08 smithi196 ceph-mon[33018]: 3.1a deep-scrub starts 2024-02-05T08:17:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:08 smithi196 ceph-mon[33018]: 3.1a deep-scrub ok 2024-02-05T08:17:08.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:08.678 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:08.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:08.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:08.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:08.810 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:08.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:08.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:08.945 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:08.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:09.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:09.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:09.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:09.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:09.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:09.240 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:09.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:09.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:09.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:09.378 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:09.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:09.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:09 smithi134 ceph-mon[30086]: pgmap v551: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 889 B/s rd, 0 B/s wr, 1.16k op/s 2024-02-05T08:17:09.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:09 smithi134 ceph-mon[30086]: 5.3 deep-scrub starts 2024-02-05T08:17:09.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:09 smithi134 ceph-mon[30086]: 5.3 deep-scrub ok 2024-02-05T08:17:09.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:09 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:17:09.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:09 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:17:09.465 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:09 smithi134 ceph-mon[39278]: pgmap v551: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 889 B/s rd, 0 B/s wr, 1.16k op/s 2024-02-05T08:17:09.466 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:09 smithi134 ceph-mon[39278]: 5.3 deep-scrub starts 2024-02-05T08:17:09.466 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:09 smithi134 ceph-mon[39278]: 5.3 deep-scrub ok 2024-02-05T08:17:09.466 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:09 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:17:09.466 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:09 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:17:09.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:09.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:09.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:09.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:09 smithi196 ceph-mon[33018]: pgmap v551: 148 pgs: 148 active+clean; 6.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 889 B/s rd, 0 B/s wr, 1.16k op/s 2024-02-05T08:17:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:09 smithi196 ceph-mon[33018]: 5.3 deep-scrub starts 2024-02-05T08:17:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:09 smithi196 ceph-mon[33018]: 5.3 deep-scrub ok 2024-02-05T08:17:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:09 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:17:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:09 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:17:09.614 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:09.653 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:09.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:09.755 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:09.788 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:09.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:09.889 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:09.921 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:09.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:10.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:10.053 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:10.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:10.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:10.126 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.5 to 1 (10000) 2024-02-05T08:17:10.155 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:10.187 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:10.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:10.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:10.329 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:10.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:10.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:10.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:10.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:10.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[30086]: 3.1c scrub starts 2024-02-05T08:17:10.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[30086]: 3.1c scrub ok 2024-02-05T08:17:10.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[30086]: 6.10 deep-scrub starts 2024-02-05T08:17:10.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[30086]: 6.10 deep-scrub ok 2024-02-05T08:17:10.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:17:10.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1262564699' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:17:10.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:17:10.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:17:10.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:17:10.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[39278]: 3.1c scrub starts 2024-02-05T08:17:10.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[39278]: 3.1c scrub ok 2024-02-05T08:17:10.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[39278]: 6.10 deep-scrub starts 2024-02-05T08:17:10.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[39278]: 6.10 deep-scrub ok 2024-02-05T08:17:10.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:17:10.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1262564699' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:17:10.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:17:10.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:17:10.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:10 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:17:10.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:10 smithi196 ceph-mon[33018]: 3.1c scrub starts 2024-02-05T08:17:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:10 smithi196 ceph-mon[33018]: 3.1c scrub ok 2024-02-05T08:17:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:10 smithi196 ceph-mon[33018]: 6.10 deep-scrub starts 2024-02-05T08:17:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:10 smithi196 ceph-mon[33018]: 6.10 deep-scrub ok 2024-02-05T08:17:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:10 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:17:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:10 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1262564699' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:17:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:10 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:17:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:10 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:17:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:10 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:17:10.587 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:10.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:10.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:10.720 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:10.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:10.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:10.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:10.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:10.957 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:10.989 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:10.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:10.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:11.072 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:11.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:11.126 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:11.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:11.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:11.228 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:11.258 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:11.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:11.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:11.360 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:11.394 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:11.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:11.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:11.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:17:11.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[30086]: pgmap v552: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 758 B/s rd, 0 B/s wr, 1.52k op/s 2024-02-05T08:17:11.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[30086]: 9.c deep-scrub starts 2024-02-05T08:17:11.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[30086]: 5.9 deep-scrub starts 2024-02-05T08:17:11.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[30086]: 5.9 deep-scrub ok 2024-02-05T08:17:11.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[30086]: 4.10 scrub starts 2024-02-05T08:17:11.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[30086]: 4.10 scrub ok 2024-02-05T08:17:11.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:17:11.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[30086]: osdmap e278: 8 total, 8 up, 7 in 2024-02-05T08:17:11.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:17:11.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:17:11.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:17:11.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[39278]: pgmap v552: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 758 B/s rd, 0 B/s wr, 1.52k op/s 2024-02-05T08:17:11.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[39278]: 9.c deep-scrub starts 2024-02-05T08:17:11.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[39278]: 5.9 deep-scrub starts 2024-02-05T08:17:11.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[39278]: 5.9 deep-scrub ok 2024-02-05T08:17:11.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[39278]: 4.10 scrub starts 2024-02-05T08:17:11.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[39278]: 4.10 scrub ok 2024-02-05T08:17:11.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:17:11.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[39278]: osdmap e278: 8 total, 8 up, 7 in 2024-02-05T08:17:11.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:17:11.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:11 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:17:11.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:11.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:11.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:11 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:17:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:11 smithi196 ceph-mon[33018]: pgmap v552: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 758 B/s rd, 0 B/s wr, 1.52k op/s 2024-02-05T08:17:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:11 smithi196 ceph-mon[33018]: 9.c deep-scrub starts 2024-02-05T08:17:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:11 smithi196 ceph-mon[33018]: 5.9 deep-scrub starts 2024-02-05T08:17:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:11 smithi196 ceph-mon[33018]: 5.9 deep-scrub ok 2024-02-05T08:17:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:11 smithi196 ceph-mon[33018]: 4.10 scrub starts 2024-02-05T08:17:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:11 smithi196 ceph-mon[33018]: 4.10 scrub ok 2024-02-05T08:17:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:11 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:17:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:11 smithi196 ceph-mon[33018]: osdmap e278: 8 total, 8 up, 7 in 2024-02-05T08:17:11.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:11 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:17:11.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:11 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:17:11.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:11.672 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:11.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:11.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:11.774 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:11.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:11.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:11.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:11.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:11.948 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:11.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:12.049 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:12.082 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:12.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:12.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:12.217 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:12.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:12.319 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:12.348 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:12.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:12.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:12.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:12.474 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:12.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:12.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:12.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[39278]: 6.2 scrub starts 2024-02-05T08:17:12.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[39278]: 6.2 scrub ok 2024-02-05T08:17:12.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[39278]: 6.1c scrub starts 2024-02-05T08:17:12.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[39278]: 6.1c scrub ok 2024-02-05T08:17:12.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[39278]: 4.16 deep-scrub starts 2024-02-05T08:17:12.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[39278]: 4.16 deep-scrub ok 2024-02-05T08:17:12.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:17:12.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:17:12.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[39278]: 4.13 scrub starts 2024-02-05T08:17:12.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[39278]: 4.13 scrub ok 2024-02-05T08:17:12.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[30086]: 6.2 scrub starts 2024-02-05T08:17:12.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[30086]: 6.2 scrub ok 2024-02-05T08:17:12.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[30086]: 6.1c scrub starts 2024-02-05T08:17:12.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[30086]: 6.1c scrub ok 2024-02-05T08:17:12.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[30086]: 4.16 deep-scrub starts 2024-02-05T08:17:12.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[30086]: 4.16 deep-scrub ok 2024-02-05T08:17:12.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:17:12.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:17:12.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[30086]: 4.13 scrub starts 2024-02-05T08:17:12.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:12 smithi134 ceph-mon[30086]: 4.13 scrub ok 2024-02-05T08:17:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:12 smithi196 ceph-mon[33018]: 6.2 scrub starts 2024-02-05T08:17:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:12 smithi196 ceph-mon[33018]: 6.2 scrub ok 2024-02-05T08:17:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:12 smithi196 ceph-mon[33018]: 6.1c scrub starts 2024-02-05T08:17:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:12 smithi196 ceph-mon[33018]: 6.1c scrub ok 2024-02-05T08:17:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:12 smithi196 ceph-mon[33018]: 4.16 deep-scrub starts 2024-02-05T08:17:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:12 smithi196 ceph-mon[33018]: 4.16 deep-scrub ok 2024-02-05T08:17:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:12 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:17:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:12 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:17:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:12 smithi196 ceph-mon[33018]: 4.13 scrub starts 2024-02-05T08:17:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:12 smithi196 ceph-mon[33018]: 4.13 scrub ok 2024-02-05T08:17:12.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:12.599 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:12.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:12.701 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:12.730 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:12.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:12.832 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:12.864 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:12.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:12.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:12.966 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:12.994 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:12.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:12.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:13.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:13.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:13.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:13.191 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:17:13.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:13.268 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:13.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:13.370 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:13.400 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:13.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:13.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:13.464 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:13 smithi196 ceph-mon[33018]: pgmap v554: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 766 B/s rd, 0 B/s wr, 891 op/s 2024-02-05T08:17:13.465 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:13 smithi196 ceph-mon[33018]: 4.11 scrub starts 2024-02-05T08:17:13.465 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:13 smithi196 ceph-mon[33018]: 4.11 scrub ok 2024-02-05T08:17:13.465 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:13 smithi196 ceph-mon[33018]: 5.c scrub starts 2024-02-05T08:17:13.465 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:13 smithi196 ceph-mon[33018]: 5.c scrub ok 2024-02-05T08:17:13.465 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:13 smithi196 ceph-mon[33018]: 4.1 scrub starts 2024-02-05T08:17:13.465 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:13 smithi196 ceph-mon[33018]: 4.1 scrub ok 2024-02-05T08:17:13.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:13.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[30086]: pgmap v554: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 766 B/s rd, 0 B/s wr, 891 op/s 2024-02-05T08:17:13.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[30086]: 4.11 scrub starts 2024-02-05T08:17:13.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[30086]: 4.11 scrub ok 2024-02-05T08:17:13.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[30086]: 5.c scrub starts 2024-02-05T08:17:13.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[30086]: 5.c scrub ok 2024-02-05T08:17:13.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[30086]: 4.1 scrub starts 2024-02-05T08:17:13.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[30086]: 4.1 scrub ok 2024-02-05T08:17:13.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[39278]: pgmap v554: 148 pgs: 148 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 766 B/s rd, 0 B/s wr, 891 op/s 2024-02-05T08:17:13.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[39278]: 4.11 scrub starts 2024-02-05T08:17:13.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[39278]: 4.11 scrub ok 2024-02-05T08:17:13.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[39278]: 5.c scrub starts 2024-02-05T08:17:13.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[39278]: 5.c scrub ok 2024-02-05T08:17:13.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[39278]: 4.1 scrub starts 2024-02-05T08:17:13.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:13 smithi134 ceph-mon[39278]: 4.1 scrub ok 2024-02-05T08:17:13.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:13.539 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:13.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:13.640 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:13.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 7 1 2024-02-05T08:17:13.674 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:13.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:13.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:13.775 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:13.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:17:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:17:13.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:13.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:13.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:13.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:13.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:13.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:13.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:14.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:14.093 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:14.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:14.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:14.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:14.227 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:14.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:14.329 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:14.370 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:14.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:14.401 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[30086]: 9.7 scrub starts 2024-02-05T08:17:14.401 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[30086]: 2.1 scrub starts 2024-02-05T08:17:14.402 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[30086]: 2.1 scrub ok 2024-02-05T08:17:14.402 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[30086]: 3.e scrub starts 2024-02-05T08:17:14.402 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[30086]: 3.e scrub ok 2024-02-05T08:17:14.402 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[30086]: 9.5 deep-scrub starts 2024-02-05T08:17:14.402 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:14.402 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[30086]: 4.f scrub starts 2024-02-05T08:17:14.402 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[30086]: 4.f scrub ok 2024-02-05T08:17:14.402 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[39278]: 9.7 scrub starts 2024-02-05T08:17:14.402 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[39278]: 2.1 scrub starts 2024-02-05T08:17:14.402 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[39278]: 2.1 scrub ok 2024-02-05T08:17:14.403 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[39278]: 3.e scrub starts 2024-02-05T08:17:14.403 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[39278]: 3.e scrub ok 2024-02-05T08:17:14.403 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[39278]: 9.5 deep-scrub starts 2024-02-05T08:17:14.403 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:14.403 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[39278]: 4.f scrub starts 2024-02-05T08:17:14.403 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:14 smithi134 ceph-mon[39278]: 4.f scrub ok 2024-02-05T08:17:14.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:14.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:14.506 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:14.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:14.507 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:17:14.508 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:17:14.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:14 smithi196 ceph-mon[33018]: 9.7 scrub starts 2024-02-05T08:17:14.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:14 smithi196 ceph-mon[33018]: 2.1 scrub starts 2024-02-05T08:17:14.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:14 smithi196 ceph-mon[33018]: 2.1 scrub ok 2024-02-05T08:17:14.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:14 smithi196 ceph-mon[33018]: 3.e scrub starts 2024-02-05T08:17:14.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:14 smithi196 ceph-mon[33018]: 3.e scrub ok 2024-02-05T08:17:14.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:14 smithi196 ceph-mon[33018]: 9.5 deep-scrub starts 2024-02-05T08:17:14.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:14 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:14.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:14 smithi196 ceph-mon[33018]: 4.f scrub starts 2024-02-05T08:17:14.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:14 smithi196 ceph-mon[33018]: 4.f scrub ok 2024-02-05T08:17:14.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:14.646 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:14.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:14.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:14.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:14.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:14.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:14.885 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:14.921 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:14.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:15.022 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:15.054 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:15.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:15.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:15.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:15.191 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:15.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:15.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:15.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[30086]: 3.2 scrub starts 2024-02-05T08:17:15.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[30086]: 3.2 scrub ok 2024-02-05T08:17:15.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[30086]: 9.7 scrub ok 2024-02-05T08:17:15.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[30086]: pgmap v555: 148 pgs: 2 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 144 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-02-05T08:17:15.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[30086]: 6.b deep-scrub starts 2024-02-05T08:17:15.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[30086]: 5.0 scrub starts 2024-02-05T08:17:15.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[30086]: 5.0 scrub ok 2024-02-05T08:17:15.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[30086]: 9.b scrub starts 2024-02-05T08:17:15.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[30086]: 3.a scrub starts 2024-02-05T08:17:15.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[30086]: 3.a scrub ok 2024-02-05T08:17:15.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[30086]: 6.b deep-scrub ok 2024-02-05T08:17:15.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[30086]: 6.1a scrub starts 2024-02-05T08:17:15.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[30086]: 6.1a scrub ok 2024-02-05T08:17:15.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[39278]: 3.2 scrub starts 2024-02-05T08:17:15.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[39278]: 3.2 scrub ok 2024-02-05T08:17:15.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[39278]: 9.7 scrub ok 2024-02-05T08:17:15.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[39278]: pgmap v555: 148 pgs: 2 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 144 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-02-05T08:17:15.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[39278]: 6.b deep-scrub starts 2024-02-05T08:17:15.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[39278]: 5.0 scrub starts 2024-02-05T08:17:15.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[39278]: 5.0 scrub ok 2024-02-05T08:17:15.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[39278]: 9.b scrub starts 2024-02-05T08:17:15.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[39278]: 3.a scrub starts 2024-02-05T08:17:15.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[39278]: 3.a scrub ok 2024-02-05T08:17:15.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[39278]: 6.b deep-scrub ok 2024-02-05T08:17:15.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[39278]: 6.1a scrub starts 2024-02-05T08:17:15.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:15 smithi134 ceph-mon[39278]: 6.1a scrub ok 2024-02-05T08:17:15.293 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:15.324 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:15.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:15.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:15.451 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:15.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:15.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:15.553 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:15 smithi196 ceph-mon[33018]: 3.2 scrub starts 2024-02-05T08:17:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:15 smithi196 ceph-mon[33018]: 3.2 scrub ok 2024-02-05T08:17:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:15 smithi196 ceph-mon[33018]: 9.7 scrub ok 2024-02-05T08:17:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:15 smithi196 ceph-mon[33018]: pgmap v555: 148 pgs: 2 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 144 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s 2024-02-05T08:17:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:15 smithi196 ceph-mon[33018]: 6.b deep-scrub starts 2024-02-05T08:17:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:15 smithi196 ceph-mon[33018]: 5.0 scrub starts 2024-02-05T08:17:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:15 smithi196 ceph-mon[33018]: 5.0 scrub ok 2024-02-05T08:17:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:15 smithi196 ceph-mon[33018]: 9.b scrub starts 2024-02-05T08:17:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:15 smithi196 ceph-mon[33018]: 3.a scrub starts 2024-02-05T08:17:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:15 smithi196 ceph-mon[33018]: 3.a scrub ok 2024-02-05T08:17:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:15 smithi196 ceph-mon[33018]: 6.b deep-scrub ok 2024-02-05T08:17:15.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:15 smithi196 ceph-mon[33018]: 6.1a scrub starts 2024-02-05T08:17:15.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:15 smithi196 ceph-mon[33018]: 6.1a scrub ok 2024-02-05T08:17:15.579 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:15.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:15.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:15.707 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:15.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:15.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:15.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:15.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:15.841 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:15.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:15.942 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:15.968 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:15.980 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:15.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:15.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:16.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T08:17:16.082 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:16.136 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:16.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:16.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:16.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:16.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:16.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:16.283 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:16 smithi134 ceph-mon[30086]: 9.b scrub ok 2024-02-05T08:17:16.283 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:16 smithi134 ceph-mon[39278]: 9.b scrub ok 2024-02-05T08:17:16.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:16.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:16.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:16.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:16.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:16.554 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:16.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:16.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:16 smithi196 ceph-mon[33018]: 9.b scrub ok 2024-02-05T08:17:16.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:16.687 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:16.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:16.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:16.820 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:16.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:16.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:16.922 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:16.962 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:16.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:17.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:17.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:17.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:17.199 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:17.231 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:17.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:17.332 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:17.361 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:17.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:17.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:17.473 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:17 smithi134 ceph-mon[30086]: pgmap v556: 148 pgs: 2 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 144 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T08:17:17.473 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:17 smithi134 ceph-mon[30086]: 9.e scrub starts 2024-02-05T08:17:17.473 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:17 smithi134 ceph-mon[39278]: pgmap v556: 148 pgs: 2 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 144 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T08:17:17.474 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:17 smithi134 ceph-mon[39278]: 9.e scrub starts 2024-02-05T08:17:17.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:17.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:17.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:17.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:17 smithi196 ceph-mon[33018]: pgmap v556: 148 pgs: 2 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 144 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T08:17:17.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:17 smithi196 ceph-mon[33018]: 9.e scrub starts 2024-02-05T08:17:17.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:17.635 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:17.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:17.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:17.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:17.777 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:17.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:17.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:17.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:17.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:17.912 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:17.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:18.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:18.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:17:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:17:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:17:18.045 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:18.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:18.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:18.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:18.176 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:17:18.183 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:18.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:18.285 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:18.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:18.320 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:18.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:18.417 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[30086]: 6.1d scrub starts 2024-02-05T08:17:18.417 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[30086]: 6.1d scrub ok 2024-02-05T08:17:18.417 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[30086]: 4.1c scrub starts 2024-02-05T08:17:18.417 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[30086]: 4.1c scrub ok 2024-02-05T08:17:18.417 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[30086]: 9.e scrub ok 2024-02-05T08:17:18.417 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1977432877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:17:18.417 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:17:18.417 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[39278]: 6.1d scrub starts 2024-02-05T08:17:18.418 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[39278]: 6.1d scrub ok 2024-02-05T08:17:18.418 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[39278]: 4.1c scrub starts 2024-02-05T08:17:18.418 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[39278]: 4.1c scrub ok 2024-02-05T08:17:18.418 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[39278]: 9.e scrub ok 2024-02-05T08:17:18.418 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1977432877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:17:18.418 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:18 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:17:18.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:18.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:18.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:18.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:18.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:18 smithi196 ceph-mon[33018]: 6.1d scrub starts 2024-02-05T08:17:18.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:18 smithi196 ceph-mon[33018]: 6.1d scrub ok 2024-02-05T08:17:18.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:18 smithi196 ceph-mon[33018]: 4.1c scrub starts 2024-02-05T08:17:18.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:18 smithi196 ceph-mon[33018]: 4.1c scrub ok 2024-02-05T08:17:18.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:18 smithi196 ceph-mon[33018]: 9.e scrub ok 2024-02-05T08:17:18.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:18 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1977432877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:17:18.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:18 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:17:18.598 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:18.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:18.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:18.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:18.733 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:18.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:18.835 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:18.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:18.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:18.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:18.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:18.998 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:18.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:19.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:19.129 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:19.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:19.231 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:19.261 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:19.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:19.276 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:19.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:19.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:19.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:19.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:19.404 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[30086]: pgmap v557: 148 pgs: 2 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 144 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-05T08:17:19.404 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[30086]: 4.14 scrub starts 2024-02-05T08:17:19.404 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[30086]: 4.14 scrub ok 2024-02-05T08:17:19.404 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[30086]: 9.d scrub starts 2024-02-05T08:17:19.404 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:17:19.404 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:17:19.404 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[30086]: osdmap e279: 8 total, 8 up, 7 in 2024-02-05T08:17:19.405 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[39278]: pgmap v557: 148 pgs: 2 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 144 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-05T08:17:19.405 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[39278]: 4.14 scrub starts 2024-02-05T08:17:19.405 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[39278]: 4.14 scrub ok 2024-02-05T08:17:19.405 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[39278]: 9.d scrub starts 2024-02-05T08:17:19.405 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:17:19.405 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:17:19.405 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:19 smithi134 ceph-mon[39278]: osdmap e279: 8 total, 8 up, 7 in 2024-02-05T08:17:19.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:19.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:19.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:19.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:19 smithi196 ceph-mon[33018]: pgmap v557: 148 pgs: 2 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 144 active+clean; 5.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.13k op/s 2024-02-05T08:17:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:19 smithi196 ceph-mon[33018]: 4.14 scrub starts 2024-02-05T08:17:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:19 smithi196 ceph-mon[33018]: 4.14 scrub ok 2024-02-05T08:17:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:19 smithi196 ceph-mon[33018]: 9.d scrub starts 2024-02-05T08:17:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:19 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:17:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:19 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:17:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:19 smithi196 ceph-mon[33018]: osdmap e279: 8 total, 8 up, 7 in 2024-02-05T08:17:19.590 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:17:19.626 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:17:19.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:19.656 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:17:19.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:19.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:19.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:19.684 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:17:19.718 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:17:19.748 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:17:19.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:19.777 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:17:19.799 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:19.806 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:19.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:19.819 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:17:19.822 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:17:19.822 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:17:19.822 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:17:19.822 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:17:19.822 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:17:19.822 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:17:19.822 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:17:19.822 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:17:19.907 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:19.940 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:19.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:20.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:20.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:20.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:20.175 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:20.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:20.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:20.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:20.309 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:20.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:20.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:20.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:20.445 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:20.476 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:20 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1915713158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:17:20.476 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:20 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1915713158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:17:20.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:20.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:20.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:20 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1915713158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:17:20.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:20.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:20.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:20.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:20.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:20.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:20.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:20.893 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:20.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:20.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:20.995 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:21.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:21.028 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:21.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:21.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:21.158 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:21.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:21.260 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:21.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:21 smithi134 ceph-mon[30086]: pgmap v559: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 977 op/s 2024-02-05T08:17:21.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:21 smithi134 ceph-mon[39278]: pgmap v559: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 977 op/s 2024-02-05T08:17:21.288 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:21.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:21.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:21.390 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:21.425 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:21.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:21.527 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:21.560 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:21.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:21.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:21 smithi196 ceph-mon[33018]: pgmap v559: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 977 op/s 2024-02-05T08:17:21.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:21.691 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:21.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:21.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:21.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:21.825 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:21.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:21.927 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:21.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:21.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:21.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:22.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:22.097 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:22.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:22.199 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:22.228 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:22.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:22.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:22.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:17:22.361 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:22.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:22.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:22.464 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:22.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:22.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:22.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:22.594 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:22.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:22.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:22.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:22.768 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:22.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:22.869 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:22.897 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:22.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:22.998 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:23.027 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:23.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:23.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:23.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:23.163 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:23.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:23.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:23.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:23.294 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:23.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:23.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:23.395 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:23.408 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:17:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:23.424 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:23.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:23.451 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:23 smithi134 ceph-mon[30086]: pgmap v560: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 940 op/s 2024-02-05T08:17:23.451 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:17:23.451 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:17:23.452 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:23 smithi134 ceph-mon[39278]: pgmap v560: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 940 op/s 2024-02-05T08:17:23.452 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:17:23.452 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:17:23.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:23 smithi196 ceph-mon[33018]: pgmap v560: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 940 op/s 2024-02-05T08:17:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:17:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:17:23.574 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:23.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:23.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:23.715 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:23.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:23.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:23.743 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:23.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:23.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:23.853 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:23.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:23.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:23.986 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:23.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:23.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:24.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:17:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:17:24.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:24.122 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:24.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:24.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:24.223 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:24.251 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:24.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:24.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:24.353 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:24.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:24.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:24.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:24.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:24.521 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:24.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:24.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:24 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:24.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:24 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:24.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:24 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:24.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:24.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:24.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:24.757 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:24.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:24.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:24.892 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:24.920 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:24.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:24.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:25.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:25.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:25.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:25.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:25.195 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:25.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:25.297 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:25.331 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:25.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:25.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:25.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:25.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:25.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:25.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:25 smithi134 ceph-mon[30086]: pgmap v561: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 804 op/s 2024-02-05T08:17:25.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:25 smithi134 ceph-mon[39278]: pgmap v561: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 804 op/s 2024-02-05T08:17:25.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:25 smithi196 ceph-mon[33018]: pgmap v561: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 804 op/s 2024-02-05T08:17:25.566 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:25.601 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:25.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:25.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:25.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:25.735 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:25.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:25.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:25.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:25.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:25.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:25.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:25.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:26.015 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:26.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:26.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:26.117 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:26.149 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:26.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:26.167 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:17:26.167 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:17:26.167 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-02-05T08:17:26.167 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:17:26.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:26.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:26.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:26.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:26.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:26.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:26.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:26.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:26.518 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:26.548 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:26.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:26.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:26.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:26.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:26.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:26.784 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:26.814 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:26.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:26.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:26.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:26.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:26.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:26.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:27.055 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:27.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:27.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:27.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:27.200 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.7 to 1 (10000) 2024-02-05T08:17:27.228 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:27.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:27.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:27.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:27.364 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:27.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:27.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:27.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:27.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:27.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:27.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:27 smithi134 ceph-mon[30086]: pgmap v562: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 804 op/s 2024-02-05T08:17:27.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:27 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4006213782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:17:27.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:27 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:17:27.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:27 smithi134 ceph-mon[39278]: pgmap v562: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 804 op/s 2024-02-05T08:17:27.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:27 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4006213782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:17:27.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:27 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:17:27.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:27 smithi196 ceph-mon[33018]: pgmap v562: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 804 op/s 2024-02-05T08:17:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:27 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4006213782' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:17:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:27 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:17:27.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:27.633 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:27.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:27.735 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:27.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:27.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:27.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:27.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:27.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:27.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:28.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:28.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:17:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:17:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:17:28.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:28.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:28.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:28.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:28.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:28.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:28.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:28.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:28.305 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:28.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:28.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:28.407 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:28.437 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:28.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:28.482 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:28 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:17:28.482 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:28 smithi134 ceph-mon[30086]: osdmap e280: 8 total, 8 up, 7 in 2024-02-05T08:17:28.483 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:28 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:17:28.483 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:28 smithi134 ceph-mon[39278]: osdmap e280: 8 total, 8 up, 7 in 2024-02-05T08:17:28.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:28.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:28 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:17:28.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:28 smithi196 ceph-mon[33018]: osdmap e280: 8 total, 8 up, 7 in 2024-02-05T08:17:28.575 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:28.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:28.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:28.677 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:28.709 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:28.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:28.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:28.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:28.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T08:17:28.844 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:28.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:28.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:28.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:28.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:28.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:28.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:29.083 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:29.144 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:29.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:29.245 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:29.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:29.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:29.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:29.412 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:29.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:29.514 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:29.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:29 smithi134 ceph-mon[30086]: pgmap v564: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 534 B/s rd, 0 B/s wr, 639 op/s 2024-02-05T08:17:29.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:29 smithi134 ceph-mon[39278]: pgmap v564: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 534 B/s rd, 0 B/s wr, 639 op/s 2024-02-05T08:17:29.546 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:29.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:29.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:29.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:29 smithi196 ceph-mon[33018]: pgmap v564: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 534 B/s rd, 0 B/s wr, 639 op/s 2024-02-05T08:17:29.647 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:29.680 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:29.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:29.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:29.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:29.814 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:29.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:29.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:29.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 6 1 2024-02-05T08:17:29.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:29.952 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:29.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:29.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:30.054 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:30.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:30.087 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:30.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:30.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:30.225 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:30.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:30.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:30.366 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:30.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:30.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:30.432 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:30.432 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:30.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:30.509 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:30.608 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:30.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:30.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:30.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:30.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:30.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:30.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:30.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:30.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:30.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:30.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:30.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:31.009 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:31.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:31.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:31.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:31.143 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:31.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:31.235 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:17:31.245 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:31.286 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:31.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:31.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:31.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:31.420 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:31.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:31.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:31.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:31.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:31 smithi134 ceph-mon[30086]: pgmap v565: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 815 op/s 2024-02-05T08:17:31.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:31 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/129266560' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:17:31.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:31 smithi134 ceph-mon[39278]: pgmap v565: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 815 op/s 2024-02-05T08:17:31.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:31 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/129266560' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:17:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:31.558 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:31.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:31 smithi196 ceph-mon[33018]: pgmap v565: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 815 op/s 2024-02-05T08:17:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:31 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/129266560' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:17:31.659 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:31.698 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:31.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:31.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:31.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:31.836 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:31.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:31.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:31.970 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:31.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:32.072 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:32.107 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:32.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:32.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:32.250 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:32.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:32.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:32 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:17:32.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:32 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/129266560' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:17:32.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:32 smithi134 ceph-mon[30086]: osdmap e281: 8 total, 8 up, 7 in 2024-02-05T08:17:32.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:32 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:17:32.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:32 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/129266560' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:17:32.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:32 smithi134 ceph-mon[39278]: osdmap e281: 8 total, 8 up, 7 in 2024-02-05T08:17:32.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:32.395 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:32.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:32.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:32.497 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:32.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:32.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:32.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:32 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:17:32.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:32 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/129266560' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:17:32.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:32 smithi196 ceph-mon[33018]: osdmap e281: 8 total, 8 up, 7 in 2024-02-05T08:17:32.636 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:32.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:32.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:32.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:32.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:32.808 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:32.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:32.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:32.943 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:32.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:32.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:33.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:33.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:33.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:33.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:33.186 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:33.221 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:33.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:33.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:33.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:33.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:33.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:33.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:33.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:33.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:33.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:33 smithi134 ceph-mon[30086]: pgmap v567: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 498 op/s 2024-02-05T08:17:33.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:33 smithi134 ceph-mon[39278]: pgmap v567: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 498 op/s 2024-02-05T08:17:33.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:33 smithi196 ceph-mon[33018]: pgmap v567: 148 pgs: 2 active+clean+scrubbing+deep, 146 active+clean; 5.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 498 op/s 2024-02-05T08:17:33.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:33.636 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:33.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:33.736 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:33.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:33.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:33.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:33.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:33.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:33.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:33.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:34.011 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:34.045 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:34.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:34.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:34.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:17:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:17:34.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:34.181 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:34.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:34.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:34.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:34.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:34.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:34.452 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:34.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:34.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:34.455 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:17:34.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:34.591 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:34.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:34.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:34.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:34.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:34.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:34.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:34.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:34.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:34.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:34.963 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:34.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:35.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:35.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:35.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:35.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:35.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:35.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:35.317 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:35.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:35.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:35.456 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:35.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:35.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:35.480 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:35 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:35.481 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:35 smithi134 ceph-mon[30086]: pgmap v568: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T08:17:35.481 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:35 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:35.481 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:35 smithi134 ceph-mon[39278]: pgmap v568: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T08:17:35.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:35 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:35 smithi196 ceph-mon[33018]: pgmap v568: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T08:17:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:35.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:35.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:35.699 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:35.735 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:35.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:35.837 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:35.872 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:35.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:35.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:35.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:36.011 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:36.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:36.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:36.113 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:36.146 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:36.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:36.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:36.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:36.261 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.6 to 1 (10000) 2024-02-05T08:17:36.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:36.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:36.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:36.411 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:17:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:36.422 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:36.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:36.445 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:17:36.476 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:17:36.506 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:17:36.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2796611143' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:17:36.507 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:36 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:17:36.507 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2796611143' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:17:36.507 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:36 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:17:36.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:36.541 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:17:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:36.559 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:36.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:36.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2796611143' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:17:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:36 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:17:36.575 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:17:36.612 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:17:36.649 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:17:36.651 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:17:36.651 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:17:36.651 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:17:36.651 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:17:36.651 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:17:36.651 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:17:36.651 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:17:36.652 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:17:36.660 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:36.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:36.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:36.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:36.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:36.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:36.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:36.931 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:36.969 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:36.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:36.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:37.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:37.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:37.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:37.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:37.245 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:37.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:37.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:37.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:37.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:37.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:37.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:37.520 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:37.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:37.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:37.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:37 smithi134 ceph-mon[30086]: pgmap v569: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-02-05T08:17:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:37 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T08:17:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:37 smithi134 ceph-mon[30086]: osdmap e282: 8 total, 8 up, 7 in 2024-02-05T08:17:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:37 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/874347957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:17:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:37 smithi134 ceph-mon[39278]: pgmap v569: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-02-05T08:17:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:37 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T08:17:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:37 smithi134 ceph-mon[39278]: osdmap e282: 8 total, 8 up, 7 in 2024-02-05T08:17:37.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:37 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/874347957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:17:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:37 smithi196 ceph-mon[33018]: pgmap v569: 148 pgs: 148 active+clean; 4.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-02-05T08:17:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:37 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T08:17:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:37 smithi196 ceph-mon[33018]: osdmap e282: 8 total, 8 up, 7 in 2024-02-05T08:17:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:37 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/874347957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:17:37.622 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:37.653 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-05T08:17:37.653 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:17:37.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:37.685 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:37.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:37.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:37.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:37.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:37.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:37.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:37.938 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:17:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:17:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:17:37.973 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:37.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:38.075 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:38.112 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:38.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:38.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:38.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:38.248 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:38.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:38.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:38.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:38.385 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:38.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:38.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:38.520 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:38.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:38.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:17:38.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:17:38.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:17:38.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:38.662 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:38.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:38.684 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:38.764 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:38.773 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:17:38.773 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:17:38.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:38.808 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:38.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:38.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:38.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:38.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:38.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:39.051 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:39.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:39.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:39.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:39.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:39.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:39.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:39.388 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:39.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:39.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:39.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:39.522 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:39.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:39.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:39 smithi134 ceph-mon[30086]: pgmap v571: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 689 op/s 2024-02-05T08:17:39.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:39 smithi134 ceph-mon[39278]: pgmap v571: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 689 op/s 2024-02-05T08:17:39.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:39 smithi196 ceph-mon[33018]: pgmap v571: 148 pgs: 148 active+clean; 4.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 689 op/s 2024-02-05T08:17:39.624 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:39.658 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:39.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:39.760 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:39.793 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:39.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:39.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:39.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:39.931 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:39.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:39.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:40.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:40.064 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:40.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:40.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:40.166 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:40.213 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:40.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:40.226 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T08:17:40.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:40.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:40.360 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:40.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:40.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:40.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:17:40.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:40.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:40.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:40.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:40.635 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:40.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:40.737 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:40.772 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:40.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:40.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:40.907 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:40.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:40.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:41.009 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:41.039 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:41.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:41.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:41.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:41.177 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:41.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:41.278 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:41.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:41.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:41.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:41.407 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:41.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:41.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:41.463 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:41.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:41.515 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:41.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:41 smithi134 ceph-mon[30086]: pgmap v572: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-02-05T08:17:41.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:41 smithi134 ceph-mon[39278]: pgmap v572: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-02-05T08:17:41.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:41 smithi196 ceph-mon[33018]: pgmap v572: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-02-05T08:17:41.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:41.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:41.597 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:41.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:41.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:41.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:41.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:41.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:41.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:41.869 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:41.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:41.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:41.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:42.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:42.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:42.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:42.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:42.147 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:42.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:42.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:42.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:42.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:42.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:42.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:42.421 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:42.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:42.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:42.564 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:42.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:42.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:42.642 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:42.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:42.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:42.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:42.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:42.845 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:42.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:42.946 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:42.979 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:42.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:42.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:43.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:43.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:43.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:43.222 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:43.257 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:43.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:43.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:43.285 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:17:43.358 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:43.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:43.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:43.497 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:43.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:43 smithi134 ceph-mon[30086]: pgmap v573: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-02-05T08:17:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:43 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2813092788' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:17:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:43 smithi134 ceph-mon[39278]: pgmap v573: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-02-05T08:17:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:43 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2813092788' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:17:43.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:43.536 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:43.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:43 smithi196 ceph-mon[33018]: pgmap v573: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s 2024-02-05T08:17:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:43 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2813092788' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:17:43.636 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:43.672 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:43.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:43.774 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:43.809 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:43.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:43.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:43.947 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:43.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:43.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:44.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:44.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:17:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:17:44.089 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:44.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:44.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:44.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:44.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:44.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:44.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:44.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:44.365 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:44.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:44.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:44.466 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:44.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:44 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:17:44.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:44 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2813092788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:17:44.495 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:44 smithi134 ceph-mon[30086]: osdmap e283: 8 total, 8 up, 7 in 2024-02-05T08:17:44.495 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:44 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:17:44.495 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:44 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2813092788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:17:44.495 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:44 smithi134 ceph-mon[39278]: osdmap e283: 8 total, 8 up, 7 in 2024-02-05T08:17:44.498 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:44.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:44.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:44 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:17:44.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:44 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2813092788' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:17:44.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:44 smithi196 ceph-mon[33018]: osdmap e283: 8 total, 8 up, 7 in 2024-02-05T08:17:44.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:44.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:44.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:44.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:44.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:44.780 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:44.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:44.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:44.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:44.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:44.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:45.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:45.055 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:45.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:45.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:45.188 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:45.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:45.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:45.329 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:45.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:45.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:45.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:45.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:45.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:45 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:45.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:45 smithi134 ceph-mon[30086]: pgmap v575: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-02-05T08:17:45.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:45 smithi134 ceph-mon[30086]: 9.d scrub starts 2024-02-05T08:17:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:45 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:45 smithi134 ceph-mon[39278]: pgmap v575: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-02-05T08:17:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:45 smithi134 ceph-mon[39278]: 9.d scrub starts 2024-02-05T08:17:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:45 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:45 smithi196 ceph-mon[33018]: pgmap v575: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s 2024-02-05T08:17:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:45 smithi196 ceph-mon[33018]: 9.d scrub starts 2024-02-05T08:17:45.567 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:45.597 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:45.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:45.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:45.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:45.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:45.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:45.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:45.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:45.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:45.977 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:46.019 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:46.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:46.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:46.070 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:17:46.080 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:17:46.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:17:46.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:46.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:46.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:46.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:46.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:46.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:46.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:46.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:46.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:46.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:46 smithi134 ceph-mon[30086]: 9.d scrub ok 2024-02-05T08:17:46.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:46 smithi134 ceph-mon[39278]: 9.d scrub ok 2024-02-05T08:17:46.534 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:46.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:46 smithi196 ceph-mon[33018]: 9.d scrub ok 2024-02-05T08:17:46.573 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:46.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:46.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:46.675 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:46.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:46.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:46.824 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:46.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:46.859 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:46.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:46.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:46.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:46.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:46.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:47.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:47.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:47.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:47.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:47.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:47.272 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:47.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:47.373 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:47.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:47.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:47.508 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:47.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:47.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:47 smithi196 ceph-mon[33018]: pgmap v576: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-02-05T08:17:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:47 smithi196 ceph-mon[33018]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:17:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:47 smithi196 ceph-mon[33018]: 9.5 scrub starts 2024-02-05T08:17:47.651 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:47.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:47.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:47.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:47 smithi134 ceph-mon[30086]: pgmap v576: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-02-05T08:17:47.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:47 smithi134 ceph-mon[30086]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:17:47.741 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:47 smithi134 ceph-mon[30086]: 9.5 scrub starts 2024-02-05T08:17:47.742 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:47 smithi134 ceph-mon[39278]: pgmap v576: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-02-05T08:17:47.742 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:47 smithi134 ceph-mon[39278]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:17:47.742 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:47 smithi134 ceph-mon[39278]: 9.5 scrub starts 2024-02-05T08:17:47.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:47.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:47.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:47.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:47.947 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:47.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:48.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:17:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:17:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:17:48.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:48.083 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:48.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:48.185 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:48.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:48.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:48.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:48.319 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:48.351 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:48.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:48.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:48.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:48.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:48.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:48.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[30086]: 5.b scrub starts 2024-02-05T08:17:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[30086]: 5.b scrub ok 2024-02-05T08:17:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[30086]: 9.c scrub starts 2024-02-05T08:17:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[30086]: 5.12 scrub starts 2024-02-05T08:17:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[30086]: 5.12 scrub ok 2024-02-05T08:17:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[30086]: 9.8 scrub starts 2024-02-05T08:17:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:17:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:17:48.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[39278]: 5.b scrub starts 2024-02-05T08:17:48.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[39278]: 5.b scrub ok 2024-02-05T08:17:48.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[39278]: 9.c scrub starts 2024-02-05T08:17:48.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[39278]: 5.12 scrub starts 2024-02-05T08:17:48.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[39278]: 5.12 scrub ok 2024-02-05T08:17:48.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[39278]: 9.8 scrub starts 2024-02-05T08:17:48.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:17:48.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:48 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:17:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:48 smithi196 ceph-mon[33018]: 5.b scrub starts 2024-02-05T08:17:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:48 smithi196 ceph-mon[33018]: 5.b scrub ok 2024-02-05T08:17:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:48 smithi196 ceph-mon[33018]: 9.c scrub starts 2024-02-05T08:17:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:48 smithi196 ceph-mon[33018]: 5.12 scrub starts 2024-02-05T08:17:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:48 smithi196 ceph-mon[33018]: 5.12 scrub ok 2024-02-05T08:17:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:48 smithi196 ceph-mon[33018]: 9.8 scrub starts 2024-02-05T08:17:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:48 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:17:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:48 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:17:48.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:48.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:48.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:48.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:48.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:48.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:48.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:48.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:48.853 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:48.887 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:48.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:48.989 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:49.020 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:49.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:49.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:49.122 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:49.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:49.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:49.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:49.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:49.290 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:49.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:49.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:49.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:49.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:49.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:49.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:49.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:49 smithi196 ceph-mon[33018]: 4.7 scrub starts 2024-02-05T08:17:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:49 smithi196 ceph-mon[33018]: 4.7 scrub ok 2024-02-05T08:17:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:49 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:17:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:49 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:17:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:49 smithi196 ceph-mon[33018]: pgmap v577: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-02-05T08:17:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:49 smithi196 ceph-mon[33018]: 9.5 scrub ok 2024-02-05T08:17:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:49 smithi196 ceph-mon[33018]: 9.c scrub ok 2024-02-05T08:17:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:49 smithi196 ceph-mon[33018]: osdmap e284: 8 total, 8 up, 7 in 2024-02-05T08:17:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:49 smithi196 ceph-mon[33018]: 3.17 scrub starts 2024-02-05T08:17:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:49 smithi196 ceph-mon[33018]: 3.17 scrub ok 2024-02-05T08:17:49.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:49.565 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:49.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:49.666 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:49.700 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:49.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:49.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:49.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[30086]: 4.7 scrub starts 2024-02-05T08:17:49.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[30086]: 4.7 scrub ok 2024-02-05T08:17:49.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:17:49.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:17:49.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[30086]: pgmap v577: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-02-05T08:17:49.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[30086]: 9.5 scrub ok 2024-02-05T08:17:49.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[30086]: 9.c scrub ok 2024-02-05T08:17:49.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[30086]: osdmap e284: 8 total, 8 up, 7 in 2024-02-05T08:17:49.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[30086]: 3.17 scrub starts 2024-02-05T08:17:49.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[30086]: 3.17 scrub ok 2024-02-05T08:17:49.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[39278]: 4.7 scrub starts 2024-02-05T08:17:49.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[39278]: 4.7 scrub ok 2024-02-05T08:17:49.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:17:49.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:17:49.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[39278]: pgmap v577: 148 pgs: 148 active+clean; 4.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-02-05T08:17:49.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[39278]: 9.5 scrub ok 2024-02-05T08:17:49.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[39278]: 9.c scrub ok 2024-02-05T08:17:49.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[39278]: osdmap e284: 8 total, 8 up, 7 in 2024-02-05T08:17:49.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[39278]: 3.17 scrub starts 2024-02-05T08:17:49.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:49 smithi134 ceph-mon[39278]: 3.17 scrub ok 2024-02-05T08:17:49.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:49.839 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:49.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:49.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:49.967 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:49.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:49.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:50.069 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:50.101 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:50.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:50.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:50.203 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:50.232 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:50.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:50.334 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:50.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:50.368 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:50.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:50.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:50.507 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:50.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:50.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:50 smithi196 ceph-mon[33018]: 9.8 scrub ok 2024-02-05T08:17:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:50 smithi196 ceph-mon[33018]: 3.10 scrub starts 2024-02-05T08:17:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:50 smithi196 ceph-mon[33018]: 3.10 scrub ok 2024-02-05T08:17:50.608 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:50.643 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:50.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:50.745 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:50.775 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:50.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:50.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:50 smithi134 ceph-mon[30086]: 9.8 scrub ok 2024-02-05T08:17:50.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:50 smithi134 ceph-mon[30086]: 3.10 scrub starts 2024-02-05T08:17:50.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:50 smithi134 ceph-mon[30086]: 3.10 scrub ok 2024-02-05T08:17:50.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:50 smithi134 ceph-mon[39278]: 9.8 scrub ok 2024-02-05T08:17:50.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:50 smithi134 ceph-mon[39278]: 3.10 scrub starts 2024-02-05T08:17:50.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:50 smithi134 ceph-mon[39278]: 3.10 scrub ok 2024-02-05T08:17:50.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:50.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:50.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:50.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:51.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:51.054 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:51.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:51.156 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:51.195 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:51.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:51.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:51.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:51.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:51.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:51.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:51.464 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:51.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:51.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:51.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:51.603 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:51.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:51.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:51.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:51.750 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:51.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:51.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:51 smithi134 ceph-mon[30086]: pgmap v579: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-02-05T08:17:51.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:51 smithi134 ceph-mon[30086]: 3.8 scrub starts 2024-02-05T08:17:51.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:51 smithi134 ceph-mon[30086]: 3.8 scrub ok 2024-02-05T08:17:51.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:51 smithi134 ceph-mon[30086]: 3.5 scrub starts 2024-02-05T08:17:51.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:51 smithi134 ceph-mon[30086]: 3.5 scrub ok 2024-02-05T08:17:51.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:51 smithi134 ceph-mon[30086]: 9.9 scrub starts 2024-02-05T08:17:51.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:51 smithi134 ceph-mon[39278]: pgmap v579: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-02-05T08:17:51.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:51 smithi134 ceph-mon[39278]: 3.8 scrub starts 2024-02-05T08:17:51.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:51 smithi134 ceph-mon[39278]: 3.8 scrub ok 2024-02-05T08:17:51.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:51 smithi134 ceph-mon[39278]: 3.5 scrub starts 2024-02-05T08:17:51.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:51 smithi134 ceph-mon[39278]: 3.5 scrub ok 2024-02-05T08:17:51.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:51 smithi134 ceph-mon[39278]: 9.9 scrub starts 2024-02-05T08:17:51.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:51 smithi196 ceph-mon[33018]: pgmap v579: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-02-05T08:17:51.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:51 smithi196 ceph-mon[33018]: 3.8 scrub starts 2024-02-05T08:17:51.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:51 smithi196 ceph-mon[33018]: 3.8 scrub ok 2024-02-05T08:17:51.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:51 smithi196 ceph-mon[33018]: 3.5 scrub starts 2024-02-05T08:17:51.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:51 smithi196 ceph-mon[33018]: 3.5 scrub ok 2024-02-05T08:17:51.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:51 smithi196 ceph-mon[33018]: 9.9 scrub starts 2024-02-05T08:17:51.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:51.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:51.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:51.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:51.916 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":576,"stamp":"2024-02-05T08:17:45.750875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4350482804,"num_objects":66573,"num_object_clones":0,"num_object_copies":199719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66573,"num_whiteouts":0,"num_read":2127,"num_read_kb":1973,"num_write":198002,"num_write_kb":8433630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":2957685,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75937,"ondisk_log_size":75937,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16017648,"kb_used_data":12881012,"kb_used_omap":10,"kb_used_meta":3136373,"kb_avail":640198416,"statfs":{"total":671965249536,"available":655563177984,"internally_reserved":0,"allocated":13190156288,"data_stored":13188022650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":3211646058},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-861208576,"num_objects":-13141,"num_object_clones":0,"num_object_copies":-39423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-13141,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13141,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.499027"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":481,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.256515+0000","last_change":"2024-02-05T08:17:14.644253+0000","last_active":"2024-02-05T08:17:36.256515+0000","last_peered":"2024-02-05T08:17:36.256515+0000","last_clean":"2024-02-05T08:17:36.256515+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:17:36.256515+0000","last_undegraded":"2024-02-05T08:17:36.256515+0000","last_fullsized":"2024-02-05T08:17:36.256515+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:14.644144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:14.644144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:35.696086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":488,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.289843+0000","last_change":"2024-02-05T08:16:59.970092+0000","last_active":"2024-02-05T08:17:43.289843+0000","last_peered":"2024-02-05T08:17:43.289843+0000","last_clean":"2024-02-05T08:17:43.289843+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:17:43.289843+0000","last_undegraded":"2024-02-05T08:17:43.289843+0000","last_fullsized":"2024-02-05T08:17:43.289843+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:59.969957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:59.969957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:01.007561+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":487,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292908+0000","last_change":"2024-02-05T08:16:46.841747+0000","last_active":"2024-02-05T08:17:43.292908+0000","last_peered":"2024-02-05T08:17:43.292908+0000","last_clean":"2024-02-05T08:17:43.292908+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:17:43.292908+0000","last_undegraded":"2024-02-05T08:17:43.292908+0000","last_fullsized":"2024-02-05T08:17:43.292908+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:46.841644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:46.841644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:06.877982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"236'19","reported_seq":542,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.256204+0000","last_change":"2024-02-05T08:16:03.779092+0000","last_active":"2024-02-05T08:17:36.256204+0000","last_peered":"2024-02-05T08:17:36.256204+0000","last_clean":"2024-02-05T08:17:36.256204+0000","last_became_active":"2024-02-05T08:12:29.675863+0000","last_became_peered":"2024-02-05T08:12:29.675863+0000","last_unstale":"2024-02-05T08:17:36.256204+0000","last_undegraded":"2024-02-05T08:17:36.256204+0000","last_fullsized":"2024-02-05T08:17:36.256204+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:16:03.778966+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:16:03.778966+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:40.868794+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":468,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.268067+0000","last_change":"2024-02-05T08:15:22.548444+0000","last_active":"2024-02-05T08:17:36.268067+0000","last_peered":"2024-02-05T08:17:36.268067+0000","last_clean":"2024-02-05T08:17:36.268067+0000","last_became_active":"2024-02-05T08:12:41.722414+0000","last_became_peered":"2024-02-05T08:12:41.722414+0000","last_unstale":"2024-02-05T08:17:36.268067+0000","last_undegraded":"2024-02-05T08:17:36.268067+0000","last_fullsized":"2024-02-05T08:17:36.268067+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.548159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:44.018977+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":489,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.256343+0000","last_change":"2024-02-05T08:16:13.845746+0000","last_active":"2024-02-05T08:17:36.256343+0000","last_peered":"2024-02-05T08:17:36.256343+0000","last_clean":"2024-02-05T08:17:36.256343+0000","last_became_active":"2024-02-05T08:12:29.676482+0000","last_became_peered":"2024-02-05T08:12:29.676482+0000","last_unstale":"2024-02-05T08:17:36.256343+0000","last_undegraded":"2024-02-05T08:17:36.256343+0000","last_fullsized":"2024-02-05T08:17:36.256343+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:13.845566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:16:13.845566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:37.779806+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":211,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.276935+0000","last_change":"2024-02-05T08:16:57.294836+0000","last_active":"2024-02-05T08:17:36.276935+0000","last_peered":"2024-02-05T08:17:36.276935+0000","last_clean":"2024-02-05T08:17:36.276935+0000","last_became_active":"2024-02-05T08:12:41.727450+0000","last_became_peered":"2024-02-05T08:12:41.727450+0000","last_unstale":"2024-02-05T08:17:36.276935+0000","last_undegraded":"2024-02-05T08:17:36.276935+0000","last_fullsized":"2024-02-05T08:17:36.276935+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:57.294696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:16:57.294696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:18.513025+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"236'34","reported_seq":392,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.256735+0000","last_change":"2024-02-05T08:17:01.432645+0000","last_active":"2024-02-05T08:17:36.256735+0000","last_peered":"2024-02-05T08:17:36.256735+0000","last_clean":"2024-02-05T08:17:36.256735+0000","last_became_active":"2024-02-05T08:12:41.745917+0000","last_became_peered":"2024-02-05T08:12:41.745917+0000","last_unstale":"2024-02-05T08:17:36.256735+0000","last_undegraded":"2024-02-05T08:17:36.256735+0000","last_fullsized":"2024-02-05T08:17:36.256735+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:17:01.432518+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:17:01.432518+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:29.324174+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":474,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.290942+0000","last_change":"2024-02-05T08:16:14.944841+0000","last_active":"2024-02-05T08:17:43.290942+0000","last_peered":"2024-02-05T08:17:43.290942+0000","last_clean":"2024-02-05T08:17:43.290942+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:17:43.290942+0000","last_undegraded":"2024-02-05T08:17:43.290942+0000","last_fullsized":"2024-02-05T08:17:43.290942+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:14.944722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:14.944722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":477,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.261920+0000","last_change":"2024-02-05T08:15:22.941148+0000","last_active":"2024-02-05T08:17:36.261920+0000","last_peered":"2024-02-05T08:17:36.261920+0000","last_clean":"2024-02-05T08:17:36.261920+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:17:36.261920+0000","last_undegraded":"2024-02-05T08:17:36.261920+0000","last_fullsized":"2024-02-05T08:17:36.261920+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.940396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.940396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":505,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.270965+0000","last_change":"2024-02-05T08:16:11.809956+0000","last_active":"2024-02-05T08:17:36.270965+0000","last_peered":"2024-02-05T08:17:36.270965+0000","last_clean":"2024-02-05T08:17:36.270965+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:17:36.270965+0000","last_undegraded":"2024-02-05T08:17:36.270965+0000","last_fullsized":"2024-02-05T08:17:36.270965+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:11.809710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:11.809710+0000","last_clean_scrub_stamp":"2024-02-05T08:16:11.809710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"263'19","reported_seq":532,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.256751+0000","last_change":"2024-02-05T08:16:07.826180+0000","last_active":"2024-02-05T08:17:36.256751+0000","last_peered":"2024-02-05T08:17:36.256751+0000","last_clean":"2024-02-05T08:17:36.256751+0000","last_became_active":"2024-02-05T08:12:41.731452+0000","last_became_peered":"2024-02-05T08:12:41.731452+0000","last_unstale":"2024-02-05T08:17:36.256751+0000","last_undegraded":"2024-02-05T08:17:36.256751+0000","last_fullsized":"2024-02-05T08:17:36.256751+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:16:07.826018+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:40.868893+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":479,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.262023+0000","last_change":"2024-02-05T08:15:56.898965+0000","last_active":"2024-02-05T08:17:36.262023+0000","last_peered":"2024-02-05T08:17:36.262023+0000","last_clean":"2024-02-05T08:17:36.262023+0000","last_became_active":"2024-02-05T08:12:41.721611+0000","last_became_peered":"2024-02-05T08:12:41.721611+0000","last_unstale":"2024-02-05T08:17:36.262023+0000","last_undegraded":"2024-02-05T08:17:36.262023+0000","last_fullsized":"2024-02-05T08:17:36.262023+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:56.898547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:56.898547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":478,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.269367+0000","last_change":"2024-02-05T08:15:18.440781+0000","last_active":"2024-02-05T08:17:36.269367+0000","last_peered":"2024-02-05T08:17:36.269367+0000","last_clean":"2024-02-05T08:17:36.269367+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:17:36.269367+0000","last_undegraded":"2024-02-05T08:17:36.269367+0000","last_fullsized":"2024-02-05T08:17:36.269367+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:18.440664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:18.440664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.010201+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":493,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.266127+0000","last_change":"2024-02-05T08:17:08.551310+0000","last_active":"2024-02-05T08:17:36.266127+0000","last_peered":"2024-02-05T08:17:36.266127+0000","last_clean":"2024-02-05T08:17:36.266127+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:17:36.266127+0000","last_undegraded":"2024-02-05T08:17:36.266127+0000","last_fullsized":"2024-02-05T08:17:36.266127+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:08.551190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:17:08.551190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:22.015564+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":506,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.257204+0000","last_change":"2024-02-05T08:16:05.810845+0000","last_active":"2024-02-05T08:17:36.257204+0000","last_peered":"2024-02-05T08:17:36.257204+0000","last_clean":"2024-02-05T08:17:36.257204+0000","last_became_active":"2024-02-05T08:12:41.730485+0000","last_became_peered":"2024-02-05T08:12:41.730485+0000","last_unstale":"2024-02-05T08:17:36.257204+0000","last_undegraded":"2024-02-05T08:17:36.257204+0000","last_fullsized":"2024-02-05T08:17:36.257204+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-02-05T08:16:05.810713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:05.810713+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:40.868860+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":486,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.257789+0000","last_change":"2024-02-05T08:15:13.039454+0000","last_active":"2024-02-05T08:17:36.257789+0000","last_peered":"2024-02-05T08:17:36.257789+0000","last_clean":"2024-02-05T08:17:36.257789+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:17:36.257789+0000","last_undegraded":"2024-02-05T08:17:36.257789+0000","last_fullsized":"2024-02-05T08:17:36.257789+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:13.039321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:15:13.039321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:40.868688+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":480,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.290801+0000","last_change":"2024-02-05T08:16:53.947563+0000","last_active":"2024-02-05T08:17:43.290801+0000","last_peered":"2024-02-05T08:17:43.290801+0000","last_clean":"2024-02-05T08:17:43.290801+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:17:43.290801+0000","last_undegraded":"2024-02-05T08:17:43.290801+0000","last_fullsized":"2024-02-05T08:17:43.290801+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:53.947466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:16:53.947466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:07.435429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":478,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292329+0000","last_change":"2024-02-05T08:15:22.341597+0000","last_active":"2024-02-05T08:17:43.292329+0000","last_peered":"2024-02-05T08:17:43.292329+0000","last_clean":"2024-02-05T08:17:43.292329+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:17:43.292329+0000","last_undegraded":"2024-02-05T08:17:43.292329+0000","last_fullsized":"2024-02-05T08:17:43.292329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.341491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.341491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.344762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"236'34","reported_seq":577,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.257615+0000","last_change":"2024-02-05T08:17:17.405954+0000","last_active":"2024-02-05T08:17:36.257615+0000","last_peered":"2024-02-05T08:17:36.257615+0000","last_clean":"2024-02-05T08:17:36.257615+0000","last_became_active":"2024-02-05T08:12:41.727027+0000","last_became_peered":"2024-02-05T08:12:41.727027+0000","last_unstale":"2024-02-05T08:17:36.257615+0000","last_undegraded":"2024-02-05T08:17:36.257615+0000","last_fullsized":"2024-02-05T08:17:36.257615+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:17:17.405818+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:17:17.405818+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:26.313166+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":474,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.277292+0000","last_change":"2024-02-05T08:16:02.397422+0000","last_active":"2024-02-05T08:17:36.277292+0000","last_peered":"2024-02-05T08:17:36.277292+0000","last_clean":"2024-02-05T08:17:36.277292+0000","last_became_active":"2024-02-05T08:12:41.713564+0000","last_became_peered":"2024-02-05T08:12:41.713564+0000","last_unstale":"2024-02-05T08:17:36.277292+0000","last_undegraded":"2024-02-05T08:17:36.277292+0000","last_fullsized":"2024-02-05T08:17:36.277292+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:02.396921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:02.396921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.125668+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":491,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.260988+0000","last_change":"2024-02-05T08:16:46.786563+0000","last_active":"2024-02-05T08:17:36.260988+0000","last_peered":"2024-02-05T08:17:36.260988+0000","last_clean":"2024-02-05T08:17:36.260988+0000","last_became_active":"2024-02-05T08:12:41.735240+0000","last_became_peered":"2024-02-05T08:12:41.735240+0000","last_unstale":"2024-02-05T08:17:36.260988+0000","last_undegraded":"2024-02-05T08:17:36.260988+0000","last_fullsized":"2024-02-05T08:17:36.260988+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:46.786289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:16:46.786289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:50.852392+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":485,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.256846+0000","last_change":"2024-02-05T08:17:07.717997+0000","last_active":"2024-02-05T08:17:36.256846+0000","last_peered":"2024-02-05T08:17:36.256846+0000","last_clean":"2024-02-05T08:17:36.256846+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:17:36.256846+0000","last_undegraded":"2024-02-05T08:17:36.256846+0000","last_fullsized":"2024-02-05T08:17:36.256846+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:17:07.717888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:14.855138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"236'23","reported_seq":307,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.290658+0000","last_change":"2024-02-05T08:16:12.024893+0000","last_active":"2024-02-05T08:17:43.290658+0000","last_peered":"2024-02-05T08:17:43.290658+0000","last_clean":"2024-02-05T08:17:43.290658+0000","last_became_active":"2024-02-05T08:12:41.716794+0000","last_became_peered":"2024-02-05T08:12:41.716794+0000","last_unstale":"2024-02-05T08:17:43.290658+0000","last_undegraded":"2024-02-05T08:17:43.290658+0000","last_fullsized":"2024-02-05T08:17:43.290658+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'23","last_scrub_stamp":"2024-02-05T08:16:12.024808+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:16:12.024808+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":492,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.272038+0000","last_change":"2024-02-05T08:17:10.980274+0000","last_active":"2024-02-05T08:17:36.272038+0000","last_peered":"2024-02-05T08:17:36.272038+0000","last_clean":"2024-02-05T08:17:36.272038+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:17:36.272038+0000","last_undegraded":"2024-02-05T08:17:36.272038+0000","last_fullsized":"2024-02-05T08:17:36.272038+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:17:10.980170+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:17:10.980170+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:15.656865+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":290,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.741666+0000","last_change":"2024-02-05T08:17:11.092748+0000","last_active":"2024-02-05T08:17:36.741666+0000","last_peered":"2024-02-05T08:17:36.741666+0000","last_clean":"2024-02-05T08:17:36.741666+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:17:36.741666+0000","last_undegraded":"2024-02-05T08:17:36.741666+0000","last_fullsized":"2024-02-05T08:17:36.741666+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:17:11.092613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:11.092613+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:15.479387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":499,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.289758+0000","last_change":"2024-02-05T08:16:49.034265+0000","last_active":"2024-02-05T08:17:43.289758+0000","last_peered":"2024-02-05T08:17:43.289758+0000","last_clean":"2024-02-05T08:17:43.289758+0000","last_became_active":"2024-02-05T08:12:41.719010+0000","last_became_peered":"2024-02-05T08:12:41.719010+0000","last_unstale":"2024-02-05T08:17:43.289758+0000","last_undegraded":"2024-02-05T08:17:43.289758+0000","last_fullsized":"2024-02-05T08:17:43.289758+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:16:49.034140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:49.034140+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:49.728450+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"236'20","reported_seq":559,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292510+0000","last_change":"2024-02-05T08:15:26.443674+0000","last_active":"2024-02-05T08:17:43.292510+0000","last_peered":"2024-02-05T08:17:43.292510+0000","last_clean":"2024-02-05T08:17:43.292510+0000","last_became_active":"2024-02-05T08:12:41.757718+0000","last_became_peered":"2024-02-05T08:12:41.757718+0000","last_unstale":"2024-02-05T08:17:43.292510+0000","last_undegraded":"2024-02-05T08:17:43.292510+0000","last_fullsized":"2024-02-05T08:17:43.292510+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:15:26.443406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:26.443406+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.344679+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":484,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.290919+0000","last_change":"2024-02-05T08:17:17.079508+0000","last_active":"2024-02-05T08:17:43.290919+0000","last_peered":"2024-02-05T08:17:43.290919+0000","last_clean":"2024-02-05T08:17:43.290919+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:17:43.290919+0000","last_undegraded":"2024-02-05T08:17:43.290919+0000","last_fullsized":"2024-02-05T08:17:43.290919+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:17.079382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:17.079382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:37.501898+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":479,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292760+0000","last_change":"2024-02-05T08:16:57.872130+0000","last_active":"2024-02-05T08:17:43.292760+0000","last_peered":"2024-02-05T08:17:43.292760+0000","last_clean":"2024-02-05T08:17:43.292760+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:17:43.292760+0000","last_undegraded":"2024-02-05T08:17:43.292760+0000","last_fullsized":"2024-02-05T08:17:43.292760+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:57.871993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:32.231861+0000","last_clean_scrub_stamp":"2024-02-05T08:16:57.871993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:17.033329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":207,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.290866+0000","last_change":"2024-02-05T08:16:52.804014+0000","last_active":"2024-02-05T08:17:43.290866+0000","last_peered":"2024-02-05T08:17:43.290866+0000","last_clean":"2024-02-05T08:17:43.290866+0000","last_became_active":"2024-02-05T08:12:41.727406+0000","last_became_peered":"2024-02-05T08:12:41.727406+0000","last_unstale":"2024-02-05T08:17:43.290866+0000","last_undegraded":"2024-02-05T08:17:43.290866+0000","last_fullsized":"2024-02-05T08:17:43.290866+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:52.803901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:16:52.803901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:59.825509+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"236'19","reported_seq":239,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.277388+0000","last_change":"2024-02-05T08:16:08.369341+0000","last_active":"2024-02-05T08:17:36.277388+0000","last_peered":"2024-02-05T08:17:36.277388+0000","last_clean":"2024-02-05T08:17:36.277388+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:17:36.277388+0000","last_undegraded":"2024-02-05T08:17:36.277388+0000","last_fullsized":"2024-02-05T08:17:36.277388+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:16:08.369158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:08.369158+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:41.168389+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":474,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.269497+0000","last_change":"2024-02-05T08:16:52.009766+0000","last_active":"2024-02-05T08:17:36.269497+0000","last_peered":"2024-02-05T08:17:36.269497+0000","last_clean":"2024-02-05T08:17:36.269497+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:17:36.269497+0000","last_undegraded":"2024-02-05T08:17:36.269497+0000","last_fullsized":"2024-02-05T08:17:36.269497+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:52.009657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:52.009657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:18.660392+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":215,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.275429+0000","last_change":"2024-02-05T08:16:55.299725+0000","last_active":"2024-02-05T08:17:36.275429+0000","last_peered":"2024-02-05T08:17:36.275429+0000","last_clean":"2024-02-05T08:17:36.275429+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:17:36.275429+0000","last_undegraded":"2024-02-05T08:17:36.275429+0000","last_fullsized":"2024-02-05T08:17:36.275429+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:55.299617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:16:55.299617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:18.988856+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":484,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.293497+0000","last_change":"2024-02-05T08:15:11.364628+0000","last_active":"2024-02-05T08:17:43.293497+0000","last_peered":"2024-02-05T08:17:43.293497+0000","last_clean":"2024-02-05T08:17:43.293497+0000","last_became_active":"2024-02-05T08:12:41.736991+0000","last_became_peered":"2024-02-05T08:12:41.736991+0000","last_unstale":"2024-02-05T08:17:43.293497+0000","last_undegraded":"2024-02-05T08:17:43.293497+0000","last_fullsized":"2024-02-05T08:17:43.293497+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:11.364552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:11.364552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.344582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":245,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292369+0000","last_change":"2024-02-05T08:17:10.054692+0000","last_active":"2024-02-05T08:17:43.292369+0000","last_peered":"2024-02-05T08:17:43.292369+0000","last_clean":"2024-02-05T08:17:43.292369+0000","last_became_active":"2024-02-05T08:12:41.717459+0000","last_became_peered":"2024-02-05T08:12:41.717459+0000","last_unstale":"2024-02-05T08:17:43.292369+0000","last_undegraded":"2024-02-05T08:17:43.292369+0000","last_fullsized":"2024-02-05T08:17:43.292369+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:17:10.054526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:10.054526+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:34.549622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":469,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.291967+0000","last_change":"2024-02-05T08:16:03.937874+0000","last_active":"2024-02-05T08:17:43.291967+0000","last_peered":"2024-02-05T08:17:43.291967+0000","last_clean":"2024-02-05T08:17:43.291967+0000","last_became_active":"2024-02-05T08:12:41.737803+0000","last_became_peered":"2024-02-05T08:12:41.737803+0000","last_unstale":"2024-02-05T08:17:43.291967+0000","last_undegraded":"2024-02-05T08:17:43.291967+0000","last_fullsized":"2024-02-05T08:17:43.291967+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:03.937759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:03.937759+0000","last_clean_scrub_stamp":"2024-02-05T08:16:03.937759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.344885+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":498,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.272586+0000","last_change":"2024-02-05T08:16:13.076225+0000","last_active":"2024-02-05T08:17:36.272586+0000","last_peered":"2024-02-05T08:17:36.272586+0000","last_clean":"2024-02-05T08:17:36.272586+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:17:36.272586+0000","last_undegraded":"2024-02-05T08:17:36.272586+0000","last_fullsized":"2024-02-05T08:17:36.272586+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:13.076114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:16:13.076114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.010277+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":493,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.267769+0000","last_change":"2024-02-05T08:16:07.831634+0000","last_active":"2024-02-05T08:17:36.267769+0000","last_peered":"2024-02-05T08:17:36.267769+0000","last_clean":"2024-02-05T08:17:36.267769+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:17:36.267769+0000","last_undegraded":"2024-02-05T08:17:36.267769+0000","last_fullsized":"2024-02-05T08:17:36.267769+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:16:07.831525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"236'24","reported_seq":442,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.269219+0000","last_change":"2024-02-05T08:17:11.945979+0000","last_active":"2024-02-05T08:17:36.269219+0000","last_peered":"2024-02-05T08:17:36.269219+0000","last_clean":"2024-02-05T08:17:36.269219+0000","last_became_active":"2024-02-05T08:12:41.718651+0000","last_became_peered":"2024-02-05T08:12:41.718651+0000","last_unstale":"2024-02-05T08:17:36.269219+0000","last_undegraded":"2024-02-05T08:17:36.269219+0000","last_fullsized":"2024-02-05T08:17:36.269219+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:17:11.945868+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:17:11.945868+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:31.340745+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":474,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.290392+0000","last_change":"2024-02-05T08:17:09.009620+0000","last_active":"2024-02-05T08:17:43.290392+0000","last_peered":"2024-02-05T08:17:43.290392+0000","last_clean":"2024-02-05T08:17:43.290392+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:17:43.290392+0000","last_undegraded":"2024-02-05T08:17:43.290392+0000","last_fullsized":"2024-02-05T08:17:43.290392+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:17:09.009398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:13.651051+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":208,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292120+0000","last_change":"2024-02-05T08:16:12.850966+0000","last_active":"2024-02-05T08:17:43.292120+0000","last_peered":"2024-02-05T08:17:43.292120+0000","last_clean":"2024-02-05T08:17:43.292120+0000","last_became_active":"2024-02-05T08:12:41.757901+0000","last_became_peered":"2024-02-05T08:12:41.757901+0000","last_unstale":"2024-02-05T08:17:43.292120+0000","last_undegraded":"2024-02-05T08:17:43.292120+0000","last_fullsized":"2024-02-05T08:17:43.292120+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:12.850323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:12.850323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.345098+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":492,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.274619+0000","last_change":"2024-02-05T08:15:10.525698+0000","last_active":"2024-02-05T08:17:36.274619+0000","last_peered":"2024-02-05T08:17:36.274619+0000","last_clean":"2024-02-05T08:17:36.274619+0000","last_became_active":"2024-02-05T08:12:41.734072+0000","last_became_peered":"2024-02-05T08:12:41.734072+0000","last_unstale":"2024-02-05T08:17:36.274619+0000","last_undegraded":"2024-02-05T08:17:36.274619+0000","last_fullsized":"2024-02-05T08:17:36.274619+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:10.525625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:10.525625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.010154+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"263'19","reported_seq":153,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.263379+0000","last_change":"2024-02-05T08:16:09.806635+0000","last_active":"2024-02-05T08:17:36.263379+0000","last_peered":"2024-02-05T08:17:36.263379+0000","last_clean":"2024-02-05T08:17:36.263379+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:17:36.263379+0000","last_undegraded":"2024-02-05T08:17:36.263379+0000","last_fullsized":"2024-02-05T08:17:36.263379+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:16:09.806380+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:09.806380+0000","last_clean_scrub_stamp":"2024-02-05T08:16:09.806380+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":199,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.291452+0000","last_change":"2024-02-05T08:15:54.851205+0000","last_active":"2024-02-05T08:17:43.291452+0000","last_peered":"2024-02-05T08:17:43.291452+0000","last_clean":"2024-02-05T08:17:43.291452+0000","last_became_active":"2024-02-05T08:12:41.737246+0000","last_became_peered":"2024-02-05T08:12:41.737246+0000","last_unstale":"2024-02-05T08:17:43.291452+0000","last_undegraded":"2024-02-05T08:17:43.291452+0000","last_fullsized":"2024-02-05T08:17:43.291452+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:54.850842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:15:54.850842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.344948+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":486,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.294976+0000","last_change":"2024-02-05T08:16:07.000448+0000","last_active":"2024-02-05T08:17:43.294976+0000","last_peered":"2024-02-05T08:17:43.294976+0000","last_clean":"2024-02-05T08:17:43.294976+0000","last_became_active":"2024-02-05T08:12:41.717285+0000","last_became_peered":"2024-02-05T08:12:41.717285+0000","last_unstale":"2024-02-05T08:17:43.294976+0000","last_undegraded":"2024-02-05T08:17:43.294976+0000","last_fullsized":"2024-02-05T08:17:43.294976+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:07.000154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:07.000154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":505,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.260783+0000","last_change":"2024-02-05T08:16:06.796972+0000","last_active":"2024-02-05T08:17:36.260783+0000","last_peered":"2024-02-05T08:17:36.260783+0000","last_clean":"2024-02-05T08:17:36.260783+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:17:36.260783+0000","last_undegraded":"2024-02-05T08:17:36.260783+0000","last_fullsized":"2024-02-05T08:17:36.260783+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:06.796588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:16:06.796588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"249'24","reported_seq":561,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.257279+0000","last_change":"2024-02-05T08:17:11.725677+0000","last_active":"2024-02-05T08:17:36.257279+0000","last_peered":"2024-02-05T08:17:36.257279+0000","last_clean":"2024-02-05T08:17:36.257279+0000","last_became_active":"2024-02-05T08:12:29.675657+0000","last_became_peered":"2024-02-05T08:12:29.675657+0000","last_unstale":"2024-02-05T08:17:36.257279+0000","last_undegraded":"2024-02-05T08:17:36.257279+0000","last_fullsized":"2024-02-05T08:17:36.257279+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:17:11.725554+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:17:11.725554+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:22.106197+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":472,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292234+0000","last_change":"2024-02-05T08:15:25.472795+0000","last_active":"2024-02-05T08:17:43.292234+0000","last_peered":"2024-02-05T08:17:43.292234+0000","last_clean":"2024-02-05T08:17:43.292234+0000","last_became_active":"2024-02-05T08:12:41.736845+0000","last_became_peered":"2024-02-05T08:12:41.736845+0000","last_unstale":"2024-02-05T08:17:43.292234+0000","last_undegraded":"2024-02-05T08:17:43.292234+0000","last_fullsized":"2024-02-05T08:17:43.292234+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.472655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.472655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.344822+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":650,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.741821+0000","last_change":"2024-02-05T08:17:00.560018+0000","last_active":"2024-02-05T08:17:36.741821+0000","last_peered":"2024-02-05T08:17:36.741821+0000","last_clean":"2024-02-05T08:17:36.741821+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:17:36.741821+0000","last_undegraded":"2024-02-05T08:17:36.741821+0000","last_fullsized":"2024-02-05T08:17:36.741821+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:17:00.559889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:00.559889+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:21.370792+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":494,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.268110+0000","last_change":"2024-02-05T08:16:58.934204+0000","last_active":"2024-02-05T08:17:36.268110+0000","last_peered":"2024-02-05T08:17:36.268110+0000","last_clean":"2024-02-05T08:17:36.268110+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:17:36.268110+0000","last_undegraded":"2024-02-05T08:17:36.268110+0000","last_fullsized":"2024-02-05T08:17:36.268110+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:58.934100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:58.934100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:16.320790+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":392,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.291574+0000","last_change":"2024-02-05T08:17:17.993610+0000","last_active":"2024-02-05T08:17:43.291574+0000","last_peered":"2024-02-05T08:17:43.291574+0000","last_clean":"2024-02-05T08:17:43.291574+0000","last_became_active":"2024-02-05T08:12:41.726861+0000","last_became_peered":"2024-02-05T08:12:41.726861+0000","last_unstale":"2024-02-05T08:17:43.291574+0000","last_undegraded":"2024-02-05T08:17:43.291574+0000","last_fullsized":"2024-02-05T08:17:43.291574+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-02-05T08:17:17.993472+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:17:17.993472+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:35.927283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":476,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.256792+0000","last_change":"2024-02-05T08:17:01.898247+0000","last_active":"2024-02-05T08:17:36.256792+0000","last_peered":"2024-02-05T08:17:36.256792+0000","last_clean":"2024-02-05T08:17:36.256792+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:17:36.256792+0000","last_undegraded":"2024-02-05T08:17:36.256792+0000","last_fullsized":"2024-02-05T08:17:36.256792+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:01.897910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:01.897910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:26.422119+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"194'9","reported_seq":296,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.741094+0000","last_change":"2024-02-05T08:15:27.464031+0000","last_active":"2024-02-05T08:17:36.741094+0000","last_peered":"2024-02-05T08:17:36.741094+0000","last_clean":"2024-02-05T08:17:36.741094+0000","last_became_active":"2024-02-05T08:12:41.728687+0000","last_became_peered":"2024-02-05T08:12:41.728687+0000","last_unstale":"2024-02-05T08:17:36.741094+0000","last_undegraded":"2024-02-05T08:17:36.741094+0000","last_fullsized":"2024-02-05T08:17:36.741094+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'9","last_scrub_stamp":"2024-02-05T08:15:27.463918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:27.463918+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:44.018919+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":484,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292065+0000","last_change":"2024-02-05T08:15:23.494414+0000","last_active":"2024-02-05T08:17:43.292065+0000","last_peered":"2024-02-05T08:17:43.292065+0000","last_clean":"2024-02-05T08:17:43.292065+0000","last_became_active":"2024-02-05T08:12:41.723383+0000","last_became_peered":"2024-02-05T08:12:41.723383+0000","last_unstale":"2024-02-05T08:17:43.292065+0000","last_undegraded":"2024-02-05T08:17:43.292065+0000","last_fullsized":"2024-02-05T08:17:43.292065+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:23.494299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:15:23.494299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.344707+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":530,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.271365+0000","last_change":"2024-02-05T08:16:53.643233+0000","last_active":"2024-02-05T08:17:36.271365+0000","last_peered":"2024-02-05T08:17:36.271365+0000","last_clean":"2024-02-05T08:17:36.271365+0000","last_became_active":"2024-02-05T08:12:41.723526+0000","last_became_peered":"2024-02-05T08:12:41.723526+0000","last_unstale":"2024-02-05T08:17:36.271365+0000","last_undegraded":"2024-02-05T08:17:36.271365+0000","last_fullsized":"2024-02-05T08:17:36.271365+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:16:53.643108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:53.643108+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:59.084301+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":489,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.257931+0000","last_change":"2024-02-05T08:17:05.439954+0000","last_active":"2024-02-05T08:17:36.257931+0000","last_peered":"2024-02-05T08:17:36.257931+0000","last_clean":"2024-02-05T08:17:36.257931+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:17:36.257931+0000","last_undegraded":"2024-02-05T08:17:36.257931+0000","last_fullsized":"2024-02-05T08:17:36.257931+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:05.439752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:05.439752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:08.725804+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":209,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.291201+0000","last_change":"2024-02-05T08:16:14.014002+0000","last_active":"2024-02-05T08:17:43.291201+0000","last_peered":"2024-02-05T08:17:43.291201+0000","last_clean":"2024-02-05T08:17:43.291201+0000","last_became_active":"2024-02-05T08:12:41.721852+0000","last_became_peered":"2024-02-05T08:12:41.721852+0000","last_unstale":"2024-02-05T08:17:43.291201+0000","last_undegraded":"2024-02-05T08:17:43.291201+0000","last_fullsized":"2024-02-05T08:17:43.291201+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:14.013888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:14.013888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":498,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.269032+0000","last_change":"2024-02-05T08:15:25.470332+0000","last_active":"2024-02-05T08:17:36.269032+0000","last_peered":"2024-02-05T08:17:36.269032+0000","last_clean":"2024-02-05T08:17:36.269032+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:17:36.269032+0000","last_undegraded":"2024-02-05T08:17:36.269032+0000","last_fullsized":"2024-02-05T08:17:36.269032+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.470117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.010233+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"278'19","reported_seq":527,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292603+0000","last_change":"2024-02-05T08:17:11.006500+0000","last_active":"2024-02-05T08:17:43.292603+0000","last_peered":"2024-02-05T08:17:43.292603+0000","last_clean":"2024-02-05T08:17:43.292603+0000","last_became_active":"2024-02-05T08:12:41.736414+0000","last_became_peered":"2024-02-05T08:12:41.736414+0000","last_unstale":"2024-02-05T08:17:43.292603+0000","last_undegraded":"2024-02-05T08:17:43.292603+0000","last_fullsized":"2024-02-05T08:17:43.292603+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:17:11.006333+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:25.749896+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":494,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.274118+0000","last_change":"2024-02-05T08:16:54.078592+0000","last_active":"2024-02-05T08:17:36.274118+0000","last_peered":"2024-02-05T08:17:36.274118+0000","last_clean":"2024-02-05T08:17:36.274118+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:17:36.274118+0000","last_undegraded":"2024-02-05T08:17:36.274118+0000","last_fullsized":"2024-02-05T08:17:36.274118+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:54.078361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:54.078361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:07.041615+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":483,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.268016+0000","last_change":"2024-02-05T08:15:55.898563+0000","last_active":"2024-02-05T08:17:36.268016+0000","last_peered":"2024-02-05T08:17:36.268016+0000","last_clean":"2024-02-05T08:17:36.268016+0000","last_became_active":"2024-02-05T08:12:41.721486+0000","last_became_peered":"2024-02-05T08:12:41.721486+0000","last_unstale":"2024-02-05T08:17:36.268016+0000","last_undegraded":"2024-02-05T08:17:36.268016+0000","last_fullsized":"2024-02-05T08:17:36.268016+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:55.898292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:15:55.898292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":215,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.275209+0000","last_change":"2024-02-05T08:15:56.286068+0000","last_active":"2024-02-05T08:17:36.275209+0000","last_peered":"2024-02-05T08:17:36.275209+0000","last_clean":"2024-02-05T08:17:36.275209+0000","last_became_active":"2024-02-05T08:12:29.684444+0000","last_became_peered":"2024-02-05T08:12:29.684444+0000","last_unstale":"2024-02-05T08:17:36.275209+0000","last_undegraded":"2024-02-05T08:17:36.275209+0000","last_fullsized":"2024-02-05T08:17:36.275209+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:56.285899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:56.285899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.125554+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'15","reported_seq":348,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.268652+0000","last_change":"2024-02-05T08:15:23.404290+0000","last_active":"2024-02-05T08:17:36.268652+0000","last_peered":"2024-02-05T08:17:36.268652+0000","last_clean":"2024-02-05T08:17:36.268652+0000","last_became_active":"2024-02-05T08:12:41.732369+0000","last_became_peered":"2024-02-05T08:12:41.732369+0000","last_unstale":"2024-02-05T08:17:36.268652+0000","last_undegraded":"2024-02-05T08:17:36.268652+0000","last_fullsized":"2024-02-05T08:17:36.268652+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:15:23.404196+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:15:23.404196+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.010139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"283'15887","reported_seq":16129,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.715856+0000","last_change":"2024-02-05T08:17:31.333627+0000","last_active":"2024-02-05T08:17:43.715856+0000","last_peered":"2024-02-05T08:17:43.715856+0000","last_clean":"2024-02-05T08:17:43.715856+0000","last_became_active":"2024-02-05T08:14:57.766609+0000","last_became_peered":"2024-02-05T08:14:57.766609+0000","last_unstale":"2024-02-05T08:17:43.715856+0000","last_undegraded":"2024-02-05T08:17:43.715856+0000","last_fullsized":"2024-02-05T08:17:43.715856+0000","mapping_epoch":239,"log_start":"278'10702","ondisk_log_start":"278'10702","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":3448,"log_size":5185,"log_dups_size":0,"ondisk_log_size":5185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280231936,"num_objects":4276,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12440,"num_write_kb":534912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":493,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.263816+0000","last_change":"2024-02-05T08:16:50.673897+0000","last_active":"2024-02-05T08:17:36.263816+0000","last_peered":"2024-02-05T08:17:36.263816+0000","last_clean":"2024-02-05T08:17:36.263816+0000","last_became_active":"2024-02-05T08:12:29.679849+0000","last_became_peered":"2024-02-05T08:12:29.679849+0000","last_unstale":"2024-02-05T08:17:36.263816+0000","last_undegraded":"2024-02-05T08:17:36.263816+0000","last_fullsized":"2024-02-05T08:17:36.263816+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:50.673646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:50.673646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:54.535977+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":506,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.268569+0000","last_change":"2024-02-05T08:17:00.989869+0000","last_active":"2024-02-05T08:17:36.268569+0000","last_peered":"2024-02-05T08:17:36.268569+0000","last_clean":"2024-02-05T08:17:36.268569+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:17:36.268569+0000","last_undegraded":"2024-02-05T08:17:36.268569+0000","last_fullsized":"2024-02-05T08:17:36.268569+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:17:00.989762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:00.989762+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:14.219383+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"262'34","reported_seq":575,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.261805+0000","last_change":"2024-02-05T08:16:06.849357+0000","last_active":"2024-02-05T08:17:36.261805+0000","last_peered":"2024-02-05T08:17:36.261805+0000","last_clean":"2024-02-05T08:17:36.261805+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:17:36.261805+0000","last_undegraded":"2024-02-05T08:17:36.261805+0000","last_fullsized":"2024-02-05T08:17:36.261805+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'34","last_scrub_stamp":"2024-02-05T08:16:06.848964+0000","last_deep_scrub":"262'34","last_deep_scrub_stamp":"2024-02-05T08:16:06.848964+0000","last_clean_scrub_stamp":"2024-02-05T08:16:06.848964+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":672,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.743881+0000","last_change":"2024-02-05T08:17:09.961319+0000","last_active":"2024-02-05T08:17:36.743881+0000","last_peered":"2024-02-05T08:17:36.743881+0000","last_clean":"2024-02-05T08:17:36.743881+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:17:36.743881+0000","last_undegraded":"2024-02-05T08:17:36.743881+0000","last_fullsized":"2024-02-05T08:17:36.743881+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:17:09.961036+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:17:09.961036+0000","last_clean_scrub_stamp":"2024-02-05T08:17:09.961036+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:28.075138+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"283'12556","reported_seq":12735,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.352324+0000","last_change":"2024-02-05T08:17:00.838377+0000","last_active":"2024-02-05T08:17:43.352324+0000","last_peered":"2024-02-05T08:17:43.352324+0000","last_clean":"2024-02-05T08:17:43.352324+0000","last_became_active":"2024-02-05T08:14:57.758070+0000","last_became_peered":"2024-02-05T08:14:57.758070+0000","last_unstale":"2024-02-05T08:17:43.352324+0000","last_undegraded":"2024-02-05T08:17:43.352324+0000","last_fullsized":"2024-02-05T08:17:43.352324+0000","mapping_epoch":239,"log_start":"270'8400","ondisk_log_start":"270'8400","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9636","last_scrub_stamp":"2024-02-05T08:17:00.838236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:00.838236+0000","objects_scrubbed":7096,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:16.720317+0000","scrub_duration":1916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269352960,"num_objects":4110,"num_object_clones":0,"num_object_copies":12330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12556,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":206,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.268185+0000","last_change":"2024-02-05T08:17:14.099450+0000","last_active":"2024-02-05T08:17:36.268185+0000","last_peered":"2024-02-05T08:17:36.268185+0000","last_clean":"2024-02-05T08:17:36.268185+0000","last_became_active":"2024-02-05T08:12:41.722853+0000","last_became_peered":"2024-02-05T08:12:41.722853+0000","last_unstale":"2024-02-05T08:17:36.268185+0000","last_undegraded":"2024-02-05T08:17:36.268185+0000","last_fullsized":"2024-02-05T08:17:36.268185+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:17:14.099323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:30.182521+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":494,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.269587+0000","last_change":"2024-02-05T08:17:12.948132+0000","last_active":"2024-02-05T08:17:36.269587+0000","last_peered":"2024-02-05T08:17:36.269587+0000","last_clean":"2024-02-05T08:17:36.269587+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:17:36.269587+0000","last_undegraded":"2024-02-05T08:17:36.269587+0000","last_fullsized":"2024-02-05T08:17:36.269587+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:12.948033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:17:12.948033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:33.426665+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":531,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.258619+0000","last_change":"2024-02-05T08:16:14.860227+0000","last_active":"2024-02-05T08:17:36.258619+0000","last_peered":"2024-02-05T08:17:36.258619+0000","last_clean":"2024-02-05T08:17:36.258619+0000","last_became_active":"2024-02-05T08:12:41.728925+0000","last_became_peered":"2024-02-05T08:12:41.728925+0000","last_unstale":"2024-02-05T08:17:36.258619+0000","last_undegraded":"2024-02-05T08:17:36.258619+0000","last_fullsized":"2024-02-05T08:17:36.258619+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-02-05T08:16:14.857110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:14.857110+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:40.868827+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":493,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.256164+0000","last_change":"2024-02-05T08:16:54.393426+0000","last_active":"2024-02-05T08:17:36.256164+0000","last_peered":"2024-02-05T08:17:36.256164+0000","last_clean":"2024-02-05T08:17:36.256164+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:17:36.256164+0000","last_undegraded":"2024-02-05T08:17:36.256164+0000","last_fullsized":"2024-02-05T08:17:36.256164+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:54.393312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:16:54.393312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:18.129621+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"282'12088","reported_seq":12249,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.065932+0000","last_change":"2024-02-05T08:17:13.631762+0000","last_active":"2024-02-05T08:17:43.065932+0000","last_peered":"2024-02-05T08:17:43.065932+0000","last_clean":"2024-02-05T08:17:43.065932+0000","last_became_active":"2024-02-05T08:14:57.763165+0000","last_became_peered":"2024-02-05T08:14:57.763165+0000","last_unstale":"2024-02-05T08:17:43.065932+0000","last_undegraded":"2024-02-05T08:17:43.065932+0000","last_fullsized":"2024-02-05T08:17:43.065932+0000","mapping_epoch":239,"log_start":"267'7400","ondisk_log_start":"267'7400","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'10307","last_scrub_stamp":"2024-02-05T08:17:13.631603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:13.631603+0000","objects_scrubbed":5907,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:20.761498+0000","scrub_duration":1163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268697600,"num_objects":4100,"num_object_clones":0,"num_object_copies":12300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12088,"num_write_kb":518016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":480,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.268802+0000","last_change":"2024-02-05T08:15:13.442673+0000","last_active":"2024-02-05T08:17:36.268802+0000","last_peered":"2024-02-05T08:17:36.268802+0000","last_clean":"2024-02-05T08:17:36.268802+0000","last_became_active":"2024-02-05T08:12:41.724830+0000","last_became_peered":"2024-02-05T08:12:41.724830+0000","last_unstale":"2024-02-05T08:17:36.268802+0000","last_undegraded":"2024-02-05T08:17:36.268802+0000","last_fullsized":"2024-02-05T08:17:36.268802+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:13.442573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:13.442573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.010168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":485,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.256733+0000","last_change":"2024-02-05T08:15:57.811341+0000","last_active":"2024-02-05T08:17:36.256733+0000","last_peered":"2024-02-05T08:17:36.256733+0000","last_clean":"2024-02-05T08:17:36.256733+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:17:36.256733+0000","last_undegraded":"2024-02-05T08:17:36.256733+0000","last_fullsized":"2024-02-05T08:17:36.256733+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:57.811220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:15:57.811220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:40.577964+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":507,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.274820+0000","last_change":"2024-02-05T08:16:57.957167+0000","last_active":"2024-02-05T08:17:36.274820+0000","last_peered":"2024-02-05T08:17:36.274820+0000","last_clean":"2024-02-05T08:17:36.274820+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:17:36.274820+0000","last_undegraded":"2024-02-05T08:17:36.274820+0000","last_fullsized":"2024-02-05T08:17:36.274820+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:57.957064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:57.957064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:26.444176+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"236'35","reported_seq":259,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.275642+0000","last_change":"2024-02-05T08:16:10.321575+0000","last_active":"2024-02-05T08:17:36.275642+0000","last_peered":"2024-02-05T08:17:36.275642+0000","last_clean":"2024-02-05T08:17:36.275642+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:17:36.275642+0000","last_undegraded":"2024-02-05T08:17:36.275642+0000","last_fullsized":"2024-02-05T08:17:36.275642+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'35","last_scrub_stamp":"2024-02-05T08:16:10.321448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:10.321448+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.125586+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"282'12355","reported_seq":12528,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.083044+0000","last_change":"2024-02-05T08:17:06.041806+0000","last_active":"2024-02-05T08:17:43.083044+0000","last_peered":"2024-02-05T08:17:43.083044+0000","last_clean":"2024-02-05T08:17:43.083044+0000","last_became_active":"2024-02-05T08:14:57.761156+0000","last_became_peered":"2024-02-05T08:14:57.761156+0000","last_unstale":"2024-02-05T08:17:43.083044+0000","last_undegraded":"2024-02-05T08:17:43.083044+0000","last_fullsized":"2024-02-05T08:17:43.083044+0000","mapping_epoch":239,"log_start":"267'7700","ondisk_log_start":"267'7700","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'9937","last_scrub_stamp":"2024-02-05T08:17:06.041540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:06.041540+0000","objects_scrubbed":6705,"log_size":4655,"log_dups_size":0,"ondisk_log_size":4655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:16.501634+0000","scrub_duration":2521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274530304,"num_objects":4189,"num_object_clones":0,"num_object_copies":12567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12355,"num_write_kb":529408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":474,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.291637+0000","last_change":"2024-02-05T08:16:49.867423+0000","last_active":"2024-02-05T08:17:43.291637+0000","last_peered":"2024-02-05T08:17:43.291637+0000","last_clean":"2024-02-05T08:17:43.291637+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:17:43.291637+0000","last_undegraded":"2024-02-05T08:17:43.291637+0000","last_fullsized":"2024-02-05T08:17:43.291637+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:49.867301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:49.867301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:58.916262+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":499,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.259732+0000","last_change":"2024-02-05T08:17:10.458367+0000","last_active":"2024-02-05T08:17:36.259732+0000","last_peered":"2024-02-05T08:17:36.259732+0000","last_clean":"2024-02-05T08:17:36.259732+0000","last_became_active":"2024-02-05T08:12:41.743888+0000","last_became_peered":"2024-02-05T08:12:41.743888+0000","last_unstale":"2024-02-05T08:17:36.259732+0000","last_undegraded":"2024-02-05T08:17:36.259732+0000","last_fullsized":"2024-02-05T08:17:36.259732+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:10.458223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:10.458223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:12.912193+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":503,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.266222+0000","last_change":"2024-02-05T08:15:58.803274+0000","last_active":"2024-02-05T08:17:36.266222+0000","last_peered":"2024-02-05T08:17:36.266222+0000","last_clean":"2024-02-05T08:17:36.266222+0000","last_became_active":"2024-02-05T08:12:41.718843+0000","last_became_peered":"2024-02-05T08:12:41.718843+0000","last_unstale":"2024-02-05T08:17:36.266222+0000","last_undegraded":"2024-02-05T08:17:36.266222+0000","last_fullsized":"2024-02-05T08:17:36.266222+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:58.803170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:58.803170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"262'19","reported_seq":511,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292594+0000","last_change":"2024-02-05T08:16:04.837042+0000","last_active":"2024-02-05T08:17:43.292594+0000","last_peered":"2024-02-05T08:17:43.292594+0000","last_clean":"2024-02-05T08:17:43.292594+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:17:43.292594+0000","last_undegraded":"2024-02-05T08:17:43.292594+0000","last_fullsized":"2024-02-05T08:17:43.292594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'19","last_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:16:04.836768+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.345040+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"283'12579","reported_seq":12678,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.729021+0000","last_change":"2024-02-05T08:14:57.766128+0000","last_active":"2024-02-05T08:17:43.729021+0000","last_peered":"2024-02-05T08:17:43.729021+0000","last_clean":"2024-02-05T08:17:43.729021+0000","last_became_active":"2024-02-05T08:14:57.765694+0000","last_became_peered":"2024-02-05T08:14:57.765694+0000","last_unstale":"2024-02-05T08:17:43.729021+0000","last_undegraded":"2024-02-05T08:17:43.729021+0000","last_fullsized":"2024-02-05T08:17:43.729021+0000","mapping_epoch":239,"log_start":"267'7400","ondisk_log_start":"267'7400","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":5179,"log_dups_size":0,"ondisk_log_size":5179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273088512,"num_objects":4167,"num_object_clones":0,"num_object_copies":12501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12579,"num_write_kb":535872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":198,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.257430+0000","last_change":"2024-02-05T08:15:29.993055+0000","last_active":"2024-02-05T08:17:36.257430+0000","last_peered":"2024-02-05T08:17:36.257430+0000","last_clean":"2024-02-05T08:17:36.257430+0000","last_became_active":"2024-02-05T08:12:41.726776+0000","last_became_peered":"2024-02-05T08:12:41.726776+0000","last_unstale":"2024-02-05T08:17:36.257430+0000","last_undegraded":"2024-02-05T08:17:36.257430+0000","last_fullsized":"2024-02-05T08:17:36.257430+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:29.992957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:29.992957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:40.868720+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":526,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.263595+0000","last_change":"2024-02-05T08:16:48.614737+0000","last_active":"2024-02-05T08:17:36.263595+0000","last_peered":"2024-02-05T08:17:36.263595+0000","last_clean":"2024-02-05T08:17:36.263595+0000","last_became_active":"2024-02-05T08:12:29.679395+0000","last_became_peered":"2024-02-05T08:12:29.679395+0000","last_unstale":"2024-02-05T08:17:36.263595+0000","last_undegraded":"2024-02-05T08:17:36.263595+0000","last_fullsized":"2024-02-05T08:17:36.263595+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:16:48.614659+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:16:48.614659+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:17.786365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":498,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.257108+0000","last_change":"2024-02-05T08:16:11.809948+0000","last_active":"2024-02-05T08:17:36.257108+0000","last_peered":"2024-02-05T08:17:36.257108+0000","last_clean":"2024-02-05T08:17:36.257108+0000","last_became_active":"2024-02-05T08:12:29.670859+0000","last_became_peered":"2024-02-05T08:12:29.670859+0000","last_unstale":"2024-02-05T08:17:36.257108+0000","last_undegraded":"2024-02-05T08:17:36.257108+0000","last_fullsized":"2024-02-05T08:17:36.257108+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:11.809845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:16:11.809845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:41.019828+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"236'51","reported_seq":746,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.290176+0000","last_change":"2024-02-05T08:15:59.844538+0000","last_active":"2024-02-05T08:17:43.290176+0000","last_peered":"2024-02-05T08:17:43.290176+0000","last_clean":"2024-02-05T08:17:43.290176+0000","last_became_active":"2024-02-05T08:12:41.719820+0000","last_became_peered":"2024-02-05T08:12:41.719820+0000","last_unstale":"2024-02-05T08:17:43.290176+0000","last_undegraded":"2024-02-05T08:17:43.290176+0000","last_fullsized":"2024-02-05T08:17:43.290176+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'51","last_scrub_stamp":"2024-02-05T08:15:59.844399+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:15:59.844399+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":471,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.296881+0000","last_change":"2024-02-05T08:16:09.774039+0000","last_active":"2024-02-05T08:17:43.296881+0000","last_peered":"2024-02-05T08:17:43.296881+0000","last_clean":"2024-02-05T08:17:43.296881+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:17:43.296881+0000","last_undegraded":"2024-02-05T08:17:43.296881+0000","last_fullsized":"2024-02-05T08:17:43.296881+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:09.773910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:09.773910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.344978+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"282'12298","reported_seq":12477,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.074899+0000","last_change":"2024-02-05T08:16:58.188469+0000","last_active":"2024-02-05T08:17:43.074899+0000","last_peered":"2024-02-05T08:17:43.074899+0000","last_clean":"2024-02-05T08:17:43.074899+0000","last_became_active":"2024-02-05T08:14:57.763184+0000","last_became_peered":"2024-02-05T08:14:57.763184+0000","last_unstale":"2024-02-05T08:17:43.074899+0000","last_undegraded":"2024-02-05T08:17:43.074899+0000","last_fullsized":"2024-02-05T08:17:43.074899+0000","mapping_epoch":239,"log_start":"267'7600","ondisk_log_start":"267'7600","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'9334","last_scrub_stamp":"2024-02-05T08:16:58.188319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:16:58.188319+0000","objects_scrubbed":7257,"log_size":4698,"log_dups_size":0,"ondisk_log_size":4698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:24.047016+0000","scrub_duration":2565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275251200,"num_objects":4200,"num_object_clones":0,"num_object_copies":12600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12298,"num_write_kb":527936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":493,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.274962+0000","last_change":"2024-02-05T08:15:20.467544+0000","last_active":"2024-02-05T08:17:36.274962+0000","last_peered":"2024-02-05T08:17:36.274962+0000","last_clean":"2024-02-05T08:17:36.274962+0000","last_became_active":"2024-02-05T08:12:41.732338+0000","last_became_peered":"2024-02-05T08:12:41.732338+0000","last_unstale":"2024-02-05T08:17:36.274962+0000","last_undegraded":"2024-02-05T08:17:36.274962+0000","last_fullsized":"2024-02-05T08:17:36.274962+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:20.467407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:20.467407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.010247+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":516,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292276+0000","last_change":"2024-02-05T08:17:06.932722+0000","last_active":"2024-02-05T08:17:43.292276+0000","last_peered":"2024-02-05T08:17:43.292276+0000","last_clean":"2024-02-05T08:17:43.292276+0000","last_became_active":"2024-02-05T08:12:29.673554+0000","last_became_peered":"2024-02-05T08:12:29.673554+0000","last_unstale":"2024-02-05T08:17:43.292276+0000","last_undegraded":"2024-02-05T08:17:43.292276+0000","last_fullsized":"2024-02-05T08:17:43.292276+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:17:06.932583+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:17:06.932583+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:31.571859+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":517,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.256644+0000","last_change":"2024-02-05T08:17:12.613722+0000","last_active":"2024-02-05T08:17:36.256644+0000","last_peered":"2024-02-05T08:17:36.256644+0000","last_clean":"2024-02-05T08:17:36.256644+0000","last_became_active":"2024-02-05T08:12:41.746289+0000","last_became_peered":"2024-02-05T08:12:41.746289+0000","last_unstale":"2024-02-05T08:17:36.256644+0000","last_undegraded":"2024-02-05T08:17:36.256644+0000","last_fullsized":"2024-02-05T08:17:36.256644+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:12.613617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:17:12.613617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:22.365708+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'17","reported_seq":499,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.257374+0000","last_change":"2024-02-05T08:15:58.770113+0000","last_active":"2024-02-05T08:17:36.257374+0000","last_peered":"2024-02-05T08:17:36.257374+0000","last_clean":"2024-02-05T08:17:36.257374+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:17:36.257374+0000","last_undegraded":"2024-02-05T08:17:36.257374+0000","last_fullsized":"2024-02-05T08:17:36.257374+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'17","last_scrub_stamp":"2024-02-05T08:15:58.770027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:58.770027+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:40.577986+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":494,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.257150+0000","last_change":"2024-02-05T08:17:03.422904+0000","last_active":"2024-02-05T08:17:36.257150+0000","last_peered":"2024-02-05T08:17:36.257150+0000","last_clean":"2024-02-05T08:17:36.257150+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:17:36.257150+0000","last_undegraded":"2024-02-05T08:17:36.257150+0000","last_fullsized":"2024-02-05T08:17:36.257150+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:03.422828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:03.422828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:05.002750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"283'12178","reported_seq":12277,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.736327+0000","last_change":"2024-02-05T08:14:57.762460+0000","last_active":"2024-02-05T08:17:43.736327+0000","last_peered":"2024-02-05T08:17:43.736327+0000","last_clean":"2024-02-05T08:17:43.736327+0000","last_became_active":"2024-02-05T08:14:57.762164+0000","last_became_peered":"2024-02-05T08:14:57.762164+0000","last_unstale":"2024-02-05T08:17:43.736327+0000","last_undegraded":"2024-02-05T08:17:43.736327+0000","last_fullsized":"2024-02-05T08:17:43.736327+0000","mapping_epoch":239,"log_start":"266'7000","ondisk_log_start":"266'7000","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":5178,"log_dups_size":0,"ondisk_log_size":5178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270270464,"num_objects":4124,"num_object_clones":0,"num_object_copies":12372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12178,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":479,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.266731+0000","last_change":"2024-02-05T08:15:55.025258+0000","last_active":"2024-02-05T08:17:36.266731+0000","last_peered":"2024-02-05T08:17:36.266731+0000","last_clean":"2024-02-05T08:17:36.266731+0000","last_became_active":"2024-02-05T08:12:41.717170+0000","last_became_peered":"2024-02-05T08:12:41.717170+0000","last_unstale":"2024-02-05T08:17:36.266731+0000","last_undegraded":"2024-02-05T08:17:36.266731+0000","last_fullsized":"2024-02-05T08:17:36.266731+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:55.025140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:55.025140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":775,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.269031+0000","last_change":"2024-02-05T08:16:15.047572+0000","last_active":"2024-02-05T08:17:36.269031+0000","last_peered":"2024-02-05T08:17:36.269031+0000","last_clean":"2024-02-05T08:17:36.269031+0000","last_became_active":"2024-02-05T08:12:32.677426+0000","last_became_peered":"2024-02-05T08:12:32.677426+0000","last_unstale":"2024-02-05T08:17:36.269031+0000","last_undegraded":"2024-02-05T08:17:36.269031+0000","last_fullsized":"2024-02-05T08:17:36.269031+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:16:15.047436+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:16:15.047436+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.010308+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":488,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.263518+0000","last_change":"2024-02-05T08:17:13.476425+0000","last_active":"2024-02-05T08:17:36.263518+0000","last_peered":"2024-02-05T08:17:36.263518+0000","last_clean":"2024-02-05T08:17:36.263518+0000","last_became_active":"2024-02-05T08:12:41.722331+0000","last_became_peered":"2024-02-05T08:12:41.722331+0000","last_unstale":"2024-02-05T08:17:36.263518+0000","last_undegraded":"2024-02-05T08:17:36.263518+0000","last_fullsized":"2024-02-05T08:17:36.263518+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:13.476308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:13.476308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:21.477845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"236'30","reported_seq":314,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.741851+0000","last_change":"2024-02-05T08:16:54.287476+0000","last_active":"2024-02-05T08:17:36.741851+0000","last_peered":"2024-02-05T08:17:36.741851+0000","last_clean":"2024-02-05T08:17:36.741851+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:17:36.741851+0000","last_undegraded":"2024-02-05T08:17:36.741851+0000","last_fullsized":"2024-02-05T08:17:36.741851+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:16:54.287331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:54.287331+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:04.985090+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":479,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.272139+0000","last_change":"2024-02-05T08:16:06.033678+0000","last_active":"2024-02-05T08:17:36.272139+0000","last_peered":"2024-02-05T08:17:36.272139+0000","last_clean":"2024-02-05T08:17:36.272139+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:17:36.272139+0000","last_undegraded":"2024-02-05T08:17:36.272139+0000","last_fullsized":"2024-02-05T08:17:36.272139+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:06.033461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:06.033461+0000","last_clean_scrub_stamp":"2024-02-05T08:16:06.033461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.010292+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"283'12496","reported_seq":12662,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.740908+0000","last_change":"2024-02-05T08:17:15.291706+0000","last_active":"2024-02-05T08:17:43.740908+0000","last_peered":"2024-02-05T08:17:43.740908+0000","last_clean":"2024-02-05T08:17:43.740908+0000","last_became_active":"2024-02-05T08:14:57.761382+0000","last_became_peered":"2024-02-05T08:14:57.761382+0000","last_unstale":"2024-02-05T08:17:43.740908+0000","last_undegraded":"2024-02-05T08:17:43.740908+0000","last_fullsized":"2024-02-05T08:17:43.740908+0000","mapping_epoch":239,"log_start":"267'7300","ondisk_log_start":"267'7300","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'10631","last_scrub_stamp":"2024-02-05T08:17:15.291551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:15.291551+0000","objects_scrubbed":5855,"log_size":5196,"log_dups_size":0,"ondisk_log_size":5196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:25.553730+0000","scrub_duration":1251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259522560,"num_objects":3960,"num_object_clones":0,"num_object_copies":11880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12496,"num_write_kb":526592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":483,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.289843+0000","last_change":"2024-02-05T08:16:03.932381+0000","last_active":"2024-02-05T08:17:43.289843+0000","last_peered":"2024-02-05T08:17:43.289843+0000","last_clean":"2024-02-05T08:17:43.289843+0000","last_became_active":"2024-02-05T08:12:41.716394+0000","last_became_peered":"2024-02-05T08:12:41.716394+0000","last_unstale":"2024-02-05T08:17:43.289843+0000","last_undegraded":"2024-02-05T08:17:43.289843+0000","last_fullsized":"2024-02-05T08:17:43.289843+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:03.932264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:03.932264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":485,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.291374+0000","last_change":"2024-02-05T08:15:15.359343+0000","last_active":"2024-02-05T08:17:43.291374+0000","last_peered":"2024-02-05T08:17:43.291374+0000","last_clean":"2024-02-05T08:17:43.291374+0000","last_became_active":"2024-02-05T08:12:41.741846+0000","last_became_peered":"2024-02-05T08:12:41.741846+0000","last_unstale":"2024-02-05T08:17:43.291374+0000","last_undegraded":"2024-02-05T08:17:43.291374+0000","last_fullsized":"2024-02-05T08:17:43.291374+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:15.359190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:15.359190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.344792+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":1282,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:41.147006+0000","last_change":"2024-02-05T08:16:53.018165+0000","last_active":"2024-02-05T08:17:41.147006+0000","last_peered":"2024-02-05T08:17:41.147006+0000","last_clean":"2024-02-05T08:17:41.147006+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:17:41.147006+0000","last_undegraded":"2024-02-05T08:17:41.147006+0000","last_fullsized":"2024-02-05T08:17:41.147006+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:16:53.018039+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:16:53.018039+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:01.775200+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":774,"num_read_kb":769,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":531,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292366+0000","last_change":"2024-02-05T08:16:01.857925+0000","last_active":"2024-02-05T08:17:43.292366+0000","last_peered":"2024-02-05T08:17:43.292366+0000","last_clean":"2024-02-05T08:17:43.292366+0000","last_became_active":"2024-02-05T08:14:24.992820+0000","last_became_peered":"2024-02-05T08:14:24.992820+0000","last_unstale":"2024-02-05T08:17:43.292366+0000","last_undegraded":"2024-02-05T08:17:43.292366+0000","last_fullsized":"2024-02-05T08:17:43.292366+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:16:01.857750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:01.857750+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":478,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.267165+0000","last_change":"2024-02-05T08:16:02.899200+0000","last_active":"2024-02-05T08:17:36.267165+0000","last_peered":"2024-02-05T08:17:36.267165+0000","last_clean":"2024-02-05T08:17:36.267165+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:17:36.267165+0000","last_undegraded":"2024-02-05T08:17:36.267165+0000","last_fullsized":"2024-02-05T08:17:36.267165+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:16:02.899084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"282'12218","reported_seq":12382,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.063543+0000","last_change":"2024-02-05T08:17:17.506968+0000","last_active":"2024-02-05T08:17:43.063543+0000","last_peered":"2024-02-05T08:17:43.063543+0000","last_clean":"2024-02-05T08:17:43.063543+0000","last_became_active":"2024-02-05T08:14:57.764021+0000","last_became_peered":"2024-02-05T08:14:57.764021+0000","last_unstale":"2024-02-05T08:17:43.063543+0000","last_undegraded":"2024-02-05T08:17:43.063543+0000","last_fullsized":"2024-02-05T08:17:43.063543+0000","mapping_epoch":239,"log_start":"267'7800","ondisk_log_start":"267'7800","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'10561","last_scrub_stamp":"2024-02-05T08:17:17.506587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:17.506587+0000","objects_scrubbed":5822,"log_size":4418,"log_dups_size":0,"ondisk_log_size":4418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:32.650366+0000","scrub_duration":1470,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270139392,"num_objects":4122,"num_object_clones":0,"num_object_copies":12366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12218,"num_write_kb":522880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":489,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.289921+0000","last_change":"2024-02-05T08:16:07.996089+0000","last_active":"2024-02-05T08:17:43.289921+0000","last_peered":"2024-02-05T08:17:43.289921+0000","last_clean":"2024-02-05T08:17:43.289921+0000","last_became_active":"2024-02-05T08:12:29.672973+0000","last_became_peered":"2024-02-05T08:12:29.672973+0000","last_unstale":"2024-02-05T08:17:43.289921+0000","last_undegraded":"2024-02-05T08:17:43.289921+0000","last_fullsized":"2024-02-05T08:17:43.289921+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:07.996005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:07.996005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":497,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.290031+0000","last_change":"2024-02-05T08:16:50.981337+0000","last_active":"2024-02-05T08:17:43.290031+0000","last_peered":"2024-02-05T08:17:43.290031+0000","last_clean":"2024-02-05T08:17:43.290031+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:17:43.290031+0000","last_undegraded":"2024-02-05T08:17:43.290031+0000","last_fullsized":"2024-02-05T08:17:43.290031+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:50.981203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:50.981203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:14.667929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":534,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292416+0000","last_change":"2024-02-05T08:15:56.779910+0000","last_active":"2024-02-05T08:17:43.292416+0000","last_peered":"2024-02-05T08:17:43.292416+0000","last_clean":"2024-02-05T08:17:43.292416+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:17:43.292416+0000","last_undegraded":"2024-02-05T08:17:43.292416+0000","last_fullsized":"2024-02-05T08:17:43.292416+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-02-05T08:15:56.779551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:56.779551+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.345009+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":215,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.275559+0000","last_change":"2024-02-05T08:15:58.238221+0000","last_active":"2024-02-05T08:17:36.275559+0000","last_peered":"2024-02-05T08:17:36.275559+0000","last_clean":"2024-02-05T08:17:36.275559+0000","last_became_active":"2024-02-05T08:12:29.688534+0000","last_became_peered":"2024-02-05T08:12:29.688534+0000","last_unstale":"2024-02-05T08:17:36.275559+0000","last_undegraded":"2024-02-05T08:17:36.275559+0000","last_fullsized":"2024-02-05T08:17:36.275559+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:58.238104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:15:58.238104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.125642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"282'12062","reported_seq":12235,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.057079+0000","last_change":"2024-02-05T08:17:01.762530+0000","last_active":"2024-02-05T08:17:43.057079+0000","last_peered":"2024-02-05T08:17:43.057079+0000","last_clean":"2024-02-05T08:17:43.057079+0000","last_became_active":"2024-02-05T08:14:57.764038+0000","last_became_peered":"2024-02-05T08:14:57.764038+0000","last_unstale":"2024-02-05T08:17:43.057079+0000","last_undegraded":"2024-02-05T08:17:43.057079+0000","last_fullsized":"2024-02-05T08:17:43.057079+0000","mapping_epoch":239,"log_start":"267'7400","ondisk_log_start":"267'7400","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9398","last_scrub_stamp":"2024-02-05T08:17:01.762104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:01.762104+0000","objects_scrubbed":6842,"log_size":4662,"log_dups_size":0,"ondisk_log_size":4662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:25.827043+0000","scrub_duration":2222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268828672,"num_objects":4102,"num_object_clones":0,"num_object_copies":12306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4102,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12062,"num_write_kb":517248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":481,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.290626+0000","last_change":"2024-02-05T08:16:00.862974+0000","last_active":"2024-02-05T08:17:43.290626+0000","last_peered":"2024-02-05T08:17:43.290626+0000","last_clean":"2024-02-05T08:17:43.290626+0000","last_became_active":"2024-02-05T08:12:41.735706+0000","last_became_peered":"2024-02-05T08:12:41.735706+0000","last_unstale":"2024-02-05T08:17:43.290626+0000","last_undegraded":"2024-02-05T08:17:43.290626+0000","last_fullsized":"2024-02-05T08:17:43.290626+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:00.862633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:00.862633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.345069+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":492,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.271265+0000","last_change":"2024-02-05T08:15:16.880868+0000","last_active":"2024-02-05T08:17:36.271265+0000","last_peered":"2024-02-05T08:17:36.271265+0000","last_clean":"2024-02-05T08:17:36.271265+0000","last_became_active":"2024-02-05T08:12:41.730010+0000","last_became_peered":"2024-02-05T08:12:41.730010+0000","last_unstale":"2024-02-05T08:17:36.271265+0000","last_undegraded":"2024-02-05T08:17:36.271265+0000","last_fullsized":"2024-02-05T08:17:36.271265+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:16.880446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:16.880446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'20","reported_seq":515,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.295005+0000","last_change":"2024-02-05T08:16:02.890238+0000","last_active":"2024-02-05T08:17:43.295005+0000","last_peered":"2024-02-05T08:17:43.295005+0000","last_clean":"2024-02-05T08:17:43.295005+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:17:43.295005+0000","last_undegraded":"2024-02-05T08:17:43.295005+0000","last_fullsized":"2024-02-05T08:17:43.295005+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:16:02.886731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:02.886731+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"277'11","reported_seq":665,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292694+0000","last_change":"2024-02-05T08:17:07.973477+0000","last_active":"2024-02-05T08:17:43.292694+0000","last_peered":"2024-02-05T08:17:43.292694+0000","last_clean":"2024-02-05T08:17:43.292694+0000","last_became_active":"2024-02-05T08:12:29.683091+0000","last_became_peered":"2024-02-05T08:12:29.683091+0000","last_unstale":"2024-02-05T08:17:43.292694+0000","last_undegraded":"2024-02-05T08:17:43.292694+0000","last_fullsized":"2024-02-05T08:17:43.292694+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_clean_scrub_stamp":"2024-02-05T08:17:07.973171+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:19.190363+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"282'16430","reported_seq":16704,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:42.837057+0000","last_change":"2024-02-05T08:17:31.249392+0000","last_active":"2024-02-05T08:17:42.837057+0000","last_peered":"2024-02-05T08:17:42.837057+0000","last_clean":"2024-02-05T08:17:42.837057+0000","last_became_active":"2024-02-05T08:14:57.761443+0000","last_became_peered":"2024-02-05T08:14:57.761443+0000","last_unstale":"2024-02-05T08:17:42.837057+0000","last_undegraded":"2024-02-05T08:17:42.837057+0000","last_fullsized":"2024-02-05T08:17:42.837057+0000","mapping_epoch":239,"log_start":"278'11802","ondisk_log_start":"278'11802","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":4323,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:45.893269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272367616,"num_objects":4156,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12126,"num_write_kb":521024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":487,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.268417+0000","last_change":"2024-02-05T08:15:29.563981+0000","last_active":"2024-02-05T08:17:36.268417+0000","last_peered":"2024-02-05T08:17:36.268417+0000","last_clean":"2024-02-05T08:17:36.268417+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:17:36.268417+0000","last_undegraded":"2024-02-05T08:17:36.268417+0000","last_fullsized":"2024-02-05T08:17:36.268417+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:29.563876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:29.563876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.010262+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":498,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292458+0000","last_change":"2024-02-05T08:16:08.716324+0000","last_active":"2024-02-05T08:17:43.292458+0000","last_peered":"2024-02-05T08:17:43.292458+0000","last_clean":"2024-02-05T08:17:43.292458+0000","last_became_active":"2024-02-05T08:11:23.200753+0000","last_became_peered":"2024-02-05T08:11:23.200753+0000","last_unstale":"2024-02-05T08:17:43.292458+0000","last_undegraded":"2024-02-05T08:17:43.292458+0000","last_fullsized":"2024-02-05T08:17:43.292458+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:16:08.716199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:08.716199+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"236'28","reported_seq":568,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.258450+0000","last_change":"2024-02-05T08:17:12.688139+0000","last_active":"2024-02-05T08:17:36.258450+0000","last_peered":"2024-02-05T08:17:36.258450+0000","last_clean":"2024-02-05T08:17:36.258450+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:17:36.258450+0000","last_undegraded":"2024-02-05T08:17:36.258450+0000","last_fullsized":"2024-02-05T08:17:36.258450+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:17:12.687898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:12.687898+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:34.476715+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"194'10","reported_seq":589,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292014+0000","last_change":"2024-02-05T08:17:14.040873+0000","last_active":"2024-02-05T08:17:43.292014+0000","last_peered":"2024-02-05T08:17:43.292014+0000","last_clean":"2024-02-05T08:17:43.292014+0000","last_became_active":"2024-02-05T08:12:41.721686+0000","last_became_peered":"2024-02-05T08:12:41.721686+0000","last_unstale":"2024-02-05T08:17:43.292014+0000","last_undegraded":"2024-02-05T08:17:43.292014+0000","last_fullsized":"2024-02-05T08:17:43.292014+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'10","last_scrub_stamp":"2024-02-05T08:17:14.040769+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:17:14.040769+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:36.262898+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"282'12269","reported_seq":12370,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.038146+0000","last_change":"2024-02-05T08:17:18.266778+0000","last_active":"2024-02-05T08:17:43.038146+0000","last_peered":"2024-02-05T08:17:43.038146+0000","last_clean":"2024-02-05T08:17:43.038146+0000","last_became_active":"2024-02-05T08:14:57.765609+0000","last_became_peered":"2024-02-05T08:14:57.765609+0000","last_unstale":"2024-02-05T08:17:43.038146+0000","last_undegraded":"2024-02-05T08:17:43.038146+0000","last_fullsized":"2024-02-05T08:17:43.038146+0000","mapping_epoch":239,"log_start":"267'7800","ondisk_log_start":"267'7800","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4469,"log_dups_size":0,"ondisk_log_size":4469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:44.018730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274726936,"num_objects":4193,"num_object_clones":0,"num_object_copies":12579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4193,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12269,"num_write_kb":526721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":488,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.256604+0000","last_change":"2024-02-05T08:17:10.764121+0000","last_active":"2024-02-05T08:17:36.256604+0000","last_peered":"2024-02-05T08:17:36.256604+0000","last_clean":"2024-02-05T08:17:36.256604+0000","last_became_active":"2024-02-05T08:12:41.731253+0000","last_became_peered":"2024-02-05T08:12:41.731253+0000","last_unstale":"2024-02-05T08:17:36.256604+0000","last_undegraded":"2024-02-05T08:17:36.256604+0000","last_fullsized":"2024-02-05T08:17:36.256604+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:10.764008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:10.764008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:12.220343+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":210,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.268361+0000","last_change":"2024-02-05T08:15:15.390114+0000","last_active":"2024-02-05T08:17:36.268361+0000","last_peered":"2024-02-05T08:17:36.268361+0000","last_clean":"2024-02-05T08:17:36.268361+0000","last_became_active":"2024-02-05T08:12:41.723699+0000","last_became_peered":"2024-02-05T08:12:41.723699+0000","last_unstale":"2024-02-05T08:17:36.268361+0000","last_undegraded":"2024-02-05T08:17:36.268361+0000","last_fullsized":"2024-02-05T08:17:36.268361+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:15:15.389990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:15.389990+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:38.010122+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"236'38","reported_seq":548,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.272503+0000","last_change":"2024-02-05T08:17:05.909449+0000","last_active":"2024-02-05T08:17:36.272503+0000","last_peered":"2024-02-05T08:17:36.272503+0000","last_clean":"2024-02-05T08:17:36.272503+0000","last_became_active":"2024-02-05T08:12:41.728044+0000","last_became_peered":"2024-02-05T08:12:41.728044+0000","last_unstale":"2024-02-05T08:17:36.272503+0000","last_undegraded":"2024-02-05T08:17:36.272503+0000","last_fullsized":"2024-02-05T08:17:36.272503+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'38","last_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:17:05.909305+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:33.494770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":484,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.258246+0000","last_change":"2024-02-05T08:16:51.793138+0000","last_active":"2024-02-05T08:17:36.258246+0000","last_peered":"2024-02-05T08:17:36.258246+0000","last_clean":"2024-02-05T08:17:36.258246+0000","last_became_active":"2024-02-05T08:12:41.720247+0000","last_became_peered":"2024-02-05T08:12:41.720247+0000","last_unstale":"2024-02-05T08:17:36.258246+0000","last_undegraded":"2024-02-05T08:17:36.258246+0000","last_fullsized":"2024-02-05T08:17:36.258246+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:51.793017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:51.793017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:02.823985+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"283'12286","reported_seq":12468,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.368437+0000","last_change":"2024-02-05T08:16:55.072732+0000","last_active":"2024-02-05T08:17:43.368437+0000","last_peered":"2024-02-05T08:17:43.368437+0000","last_clean":"2024-02-05T08:17:43.368437+0000","last_became_active":"2024-02-05T08:14:57.764174+0000","last_became_peered":"2024-02-05T08:14:57.764174+0000","last_unstale":"2024-02-05T08:17:43.368437+0000","last_undegraded":"2024-02-05T08:17:43.368437+0000","last_fullsized":"2024-02-05T08:17:43.368437+0000","mapping_epoch":239,"log_start":"268'8100","ondisk_log_start":"268'8100","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'9088","last_scrub_stamp":"2024-02-05T08:16:55.072580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:16:55.072580+0000","objects_scrubbed":7413,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:21.023713+0000","scrub_duration":2137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270794752,"num_objects":4132,"num_object_clones":0,"num_object_copies":12396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12286,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":481,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.270883+0000","last_change":"2024-02-05T08:16:12.876608+0000","last_active":"2024-02-05T08:17:36.270883+0000","last_peered":"2024-02-05T08:17:36.270883+0000","last_clean":"2024-02-05T08:17:36.270883+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:17:36.270883+0000","last_undegraded":"2024-02-05T08:17:36.270883+0000","last_fullsized":"2024-02-05T08:17:36.270883+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:12.876345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:12.876345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":642,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.744641+0000","last_change":"2024-02-05T08:16:49.737238+0000","last_active":"2024-02-05T08:17:36.744641+0000","last_peered":"2024-02-05T08:17:36.744641+0000","last_clean":"2024-02-05T08:17:36.744641+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:17:36.744641+0000","last_undegraded":"2024-02-05T08:17:36.744641+0000","last_fullsized":"2024-02-05T08:17:36.744641+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:16:49.733743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:49.733743+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:03.692915+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":208,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292458+0000","last_change":"2024-02-05T08:16:11.024366+0000","last_active":"2024-02-05T08:17:43.292458+0000","last_peered":"2024-02-05T08:17:43.292458+0000","last_clean":"2024-02-05T08:17:43.292458+0000","last_became_active":"2024-02-05T08:12:41.724969+0000","last_became_peered":"2024-02-05T08:12:41.724969+0000","last_unstale":"2024-02-05T08:17:43.292458+0000","last_undegraded":"2024-02-05T08:17:43.292458+0000","last_fullsized":"2024-02-05T08:17:43.292458+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:11.024176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:11.024176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"236'27","reported_seq":557,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.291043+0000","last_change":"2024-02-05T08:17:13.636942+0000","last_active":"2024-02-05T08:17:43.291043+0000","last_peered":"2024-02-05T08:17:43.291043+0000","last_clean":"2024-02-05T08:17:43.291043+0000","last_became_active":"2024-02-05T08:12:41.725007+0000","last_became_peered":"2024-02-05T08:12:41.725007+0000","last_unstale":"2024-02-05T08:17:43.291043+0000","last_undegraded":"2024-02-05T08:17:43.291043+0000","last_fullsized":"2024-02-05T08:17:43.291043+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:17:13.636688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:13.636688+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:25.647442+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"282'12459","reported_seq":12556,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.055135+0000","last_change":"2024-02-05T08:14:57.765880+0000","last_active":"2024-02-05T08:17:43.055135+0000","last_peered":"2024-02-05T08:17:43.055135+0000","last_clean":"2024-02-05T08:17:43.055135+0000","last_became_active":"2024-02-05T08:14:57.765255+0000","last_became_peered":"2024-02-05T08:14:57.765255+0000","last_unstale":"2024-02-05T08:17:43.055135+0000","last_undegraded":"2024-02-05T08:17:43.055135+0000","last_fullsized":"2024-02-05T08:17:43.055135+0000","mapping_epoch":239,"log_start":"268'8000","ondisk_log_start":"268'8000","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:14:56.736469+0000","objects_scrubbed":0,"log_size":4459,"log_dups_size":0,"ondisk_log_size":4459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:44.018767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272695296,"num_objects":4161,"num_object_clones":0,"num_object_copies":12483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12459,"num_write_kb":531840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":474,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.276073+0000","last_change":"2024-02-05T08:16:50.260245+0000","last_active":"2024-02-05T08:17:36.276073+0000","last_peered":"2024-02-05T08:17:36.276073+0000","last_clean":"2024-02-05T08:17:36.276073+0000","last_became_active":"2024-02-05T08:12:41.719835+0000","last_became_peered":"2024-02-05T08:12:41.719835+0000","last_unstale":"2024-02-05T08:17:36.276073+0000","last_undegraded":"2024-02-05T08:17:36.276073+0000","last_fullsized":"2024-02-05T08:17:36.276073+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:50.260123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:16:50.260123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:06.973752+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":497,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.267041+0000","last_change":"2024-02-05T08:16:49.669406+0000","last_active":"2024-02-05T08:17:36.267041+0000","last_peered":"2024-02-05T08:17:36.267041+0000","last_clean":"2024-02-05T08:17:36.267041+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:17:36.267041+0000","last_undegraded":"2024-02-05T08:17:36.267041+0000","last_fullsized":"2024-02-05T08:17:36.267041+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:49.669303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:49.669303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:02.029690+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":493,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.258319+0000","last_change":"2024-02-05T08:16:55.778045+0000","last_active":"2024-02-05T08:17:36.258319+0000","last_peered":"2024-02-05T08:17:36.258319+0000","last_clean":"2024-02-05T08:17:36.258319+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:17:36.258319+0000","last_undegraded":"2024-02-05T08:17:36.258319+0000","last_fullsized":"2024-02-05T08:17:36.258319+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:55.777839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:16:55.777839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:58.223648+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"236'19","reported_seq":548,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.257551+0000","last_change":"2024-02-05T08:16:12.843894+0000","last_active":"2024-02-05T08:17:36.257551+0000","last_peered":"2024-02-05T08:17:36.257551+0000","last_clean":"2024-02-05T08:17:36.257551+0000","last_became_active":"2024-02-05T08:12:29.666284+0000","last_became_peered":"2024-02-05T08:12:29.666284+0000","last_unstale":"2024-02-05T08:17:36.257551+0000","last_undegraded":"2024-02-05T08:17:36.257551+0000","last_fullsized":"2024-02-05T08:17:36.257551+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:16:12.843751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:12.843751+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:40.868927+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"283'12426","reported_seq":12608,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.367955+0000","last_change":"2024-02-05T08:16:53.913489+0000","last_active":"2024-02-05T08:17:43.367955+0000","last_peered":"2024-02-05T08:17:43.367955+0000","last_clean":"2024-02-05T08:17:43.367955+0000","last_became_active":"2024-02-05T08:14:57.764870+0000","last_became_peered":"2024-02-05T08:14:57.764870+0000","last_unstale":"2024-02-05T08:17:43.367955+0000","last_undegraded":"2024-02-05T08:17:43.367955+0000","last_fullsized":"2024-02-05T08:17:43.367955+0000","mapping_epoch":239,"log_start":"269'8300","ondisk_log_start":"269'8300","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'9083","last_scrub_stamp":"2024-02-05T08:16:53.913296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:16:53.913296+0000","objects_scrubbed":7620,"log_size":4126,"log_dups_size":0,"ondisk_log_size":4126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:16.578086+0000","scrub_duration":1959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276168704,"num_objects":4214,"num_object_clones":0,"num_object_copies":12642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12426,"num_write_kb":532480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":500,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.257734+0000","last_change":"2024-02-05T08:17:09.581135+0000","last_active":"2024-02-05T08:17:36.257734+0000","last_peered":"2024-02-05T08:17:36.257734+0000","last_clean":"2024-02-05T08:17:36.257734+0000","last_became_active":"2024-02-05T08:12:41.737224+0000","last_became_peered":"2024-02-05T08:12:41.737224+0000","last_unstale":"2024-02-05T08:17:36.257734+0000","last_undegraded":"2024-02-05T08:17:36.257734+0000","last_fullsized":"2024-02-05T08:17:36.257734+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:17:09.581011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:09.581011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:10.528750+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":480,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.291663+0000","last_change":"2024-02-05T08:17:12.127732+0000","last_active":"2024-02-05T08:17:43.291663+0000","last_peered":"2024-02-05T08:17:43.291663+0000","last_clean":"2024-02-05T08:17:43.291663+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:17:43.291663+0000","last_undegraded":"2024-02-05T08:17:43.291663+0000","last_fullsized":"2024-02-05T08:17:43.291663+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:12.127605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:12.127605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:31.758118+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":223,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.275728+0000","last_change":"2024-02-05T08:17:14.087240+0000","last_active":"2024-02-05T08:17:36.275728+0000","last_peered":"2024-02-05T08:17:36.275728+0000","last_clean":"2024-02-05T08:17:36.275728+0000","last_became_active":"2024-02-05T08:12:29.678501+0000","last_became_peered":"2024-02-05T08:12:29.678501+0000","last_unstale":"2024-02-05T08:17:36.275728+0000","last_undegraded":"2024-02-05T08:17:36.275728+0000","last_fullsized":"2024-02-05T08:17:36.275728+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:14.087110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:17:14.087110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:32.049970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"236'31","reported_seq":530,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.257086+0000","last_change":"2024-02-05T08:16:58.775056+0000","last_active":"2024-02-05T08:17:36.257086+0000","last_peered":"2024-02-05T08:17:36.257086+0000","last_clean":"2024-02-05T08:17:36.257086+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:17:36.257086+0000","last_undegraded":"2024-02-05T08:17:36.257086+0000","last_fullsized":"2024-02-05T08:17:36.257086+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'31","last_scrub_stamp":"2024-02-05T08:16:58.774921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:58.774921+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:17.545014+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"282'12265","reported_seq":12428,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:42.606999+0000","last_change":"2024-02-05T08:16:15.159977+0000","last_active":"2024-02-05T08:17:42.606999+0000","last_peered":"2024-02-05T08:17:42.606999+0000","last_clean":"2024-02-05T08:17:42.606999+0000","last_became_active":"2024-02-05T08:14:57.762652+0000","last_became_peered":"2024-02-05T08:14:57.762652+0000","last_unstale":"2024-02-05T08:17:42.606999+0000","last_undegraded":"2024-02-05T08:17:42.606999+0000","last_fullsized":"2024-02-05T08:17:42.606999+0000","mapping_epoch":239,"log_start":"267'7200","ondisk_log_start":"267'7200","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'5960","last_scrub_stamp":"2024-02-05T08:16:15.159720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:16:15.159720+0000","objects_scrubbed":5912,"log_size":5065,"log_dups_size":0,"ondisk_log_size":5065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:46.280848+0000","scrub_duration":1865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273219584,"num_objects":4169,"num_object_clones":0,"num_object_copies":12507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12265,"num_write_kb":525888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":473,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.257989+0000","last_change":"2024-02-05T08:16:52.875421+0000","last_active":"2024-02-05T08:17:36.257989+0000","last_peered":"2024-02-05T08:17:36.257989+0000","last_clean":"2024-02-05T08:17:36.257989+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:17:36.257989+0000","last_undegraded":"2024-02-05T08:17:36.257989+0000","last_fullsized":"2024-02-05T08:17:36.257989+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:52.874790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:16:52.874790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:04.742008+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":659,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.741323+0000","last_change":"2024-02-05T08:16:53.367751+0000","last_active":"2024-02-05T08:17:36.741323+0000","last_peered":"2024-02-05T08:17:36.741323+0000","last_clean":"2024-02-05T08:17:36.741323+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:17:36.741323+0000","last_undegraded":"2024-02-05T08:17:36.741323+0000","last_fullsized":"2024-02-05T08:17:36.741323+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:16:53.367552+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:16:53.367552+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:07.009993+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":199,"reported_epoch":283,"state":"active+clean","last_fresh":"2024-02-05T08:17:43.292509+0000","last_change":"2024-02-05T08:15:29.437631+0000","last_active":"2024-02-05T08:17:43.292509+0000","last_peered":"2024-02-05T08:17:43.292509+0000","last_clean":"2024-02-05T08:17:43.292509+0000","last_became_active":"2024-02-05T08:12:41.757159+0000","last_became_peered":"2024-02-05T08:12:41.757159+0000","last_unstale":"2024-02-05T08:17:43.292509+0000","last_undegraded":"2024-02-05T08:17:43.292509+0000","last_fullsized":"2024-02-05T08:17:43.292509+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:29.437487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:29.437487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:43.344735+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"236'20","reported_seq":565,"reported_epoch":282,"state":"active+clean","last_fresh":"2024-02-05T08:17:36.260906+0000","last_change":"2024-02-05T08:16:01.862035+0000","last_active":"2024-02-05T08:17:36.260906+0000","last_peered":"2024-02-05T08:17:36.260906+0000","last_clean":"2024-02-05T08:17:36.260906+0000","last_became_active":"2024-02-05T08:12:41.727271+0000","last_became_peered":"2024-02-05T08:12:41.727271+0000","last_unstale":"2024-02-05T08:17:36.260906+0000","last_undegraded":"2024-02-05T08:17:36.260906+0000","last_fullsized":"2024-02-05T08:17:36.260906+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:16:01.861532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:01.861532+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:40.868755+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4349886488,"num_objects":66375,"num_object_clones":0,"num_object_copies":199125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66375,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":197101,"num_write_kb":8431169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13076213760,"data_stored":13076201544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74948,"ondisk_log_size":74948,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":774,"num_read_kb":769,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1123,"num_read_kb":880,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":4032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":12894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":764,"ondisk_log_size":764,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037921,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3414740,"kb_used_data":2946252,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":90330412,"statfs":{"total":95995035648,"available":92498341888,"internally_reserved":0,"allocated":3016962048,"data_stored":3016661612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":479721930},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:16:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5649999999999999,"5min":1.651,"15min":1.3919999999999999},"min":{"1min":0.33000000000000002,"5min":0.22,"15min":0.217},"max":{"1min":7.4820000000000002,"5min":52.488999999999997,"15min":60.127000000000002},"last":3.016},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.72,"15min":1.4139999999999999},"min":{"1min":0.24099999999999999,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":5.548,"5min":52.335000000000001,"15min":58.283000000000001},"last":1.482}]},{"osd":1,"last update":"Mon Feb 5 08:16:51 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.702,"15min":1.359},"min":{"1min":0.23699999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":4.1769999999999996,"5min":32.682000000000002,"15min":38.399999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.5589999999999999,"5min":1.5820000000000001,"15min":1.218},"min":{"1min":0.33500000000000002,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":7.5380000000000003,"5min":32.412999999999997,"15min":38.195999999999998},"last":3.0470000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:16:51 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.157,"15min":0.92800000000000005},"min":{"1min":0.35799999999999998,"5min":0.26800000000000002,"15min":0.24099999999999999},"max":{"1min":7.5709999999999997,"5min":7.5709999999999997,"15min":7.6779999999999999},"last":3.0710000000000002},{"interface":"front","average":{"1min":1.9039999999999999,"5min":1.335,"15min":1.0409999999999999},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.24299999999999999},"max":{"1min":5.9400000000000004,"5min":5.9400000000000004,"15min":8.9100000000000001},"last":0.40899999999999997}]},{"osd":3,"last update":"Mon Feb 5 08:16:51 2024","interfaces":[{"interface":"back","average":{"1min":1.8300000000000001,"5min":1.1699999999999999,"15min":1.2589999999999999},"min":{"1min":0.41899999999999998,"5min":0.249,"15min":0.249},"max":{"1min":5.984,"5min":5.984,"15min":58.515000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.139,"15min":1.22},"min":{"1min":0.314,"5min":0.27400000000000002,"15min":0.27000000000000002},"max":{"1min":2.5449999999999999,"5min":5.6879999999999997,"15min":58.225999999999999},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:16:51 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.6950000000000001,"15min":0.84199999999999997},"min":{"1min":0.29999999999999999,"5min":0.23699999999999999,"15min":0.22},"max":{"1min":7.6340000000000003,"5min":49.601999999999997,"15min":49.601999999999997},"last":2.9670000000000001},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.7370000000000001,"15min":0.86299999999999999},"min":{"1min":0.34799999999999998,"5min":0.28100000000000003,"15min":0.249},"max":{"1min":7.8399999999999999,"5min":49.795999999999999,"15min":49.795999999999999},"last":0.443}]},{"osd":5,"last update":"Mon Feb 5 08:16:51 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.788,"15min":0.89600000000000002},"min":{"1min":0.46500000000000002,"5min":0.251,"15min":0.24299999999999999},"max":{"1min":4.3360000000000003,"5min":49.914000000000001,"15min":49.914000000000001},"last":1.4399999999999999},{"interface":"front","average":{"1min":2.1930000000000001,"5min":1.6599999999999999,"15min":0.874},"min":{"1min":0.45800000000000002,"5min":0.314,"15min":0.27000000000000002},"max":{"1min":7.6040000000000001,"5min":52.256,"15min":52.256},"last":1.526}]},{"osd":6,"last update":"Mon Feb 5 08:16:51 2024","interfaces":[{"interface":"back","average":{"1min":1.653,"5min":1.167,"15min":0.71899999999999997},"min":{"1min":0.40799999999999997,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.722,"5min":8.0050000000000008,"15min":8.0050000000000008},"last":0.38900000000000001},{"interface":"front","average":{"1min":2.028,"5min":1.125,"15min":0.69599999999999995},"min":{"1min":0.45400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":7.8029999999999999,"5min":7.8029999999999999,"15min":7.8029999999999999},"last":0.70399999999999996}]}]},{"osd":6,"up_from":51,"seq":219043332263,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1082020,"kb_used_data":807900,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92663132,"statfs":{"total":95995035648,"available":94887047168,"internally_reserved":0,"allocated":827289600,"data_stored":826972086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":4.1929999999999996,"5min":3.2610000000000001,"15min":1.657},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":76.123000000000005,"5min":102.819,"15min":102.819},"last":5.7140000000000004},{"interface":"front","average":{"1min":4.2969999999999997,"5min":3.0819999999999999,"15min":1.724},"min":{"1min":0.249,"5min":0.214,"15min":0.214},"max":{"1min":76.037000000000006,"5min":102.75,"15min":102.75},"last":0.498}]},{"osd":1,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.6439999999999999,"15min":1.1299999999999999},"min":{"1min":0.30599999999999999,"5min":0.24199999999999999,"15min":0.22700000000000001},"max":{"1min":3.2200000000000002,"5min":23.773,"15min":23.773},"last":1.4239999999999999},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.552,"15min":1.0920000000000001},"min":{"1min":0.28999999999999998,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":4.4509999999999996,"5min":23.719000000000001,"15min":23.719000000000001},"last":0.90600000000000003}]},{"osd":2,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.22,"15min":1.016},"min":{"1min":0.27700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.524,"5min":22.503,"15min":22.503},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.96299999999999997,"5min":1.0409999999999999,"15min":0.93799999999999994},"min":{"1min":0.27300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.1720000000000002,"5min":22.466000000000001,"15min":22.466000000000001},"last":0.52900000000000003}]},{"osd":3,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":1.905,"15min":1.288},"min":{"1min":0.33600000000000002,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":2.75,"5min":102.438,"15min":102.438},"last":1.399},{"interface":"front","average":{"1min":1.216,"5min":2.0089999999999999,"15min":1.3080000000000001},"min":{"1min":0.32200000000000001,"5min":0.316,"15min":0.29099999999999998},"max":{"1min":4.4939999999999998,"5min":102.288,"15min":102.288},"last":0.96299999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.95099999999999996,"15min":0.60499999999999998},"min":{"1min":0.29199999999999998,"5min":0.26700000000000002,"15min":0.223},"max":{"1min":3.141,"5min":9.8970000000000002,"15min":9.8970000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.93100000000000005,"15min":0.62},"min":{"1min":0.316,"5min":0.28599999999999998,"15min":0.26600000000000001},"max":{"1min":3.4079999999999999,"5min":4.9589999999999996,"15min":4.9589999999999996},"last":1.3280000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.7649999999999999,"15min":0.92700000000000005},"min":{"1min":0.29299999999999998,"5min":0.25,"15min":0.24199999999999999},"max":{"1min":3.7709999999999999,"5min":43.643000000000001,"15min":43.643000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.083,"5min":1.641,"15min":0.91100000000000003},"min":{"1min":0.376,"5min":0.307,"15min":0.30399999999999999},"max":{"1min":3.617,"5min":43.893000000000001,"15min":43.893000000000001},"last":1.369}]},{"osd":7,"last update":"Mon Feb 5 08:16:59 2024","interfaces":[{"interface":"back","average":{"1min":3.2530000000000001,"5min":1.2949999999999999,"15min":0.77900000000000003},"min":{"1min":0.376,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":24.728999999999999,"5min":24.728999999999999,"15min":24.728999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":2.6589999999999998,"5min":1.2010000000000001,"15min":0.75800000000000001},"min":{"1min":0.375,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":21.91,"5min":21.91,"15min":21.91},"last":1.986}]}]},{"osd":1,"up_from":13,"seq":55834575040,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2924904,"kb_used_data":2402080,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":90820248,"statfs":{"total":95995035648,"available":92999933952,"internally_reserved":0,"allocated":2459729920,"data_stored":2459408168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:17:16 2024","interfaces":[{"interface":"back","average":{"1min":2.8250000000000002,"5min":1.716,"15min":1.5800000000000001},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.14699999999999999},"max":{"1min":36.462000000000003,"5min":36.462000000000003,"15min":86.105000000000004},"last":0.27400000000000002},{"interface":"front","average":{"1min":2.593,"5min":1.8480000000000001,"15min":1.8220000000000001},"min":{"1min":0.21099999999999999,"5min":0.20699999999999999,"15min":0.16},"max":{"1min":35.878,"5min":35.878,"15min":85.665000000000006},"last":0.55000000000000004}]},{"osd":2,"last update":"Mon Feb 5 08:16:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.4359999999999999,"15min":1.333},"min":{"1min":0.22900000000000001,"5min":0.219,"15min":0.17799999999999999},"max":{"1min":4.2859999999999996,"5min":19.016999999999999,"15min":49.030999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.6299999999999999,"15min":1.51},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":3.589,"5min":20.434000000000001,"15min":48.984000000000002},"last":0.59299999999999997}]},{"osd":3,"last update":"Mon Feb 5 08:17:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.546,"15min":1.488},"min":{"1min":0.222,"5min":0.222,"15min":0.20599999999999999},"max":{"1min":5.3289999999999997,"5min":31.896000000000001,"15min":49.067999999999998},"last":0.311},{"interface":"front","average":{"1min":0.78300000000000003,"5min":1.304,"15min":1.427},"min":{"1min":0.27200000000000002,"5min":0.27100000000000002,"15min":0.24199999999999999},"max":{"1min":2.8839999999999999,"5min":32.018999999999998,"15min":48.734999999999999},"last":0.35999999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:17:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.706,"15min":1.1359999999999999},"min":{"1min":0.36699999999999999,"5min":0.33100000000000002,"15min":0.26000000000000001},"max":{"1min":3.0190000000000001,"5min":25.952000000000002,"15min":25.952000000000002},"last":0.623},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.762,"15min":1.232},"min":{"1min":0.38500000000000001,"5min":0.31,"15min":0.27100000000000002},"max":{"1min":3.613,"5min":25.922000000000001,"15min":25.922000000000001},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:16:53 2024","interfaces":[{"interface":"back","average":{"1min":2.3330000000000002,"5min":2.1739999999999999,"15min":1.298},"min":{"1min":0.434,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":9.6820000000000004,"5min":54.552999999999997,"15min":54.552999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.296,"5min":2.0139999999999998,"15min":1.3380000000000001},"min":{"1min":0.41899999999999998,"5min":0.32500000000000001,"15min":0.28999999999999998},"max":{"1min":5.3659999999999997,"5min":55.811,"15min":55.811},"last":0.745}]},{"osd":6,"last update":"Mon Feb 5 08:17:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.6040000000000001,"15min":1.1930000000000001},"min":{"1min":0.35599999999999998,"5min":0.35199999999999998,"15min":0.33100000000000002},"max":{"1min":3.6819999999999999,"5min":54.384,"15min":54.384},"last":1.8180000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.8129999999999999,"15min":1.22},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.34899999999999998},"max":{"1min":3.5419999999999998,"5min":55.210999999999999,"15min":55.210999999999999},"last":0.42799999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:16:46 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.4379999999999999,"15min":1.194},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":4.4660000000000002,"5min":7.9390000000000001,"15min":13.749000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.522,"5min":1.478,"15min":1.1559999999999999},"min":{"1min":0.41699999999999998,"5min":0.379,"15min":0.379},"max":{"1min":3.9470000000000001,"5min":7.9640000000000004,"15min":7.9640000000000004},"last":1.0740000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738565,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2089636,"kb_used_data":1593368,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":91655516,"statfs":{"total":95995035648,"available":93855248384,"internally_reserved":0,"allocated":1631608832,"data_stored":1631304053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":1.2909999999999999,"15min":1.76},"min":{"1min":0.189,"5min":0.17499999999999999,"15min":0.152},"max":{"1min":2.5600000000000001,"5min":15.539,"15min":66.772999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.623,"5min":1.4730000000000001,"15min":1.4810000000000001},"min":{"1min":0.23300000000000001,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":1.9790000000000001,"5min":15.327999999999999,"15min":46.198999999999998},"last":1.413}]},{"osd":2,"last update":"Mon Feb 5 08:16:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.681,"15min":1.46},"min":{"1min":0.24399999999999999,"5min":0.22500000000000001,"15min":0.216},"max":{"1min":3.9830000000000001,"5min":12.074,"15min":46.170999999999999},"last":1.4410000000000001},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.7230000000000001,"15min":1.6040000000000001},"min":{"1min":0.20799999999999999,"5min":0.20399999999999999,"15min":0.19400000000000001},"max":{"1min":2.613,"5min":15.569000000000001,"15min":46.543999999999997},"last":0.47399999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.323,"15min":1.657},"min":{"1min":0.27100000000000002,"5min":0.26100000000000001,"15min":0.21099999999999999},"max":{"1min":2.722,"5min":7.3929999999999998,"15min":67.888000000000005},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.5349999999999999,"15min":1.956},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22600000000000001},"max":{"1min":1.6910000000000001,"5min":15.494,"15min":66.891999999999996},"last":0.40799999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:17:42 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.667,"15min":1.2190000000000001},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":2.6739999999999999,"5min":22.920000000000002,"15min":22.920000000000002},"last":0.317},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.4750000000000001,"15min":1.1930000000000001},"min":{"1min":0.39400000000000002,"5min":0.33800000000000002,"15min":0.26900000000000002},"max":{"1min":2.7610000000000001,"5min":22.983000000000001,"15min":22.983000000000001},"last":0.84899999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:17:03 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.6990000000000001,"15min":1.2809999999999999},"min":{"1min":0.38900000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.2939999999999996,"5min":22.800999999999998,"15min":22.800999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":1.49,"5min":1.631,"15min":1.2030000000000001},"min":{"1min":0.44900000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.8050000000000002,"5min":22.744,"15min":22.744},"last":0.95799999999999996}]},{"osd":6,"last update":"Mon Feb 5 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.3080000000000001,"15min":1.1419999999999999},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.79,"5min":7.8220000000000001,"15min":12.967000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.335,"5min":1.429,"15min":1.228},"min":{"1min":0.34200000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.1210000000000004,"5min":15.721,"15min":15.721},"last":0.66600000000000004}]},{"osd":7,"last update":"Mon Feb 5 08:16:49 2024","interfaces":[{"interface":"back","average":{"1min":5.6779999999999999,"5min":2.645,"15min":1.48},"min":{"1min":0.38100000000000001,"5min":0.317,"15min":0.317},"max":{"1min":68.241,"5min":68.241,"15min":68.241},"last":0.51700000000000002},{"interface":"front","average":{"1min":5.7809999999999997,"5min":2.556,"15min":1.5249999999999999},"min":{"1min":0.44600000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":68.980000000000004,"5min":68.980000000000004,"15min":68.980000000000004},"last":0.88300000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411515,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1825944,"kb_used_data":1343568,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":91919208,"statfs":{"total":95995035648,"available":94125268992,"internally_reserved":0,"allocated":1375813632,"data_stored":1375507948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":493943238},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:16:50 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":3.3130000000000002,"15min":1.9610000000000001},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.7100000000000009,"5min":107.155,"15min":107.155},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.325,"5min":3.6099999999999999,"15min":2.105},"min":{"1min":0.23100000000000001,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":8.4710000000000001,"5min":107.221,"15min":107.221},"last":0.36099999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.776,"15min":1.3859999999999999},"min":{"1min":0.216,"5min":0.19400000000000001,"15min":0.17999999999999999},"max":{"1min":2.5720000000000001,"5min":55.942,"15min":55.942},"last":0.44900000000000001},{"interface":"front","average":{"1min":1.843,"5min":2.1589999999999998,"15min":1.522},"min":{"1min":0.29999999999999999,"5min":0.22900000000000001,"15min":0.193},"max":{"1min":10.871,"5min":55.960999999999999,"15min":55.960999999999999},"last":0.433}]},{"osd":3,"last update":"Mon Feb 5 08:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.28,"15min":1.7490000000000001},"min":{"1min":0.30299999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.3629999999999995,"5min":8.6590000000000007,"15min":68.108000000000004},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":1.05,"15min":1.554},"min":{"1min":0.251,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":1.1699999999999999,"5min":6.8600000000000003,"15min":67.945999999999998},"last":0.70299999999999996}]},{"osd":4,"last update":"Mon Feb 5 08:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.304,"15min":1.3300000000000001},"min":{"1min":0.317,"5min":0.28399999999999997,"15min":0.27400000000000002},"max":{"1min":3.5550000000000002,"5min":8.2620000000000005,"15min":31.596},"last":0.317},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.2,"15min":1.1479999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.252},"max":{"1min":2.6030000000000002,"5min":6.0220000000000002,"15min":31.57},"last":0.74299999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:17:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.462,"15min":1.3380000000000001},"min":{"1min":0.38,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":7.077,"5min":7.077,"15min":58.799999999999997},"last":1.397},{"interface":"front","average":{"1min":1.095,"5min":1.3220000000000001,"15min":1.29},"min":{"1min":0.34000000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.4809999999999999,"5min":7.3170000000000002,"15min":58.615000000000002},"last":0.75700000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.2549999999999999,"15min":1.1890000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.2999999999999998,"5min":8.3089999999999993,"15min":15.859},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.999,"5min":1.1319999999999999,"15min":1.0249999999999999},"min":{"1min":0.41399999999999998,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":4.0140000000000002,"5min":6.0570000000000004,"15min":6.1820000000000004},"last":0.82999999999999996}]},{"osd":7,"last update":"Mon Feb 5 08:16:50 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.3129999999999999,"15min":1.1419999999999999},"min":{"1min":0.45600000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.8940000000000001,"5min":7.3600000000000003,"15min":15.986000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":2.226,"5min":1.657,"15min":1.2929999999999999},"min":{"1min":0.42499999999999999,"5min":0.40400000000000003,"15min":0.35299999999999998},"max":{"1min":8.3369999999999997,"5min":8.3369999999999997,"15min":15.951000000000001},"last":3.1200000000000001}]}]},{"osd":3,"up_from":0,"seq":115964117174,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953649,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1475340,"kb_used_data":1083204,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92269812,"statfs":{"total":95995035648,"available":94484287488,"internally_reserved":0,"allocated":1109200896,"data_stored":1108906621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":1.7969999999999999,"15min":1.5369999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20799999999999999},"max":{"1min":3.1669999999999998,"5min":39.552,"15min":73.078999999999994},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.861,"15min":1.5349999999999999},"min":{"1min":0.249,"5min":0.214,"15min":0.21099999999999999},"max":{"1min":5.625,"5min":39.350999999999999,"15min":73.412000000000006},"last":1.2829999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":2.6869999999999998,"15min":1.4470000000000001},"min":{"1min":0.252,"5min":0.24299999999999999,"15min":0.21099999999999999},"max":{"1min":3.371,"5min":125.054,"15min":125.054},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":2.7570000000000001,"15min":1.3700000000000001},"min":{"1min":0.28499999999999998,"5min":0.26100000000000001,"15min":0.23499999999999999},"max":{"1min":3.2240000000000002,"5min":125.08799999999999,"15min":125.08799999999999},"last":0.56200000000000006}]},{"osd":2,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":1.0840000000000001,"15min":1.2170000000000001},"min":{"1min":0.26500000000000001,"5min":0.247,"15min":0.24199999999999999},"max":{"1min":3.198,"5min":4.6040000000000001,"15min":96.849999999999994},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.248,"5min":1.236,"15min":1.288},"min":{"1min":0.28499999999999998,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":96.143000000000001},"last":1.6539999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.5660000000000001,"15min":1.776},"min":{"1min":0.33200000000000002,"5min":0.28599999999999998,"15min":0.24399999999999999},"max":{"1min":5.5460000000000003,"5min":38.985999999999997,"15min":73.602999999999994},"last":1.589},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.4850000000000001,"15min":1.782},"min":{"1min":0.309,"5min":0.309,"15min":0.27200000000000002},"max":{"1min":3.3079999999999998,"5min":39.290999999999997,"15min":74.817999999999998},"last":1.347}]},{"osd":5,"last update":"Mon Feb 5 08:17:08 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.4650000000000001,"15min":0.76300000000000001},"min":{"1min":0.29399999999999998,"5min":0.25800000000000001,"15min":0.224},"max":{"1min":9.157,"5min":34.167000000000002,"15min":34.167000000000002},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.546,"15min":0.81200000000000006},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":5.6550000000000002,"5min":36.051000000000002,"15min":36.051000000000002},"last":1.698}]},{"osd":6,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.028,"15min":0.65000000000000002},"min":{"1min":0.35799999999999998,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":5.5910000000000002,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":1.3200000000000001},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.91100000000000003,"15min":0.61199999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.3450000000000002,"5min":5.5049999999999999,"15min":5.5049999999999999},"last":1.3799999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:16:40 2024","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.006,"15min":0.65400000000000003},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.27000000000000002},"max":{"1min":4.25,"5min":4.4859999999999998,"15min":4.4859999999999998},"last":1.7490000000000001},{"interface":"front","average":{"1min":1.173,"5min":0.93999999999999995,"15min":0.65300000000000002},"min":{"1min":0.36099999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.6720000000000002,"5min":5.7450000000000001,"15min":5.7450000000000001},"last":1.4159999999999999}]}]},{"osd":5,"up_from":42,"seq":180388626603,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3205064,"kb_used_data":2704640,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":90540088,"statfs":{"total":95995035648,"available":92713050112,"internally_reserved":0,"allocated":2769551360,"data_stored":2769262162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:16:52 2024","interfaces":[{"interface":"back","average":{"1min":2.8300000000000001,"5min":1.6950000000000001,"15min":1.744},"min":{"1min":0.25,"5min":0.222,"15min":0.19400000000000001},"max":{"1min":24.684000000000001,"5min":24.684000000000001,"15min":143.06299999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.5609999999999999,"5min":1.528,"15min":1.681},"min":{"1min":0.248,"5min":0.24299999999999999,"15min":0.20499999999999999},"max":{"1min":24.806000000000001,"5min":24.806000000000001,"15min":143.09700000000001},"last":0.76600000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:16:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.2110000000000001,"15min":1.0429999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.215},"max":{"1min":5.8940000000000001,"5min":5.9000000000000004,"15min":46.200000000000003},"last":3.9769999999999999},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.2250000000000001,"15min":1.121},"min":{"1min":0.26800000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.3519999999999999,"5min":5.4809999999999999,"15min":46.343000000000004},"last":2.3370000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:16:52 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.206,"15min":0.98299999999999998},"min":{"1min":0.29599999999999999,"5min":0.26700000000000002,"15min":0.246},"max":{"1min":4.9539999999999997,"5min":4.9539999999999997,"15min":10.733000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.28,"5min":1.3720000000000001,"15min":1.0469999999999999},"min":{"1min":0.26900000000000002,"5min":0.26700000000000002,"15min":0.26400000000000001},"max":{"1min":5.8250000000000002,"5min":8.0220000000000002,"15min":8.0220000000000002},"last":1.4390000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:16:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.111,"15min":1.6080000000000001},"min":{"1min":0.35899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.9260000000000002,"5min":4.9260000000000002,"15min":143.03399999999999},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.5920000000000001,"15min":1.73},"min":{"1min":0.42699999999999999,"5min":0.32600000000000001,"15min":0.28100000000000003},"max":{"1min":5.8609999999999998,"5min":7.5270000000000001,"15min":143.46799999999999},"last":0.78400000000000003}]},{"osd":4,"last update":"Mon Feb 5 08:16:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.107,"15min":0.65500000000000003},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":3.4199999999999999,"5min":5.3769999999999998,"15min":5.3769999999999998},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.599,"5min":1.002,"15min":0.63},"min":{"1min":0.33400000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":4.8849999999999998,"5min":4.8849999999999998,"15min":4.8849999999999998},"last":0.46500000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.198,"15min":0.71099999999999997},"min":{"1min":0.35799999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":7.5439999999999996,"5min":7.5439999999999996,"15min":7.5439999999999996},"last":0.432},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.1830000000000001,"15min":0.73299999999999998},"min":{"1min":0.38100000000000001,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":7.194,"5min":7.194,"15min":7.194},"last":0.68400000000000005}]},{"osd":7,"last update":"Mon Feb 5 08:16:52 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.3169999999999999,"15min":0.77100000000000002},"min":{"1min":0.45800000000000002,"5min":0.30399999999999999,"15min":0.25900000000000001},"max":{"1min":6.6210000000000004,"5min":9.6940000000000008,"15min":9.6940000000000008},"last":0.753},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.155,"15min":0.72699999999999998},"min":{"1min":0.44600000000000001,"5min":0.318,"15min":0.28199999999999997},"max":{"1min":4.0190000000000001,"5min":8.0289999999999999,"15min":8.0289999999999999},"last":0.80000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1617756160,"data_stored":1617756160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2443513856,"data_stored":2443509784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1361117184,"data_stored":1361117184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1089142784,"data_stored":1089142784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2732331008,"data_stored":2732326936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":820314112,"data_stored":820314112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3012038656,"data_stored":3012034584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:17:51.925 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:17:51.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:52.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:52.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:52.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:52.160 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:52.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:52.262 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:52.293 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:52.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:52.395 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:52.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:52.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:52.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:52.470 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:52 smithi134 ceph-mon[30086]: 9.3 scrub starts 2024-02-05T08:17:52.470 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:52 smithi134 ceph-mon[30086]: 5.1b scrub starts 2024-02-05T08:17:52.470 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:52 smithi134 ceph-mon[30086]: 5.1b scrub ok 2024-02-05T08:17:52.470 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:52 smithi134 ceph-mon[39278]: 9.3 scrub starts 2024-02-05T08:17:52.470 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:52 smithi134 ceph-mon[39278]: 5.1b scrub starts 2024-02-05T08:17:52.470 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:52 smithi134 ceph-mon[39278]: 5.1b scrub ok 2024-02-05T08:17:52.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:52.573 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:52.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:52.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:52.675 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:52.717 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:52.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:52.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:52 smithi196 ceph-mon[33018]: 9.3 scrub starts 2024-02-05T08:17:52.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:52 smithi196 ceph-mon[33018]: 5.1b scrub starts 2024-02-05T08:17:52.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:52 smithi196 ceph-mon[33018]: 5.1b scrub ok 2024-02-05T08:17:52.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:52.855 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:52.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:52.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:52.957 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:52.993 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:52.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:52.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:53.016 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:17:53.016 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:17:53.016 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:17:53.016 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:17:53.017 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:17:53.017 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:17:53.017 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:17:53.049 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:17:53.066 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:17:53.066 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:17:53.066 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:17:53.066 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:17:53.066 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:17:53.066 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:17:53.066 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:17:53.066 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:17:53.072 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:53.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:53.135 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:53.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:53.237 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:53.271 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:53.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:53.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:53.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:53.400 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:53.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:53.501 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:53.531 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:53.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:53.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:53.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-mon[33018]: pgmap v580: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 604 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T08:17:53.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-mon[33018]: 6.1e scrub starts 2024-02-05T08:17:53.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-mon[33018]: 6.1e scrub ok 2024-02-05T08:17:53.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-mon[33018]: 9.9 scrub ok 2024-02-05T08:17:53.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-mon[33018]: 5.17 scrub starts 2024-02-05T08:17:53.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-mon[33018]: 5.17 scrub ok 2024-02-05T08:17:53.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-mon[33018]: 3.7 scrub starts 2024-02-05T08:17:53.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-mon[33018]: 3.7 scrub ok 2024-02-05T08:17:53.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-mon[33018]: 9.3 scrub ok 2024-02-05T08:17:53.623 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-mon[33018]: 4.1e scrub starts 2024-02-05T08:17:53.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-mon[33018]: 4.1e scrub ok 2024-02-05T08:17:53.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3495588136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:17:53.624 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:17:53.632 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:53.664 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:53.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:53.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[30086]: pgmap v580: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 604 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T08:17:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[30086]: 6.1e scrub starts 2024-02-05T08:17:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[30086]: 6.1e scrub ok 2024-02-05T08:17:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[30086]: 9.9 scrub ok 2024-02-05T08:17:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[30086]: 5.17 scrub starts 2024-02-05T08:17:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[30086]: 5.17 scrub ok 2024-02-05T08:17:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[30086]: 3.7 scrub starts 2024-02-05T08:17:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[30086]: 3.7 scrub ok 2024-02-05T08:17:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[30086]: 9.3 scrub ok 2024-02-05T08:17:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[30086]: 4.1e scrub starts 2024-02-05T08:17:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[30086]: 4.1e scrub ok 2024-02-05T08:17:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3495588136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:17:53.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:17:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[39278]: pgmap v580: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 604 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T08:17:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[39278]: 6.1e scrub starts 2024-02-05T08:17:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[39278]: 6.1e scrub ok 2024-02-05T08:17:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[39278]: 9.9 scrub ok 2024-02-05T08:17:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[39278]: 5.17 scrub starts 2024-02-05T08:17:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[39278]: 5.17 scrub ok 2024-02-05T08:17:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[39278]: 3.7 scrub starts 2024-02-05T08:17:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[39278]: 3.7 scrub ok 2024-02-05T08:17:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[39278]: 9.3 scrub ok 2024-02-05T08:17:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[39278]: 4.1e scrub starts 2024-02-05T08:17:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[39278]: 4.1e scrub ok 2024-02-05T08:17:53.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3495588136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:17:53.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:17:53.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:53.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:53.904 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:53.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:53.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:54.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:54.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:17:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:17:54.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:54.075 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:54.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:54.176 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:54.210 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:54.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:54.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:54.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:54.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:54.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:54.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:54.490 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:54.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:54.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:54.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:54.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:54.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:54.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:54.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:54.764 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:54.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:54.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:54.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:54 smithi134 ceph-mon[30086]: 4.2 scrub starts 2024-02-05T08:17:54.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:54 smithi134 ceph-mon[30086]: 4.2 scrub ok 2024-02-05T08:17:54.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:54 smithi134 ceph-mon[30086]: 3.c scrub starts 2024-02-05T08:17:54.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:54 smithi134 ceph-mon[30086]: 3.c scrub ok 2024-02-05T08:17:54.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:54 smithi134 ceph-mon[39278]: 4.2 scrub starts 2024-02-05T08:17:54.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:54 smithi134 ceph-mon[39278]: 4.2 scrub ok 2024-02-05T08:17:54.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:54 smithi134 ceph-mon[39278]: 3.c scrub starts 2024-02-05T08:17:54.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:54 smithi134 ceph-mon[39278]: 3.c scrub ok 2024-02-05T08:17:54.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:54 smithi196 ceph-mon[33018]: 4.2 scrub starts 2024-02-05T08:17:54.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:54 smithi196 ceph-mon[33018]: 4.2 scrub ok 2024-02-05T08:17:54.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:54 smithi196 ceph-mon[33018]: 3.c scrub starts 2024-02-05T08:17:54.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:54 smithi196 ceph-mon[33018]: 3.c scrub ok 2024-02-05T08:17:54.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:54.898 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:54.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:55.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:55.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:55.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:55.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:55.120 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:17:55.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:55.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:55.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:55.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:55.308 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:55.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:55.410 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:55.444 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:55.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:55.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:55.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:55.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:55.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:55.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:55.689 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:55.689 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[30086]: pgmap v581: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-02-05T08:17:55.689 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[30086]: 4.17 scrub starts 2024-02-05T08:17:55.689 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[30086]: 4.17 scrub ok 2024-02-05T08:17:55.689 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:17:55.689 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:17:55.689 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[30086]: 6.6 scrub starts 2024-02-05T08:17:55.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:55.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[39278]: pgmap v581: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-02-05T08:17:55.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[39278]: 4.17 scrub starts 2024-02-05T08:17:55.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[39278]: 4.17 scrub ok 2024-02-05T08:17:55.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:17:55.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:17:55.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:55 smithi134 ceph-mon[39278]: 6.6 scrub starts 2024-02-05T08:17:55.725 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:55.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:55 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:17:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:55 smithi196 ceph-mon[33018]: pgmap v581: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-02-05T08:17:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:55 smithi196 ceph-mon[33018]: 4.17 scrub starts 2024-02-05T08:17:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:55 smithi196 ceph-mon[33018]: 4.17 scrub ok 2024-02-05T08:17:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:55 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:17:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:55 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:17:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:55 smithi196 ceph-mon[33018]: 6.6 scrub starts 2024-02-05T08:17:55.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:55.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:55.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:55.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:55.868 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:17:55.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:56.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:56.011 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:56.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:56.112 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:56.167 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:56.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:56.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:56.269 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:56.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:56.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:56.408 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:56.452 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:56.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:56.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:56 smithi134 ceph-mon[30086]: 6.6 scrub ok 2024-02-05T08:17:56.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:56 smithi134 ceph-mon[39278]: 6.6 scrub ok 2024-02-05T08:17:56.554 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:56.577 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:56.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:56.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:56.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:56.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:56.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:56.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:56 smithi196 ceph-mon[33018]: 6.6 scrub ok 2024-02-05T08:17:56.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:56.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:56.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:56.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:56.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:56.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:56.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:57.082 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:57.113 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:57.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:57.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:57.214 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:57.247 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:57.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:57.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:57.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:57.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:57.386 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:57.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:57.395 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:17:57.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:57.523 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:57.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:57.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:57.624 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:17:57.656 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:57.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:57.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:17:57.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:57 smithi134 ceph-mon[30086]: pgmap v582: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-02-05T08:17:57.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:57 smithi134 ceph-mon[30086]: 4.4 scrub starts 2024-02-05T08:17:57.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:57 smithi134 ceph-mon[30086]: 4.4 scrub ok 2024-02-05T08:17:57.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:57 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2948966603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:17:57.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:57 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:17:57.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:57 smithi134 ceph-mon[39278]: pgmap v582: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-02-05T08:17:57.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:57 smithi134 ceph-mon[39278]: 4.4 scrub starts 2024-02-05T08:17:57.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:57 smithi134 ceph-mon[39278]: 4.4 scrub ok 2024-02-05T08:17:57.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:57 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2948966603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:17:57.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:57 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:17:57.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:57.795 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:57.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:57.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:57 smithi196 ceph-mon[33018]: pgmap v582: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.20k op/s 2024-02-05T08:17:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:57 smithi196 ceph-mon[33018]: 4.4 scrub starts 2024-02-05T08:17:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:57 smithi196 ceph-mon[33018]: 4.4 scrub ok 2024-02-05T08:17:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:57 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2948966603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:17:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:57 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:17:57.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:57.935 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:57.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:57.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:58.021 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:17:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:17:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:17:58.037 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:58.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:58.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:58.178 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:17:58.212 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:58.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:17:58.314 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:58.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:58.352 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:58.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:58.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:58.513 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:58.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:58.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:58.543 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:58 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:17:58.544 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:58 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:17:58.544 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:58 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:17:58.544 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:58 smithi134 ceph-mon[30086]: osdmap e285: 8 total, 8 up, 7 in 2024-02-05T08:17:58.544 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:58 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:17:58.544 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:58 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:17:58.544 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:58 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:17:58.544 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:58 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:17:58.544 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:58 smithi134 ceph-mon[39278]: osdmap e285: 8 total, 8 up, 7 in 2024-02-05T08:17:58.544 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:58 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:17:58.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:58.646 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:58.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:58.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:17:58.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:58.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:58.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:17:58.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:58 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:17:58.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:58 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:17:58.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:58 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:17:58.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:58 smithi196 ceph-mon[33018]: osdmap e285: 8 total, 8 up, 7 in 2024-02-05T08:17:58.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:58 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:17:58.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:17:58.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:58.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:58.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:17:59.043 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:59.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:59.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:59.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:59.219 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:59.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:59.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:59.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:17:59.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:59.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:17:59.457 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:17:59.500 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:59.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:17:59.504 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:17:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:17:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:17:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:17:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106678 7.e 7:7f897189:::benchmark_data_smithi134_83016_object106677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:17:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.632287+0000", 2024-02-05T08:17:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 582.86755586200002, 2024-02-05T08:17:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14517614000000001, 2024-02-05T08:17:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:17:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:17:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:17:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106678 2024-02-05T08:17:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632287+0000", 2024-02-05T08:17:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632287+0000", 2024-02-05T08:17:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632288+0000", 2024-02-05T08:17:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3e-06 2024-02-05T08:17:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632314+0000", 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5622e-05 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632326+0000", 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1901e-05 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.632333+0000", 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.08e-06 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633044+0000", 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071079200000000002 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633761+0000", 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00071729699999999999 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633976+0000", 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021524099999999999 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650870+0000", 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.016893551 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650906+0000", 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6427000000000002e-05 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777225+0000", 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.126318758 2024-02-05T08:17:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777321+0000", 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6436999999999998e-05 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777463+0000", 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000141734 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.14802.0:106682 7.e 7:7cfa27eb:::benchmark_data_smithi134_83016_object106681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:08:16.633782+0000", 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 582.86606093299997, 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14385479700000001, 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.14802", 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1257738680", 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 106682 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633782+0000", 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633782+0000", 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633783+0000", 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1090000000000001e-06 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633813+0000", 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9856999999999999e-05 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633823+0000", 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0222e-05 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.633830+0000", 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0639999999999996e-06 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642537+0000", 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0087064080000000005 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642683+0000", 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000146527 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.642858+0000", 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017454499999999999 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.650945+0000", 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0080870500000000001 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.769282+0000", 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.118337331 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777475+0000", 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0081926900000000007 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777545+0000", 2024-02-05T08:17:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0735000000000006e-05 2024-02-05T08:17:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:08:16.777637+0000", 2024-02-05T08:17:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1259000000000002e-05 2024-02-05T08:17:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:131934 8.f 8:fd4b76ad:::benchmark_data_smithi134_115836_object15274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:17:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:58.583316+0000", 2024-02-05T08:17:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 300.916526675, 2024-02-05T08:17:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15811914799999999, 2024-02-05T08:17:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:17:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:17:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:17:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 131934 2024-02-05T08:17:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583316+0000", 2024-02-05T08:17:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583316+0000", 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583318+0000", 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.593e-06 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583324+0000", 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3620000000000002e-06 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583342+0000", 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7396000000000001e-05 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583351+0000", 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7429999999999997e-06 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.589777+0000", 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0064258070000000004 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.589907+0000", 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000129913 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.590038+0000", 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013120100000000001 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.595790+0000", 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0057520330000000001 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596232+0000", 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000442075 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600111+0000", 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003878898 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.603530+0000", 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0034189889999999999 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.741275+0000", 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13774504400000001 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.741435+0000", 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000160094 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:131938 8.f 8:fc014979:::benchmark_data_smithi134_115836_object15278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:58.588880+0000", 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 300.91096282500001, 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15328541100000001, 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 131938 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588880+0000", 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588880+0000", 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588882+0000", 2024-02-05T08:17:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6500000000000001e-06 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588888+0000", 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0839999999999997e-06 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588902+0000", 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4208e-05 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588912+0000", 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7839999999999998e-06 2024-02-05T08:17:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.592812+0000", 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003899881 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.592939+0000", 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012735700000000001 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.593065+0000", 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000125659 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.599122+0000", 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0060574239999999996 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600149+0000", 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001027095 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600698+0000", 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00054859500000000003 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.741935+0000", 2024-02-05T08:17:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.141237063 2024-02-05T08:17:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.742052+0000", 2024-02-05T08:17:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000117544 2024-02-05T08:17:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.742165+0000", 2024-02-05T08:17:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011306699999999999 2024-02-05T08:17:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:131939 8.f 8:fbb4e894:::benchmark_data_smithi134_115836_object15279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:17:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:58.588930+0000", 2024-02-05T08:17:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 300.91091261399998, 2024-02-05T08:17:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15448906300000001, 2024-02-05T08:17:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:17:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:17:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:17:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 131939 2024-02-05T08:17:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588930+0000", 2024-02-05T08:17:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588930+0000", 2024-02-05T08:17:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588931+0000", 2024-02-05T08:17:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0800000000000005e-07 2024-02-05T08:17:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588933+0000", 2024-02-05T08:17:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5749999999999999e-06 2024-02-05T08:17:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588941+0000", 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9489999999999996e-06 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588947+0000", 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8749999999999997e-06 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596264+0000", 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0073165060000000004 2024-02-05T08:17:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596396+0000", 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013257000000000001 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596529+0000", 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013312400000000001 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600168+0000", 2024-02-05T08:17:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036390340000000002 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600367+0000", 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019905500000000001 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600806+0000", 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00043826 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.743237+0000", 2024-02-05T08:17:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14243134299999999 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.743303+0000", 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6328999999999995e-05 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.743419+0000", 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011593499999999999 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDPGPushReply(8.f 198/194 [PushReplyOp(8:f4899214:::benchmark_data_smithi134_115836_object113206:head)])", 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:59.890203+0000", 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 299.60964030399998, 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14734050700000001, 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890203+0000", 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890203+0000", 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890203+0000", 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9400000000000001e-07 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890205+0000", 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.627e-06 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890211+0000", 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5439999999999998e-06 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890216+0000", 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1460000000000004e-06 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:59.890262+0000", 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5782999999999999e-05 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:00.037543+0000", 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14728111299999999 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:134926 8.f 8:f72340dc:::benchmark_data_smithi134_115836_object18266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.535476+0000", 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 296.96436726100001, 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16460388500000001, 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:17:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 134926 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535476+0000", 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535476+0000", 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535477+0000", 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8160000000000001e-06 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535482+0000", 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9400000000000001e-06 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535494+0000", 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1657999999999999e-05 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535503+0000", 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7129999999999994e-06 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535674+0000", 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017146300000000001 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535806+0000", 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000131707 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:17:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535947+0000", 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014141300000000001 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.539323+0000", 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033756630000000001 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.539337+0000", 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4443e-05 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.540469+0000", 2024-02-05T08:17:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011320169999999999 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.699796+0000", 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15932692800000001 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.699972+0000", 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000175513 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.700079+0000", 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107611 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15087.0:134935 8.d e199/157 8:bc5808d6:::benchmark_data_smithi134_115836_object18275:head v 199'8633, mlcod=199'8631)", 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.544759+0000", 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 296.95508348499999, 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15565003399999999, 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.544759+0000", 2024-02-05T08:17:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.544759+0000", 2024-02-05T08:17:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.544760+0000", 2024-02-05T08:17:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.06e-07 2024-02-05T08:17:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.544764+0000", 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1629999999999998e-06 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.694991+0000", 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15022724000000001 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.694999+0000", 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2339999999999994e-06 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.695129+0000", 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012998799999999999 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.695151+0000", 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1145000000000002e-05 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.700293+0000", 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0051418979999999998 2024-02-05T08:17:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.700409+0000", 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011686 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:161420 8.f 8:f12e680b:::benchmark_data_smithi134_115836_object44760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:35.294162+0000", 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 264.20568042299999, 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14680094699999999, 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 161420 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294162+0000", 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294162+0000", 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294163+0000", 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8899999999999999e-07 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294167+0000", 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8290000000000001e-06 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294181+0000", 2024-02-05T08:17:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3961000000000001e-05 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294188+0000", 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.939e-06 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.297429+0000", 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032408910000000001 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.297540+0000", 2024-02-05T08:17:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000110961 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.297646+0000", 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010610100000000001 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.300140+0000", 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0024944670000000002 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.301421+0000", 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001281157 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.440869+0000", 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.139447667 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.440927+0000", 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7918999999999999e-05 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.440963+0000", 2024-02-05T08:17:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6365999999999997e-05 2024-02-05T08:17:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:162662 8.f 8:f41fb899:::benchmark_data_smithi134_115836_object46002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-02-05T08:17:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:36.809228+0000", 2024-02-05T08:17:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 262.69061462299999, 2024-02-05T08:17:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16428367499999999, 2024-02-05T08:17:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:17:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:17:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:17:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 162662 2024-02-05T08:17:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809228+0000", 2024-02-05T08:17:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809228+0000", 2024-02-05T08:17:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809230+0000", 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.488e-06 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809239+0000", 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2180000000000005e-06 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809248+0000", 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4040000000000001e-06 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809255+0000", 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5119999999999997e-06 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809371+0000", 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115905 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809483+0000", 2024-02-05T08:17:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000112136 2024-02-05T08:17:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:17:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809632+0000", 2024-02-05T08:17:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000149285 2024-02-05T08:17:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:17:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.815527+0000", 2024-02-05T08:17:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0058951819999999997 2024-02-05T08:17:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.819817+0000", 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042893899999999997 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973380+0000", 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15356336900000001 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973474+0000", 2024-02-05T08:17:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3459000000000001e-05 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973512+0000", 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8327000000000001e-05 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:162673 8.f 8:f0c11af9:::benchmark_data_smithi134_115836_object46013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:36.816303+0000", 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 262.683540204, 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.157358891, 2024-02-05T08:17:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 162673 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816303+0000", 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816303+0000", 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816304+0000", 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2449999999999999e-06 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816308+0000", 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2679999999999996e-06 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816318+0000", 2024-02-05T08:17:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4779999999999997e-06 2024-02-05T08:17:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816324+0000", 2024-02-05T08:17:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.725e-06 2024-02-05T08:17:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816371+0000", 2024-02-05T08:17:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6983000000000001e-05 2024-02-05T08:17:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816476+0000", 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104721 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816581+0000", 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010494500000000001 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.823858+0000", 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0072768090000000004 2024-02-05T08:17:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.823889+0000", 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1165999999999999e-05 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973555+0000", 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14966643499999999 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973629+0000", 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3936000000000005e-05 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973662+0000", 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.218e-05 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:162674 8.f 8:f8e2bae8:::benchmark_data_smithi134_115836_object46014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:36.818247+0000", 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 262.68159614899997, 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.155548574, 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:17:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:17:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:17:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 162674 2024-02-05T08:17:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818247+0000", 2024-02-05T08:17:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818247+0000", 2024-02-05T08:17:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818248+0000", 2024-02-05T08:17:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.113e-06 2024-02-05T08:17:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818251+0000", 2024-02-05T08:17:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.647e-06 2024-02-05T08:17:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818262+0000", 2024-02-05T08:17:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0519e-05 2024-02-05T08:17:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818269+0000", 2024-02-05T08:17:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3679999999999996e-06 2024-02-05T08:17:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.819987+0000", 2024-02-05T08:17:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017176610000000001 2024-02-05T08:17:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.820094+0000", 2024-02-05T08:17:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000106597 2024-02-05T08:17:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:17:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.820209+0000", 2024-02-05T08:17:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114962 2024-02-05T08:17:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:17:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.823966+0000", 2024-02-05T08:17:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0037573390000000002 2024-02-05T08:17:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.825554+0000", 2024-02-05T08:17:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001587647 2024-02-05T08:17:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973698+0000", 2024-02-05T08:17:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14814437499999999 2024-02-05T08:17:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973765+0000", 2024-02-05T08:17:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7377000000000002e-05 2024-02-05T08:17:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973795+0000", 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9969000000000001e-05 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:197385 8.f 8:f14f197d:::benchmark_data_smithi134_115836_object80725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:14:07.714003+0000", 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 231.78583970400001, 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14987962099999999, 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:17:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:17:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:17:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 197385 2024-02-05T08:17:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714003+0000", 2024-02-05T08:17:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714003+0000", 2024-02-05T08:17:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714004+0000", 2024-02-05T08:17:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.173e-06 2024-02-05T08:17:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714008+0000", 2024-02-05T08:17:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8130000000000002e-06 2024-02-05T08:17:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714019+0000", 2024-02-05T08:17:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1114999999999999e-05 2024-02-05T08:17:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714026+0000", 2024-02-05T08:17:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2690000000000002e-06 2024-02-05T08:17:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.717045+0000", 2024-02-05T08:17:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030182009999999999 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.717183+0000", 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013826499999999999 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.717300+0000", 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000116941 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.724371+0000", 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0070706450000000004 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.726733+0000", 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023622299999999999 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.863757+0000", 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13702392899999999 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.863841+0000", 2024-02-05T08:17:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4518999999999995e-05 2024-02-05T08:17:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.863883+0000", 2024-02-05T08:17:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1520999999999998e-05 2024-02-05T08:17:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:197386 8.f 8:ffb8808e:::benchmark_data_smithi134_115836_object80726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-02-05T08:17:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:14:07.715023+0000", 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 231.78482014100001, 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15136122299999999, 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 197386 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715023+0000", 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715023+0000", 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715023+0000", 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6700000000000003e-07 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715027+0000", 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7100000000000001e-06 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715038+0000", 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1262e-05 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715046+0000", 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7919999999999993e-06 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.720250+0000", 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0052037560000000004 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.721564+0000", 2024-02-05T08:17:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001314052 2024-02-05T08:17:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:17:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.721676+0000", 2024-02-05T08:17:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000112118 2024-02-05T08:17:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:17:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.726778+0000", 2024-02-05T08:17:59.585 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:59 smithi134 ceph-mon[30086]: pgmap v584: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 541 B/s rd, 0 B/s wr, 943 op/s 2024-02-05T08:17:59.585 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:59 smithi134 ceph-mon[30086]: 4.1d scrub starts 2024-02-05T08:17:59.585 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:59 smithi134 ceph-mon[30086]: 4.1d scrub ok 2024-02-05T08:17:59.585 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:59 smithi134 ceph-mon[30086]: 3.15 scrub starts 2024-02-05T08:17:59.585 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:59 smithi134 ceph-mon[30086]: 3.15 scrub ok 2024-02-05T08:17:59.585 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:17:59 smithi134 ceph-mon[30086]: 9.1 scrub starts 2024-02-05T08:17:59.585 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:59 smithi134 ceph-mon[39278]: pgmap v584: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 541 B/s rd, 0 B/s wr, 943 op/s 2024-02-05T08:17:59.585 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:59 smithi134 ceph-mon[39278]: 4.1d scrub starts 2024-02-05T08:17:59.585 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:59 smithi134 ceph-mon[39278]: 4.1d scrub ok 2024-02-05T08:17:59.586 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:59 smithi134 ceph-mon[39278]: 3.15 scrub starts 2024-02-05T08:17:59.586 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:59 smithi134 ceph-mon[39278]: 3.15 scrub ok 2024-02-05T08:17:59.586 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:17:59 smithi134 ceph-mon[39278]: 9.1 scrub starts 2024-02-05T08:17:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0051023309999999999 2024-02-05T08:17:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.728765+0000", 2024-02-05T08:17:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0019865550000000001 2024-02-05T08:17:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866283+0000", 2024-02-05T08:17:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.137518106 2024-02-05T08:17:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866340+0000", 2024-02-05T08:17:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7144e-05 2024-02-05T08:17:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866384+0000", 2024-02-05T08:17:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3630000000000001e-05 2024-02-05T08:17:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:197388 8.f 8:fbe8cc34:::benchmark_data_smithi134_115836_object80728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-02-05T08:17:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:14:07.718059+0000", 2024-02-05T08:17:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 231.78178390400001, 2024-02-05T08:17:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.148440035, 2024-02-05T08:17:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:17:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:17:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:17:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 197388 2024-02-05T08:17:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.718059+0000", 2024-02-05T08:17:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.718059+0000", 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.718060+0000", 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1739999999999999e-06 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.718071+0000", 2024-02-05T08:17:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1243999999999999e-05 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.718081+0000", 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.251e-06 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.718087+0000", 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7209999999999997e-06 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.724408+0000", 2024-02-05T08:17:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0063203469999999996 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.724521+0000", 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000112916 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.724634+0000", 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000113113 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:17:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.728684+0000", 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0040504169999999997 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.730701+0000", 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0020171120000000002 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866426+0000", 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13572458000000001 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866469+0000", 2024-02-05T08:17:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2790000000000002e-05 2024-02-05T08:17:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866499+0000", 2024-02-05T08:17:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0369999999999999e-05 2024-02-05T08:17:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.25223.0:153623 9.7 e273/239 9:e98c86db:::benchmark_data_smithi134_144349_object21883:head v 273'9475, mlcod=273'9472)", 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:17:01.828923+0000", 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 57.670920262000003, 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.041648350000000001, 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.828923+0000", 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.828923+0000", 2024-02-05T08:17:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.828924+0000", 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4200000000000004e-07 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.828931+0000", 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1269999999999997e-06 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.828941+0000", 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0688e-05 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.828947+0000", 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9340000000000003e-06 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.829000+0000", 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3035999999999998e-05 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.829018+0000", 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8159e-05 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.870552+0000", 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.041533217999999997 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.870571+0000", 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9245999999999998e-05 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.25223.0:153630 9.f e273/239 9:fdd253b8:::benchmark_data_smithi134_144349_object21890:head v 273'9403, mlcod=273'9401)", 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:17:01.829510+0000", 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 57.670332682999998, 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.041079161000000003, 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.829510+0000", 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.829510+0000", 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.829511+0000", 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2600000000000002e-07 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.829517+0000", 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8749999999999997e-06 2024-02-05T08:17:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.829529+0000", 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2483999999999999e-05 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.829537+0000", 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4259999999999997e-06 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.829583+0000", 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6566000000000001e-05 2024-02-05T08:17:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.829602+0000", 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8742000000000001e-05 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.870573+0000", 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.040971295999999997 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.870589+0000", 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6146000000000001e-05 2024-02-05T08:17:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.25223.0:153631 9.7 e273/239 9:e5982d27:::benchmark_data_smithi134_144349_object21891:head v 273'9476, mlcod=273'9472)", 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:17:01.835569+0000", 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 57.664273588, 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.035044154000000001, 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.835569+0000", 2024-02-05T08:17:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.835569+0000", 2024-02-05T08:17:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.835570+0000", 2024-02-05T08:17:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5899999999999995e-07 2024-02-05T08:17:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.835577+0000", 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7519999999999999e-06 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.835588+0000", 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1299e-05 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.835595+0000", 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6270000000000002e-06 2024-02-05T08:17:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.835637+0000", 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2039999999999997e-05 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.835654+0000", 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6985000000000001e-05 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.870591+0000", 2024-02-05T08:17:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.034937098999999999 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:01.870613+0000", 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2592999999999999e-05 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.25223.0:186705 9.f e281/239 9:f8129e40:::benchmark_data_smithi134_144349_object54965:head v 281'11420, mlcod=281'11415)", 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:17:35.333377+0000", 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 24.166465612, 2024-02-05T08:17:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.027698189000000002, 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.333377+0000", 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.333377+0000", 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.333378+0000", 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4099999999999999e-07 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.333383+0000", 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7650000000000001e-06 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.333398+0000", 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5838999999999999e-05 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.333404+0000", 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9460000000000003e-06 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.336661+0000", 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032563990000000001 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.336684+0000", 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3686e-05 2024-02-05T08:17:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.360349+0000", 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.023664803000000002 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.361075+0000", 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00072621000000000003 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.25223.0:186889 9.f e281/239 9:fd5009a0:::benchmark_data_smithi134_144349_object55149:head v 281'11435, mlcod=281'11430)", 2024-02-05T08:17:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:17:35.486832+0000", 2024-02-05T08:17:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 24.013010462, 2024-02-05T08:17:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.027032031000000002, 2024-02-05T08:17:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:17:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:17:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:17:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:17:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.486832+0000", 2024-02-05T08:17:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.486832+0000", 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.486833+0000", 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6199999999999998e-07 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.486842+0000", 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6689999999999995e-06 2024-02-05T08:17:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:17:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.486851+0000", 2024-02-05T08:17:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0559999999999994e-06 2024-02-05T08:17:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:17:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.486857+0000", 2024-02-05T08:17:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4810000000000003e-06 2024-02-05T08:17:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.498835+0000", 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.011978150999999999 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.498887+0000", 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2033000000000001e-05 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.513702+0000", 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.014814802 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:17:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:17:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:17:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:17:35.513864+0000", 2024-02-05T08:17:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016227700000000001 2024-02-05T08:17:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:17:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:17:59.616 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:17:59.616 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:59.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:59.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:17:59.755 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:17:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:17:59.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:59.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:17:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:59 smithi196 ceph-mon[33018]: pgmap v584: 148 pgs: 148 active+clean; 3.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 541 B/s rd, 0 B/s wr, 943 op/s 2024-02-05T08:17:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:59 smithi196 ceph-mon[33018]: 4.1d scrub starts 2024-02-05T08:17:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:59 smithi196 ceph-mon[33018]: 4.1d scrub ok 2024-02-05T08:17:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:59 smithi196 ceph-mon[33018]: 3.15 scrub starts 2024-02-05T08:17:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:59 smithi196 ceph-mon[33018]: 3.15 scrub ok 2024-02-05T08:17:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:17:59 smithi196 ceph-mon[33018]: 9.1 scrub starts 2024-02-05T08:17:59.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:17:59.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:17:59.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:17:59.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:00.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:00.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:00.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:00.168 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:00.204 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:00.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:00.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:00.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:00.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:00.339 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:00.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:00.440 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:00.472 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:00.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:00.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:00.609 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:00.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:00.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:00.743 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:00.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:00.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:00.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:00 smithi134 ceph-mon[30086]: 9.1 scrub ok 2024-02-05T08:18:00.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:00 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:18:00.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:00 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:18:00.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:00 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:18:00.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:00 smithi134 ceph-mon[39278]: 9.1 scrub ok 2024-02-05T08:18:00.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:00 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:18:00.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:00 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:18:00.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:00 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:18:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:00 smithi196 ceph-mon[33018]: 9.1 scrub ok 2024-02-05T08:18:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:00 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:18:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:00 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:18:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:00 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:18:00.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:00.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:00.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:00.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:00.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:01.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:01.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:01.075 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:18:01.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:01.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:01.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:01.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:01.281 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:01.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:01.383 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:01.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:01.425 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:01.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:01.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:01.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:01.561 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:01.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:01.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:01.771 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:01 smithi134 ceph-mon[30086]: pgmap v585: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T08:18:01.771 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:01 smithi134 ceph-mon[30086]: 6.8 scrub starts 2024-02-05T08:18:01.771 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:01 smithi134 ceph-mon[30086]: 6.8 scrub ok 2024-02-05T08:18:01.771 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:01 smithi134 ceph-mon[30086]: 4.c scrub starts 2024-02-05T08:18:01.771 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:01 smithi134 ceph-mon[30086]: 4.c scrub ok 2024-02-05T08:18:01.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:01 smithi134 ceph-mon[39278]: pgmap v585: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T08:18:01.771 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:01 smithi134 ceph-mon[39278]: 6.8 scrub starts 2024-02-05T08:18:01.772 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:01 smithi134 ceph-mon[39278]: 6.8 scrub ok 2024-02-05T08:18:01.772 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:01 smithi134 ceph-mon[39278]: 4.c scrub starts 2024-02-05T08:18:01.772 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:01 smithi134 ceph-mon[39278]: 4.c scrub ok 2024-02-05T08:18:01.802 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T08:18:01.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:01 smithi196 ceph-mon[33018]: pgmap v585: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T08:18:01.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:01 smithi196 ceph-mon[33018]: 6.8 scrub starts 2024-02-05T08:18:01.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:01 smithi196 ceph-mon[33018]: 6.8 scrub ok 2024-02-05T08:18:01.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:01 smithi196 ceph-mon[33018]: 4.c scrub starts 2024-02-05T08:18:01.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:01 smithi196 ceph-mon[33018]: 4.c scrub ok 2024-02-05T08:18:01.824 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:01.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:01.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:01.927 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:01.963 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:01.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:02.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:02.104 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:02.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:02.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:02.263 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:02.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:02.364 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:02.364 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:02.398 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:02.439 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:02.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:02.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:02.457 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:02.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:02.580 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:02 smithi134 ceph-mon[30086]: 6.1 scrub starts 2024-02-05T08:18:02.580 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:02 smithi134 ceph-mon[30086]: 6.1 scrub ok 2024-02-05T08:18:02.580 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:02 smithi134 ceph-mon[30086]: 5.2 scrub starts 2024-02-05T08:18:02.581 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:02 smithi134 ceph-mon[30086]: 5.2 scrub ok 2024-02-05T08:18:02.581 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:02 smithi134 ceph-mon[30086]: 4.18 scrub starts 2024-02-05T08:18:02.581 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:02 smithi134 ceph-mon[30086]: 4.18 scrub ok 2024-02-05T08:18:02.581 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:02 smithi134 ceph-mon[39278]: 6.1 scrub starts 2024-02-05T08:18:02.581 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:02 smithi134 ceph-mon[39278]: 6.1 scrub ok 2024-02-05T08:18:02.581 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:02 smithi134 ceph-mon[39278]: 5.2 scrub starts 2024-02-05T08:18:02.581 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:02 smithi134 ceph-mon[39278]: 5.2 scrub ok 2024-02-05T08:18:02.581 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:02 smithi134 ceph-mon[39278]: 4.18 scrub starts 2024-02-05T08:18:02.581 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:02 smithi134 ceph-mon[39278]: 4.18 scrub ok 2024-02-05T08:18:02.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:02.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:02.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:02.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:02.722 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:18:02.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:02.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:02.740 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:18:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:02 smithi196 ceph-mon[33018]: 6.1 scrub starts 2024-02-05T08:18:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:02 smithi196 ceph-mon[33018]: 6.1 scrub ok 2024-02-05T08:18:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:02 smithi196 ceph-mon[33018]: 5.2 scrub starts 2024-02-05T08:18:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:02 smithi196 ceph-mon[33018]: 5.2 scrub ok 2024-02-05T08:18:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:02 smithi196 ceph-mon[33018]: 4.18 scrub starts 2024-02-05T08:18:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:02 smithi196 ceph-mon[33018]: 4.18 scrub ok 2024-02-05T08:18:02.835 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:02.868 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:02.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:02.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:02.970 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:03.004 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:03.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:03.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:03.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:03.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:03.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:03.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:03.279 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:03.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:03.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:03.381 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:03.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:03.425 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:03.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:03.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[30086]: pgmap v586: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T08:18:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[30086]: 6.4 scrub starts 2024-02-05T08:18:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[30086]: 6.4 scrub ok 2024-02-05T08:18:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[30086]: 4.9 scrub starts 2024-02-05T08:18:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[30086]: 4.9 scrub ok 2024-02-05T08:18:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[30086]: 5.7 scrub starts 2024-02-05T08:18:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[30086]: 5.7 scrub ok 2024-02-05T08:18:03.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[39278]: pgmap v586: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T08:18:03.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[39278]: 6.4 scrub starts 2024-02-05T08:18:03.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[39278]: 6.4 scrub ok 2024-02-05T08:18:03.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[39278]: 4.9 scrub starts 2024-02-05T08:18:03.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[39278]: 4.9 scrub ok 2024-02-05T08:18:03.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[39278]: 5.7 scrub starts 2024-02-05T08:18:03.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:03 smithi134 ceph-mon[39278]: 5.7 scrub ok 2024-02-05T08:18:03.557 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:03.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:03.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:03.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:03.692 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:03.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:03.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:03.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:03.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:18:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:18:03.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:03 smithi196 ceph-mon[33018]: pgmap v586: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T08:18:03.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:03 smithi196 ceph-mon[33018]: 6.4 scrub starts 2024-02-05T08:18:03.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:03 smithi196 ceph-mon[33018]: 6.4 scrub ok 2024-02-05T08:18:03.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:03 smithi196 ceph-mon[33018]: 4.9 scrub starts 2024-02-05T08:18:03.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:03 smithi196 ceph-mon[33018]: 4.9 scrub ok 2024-02-05T08:18:03.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:03 smithi196 ceph-mon[33018]: 5.7 scrub starts 2024-02-05T08:18:03.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:03 smithi196 ceph-mon[33018]: 5.7 scrub ok 2024-02-05T08:18:03.829 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:03.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:03.931 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:03.966 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:03.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:04.067 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:04.109 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:04.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:04.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:04.210 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:04.246 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:04.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:04.348 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:04.380 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:04.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:04.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:04.482 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:04.511 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:04.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:04.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:04.648 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:04.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:04.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:04 smithi134 ceph-mon[30086]: from='client.25436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:18:04.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:04 smithi134 ceph-mon[30086]: 5.14 scrub starts 2024-02-05T08:18:04.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:04 smithi134 ceph-mon[30086]: 5.14 scrub ok 2024-02-05T08:18:04.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:04 smithi134 ceph-mon[30086]: 3.12 scrub starts 2024-02-05T08:18:04.746 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:04 smithi134 ceph-mon[30086]: 3.12 scrub ok 2024-02-05T08:18:04.748 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:04 smithi134 ceph-mon[39278]: from='client.25436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:18:04.748 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:04 smithi134 ceph-mon[39278]: 5.14 scrub starts 2024-02-05T08:18:04.748 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:04 smithi134 ceph-mon[39278]: 5.14 scrub ok 2024-02-05T08:18:04.748 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:04 smithi134 ceph-mon[39278]: 3.12 scrub starts 2024-02-05T08:18:04.748 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:04 smithi134 ceph-mon[39278]: 3.12 scrub ok 2024-02-05T08:18:04.750 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:04.786 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:04.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:04.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:04.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:04 smithi196 ceph-mon[33018]: from='client.25436 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:18:04.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:04 smithi196 ceph-mon[33018]: 5.14 scrub starts 2024-02-05T08:18:04.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:04 smithi196 ceph-mon[33018]: 5.14 scrub ok 2024-02-05T08:18:04.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:04 smithi196 ceph-mon[33018]: 3.12 scrub starts 2024-02-05T08:18:04.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:04 smithi196 ceph-mon[33018]: 3.12 scrub ok 2024-02-05T08:18:04.888 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:04.934 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:04.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:04.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:05.036 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:05.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:05.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:05.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:05.217 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:05.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:05.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:05.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:05.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:05.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:05.458 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:05.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:05.495 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:05.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:05.596 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:05.631 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:05.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:05.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:05.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:05.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:05.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:05.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:05.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:05.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[30086]: pgmap v587: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T08:18:05.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[30086]: 3.19 scrub starts 2024-02-05T08:18:05.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[30086]: 3.19 scrub ok 2024-02-05T08:18:05.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[30086]: 5.1a scrub starts 2024-02-05T08:18:05.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[30086]: 5.1a scrub ok 2024-02-05T08:18:05.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[30086]: 6.1f deep-scrub starts 2024-02-05T08:18:05.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[30086]: 6.1f deep-scrub ok 2024-02-05T08:18:05.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:05.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[39278]: pgmap v587: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T08:18:05.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[39278]: 3.19 scrub starts 2024-02-05T08:18:05.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[39278]: 3.19 scrub ok 2024-02-05T08:18:05.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[39278]: 5.1a scrub starts 2024-02-05T08:18:05.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[39278]: 5.1a scrub ok 2024-02-05T08:18:05.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[39278]: 6.1f deep-scrub starts 2024-02-05T08:18:05.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:05 smithi134 ceph-mon[39278]: 6.1f deep-scrub ok 2024-02-05T08:18:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:05 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:05 smithi196 ceph-mon[33018]: pgmap v587: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T08:18:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:05 smithi196 ceph-mon[33018]: 3.19 scrub starts 2024-02-05T08:18:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:05 smithi196 ceph-mon[33018]: 3.19 scrub ok 2024-02-05T08:18:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:05 smithi196 ceph-mon[33018]: 5.1a scrub starts 2024-02-05T08:18:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:05 smithi196 ceph-mon[33018]: 5.1a scrub ok 2024-02-05T08:18:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:05 smithi196 ceph-mon[33018]: 6.1f deep-scrub starts 2024-02-05T08:18:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:05 smithi196 ceph-mon[33018]: 6.1f deep-scrub ok 2024-02-05T08:18:05.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:05.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:05.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:05.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:06.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:06.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:06.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:06.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:06.174 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":586,"stamp":"2024-02-05T08:18:01.755229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3151108468,"num_objects":48272,"num_object_clones":0,"num_object_copies":144816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48272,"num_whiteouts":0,"num_read":2142,"num_read_kb":1988,"num_write":216303,"num_write_kb":8433630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":2957685,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76138,"ondisk_log_size":76138,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":12542672,"kb_used_data":9348940,"kb_used_omap":10,"kb_used_meta":3193269,"kb_avail":643673392,"statfs":{"total":671965249536,"available":659121553408,"internally_reserved":0,"allocated":9573314560,"data_stored":9571108506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":3269907562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,7,4,5,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-805830656,"num_objects":-12296,"num_object_clones":0,"num_object_copies":-36888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12296,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12296,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002922"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":487,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.392655+0000","last_change":"2024-02-05T08:17:14.644253+0000","last_active":"2024-02-05T08:17:57.392655+0000","last_peered":"2024-02-05T08:17:57.392655+0000","last_clean":"2024-02-05T08:17:57.392655+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:17:57.392655+0000","last_undegraded":"2024-02-05T08:17:57.392655+0000","last_fullsized":"2024-02-05T08:17:57.392655+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:14.644144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:14.644144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:35.696086+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":492,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.388949+0000","last_change":"2024-02-05T08:16:59.970092+0000","last_active":"2024-02-05T08:17:57.388949+0000","last_peered":"2024-02-05T08:17:57.388949+0000","last_clean":"2024-02-05T08:17:57.388949+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:17:57.388949+0000","last_undegraded":"2024-02-05T08:17:57.388949+0000","last_fullsized":"2024-02-05T08:17:57.388949+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:59.969957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:59.969957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:01.007561+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":491,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398795+0000","last_change":"2024-02-05T08:16:46.841747+0000","last_active":"2024-02-05T08:17:57.398795+0000","last_peered":"2024-02-05T08:17:57.398795+0000","last_clean":"2024-02-05T08:17:57.398795+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:17:57.398795+0000","last_undegraded":"2024-02-05T08:17:57.398795+0000","last_fullsized":"2024-02-05T08:17:57.398795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:46.841644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:46.841644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:06.877982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"236'19","reported_seq":548,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.392071+0000","last_change":"2024-02-05T08:16:03.779092+0000","last_active":"2024-02-05T08:17:57.392071+0000","last_peered":"2024-02-05T08:17:57.392071+0000","last_clean":"2024-02-05T08:17:57.392071+0000","last_became_active":"2024-02-05T08:12:29.675863+0000","last_became_peered":"2024-02-05T08:12:29.675863+0000","last_unstale":"2024-02-05T08:17:57.392071+0000","last_undegraded":"2024-02-05T08:17:57.392071+0000","last_fullsized":"2024-02-05T08:17:57.392071+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:16:03.778966+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:16:03.778966+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":474,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.406049+0000","last_change":"2024-02-05T08:15:22.548444+0000","last_active":"2024-02-05T08:17:57.406049+0000","last_peered":"2024-02-05T08:17:57.406049+0000","last_clean":"2024-02-05T08:17:57.406049+0000","last_became_active":"2024-02-05T08:12:41.722414+0000","last_became_peered":"2024-02-05T08:12:41.722414+0000","last_unstale":"2024-02-05T08:17:57.406049+0000","last_undegraded":"2024-02-05T08:17:57.406049+0000","last_fullsized":"2024-02-05T08:17:57.406049+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.548159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":495,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.392224+0000","last_change":"2024-02-05T08:16:13.845746+0000","last_active":"2024-02-05T08:17:57.392224+0000","last_peered":"2024-02-05T08:17:57.392224+0000","last_clean":"2024-02-05T08:17:57.392224+0000","last_became_active":"2024-02-05T08:12:29.676482+0000","last_became_peered":"2024-02-05T08:12:29.676482+0000","last_unstale":"2024-02-05T08:17:57.392224+0000","last_undegraded":"2024-02-05T08:17:57.392224+0000","last_fullsized":"2024-02-05T08:17:57.392224+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:13.845566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:16:13.845566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":217,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.401022+0000","last_change":"2024-02-05T08:16:57.294836+0000","last_active":"2024-02-05T08:17:57.401022+0000","last_peered":"2024-02-05T08:17:57.401022+0000","last_clean":"2024-02-05T08:17:57.401022+0000","last_became_active":"2024-02-05T08:12:41.727450+0000","last_became_peered":"2024-02-05T08:12:41.727450+0000","last_unstale":"2024-02-05T08:17:57.401022+0000","last_undegraded":"2024-02-05T08:17:57.401022+0000","last_fullsized":"2024-02-05T08:17:57.401022+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:57.294696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:16:57.294696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:18.513025+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"236'34","reported_seq":398,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.388674+0000","last_change":"2024-02-05T08:17:01.432645+0000","last_active":"2024-02-05T08:17:57.388674+0000","last_peered":"2024-02-05T08:17:57.388674+0000","last_clean":"2024-02-05T08:17:57.388674+0000","last_became_active":"2024-02-05T08:12:41.745917+0000","last_became_peered":"2024-02-05T08:12:41.745917+0000","last_unstale":"2024-02-05T08:17:57.388674+0000","last_undegraded":"2024-02-05T08:17:57.388674+0000","last_fullsized":"2024-02-05T08:17:57.388674+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:17:01.432518+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:17:01.432518+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:29.324174+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":478,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.397193+0000","last_change":"2024-02-05T08:16:14.944841+0000","last_active":"2024-02-05T08:17:57.397193+0000","last_peered":"2024-02-05T08:17:57.397193+0000","last_clean":"2024-02-05T08:17:57.397193+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:17:57.397193+0000","last_undegraded":"2024-02-05T08:17:57.397193+0000","last_fullsized":"2024-02-05T08:17:57.397193+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:14.944722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:14.944722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":491,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.397297+0000","last_change":"2024-02-05T08:17:51.448848+0000","last_active":"2024-02-05T08:17:57.397297+0000","last_peered":"2024-02-05T08:17:57.397297+0000","last_clean":"2024-02-05T08:17:57.397297+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:17:57.397297+0000","last_undegraded":"2024-02-05T08:17:57.397297+0000","last_fullsized":"2024-02-05T08:17:57.397297+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:51.448603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:17:51.448603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:18.582760+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":511,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400686+0000","last_change":"2024-02-05T08:16:11.809956+0000","last_active":"2024-02-05T08:17:57.400686+0000","last_peered":"2024-02-05T08:17:57.400686+0000","last_clean":"2024-02-05T08:17:57.400686+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:17:57.400686+0000","last_undegraded":"2024-02-05T08:17:57.400686+0000","last_fullsized":"2024-02-05T08:17:57.400686+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:11.809710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:11.809710+0000","last_clean_scrub_stamp":"2024-02-05T08:16:11.809710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"263'19","reported_seq":538,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.392849+0000","last_change":"2024-02-05T08:16:07.826180+0000","last_active":"2024-02-05T08:17:57.392849+0000","last_peered":"2024-02-05T08:17:57.392849+0000","last_clean":"2024-02-05T08:17:57.392849+0000","last_became_active":"2024-02-05T08:12:41.731452+0000","last_became_peered":"2024-02-05T08:12:41.731452+0000","last_unstale":"2024-02-05T08:17:57.392849+0000","last_undegraded":"2024-02-05T08:17:57.392849+0000","last_fullsized":"2024-02-05T08:17:57.392849+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:16:07.826018+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":485,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.397322+0000","last_change":"2024-02-05T08:15:56.898965+0000","last_active":"2024-02-05T08:17:57.397322+0000","last_peered":"2024-02-05T08:17:57.397322+0000","last_clean":"2024-02-05T08:17:57.397322+0000","last_became_active":"2024-02-05T08:12:41.721611+0000","last_became_peered":"2024-02-05T08:12:41.721611+0000","last_unstale":"2024-02-05T08:17:57.397322+0000","last_undegraded":"2024-02-05T08:17:57.397322+0000","last_fullsized":"2024-02-05T08:17:57.397322+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:56.898547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:56.898547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":484,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400151+0000","last_change":"2024-02-05T08:15:18.440781+0000","last_active":"2024-02-05T08:17:57.400151+0000","last_peered":"2024-02-05T08:17:57.400151+0000","last_clean":"2024-02-05T08:17:57.400151+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:17:57.400151+0000","last_undegraded":"2024-02-05T08:17:57.400151+0000","last_fullsized":"2024-02-05T08:17:57.400151+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:18.440664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:18.440664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":499,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.396841+0000","last_change":"2024-02-05T08:17:08.551310+0000","last_active":"2024-02-05T08:17:57.396841+0000","last_peered":"2024-02-05T08:17:57.396841+0000","last_clean":"2024-02-05T08:17:57.396841+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:17:57.396841+0000","last_undegraded":"2024-02-05T08:17:57.396841+0000","last_fullsized":"2024-02-05T08:17:57.396841+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:08.551190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:17:08.551190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:22.015564+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":512,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.393125+0000","last_change":"2024-02-05T08:16:05.810845+0000","last_active":"2024-02-05T08:17:57.393125+0000","last_peered":"2024-02-05T08:17:57.393125+0000","last_clean":"2024-02-05T08:17:57.393125+0000","last_became_active":"2024-02-05T08:12:41.730485+0000","last_became_peered":"2024-02-05T08:12:41.730485+0000","last_unstale":"2024-02-05T08:17:57.393125+0000","last_undegraded":"2024-02-05T08:17:57.393125+0000","last_fullsized":"2024-02-05T08:17:57.393125+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-02-05T08:16:05.810713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:05.810713+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":500,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.408532+0000","last_change":"2024-02-05T08:17:51.825665+0000","last_active":"2024-02-05T08:17:57.408532+0000","last_peered":"2024-02-05T08:17:57.408532+0000","last_clean":"2024-02-05T08:17:57.408532+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:17:57.408532+0000","last_undegraded":"2024-02-05T08:17:57.408532+0000","last_fullsized":"2024-02-05T08:17:57.408532+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:51.825345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:17:51.825345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:20.235472+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":484,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.391713+0000","last_change":"2024-02-05T08:16:53.947563+0000","last_active":"2024-02-05T08:17:57.391713+0000","last_peered":"2024-02-05T08:17:57.391713+0000","last_clean":"2024-02-05T08:17:57.391713+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:17:57.391713+0000","last_undegraded":"2024-02-05T08:17:57.391713+0000","last_fullsized":"2024-02-05T08:17:57.391713+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:53.947466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:16:53.947466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:07.435429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":482,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398663+0000","last_change":"2024-02-05T08:15:22.341597+0000","last_active":"2024-02-05T08:17:57.398663+0000","last_peered":"2024-02-05T08:17:57.398663+0000","last_clean":"2024-02-05T08:17:57.398663+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:17:57.398663+0000","last_undegraded":"2024-02-05T08:17:57.398663+0000","last_fullsized":"2024-02-05T08:17:57.398663+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.341491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.341491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"236'34","reported_seq":583,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.389727+0000","last_change":"2024-02-05T08:17:17.405954+0000","last_active":"2024-02-05T08:17:57.389727+0000","last_peered":"2024-02-05T08:17:57.389727+0000","last_clean":"2024-02-05T08:17:57.389727+0000","last_became_active":"2024-02-05T08:12:41.727027+0000","last_became_peered":"2024-02-05T08:12:41.727027+0000","last_unstale":"2024-02-05T08:17:57.389727+0000","last_undegraded":"2024-02-05T08:17:57.389727+0000","last_fullsized":"2024-02-05T08:17:57.389727+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:17:17.405818+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:17:17.405818+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:26.313166+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":480,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.401318+0000","last_change":"2024-02-05T08:16:02.397422+0000","last_active":"2024-02-05T08:17:57.401318+0000","last_peered":"2024-02-05T08:17:57.401318+0000","last_clean":"2024-02-05T08:17:57.401318+0000","last_became_active":"2024-02-05T08:12:41.713564+0000","last_became_peered":"2024-02-05T08:12:41.713564+0000","last_unstale":"2024-02-05T08:17:57.401318+0000","last_undegraded":"2024-02-05T08:17:57.401318+0000","last_fullsized":"2024-02-05T08:17:57.401318+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:02.396921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:02.396921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":497,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.393097+0000","last_change":"2024-02-05T08:16:46.786563+0000","last_active":"2024-02-05T08:17:57.393097+0000","last_peered":"2024-02-05T08:17:57.393097+0000","last_clean":"2024-02-05T08:17:57.393097+0000","last_became_active":"2024-02-05T08:12:41.735240+0000","last_became_peered":"2024-02-05T08:12:41.735240+0000","last_unstale":"2024-02-05T08:17:57.393097+0000","last_undegraded":"2024-02-05T08:17:57.393097+0000","last_fullsized":"2024-02-05T08:17:57.393097+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:46.786289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:16:46.786289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":491,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.392871+0000","last_change":"2024-02-05T08:17:07.717997+0000","last_active":"2024-02-05T08:17:57.392871+0000","last_peered":"2024-02-05T08:17:57.392871+0000","last_clean":"2024-02-05T08:17:57.392871+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:17:57.392871+0000","last_undegraded":"2024-02-05T08:17:57.392871+0000","last_fullsized":"2024-02-05T08:17:57.392871+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:17:07.717888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:14.855138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"236'23","reported_seq":319,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.012903+0000","last_change":"2024-02-05T08:17:58.012952+0000","last_active":"2024-02-05T08:17:58.012903+0000","last_peered":"2024-02-05T08:17:58.012903+0000","last_clean":"2024-02-05T08:17:58.012903+0000","last_became_active":"2024-02-05T08:12:41.716794+0000","last_became_peered":"2024-02-05T08:12:41.716794+0000","last_unstale":"2024-02-05T08:17:58.012903+0000","last_undegraded":"2024-02-05T08:17:58.012903+0000","last_fullsized":"2024-02-05T08:17:58.012903+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'23","last_scrub_stamp":"2024-02-05T08:17:58.012827+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:17:58.012827+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:14.876020+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":498,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400822+0000","last_change":"2024-02-05T08:17:10.980274+0000","last_active":"2024-02-05T08:17:57.400822+0000","last_peered":"2024-02-05T08:17:57.400822+0000","last_clean":"2024-02-05T08:17:57.400822+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:17:57.400822+0000","last_undegraded":"2024-02-05T08:17:57.400822+0000","last_fullsized":"2024-02-05T08:17:57.400822+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:17:10.980170+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:17:10.980170+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:15.656865+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":299,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.401600+0000","last_change":"2024-02-05T08:17:11.092748+0000","last_active":"2024-02-05T08:17:57.401600+0000","last_peered":"2024-02-05T08:17:57.401600+0000","last_clean":"2024-02-05T08:17:57.401600+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:17:57.401600+0000","last_undegraded":"2024-02-05T08:17:57.401600+0000","last_fullsized":"2024-02-05T08:17:57.401600+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:17:11.092613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:11.092613+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:15.479387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":503,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.388866+0000","last_change":"2024-02-05T08:16:49.034265+0000","last_active":"2024-02-05T08:17:57.388866+0000","last_peered":"2024-02-05T08:17:57.388866+0000","last_clean":"2024-02-05T08:17:57.388866+0000","last_became_active":"2024-02-05T08:12:41.719010+0000","last_became_peered":"2024-02-05T08:12:41.719010+0000","last_unstale":"2024-02-05T08:17:57.388866+0000","last_undegraded":"2024-02-05T08:17:57.388866+0000","last_fullsized":"2024-02-05T08:17:57.388866+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:16:49.034140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:49.034140+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"236'20","reported_seq":571,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398704+0000","last_change":"2024-02-05T08:17:52.347679+0000","last_active":"2024-02-05T08:17:57.398704+0000","last_peered":"2024-02-05T08:17:57.398704+0000","last_clean":"2024-02-05T08:17:57.398704+0000","last_became_active":"2024-02-05T08:12:41.757718+0000","last_became_peered":"2024-02-05T08:12:41.757718+0000","last_unstale":"2024-02-05T08:17:57.398704+0000","last_undegraded":"2024-02-05T08:17:57.398704+0000","last_fullsized":"2024-02-05T08:17:57.398704+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:17:52.347554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:52.347554+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:15.747040+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":488,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.391834+0000","last_change":"2024-02-05T08:17:17.079508+0000","last_active":"2024-02-05T08:17:57.391834+0000","last_peered":"2024-02-05T08:17:57.391834+0000","last_clean":"2024-02-05T08:17:57.391834+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:17:57.391834+0000","last_undegraded":"2024-02-05T08:17:57.391834+0000","last_fullsized":"2024-02-05T08:17:57.391834+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:17.079382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:17.079382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:37.501898+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":483,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398839+0000","last_change":"2024-02-05T08:16:57.872130+0000","last_active":"2024-02-05T08:17:57.398839+0000","last_peered":"2024-02-05T08:17:57.398839+0000","last_clean":"2024-02-05T08:17:57.398839+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:17:57.398839+0000","last_undegraded":"2024-02-05T08:17:57.398839+0000","last_fullsized":"2024-02-05T08:17:57.398839+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:57.871993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:32.231861+0000","last_clean_scrub_stamp":"2024-02-05T08:16:57.871993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:17.033329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":211,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.397118+0000","last_change":"2024-02-05T08:16:52.804014+0000","last_active":"2024-02-05T08:17:57.397118+0000","last_peered":"2024-02-05T08:17:57.397118+0000","last_clean":"2024-02-05T08:17:57.397118+0000","last_became_active":"2024-02-05T08:12:41.727406+0000","last_became_peered":"2024-02-05T08:12:41.727406+0000","last_unstale":"2024-02-05T08:17:57.397118+0000","last_undegraded":"2024-02-05T08:17:57.397118+0000","last_fullsized":"2024-02-05T08:17:57.397118+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:52.803901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:16:52.803901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:59.825509+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"236'19","reported_seq":245,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.401403+0000","last_change":"2024-02-05T08:16:08.369341+0000","last_active":"2024-02-05T08:17:57.401403+0000","last_peered":"2024-02-05T08:17:57.401403+0000","last_clean":"2024-02-05T08:17:57.401403+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:17:57.401403+0000","last_undegraded":"2024-02-05T08:17:57.401403+0000","last_fullsized":"2024-02-05T08:17:57.401403+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:16:08.369158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:08.369158+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":480,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400302+0000","last_change":"2024-02-05T08:16:52.009766+0000","last_active":"2024-02-05T08:17:57.400302+0000","last_peered":"2024-02-05T08:17:57.400302+0000","last_clean":"2024-02-05T08:17:57.400302+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:17:57.400302+0000","last_undegraded":"2024-02-05T08:17:57.400302+0000","last_fullsized":"2024-02-05T08:17:57.400302+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:52.009657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:52.009657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:18.660392+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":221,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400140+0000","last_change":"2024-02-05T08:16:55.299725+0000","last_active":"2024-02-05T08:17:57.400140+0000","last_peered":"2024-02-05T08:17:57.400140+0000","last_clean":"2024-02-05T08:17:57.400140+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:17:57.400140+0000","last_undegraded":"2024-02-05T08:17:57.400140+0000","last_fullsized":"2024-02-05T08:17:57.400140+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:55.299617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:16:55.299617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:18.988856+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":496,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398996+0000","last_change":"2024-02-05T08:17:48.387440+0000","last_active":"2024-02-05T08:17:57.398996+0000","last_peered":"2024-02-05T08:17:57.398996+0000","last_clean":"2024-02-05T08:17:57.398996+0000","last_became_active":"2024-02-05T08:12:41.736991+0000","last_became_peered":"2024-02-05T08:12:41.736991+0000","last_unstale":"2024-02-05T08:17:57.398996+0000","last_undegraded":"2024-02-05T08:17:57.398996+0000","last_fullsized":"2024-02-05T08:17:57.398996+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:48.387323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:48.387323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:51.635436+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":249,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.388680+0000","last_change":"2024-02-05T08:17:10.054692+0000","last_active":"2024-02-05T08:17:57.388680+0000","last_peered":"2024-02-05T08:17:57.388680+0000","last_clean":"2024-02-05T08:17:57.388680+0000","last_became_active":"2024-02-05T08:12:41.717459+0000","last_became_peered":"2024-02-05T08:12:41.717459+0000","last_unstale":"2024-02-05T08:17:57.388680+0000","last_undegraded":"2024-02-05T08:17:57.388680+0000","last_fullsized":"2024-02-05T08:17:57.388680+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:17:10.054526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:10.054526+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:34.549622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":473,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398278+0000","last_change":"2024-02-05T08:16:03.937874+0000","last_active":"2024-02-05T08:17:57.398278+0000","last_peered":"2024-02-05T08:17:57.398278+0000","last_clean":"2024-02-05T08:17:57.398278+0000","last_became_active":"2024-02-05T08:12:41.737803+0000","last_became_peered":"2024-02-05T08:12:41.737803+0000","last_unstale":"2024-02-05T08:17:57.398278+0000","last_undegraded":"2024-02-05T08:17:57.398278+0000","last_fullsized":"2024-02-05T08:17:57.398278+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:03.937759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:03.937759+0000","last_clean_scrub_stamp":"2024-02-05T08:16:03.937759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":504,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.401387+0000","last_change":"2024-02-05T08:16:13.076225+0000","last_active":"2024-02-05T08:17:57.401387+0000","last_peered":"2024-02-05T08:17:57.401387+0000","last_clean":"2024-02-05T08:17:57.401387+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:17:57.401387+0000","last_undegraded":"2024-02-05T08:17:57.401387+0000","last_fullsized":"2024-02-05T08:17:57.401387+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:13.076114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:16:13.076114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":499,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.396606+0000","last_change":"2024-02-05T08:16:07.831634+0000","last_active":"2024-02-05T08:17:57.396606+0000","last_peered":"2024-02-05T08:17:57.396606+0000","last_clean":"2024-02-05T08:17:57.396606+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:17:57.396606+0000","last_undegraded":"2024-02-05T08:17:57.396606+0000","last_fullsized":"2024-02-05T08:17:57.396606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:16:07.831525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"236'24","reported_seq":448,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.402318+0000","last_change":"2024-02-05T08:17:11.945979+0000","last_active":"2024-02-05T08:17:57.402318+0000","last_peered":"2024-02-05T08:17:57.402318+0000","last_clean":"2024-02-05T08:17:57.402318+0000","last_became_active":"2024-02-05T08:12:41.718651+0000","last_became_peered":"2024-02-05T08:12:41.718651+0000","last_unstale":"2024-02-05T08:17:57.402318+0000","last_undegraded":"2024-02-05T08:17:57.402318+0000","last_fullsized":"2024-02-05T08:17:57.402318+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:17:11.945868+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:17:11.945868+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:31.340745+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":478,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.396760+0000","last_change":"2024-02-05T08:17:09.009620+0000","last_active":"2024-02-05T08:17:57.396760+0000","last_peered":"2024-02-05T08:17:57.396760+0000","last_clean":"2024-02-05T08:17:57.396760+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:17:57.396760+0000","last_undegraded":"2024-02-05T08:17:57.396760+0000","last_fullsized":"2024-02-05T08:17:57.396760+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:17:09.009398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:13.651051+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":212,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398347+0000","last_change":"2024-02-05T08:16:12.850966+0000","last_active":"2024-02-05T08:17:57.398347+0000","last_peered":"2024-02-05T08:17:57.398347+0000","last_clean":"2024-02-05T08:17:57.398347+0000","last_became_active":"2024-02-05T08:12:41.757901+0000","last_became_peered":"2024-02-05T08:12:41.757901+0000","last_unstale":"2024-02-05T08:17:57.398347+0000","last_undegraded":"2024-02-05T08:17:57.398347+0000","last_fullsized":"2024-02-05T08:17:57.398347+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:12.850323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:12.850323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":506,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.070150+0000","last_change":"2024-02-05T08:17:58.070182+0000","last_active":"2024-02-05T08:17:58.070150+0000","last_peered":"2024-02-05T08:17:58.070150+0000","last_clean":"2024-02-05T08:17:58.070150+0000","last_became_active":"2024-02-05T08:12:41.734072+0000","last_became_peered":"2024-02-05T08:12:41.734072+0000","last_unstale":"2024-02-05T08:17:58.070150+0000","last_undegraded":"2024-02-05T08:17:58.070150+0000","last_fullsized":"2024-02-05T08:17:58.070150+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:58.070086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:58.070086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:21.992767+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"263'19","reported_seq":159,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.396272+0000","last_change":"2024-02-05T08:16:09.806635+0000","last_active":"2024-02-05T08:17:57.396272+0000","last_peered":"2024-02-05T08:17:57.396272+0000","last_clean":"2024-02-05T08:17:57.396272+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:17:57.396272+0000","last_undegraded":"2024-02-05T08:17:57.396272+0000","last_fullsized":"2024-02-05T08:17:57.396272+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:16:09.806380+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:09.806380+0000","last_clean_scrub_stamp":"2024-02-05T08:16:09.806380+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":203,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.397386+0000","last_change":"2024-02-05T08:15:54.851205+0000","last_active":"2024-02-05T08:17:57.397386+0000","last_peered":"2024-02-05T08:17:57.397386+0000","last_clean":"2024-02-05T08:17:57.397386+0000","last_became_active":"2024-02-05T08:12:41.737246+0000","last_became_peered":"2024-02-05T08:12:41.737246+0000","last_unstale":"2024-02-05T08:17:57.397386+0000","last_undegraded":"2024-02-05T08:17:57.397386+0000","last_fullsized":"2024-02-05T08:17:57.397386+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:54.850842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:15:54.850842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":498,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.391870+0000","last_change":"2024-02-05T08:17:47.016140+0000","last_active":"2024-02-05T08:17:57.391870+0000","last_peered":"2024-02-05T08:17:57.391870+0000","last_clean":"2024-02-05T08:17:57.391870+0000","last_became_active":"2024-02-05T08:12:41.717285+0000","last_became_peered":"2024-02-05T08:12:41.717285+0000","last_unstale":"2024-02-05T08:17:57.391870+0000","last_undegraded":"2024-02-05T08:17:57.391870+0000","last_fullsized":"2024-02-05T08:17:57.391870+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:47.016029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:47.016029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:03.710554+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":511,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.392880+0000","last_change":"2024-02-05T08:16:06.796972+0000","last_active":"2024-02-05T08:17:57.392880+0000","last_peered":"2024-02-05T08:17:57.392880+0000","last_clean":"2024-02-05T08:17:57.392880+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:17:57.392880+0000","last_undegraded":"2024-02-05T08:17:57.392880+0000","last_fullsized":"2024-02-05T08:17:57.392880+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:06.796588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:16:06.796588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"249'24","reported_seq":567,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.393217+0000","last_change":"2024-02-05T08:17:11.725677+0000","last_active":"2024-02-05T08:17:57.393217+0000","last_peered":"2024-02-05T08:17:57.393217+0000","last_clean":"2024-02-05T08:17:57.393217+0000","last_became_active":"2024-02-05T08:12:29.675657+0000","last_became_peered":"2024-02-05T08:12:29.675657+0000","last_unstale":"2024-02-05T08:17:57.393217+0000","last_undegraded":"2024-02-05T08:17:57.393217+0000","last_fullsized":"2024-02-05T08:17:57.393217+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:17:11.725554+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:17:11.725554+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:22.106197+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":476,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398559+0000","last_change":"2024-02-05T08:15:25.472795+0000","last_active":"2024-02-05T08:17:57.398559+0000","last_peered":"2024-02-05T08:17:57.398559+0000","last_clean":"2024-02-05T08:17:57.398559+0000","last_became_active":"2024-02-05T08:12:41.736845+0000","last_became_peered":"2024-02-05T08:12:41.736845+0000","last_unstale":"2024-02-05T08:17:57.398559+0000","last_undegraded":"2024-02-05T08:17:57.398559+0000","last_fullsized":"2024-02-05T08:17:57.398559+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.472655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.472655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":659,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400772+0000","last_change":"2024-02-05T08:17:00.560018+0000","last_active":"2024-02-05T08:17:57.400772+0000","last_peered":"2024-02-05T08:17:57.400772+0000","last_clean":"2024-02-05T08:17:57.400772+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:17:57.400772+0000","last_undegraded":"2024-02-05T08:17:57.400772+0000","last_fullsized":"2024-02-05T08:17:57.400772+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:17:00.559889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:00.559889+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:21.370792+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":500,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.406036+0000","last_change":"2024-02-05T08:16:58.934204+0000","last_active":"2024-02-05T08:17:57.406036+0000","last_peered":"2024-02-05T08:17:57.406036+0000","last_clean":"2024-02-05T08:17:57.406036+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:17:57.406036+0000","last_undegraded":"2024-02-05T08:17:57.406036+0000","last_fullsized":"2024-02-05T08:17:57.406036+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:58.934100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:58.934100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:16.320790+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":396,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.390278+0000","last_change":"2024-02-05T08:17:17.993610+0000","last_active":"2024-02-05T08:17:57.390278+0000","last_peered":"2024-02-05T08:17:57.390278+0000","last_clean":"2024-02-05T08:17:57.390278+0000","last_became_active":"2024-02-05T08:12:41.726861+0000","last_became_peered":"2024-02-05T08:12:41.726861+0000","last_unstale":"2024-02-05T08:17:57.390278+0000","last_undegraded":"2024-02-05T08:17:57.390278+0000","last_fullsized":"2024-02-05T08:17:57.390278+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-02-05T08:17:17.993472+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:17:17.993472+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:35.927283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":482,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.393566+0000","last_change":"2024-02-05T08:17:01.898247+0000","last_active":"2024-02-05T08:17:57.393566+0000","last_peered":"2024-02-05T08:17:57.393566+0000","last_clean":"2024-02-05T08:17:57.393566+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:17:57.393566+0000","last_undegraded":"2024-02-05T08:17:57.393566+0000","last_fullsized":"2024-02-05T08:17:57.393566+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:01.897910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:01.897910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:26.422119+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"194'9","reported_seq":305,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400543+0000","last_change":"2024-02-05T08:15:27.464031+0000","last_active":"2024-02-05T08:17:57.400543+0000","last_peered":"2024-02-05T08:17:57.400543+0000","last_clean":"2024-02-05T08:17:57.400543+0000","last_became_active":"2024-02-05T08:12:41.728687+0000","last_became_peered":"2024-02-05T08:12:41.728687+0000","last_unstale":"2024-02-05T08:17:57.400543+0000","last_undegraded":"2024-02-05T08:17:57.400543+0000","last_fullsized":"2024-02-05T08:17:57.400543+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'9","last_scrub_stamp":"2024-02-05T08:15:27.463918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:15:27.463918+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":488,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.396975+0000","last_change":"2024-02-05T08:15:23.494414+0000","last_active":"2024-02-05T08:17:57.396975+0000","last_peered":"2024-02-05T08:17:57.396975+0000","last_clean":"2024-02-05T08:17:57.396975+0000","last_became_active":"2024-02-05T08:12:41.723383+0000","last_became_peered":"2024-02-05T08:12:41.723383+0000","last_unstale":"2024-02-05T08:17:57.396975+0000","last_undegraded":"2024-02-05T08:17:57.396975+0000","last_fullsized":"2024-02-05T08:17:57.396975+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:23.494299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:15:23.494299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":536,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400996+0000","last_change":"2024-02-05T08:16:53.643233+0000","last_active":"2024-02-05T08:17:57.400996+0000","last_peered":"2024-02-05T08:17:57.400996+0000","last_clean":"2024-02-05T08:17:57.400996+0000","last_became_active":"2024-02-05T08:12:41.723526+0000","last_became_peered":"2024-02-05T08:12:41.723526+0000","last_unstale":"2024-02-05T08:17:57.400996+0000","last_undegraded":"2024-02-05T08:17:57.400996+0000","last_fullsized":"2024-02-05T08:17:57.400996+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:16:53.643108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:53.643108+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:59.084301+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":495,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.390034+0000","last_change":"2024-02-05T08:17:05.439954+0000","last_active":"2024-02-05T08:17:57.390034+0000","last_peered":"2024-02-05T08:17:57.390034+0000","last_clean":"2024-02-05T08:17:57.390034+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:17:57.390034+0000","last_undegraded":"2024-02-05T08:17:57.390034+0000","last_fullsized":"2024-02-05T08:17:57.390034+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:05.439752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:05.439752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:08.725804+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":221,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.389646+0000","last_change":"2024-02-05T08:17:51.982654+0000","last_active":"2024-02-05T08:17:57.389646+0000","last_peered":"2024-02-05T08:17:57.389646+0000","last_clean":"2024-02-05T08:17:57.389646+0000","last_became_active":"2024-02-05T08:12:41.721852+0000","last_became_peered":"2024-02-05T08:12:41.721852+0000","last_unstale":"2024-02-05T08:17:57.389646+0000","last_undegraded":"2024-02-05T08:17:57.389646+0000","last_fullsized":"2024-02-05T08:17:57.389646+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:51.982489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:51.982489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:03.172111+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":504,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.402568+0000","last_change":"2024-02-05T08:15:25.470332+0000","last_active":"2024-02-05T08:17:57.402568+0000","last_peered":"2024-02-05T08:17:57.402568+0000","last_clean":"2024-02-05T08:17:57.402568+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:17:57.402568+0000","last_undegraded":"2024-02-05T08:17:57.402568+0000","last_fullsized":"2024-02-05T08:17:57.402568+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.470117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"278'19","reported_seq":531,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398753+0000","last_change":"2024-02-05T08:17:11.006500+0000","last_active":"2024-02-05T08:17:57.398753+0000","last_peered":"2024-02-05T08:17:57.398753+0000","last_clean":"2024-02-05T08:17:57.398753+0000","last_became_active":"2024-02-05T08:12:41.736414+0000","last_became_peered":"2024-02-05T08:12:41.736414+0000","last_unstale":"2024-02-05T08:17:57.398753+0000","last_undegraded":"2024-02-05T08:17:57.398753+0000","last_fullsized":"2024-02-05T08:17:57.398753+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:17:11.006333+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:25.749896+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":500,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400525+0000","last_change":"2024-02-05T08:16:54.078592+0000","last_active":"2024-02-05T08:17:57.400525+0000","last_peered":"2024-02-05T08:17:57.400525+0000","last_clean":"2024-02-05T08:17:57.400525+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:17:57.400525+0000","last_undegraded":"2024-02-05T08:17:57.400525+0000","last_fullsized":"2024-02-05T08:17:57.400525+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:54.078361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:54.078361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:07.041615+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":489,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.396840+0000","last_change":"2024-02-05T08:15:55.898563+0000","last_active":"2024-02-05T08:17:57.396840+0000","last_peered":"2024-02-05T08:17:57.396840+0000","last_clean":"2024-02-05T08:17:57.396840+0000","last_became_active":"2024-02-05T08:12:41.721486+0000","last_became_peered":"2024-02-05T08:12:41.721486+0000","last_unstale":"2024-02-05T08:17:57.396840+0000","last_undegraded":"2024-02-05T08:17:57.396840+0000","last_fullsized":"2024-02-05T08:17:57.396840+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:55.898292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:15:55.898292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":229,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.399078+0000","last_change":"2024-02-05T08:17:49.211808+0000","last_active":"2024-02-05T08:17:57.399078+0000","last_peered":"2024-02-05T08:17:57.399078+0000","last_clean":"2024-02-05T08:17:57.399078+0000","last_became_active":"2024-02-05T08:12:29.684444+0000","last_became_peered":"2024-02-05T08:12:29.684444+0000","last_unstale":"2024-02-05T08:17:57.399078+0000","last_undegraded":"2024-02-05T08:17:57.399078+0000","last_fullsized":"2024-02-05T08:17:57.399078+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:49.211690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:49.211690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:10.018813+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'15","reported_seq":362,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400870+0000","last_change":"2024-02-05T08:17:54.101977+0000","last_active":"2024-02-05T08:17:57.400870+0000","last_peered":"2024-02-05T08:17:57.400870+0000","last_clean":"2024-02-05T08:17:57.400870+0000","last_became_active":"2024-02-05T08:12:41.732369+0000","last_became_peered":"2024-02-05T08:12:41.732369+0000","last_unstale":"2024-02-05T08:17:57.400870+0000","last_undegraded":"2024-02-05T08:17:57.400870+0000","last_fullsized":"2024-02-05T08:17:57.400870+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:17:54.101749+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.101749+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:13.347735+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"285'17049","reported_seq":17344,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.736476+0000","last_change":"2024-02-05T08:17:47.872971+0000","last_active":"2024-02-05T08:17:58.736476+0000","last_peered":"2024-02-05T08:17:58.736476+0000","last_clean":"2024-02-05T08:17:58.736476+0000","last_became_active":"2024-02-05T08:14:57.766609+0000","last_became_peered":"2024-02-05T08:14:57.766609+0000","last_unstale":"2024-02-05T08:17:58.736476+0000","last_undegraded":"2024-02-05T08:17:58.736476+0000","last_fullsized":"2024-02-05T08:17:58.736476+0000","mapping_epoch":239,"log_start":"279'11802","ondisk_log_start":"279'11802","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'16197","last_scrub_stamp":"2024-02-05T08:17:47.872744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:47.872744+0000","objects_scrubbed":4015,"log_size":5247,"log_dups_size":0,"ondisk_log_size":5247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:12.341039+0000","scrub_duration":1552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204079104,"num_objects":3114,"num_object_clones":0,"num_object_copies":9342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13602,"num_write_kb":534912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":500,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.396792+0000","last_change":"2024-02-05T08:16:50.673897+0000","last_active":"2024-02-05T08:17:57.396792+0000","last_peered":"2024-02-05T08:17:57.396792+0000","last_clean":"2024-02-05T08:17:57.396792+0000","last_became_active":"2024-02-05T08:12:29.679849+0000","last_became_peered":"2024-02-05T08:12:29.679849+0000","last_unstale":"2024-02-05T08:17:57.396792+0000","last_undegraded":"2024-02-05T08:17:57.396792+0000","last_fullsized":"2024-02-05T08:17:57.396792+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:50.673646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:50.673646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":512,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400774+0000","last_change":"2024-02-05T08:17:00.989869+0000","last_active":"2024-02-05T08:17:57.400774+0000","last_peered":"2024-02-05T08:17:57.400774+0000","last_clean":"2024-02-05T08:17:57.400774+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:17:57.400774+0000","last_undegraded":"2024-02-05T08:17:57.400774+0000","last_fullsized":"2024-02-05T08:17:57.400774+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:17:00.989762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:00.989762+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:14.219383+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"262'34","reported_seq":582,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398660+0000","last_change":"2024-02-05T08:16:06.849357+0000","last_active":"2024-02-05T08:17:57.398660+0000","last_peered":"2024-02-05T08:17:57.398660+0000","last_clean":"2024-02-05T08:17:57.398660+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:17:57.398660+0000","last_undegraded":"2024-02-05T08:17:57.398660+0000","last_fullsized":"2024-02-05T08:17:57.398660+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'34","last_scrub_stamp":"2024-02-05T08:16:06.848964+0000","last_deep_scrub":"262'34","last_deep_scrub_stamp":"2024-02-05T08:16:06.848964+0000","last_clean_scrub_stamp":"2024-02-05T08:16:06.848964+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":682,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.402423+0000","last_change":"2024-02-05T08:17:09.961319+0000","last_active":"2024-02-05T08:17:57.402423+0000","last_peered":"2024-02-05T08:17:57.402423+0000","last_clean":"2024-02-05T08:17:57.402423+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:17:57.402423+0000","last_undegraded":"2024-02-05T08:17:57.402423+0000","last_fullsized":"2024-02-05T08:17:57.402423+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:17:09.961036+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:17:09.961036+0000","last_clean_scrub_stamp":"2024-02-05T08:17:09.961036+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:28.075138+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"285'13653","reported_seq":13836,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.347087+0000","last_change":"2024-02-05T08:17:00.838377+0000","last_active":"2024-02-05T08:17:58.347087+0000","last_peered":"2024-02-05T08:17:58.347087+0000","last_clean":"2024-02-05T08:17:58.347087+0000","last_became_active":"2024-02-05T08:14:57.758070+0000","last_became_peered":"2024-02-05T08:14:57.758070+0000","last_unstale":"2024-02-05T08:17:58.347087+0000","last_undegraded":"2024-02-05T08:17:58.347087+0000","last_fullsized":"2024-02-05T08:17:58.347087+0000","mapping_epoch":239,"log_start":"272'9500","ondisk_log_start":"272'9500","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9636","last_scrub_stamp":"2024-02-05T08:17:00.838236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:00.838236+0000","objects_scrubbed":7096,"log_size":4153,"log_dups_size":0,"ondisk_log_size":4153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:16.720317+0000","scrub_duration":1916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197459968,"num_objects":3013,"num_object_clones":0,"num_object_copies":9039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13653,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":212,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.406120+0000","last_change":"2024-02-05T08:17:14.099450+0000","last_active":"2024-02-05T08:17:57.406120+0000","last_peered":"2024-02-05T08:17:57.406120+0000","last_clean":"2024-02-05T08:17:57.406120+0000","last_became_active":"2024-02-05T08:12:41.722853+0000","last_became_peered":"2024-02-05T08:12:41.722853+0000","last_unstale":"2024-02-05T08:17:57.406120+0000","last_undegraded":"2024-02-05T08:17:57.406120+0000","last_fullsized":"2024-02-05T08:17:57.406120+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:17:14.099323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:30.182521+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":500,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400428+0000","last_change":"2024-02-05T08:17:12.948132+0000","last_active":"2024-02-05T08:17:57.400428+0000","last_peered":"2024-02-05T08:17:57.400428+0000","last_clean":"2024-02-05T08:17:57.400428+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:17:57.400428+0000","last_undegraded":"2024-02-05T08:17:57.400428+0000","last_fullsized":"2024-02-05T08:17:57.400428+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:12.948033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:17:12.948033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:33.426665+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":538,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.393183+0000","last_change":"2024-02-05T08:16:14.860227+0000","last_active":"2024-02-05T08:17:57.393183+0000","last_peered":"2024-02-05T08:17:57.393183+0000","last_clean":"2024-02-05T08:17:57.393183+0000","last_became_active":"2024-02-05T08:12:41.728925+0000","last_became_peered":"2024-02-05T08:12:41.728925+0000","last_unstale":"2024-02-05T08:17:57.393183+0000","last_undegraded":"2024-02-05T08:17:57.393183+0000","last_fullsized":"2024-02-05T08:17:57.393183+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-02-05T08:16:14.857110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:14.857110+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":500,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.388643+0000","last_change":"2024-02-05T08:16:54.393426+0000","last_active":"2024-02-05T08:17:57.388643+0000","last_peered":"2024-02-05T08:17:57.388643+0000","last_clean":"2024-02-05T08:17:57.388643+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:17:57.388643+0000","last_undegraded":"2024-02-05T08:17:57.388643+0000","last_fullsized":"2024-02-05T08:17:57.388643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:54.393312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:16:54.393312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:18.129621+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"285'13218","reported_seq":13388,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.096513+0000","last_change":"2024-02-05T08:17:13.631762+0000","last_active":"2024-02-05T08:17:58.096513+0000","last_peered":"2024-02-05T08:17:58.096513+0000","last_clean":"2024-02-05T08:17:58.096513+0000","last_became_active":"2024-02-05T08:14:57.763165+0000","last_became_peered":"2024-02-05T08:14:57.763165+0000","last_unstale":"2024-02-05T08:17:58.096513+0000","last_undegraded":"2024-02-05T08:17:58.096513+0000","last_fullsized":"2024-02-05T08:17:58.096513+0000","mapping_epoch":239,"log_start":"271'8600","ondisk_log_start":"271'8600","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'10307","last_scrub_stamp":"2024-02-05T08:17:13.631603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:13.631603+0000","objects_scrubbed":5907,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:20.761498+0000","scrub_duration":1163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194641920,"num_objects":2970,"num_object_clones":0,"num_object_copies":8910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13218,"num_write_kb":518016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":487,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.399942+0000","last_change":"2024-02-05T08:15:13.442673+0000","last_active":"2024-02-05T08:17:57.399942+0000","last_peered":"2024-02-05T08:17:57.399942+0000","last_clean":"2024-02-05T08:17:57.399942+0000","last_became_active":"2024-02-05T08:12:41.724830+0000","last_became_peered":"2024-02-05T08:12:41.724830+0000","last_unstale":"2024-02-05T08:17:57.399942+0000","last_undegraded":"2024-02-05T08:17:57.399942+0000","last_fullsized":"2024-02-05T08:17:57.399942+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:13.442573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:13.442573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":499,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.389055+0000","last_change":"2024-02-05T08:17:46.541994+0000","last_active":"2024-02-05T08:17:57.389055+0000","last_peered":"2024-02-05T08:17:57.389055+0000","last_clean":"2024-02-05T08:17:57.389055+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:17:57.389055+0000","last_undegraded":"2024-02-05T08:17:57.389055+0000","last_fullsized":"2024-02-05T08:17:57.389055+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:46.541893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:17:46.541893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:49.504701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":513,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.401957+0000","last_change":"2024-02-05T08:16:57.957167+0000","last_active":"2024-02-05T08:17:57.401957+0000","last_peered":"2024-02-05T08:17:57.401957+0000","last_clean":"2024-02-05T08:17:57.401957+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:17:57.401957+0000","last_undegraded":"2024-02-05T08:17:57.401957+0000","last_fullsized":"2024-02-05T08:17:57.401957+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:57.957064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:57.957064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:26.444176+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"236'35","reported_seq":273,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.399672+0000","last_change":"2024-02-05T08:17:54.291897+0000","last_active":"2024-02-05T08:17:57.399672+0000","last_peered":"2024-02-05T08:17:57.399672+0000","last_clean":"2024-02-05T08:17:57.399672+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:17:57.399672+0000","last_undegraded":"2024-02-05T08:17:57.399672+0000","last_fullsized":"2024-02-05T08:17:57.399672+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'35","last_scrub_stamp":"2024-02-05T08:17:54.291742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.291742+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:09.781739+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"285'13471","reported_seq":13650,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.088408+0000","last_change":"2024-02-05T08:17:06.041806+0000","last_active":"2024-02-05T08:17:58.088408+0000","last_peered":"2024-02-05T08:17:58.088408+0000","last_clean":"2024-02-05T08:17:58.088408+0000","last_became_active":"2024-02-05T08:14:57.761156+0000","last_became_peered":"2024-02-05T08:14:57.761156+0000","last_unstale":"2024-02-05T08:17:58.088408+0000","last_undegraded":"2024-02-05T08:17:58.088408+0000","last_fullsized":"2024-02-05T08:17:58.088408+0000","mapping_epoch":239,"log_start":"271'8800","ondisk_log_start":"271'8800","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'9937","last_scrub_stamp":"2024-02-05T08:17:06.041540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:06.041540+0000","objects_scrubbed":6705,"log_size":4671,"log_dups_size":0,"ondisk_log_size":4671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:16.501634+0000","scrub_duration":2521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201392128,"num_objects":3073,"num_object_clones":0,"num_object_copies":9219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13471,"num_write_kb":529408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":479,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.397516+0000","last_change":"2024-02-05T08:16:49.867423+0000","last_active":"2024-02-05T08:17:57.397516+0000","last_peered":"2024-02-05T08:17:57.397516+0000","last_clean":"2024-02-05T08:17:57.397516+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:17:57.397516+0000","last_undegraded":"2024-02-05T08:17:57.397516+0000","last_fullsized":"2024-02-05T08:17:57.397516+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:49.867301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:49.867301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:58.916262+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":506,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.389097+0000","last_change":"2024-02-05T08:17:10.458367+0000","last_active":"2024-02-05T08:17:57.389097+0000","last_peered":"2024-02-05T08:17:57.389097+0000","last_clean":"2024-02-05T08:17:57.389097+0000","last_became_active":"2024-02-05T08:12:41.743888+0000","last_became_peered":"2024-02-05T08:12:41.743888+0000","last_unstale":"2024-02-05T08:17:57.389097+0000","last_undegraded":"2024-02-05T08:17:57.389097+0000","last_fullsized":"2024-02-05T08:17:57.389097+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:10.458223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:10.458223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:12.912193+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":517,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.396608+0000","last_change":"2024-02-05T08:17:53.388494+0000","last_active":"2024-02-05T08:17:57.396608+0000","last_peered":"2024-02-05T08:17:57.396608+0000","last_clean":"2024-02-05T08:17:57.396608+0000","last_became_active":"2024-02-05T08:12:41.718843+0000","last_became_peered":"2024-02-05T08:12:41.718843+0000","last_unstale":"2024-02-05T08:17:57.396608+0000","last_undegraded":"2024-02-05T08:17:57.396608+0000","last_fullsized":"2024-02-05T08:17:57.396608+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:53.388314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:53.388314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:17.851953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"262'19","reported_seq":515,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398933+0000","last_change":"2024-02-05T08:16:04.837042+0000","last_active":"2024-02-05T08:17:57.398933+0000","last_peered":"2024-02-05T08:17:57.398933+0000","last_clean":"2024-02-05T08:17:57.398933+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:17:57.398933+0000","last_undegraded":"2024-02-05T08:17:57.398933+0000","last_fullsized":"2024-02-05T08:17:57.398933+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'19","last_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:16:04.836768+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"285'13772","reported_seq":13921,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.729478+0000","last_change":"2024-02-05T08:17:51.835234+0000","last_active":"2024-02-05T08:17:58.729478+0000","last_peered":"2024-02-05T08:17:58.729478+0000","last_clean":"2024-02-05T08:17:58.729478+0000","last_became_active":"2024-02-05T08:14:57.765694+0000","last_became_peered":"2024-02-05T08:14:57.765694+0000","last_unstale":"2024-02-05T08:17:58.729478+0000","last_undegraded":"2024-02-05T08:17:58.729478+0000","last_fullsized":"2024-02-05T08:17:58.729478+0000","mapping_epoch":239,"log_start":"270'8500","ondisk_log_start":"270'8500","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'13220","last_scrub_stamp":"2024-02-05T08:17:51.835033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:51.835033+0000","objects_scrubbed":3578,"log_size":5272,"log_dups_size":0,"ondisk_log_size":5272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:59.083208+0000","scrub_duration":1459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194904064,"num_objects":2974,"num_object_clones":0,"num_object_copies":8922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13772,"num_write_kb":535872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":212,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.405370+0000","last_change":"2024-02-05T08:17:54.817747+0000","last_active":"2024-02-05T08:17:57.405370+0000","last_peered":"2024-02-05T08:17:57.405370+0000","last_clean":"2024-02-05T08:17:57.405370+0000","last_became_active":"2024-02-05T08:12:41.726776+0000","last_became_peered":"2024-02-05T08:12:41.726776+0000","last_unstale":"2024-02-05T08:17:57.405370+0000","last_undegraded":"2024-02-05T08:17:57.405370+0000","last_fullsized":"2024-02-05T08:17:57.405370+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:54.817641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.817641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:24.064027+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":532,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.396541+0000","last_change":"2024-02-05T08:16:48.614737+0000","last_active":"2024-02-05T08:17:57.396541+0000","last_peered":"2024-02-05T08:17:57.396541+0000","last_clean":"2024-02-05T08:17:57.396541+0000","last_became_active":"2024-02-05T08:12:29.679395+0000","last_became_peered":"2024-02-05T08:12:29.679395+0000","last_unstale":"2024-02-05T08:17:57.396541+0000","last_undegraded":"2024-02-05T08:17:57.396541+0000","last_fullsized":"2024-02-05T08:17:57.396541+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:16:48.614659+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:16:48.614659+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:17.786365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":504,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.393027+0000","last_change":"2024-02-05T08:16:11.809948+0000","last_active":"2024-02-05T08:17:57.393027+0000","last_peered":"2024-02-05T08:17:57.393027+0000","last_clean":"2024-02-05T08:17:57.393027+0000","last_became_active":"2024-02-05T08:12:29.670859+0000","last_became_peered":"2024-02-05T08:12:29.670859+0000","last_unstale":"2024-02-05T08:17:57.393027+0000","last_undegraded":"2024-02-05T08:17:57.393027+0000","last_fullsized":"2024-02-05T08:17:57.393027+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:11.809845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:16:11.809845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"236'51","reported_seq":761,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.390183+0000","last_change":"2024-02-05T08:17:56.019643+0000","last_active":"2024-02-05T08:17:57.390183+0000","last_peered":"2024-02-05T08:17:57.390183+0000","last_clean":"2024-02-05T08:17:57.390183+0000","last_became_active":"2024-02-05T08:12:41.719820+0000","last_became_peered":"2024-02-05T08:12:41.719820+0000","last_unstale":"2024-02-05T08:17:57.390183+0000","last_undegraded":"2024-02-05T08:17:57.390183+0000","last_fullsized":"2024-02-05T08:17:57.390183+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'51","last_scrub_stamp":"2024-02-05T08:17:56.019516+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:17:56.019516+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:15.129435+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":475,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398528+0000","last_change":"2024-02-05T08:16:09.774039+0000","last_active":"2024-02-05T08:17:57.398528+0000","last_peered":"2024-02-05T08:17:57.398528+0000","last_clean":"2024-02-05T08:17:57.398528+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:17:57.398528+0000","last_undegraded":"2024-02-05T08:17:57.398528+0000","last_fullsized":"2024-02-05T08:17:57.398528+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:09.773910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:09.773910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"285'13449","reported_seq":13634,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.060835+0000","last_change":"2024-02-05T08:16:58.188469+0000","last_active":"2024-02-05T08:17:58.060835+0000","last_peered":"2024-02-05T08:17:58.060835+0000","last_clean":"2024-02-05T08:17:58.060835+0000","last_became_active":"2024-02-05T08:14:57.763184+0000","last_became_peered":"2024-02-05T08:14:57.763184+0000","last_unstale":"2024-02-05T08:17:58.060835+0000","last_undegraded":"2024-02-05T08:17:58.060835+0000","last_fullsized":"2024-02-05T08:17:58.060835+0000","mapping_epoch":239,"log_start":"271'8800","ondisk_log_start":"271'8800","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'9334","last_scrub_stamp":"2024-02-05T08:16:58.188319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:16:58.188319+0000","objects_scrubbed":7257,"log_size":4649,"log_dups_size":0,"ondisk_log_size":4649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:24.047016+0000","scrub_duration":2565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199819264,"num_objects":3049,"num_object_clones":0,"num_object_copies":9147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13449,"num_write_kb":527936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":499,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.402088+0000","last_change":"2024-02-05T08:15:20.467544+0000","last_active":"2024-02-05T08:17:57.402088+0000","last_peered":"2024-02-05T08:17:57.402088+0000","last_clean":"2024-02-05T08:17:57.402088+0000","last_became_active":"2024-02-05T08:12:41.732338+0000","last_became_peered":"2024-02-05T08:12:41.732338+0000","last_unstale":"2024-02-05T08:17:57.402088+0000","last_undegraded":"2024-02-05T08:17:57.402088+0000","last_fullsized":"2024-02-05T08:17:57.402088+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:20.467407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:20.467407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":520,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.388596+0000","last_change":"2024-02-05T08:17:06.932722+0000","last_active":"2024-02-05T08:17:57.388596+0000","last_peered":"2024-02-05T08:17:57.388596+0000","last_clean":"2024-02-05T08:17:57.388596+0000","last_became_active":"2024-02-05T08:12:29.673554+0000","last_became_peered":"2024-02-05T08:12:29.673554+0000","last_unstale":"2024-02-05T08:17:57.388596+0000","last_undegraded":"2024-02-05T08:17:57.388596+0000","last_fullsized":"2024-02-05T08:17:57.388596+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:17:06.932583+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:17:06.932583+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:31.571859+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":523,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.388594+0000","last_change":"2024-02-05T08:17:12.613722+0000","last_active":"2024-02-05T08:17:57.388594+0000","last_peered":"2024-02-05T08:17:57.388594+0000","last_clean":"2024-02-05T08:17:57.388594+0000","last_became_active":"2024-02-05T08:12:41.746289+0000","last_became_peered":"2024-02-05T08:12:41.746289+0000","last_unstale":"2024-02-05T08:17:57.388594+0000","last_undegraded":"2024-02-05T08:17:57.388594+0000","last_fullsized":"2024-02-05T08:17:57.388594+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:12.613617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:39.497927+0000","last_clean_scrub_stamp":"2024-02-05T08:17:12.613617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:22.365708+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'17","reported_seq":513,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.392777+0000","last_change":"2024-02-05T08:17:47.561079+0000","last_active":"2024-02-05T08:17:57.392777+0000","last_peered":"2024-02-05T08:17:57.392777+0000","last_clean":"2024-02-05T08:17:57.392777+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:17:57.392777+0000","last_undegraded":"2024-02-05T08:17:57.392777+0000","last_fullsized":"2024-02-05T08:17:57.392777+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'17","last_scrub_stamp":"2024-02-05T08:17:47.560963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:47.560963+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:53.181518+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":500,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.390389+0000","last_change":"2024-02-05T08:17:03.422904+0000","last_active":"2024-02-05T08:17:57.390389+0000","last_peered":"2024-02-05T08:17:57.390389+0000","last_clean":"2024-02-05T08:17:57.390389+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:17:57.390389+0000","last_undegraded":"2024-02-05T08:17:57.390389+0000","last_fullsized":"2024-02-05T08:17:57.390389+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:03.422828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:03.422828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:05.002750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"285'13336","reported_seq":13486,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.738013+0000","last_change":"2024-02-05T08:17:48.702277+0000","last_active":"2024-02-05T08:17:58.738013+0000","last_peered":"2024-02-05T08:17:58.738013+0000","last_clean":"2024-02-05T08:17:58.738013+0000","last_became_active":"2024-02-05T08:14:57.762164+0000","last_became_peered":"2024-02-05T08:14:57.762164+0000","last_unstale":"2024-02-05T08:17:58.738013+0000","last_undegraded":"2024-02-05T08:17:58.738013+0000","last_fullsized":"2024-02-05T08:17:58.738013+0000","mapping_epoch":239,"log_start":"268'8100","ondisk_log_start":"268'8100","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'12548","last_scrub_stamp":"2024-02-05T08:17:48.702005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:48.702005+0000","objects_scrubbed":3805,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:10.411578+0000","scrub_duration":1352,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194379776,"num_objects":2966,"num_object_clones":0,"num_object_copies":8898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13336,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":486,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398049+0000","last_change":"2024-02-05T08:15:55.025258+0000","last_active":"2024-02-05T08:17:57.398049+0000","last_peered":"2024-02-05T08:17:57.398049+0000","last_clean":"2024-02-05T08:17:57.398049+0000","last_became_active":"2024-02-05T08:12:41.717170+0000","last_became_peered":"2024-02-05T08:12:41.717170+0000","last_unstale":"2024-02-05T08:17:57.398049+0000","last_undegraded":"2024-02-05T08:17:57.398049+0000","last_fullsized":"2024-02-05T08:17:57.398049+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:55.025140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:55.025140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":781,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.401108+0000","last_change":"2024-02-05T08:16:15.047572+0000","last_active":"2024-02-05T08:17:57.401108+0000","last_peered":"2024-02-05T08:17:57.401108+0000","last_clean":"2024-02-05T08:17:57.401108+0000","last_became_active":"2024-02-05T08:12:32.677426+0000","last_became_peered":"2024-02-05T08:12:32.677426+0000","last_unstale":"2024-02-05T08:17:57.401108+0000","last_undegraded":"2024-02-05T08:17:57.401108+0000","last_fullsized":"2024-02-05T08:17:57.401108+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:16:15.047436+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:16:15.047436+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":494,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.396458+0000","last_change":"2024-02-05T08:17:13.476425+0000","last_active":"2024-02-05T08:17:57.396458+0000","last_peered":"2024-02-05T08:17:57.396458+0000","last_clean":"2024-02-05T08:17:57.396458+0000","last_became_active":"2024-02-05T08:12:41.722331+0000","last_became_peered":"2024-02-05T08:12:41.722331+0000","last_unstale":"2024-02-05T08:17:57.396458+0000","last_undegraded":"2024-02-05T08:17:57.396458+0000","last_fullsized":"2024-02-05T08:17:57.396458+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:13.476308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:13.476308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:21.477845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"236'30","reported_seq":323,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.401833+0000","last_change":"2024-02-05T08:16:54.287476+0000","last_active":"2024-02-05T08:17:57.401833+0000","last_peered":"2024-02-05T08:17:57.401833+0000","last_clean":"2024-02-05T08:17:57.401833+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:17:57.401833+0000","last_undegraded":"2024-02-05T08:17:57.401833+0000","last_fullsized":"2024-02-05T08:17:57.401833+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:16:54.287331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:54.287331+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:04.985090+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":485,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400923+0000","last_change":"2024-02-05T08:16:06.033678+0000","last_active":"2024-02-05T08:17:57.400923+0000","last_peered":"2024-02-05T08:17:57.400923+0000","last_clean":"2024-02-05T08:17:57.400923+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:17:57.400923+0000","last_undegraded":"2024-02-05T08:17:57.400923+0000","last_fullsized":"2024-02-05T08:17:57.400923+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:06.033461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:06.033461+0000","last_clean_scrub_stamp":"2024-02-05T08:16:06.033461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"285'13605","reported_seq":13775,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.729467+0000","last_change":"2024-02-05T08:17:15.291706+0000","last_active":"2024-02-05T08:17:58.729467+0000","last_peered":"2024-02-05T08:17:58.729467+0000","last_clean":"2024-02-05T08:17:58.729467+0000","last_became_active":"2024-02-05T08:14:57.761382+0000","last_became_peered":"2024-02-05T08:14:57.761382+0000","last_unstale":"2024-02-05T08:17:58.729467+0000","last_undegraded":"2024-02-05T08:17:58.729467+0000","last_fullsized":"2024-02-05T08:17:58.729467+0000","mapping_epoch":239,"log_start":"271'8400","ondisk_log_start":"271'8400","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'10631","last_scrub_stamp":"2024-02-05T08:17:15.291551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:15.291551+0000","objects_scrubbed":5855,"log_size":5205,"log_dups_size":0,"ondisk_log_size":5205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:25.553730+0000","scrub_duration":1251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186843136,"num_objects":2851,"num_object_clones":0,"num_object_copies":8553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2851,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13605,"num_write_kb":526592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":487,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.389832+0000","last_change":"2024-02-05T08:16:03.932381+0000","last_active":"2024-02-05T08:17:57.389832+0000","last_peered":"2024-02-05T08:17:57.389832+0000","last_clean":"2024-02-05T08:17:57.389832+0000","last_became_active":"2024-02-05T08:12:41.716394+0000","last_became_peered":"2024-02-05T08:12:41.716394+0000","last_unstale":"2024-02-05T08:17:57.389832+0000","last_undegraded":"2024-02-05T08:17:57.389832+0000","last_fullsized":"2024-02-05T08:17:57.389832+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:03.932264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:03.932264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":489,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.397335+0000","last_change":"2024-02-05T08:15:15.359343+0000","last_active":"2024-02-05T08:17:57.397335+0000","last_peered":"2024-02-05T08:17:57.397335+0000","last_clean":"2024-02-05T08:17:57.397335+0000","last_became_active":"2024-02-05T08:12:41.741846+0000","last_became_peered":"2024-02-05T08:12:41.741846+0000","last_unstale":"2024-02-05T08:17:57.397335+0000","last_undegraded":"2024-02-05T08:17:57.397335+0000","last_fullsized":"2024-02-05T08:17:57.397335+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:15.359190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:15.359190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":1303,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.401007+0000","last_change":"2024-02-05T08:16:53.018165+0000","last_active":"2024-02-05T08:17:57.401007+0000","last_peered":"2024-02-05T08:17:57.401007+0000","last_clean":"2024-02-05T08:17:57.401007+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:17:57.401007+0000","last_undegraded":"2024-02-05T08:17:57.401007+0000","last_fullsized":"2024-02-05T08:17:57.401007+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:16:53.018039+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:16:53.018039+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:01.775200+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":789,"num_read_kb":784,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":535,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398611+0000","last_change":"2024-02-05T08:16:01.857925+0000","last_active":"2024-02-05T08:17:57.398611+0000","last_peered":"2024-02-05T08:17:57.398611+0000","last_clean":"2024-02-05T08:17:57.398611+0000","last_became_active":"2024-02-05T08:14:24.992820+0000","last_became_peered":"2024-02-05T08:14:24.992820+0000","last_unstale":"2024-02-05T08:17:57.398611+0000","last_undegraded":"2024-02-05T08:17:57.398611+0000","last_fullsized":"2024-02-05T08:17:57.398611+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:16:01.857750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:01.857750+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":484,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398398+0000","last_change":"2024-02-05T08:16:02.899200+0000","last_active":"2024-02-05T08:17:57.398398+0000","last_peered":"2024-02-05T08:17:57.398398+0000","last_clean":"2024-02-05T08:17:57.398398+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:17:57.398398+0000","last_undegraded":"2024-02-05T08:17:57.398398+0000","last_fullsized":"2024-02-05T08:17:57.398398+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:16:02.899084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"285'13389","reported_seq":13559,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.053800+0000","last_change":"2024-02-05T08:17:17.506968+0000","last_active":"2024-02-05T08:17:58.053800+0000","last_peered":"2024-02-05T08:17:58.053800+0000","last_clean":"2024-02-05T08:17:58.053800+0000","last_became_active":"2024-02-05T08:14:57.764021+0000","last_became_peered":"2024-02-05T08:14:57.764021+0000","last_unstale":"2024-02-05T08:17:58.053800+0000","last_undegraded":"2024-02-05T08:17:58.053800+0000","last_fullsized":"2024-02-05T08:17:58.053800+0000","mapping_epoch":239,"log_start":"271'8900","ondisk_log_start":"271'8900","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'10561","last_scrub_stamp":"2024-02-05T08:17:17.506587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:17.506587+0000","objects_scrubbed":5822,"log_size":4489,"log_dups_size":0,"ondisk_log_size":4489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:32.650366+0000","scrub_duration":1470,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193396736,"num_objects":2951,"num_object_clones":0,"num_object_copies":8853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13389,"num_write_kb":522880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":493,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.389020+0000","last_change":"2024-02-05T08:16:07.996089+0000","last_active":"2024-02-05T08:17:57.389020+0000","last_peered":"2024-02-05T08:17:57.389020+0000","last_clean":"2024-02-05T08:17:57.389020+0000","last_became_active":"2024-02-05T08:12:29.672973+0000","last_became_peered":"2024-02-05T08:12:29.672973+0000","last_unstale":"2024-02-05T08:17:57.389020+0000","last_undegraded":"2024-02-05T08:17:57.389020+0000","last_fullsized":"2024-02-05T08:17:57.389020+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:07.996005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:07.996005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":501,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.390034+0000","last_change":"2024-02-05T08:16:50.981337+0000","last_active":"2024-02-05T08:17:57.390034+0000","last_peered":"2024-02-05T08:17:57.390034+0000","last_clean":"2024-02-05T08:17:57.390034+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:17:57.390034+0000","last_undegraded":"2024-02-05T08:17:57.390034+0000","last_fullsized":"2024-02-05T08:17:57.390034+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:50.981203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:50.981203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:14.667929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":538,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398748+0000","last_change":"2024-02-05T08:15:56.779910+0000","last_active":"2024-02-05T08:17:57.398748+0000","last_peered":"2024-02-05T08:17:57.398748+0000","last_clean":"2024-02-05T08:17:57.398748+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:17:57.398748+0000","last_undegraded":"2024-02-05T08:17:57.398748+0000","last_fullsized":"2024-02-05T08:17:57.398748+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-02-05T08:15:56.779551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:56.779551+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":221,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.399596+0000","last_change":"2024-02-05T08:15:58.238221+0000","last_active":"2024-02-05T08:17:57.399596+0000","last_peered":"2024-02-05T08:17:57.399596+0000","last_clean":"2024-02-05T08:17:57.399596+0000","last_became_active":"2024-02-05T08:12:29.688534+0000","last_became_peered":"2024-02-05T08:12:29.688534+0000","last_unstale":"2024-02-05T08:17:57.399596+0000","last_undegraded":"2024-02-05T08:17:57.399596+0000","last_fullsized":"2024-02-05T08:17:57.399596+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:58.238104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:15:58.238104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"285'13218","reported_seq":13397,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.071082+0000","last_change":"2024-02-05T08:17:01.762530+0000","last_active":"2024-02-05T08:17:58.071082+0000","last_peered":"2024-02-05T08:17:58.071082+0000","last_clean":"2024-02-05T08:17:58.071082+0000","last_became_active":"2024-02-05T08:14:57.764038+0000","last_became_peered":"2024-02-05T08:14:57.764038+0000","last_unstale":"2024-02-05T08:17:58.071082+0000","last_undegraded":"2024-02-05T08:17:58.071082+0000","last_fullsized":"2024-02-05T08:17:58.071082+0000","mapping_epoch":239,"log_start":"271'8600","ondisk_log_start":"271'8600","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9398","last_scrub_stamp":"2024-02-05T08:17:01.762104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:01.762104+0000","objects_scrubbed":6842,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:25.827043+0000","scrub_duration":2222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193069056,"num_objects":2946,"num_object_clones":0,"num_object_copies":8838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13218,"num_write_kb":517248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":485,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.396861+0000","last_change":"2024-02-05T08:16:00.862974+0000","last_active":"2024-02-05T08:17:57.396861+0000","last_peered":"2024-02-05T08:17:57.396861+0000","last_clean":"2024-02-05T08:17:57.396861+0000","last_became_active":"2024-02-05T08:12:41.735706+0000","last_became_peered":"2024-02-05T08:12:41.735706+0000","last_unstale":"2024-02-05T08:17:57.396861+0000","last_undegraded":"2024-02-05T08:17:57.396861+0000","last_fullsized":"2024-02-05T08:17:57.396861+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:00.862633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:00.862633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":506,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400860+0000","last_change":"2024-02-05T08:17:50.387634+0000","last_active":"2024-02-05T08:17:57.400860+0000","last_peered":"2024-02-05T08:17:57.400860+0000","last_clean":"2024-02-05T08:17:57.400860+0000","last_became_active":"2024-02-05T08:12:41.730010+0000","last_became_peered":"2024-02-05T08:12:41.730010+0000","last_unstale":"2024-02-05T08:17:57.400860+0000","last_undegraded":"2024-02-05T08:17:57.400860+0000","last_fullsized":"2024-02-05T08:17:57.400860+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:50.387417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:50.387417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:54.040152+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'20","reported_seq":527,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.391963+0000","last_change":"2024-02-05T08:17:52.948101+0000","last_active":"2024-02-05T08:17:57.391963+0000","last_peered":"2024-02-05T08:17:57.391963+0000","last_clean":"2024-02-05T08:17:57.391963+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:17:57.391963+0000","last_undegraded":"2024-02-05T08:17:57.391963+0000","last_fullsized":"2024-02-05T08:17:57.391963+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:17:52.947966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:52.947966+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:06.320713+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"277'11","reported_seq":672,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398974+0000","last_change":"2024-02-05T08:17:07.973477+0000","last_active":"2024-02-05T08:17:57.398974+0000","last_peered":"2024-02-05T08:17:57.398974+0000","last_clean":"2024-02-05T08:17:57.398974+0000","last_became_active":"2024-02-05T08:12:29.683091+0000","last_became_peered":"2024-02-05T08:12:29.683091+0000","last_unstale":"2024-02-05T08:17:57.398974+0000","last_undegraded":"2024-02-05T08:17:57.398974+0000","last_fullsized":"2024-02-05T08:17:57.398974+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_clean_scrub_stamp":"2024-02-05T08:17:07.973171+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:19.190363+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"285'17572","reported_seq":17901,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.809870+0000","last_change":"2024-02-05T08:17:48.136219+0000","last_active":"2024-02-05T08:17:57.809870+0000","last_peered":"2024-02-05T08:17:57.809870+0000","last_clean":"2024-02-05T08:17:57.809870+0000","last_became_active":"2024-02-05T08:14:57.761443+0000","last_became_peered":"2024-02-05T08:14:57.761443+0000","last_unstale":"2024-02-05T08:17:57.809870+0000","last_undegraded":"2024-02-05T08:17:57.809870+0000","last_fullsized":"2024-02-05T08:17:57.809870+0000","mapping_epoch":239,"log_start":"279'12902","ondisk_log_start":"279'12902","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'16825","last_scrub_stamp":"2024-02-05T08:17:48.136083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:48.136083+0000","objects_scrubbed":3810,"log_size":4670,"log_dups_size":0,"ondisk_log_size":4670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:17.460613+0000","scrub_duration":1270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197525504,"num_objects":3014,"num_object_clones":0,"num_object_copies":9042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13268,"num_write_kb":521024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":493,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.406305+0000","last_change":"2024-02-05T08:15:29.563981+0000","last_active":"2024-02-05T08:17:57.406305+0000","last_peered":"2024-02-05T08:17:57.406305+0000","last_clean":"2024-02-05T08:17:57.406305+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:17:57.406305+0000","last_undegraded":"2024-02-05T08:17:57.406305+0000","last_fullsized":"2024-02-05T08:17:57.406305+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:29.563876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:29.563876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":502,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398663+0000","last_change":"2024-02-05T08:16:08.716324+0000","last_active":"2024-02-05T08:17:57.398663+0000","last_peered":"2024-02-05T08:17:57.398663+0000","last_clean":"2024-02-05T08:17:57.398663+0000","last_became_active":"2024-02-05T08:11:23.200753+0000","last_became_peered":"2024-02-05T08:11:23.200753+0000","last_unstale":"2024-02-05T08:17:57.398663+0000","last_undegraded":"2024-02-05T08:17:57.398663+0000","last_fullsized":"2024-02-05T08:17:57.398663+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:16:08.716199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:08.716199+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"236'28","reported_seq":574,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.392685+0000","last_change":"2024-02-05T08:17:12.688139+0000","last_active":"2024-02-05T08:17:57.392685+0000","last_peered":"2024-02-05T08:17:57.392685+0000","last_clean":"2024-02-05T08:17:57.392685+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:17:57.392685+0000","last_undegraded":"2024-02-05T08:17:57.392685+0000","last_fullsized":"2024-02-05T08:17:57.392685+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:17:12.687898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:12.687898+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:34.476715+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"194'10","reported_seq":596,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.388373+0000","last_change":"2024-02-05T08:17:14.040873+0000","last_active":"2024-02-05T08:17:57.388373+0000","last_peered":"2024-02-05T08:17:57.388373+0000","last_clean":"2024-02-05T08:17:57.388373+0000","last_became_active":"2024-02-05T08:12:41.721686+0000","last_became_peered":"2024-02-05T08:12:41.721686+0000","last_unstale":"2024-02-05T08:17:57.388373+0000","last_undegraded":"2024-02-05T08:17:57.388373+0000","last_fullsized":"2024-02-05T08:17:57.388373+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'10","last_scrub_stamp":"2024-02-05T08:17:14.040769+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:17:14.040769+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:36.262898+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"285'13416","reported_seq":13570,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.055092+0000","last_change":"2024-02-05T08:17:45.599157+0000","last_active":"2024-02-05T08:17:58.055092+0000","last_peered":"2024-02-05T08:17:58.055092+0000","last_clean":"2024-02-05T08:17:58.055092+0000","last_became_active":"2024-02-05T08:14:57.765609+0000","last_became_peered":"2024-02-05T08:14:57.765609+0000","last_unstale":"2024-02-05T08:17:58.055092+0000","last_undegraded":"2024-02-05T08:17:58.055092+0000","last_fullsized":"2024-02-05T08:17:58.055092+0000","mapping_epoch":239,"log_start":"271'9000","ondisk_log_start":"271'9000","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'12489","last_scrub_stamp":"2024-02-05T08:17:45.598921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:45.598921+0000","objects_scrubbed":4027,"log_size":4416,"log_dups_size":0,"ondisk_log_size":4416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:13.318391+0000","scrub_duration":1561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199557144,"num_objects":3046,"num_object_clones":0,"num_object_copies":9138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13416,"num_write_kb":526721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":494,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.392742+0000","last_change":"2024-02-05T08:17:10.764121+0000","last_active":"2024-02-05T08:17:57.392742+0000","last_peered":"2024-02-05T08:17:57.392742+0000","last_clean":"2024-02-05T08:17:57.392742+0000","last_became_active":"2024-02-05T08:12:41.731253+0000","last_became_peered":"2024-02-05T08:12:41.731253+0000","last_unstale":"2024-02-05T08:17:57.392742+0000","last_undegraded":"2024-02-05T08:17:57.392742+0000","last_fullsized":"2024-02-05T08:17:57.392742+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:10.764008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:10.764008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:12.220343+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":224,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400572+0000","last_change":"2024-02-05T08:17:52.089819+0000","last_active":"2024-02-05T08:17:57.400572+0000","last_peered":"2024-02-05T08:17:57.400572+0000","last_clean":"2024-02-05T08:17:57.400572+0000","last_became_active":"2024-02-05T08:12:41.723699+0000","last_became_peered":"2024-02-05T08:12:41.723699+0000","last_unstale":"2024-02-05T08:17:57.400572+0000","last_undegraded":"2024-02-05T08:17:57.400572+0000","last_fullsized":"2024-02-05T08:17:57.400572+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:17:52.089740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:52.089740+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:53.221624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"236'38","reported_seq":554,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.401295+0000","last_change":"2024-02-05T08:17:05.909449+0000","last_active":"2024-02-05T08:17:57.401295+0000","last_peered":"2024-02-05T08:17:57.401295+0000","last_clean":"2024-02-05T08:17:57.401295+0000","last_became_active":"2024-02-05T08:12:41.728044+0000","last_became_peered":"2024-02-05T08:12:41.728044+0000","last_unstale":"2024-02-05T08:17:57.401295+0000","last_undegraded":"2024-02-05T08:17:57.401295+0000","last_fullsized":"2024-02-05T08:17:57.401295+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'38","last_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:17:05.909305+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:33.494770+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":490,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.392465+0000","last_change":"2024-02-05T08:16:51.793138+0000","last_active":"2024-02-05T08:17:57.392465+0000","last_peered":"2024-02-05T08:17:57.392465+0000","last_clean":"2024-02-05T08:17:57.392465+0000","last_became_active":"2024-02-05T08:12:41.720247+0000","last_became_peered":"2024-02-05T08:12:41.720247+0000","last_unstale":"2024-02-05T08:17:57.392465+0000","last_undegraded":"2024-02-05T08:17:57.392465+0000","last_fullsized":"2024-02-05T08:17:57.392465+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:51.793017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:51.793017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:02.823985+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"285'13409","reported_seq":13595,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.353938+0000","last_change":"2024-02-05T08:16:55.072732+0000","last_active":"2024-02-05T08:17:58.353938+0000","last_peered":"2024-02-05T08:17:58.353938+0000","last_clean":"2024-02-05T08:17:58.353938+0000","last_became_active":"2024-02-05T08:14:57.764174+0000","last_became_peered":"2024-02-05T08:14:57.764174+0000","last_unstale":"2024-02-05T08:17:58.353938+0000","last_undegraded":"2024-02-05T08:17:58.353938+0000","last_fullsized":"2024-02-05T08:17:58.353938+0000","mapping_epoch":239,"log_start":"271'9200","ondisk_log_start":"271'9200","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'9088","last_scrub_stamp":"2024-02-05T08:16:55.072580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:16:55.072580+0000","objects_scrubbed":7413,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:21.023713+0000","scrub_duration":2137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197197824,"num_objects":3009,"num_object_clones":0,"num_object_copies":9027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13409,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":487,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.399295+0000","last_change":"2024-02-05T08:16:12.876608+0000","last_active":"2024-02-05T08:17:57.399295+0000","last_peered":"2024-02-05T08:17:57.399295+0000","last_clean":"2024-02-05T08:17:57.399295+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:17:57.399295+0000","last_undegraded":"2024-02-05T08:17:57.399295+0000","last_fullsized":"2024-02-05T08:17:57.399295+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:12.876345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:12.876345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":651,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.408728+0000","last_change":"2024-02-05T08:16:49.737238+0000","last_active":"2024-02-05T08:17:57.408728+0000","last_peered":"2024-02-05T08:17:57.408728+0000","last_clean":"2024-02-05T08:17:57.408728+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:17:57.408728+0000","last_undegraded":"2024-02-05T08:17:57.408728+0000","last_fullsized":"2024-02-05T08:17:57.408728+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:16:49.733743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:49.733743+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:03.692915+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":220,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.388762+0000","last_change":"2024-02-05T08:17:50.018933+0000","last_active":"2024-02-05T08:17:57.388762+0000","last_peered":"2024-02-05T08:17:57.388762+0000","last_clean":"2024-02-05T08:17:57.388762+0000","last_became_active":"2024-02-05T08:12:41.724969+0000","last_became_peered":"2024-02-05T08:12:41.724969+0000","last_unstale":"2024-02-05T08:17:57.388762+0000","last_undegraded":"2024-02-05T08:17:57.388762+0000","last_fullsized":"2024-02-05T08:17:57.388762+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:50.018829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:50.018829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:01.189055+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"236'27","reported_seq":561,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.389546+0000","last_change":"2024-02-05T08:17:13.636942+0000","last_active":"2024-02-05T08:17:57.389546+0000","last_peered":"2024-02-05T08:17:57.389546+0000","last_clean":"2024-02-05T08:17:57.389546+0000","last_became_active":"2024-02-05T08:12:41.725007+0000","last_became_peered":"2024-02-05T08:12:41.725007+0000","last_unstale":"2024-02-05T08:17:57.389546+0000","last_undegraded":"2024-02-05T08:17:57.389546+0000","last_fullsized":"2024-02-05T08:17:57.389546+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:17:13.636688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:13.636688+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:25.647442+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"285'13627","reported_seq":13773,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.041702+0000","last_change":"2024-02-05T08:17:52.291887+0000","last_active":"2024-02-05T08:17:58.041702+0000","last_peered":"2024-02-05T08:17:58.041702+0000","last_clean":"2024-02-05T08:17:58.041702+0000","last_became_active":"2024-02-05T08:14:57.765255+0000","last_became_peered":"2024-02-05T08:14:57.765255+0000","last_unstale":"2024-02-05T08:17:58.041702+0000","last_undegraded":"2024-02-05T08:17:58.041702+0000","last_fullsized":"2024-02-05T08:17:58.041702+0000","mapping_epoch":239,"log_start":"271'9200","ondisk_log_start":"271'9200","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'13143","last_scrub_stamp":"2024-02-05T08:17:52.291706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:52.291706+0000","objects_scrubbed":3510,"log_size":4427,"log_dups_size":0,"ondisk_log_size":4427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:54.016672+0000","scrub_duration":1208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196149248,"num_objects":2993,"num_object_clones":0,"num_object_copies":8979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13627,"num_write_kb":531840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":480,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.400497+0000","last_change":"2024-02-05T08:16:50.260245+0000","last_active":"2024-02-05T08:17:57.400497+0000","last_peered":"2024-02-05T08:17:57.400497+0000","last_clean":"2024-02-05T08:17:57.400497+0000","last_became_active":"2024-02-05T08:12:41.719835+0000","last_became_peered":"2024-02-05T08:12:41.719835+0000","last_unstale":"2024-02-05T08:17:57.400497+0000","last_undegraded":"2024-02-05T08:17:57.400497+0000","last_fullsized":"2024-02-05T08:17:57.400497+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:50.260123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:16:50.260123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:06.973752+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":503,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398314+0000","last_change":"2024-02-05T08:16:49.669406+0000","last_active":"2024-02-05T08:17:57.398314+0000","last_peered":"2024-02-05T08:17:57.398314+0000","last_clean":"2024-02-05T08:17:57.398314+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:17:57.398314+0000","last_undegraded":"2024-02-05T08:17:57.398314+0000","last_fullsized":"2024-02-05T08:17:57.398314+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:49.669303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:49.669303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:02.029690+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":500,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.392590+0000","last_change":"2024-02-05T08:16:55.778045+0000","last_active":"2024-02-05T08:17:57.392590+0000","last_peered":"2024-02-05T08:17:57.392590+0000","last_clean":"2024-02-05T08:17:57.392590+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:17:57.392590+0000","last_undegraded":"2024-02-05T08:17:57.392590+0000","last_fullsized":"2024-02-05T08:17:57.392590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:55.777839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:16:55.777839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:17:58.223648+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"236'19","reported_seq":554,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.408401+0000","last_change":"2024-02-05T08:16:12.843894+0000","last_active":"2024-02-05T08:17:57.408401+0000","last_peered":"2024-02-05T08:17:57.408401+0000","last_clean":"2024-02-05T08:17:57.408401+0000","last_became_active":"2024-02-05T08:12:29.666284+0000","last_became_peered":"2024-02-05T08:12:29.666284+0000","last_unstale":"2024-02-05T08:17:57.408401+0000","last_undegraded":"2024-02-05T08:17:57.408401+0000","last_fullsized":"2024-02-05T08:17:57.408401+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:16:12.843751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:12.843751+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"285'13570","reported_seq":13757,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:58.314379+0000","last_change":"2024-02-05T08:16:53.913489+0000","last_active":"2024-02-05T08:17:58.314379+0000","last_peered":"2024-02-05T08:17:58.314379+0000","last_clean":"2024-02-05T08:17:58.314379+0000","last_became_active":"2024-02-05T08:14:57.764870+0000","last_became_peered":"2024-02-05T08:14:57.764870+0000","last_unstale":"2024-02-05T08:17:58.314379+0000","last_undegraded":"2024-02-05T08:17:58.314379+0000","last_fullsized":"2024-02-05T08:17:58.314379+0000","mapping_epoch":239,"log_start":"272'9400","ondisk_log_start":"272'9400","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'9083","last_scrub_stamp":"2024-02-05T08:16:53.913296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:16:53.913296+0000","objects_scrubbed":7620,"log_size":4170,"log_dups_size":0,"ondisk_log_size":4170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:16.578086+0000","scrub_duration":1959,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201195520,"num_objects":3070,"num_object_clones":0,"num_object_copies":9210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13570,"num_write_kb":532480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":506,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.392925+0000","last_change":"2024-02-05T08:17:09.581135+0000","last_active":"2024-02-05T08:17:57.392925+0000","last_peered":"2024-02-05T08:17:57.392925+0000","last_clean":"2024-02-05T08:17:57.392925+0000","last_became_active":"2024-02-05T08:12:41.737224+0000","last_became_peered":"2024-02-05T08:12:41.737224+0000","last_unstale":"2024-02-05T08:17:57.392925+0000","last_undegraded":"2024-02-05T08:17:57.392925+0000","last_fullsized":"2024-02-05T08:17:57.392925+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:17:09.581011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:09.581011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:10.528750+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":484,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.392239+0000","last_change":"2024-02-05T08:17:12.127732+0000","last_active":"2024-02-05T08:17:57.392239+0000","last_peered":"2024-02-05T08:17:57.392239+0000","last_clean":"2024-02-05T08:17:57.392239+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:17:57.392239+0000","last_undegraded":"2024-02-05T08:17:57.392239+0000","last_fullsized":"2024-02-05T08:17:57.392239+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:12.127605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:12.127605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:31.758118+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":229,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.399754+0000","last_change":"2024-02-05T08:17:14.087240+0000","last_active":"2024-02-05T08:17:57.399754+0000","last_peered":"2024-02-05T08:17:57.399754+0000","last_clean":"2024-02-05T08:17:57.399754+0000","last_became_active":"2024-02-05T08:12:29.678501+0000","last_became_peered":"2024-02-05T08:12:29.678501+0000","last_unstale":"2024-02-05T08:17:57.399754+0000","last_undegraded":"2024-02-05T08:17:57.399754+0000","last_fullsized":"2024-02-05T08:17:57.399754+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:14.087110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:17:14.087110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:32.049970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"236'31","reported_seq":536,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.393464+0000","last_change":"2024-02-05T08:16:58.775056+0000","last_active":"2024-02-05T08:17:57.393464+0000","last_peered":"2024-02-05T08:17:57.393464+0000","last_clean":"2024-02-05T08:17:57.393464+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:17:57.393464+0000","last_undegraded":"2024-02-05T08:17:57.393464+0000","last_fullsized":"2024-02-05T08:17:57.393464+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'31","last_scrub_stamp":"2024-02-05T08:16:58.774921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:58.774921+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:17.545014+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"285'13399","reported_seq":13568,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.632128+0000","last_change":"2024-02-05T08:16:15.159977+0000","last_active":"2024-02-05T08:17:57.632128+0000","last_peered":"2024-02-05T08:17:57.632128+0000","last_clean":"2024-02-05T08:17:57.632128+0000","last_became_active":"2024-02-05T08:14:57.762652+0000","last_became_peered":"2024-02-05T08:14:57.762652+0000","last_unstale":"2024-02-05T08:17:57.632128+0000","last_undegraded":"2024-02-05T08:17:57.632128+0000","last_fullsized":"2024-02-05T08:17:57.632128+0000","mapping_epoch":239,"log_start":"270'8300","ondisk_log_start":"270'8300","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'5960","last_scrub_stamp":"2024-02-05T08:16:15.159720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:16:15.159720+0000","objects_scrubbed":5912,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1865,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198901760,"num_objects":3035,"num_object_clones":0,"num_object_copies":9105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13399,"num_write_kb":525888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":479,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.408861+0000","last_change":"2024-02-05T08:16:52.875421+0000","last_active":"2024-02-05T08:17:57.408861+0000","last_peered":"2024-02-05T08:17:57.408861+0000","last_clean":"2024-02-05T08:17:57.408861+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:17:57.408861+0000","last_undegraded":"2024-02-05T08:17:57.408861+0000","last_fullsized":"2024-02-05T08:17:57.408861+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:52.874790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:16:52.874790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:04.742008+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":668,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.390246+0000","last_change":"2024-02-05T08:16:53.367751+0000","last_active":"2024-02-05T08:17:57.390246+0000","last_peered":"2024-02-05T08:17:57.390246+0000","last_clean":"2024-02-05T08:17:57.390246+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:17:57.390246+0000","last_undegraded":"2024-02-05T08:17:57.390246+0000","last_fullsized":"2024-02-05T08:17:57.390246+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:16:53.367552+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:16:53.367552+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:07.009993+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":203,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.398845+0000","last_change":"2024-02-05T08:15:29.437631+0000","last_active":"2024-02-05T08:17:57.398845+0000","last_peered":"2024-02-05T08:17:57.398845+0000","last_clean":"2024-02-05T08:17:57.398845+0000","last_became_active":"2024-02-05T08:12:41.757159+0000","last_became_peered":"2024-02-05T08:12:41.757159+0000","last_unstale":"2024-02-05T08:17:57.398845+0000","last_undegraded":"2024-02-05T08:17:57.398845+0000","last_fullsized":"2024-02-05T08:17:57.398845+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:29.437487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:29.437487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"236'20","reported_seq":571,"reported_epoch":285,"state":"active+clean","last_fresh":"2024-02-05T08:17:57.393008+0000","last_change":"2024-02-05T08:16:01.862035+0000","last_active":"2024-02-05T08:17:57.393008+0000","last_peered":"2024-02-05T08:17:57.393008+0000","last_clean":"2024-02-05T08:17:57.393008+0000","last_became_active":"2024-02-05T08:12:41.727271+0000","last_became_peered":"2024-02-05T08:12:41.727271+0000","last_unstale":"2024-02-05T08:17:57.393008+0000","last_undegraded":"2024-02-05T08:17:57.393008+0000","last_fullsized":"2024-02-05T08:17:57.393008+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:16:01.861532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:01.861532+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3150512152,"num_objects":48074,"num_object_clones":0,"num_object_copies":144222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48074,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215402,"num_write_kb":8431169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9470095360,"data_stored":9470083144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75149,"ondisk_log_size":75149,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":789,"num_read_kb":784,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":5288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1123,"num_read_kb":880,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":223,"num_bytes_recovered":4032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487424,"data_stored":12894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":764,"ondisk_log_size":764,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037924,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2609336,"kb_used_data":2131056,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":91135816,"statfs":{"total":95995035648,"available":93323075584,"internally_reserved":0,"allocated":2182201344,"data_stored":2181888822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":489683402},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":2.109,"5min":1.855,"15min":1.4950000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":27.376000000000001,"5min":52.488999999999997,"15min":60.127000000000002},"last":0.378},{"interface":"front","average":{"1min":2.782,"5min":2.0870000000000002,"15min":1.5640000000000001},"min":{"1min":0.27100000000000002,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":27.456,"5min":52.335000000000001,"15min":58.283000000000001},"last":0.27600000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":1.617,"15min":1.476},"min":{"1min":0.252,"5min":0.23400000000000001,"15min":0.22900000000000001},"max":{"1min":27.518999999999998,"5min":27.518999999999998,"15min":38.399999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.4890000000000001,"15min":1.333},"min":{"1min":0.27600000000000002,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":27.503,"5min":27.503,"15min":38.195999999999998},"last":2.8530000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.1559999999999999,"15min":0.95599999999999996},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.24099999999999999},"max":{"1min":3.238,"5min":7.5709999999999997,"15min":7.6779999999999999},"last":2.6150000000000002},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.226,"15min":1.0740000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.24299999999999999},"max":{"1min":3.9820000000000002,"5min":5.9400000000000004,"15min":8.9100000000000001},"last":0.55400000000000005}]},{"osd":3,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.117,"15min":1.284},"min":{"1min":0.29699999999999999,"5min":0.28199999999999997,"15min":0.249},"max":{"1min":4.0209999999999999,"5min":5.984,"15min":58.515000000000001},"last":2.4540000000000002},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.0880000000000001,"15min":1.2609999999999999},"min":{"1min":0.35799999999999998,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":4.0960000000000001,"5min":5.391,"15min":58.225999999999999},"last":2.3730000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.571,"15min":0.876},"min":{"1min":0.36799999999999999,"5min":0.26500000000000001,"15min":0.22},"max":{"1min":2.9670000000000001,"5min":49.601999999999997,"15min":49.601999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.617,"15min":0.90400000000000003},"min":{"1min":0.28199999999999997,"5min":0.28100000000000003,"15min":0.249},"max":{"1min":4.0510000000000002,"5min":49.795999999999999,"15min":49.795999999999999},"last":0.39400000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.9570000000000001,"15min":0.95899999999999996},"min":{"1min":0.32100000000000001,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":5.1769999999999996,"5min":49.914000000000001,"15min":49.914000000000001},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.617,"5min":1.879,"15min":0.94899999999999995},"min":{"1min":0.432,"5min":0.314,"15min":0.27000000000000002},"max":{"1min":7.2350000000000003,"5min":52.256,"15min":52.256},"last":0.72799999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.296,"15min":0.76000000000000001},"min":{"1min":0.28799999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.1840000000000002,"5min":8.0050000000000008,"15min":8.0050000000000008},"last":0.443},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.2210000000000001,"15min":0.73499999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.27300000000000002},"max":{"1min":3.9420000000000002,"5min":7.8029999999999999,"15min":7.8029999999999999},"last":0.58599999999999997}]}]},{"osd":6,"up_from":51,"seq":219043332266,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":861308,"kb_used_data":587188,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92883844,"statfs":{"total":95995035648,"available":95113056256,"internally_reserved":0,"allocated":601280512,"data_stored":600954974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":280689094},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":4.1929999999999996,"5min":3.2610000000000001,"15min":1.657},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":76.123000000000005,"5min":102.819,"15min":102.819},"last":0.33300000000000002},{"interface":"front","average":{"1min":4.2969999999999997,"5min":3.0819999999999999,"15min":1.724},"min":{"1min":0.249,"5min":0.214,"15min":0.214},"max":{"1min":76.037000000000006,"5min":102.75,"15min":102.75},"last":0.48799999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.6439999999999999,"15min":1.1299999999999999},"min":{"1min":0.30599999999999999,"5min":0.24199999999999999,"15min":0.22700000000000001},"max":{"1min":3.2200000000000002,"5min":23.773,"15min":23.773},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":1.552,"15min":1.0920000000000001},"min":{"1min":0.28999999999999998,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":4.4509999999999996,"5min":23.719000000000001,"15min":23.719000000000001},"last":0.42299999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.22,"15min":1.016},"min":{"1min":0.27700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.524,"5min":22.503,"15min":22.503},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.96299999999999997,"5min":1.0409999999999999,"15min":0.93799999999999994},"min":{"1min":0.27300000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.1720000000000002,"5min":22.466000000000001,"15min":22.466000000000001},"last":0.52200000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":1.905,"15min":1.288},"min":{"1min":0.33600000000000002,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":2.75,"5min":102.438,"15min":102.438},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.216,"5min":2.0089999999999999,"15min":1.3080000000000001},"min":{"1min":0.32200000000000001,"5min":0.316,"15min":0.29099999999999998},"max":{"1min":4.4939999999999998,"5min":102.288,"15min":102.288},"last":0.47399999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.95099999999999996,"15min":0.60499999999999998},"min":{"1min":0.29199999999999998,"5min":0.26700000000000002,"15min":0.223},"max":{"1min":3.141,"5min":9.8970000000000002,"15min":9.8970000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.93100000000000005,"15min":0.62},"min":{"1min":0.316,"5min":0.28599999999999998,"15min":0.26600000000000001},"max":{"1min":3.4079999999999999,"5min":4.9589999999999996,"15min":4.9589999999999996},"last":0.36599999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.7649999999999999,"15min":0.92700000000000005},"min":{"1min":0.29299999999999998,"5min":0.25,"15min":0.24199999999999999},"max":{"1min":3.7709999999999999,"5min":43.643000000000001,"15min":43.643000000000001},"last":0.378},{"interface":"front","average":{"1min":1.083,"5min":1.641,"15min":0.91100000000000003},"min":{"1min":0.376,"5min":0.307,"15min":0.30399999999999999},"max":{"1min":3.617,"5min":43.893000000000001,"15min":43.893000000000001},"last":0.63400000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:16:59 2024","interfaces":[{"interface":"back","average":{"1min":3.2530000000000001,"5min":1.2949999999999999,"15min":0.77900000000000003},"min":{"1min":0.376,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":24.728999999999999,"5min":24.728999999999999,"15min":24.728999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":2.6589999999999998,"5min":1.2010000000000001,"15min":0.75800000000000001},"min":{"1min":0.375,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":21.91,"5min":21.91,"15min":21.91},"last":0.99099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575043,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2273232,"kb_used_data":1746116,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91471920,"statfs":{"total":95995035648,"available":93667246080,"internally_reserved":0,"allocated":1788022784,"data_stored":1787688946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:17:16 2024","interfaces":[{"interface":"back","average":{"1min":2.8250000000000002,"5min":1.716,"15min":1.5800000000000001},"min":{"1min":0.219,"5min":0.19900000000000001,"15min":0.14699999999999999},"max":{"1min":36.462000000000003,"5min":36.462000000000003,"15min":86.105000000000004},"last":0.435},{"interface":"front","average":{"1min":2.593,"5min":1.8480000000000001,"15min":1.8220000000000001},"min":{"1min":0.21099999999999999,"5min":0.20699999999999999,"15min":0.16},"max":{"1min":35.878,"5min":35.878,"15min":85.665000000000006},"last":0.53000000000000003}]},{"osd":2,"last update":"Mon Feb 5 08:17:50 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":1.298,"15min":1.3620000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.17799999999999999},"max":{"1min":3.5299999999999998,"5min":19.016999999999999,"15min":49.030999999999999},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.5249999999999999,"15min":1.544},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":2.351,"5min":20.434000000000001,"15min":48.984000000000002},"last":0.45100000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:17:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.546,"15min":1.488},"min":{"1min":0.222,"5min":0.222,"15min":0.20599999999999999},"max":{"1min":5.3289999999999997,"5min":31.896000000000001,"15min":49.067999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.78300000000000003,"5min":1.304,"15min":1.427},"min":{"1min":0.27200000000000002,"5min":0.27100000000000002,"15min":0.24199999999999999},"max":{"1min":2.8839999999999999,"5min":32.018999999999998,"15min":48.734999999999999},"last":0.373}]},{"osd":4,"last update":"Mon Feb 5 08:17:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.706,"15min":1.1359999999999999},"min":{"1min":0.36699999999999999,"5min":0.33100000000000002,"15min":0.26000000000000001},"max":{"1min":3.0190000000000001,"5min":25.952000000000002,"15min":25.952000000000002},"last":24.196999999999999},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.762,"15min":1.232},"min":{"1min":0.38500000000000001,"5min":0.31,"15min":0.27100000000000002},"max":{"1min":3.613,"5min":25.922000000000001,"15min":25.922000000000001},"last":24.215}]},{"osd":5,"last update":"Mon Feb 5 08:17:56 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":2.2050000000000001,"15min":1.361},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.28699999999999998},"max":{"1min":4.8200000000000003,"5min":54.552999999999997,"15min":54.552999999999997},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.298,"5min":2.0840000000000001,"15min":1.387},"min":{"1min":0.311,"5min":0.311,"15min":0.28999999999999998},"max":{"1min":2.972,"5min":55.811,"15min":55.811},"last":0.33100000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:17:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.6040000000000001,"15min":1.1930000000000001},"min":{"1min":0.35599999999999998,"5min":0.35199999999999998,"15min":0.33100000000000002},"max":{"1min":3.6819999999999999,"5min":54.384,"15min":54.384},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.159,"5min":1.8129999999999999,"15min":1.22},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.34899999999999998},"max":{"1min":3.5419999999999998,"5min":55.210999999999999,"15min":55.210999999999999},"last":0.51700000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:17:50 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.5329999999999999,"15min":1.2470000000000001},"min":{"1min":0.34300000000000003,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":4.2400000000000002,"5min":7.9390000000000001,"15min":13.749000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.444,"5min":1.474,"15min":1.2110000000000001},"min":{"1min":0.44800000000000001,"5min":0.379,"15min":0.379},"max":{"1min":3.601,"5min":7.9640000000000004,"15min":7.9640000000000004},"last":0.80100000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738568,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1658040,"kb_used_data":1149296,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92087112,"statfs":{"total":95995035648,"available":94297202688,"internally_reserved":0,"allocated":1176879104,"data_stored":1176566301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":1.2909999999999999,"15min":1.76},"min":{"1min":0.189,"5min":0.17499999999999999,"15min":0.152},"max":{"1min":2.5600000000000001,"5min":15.539,"15min":66.772999999999996},"last":0.874},{"interface":"front","average":{"1min":0.623,"5min":1.4730000000000001,"15min":1.4810000000000001},"min":{"1min":0.23300000000000001,"5min":0.18099999999999999,"15min":0.17199999999999999},"max":{"1min":1.9790000000000001,"5min":15.327999999999999,"15min":46.198999999999998},"last":0.72599999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:17:49 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.4590000000000001,"15min":1.496},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.649,"5min":12.074,"15min":46.170999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.3500000000000001,"15min":1.625},"min":{"1min":0.26100000000000001,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":2.2890000000000001,"5min":15.569000000000001,"15min":46.543999999999997},"last":0.73899999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.323,"15min":1.657},"min":{"1min":0.27100000000000002,"5min":0.26100000000000001,"15min":0.21099999999999999},"max":{"1min":2.722,"5min":7.3929999999999998,"15min":67.888000000000005},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.5349999999999999,"15min":1.956},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.22600000000000001},"max":{"1min":1.6910000000000001,"5min":15.494,"15min":66.891999999999996},"last":0.84399999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:17:42 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.667,"15min":1.2190000000000001},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":2.6739999999999999,"5min":22.920000000000002,"15min":22.920000000000002},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.4750000000000001,"15min":1.1930000000000001},"min":{"1min":0.39400000000000002,"5min":0.33800000000000002,"15min":0.26900000000000002},"max":{"1min":2.7610000000000001,"5min":22.983000000000001,"15min":22.983000000000001},"last":0.76100000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:17:03 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.6990000000000001,"15min":1.2809999999999999},"min":{"1min":0.38900000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.2939999999999996,"5min":22.800999999999998,"15min":22.800999999999998},"last":3.996},{"interface":"front","average":{"1min":1.49,"5min":1.631,"15min":1.2030000000000001},"min":{"1min":0.44900000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":3.8050000000000002,"5min":22.744,"15min":22.744},"last":3.9780000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.3080000000000001,"15min":1.1419999999999999},"min":{"1min":0.36799999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.79,"5min":7.8220000000000001,"15min":12.967000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.335,"5min":1.429,"15min":1.228},"min":{"1min":0.34200000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.1210000000000004,"5min":15.721,"15min":15.721},"last":3.6619999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:17:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":2.552,"15min":1.54},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.9820000000000002,"5min":68.241,"15min":68.241},"last":0.505},{"interface":"front","average":{"1min":1.3839999999999999,"5min":2.395,"15min":1.577},"min":{"1min":0.40400000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":3.1339999999999999,"5min":68.980000000000004,"15min":68.980000000000004},"last":3.7050000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411518,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1470656,"kb_used_data":984116,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92274496,"statfs":{"total":95995035648,"available":94489083904,"internally_reserved":0,"allocated":1007734784,"data_stored":1007417014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":498203078},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":2.3450000000000002,"15min":2.024},"min":{"1min":0.20499999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.638,"5min":107.155,"15min":107.155},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":2.6360000000000001,"15min":2.173},"min":{"1min":0.22600000000000001,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":10.782999999999999,"5min":107.221,"15min":107.221},"last":0.36799999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.054,"15min":1.446},"min":{"1min":0.23599999999999999,"5min":0.19400000000000001,"15min":0.17999999999999999},"max":{"1min":12.394,"5min":12.394,"15min":55.942},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.272,"5min":1.3029999999999999,"15min":1.5820000000000001},"min":{"1min":0.28799999999999998,"5min":0.23400000000000001,"15min":0.193},"max":{"1min":10.752000000000001,"5min":10.871,"15min":55.960999999999999},"last":0.54100000000000004}]},{"osd":3,"last update":"Mon Feb 5 08:17:15 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.28,"15min":1.7490000000000001},"min":{"1min":0.30299999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":8.3629999999999995,"5min":8.6590000000000007,"15min":68.108000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":1.05,"15min":1.554},"min":{"1min":0.251,"5min":0.23999999999999999,"15min":0.22900000000000001},"max":{"1min":1.1699999999999999,"5min":6.8600000000000003,"15min":67.945999999999998},"last":0.30599999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:17:37 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.304,"15min":1.3300000000000001},"min":{"1min":0.317,"5min":0.28399999999999997,"15min":0.27400000000000002},"max":{"1min":3.5550000000000002,"5min":8.2620000000000005,"15min":31.596},"last":2.4780000000000002},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.2,"15min":1.1479999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.252},"max":{"1min":2.6030000000000002,"5min":6.0220000000000002,"15min":31.57},"last":0.32800000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:17:00 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.462,"15min":1.3380000000000001},"min":{"1min":0.38,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":7.077,"5min":7.077,"15min":58.799999999999997},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.095,"5min":1.3220000000000001,"15min":1.29},"min":{"1min":0.34000000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.4809999999999999,"5min":7.3170000000000002,"15min":58.615000000000002},"last":2.5449999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.2549999999999999,"15min":1.1890000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.2999999999999998,"5min":8.3089999999999993,"15min":15.859},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.999,"5min":1.1319999999999999,"15min":1.0249999999999999},"min":{"1min":0.41399999999999998,"5min":0.35899999999999999,"15min":0.32900000000000001},"max":{"1min":4.0140000000000002,"5min":6.0570000000000004,"15min":6.1820000000000004},"last":3.4860000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.2270000000000001,"15min":1.194},"min":{"1min":0.34699999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":7.7530000000000001,"5min":7.7530000000000001,"15min":15.986000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.649,"5min":1.587,"15min":1.3580000000000001},"min":{"1min":0.503,"5min":0.41699999999999998,"15min":0.35299999999999998},"max":{"1min":4.2160000000000002,"5min":8.3369999999999997,"15min":15.951000000000001},"last":0.70499999999999996}]}]},{"osd":3,"up_from":0,"seq":115964117177,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":32,"seq":137438953652,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1189748,"kb_used_data":775400,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92555404,"statfs":{"total":95995035648,"available":94776733696,"internally_reserved":0,"allocated":794009600,"data_stored":793703239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":1.7969999999999999,"15min":1.5369999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20799999999999999},"max":{"1min":3.1669999999999998,"5min":39.552,"15min":73.078999999999994},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.861,"15min":1.5349999999999999},"min":{"1min":0.249,"5min":0.214,"15min":0.21099999999999999},"max":{"1min":5.625,"5min":39.350999999999999,"15min":73.412000000000006},"last":4.1239999999999997}]},{"osd":1,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":2.6869999999999998,"15min":1.4470000000000001},"min":{"1min":0.252,"5min":0.24299999999999999,"15min":0.21099999999999999},"max":{"1min":3.371,"5min":125.054,"15min":125.054},"last":3.3690000000000002},{"interface":"front","average":{"1min":0.82099999999999995,"5min":2.7570000000000001,"15min":1.3700000000000001},"min":{"1min":0.28499999999999998,"5min":0.26100000000000001,"15min":0.23499999999999999},"max":{"1min":3.2240000000000002,"5min":125.08799999999999,"15min":125.08799999999999},"last":1.371}]},{"osd":2,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":1.0840000000000001,"15min":1.2170000000000001},"min":{"1min":0.26500000000000001,"5min":0.247,"15min":0.24199999999999999},"max":{"1min":3.198,"5min":4.6040000000000001,"15min":96.849999999999994},"last":4.8159999999999998},{"interface":"front","average":{"1min":1.248,"5min":1.236,"15min":1.288},"min":{"1min":0.28499999999999998,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":5.6879999999999997,"5min":5.6879999999999997,"15min":96.143000000000001},"last":4.0940000000000003}]},{"osd":3,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.5660000000000001,"15min":1.776},"min":{"1min":0.33200000000000002,"5min":0.28599999999999998,"15min":0.24399999999999999},"max":{"1min":5.5460000000000003,"5min":38.985999999999997,"15min":73.602999999999994},"last":4.0670000000000002},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.4850000000000001,"15min":1.782},"min":{"1min":0.309,"5min":0.309,"15min":0.27200000000000002},"max":{"1min":3.3079999999999998,"5min":39.290999999999997,"15min":74.817999999999998},"last":3.4350000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:17:08 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.4650000000000001,"15min":0.76300000000000001},"min":{"1min":0.29399999999999998,"5min":0.25800000000000001,"15min":0.224},"max":{"1min":9.157,"5min":34.167000000000002,"15min":34.167000000000002},"last":3.754},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.546,"15min":0.81200000000000006},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":5.6550000000000002,"5min":36.051000000000002,"15min":36.051000000000002},"last":3.9990000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:17:26 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.028,"15min":0.65000000000000002},"min":{"1min":0.35799999999999998,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":5.5910000000000002,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":4.0380000000000003},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.91100000000000003,"15min":0.61199999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.3450000000000002,"5min":5.5049999999999999,"15min":5.5049999999999999},"last":3.411}]},{"osd":7,"last update":"Mon Feb 5 08:17:43 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.1950000000000001,"15min":0.72099999999999997},"min":{"1min":0.38,"5min":0.314,"15min":0.27000000000000002},"max":{"1min":4.8129999999999997,"5min":4.8129999999999997,"15min":4.8129999999999997},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.0820000000000001,"15min":0.70199999999999996},"min":{"1min":0.40500000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.3940000000000001,"5min":5.7450000000000001,"15min":5.7450000000000001},"last":4.6529999999999996}]}]},{"osd":5,"up_from":42,"seq":180388626606,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2480352,"kb_used_data":1975768,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":91264800,"statfs":{"total":95995035648,"available":93455155200,"internally_reserved":0,"allocated":2023186432,"data_stored":2022889210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":3.0779999999999998,"5min":2.0630000000000002,"15min":1.9119999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.19400000000000001},"max":{"1min":30.065999999999999,"5min":30.065999999999999,"15min":143.06299999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":2.831,"5min":1.806,"15min":1.8340000000000001},"min":{"1min":0.308,"5min":0.24299999999999999,"15min":0.20499999999999999},"max":{"1min":30.173999999999999,"5min":30.173999999999999,"15min":143.09700000000001},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.889,"5min":1.4359999999999999,"15min":1.1299999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.215},"max":{"1min":4.141,"5min":5.9000000000000004,"15min":46.200000000000003},"last":0.501},{"interface":"front","average":{"1min":1.675,"5min":1.1799999999999999,"15min":1.196},"min":{"1min":0.28699999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.2210000000000001,"5min":5.2210000000000001,"15min":46.343000000000004},"last":0.35499999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.2769999999999999,"15min":1.0449999999999999},"min":{"1min":0.34100000000000003,"5min":0.28999999999999998,"15min":0.246},"max":{"1min":7.1159999999999997,"5min":7.1159999999999997,"15min":10.733000000000001},"last":1.5229999999999999},{"interface":"front","average":{"1min":2.008,"5min":1.476,"15min":1.143},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.79,"5min":5.8250000000000002,"15min":8.0220000000000002},"last":0.52300000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6950000000000001,"5min":1.1990000000000001,"15min":1.6839999999999999},"min":{"1min":0.34699999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":7.165,"5min":7.165,"15min":143.03399999999999},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.748,"5min":1.581,"15min":1.806},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.28100000000000003},"max":{"1min":4.1929999999999996,"5min":7.0309999999999997,"15min":143.46799999999999},"last":0.58099999999999996}]},{"osd":4,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.3640000000000001,"15min":0.749},"min":{"1min":0.40000000000000002,"5min":0.24399999999999999,"15min":0.23699999999999999},"max":{"1min":5.0819999999999999,"5min":5.3769999999999998,"15min":5.3769999999999998},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.6220000000000001,"5min":1.248,"15min":0.71099999999999997},"min":{"1min":0.29899999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":7.2380000000000004,"5min":7.2380000000000004,"15min":7.2380000000000004},"last":1.4890000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:17:27 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.198,"15min":0.71099999999999997},"min":{"1min":0.35799999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":7.5439999999999996,"5min":7.5439999999999996,"15min":7.5439999999999996},"last":2.3170000000000002},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.1830000000000001,"15min":0.73299999999999998},"min":{"1min":0.38100000000000001,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":7.194,"5min":7.194,"15min":7.194},"last":1.6910000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.371,"5min":1.6819999999999999,"15min":0.89800000000000002},"min":{"1min":0.41399999999999998,"5min":0.30399999999999999,"15min":0.25900000000000001},"max":{"1min":7.0330000000000004,"5min":9.6940000000000008,"15min":9.6940000000000008},"last":2.2679999999999998},{"interface":"front","average":{"1min":2.274,"5min":1.5149999999999999,"15min":0.84699999999999998},"min":{"1min":0.65000000000000002,"5min":0.318,"15min":0.28199999999999997},"max":{"1min":5.2670000000000003,"5min":8.0289999999999999,"15min":8.0289999999999999},"last":1.0029999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1169686528,"data_stored":1169686528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1770524672,"data_stored":1770520600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":988872704,"data_stored":988872704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":789250048,"data_stored":789250048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1977290752,"data_stored":1977286680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":595197952,"data_stored":595197952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2179272704,"data_stored":2179268632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:18:06.184 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-05T08:18:06.185 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:18:06.185 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-02-05T08:18:06.185 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-05T08:18:06.185 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-05T08:18:06.185 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-02-05T08:18:06.188 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:06.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:06.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:06.335 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:06.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:06.437 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:06.478 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:06.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:06.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:06 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:18:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:06 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:18:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:06 smithi196 ceph-mon[33018]: 6.7 scrub starts 2024-02-05T08:18:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:06 smithi196 ceph-mon[33018]: 6.7 scrub ok 2024-02-05T08:18:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:06 smithi196 ceph-mon[33018]: 4.1b scrub starts 2024-02-05T08:18:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:06 smithi196 ceph-mon[33018]: 4.1b scrub ok 2024-02-05T08:18:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:06 smithi196 ceph-mon[33018]: osdmap e286: 8 total, 8 up, 7 in 2024-02-05T08:18:06.580 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:06.592 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:18:06.616 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:06.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:06.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:06.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:06.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:06.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:06.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:06.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:18:06.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:18:06.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[30086]: 6.7 scrub starts 2024-02-05T08:18:06.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[30086]: 6.7 scrub ok 2024-02-05T08:18:06.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[30086]: 4.1b scrub starts 2024-02-05T08:18:06.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[30086]: 4.1b scrub ok 2024-02-05T08:18:06.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[30086]: osdmap e286: 8 total, 8 up, 7 in 2024-02-05T08:18:06.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:18:06.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:18:06.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[39278]: 6.7 scrub starts 2024-02-05T08:18:06.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[39278]: 6.7 scrub ok 2024-02-05T08:18:06.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[39278]: 4.1b scrub starts 2024-02-05T08:18:06.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[39278]: 4.1b scrub ok 2024-02-05T08:18:06.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:06 smithi134 ceph-mon[39278]: osdmap e286: 8 total, 8 up, 7 in 2024-02-05T08:18:06.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:06.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:06.898 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:06.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:06.999 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:07.034 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:07.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:07.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:07.136 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:07.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:07.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:07.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:07.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:07.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:07.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:07.417 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:07.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:07.458 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:07.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:07.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:07.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:07 smithi196 ceph-mon[33018]: pgmap v588: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T08:18:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:07 smithi196 ceph-mon[33018]: 3.1d scrub starts 2024-02-05T08:18:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:07 smithi196 ceph-mon[33018]: 3.1d scrub ok 2024-02-05T08:18:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:07 smithi196 ceph-mon[33018]: 4.1a scrub starts 2024-02-05T08:18:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:07 smithi196 ceph-mon[33018]: 4.1a scrub ok 2024-02-05T08:18:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:07 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:18:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:07 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:18:07.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:07.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:07.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:07.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:07.731 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:07.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:07.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[30086]: pgmap v588: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T08:18:07.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[30086]: 3.1d scrub starts 2024-02-05T08:18:07.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[30086]: 3.1d scrub ok 2024-02-05T08:18:07.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[30086]: 4.1a scrub starts 2024-02-05T08:18:07.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[30086]: 4.1a scrub ok 2024-02-05T08:18:07.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:18:07.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:18:07.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[39278]: pgmap v588: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T08:18:07.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[39278]: 3.1d scrub starts 2024-02-05T08:18:07.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[39278]: 3.1d scrub ok 2024-02-05T08:18:07.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[39278]: 4.1a scrub starts 2024-02-05T08:18:07.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[39278]: 4.1a scrub ok 2024-02-05T08:18:07.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:18:07.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:18:07.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:07.873 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:07.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:07.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:07.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:08.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:08.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:08.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:18:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:18:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:18:08.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:08.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:08.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:08.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:08.249 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:08.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:08.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:08.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:08.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:08.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:08.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:08.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:08.557 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:08.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:08.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:08.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:08 smithi196 ceph-mon[33018]: 6.1b scrub starts 2024-02-05T08:18:08.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:08 smithi196 ceph-mon[33018]: 6.1b scrub ok 2024-02-05T08:18:08.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:08 smithi196 ceph-mon[33018]: 5.16 scrub starts 2024-02-05T08:18:08.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:08 smithi196 ceph-mon[33018]: 5.16 scrub ok 2024-02-05T08:18:08.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:08 smithi196 ceph-mon[33018]: osdmap e287: 8 total, 8 up, 7 in 2024-02-05T08:18:08.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:18:08.586 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:18:08.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[30086]: 6.1b scrub starts 2024-02-05T08:18:08.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[30086]: 6.1b scrub ok 2024-02-05T08:18:08.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[30086]: 5.16 scrub starts 2024-02-05T08:18:08.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[30086]: 5.16 scrub ok 2024-02-05T08:18:08.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[30086]: osdmap e287: 8 total, 8 up, 7 in 2024-02-05T08:18:08.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:18:08.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:18:08.651 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[39278]: 6.1b scrub starts 2024-02-05T08:18:08.652 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[39278]: 6.1b scrub ok 2024-02-05T08:18:08.652 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[39278]: 5.16 scrub starts 2024-02-05T08:18:08.652 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[39278]: 5.16 scrub ok 2024-02-05T08:18:08.652 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[39278]: osdmap e287: 8 total, 8 up, 7 in 2024-02-05T08:18:08.652 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:18:08.652 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:18:08.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:08.697 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:08.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:08.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:08.725 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:18:08.725 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:18:08.725 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-02-05T08:18:08.725 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:18:08.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:08.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:08.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:08.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:08.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:08.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:08.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:08.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:09.081 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:09.096 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:18:09.096 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "" 2024-02-05T08:18:09.096 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:18:09.113 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:09.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:09.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:09.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:09.249 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:09.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:09.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:09.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:09.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:09.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:09.490 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:18:09.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:09.504 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:09 smithi134 ceph-mon[30086]: pgmap v591: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 814 op/s 2024-02-05T08:18:09.504 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:09 smithi134 ceph-mon[30086]: 4.1f scrub starts 2024-02-05T08:18:09.504 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:09 smithi134 ceph-mon[30086]: 4.1f scrub ok 2024-02-05T08:18:09.504 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:09 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3717767344' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:18:09.504 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:09 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:18:09.505 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:09 smithi134 ceph-mon[39278]: pgmap v591: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 814 op/s 2024-02-05T08:18:09.505 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:09 smithi134 ceph-mon[39278]: 4.1f scrub starts 2024-02-05T08:18:09.505 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:09 smithi134 ceph-mon[39278]: 4.1f scrub ok 2024-02-05T08:18:09.505 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:09 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3717767344' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:18:09.505 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:09 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:18:09.529 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:09.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:09.631 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:09.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T08:18:09.666 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:09.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:09.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:09.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:09 smithi196 ceph-mon[33018]: pgmap v591: 148 pgs: 148 active+clean; 2.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 814 op/s 2024-02-05T08:18:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:09 smithi196 ceph-mon[33018]: 4.1f scrub starts 2024-02-05T08:18:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:09 smithi196 ceph-mon[33018]: 4.1f scrub ok 2024-02-05T08:18:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:09 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3717767344' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:18:09.777 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:09 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:18:09.807 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:09.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:09.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:09.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:09.956 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:09.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:10.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:10.092 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:10.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:10.193 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:10.231 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:10.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:10.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:10.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:10.364 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:10.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:10.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:10.466 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:10.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:10.501 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:10.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:10.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:10.607 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:18:10.607 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:18:10.607 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:18:10.607 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:18:10.607 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[39278]: 3.11 scrub starts 2024-02-05T08:18:10.607 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[39278]: 3.11 scrub ok 2024-02-05T08:18:10.607 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[39278]: 4.8 scrub starts 2024-02-05T08:18:10.607 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[39278]: 4.8 scrub ok 2024-02-05T08:18:10.607 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:10.607 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:18:10.607 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[39278]: osdmap e288: 8 total, 8 up, 7 in 2024-02-05T08:18:10.608 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:18:10.608 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:18:10.608 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:18:10.608 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:18:10.608 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[30086]: 3.11 scrub starts 2024-02-05T08:18:10.608 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[30086]: 3.11 scrub ok 2024-02-05T08:18:10.608 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[30086]: 4.8 scrub starts 2024-02-05T08:18:10.608 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[30086]: 4.8 scrub ok 2024-02-05T08:18:10.608 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:10.608 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:18:10.608 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:10 smithi134 ceph-mon[30086]: osdmap e288: 8 total, 8 up, 7 in 2024-02-05T08:18:10.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:10.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:10.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:10.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:10.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:10.777 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:10.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:10 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:18:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:10 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:18:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:10 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:18:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:10 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:18:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:10 smithi196 ceph-mon[33018]: 3.11 scrub starts 2024-02-05T08:18:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:10 smithi196 ceph-mon[33018]: 3.11 scrub ok 2024-02-05T08:18:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:10 smithi196 ceph-mon[33018]: 4.8 scrub starts 2024-02-05T08:18:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:10 smithi196 ceph-mon[33018]: 4.8 scrub ok 2024-02-05T08:18:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:10 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:10 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:18:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:10 smithi196 ceph-mon[33018]: osdmap e288: 8 total, 8 up, 7 in 2024-02-05T08:18:10.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:10.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:10.921 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:10.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:10.952 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:10.991 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:18:11.022 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:11.029 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:18:11.061 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:11.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:11.069 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:18:11.101 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:18:11.140 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:18:11.163 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:11.169 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:18:11.199 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:18:11.200 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:11.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:11.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:11.244 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:18:11.249 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:18:11.249 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:18:11.249 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:18:11.249 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:18:11.249 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:18:11.249 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:18:11.249 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:18:11.249 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:18:11.302 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:11.340 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:11.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:11.441 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:11.471 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:11.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:11.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:11.607 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:11.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:11.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:11.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:11.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:11.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:11.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:11 smithi134 ceph-mon[30086]: pgmap v593: 148 pgs: 148 active+clean; 2.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T08:18:11.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:11 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/433438214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:11.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:11 smithi134 ceph-mon[39278]: pgmap v593: 148 pgs: 148 active+clean; 2.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T08:18:11.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:11 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/433438214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:11 smithi196 ceph-mon[33018]: pgmap v593: 148 pgs: 148 active+clean; 2.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T08:18:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:11 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/433438214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:11.845 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:11.879 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:11.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:11.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:11.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:12.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:12.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:12.117 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:12.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:12.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:12.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:12.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:12.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:12.286 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:12.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:12.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:12.422 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:12.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:12.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:12.557 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:12.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:12.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:18:12.659 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:12.695 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:12.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:12.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:12.797 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:12.838 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:12.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:12.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:12.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:12.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:12.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:13.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:13.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:13.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:13.218 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:13.249 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:13.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:13.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:13.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:13.413 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:13.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:13.514 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:13.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:13.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:13.641 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:13 smithi134 ceph-mon[30086]: pgmap v594: 148 pgs: 148 active+clean; 2.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-02-05T08:18:13.641 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:13 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:18:13.641 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:13 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:18:13.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:13 smithi134 ceph-mon[39278]: pgmap v594: 148 pgs: 148 active+clean; 2.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-02-05T08:18:13.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:13 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:18:13.641 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:13 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:18:13.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:13.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:13.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:13.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:13.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:13.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:13 smithi196 ceph-mon[33018]: pgmap v594: 148 pgs: 148 active+clean; 2.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.07k op/s 2024-02-05T08:18:13.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:13 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:18:13.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:13 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:18:13.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:18:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:18:13.822 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:13.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:13.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:13.934 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:13.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:13.979 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:13.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:13.989 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:18:13.989 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:18:14.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:14.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:14.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:14.222 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:14.257 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:14.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:14.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:14.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:14.400 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:14.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:14.492 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:18:14.500 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:14.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:14 smithi196 ceph-mon[33018]: osdmap e289: 8 total, 8 up, 7 in 2024-02-05T08:18:14.568 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:14.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:14.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:14.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:14.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:14.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:14 smithi134 ceph-mon[30086]: osdmap e289: 8 total, 8 up, 7 in 2024-02-05T08:18:14.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:14 smithi134 ceph-mon[39278]: osdmap e289: 8 total, 8 up, 7 in 2024-02-05T08:18:14.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:14.828 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:18:14.828 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:18:14.828 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-02-05T08:18:14.828 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-05T08:18:14.828 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-05T08:18:14.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-02-05T08:18:14.839 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:14.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:14.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:14.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:14.970 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:14.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:14.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:15.035 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T08:18:15.072 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:15.114 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:15.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:15.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:15.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:15.254 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:15.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:15.266 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:18:15.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:15.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:15.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:15.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:15.498 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:15.533 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:15.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:15.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:15.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:15 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:15.582 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:15 smithi196 ceph-mon[33018]: pgmap v596: 148 pgs: 148 active+clean; 1.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.78k op/s 2024-02-05T08:18:15.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:15.674 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:15.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:15.775 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:15.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:15 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:15.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:15 smithi134 ceph-mon[30086]: pgmap v596: 148 pgs: 148 active+clean; 1.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.78k op/s 2024-02-05T08:18:15.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:15 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:15.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:15 smithi134 ceph-mon[39278]: pgmap v596: 148 pgs: 148 active+clean; 1.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.78k op/s 2024-02-05T08:18:15.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:15.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:15.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:15.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:15.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:15.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:16.056 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:16.101 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:16.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:16.146 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:16.156 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:16.201 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:16.249 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:16.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:16.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:16.386 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:16.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:16.488 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:16.519 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:16.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:16.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:16.621 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:16.650 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:16.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:16.751 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:16.785 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:16.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:16.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:16.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:16.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:16.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:17.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:17.077 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:17.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:17.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:17.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:17.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:17.222 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:17.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:17.276 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:18:17.276 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "" 2024-02-05T08:18:17.276 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:18:17.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:17.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:17.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:17.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:17.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:17.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:17.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:17.594 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:17.630 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:17.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:17.641 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:17 smithi196 ceph-mon[33018]: pgmap v597: 148 pgs: 148 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-02-05T08:18:17.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:17.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:17.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:17.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:17 smithi134 ceph-mon[30086]: pgmap v597: 148 pgs: 148 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-02-05T08:18:17.776 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:18:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:18:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:18:17.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:17 smithi134 ceph-mon[39278]: pgmap v597: 148 pgs: 148 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-02-05T08:18:17.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:17.899 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:17.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:17.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:18.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:18.032 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:18.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:18.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:18.168 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:18.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:18.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:18.270 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:18.304 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:18.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:18.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:18.406 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:18.446 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:18.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:18.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:18.542 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:18:18.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:18.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:18.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:18.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:18.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:18.725 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:18.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:18.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:18 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/669535319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:18 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:18.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:18.862 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:18.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:18.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:18.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:18.995 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:18.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:19.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:18 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/669535319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:19.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:18 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:19.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:18 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/669535319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:19.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:18 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:19.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:19.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:19.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:19.233 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:19.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:19.269 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:19.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:19.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:19.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:19.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:19.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:19.546 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:19.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:19.647 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:19.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:19.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:19.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:19.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:19 smithi196 ceph-mon[33018]: pgmap v598: 148 pgs: 148 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 618 B/s rd, 0 B/s wr, 820 op/s 2024-02-05T08:18:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:19 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:19 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:18:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:19 smithi196 ceph-mon[33018]: osdmap e290: 8 total, 8 up, 7 in 2024-02-05T08:18:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:19.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:19.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:19.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:19.945 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:19.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:20.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:19 smithi134 ceph-mon[30086]: pgmap v598: 148 pgs: 148 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 618 B/s rd, 0 B/s wr, 820 op/s 2024-02-05T08:18:20.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:19 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:19 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:18:20.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:19 smithi134 ceph-mon[30086]: osdmap e290: 8 total, 8 up, 7 in 2024-02-05T08:18:20.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:19 smithi134 ceph-mon[39278]: pgmap v598: 148 pgs: 148 active+clean; 1.8 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 618 B/s rd, 0 B/s wr, 820 op/s 2024-02-05T08:18:20.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:19 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:20.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:19 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:18:20.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:19 smithi134 ceph-mon[39278]: osdmap e290: 8 total, 8 up, 7 in 2024-02-05T08:18:20.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:20.087 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:20.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:20.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:20.189 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:20.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:20.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:20.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:20.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:20.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:20.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:20.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:20.497 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:20.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:20.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:20.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:20.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:20.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:20.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:20.742 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:20.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:20.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:20.884 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:20.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:20.921 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:20.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:21.022 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:21.057 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:21.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:21.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:21.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:21.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:21.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:21.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:21.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:21.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:21.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:21.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:21.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:21.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:21.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:21.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:21.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:21.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:21.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:21.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:21.741 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:21.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:21.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:21 smithi196 ceph-mon[33018]: pgmap v600: 148 pgs: 148 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-02-05T08:18:21.826 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:21 smithi134 ceph-mon[30086]: pgmap v600: 148 pgs: 148 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-02-05T08:18:21.826 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:21 smithi134 ceph-mon[39278]: pgmap v600: 148 pgs: 148 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-02-05T08:18:21.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:21.874 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:21.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:21.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:22.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:22.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:22.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:22.145 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:22.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:22.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:22.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:22.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:22.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:22.330 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:18:22.330 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:18:22.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:22.415 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:22.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:22.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:22.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:22.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:22.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:22.651 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:22.687 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:22.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:22.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:22.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:22.828 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:22.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:22.930 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:22.964 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:22.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:22.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:23.050 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:18:23.050 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:18:23.050 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-02-05T08:18:23.050 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd in 3 2024-02-05T08:18:23.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:23.132 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:23.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:23.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:23.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:23.274 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:23.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:23.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:23.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:23.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:23.512 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:23.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:23.548 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:23.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:23.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:23.685 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:23.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:23.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:23.705 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:18:23.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:23.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:18:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:18:23.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:23 smithi196 ceph-mon[33018]: pgmap v601: 148 pgs: 148 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 620 B/s rd, 0 B/s wr, 931 op/s 2024-02-05T08:18:23.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:23 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:18:23.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:23 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:18:23.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:23 smithi196 ceph-mon[33018]: osdmap e291: 8 total, 8 up, 7 in 2024-02-05T08:18:23.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:18:23.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:18:23.816 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:23.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:23.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:23.918 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:23.958 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:23.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:23.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:23 smithi134 ceph-mon[30086]: pgmap v601: 148 pgs: 148 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 620 B/s rd, 0 B/s wr, 931 op/s 2024-02-05T08:18:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:23 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:18:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:23 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:18:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:23 smithi134 ceph-mon[30086]: osdmap e291: 8 total, 8 up, 7 in 2024-02-05T08:18:24.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:18:24.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:18:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:23 smithi134 ceph-mon[39278]: pgmap v601: 148 pgs: 148 active+clean; 1.4 GiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 620 B/s rd, 0 B/s wr, 931 op/s 2024-02-05T08:18:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:23 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:18:24.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:23 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:18:24.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:23 smithi134 ceph-mon[39278]: osdmap e291: 8 total, 8 up, 7 in 2024-02-05T08:18:24.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:18:24.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:18:24.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:24.093 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:24.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:24.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:24.227 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:24.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:24.329 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:24.365 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:24.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:24.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:24.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:24.502 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:24.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:24.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:24.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:24.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:24.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:24.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:24.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:24.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:24.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:24.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:24.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:24.910 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:24.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:24.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:25.011 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:25.044 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:25.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:25.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:25.182 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:25.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:25.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:25.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:25.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:25.425 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:25.459 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T08:18:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:25.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:25.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:25.494 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:25.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:25.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:25.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:25.667 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:25 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:25.668 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:25 smithi134 ceph-mon[30086]: pgmap v603: 148 pgs: 148 active+clean; 1.1 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T08:18:25.668 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:25 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:25.668 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:25 smithi134 ceph-mon[39278]: pgmap v603: 148 pgs: 148 active+clean; 1.1 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T08:18:25.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:25.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:25.755 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:25.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:25 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:25.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:25 smithi196 ceph-mon[33018]: pgmap v603: 148 pgs: 148 active+clean; 1.1 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T08:18:25.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:25.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:25.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:25.922 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:18:26.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:26.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:26.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:26.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:26.170 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:18:26.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:26.186 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:26.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:26.190 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:18:26.209 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:18:26.229 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:18:26.256 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:18:26.286 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:18:26.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:26.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:26.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:26.326 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:18:26.369 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:18:26.374 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:18:26.374 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:18:26.374 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:18:26.374 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:18:26.374 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:18:26.374 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:18:26.374 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:18:26.374 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:18:26.421 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:26.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:26.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:26.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:26.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:26.578 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:26.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:26.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:26.679 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:26.706 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:26.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:26.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:26 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:18:26.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:26 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:18:26.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:26 smithi134 ceph-mon[30086]: osdmap e292: 8 total, 8 up, 7 in 2024-02-05T08:18:26.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:26 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3168540497' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:26.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:26 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:18:26.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:26 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:18:26.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:26 smithi134 ceph-mon[39278]: osdmap e292: 8 total, 8 up, 7 in 2024-02-05T08:18:26.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:26 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3168540497' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:26.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:26.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:26.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:26.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:26.943 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:26.967 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:26.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:26 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:18:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:26 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:18:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:26 smithi196 ceph-mon[33018]: osdmap e292: 8 total, 8 up, 7 in 2024-02-05T08:18:27.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:26 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3168540497' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:27.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:27.102 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:27.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:27.204 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:27.237 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:27.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:27.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:27.339 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:27.375 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:27.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:27.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:27.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:27.515 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:27.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:27.616 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:27.648 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:27.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:27.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:27.722 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:27 smithi134 ceph-mon[30086]: pgmap v604: 148 pgs: 148 active+clean; 1.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-02-05T08:18:27.723 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:27 smithi134 ceph-mon[39278]: pgmap v604: 148 pgs: 148 active+clean; 1.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-02-05T08:18:27.750 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:27.791 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:27.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:27.892 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:27.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:27.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:27.956 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:28.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:18:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:18:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:18:28.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:28.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:27 smithi196 ceph-mon[33018]: pgmap v604: 148 pgs: 148 active+clean; 1.0 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-02-05T08:18:28.079 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:28.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:28.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:28.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:28.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:28.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:28.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:28.313 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:28.347 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:28.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:28.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:28.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:28.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:28.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:28.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:28.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:28.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:28.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:28.722 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:28.762 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:28.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:28.863 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:28.904 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:28.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:29.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:29.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:29.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:29.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:29.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:29.177 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:29.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:29.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:29.279 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:29.310 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:29.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:29.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:29.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:29.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:29.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:29.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:29.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:29.580 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:29.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:29.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:29.607 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:18:29.649 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:29 smithi196 ceph-mon[33018]: pgmap v606: 148 pgs: 148 active+clean; 1011 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 864 op/s 2024-02-05T08:18:29.650 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:29 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2326201483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:18:29.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:29.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:29.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:29.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:29.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:29.871 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:29.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:29.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:29.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:30.023 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:30.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:30.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:30.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:29 smithi134 ceph-mon[30086]: pgmap v606: 148 pgs: 148 active+clean; 1011 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 864 op/s 2024-02-05T08:18:30.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:29 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2326201483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:18:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:29 smithi134 ceph-mon[39278]: pgmap v606: 148 pgs: 148 active+clean; 1011 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 864 op/s 2024-02-05T08:18:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:29 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2326201483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:18:30.124 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:30.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:30.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:30.189 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:18:30.189 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:18:30.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.189 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:255125 9.e e293/239 9:71b0743d:::benchmark_data_smithi134_144349_object123385:head v 293'15814, mlcod=293'15811)", 2024-02-05T08:18:30.189 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:18:30.181576+0000", 2024-02-05T08:18:30.189 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0076010510000000002, 2024-02-05T08:18:30.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0076324870000000003, 2024-02-05T08:18:30.190 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:30.190 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:30.190 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:30.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:30.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.181576+0000", 2024-02-05T08:18:30.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:30.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:30.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.181576+0000", 2024-02-05T08:18:30.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:30.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:30.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.181577+0000", 2024-02-05T08:18:30.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1060000000000001e-06 2024-02-05T08:18:30.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:30.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.181584+0000", 2024-02-05T08:18:30.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7360000000000004e-06 2024-02-05T08:18:30.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:30.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.181601+0000", 2024-02-05T08:18:30.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7207999999999999e-05 2024-02-05T08:18:30.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:30.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.181610+0000", 2024-02-05T08:18:30.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0499999999999997e-06 2024-02-05T08:18:30.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:30.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.183707+0000", 2024-02-05T08:18:30.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020968990000000002 2024-02-05T08:18:30.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:30.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.183762+0000", 2024-02-05T08:18:30.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4688999999999998e-05 2024-02-05T08:18:30.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:30.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.189103+0000", 2024-02-05T08:18:30.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053410669999999997 2024-02-05T08:18:30.194 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:30.194 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:30.194 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:30.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.194 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:255130 9.e e293/239 9:7e261eeb:::benchmark_data_smithi134_144349_object123390:head v 293'15815, mlcod=293'15811)", 2024-02-05T08:18:30.194 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:18:30.184282+0000", 2024-02-05T08:18:30.194 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0048955090000000001, 2024-02-05T08:18:30.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050274129999999997, 2024-02-05T08:18:30.195 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:30.195 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:30.195 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:30.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:30.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.184282+0000", 2024-02-05T08:18:30.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:30.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:30.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.184282+0000", 2024-02-05T08:18:30.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:30.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:30.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.184283+0000", 2024-02-05T08:18:30.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4199999999999995e-07 2024-02-05T08:18:30.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:30.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.184290+0000", 2024-02-05T08:18:30.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4309999999999997e-06 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.184307+0000", 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6752000000000002e-05 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.184315+0000", 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4889999999999995e-06 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:30.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.186593+0000", 2024-02-05T08:18:30.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002277753 2024-02-05T08:18:30.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:30.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.186638+0000", 2024-02-05T08:18:30.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4629999999999998e-05 2024-02-05T08:18:30.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:30.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:30.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:30.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:18:30.189300+0000", 2024-02-05T08:18:30.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026619730000000002 2024-02-05T08:18:30.198 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:30.199 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:30.199 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:30.199 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:30.199 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-02-05T08:18:30.199 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 2 2024-02-05T08:18:30.199 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:18:30.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:30.297 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:30.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:30.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:30.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:30.437 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:30.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:30.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:18:30.538 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:30.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:30.621 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:30.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:30.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:30.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:30.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:30.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:30.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:30 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:30.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:30 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2326201483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:18:30.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:30 smithi134 ceph-mon[30086]: osdmap e293: 8 total, 8 up, 7 in 2024-02-05T08:18:30.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:30 smithi134 ceph-mon[30086]: 5.d scrub starts 2024-02-05T08:18:30.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:30 smithi134 ceph-mon[30086]: 5.d scrub ok 2024-02-05T08:18:30.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:30 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:30.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:30 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2326201483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:18:30.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:30 smithi134 ceph-mon[39278]: osdmap e293: 8 total, 8 up, 7 in 2024-02-05T08:18:30.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:30 smithi134 ceph-mon[39278]: 5.d scrub starts 2024-02-05T08:18:30.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:30 smithi134 ceph-mon[39278]: 5.d scrub ok 2024-02-05T08:18:30.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:30.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:30.895 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:30.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:30.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:31.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:31.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:31.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:30 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:30 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2326201483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:18:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:30 smithi196 ceph-mon[33018]: osdmap e293: 8 total, 8 up, 7 in 2024-02-05T08:18:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:30 smithi196 ceph-mon[33018]: 5.d scrub starts 2024-02-05T08:18:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:30 smithi196 ceph-mon[33018]: 5.d scrub ok 2024-02-05T08:18:31.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:31.173 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:31.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:31.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T08:18:31.275 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:31.306 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:31.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:31.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:31.407 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:31.437 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:31.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:31.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:31.591 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:31.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:31.684 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:18:31.693 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:31.723 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:31.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:31.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:31.825 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:31.856 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:31.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:31.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:31.860 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:31 smithi196 ceph-mon[33018]: pgmap v608: 148 pgs: 148 active+clean; 667 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-02-05T08:18:31.942 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:31 smithi134 ceph-mon[30086]: pgmap v608: 148 pgs: 148 active+clean; 667 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-02-05T08:18:31.942 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:31 smithi134 ceph-mon[39278]: pgmap v608: 148 pgs: 148 active+clean; 667 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-02-05T08:18:31.957 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:31.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:31.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:32.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:32.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:32.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:32.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:32.236 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:32.268 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:32.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:32.349 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:32.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:32.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:32.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:32.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:32.546 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:32.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:32.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:32.648 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:32.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:32.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:32.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:32.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:32 smithi134 ceph-mon[30086]: 6.b scrub starts 2024-02-05T08:18:32.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:32 smithi134 ceph-mon[30086]: 6.b scrub ok 2024-02-05T08:18:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:32 smithi134 ceph-mon[39278]: 6.b scrub starts 2024-02-05T08:18:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:32 smithi134 ceph-mon[39278]: 6.b scrub ok 2024-02-05T08:18:32.782 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:32.818 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:32.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:32.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:32.920 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:32.959 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:32.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:33.061 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:33.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:32 smithi196 ceph-mon[33018]: 6.b scrub starts 2024-02-05T08:18:33.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:32 smithi196 ceph-mon[33018]: 6.b scrub ok 2024-02-05T08:18:33.097 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:33.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:33.116 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:33 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a[30063]: 2024-02-05T08:18:33.005+0000 7f4523bf3700 -1 mon.a@0(leader).osd e293 definitely_dead 0 2024-02-05T08:18:33.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:33.248 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:33.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:33.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:33.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:33.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:33.496 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:33.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:33.534 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:33.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:33.614 INFO:teuthology.orchestra.run.smithi134.stderr:marked in osd.3. 2024-02-05T08:18:33.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:33.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:33.673 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:33.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:33.773 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:33.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:33 smithi134 ceph-mon[30086]: pgmap v609: 148 pgs: 148 active+clean; 667 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 936 op/s 2024-02-05T08:18:33.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:33 smithi134 ceph-mon[30086]: 6.19 scrub starts 2024-02-05T08:18:33.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:33 smithi134 ceph-mon[30086]: 6.19 scrub ok 2024-02-05T08:18:33.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:33 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:18:33.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:33 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:18:33.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:33 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1849892610' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-05T08:18:33.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:33 smithi134 ceph-mon[39278]: pgmap v609: 148 pgs: 148 active+clean; 667 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 936 op/s 2024-02-05T08:18:33.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:33 smithi134 ceph-mon[39278]: 6.19 scrub starts 2024-02-05T08:18:33.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:33 smithi134 ceph-mon[39278]: 6.19 scrub ok 2024-02-05T08:18:33.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:33 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:18:33.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:33 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:18:33.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:33 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1849892610' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-05T08:18:33.795 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:33 smithi196 ceph-mon[33018]: pgmap v609: 148 pgs: 148 active+clean; 667 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 936 op/s 2024-02-05T08:18:33.795 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:33 smithi196 ceph-mon[33018]: 6.19 scrub starts 2024-02-05T08:18:33.795 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:33 smithi196 ceph-mon[33018]: 6.19 scrub ok 2024-02-05T08:18:33.796 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:33 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:18:33.796 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:33 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:18:33.796 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:33 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1849892610' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-02-05T08:18:33.796 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:18:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:18:33.797 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:33.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:33.899 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:33.921 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:18:33.921 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:18:33.921 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T08:18:33.921 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T08:18:33.921 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:18:33.923 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:33.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:33.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:33.986 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:18:34.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:34.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:34.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:34.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:34.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:34.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:34.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:34.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:34.300 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:34.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:34.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:34.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:34.416 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-02-05T08:18:34.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:34.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:34.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:34.550 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:34.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:34.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:34.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T08:18:34.642 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[30086]: 4.11 scrub starts 2024-02-05T08:18:34.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:34.703 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:34.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:34.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:34.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:34 smithi196 ceph-mon[33018]: 4.11 scrub starts 2024-02-05T08:18:34.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:34 smithi196 ceph-mon[33018]: 4.11 scrub ok 2024-02-05T08:18:34.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:34 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1849892610' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-05T08:18:34.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:34 smithi196 ceph-mon[33018]: osdmap e294: 8 total, 8 up, 8 in 2024-02-05T08:18:34.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:34 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:34.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:34 smithi196 ceph-mon[33018]: 5.8 scrub starts 2024-02-05T08:18:34.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:34 smithi196 ceph-mon[33018]: 5.8 scrub ok 2024-02-05T08:18:34.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:34 smithi196 ceph-mon[33018]: pgmap v611: 148 pgs: 148 active+clean; 378 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-02-05T08:18:34.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:34 smithi196 ceph-mon[33018]: osdmap e295: 8 total, 8 up, 8 in 2024-02-05T08:18:34.830 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:34.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:34.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:34.831 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:18:34.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:34.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:34.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:34.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:34.971 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[30086]: 4.11 scrub ok 2024-02-05T08:18:34.971 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1849892610' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-05T08:18:34.971 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[30086]: osdmap e294: 8 total, 8 up, 8 in 2024-02-05T08:18:34.971 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:34.971 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[30086]: 5.8 scrub starts 2024-02-05T08:18:34.971 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[30086]: 5.8 scrub ok 2024-02-05T08:18:34.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[30086]: pgmap v611: 148 pgs: 148 active+clean; 378 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-02-05T08:18:34.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[30086]: osdmap e295: 8 total, 8 up, 8 in 2024-02-05T08:18:34.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[39278]: 4.11 scrub starts 2024-02-05T08:18:34.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[39278]: 4.11 scrub ok 2024-02-05T08:18:34.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1849892610' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-02-05T08:18:34.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[39278]: osdmap e294: 8 total, 8 up, 8 in 2024-02-05T08:18:34.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:34.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[39278]: 5.8 scrub starts 2024-02-05T08:18:34.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[39278]: 5.8 scrub ok 2024-02-05T08:18:34.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[39278]: pgmap v611: 148 pgs: 148 active+clean; 378 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-02-05T08:18:34.972 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:34 smithi134 ceph-mon[39278]: osdmap e295: 8 total, 8 up, 8 in 2024-02-05T08:18:35.007 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:18:35.017 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:35.027 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:18:35.048 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:18:35.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:35.069 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:18:35.081 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:35.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:35.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:35.089 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:18:35.109 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:18:35.129 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:18:35.148 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:18:35.150 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:18:35.150 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:18:35.150 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:18:35.150 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:18:35.150 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:18:35.150 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:18:35.150 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:18:35.150 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:18:35.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:35.206 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:35.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:35.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:35.332 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:35.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:35.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:35.434 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:35.458 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:35.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:35.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:35.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:35.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:35.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:35.685 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:35.755 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:35.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:35.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:35.857 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:35.881 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:35.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:35.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:35.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:35.989 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[30086]: 5.19 scrub starts 2024-02-05T08:18:35.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[30086]: 5.19 scrub ok 2024-02-05T08:18:35.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[30086]: 6.d scrub starts 2024-02-05T08:18:35.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[30086]: 6.d scrub ok 2024-02-05T08:18:35.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[30086]: 3.b scrub starts 2024-02-05T08:18:35.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[30086]: 3.b scrub ok 2024-02-05T08:18:35.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/222651087' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:35.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[30086]: 6.5 scrub starts 2024-02-05T08:18:35.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[30086]: 6.5 scrub ok 2024-02-05T08:18:35.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[30086]: 9.8 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16063 2024-02-05T08:18:35.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[30086]: 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 293'20337 2024-02-05T08:18:35.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[30086]: osdmap e296: 8 total, 8 up, 8 in 2024-02-05T08:18:35.991 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[30086]: 9.3 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16344 2024-02-05T08:18:35.991 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[39278]: 5.19 scrub starts 2024-02-05T08:18:35.991 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[39278]: 5.19 scrub ok 2024-02-05T08:18:35.991 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[39278]: 6.d scrub starts 2024-02-05T08:18:35.991 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[39278]: 6.d scrub ok 2024-02-05T08:18:35.991 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[39278]: 3.b scrub starts 2024-02-05T08:18:35.991 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[39278]: 3.b scrub ok 2024-02-05T08:18:35.991 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/222651087' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:35.991 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[39278]: 6.5 scrub starts 2024-02-05T08:18:35.991 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[39278]: 6.5 scrub ok 2024-02-05T08:18:35.991 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[39278]: 9.8 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16063 2024-02-05T08:18:35.992 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[39278]: 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 293'20337 2024-02-05T08:18:35.992 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[39278]: osdmap e296: 8 total, 8 up, 8 in 2024-02-05T08:18:35.992 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:35 smithi134 ceph-mon[39278]: 9.3 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16344 2024-02-05T08:18:36.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:36.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:36.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:36.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:35 smithi196 ceph-mon[33018]: 5.19 scrub starts 2024-02-05T08:18:36.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:35 smithi196 ceph-mon[33018]: 5.19 scrub ok 2024-02-05T08:18:36.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:35 smithi196 ceph-mon[33018]: 6.d scrub starts 2024-02-05T08:18:36.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:35 smithi196 ceph-mon[33018]: 6.d scrub ok 2024-02-05T08:18:36.023 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:35 smithi196 ceph-mon[33018]: 3.b scrub starts 2024-02-05T08:18:36.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:35 smithi196 ceph-mon[33018]: 3.b scrub ok 2024-02-05T08:18:36.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:35 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/222651087' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:36.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:35 smithi196 ceph-mon[33018]: 6.5 scrub starts 2024-02-05T08:18:36.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:35 smithi196 ceph-mon[33018]: 6.5 scrub ok 2024-02-05T08:18:36.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:35 smithi196 ceph-mon[33018]: 9.8 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16063 2024-02-05T08:18:36.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:35 smithi196 ceph-mon[33018]: 9.c starting backfill to osd.3 from (0'0,0'0] MAX to 293'20337 2024-02-05T08:18:36.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:35 smithi196 ceph-mon[33018]: osdmap e296: 8 total, 8 up, 8 in 2024-02-05T08:18:36.024 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:35 smithi196 ceph-mon[33018]: 9.3 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16344 2024-02-05T08:18:36.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:36.168 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:36.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:36.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:36.270 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:36.311 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:36.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:36.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:36.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:36.451 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:36.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:36.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:36.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:36.590 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:36.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:36.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:36.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:36.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:36.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:36.798 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:18:36.798 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:18:36.837 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:36.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:36 smithi196 ceph-mon[33018]: 5.1d scrub starts 2024-02-05T08:18:36.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:36 smithi196 ceph-mon[33018]: 5.1d scrub ok 2024-02-05T08:18:36.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:36 smithi196 ceph-mon[33018]: 9.2 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16144 2024-02-05T08:18:36.864 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:36 smithi196 ceph-mon[33018]: 9.7 starting backfill to osd.3 from (0'0,0'0] MAX to 293'15946 2024-02-05T08:18:36.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:36 smithi196 ceph-mon[33018]: 9.4 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16412 2024-02-05T08:18:36.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:36 smithi196 ceph-mon[33018]: 9.1 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16180 2024-02-05T08:18:36.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:36 smithi196 ceph-mon[33018]: 9.f starting backfill to osd.3 from (0'0,0'0] MAX to 293'15924 2024-02-05T08:18:36.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:36 smithi196 ceph-mon[33018]: pgmap v614: 148 pgs: 7 peering, 141 active+clean; 309 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s 2024-02-05T08:18:36.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:36 smithi196 ceph-mon[33018]: 6.3 scrub starts 2024-02-05T08:18:36.865 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:36 smithi196 ceph-mon[33018]: 6.3 scrub ok 2024-02-05T08:18:36.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:36.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:36.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:37.013 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:37.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:37.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[30086]: 5.1d scrub starts 2024-02-05T08:18:37.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[30086]: 5.1d scrub ok 2024-02-05T08:18:37.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[30086]: 9.2 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16144 2024-02-05T08:18:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[30086]: 9.7 starting backfill to osd.3 from (0'0,0'0] MAX to 293'15946 2024-02-05T08:18:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[30086]: 9.4 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16412 2024-02-05T08:18:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[30086]: 9.1 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16180 2024-02-05T08:18:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[30086]: 9.f starting backfill to osd.3 from (0'0,0'0] MAX to 293'15924 2024-02-05T08:18:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[30086]: pgmap v614: 148 pgs: 7 peering, 141 active+clean; 309 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s 2024-02-05T08:18:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[30086]: 6.3 scrub starts 2024-02-05T08:18:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[30086]: 6.3 scrub ok 2024-02-05T08:18:37.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[39278]: 5.1d scrub starts 2024-02-05T08:18:37.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[39278]: 5.1d scrub ok 2024-02-05T08:18:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[39278]: 9.2 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16144 2024-02-05T08:18:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[39278]: 9.7 starting backfill to osd.3 from (0'0,0'0] MAX to 293'15946 2024-02-05T08:18:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[39278]: 9.4 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16412 2024-02-05T08:18:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[39278]: 9.1 starting backfill to osd.3 from (0'0,0'0] MAX to 293'16180 2024-02-05T08:18:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[39278]: 9.f starting backfill to osd.3 from (0'0,0'0] MAX to 293'15924 2024-02-05T08:18:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[39278]: pgmap v614: 148 pgs: 7 peering, 141 active+clean; 309 MiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s 2024-02-05T08:18:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[39278]: 6.3 scrub starts 2024-02-05T08:18:37.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:36 smithi134 ceph-mon[39278]: 6.3 scrub ok 2024-02-05T08:18:37.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:37.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:37.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:37.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:37.284 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:37.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:37.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:37.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:37.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:37.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:37.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:37.558 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:37.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:37.585 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T08:18:37.660 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:37.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:37.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:37.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:37.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:37.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:37.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:37.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:37.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:37.972 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:18:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:37.994 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:37.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:38.014 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:18:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:18:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:18:38.014 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:37 smithi134 ceph-mon[30086]: 4.e scrub starts 2024-02-05T08:18:38.015 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:37 smithi134 ceph-mon[30086]: 4.e scrub ok 2024-02-05T08:18:38.015 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:37 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:18:38.015 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:37 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:18:38.015 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:37 smithi134 ceph-mon[30086]: 3.1 scrub starts 2024-02-05T08:18:38.015 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:37 smithi134 ceph-mon[30086]: 3.1 scrub ok 2024-02-05T08:18:38.015 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:37 smithi134 ceph-mon[39278]: 4.e scrub starts 2024-02-05T08:18:38.015 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:37 smithi134 ceph-mon[39278]: 4.e scrub ok 2024-02-05T08:18:38.015 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:37 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:18:38.015 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:37 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:18:38.015 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:37 smithi134 ceph-mon[39278]: 3.1 scrub starts 2024-02-05T08:18:38.015 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:37 smithi134 ceph-mon[39278]: 3.1 scrub ok 2024-02-05T08:18:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:37 smithi196 ceph-mon[33018]: 4.e scrub starts 2024-02-05T08:18:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:37 smithi196 ceph-mon[33018]: 4.e scrub ok 2024-02-05T08:18:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:37 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:18:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:37 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:18:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:37 smithi196 ceph-mon[33018]: 3.1 scrub starts 2024-02-05T08:18:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:37 smithi196 ceph-mon[33018]: 3.1 scrub ok 2024-02-05T08:18:38.095 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:38.118 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:38.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:38.220 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:38.244 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:38.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:38.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:38.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:38.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:38.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:38.495 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:38.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:38.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:38.596 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:38.620 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:38.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:38.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:38.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:38.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:38.829 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:18:38.829 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: 4.5 scrub starts 2024-02-05T08:18:38.829 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: 4.5 scrub ok 2024-02-05T08:18:38.829 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: 3.4 scrub starts 2024-02-05T08:18:38.830 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: 3.4 scrub ok 2024-02-05T08:18:38.830 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: 3.14 scrub starts 2024-02-05T08:18:38.830 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: 3.14 scrub ok 2024-02-05T08:18:38.830 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: pgmap v615: 148 pgs: 7 peering, 141 active+clean; 309 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s 2024-02-05T08:18:38.830 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: osdmap e297: 8 total, 8 up, 8 in 2024-02-05T08:18:38.830 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: 6.11 scrub starts 2024-02-05T08:18:38.830 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: 6.11 scrub ok 2024-02-05T08:18:38.830 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: 5.10 scrub starts 2024-02-05T08:18:38.830 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: 5.10 scrub ok 2024-02-05T08:18:38.830 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:18:38.830 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1793616111' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:38.830 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: 4.5 scrub starts 2024-02-05T08:18:38.830 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: 4.5 scrub ok 2024-02-05T08:18:38.831 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: 3.4 scrub starts 2024-02-05T08:18:38.831 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: 3.4 scrub ok 2024-02-05T08:18:38.831 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: 3.14 scrub starts 2024-02-05T08:18:38.831 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: 3.14 scrub ok 2024-02-05T08:18:38.831 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: pgmap v615: 148 pgs: 7 peering, 141 active+clean; 309 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s 2024-02-05T08:18:38.831 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: osdmap e297: 8 total, 8 up, 8 in 2024-02-05T08:18:38.831 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: 6.11 scrub starts 2024-02-05T08:18:38.831 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: 6.11 scrub ok 2024-02-05T08:18:38.831 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: 5.10 scrub starts 2024-02-05T08:18:38.831 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: 5.10 scrub ok 2024-02-05T08:18:38.831 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:18:38.831 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:38 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1793616111' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:38.848 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:38.883 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:38.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:38.984 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:39.021 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:39.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:39.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: 4.5 scrub starts 2024-02-05T08:18:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: 4.5 scrub ok 2024-02-05T08:18:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: 3.4 scrub starts 2024-02-05T08:18:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: 3.4 scrub ok 2024-02-05T08:18:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: 3.14 scrub starts 2024-02-05T08:18:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: 3.14 scrub ok 2024-02-05T08:18:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: pgmap v615: 148 pgs: 7 peering, 141 active+clean; 309 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s 2024-02-05T08:18:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: osdmap e297: 8 total, 8 up, 8 in 2024-02-05T08:18:39.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: 6.11 scrub starts 2024-02-05T08:18:39.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: 6.11 scrub ok 2024-02-05T08:18:39.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: 5.10 scrub starts 2024-02-05T08:18:39.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: 5.10 scrub ok 2024-02-05T08:18:39.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:18:39.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:38 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1793616111' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:39.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:39.165 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:39.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:39.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:18:39.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:39.301 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:39.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:39.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:39.417 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:18:39.417 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:18:39.418 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-02-05T08:18:39.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:18:39.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:39.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:39.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:39.623 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:39.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:39.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:39.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:39.733 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:39.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:39.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:39 smithi196 ceph-mon[33018]: 9.0 scrub starts 2024-02-05T08:18:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:39 smithi196 ceph-mon[33018]: 9.0 scrub ok 2024-02-05T08:18:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:39 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:18:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:39 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:18:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:39 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1793616111' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:18:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:39 smithi196 ceph-mon[33018]: osdmap e298: 8 total, 8 up, 8 in 2024-02-05T08:18:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:39 smithi196 ceph-mon[33018]: 5.18 scrub starts 2024-02-05T08:18:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:39 smithi196 ceph-mon[33018]: 5.18 scrub ok 2024-02-05T08:18:39.860 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:39.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[39278]: 9.0 scrub starts 2024-02-05T08:18:39.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[39278]: 9.0 scrub ok 2024-02-05T08:18:39.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:18:39.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:18:39.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1793616111' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:18:39.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[39278]: osdmap e298: 8 total, 8 up, 8 in 2024-02-05T08:18:39.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[39278]: 5.18 scrub starts 2024-02-05T08:18:39.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[39278]: 5.18 scrub ok 2024-02-05T08:18:39.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[30086]: 9.0 scrub starts 2024-02-05T08:18:39.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[30086]: 9.0 scrub ok 2024-02-05T08:18:39.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:18:39.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:18:39.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1793616111' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:18:39.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[30086]: osdmap e298: 8 total, 8 up, 8 in 2024-02-05T08:18:39.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[30086]: 5.18 scrub starts 2024-02-05T08:18:39.890 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:39 smithi134 ceph-mon[30086]: 5.18 scrub ok 2024-02-05T08:18:39.896 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:39.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:39.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:39.997 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:40.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:40.039 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:40.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:40.139 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:40.184 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:40.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:40.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:40.286 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:40.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:40.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:40.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:40.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:40.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:40.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:40.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:40.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:40.592 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:40.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:40.605 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:40.609 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:18:40.610 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:18:40.694 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:40.731 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:40.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:40.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:40.806 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Removed 131737 objects 2024-02-05T08:18:40.806 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Clean up completed and total clean up time :116.308 2024-02-05T08:18:40.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:40.856 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:40.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:40.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:40.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:40.981 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:40.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:40.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:41.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[30086]: 2.1 deep-scrub starts 2024-02-05T08:18:41.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[30086]: 2.1 deep-scrub ok 2024-02-05T08:18:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[30086]: pgmap v618: 148 pgs: 3 peering, 5 active+remapped+backfill_toofull, 140 active+clean; 128 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 198 B/s rd, 0 B/s wr, 560 op/s; 651/6735 objects misplaced (9.666%); 6.9 MiB/s, 129 objects/s recovering 2024-02-05T08:18:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[30086]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:18:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[30086]: 6.13 scrub starts 2024-02-05T08:18:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[30086]: 6.13 scrub ok 2024-02-05T08:18:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[30086]: 6.16 scrub starts 2024-02-05T08:18:41.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[30086]: 6.16 scrub ok 2024-02-05T08:18:41.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[39278]: 2.1 deep-scrub starts 2024-02-05T08:18:41.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[39278]: 2.1 deep-scrub ok 2024-02-05T08:18:41.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[39278]: pgmap v618: 148 pgs: 3 peering, 5 active+remapped+backfill_toofull, 140 active+clean; 128 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 198 B/s rd, 0 B/s wr, 560 op/s; 651/6735 objects misplaced (9.666%); 6.9 MiB/s, 129 objects/s recovering 2024-02-05T08:18:41.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[39278]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:18:41.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[39278]: 6.13 scrub starts 2024-02-05T08:18:41.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[39278]: 6.13 scrub ok 2024-02-05T08:18:41.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[39278]: 6.16 scrub starts 2024-02-05T08:18:41.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:40 smithi134 ceph-mon[39278]: 6.16 scrub ok 2024-02-05T08:18:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:40 smithi196 ceph-mon[33018]: 2.1 deep-scrub starts 2024-02-05T08:18:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:40 smithi196 ceph-mon[33018]: 2.1 deep-scrub ok 2024-02-05T08:18:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:40 smithi196 ceph-mon[33018]: pgmap v618: 148 pgs: 3 peering, 5 active+remapped+backfill_toofull, 140 active+clean; 128 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 198 B/s rd, 0 B/s wr, 560 op/s; 651/6735 objects misplaced (9.666%); 6.9 MiB/s, 129 objects/s recovering 2024-02-05T08:18:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:40 smithi196 ceph-mon[33018]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:18:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:40 smithi196 ceph-mon[33018]: 6.13 scrub starts 2024-02-05T08:18:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:40 smithi196 ceph-mon[33018]: 6.13 scrub ok 2024-02-05T08:18:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:40 smithi196 ceph-mon[33018]: 6.16 scrub starts 2024-02-05T08:18:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:40 smithi196 ceph-mon[33018]: 6.16 scrub ok 2024-02-05T08:18:41.083 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:41.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:41.108 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:41.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:41.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:41.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:41.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:41.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:41.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T08:18:41.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:41.358 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:41.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:41.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:41.460 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:41.483 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:41.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:41.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:41.542 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:18:41.585 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:41.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:41.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:41.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:41.649 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:18:41.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:41.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:41.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:41.834 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:41.859 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:41.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:41.916 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:41 smithi196 ceph-mon[33018]: 4.f scrub starts 2024-02-05T08:18:41.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:41 smithi196 ceph-mon[33018]: 4.f scrub ok 2024-02-05T08:18:41.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:41 smithi196 ceph-mon[33018]: 3.3 scrub starts 2024-02-05T08:18:41.917 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:41 smithi196 ceph-mon[33018]: 3.3 scrub ok 2024-02-05T08:18:41.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:41.982 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:41.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:41.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:42.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:42.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:42.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:42.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:42.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:42.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:42.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:42.250 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:41 smithi134 ceph-mon[30086]: 4.f scrub starts 2024-02-05T08:18:42.250 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:41 smithi134 ceph-mon[30086]: 4.f scrub ok 2024-02-05T08:18:42.250 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:41 smithi134 ceph-mon[30086]: 3.3 scrub starts 2024-02-05T08:18:42.250 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:41 smithi134 ceph-mon[30086]: 3.3 scrub ok 2024-02-05T08:18:42.250 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:41 smithi134 ceph-mon[39278]: 4.f scrub starts 2024-02-05T08:18:42.250 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:41 smithi134 ceph-mon[39278]: 4.f scrub ok 2024-02-05T08:18:42.250 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:41 smithi134 ceph-mon[39278]: 3.3 scrub starts 2024-02-05T08:18:42.251 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:41 smithi134 ceph-mon[39278]: 3.3 scrub ok 2024-02-05T08:18:42.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:42.359 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:42.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:42.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:42.485 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:42.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:42.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:42.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:42.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:42.612 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:42.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:42.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:42.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:42.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:42.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:42.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:42.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:42.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:42.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:42.916 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[30086]: 3.16 scrub starts 2024-02-05T08:18:42.916 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[30086]: 3.16 scrub ok 2024-02-05T08:18:42.916 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:18:42.916 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:18:42.916 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[30086]: pgmap v619: 148 pgs: 3 peering, 5 active+remapped+backfill_toofull, 140 active+clean; 128 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 166 B/s rd, 0 B/s wr, 469 op/s; 651/6735 objects misplaced (9.666%); 5.8 MiB/s, 108 objects/s recovering 2024-02-05T08:18:42.916 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[30086]: 5.1 scrub starts 2024-02-05T08:18:42.916 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[30086]: 5.1 scrub ok 2024-02-05T08:18:42.916 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[39278]: 3.16 scrub starts 2024-02-05T08:18:42.917 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[39278]: 3.16 scrub ok 2024-02-05T08:18:42.917 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:18:42.917 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:18:42.917 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[39278]: pgmap v619: 148 pgs: 3 peering, 5 active+remapped+backfill_toofull, 140 active+clean; 128 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 166 B/s rd, 0 B/s wr, 469 op/s; 651/6735 objects misplaced (9.666%); 5.8 MiB/s, 108 objects/s recovering 2024-02-05T08:18:42.917 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[39278]: 5.1 scrub starts 2024-02-05T08:18:42.917 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:42 smithi134 ceph-mon[39278]: 5.1 scrub ok 2024-02-05T08:18:42.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:42.989 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:42.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:42 smithi196 ceph-mon[33018]: 3.16 scrub starts 2024-02-05T08:18:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:42 smithi196 ceph-mon[33018]: 3.16 scrub ok 2024-02-05T08:18:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:42 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:18:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:42 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:18:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:42 smithi196 ceph-mon[33018]: pgmap v619: 148 pgs: 3 peering, 5 active+remapped+backfill_toofull, 140 active+clean; 128 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 166 B/s rd, 0 B/s wr, 469 op/s; 651/6735 objects misplaced (9.666%); 5.8 MiB/s, 108 objects/s recovering 2024-02-05T08:18:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:42 smithi196 ceph-mon[33018]: 5.1 scrub starts 2024-02-05T08:18:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:42 smithi196 ceph-mon[33018]: 5.1 scrub ok 2024-02-05T08:18:43.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:43.115 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:43.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:43.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:43.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:43.241 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:43.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:43.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:43.361 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:18:43.361 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:18:43.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:43.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:43.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:43.406 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:18:43.428 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:18:43.448 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:18:43.467 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:18:43.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:43.487 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:18:43.492 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:43.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:43.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:43.507 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:18:43.528 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:18:43.548 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:18:43.549 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:18:43.550 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:18:43.550 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:18:43.550 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:18:43.550 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:18:43.550 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:18:43.550 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:18:43.550 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:18:43.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:43.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:43.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:43.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:43.719 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:43.744 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:43.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:43.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:43.834 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:18:43.837 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:18:43.838 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:43.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:43.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:43.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:43.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:43.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:43.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:44.056 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:18:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:18:44.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:43 smithi196 ceph-mon[33018]: 4.12 deep-scrub starts 2024-02-05T08:18:44.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:43 smithi196 ceph-mon[33018]: 4.12 deep-scrub ok 2024-02-05T08:18:44.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:43 smithi196 ceph-mon[33018]: 5.4 deep-scrub starts 2024-02-05T08:18:44.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:43 smithi196 ceph-mon[33018]: 5.4 deep-scrub ok 2024-02-05T08:18:44.056 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:43 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2933790462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:44.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T08:18:44.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:44.121 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:44.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:44.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:43 smithi134 ceph-mon[30086]: 4.12 deep-scrub starts 2024-02-05T08:18:44.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:43 smithi134 ceph-mon[30086]: 4.12 deep-scrub ok 2024-02-05T08:18:44.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:43 smithi134 ceph-mon[30086]: 5.4 deep-scrub starts 2024-02-05T08:18:44.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:43 smithi134 ceph-mon[30086]: 5.4 deep-scrub ok 2024-02-05T08:18:44.189 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:43 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2933790462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:44.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:43 smithi134 ceph-mon[39278]: 4.12 deep-scrub starts 2024-02-05T08:18:44.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:43 smithi134 ceph-mon[39278]: 4.12 deep-scrub ok 2024-02-05T08:18:44.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:43 smithi134 ceph-mon[39278]: 5.4 deep-scrub starts 2024-02-05T08:18:44.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:43 smithi134 ceph-mon[39278]: 5.4 deep-scrub ok 2024-02-05T08:18:44.190 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:43 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2933790462' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:44.223 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:44.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:44.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:44.326 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:18:44.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:44.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:44.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:44.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:44.500 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:44.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:44.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:44.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:44.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:44.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:44.728 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:44.752 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:44.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:44.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:44.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:44.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:44.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:44.918 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[30086]: 3.0 scrub starts 2024-02-05T08:18:44.918 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[30086]: 3.0 scrub ok 2024-02-05T08:18:44.918 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:44.918 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[30086]: pgmap v620: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 26 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 567 op/s; 4.5 MiB/s, 83 objects/s recovering 2024-02-05T08:18:44.918 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[30086]: from='client.25487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:18:44.918 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[30086]: 5.13 scrub starts 2024-02-05T08:18:44.919 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[30086]: 5.13 scrub ok 2024-02-05T08:18:44.919 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[30086]: 3.1b scrub starts 2024-02-05T08:18:44.919 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[30086]: 3.1b scrub ok 2024-02-05T08:18:44.919 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[39278]: 3.0 scrub starts 2024-02-05T08:18:44.919 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[39278]: 3.0 scrub ok 2024-02-05T08:18:44.919 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:44.919 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[39278]: pgmap v620: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 26 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 567 op/s; 4.5 MiB/s, 83 objects/s recovering 2024-02-05T08:18:44.919 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[39278]: from='client.25487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:18:44.919 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[39278]: 5.13 scrub starts 2024-02-05T08:18:44.919 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[39278]: 5.13 scrub ok 2024-02-05T08:18:44.919 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[39278]: 3.1b scrub starts 2024-02-05T08:18:44.919 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:44 smithi134 ceph-mon[39278]: 3.1b scrub ok 2024-02-05T08:18:44.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:45.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:45.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:45.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:44 smithi196 ceph-mon[33018]: 3.0 scrub starts 2024-02-05T08:18:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:44 smithi196 ceph-mon[33018]: 3.0 scrub ok 2024-02-05T08:18:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:44 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:44 smithi196 ceph-mon[33018]: pgmap v620: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 26 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 567 op/s; 4.5 MiB/s, 83 objects/s recovering 2024-02-05T08:18:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:44 smithi196 ceph-mon[33018]: from='client.25487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:18:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:44 smithi196 ceph-mon[33018]: 5.13 scrub starts 2024-02-05T08:18:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:44 smithi196 ceph-mon[33018]: 5.13 scrub ok 2024-02-05T08:18:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:44 smithi196 ceph-mon[33018]: 3.1b scrub starts 2024-02-05T08:18:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:44 smithi196 ceph-mon[33018]: 3.1b scrub ok 2024-02-05T08:18:45.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:45.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:45.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:45.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:45.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:45.254 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:45.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:45.355 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:45.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:45.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:45.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:45.481 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:45.506 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:45.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:45.608 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:45.633 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:45.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:45.734 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:45.758 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:45.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:45.841 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":620,"stamp":"2024-02-05T08:18:43.765696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":27007324,"num_objects":601,"num_object_clones":0,"num_object_copies":1803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":601,"num_whiteouts":0,"num_read":2183,"num_read_kb":2029,"num_write":263974,"num_write_kb":8433630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":914,"num_bytes_recovered":40380698,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78375,"ondisk_log_size":78375,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":450,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3786772,"kb_used_data":56912,"kb_used_omap":12,"kb_used_meta":3729779,"kb_avail":746174444,"statfs":{"total":767960285184,"available":764082630656,"internally_reserved":0,"allocated":58277888,"data_stored":55710109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":3819294259},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-296943640,"num_objects":-4532,"num_object_clones":0,"num_object_copies":-13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4532,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":4532,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":666,"num_bytes_recovered":37423013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002567"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":513,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.822920+0000","last_change":"2024-02-05T08:17:14.644253+0000","last_active":"2024-02-05T08:18:38.822920+0000","last_peered":"2024-02-05T08:18:38.822920+0000","last_clean":"2024-02-05T08:18:38.822920+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:18:38.822920+0000","last_undegraded":"2024-02-05T08:18:38.822920+0000","last_fullsized":"2024-02-05T08:18:38.822920+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:14.644144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:14.644144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":526,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.822555+0000","last_change":"2024-02-05T08:18:34.301578+0000","last_active":"2024-02-05T08:18:38.822555+0000","last_peered":"2024-02-05T08:18:38.822555+0000","last_clean":"2024-02-05T08:18:38.822555+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:18:38.822555+0000","last_undegraded":"2024-02-05T08:18:38.822555+0000","last_fullsized":"2024-02-05T08:18:38.822555+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.301502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.301502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:37.136866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":515,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.822669+0000","last_change":"2024-02-05T08:16:46.841747+0000","last_active":"2024-02-05T08:18:37.822669+0000","last_peered":"2024-02-05T08:18:37.822669+0000","last_clean":"2024-02-05T08:18:37.822669+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:18:37.822669+0000","last_undegraded":"2024-02-05T08:18:37.822669+0000","last_fullsized":"2024-02-05T08:18:37.822669+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:46.841644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:46.841644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"236'19","reported_seq":582,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.822707+0000","last_change":"2024-02-05T08:18:01.603030+0000","last_active":"2024-02-05T08:18:38.822707+0000","last_peered":"2024-02-05T08:18:38.822707+0000","last_clean":"2024-02-05T08:18:38.822707+0000","last_became_active":"2024-02-05T08:12:29.675863+0000","last_became_peered":"2024-02-05T08:12:29.675863+0000","last_unstale":"2024-02-05T08:18:38.822707+0000","last_undegraded":"2024-02-05T08:18:38.822707+0000","last_fullsized":"2024-02-05T08:18:38.822707+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:01.602887+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:18:01.602887+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:11.468872+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":17,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827258+0000","last_change":"2024-02-05T08:18:34.633406+0000","last_active":"2024-02-05T08:18:38.827258+0000","last_peered":"2024-02-05T08:18:38.827258+0000","last_clean":"2024-02-05T08:18:38.827258+0000","last_became_active":"2024-02-05T08:18:34.633083+0000","last_became_peered":"2024-02-05T08:18:34.633083+0000","last_unstale":"2024-02-05T08:18:38.827258+0000","last_undegraded":"2024-02-05T08:18:38.827258+0000","last_fullsized":"2024-02-05T08:18:38.827258+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:07.163073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:18:07.163073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:24.405136+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":529,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:39.536102+0000","last_change":"2024-02-05T08:18:39.536133+0000","last_active":"2024-02-05T08:18:39.536102+0000","last_peered":"2024-02-05T08:18:39.536102+0000","last_clean":"2024-02-05T08:18:39.536102+0000","last_became_active":"2024-02-05T08:12:29.676482+0000","last_became_peered":"2024-02-05T08:12:29.676482+0000","last_unstale":"2024-02-05T08:18:39.536102+0000","last_undegraded":"2024-02-05T08:18:39.536102+0000","last_fullsized":"2024-02-05T08:18:39.536102+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:39.536022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:18:39.536022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:45.419854+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827034+0000","last_change":"2024-02-05T08:18:34.668081+0000","last_active":"2024-02-05T08:18:38.827034+0000","last_peered":"2024-02-05T08:18:38.827034+0000","last_clean":"2024-02-05T08:18:38.827034+0000","last_became_active":"2024-02-05T08:18:34.667678+0000","last_became_peered":"2024-02-05T08:18:34.667678+0000","last_unstale":"2024-02-05T08:18:38.827034+0000","last_undegraded":"2024-02-05T08:18:38.827034+0000","last_fullsized":"2024-02-05T08:18:38.827034+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:57.294696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:16:57.294696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"236'34","reported_seq":33,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825906+0000","last_change":"2024-02-05T08:18:34.705538+0000","last_active":"2024-02-05T08:18:38.825906+0000","last_peered":"2024-02-05T08:18:38.825906+0000","last_clean":"2024-02-05T08:18:38.825906+0000","last_became_active":"2024-02-05T08:18:34.649433+0000","last_became_peered":"2024-02-05T08:18:34.649433+0000","last_unstale":"2024-02-05T08:18:38.825906+0000","last_undegraded":"2024-02-05T08:18:38.825906+0000","last_fullsized":"2024-02-05T08:18:38.825906+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:17:01.432518+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:17:01.432518+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":502,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.824766+0000","last_change":"2024-02-05T08:16:14.944841+0000","last_active":"2024-02-05T08:18:37.824766+0000","last_peered":"2024-02-05T08:18:37.824766+0000","last_clean":"2024-02-05T08:18:37.824766+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:18:37.824766+0000","last_undegraded":"2024-02-05T08:18:37.824766+0000","last_fullsized":"2024-02-05T08:18:37.824766+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:14.944722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:14.944722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":517,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825896+0000","last_change":"2024-02-05T08:17:51.448848+0000","last_active":"2024-02-05T08:18:38.825896+0000","last_peered":"2024-02-05T08:18:38.825896+0000","last_clean":"2024-02-05T08:18:38.825896+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:18:38.825896+0000","last_undegraded":"2024-02-05T08:18:38.825896+0000","last_fullsized":"2024-02-05T08:18:38.825896+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:51.448603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:17:51.448603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:18.582760+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":545,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826172+0000","last_change":"2024-02-05T08:18:06.405438+0000","last_active":"2024-02-05T08:18:38.826172+0000","last_peered":"2024-02-05T08:18:38.826172+0000","last_clean":"2024-02-05T08:18:38.826172+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:18:38.826172+0000","last_undegraded":"2024-02-05T08:18:38.826172+0000","last_fullsized":"2024-02-05T08:18:38.826172+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:06.403893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:11.809710+0000","last_clean_scrub_stamp":"2024-02-05T08:18:06.403893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:24.374178+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"263'19","reported_seq":588,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823044+0000","last_change":"2024-02-05T08:18:34.732396+0000","last_active":"2024-02-05T08:18:38.823044+0000","last_peered":"2024-02-05T08:18:38.823044+0000","last_clean":"2024-02-05T08:18:38.823044+0000","last_became_active":"2024-02-05T08:18:34.636165+0000","last_became_peered":"2024-02-05T08:18:34.636165+0000","last_unstale":"2024-02-05T08:18:38.823044+0000","last_undegraded":"2024-02-05T08:18:38.823044+0000","last_fullsized":"2024-02-05T08:18:38.823044+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:18:06.570361+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:18:06.570361+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:33.261052+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":525,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825675+0000","last_change":"2024-02-05T08:18:34.635235+0000","last_active":"2024-02-05T08:18:38.825675+0000","last_peered":"2024-02-05T08:18:38.825675+0000","last_clean":"2024-02-05T08:18:38.825675+0000","last_became_active":"2024-02-05T08:18:34.634934+0000","last_became_peered":"2024-02-05T08:18:34.634934+0000","last_unstale":"2024-02-05T08:18:38.825675+0000","last_undegraded":"2024-02-05T08:18:38.825675+0000","last_fullsized":"2024-02-05T08:18:38.825675+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:32.611856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:32.611856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:48.401031+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":518,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.861651+0000","last_change":"2024-02-05T08:18:04.094625+0000","last_active":"2024-02-05T08:18:38.861651+0000","last_peered":"2024-02-05T08:18:38.861651+0000","last_clean":"2024-02-05T08:18:38.861651+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:18:38.861651+0000","last_undegraded":"2024-02-05T08:18:38.861651+0000","last_fullsized":"2024-02-05T08:18:38.861651+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:04.094513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:04.094513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:10.106054+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":525,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825828+0000","last_change":"2024-02-05T08:17:08.551310+0000","last_active":"2024-02-05T08:18:38.825828+0000","last_peered":"2024-02-05T08:18:38.825828+0000","last_clean":"2024-02-05T08:18:38.825828+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:18:38.825828+0000","last_undegraded":"2024-02-05T08:18:38.825828+0000","last_fullsized":"2024-02-05T08:18:38.825828+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:08.551190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:17:08.551190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":558,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823354+0000","last_change":"2024-02-05T08:18:34.722007+0000","last_active":"2024-02-05T08:18:38.823354+0000","last_peered":"2024-02-05T08:18:38.823354+0000","last_clean":"2024-02-05T08:18:38.823354+0000","last_became_active":"2024-02-05T08:18:34.636037+0000","last_became_peered":"2024-02-05T08:18:34.636037+0000","last_unstale":"2024-02-05T08:18:38.823354+0000","last_undegraded":"2024-02-05T08:18:38.823354+0000","last_fullsized":"2024-02-05T08:18:38.823354+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-02-05T08:18:05.582303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:05.582303+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:21.826032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":526,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823663+0000","last_change":"2024-02-05T08:17:51.825665+0000","last_active":"2024-02-05T08:18:38.823663+0000","last_peered":"2024-02-05T08:18:38.823663+0000","last_clean":"2024-02-05T08:18:38.823663+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:18:38.823663+0000","last_undegraded":"2024-02-05T08:18:38.823663+0000","last_fullsized":"2024-02-05T08:18:38.823663+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:51.825345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:17:51.825345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:20.235472+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":518,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823705+0000","last_change":"2024-02-05T08:18:35.317828+0000","last_active":"2024-02-05T08:18:38.823705+0000","last_peered":"2024-02-05T08:18:38.823705+0000","last_clean":"2024-02-05T08:18:38.823705+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:18:38.823705+0000","last_undegraded":"2024-02-05T08:18:38.823705+0000","last_fullsized":"2024-02-05T08:18:38.823705+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:35.317741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:18:35.317741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:35.854280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":506,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.823600+0000","last_change":"2024-02-05T08:15:22.341597+0000","last_active":"2024-02-05T08:18:37.823600+0000","last_peered":"2024-02-05T08:18:37.823600+0000","last_clean":"2024-02-05T08:18:37.823600+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:18:37.823600+0000","last_undegraded":"2024-02-05T08:18:37.823600+0000","last_fullsized":"2024-02-05T08:18:37.823600+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:22.341491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:15:22.341491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"236'34","reported_seq":631,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.824403+0000","last_change":"2024-02-05T08:18:34.748576+0000","last_active":"2024-02-05T08:18:38.824403+0000","last_peered":"2024-02-05T08:18:38.824403+0000","last_clean":"2024-02-05T08:18:38.824403+0000","last_became_active":"2024-02-05T08:18:34.669030+0000","last_became_peered":"2024-02-05T08:18:34.669030+0000","last_unstale":"2024-02-05T08:18:38.824403+0000","last_undegraded":"2024-02-05T08:18:38.824403+0000","last_fullsized":"2024-02-05T08:18:38.824403+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:17:17.405818+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:17:17.405818+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:40.102728+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":17,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827634+0000","last_change":"2024-02-05T08:18:34.642751+0000","last_active":"2024-02-05T08:18:38.827634+0000","last_peered":"2024-02-05T08:18:38.827634+0000","last_clean":"2024-02-05T08:18:38.827634+0000","last_became_active":"2024-02-05T08:18:34.642494+0000","last_became_peered":"2024-02-05T08:18:34.642494+0000","last_unstale":"2024-02-05T08:18:38.827634+0000","last_undegraded":"2024-02-05T08:18:38.827634+0000","last_fullsized":"2024-02-05T08:18:38.827634+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_clean_scrub_stamp":"2024-02-05T08:18:04.435264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:21.971615+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":529,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823765+0000","last_change":"2024-02-05T08:18:34.639113+0000","last_active":"2024-02-05T08:18:38.823765+0000","last_peered":"2024-02-05T08:18:38.823765+0000","last_clean":"2024-02-05T08:18:38.823765+0000","last_became_active":"2024-02-05T08:18:34.638652+0000","last_became_peered":"2024-02-05T08:18:34.638652+0000","last_unstale":"2024-02-05T08:18:38.823765+0000","last_undegraded":"2024-02-05T08:18:38.823765+0000","last_fullsized":"2024-02-05T08:18:38.823765+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:46.786289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:16:46.786289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":517,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823092+0000","last_change":"2024-02-05T08:17:07.717997+0000","last_active":"2024-02-05T08:18:38.823092+0000","last_peered":"2024-02-05T08:18:38.823092+0000","last_clean":"2024-02-05T08:18:38.823092+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:18:38.823092+0000","last_undegraded":"2024-02-05T08:18:38.823092+0000","last_fullsized":"2024-02-05T08:18:38.823092+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:17:07.717888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"236'23","reported_seq":29,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826692+0000","last_change":"2024-02-05T08:18:34.763568+0000","last_active":"2024-02-05T08:18:38.826692+0000","last_peered":"2024-02-05T08:18:38.826692+0000","last_clean":"2024-02-05T08:18:38.826692+0000","last_became_active":"2024-02-05T08:18:34.663801+0000","last_became_peered":"2024-02-05T08:18:34.663801+0000","last_unstale":"2024-02-05T08:18:38.826692+0000","last_undegraded":"2024-02-05T08:18:38.826692+0000","last_fullsized":"2024-02-05T08:18:38.826692+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'23","last_scrub_stamp":"2024-02-05T08:17:58.012827+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:17:58.012827+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:24.914901+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":524,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.860695+0000","last_change":"2024-02-05T08:17:10.980274+0000","last_active":"2024-02-05T08:18:38.860695+0000","last_peered":"2024-02-05T08:18:38.860695+0000","last_clean":"2024-02-05T08:18:38.860695+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:18:38.860695+0000","last_undegraded":"2024-02-05T08:18:38.860695+0000","last_fullsized":"2024-02-05T08:18:38.860695+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:17:10.980170+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:17:10.980170+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":334,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.897596+0000","last_change":"2024-02-05T08:17:11.092748+0000","last_active":"2024-02-05T08:18:38.897596+0000","last_peered":"2024-02-05T08:18:38.897596+0000","last_clean":"2024-02-05T08:18:38.897596+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:18:38.897596+0000","last_undegraded":"2024-02-05T08:18:38.897596+0000","last_fullsized":"2024-02-05T08:18:38.897596+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:17:11.092613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:11.092613+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":547,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.822459+0000","last_change":"2024-02-05T08:18:34.676826+0000","last_active":"2024-02-05T08:18:38.822459+0000","last_peered":"2024-02-05T08:18:38.822459+0000","last_clean":"2024-02-05T08:18:38.822459+0000","last_became_active":"2024-02-05T08:18:34.657282+0000","last_became_peered":"2024-02-05T08:18:34.657282+0000","last_unstale":"2024-02-05T08:18:38.822459+0000","last_undegraded":"2024-02-05T08:18:38.822459+0000","last_fullsized":"2024-02-05T08:18:38.822459+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:18:03.966715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:18:03.966715+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:23.333028+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"236'20","reported_seq":613,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.822329+0000","last_change":"2024-02-05T08:18:34.714282+0000","last_active":"2024-02-05T08:18:37.822329+0000","last_peered":"2024-02-05T08:18:37.822329+0000","last_clean":"2024-02-05T08:18:37.822329+0000","last_became_active":"2024-02-05T08:18:34.645441+0000","last_became_peered":"2024-02-05T08:18:34.645441+0000","last_unstale":"2024-02-05T08:18:37.822329+0000","last_undegraded":"2024-02-05T08:18:37.822329+0000","last_fullsized":"2024-02-05T08:18:37.822329+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:17:52.347554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:52.347554+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:11.317085+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":514,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.824138+0000","last_change":"2024-02-05T08:17:17.079508+0000","last_active":"2024-02-05T08:18:38.824138+0000","last_peered":"2024-02-05T08:18:38.824138+0000","last_clean":"2024-02-05T08:18:38.824138+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:18:38.824138+0000","last_undegraded":"2024-02-05T08:18:38.824138+0000","last_fullsized":"2024-02-05T08:18:38.824138+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:17.079382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:17.079382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":507,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.822572+0000","last_change":"2024-02-05T08:16:57.872130+0000","last_active":"2024-02-05T08:18:37.822572+0000","last_peered":"2024-02-05T08:18:37.822572+0000","last_clean":"2024-02-05T08:18:37.822572+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:18:37.822572+0000","last_undegraded":"2024-02-05T08:18:37.822572+0000","last_fullsized":"2024-02-05T08:18:37.822572+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:57.871993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:32.231861+0000","last_clean_scrub_stamp":"2024-02-05T08:16:57.871993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825574+0000","last_change":"2024-02-05T08:18:34.654550+0000","last_active":"2024-02-05T08:18:38.825574+0000","last_peered":"2024-02-05T08:18:38.825574+0000","last_clean":"2024-02-05T08:18:38.825574+0000","last_became_active":"2024-02-05T08:18:34.653773+0000","last_became_peered":"2024-02-05T08:18:34.653773+0000","last_unstale":"2024-02-05T08:18:38.825574+0000","last_undegraded":"2024-02-05T08:18:38.825574+0000","last_fullsized":"2024-02-05T08:18:38.825574+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:52.803901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:16:52.803901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"236'19","reported_seq":279,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.897373+0000","last_change":"2024-02-05T08:18:08.410519+0000","last_active":"2024-02-05T08:18:38.897373+0000","last_peered":"2024-02-05T08:18:38.897373+0000","last_clean":"2024-02-05T08:18:38.897373+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:18:38.897373+0000","last_undegraded":"2024-02-05T08:18:38.897373+0000","last_fullsized":"2024-02-05T08:18:38.897373+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:08.410382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:08.410382+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:34.374145+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":506,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.861681+0000","last_change":"2024-02-05T08:16:52.009766+0000","last_active":"2024-02-05T08:18:38.861681+0000","last_peered":"2024-02-05T08:18:38.861681+0000","last_clean":"2024-02-05T08:18:38.861681+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:18:38.861681+0000","last_undegraded":"2024-02-05T08:18:38.861681+0000","last_fullsized":"2024-02-05T08:18:38.861681+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:52.009657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:52.009657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":247,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.896458+0000","last_change":"2024-02-05T08:16:55.299725+0000","last_active":"2024-02-05T08:18:38.896458+0000","last_peered":"2024-02-05T08:18:38.896458+0000","last_clean":"2024-02-05T08:18:38.896458+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:18:38.896458+0000","last_undegraded":"2024-02-05T08:18:38.896458+0000","last_fullsized":"2024-02-05T08:18:38.896458+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:55.299617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:16:55.299617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":526,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.822864+0000","last_change":"2024-02-05T08:18:34.644923+0000","last_active":"2024-02-05T08:18:37.822864+0000","last_peered":"2024-02-05T08:18:37.822864+0000","last_clean":"2024-02-05T08:18:37.822864+0000","last_became_active":"2024-02-05T08:18:34.644537+0000","last_became_peered":"2024-02-05T08:18:34.644537+0000","last_unstale":"2024-02-05T08:18:37.822864+0000","last_undegraded":"2024-02-05T08:18:37.822864+0000","last_fullsized":"2024-02-05T08:18:37.822864+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:48.387323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:48.387323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:53.789999+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":23,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825839+0000","last_change":"2024-02-05T08:18:34.725602+0000","last_active":"2024-02-05T08:18:38.825839+0000","last_peered":"2024-02-05T08:18:38.825839+0000","last_clean":"2024-02-05T08:18:38.825839+0000","last_became_active":"2024-02-05T08:18:34.654438+0000","last_became_peered":"2024-02-05T08:18:34.654438+0000","last_unstale":"2024-02-05T08:18:38.825839+0000","last_undegraded":"2024-02-05T08:18:38.825839+0000","last_fullsized":"2024-02-05T08:18:38.825839+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:17:10.054526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:10.054526+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":25,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:40.134772+0000","last_change":"2024-02-05T08:18:40.134914+0000","last_active":"2024-02-05T08:18:40.134772+0000","last_peered":"2024-02-05T08:18:40.134772+0000","last_clean":"2024-02-05T08:18:40.134772+0000","last_became_active":"2024-02-05T08:18:34.651979+0000","last_became_peered":"2024-02-05T08:18:34.651979+0000","last_unstale":"2024-02-05T08:18:40.134772+0000","last_undegraded":"2024-02-05T08:18:40.134772+0000","last_fullsized":"2024-02-05T08:18:40.134772+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:40.134574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:03.937759+0000","last_clean_scrub_stamp":"2024-02-05T08:18:40.134574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:55.004963+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":538,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827438+0000","last_change":"2024-02-05T08:18:38.014240+0000","last_active":"2024-02-05T08:18:38.827438+0000","last_peered":"2024-02-05T08:18:38.827438+0000","last_clean":"2024-02-05T08:18:38.827438+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:18:38.827438+0000","last_undegraded":"2024-02-05T08:18:38.827438+0000","last_fullsized":"2024-02-05T08:18:38.827438+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:38.014162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:18:38.014162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:40.648191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":533,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:41.576937+0000","last_change":"2024-02-05T08:18:41.576956+0000","last_active":"2024-02-05T08:18:41.576937+0000","last_peered":"2024-02-05T08:18:41.576937+0000","last_clean":"2024-02-05T08:18:41.576937+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:18:41.576937+0000","last_undegraded":"2024-02-05T08:18:41.576937+0000","last_fullsized":"2024-02-05T08:18:41.576937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:41.576882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:18:41.576882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:00.332588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"236'24","reported_seq":29,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826260+0000","last_change":"2024-02-05T08:18:34.675763+0000","last_active":"2024-02-05T08:18:38.826260+0000","last_peered":"2024-02-05T08:18:38.826260+0000","last_clean":"2024-02-05T08:18:38.826260+0000","last_became_active":"2024-02-05T08:18:34.631912+0000","last_became_peered":"2024-02-05T08:18:34.631912+0000","last_unstale":"2024-02-05T08:18:38.826260+0000","last_undegraded":"2024-02-05T08:18:38.826260+0000","last_fullsized":"2024-02-05T08:18:38.826260+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:18:33.096912+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:18:33.096912+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:49.851211+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":502,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.824365+0000","last_change":"2024-02-05T08:17:09.009620+0000","last_active":"2024-02-05T08:18:37.824365+0000","last_peered":"2024-02-05T08:18:37.824365+0000","last_clean":"2024-02-05T08:18:37.824365+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:18:37.824365+0000","last_undegraded":"2024-02-05T08:18:37.824365+0000","last_fullsized":"2024-02-05T08:18:37.824365+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:17:09.009398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":17,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827140+0000","last_change":"2024-02-05T08:18:34.652343+0000","last_active":"2024-02-05T08:18:38.827140+0000","last_peered":"2024-02-05T08:18:38.827140+0000","last_clean":"2024-02-05T08:18:38.827140+0000","last_became_active":"2024-02-05T08:18:34.652065+0000","last_became_peered":"2024-02-05T08:18:34.652065+0000","last_unstale":"2024-02-05T08:18:38.827140+0000","last_undegraded":"2024-02-05T08:18:38.827140+0000","last_fullsized":"2024-02-05T08:18:38.827140+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:12.850323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:12.850323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":538,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.861002+0000","last_change":"2024-02-05T08:18:34.642703+0000","last_active":"2024-02-05T08:18:38.861002+0000","last_peered":"2024-02-05T08:18:38.861002+0000","last_clean":"2024-02-05T08:18:38.861002+0000","last_became_active":"2024-02-05T08:18:34.642387+0000","last_became_peered":"2024-02-05T08:18:34.642387+0000","last_unstale":"2024-02-05T08:18:38.861002+0000","last_undegraded":"2024-02-05T08:18:38.861002+0000","last_fullsized":"2024-02-05T08:18:38.861002+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:58.070086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:58.070086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:24.009481+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"263'19","reported_seq":193,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:42.566604+0000","last_change":"2024-02-05T08:18:42.566630+0000","last_active":"2024-02-05T08:18:42.566604+0000","last_peered":"2024-02-05T08:18:42.566604+0000","last_clean":"2024-02-05T08:18:42.566604+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:18:42.566604+0000","last_undegraded":"2024-02-05T08:18:42.566604+0000","last_fullsized":"2024-02-05T08:18:42.566604+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_clean_scrub_stamp":"2024-02-05T08:18:42.566537+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:48.083453+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":25,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826110+0000","last_change":"2024-02-05T08:18:37.946586+0000","last_active":"2024-02-05T08:18:38.826110+0000","last_peered":"2024-02-05T08:18:38.826110+0000","last_clean":"2024-02-05T08:18:38.826110+0000","last_became_active":"2024-02-05T08:18:34.650359+0000","last_became_peered":"2024-02-05T08:18:34.650359+0000","last_unstale":"2024-02-05T08:18:38.826110+0000","last_undegraded":"2024-02-05T08:18:38.826110+0000","last_fullsized":"2024-02-05T08:18:38.826110+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:37.946440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.946440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:54.089385+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":530,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823523+0000","last_change":"2024-02-05T08:18:34.661036+0000","last_active":"2024-02-05T08:18:38.823523+0000","last_peered":"2024-02-05T08:18:38.823523+0000","last_clean":"2024-02-05T08:18:38.823523+0000","last_became_active":"2024-02-05T08:18:34.660727+0000","last_became_peered":"2024-02-05T08:18:34.660727+0000","last_unstale":"2024-02-05T08:18:38.823523+0000","last_undegraded":"2024-02-05T08:18:38.823523+0000","last_fullsized":"2024-02-05T08:18:38.823523+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:47.016029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:47.016029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:16.487179+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":545,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823663+0000","last_change":"2024-02-05T08:18:37.567025+0000","last_active":"2024-02-05T08:18:38.823663+0000","last_peered":"2024-02-05T08:18:38.823663+0000","last_clean":"2024-02-05T08:18:38.823663+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:18:38.823663+0000","last_undegraded":"2024-02-05T08:18:38.823663+0000","last_fullsized":"2024-02-05T08:18:38.823663+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:37.566764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.566764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:59.493231+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"249'24","reported_seq":593,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823434+0000","last_change":"2024-02-05T08:17:11.725677+0000","last_active":"2024-02-05T08:18:38.823434+0000","last_peered":"2024-02-05T08:18:38.823434+0000","last_clean":"2024-02-05T08:18:38.823434+0000","last_became_active":"2024-02-05T08:12:29.675657+0000","last_became_peered":"2024-02-05T08:12:29.675657+0000","last_unstale":"2024-02-05T08:18:38.823434+0000","last_undegraded":"2024-02-05T08:18:38.823434+0000","last_fullsized":"2024-02-05T08:18:38.823434+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:17:11.725554+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:17:11.725554+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":506,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.822021+0000","last_change":"2024-02-05T08:18:34.646238+0000","last_active":"2024-02-05T08:18:37.822021+0000","last_peered":"2024-02-05T08:18:37.822021+0000","last_clean":"2024-02-05T08:18:37.822021+0000","last_became_active":"2024-02-05T08:18:34.645027+0000","last_became_peered":"2024-02-05T08:18:34.645027+0000","last_unstale":"2024-02-05T08:18:37.822021+0000","last_undegraded":"2024-02-05T08:18:37.822021+0000","last_fullsized":"2024-02-05T08:18:37.822021+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:15:25.472655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:15:25.472655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":694,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826384+0000","last_change":"2024-02-05T08:17:00.560018+0000","last_active":"2024-02-05T08:18:38.826384+0000","last_peered":"2024-02-05T08:18:38.826384+0000","last_clean":"2024-02-05T08:18:38.826384+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:18:38.826384+0000","last_undegraded":"2024-02-05T08:18:38.826384+0000","last_fullsized":"2024-02-05T08:18:38.826384+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:17:00.559889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:00.559889+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":527,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827633+0000","last_change":"2024-02-05T08:16:58.934204+0000","last_active":"2024-02-05T08:18:38.827633+0000","last_peered":"2024-02-05T08:18:38.827633+0000","last_clean":"2024-02-05T08:18:38.827633+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:18:38.827633+0000","last_undegraded":"2024-02-05T08:18:38.827633+0000","last_fullsized":"2024-02-05T08:18:38.827633+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:58.934100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:58.934100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":29,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.833526+0000","last_change":"2024-02-05T08:18:34.765274+0000","last_active":"2024-02-05T08:18:38.833526+0000","last_peered":"2024-02-05T08:18:38.833526+0000","last_clean":"2024-02-05T08:18:38.833526+0000","last_became_active":"2024-02-05T08:18:34.655959+0000","last_became_peered":"2024-02-05T08:18:34.655959+0000","last_unstale":"2024-02-05T08:18:38.833526+0000","last_undegraded":"2024-02-05T08:18:38.833526+0000","last_fullsized":"2024-02-05T08:18:38.833526+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-02-05T08:17:17.993472+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:17:17.993472+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:42.532119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":508,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823608+0000","last_change":"2024-02-05T08:17:01.898247+0000","last_active":"2024-02-05T08:18:38.823608+0000","last_peered":"2024-02-05T08:18:38.823608+0000","last_clean":"2024-02-05T08:18:38.823608+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:18:38.823608+0000","last_undegraded":"2024-02-05T08:18:38.823608+0000","last_fullsized":"2024-02-05T08:18:38.823608+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:01.897910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:01.897910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"295'10","reported_seq":24,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.833688+0000","last_change":"2024-02-05T08:18:34.681440+0000","last_active":"2024-02-05T08:18:38.833688+0000","last_peered":"2024-02-05T08:18:38.833688+0000","last_clean":"2024-02-05T08:18:38.833688+0000","last_became_active":"2024-02-05T08:18:34.644517+0000","last_became_peered":"2024-02-05T08:18:34.644517+0000","last_unstale":"2024-02-05T08:18:38.833688+0000","last_undegraded":"2024-02-05T08:18:38.833688+0000","last_fullsized":"2024-02-05T08:18:38.833688+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'9","last_scrub_stamp":"2024-02-05T08:18:03.113117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:03.113117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:19.362542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":526,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.823193+0000","last_change":"2024-02-05T08:18:34.636065+0000","last_active":"2024-02-05T08:18:37.823193+0000","last_peered":"2024-02-05T08:18:37.823193+0000","last_clean":"2024-02-05T08:18:37.823193+0000","last_became_active":"2024-02-05T08:18:34.635793+0000","last_became_peered":"2024-02-05T08:18:34.635793+0000","last_unstale":"2024-02-05T08:18:37.823193+0000","last_undegraded":"2024-02-05T08:18:37.823193+0000","last_fullsized":"2024-02-05T08:18:37.823193+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:03.472950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:18:03.472950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:21.050332+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":578,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826494+0000","last_change":"2024-02-05T08:18:34.673468+0000","last_active":"2024-02-05T08:18:38.826494+0000","last_peered":"2024-02-05T08:18:38.826494+0000","last_clean":"2024-02-05T08:18:38.826494+0000","last_became_active":"2024-02-05T08:18:34.640741+0000","last_became_peered":"2024-02-05T08:18:34.640741+0000","last_unstale":"2024-02-05T08:18:38.826494+0000","last_undegraded":"2024-02-05T08:18:38.826494+0000","last_fullsized":"2024-02-05T08:18:38.826494+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:16:53.643108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:53.643108+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":529,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.824712+0000","last_change":"2024-02-05T08:18:32.704540+0000","last_active":"2024-02-05T08:18:38.824712+0000","last_peered":"2024-02-05T08:18:38.824712+0000","last_clean":"2024-02-05T08:18:38.824712+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:18:38.824712+0000","last_undegraded":"2024-02-05T08:18:38.824712+0000","last_fullsized":"2024-02-05T08:18:38.824712+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:32.704432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:32.704432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:51.640587+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":17,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827734+0000","last_change":"2024-02-05T08:18:34.656052+0000","last_active":"2024-02-05T08:18:38.827734+0000","last_peered":"2024-02-05T08:18:38.827734+0000","last_clean":"2024-02-05T08:18:38.827734+0000","last_became_active":"2024-02-05T08:18:34.655760+0000","last_became_peered":"2024-02-05T08:18:34.655760+0000","last_unstale":"2024-02-05T08:18:38.827734+0000","last_undegraded":"2024-02-05T08:18:38.827734+0000","last_fullsized":"2024-02-05T08:18:38.827734+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:51.982489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:51.982489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:56.866469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":539,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827366+0000","last_change":"2024-02-05T08:18:09.089321+0000","last_active":"2024-02-05T08:18:38.827366+0000","last_peered":"2024-02-05T08:18:38.827366+0000","last_clean":"2024-02-05T08:18:38.827366+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:18:38.827366+0000","last_undegraded":"2024-02-05T08:18:38.827366+0000","last_fullsized":"2024-02-05T08:18:38.827366+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:09.089099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:18:09.089099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:24.145359+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"278'19","reported_seq":571,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.822420+0000","last_change":"2024-02-05T08:18:34.709290+0000","last_active":"2024-02-05T08:18:37.822420+0000","last_peered":"2024-02-05T08:18:37.822420+0000","last_clean":"2024-02-05T08:18:37.822420+0000","last_became_active":"2024-02-05T08:18:34.645881+0000","last_became_peered":"2024-02-05T08:18:34.645881+0000","last_unstale":"2024-02-05T08:18:37.822420+0000","last_undegraded":"2024-02-05T08:18:37.822420+0000","last_fullsized":"2024-02-05T08:18:37.822420+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:17:11.006333+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":526,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827680+0000","last_change":"2024-02-05T08:16:54.078592+0000","last_active":"2024-02-05T08:18:38.827680+0000","last_peered":"2024-02-05T08:18:38.827680+0000","last_clean":"2024-02-05T08:18:38.827680+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:18:38.827680+0000","last_undegraded":"2024-02-05T08:18:38.827680+0000","last_fullsized":"2024-02-05T08:18:38.827680+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:54.078361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:54.078361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":529,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826428+0000","last_change":"2024-02-05T08:18:34.641754+0000","last_active":"2024-02-05T08:18:38.826428+0000","last_peered":"2024-02-05T08:18:38.826428+0000","last_clean":"2024-02-05T08:18:38.826428+0000","last_became_active":"2024-02-05T08:18:34.640731+0000","last_became_peered":"2024-02-05T08:18:34.640731+0000","last_unstale":"2024-02-05T08:18:38.826428+0000","last_undegraded":"2024-02-05T08:18:38.826428+0000","last_fullsized":"2024-02-05T08:18:38.826428+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:07.366823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:18:07.366823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:08.293796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":255,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.897208+0000","last_change":"2024-02-05T08:17:49.211808+0000","last_active":"2024-02-05T08:18:38.897208+0000","last_peered":"2024-02-05T08:18:38.897208+0000","last_clean":"2024-02-05T08:18:38.897208+0000","last_became_active":"2024-02-05T08:12:29.684444+0000","last_became_peered":"2024-02-05T08:12:29.684444+0000","last_unstale":"2024-02-05T08:18:38.897208+0000","last_undegraded":"2024-02-05T08:18:38.897208+0000","last_fullsized":"2024-02-05T08:18:38.897208+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:49.211690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:49.211690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:10.018813+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'15","reported_seq":25,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827348+0000","last_change":"2024-02-05T08:18:34.759559+0000","last_active":"2024-02-05T08:18:38.827348+0000","last_peered":"2024-02-05T08:18:38.827348+0000","last_clean":"2024-02-05T08:18:38.827348+0000","last_became_active":"2024-02-05T08:18:34.655119+0000","last_became_peered":"2024-02-05T08:18:34.655119+0000","last_unstale":"2024-02-05T08:18:38.827348+0000","last_undegraded":"2024-02-05T08:18:38.827348+0000","last_fullsized":"2024-02-05T08:18:38.827348+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:17:54.101749+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.101749+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:22.848826+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"297'20032","reported_seq":20351,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.871312+0000","last_change":"2024-02-05T08:17:47.872971+0000","last_active":"2024-02-05T08:18:37.871312+0000","last_peered":"2024-02-05T08:18:37.871312+0000","last_clean":"2024-02-05T08:18:37.871312+0000","last_became_active":"2024-02-05T08:14:57.766609+0000","last_became_peered":"2024-02-05T08:14:57.766609+0000","last_unstale":"2024-02-05T08:18:37.871312+0000","last_undegraded":"2024-02-05T08:18:37.871312+0000","last_fullsized":"2024-02-05T08:18:37.871312+0000","mapping_epoch":239,"log_start":"280'14702","ondisk_log_start":"280'14702","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'16197","last_scrub_stamp":"2024-02-05T08:17:47.872744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:47.872744+0000","objects_scrubbed":4015,"log_size":5330,"log_dups_size":0,"ondisk_log_size":5330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:12.341039+0000","scrub_duration":1552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8585216,"num_objects":131,"num_object_clones":0,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16585,"num_write_kb":534912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":526,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826351+0000","last_change":"2024-02-05T08:16:50.673897+0000","last_active":"2024-02-05T08:18:38.826351+0000","last_peered":"2024-02-05T08:18:38.826351+0000","last_clean":"2024-02-05T08:18:38.826351+0000","last_became_active":"2024-02-05T08:12:29.679849+0000","last_became_peered":"2024-02-05T08:12:29.679849+0000","last_unstale":"2024-02-05T08:18:38.826351+0000","last_undegraded":"2024-02-05T08:18:38.826351+0000","last_fullsized":"2024-02-05T08:18:38.826351+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:50.673646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:50.673646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":539,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.864199+0000","last_change":"2024-02-05T08:17:00.989869+0000","last_active":"2024-02-05T08:18:38.864199+0000","last_peered":"2024-02-05T08:18:38.864199+0000","last_clean":"2024-02-05T08:18:38.864199+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:18:38.864199+0000","last_undegraded":"2024-02-05T08:18:38.864199+0000","last_fullsized":"2024-02-05T08:18:38.864199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:17:00.989762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:00.989762+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"262'34","reported_seq":616,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825651+0000","last_change":"2024-02-05T08:18:09.398357+0000","last_active":"2024-02-05T08:18:38.825651+0000","last_peered":"2024-02-05T08:18:38.825651+0000","last_clean":"2024-02-05T08:18:38.825651+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:18:38.825651+0000","last_undegraded":"2024-02-05T08:18:38.825651+0000","last_fullsized":"2024-02-05T08:18:38.825651+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'34","last_scrub_stamp":"2024-02-05T08:18:09.398226+0000","last_deep_scrub":"262'34","last_deep_scrub_stamp":"2024-02-05T08:16:06.848964+0000","last_clean_scrub_stamp":"2024-02-05T08:18:09.398226+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:24.751532+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":717,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.861592+0000","last_change":"2024-02-05T08:17:09.961319+0000","last_active":"2024-02-05T08:18:38.861592+0000","last_peered":"2024-02-05T08:18:38.861592+0000","last_clean":"2024-02-05T08:18:38.861592+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:18:38.861592+0000","last_undegraded":"2024-02-05T08:18:38.861592+0000","last_fullsized":"2024-02-05T08:18:38.861592+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:17:09.961036+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:17:09.961036+0000","last_clean_scrub_stamp":"2024-02-05T08:17:09.961036+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"298'16666","reported_seq":16882,"reported_epoch":298,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T08:18:40.779826+0000","last_change":"2024-02-05T08:18:35.969638+0000","last_active":"2024-02-05T08:18:40.779826+0000","last_peered":"2024-02-05T08:18:40.779826+0000","last_clean":"2024-02-05T08:18:33.602313+0000","last_became_active":"2024-02-05T08:18:35.939928+0000","last_became_peered":"2024-02-05T08:18:35.939928+0000","last_unstale":"2024-02-05T08:18:40.779826+0000","last_undegraded":"2024-02-05T08:18:40.779826+0000","last_fullsized":"2024-02-05T08:18:40.779826+0000","mapping_epoch":295,"log_start":"282'12300","ondisk_log_start":"282'12300","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9636","last_scrub_stamp":"2024-02-05T08:17:00.838236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:00.838236+0000","objects_scrubbed":7096,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16666,"num_write_kb":533312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,0],"avail_no_missing":["1","0","3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":17,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827336+0000","last_change":"2024-02-05T08:18:34.654626+0000","last_active":"2024-02-05T08:18:38.827336+0000","last_peered":"2024-02-05T08:18:38.827336+0000","last_clean":"2024-02-05T08:18:38.827336+0000","last_became_active":"2024-02-05T08:18:34.653669+0000","last_became_peered":"2024-02-05T08:18:34.653669+0000","last_unstale":"2024-02-05T08:18:38.827336+0000","last_undegraded":"2024-02-05T08:18:38.827336+0000","last_fullsized":"2024-02-05T08:18:38.827336+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:31.245736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:18:31.245736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:44.428368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":527,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.861273+0000","last_change":"2024-02-05T08:17:12.948132+0000","last_active":"2024-02-05T08:18:38.861273+0000","last_peered":"2024-02-05T08:18:38.861273+0000","last_clean":"2024-02-05T08:18:38.861273+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:18:38.861273+0000","last_undegraded":"2024-02-05T08:18:38.861273+0000","last_fullsized":"2024-02-05T08:18:38.861273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:12.948033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:17:12.948033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":590,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823424+0000","last_change":"2024-02-05T08:18:34.747769+0000","last_active":"2024-02-05T08:18:38.823424+0000","last_peered":"2024-02-05T08:18:38.823424+0000","last_clean":"2024-02-05T08:18:38.823424+0000","last_became_active":"2024-02-05T08:18:34.644717+0000","last_became_peered":"2024-02-05T08:18:34.644717+0000","last_unstale":"2024-02-05T08:18:38.823424+0000","last_undegraded":"2024-02-05T08:18:38.823424+0000","last_fullsized":"2024-02-05T08:18:38.823424+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-02-05T08:18:02.629366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:02.629366+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:07.486978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":534,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.822826+0000","last_change":"2024-02-05T08:18:33.725725+0000","last_active":"2024-02-05T08:18:38.822826+0000","last_peered":"2024-02-05T08:18:38.822826+0000","last_clean":"2024-02-05T08:18:38.822826+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:18:38.822826+0000","last_undegraded":"2024-02-05T08:18:38.822826+0000","last_fullsized":"2024-02-05T08:18:38.822826+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:33.725647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:18:33.725647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:53.197171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"298'16188","reported_seq":16779,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:40.792085+0000","last_change":"2024-02-05T08:18:38.850481+0000","last_active":"2024-02-05T08:18:40.792085+0000","last_peered":"2024-02-05T08:18:40.792085+0000","last_clean":"2024-02-05T08:18:40.792085+0000","last_became_active":"2024-02-05T08:18:38.828002+0000","last_became_peered":"2024-02-05T08:18:38.828002+0000","last_unstale":"2024-02-05T08:18:40.792085+0000","last_undegraded":"2024-02-05T08:18:40.792085+0000","last_fullsized":"2024-02-05T08:18:40.792085+0000","mapping_epoch":297,"log_start":"281'11300","ondisk_log_start":"281'11300","created":239,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'10307","last_scrub_stamp":"2024-02-05T08:17:13.631603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:13.631603+0000","objects_scrubbed":5907,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16188,"num_write_kb":518016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":12451840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":527,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827083+0000","last_change":"2024-02-05T08:18:34.642584+0000","last_active":"2024-02-05T08:18:38.827083+0000","last_peered":"2024-02-05T08:18:38.827083+0000","last_clean":"2024-02-05T08:18:38.827083+0000","last_became_active":"2024-02-05T08:18:34.642239+0000","last_became_peered":"2024-02-05T08:18:34.642239+0000","last_unstale":"2024-02-05T08:18:38.827083+0000","last_undegraded":"2024-02-05T08:18:38.827083+0000","last_fullsized":"2024-02-05T08:18:38.827083+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:00.084681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:00.084681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:12.340840+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":526,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.824238+0000","last_change":"2024-02-05T08:17:46.541994+0000","last_active":"2024-02-05T08:18:38.824238+0000","last_peered":"2024-02-05T08:18:38.824238+0000","last_clean":"2024-02-05T08:18:38.824238+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:18:38.824238+0000","last_undegraded":"2024-02-05T08:18:38.824238+0000","last_fullsized":"2024-02-05T08:18:38.824238+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:46.541893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:17:46.541893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:49.504701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":540,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.861460+0000","last_change":"2024-02-05T08:16:57.957167+0000","last_active":"2024-02-05T08:18:38.861460+0000","last_peered":"2024-02-05T08:18:38.861460+0000","last_clean":"2024-02-05T08:18:38.861460+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:18:38.861460+0000","last_undegraded":"2024-02-05T08:18:38.861460+0000","last_fullsized":"2024-02-05T08:18:38.861460+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:57.957064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:57.957064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"236'35","reported_seq":299,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.896739+0000","last_change":"2024-02-05T08:17:54.291897+0000","last_active":"2024-02-05T08:18:38.896739+0000","last_peered":"2024-02-05T08:18:38.896739+0000","last_clean":"2024-02-05T08:18:38.896739+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:18:38.896739+0000","last_undegraded":"2024-02-05T08:18:38.896739+0000","last_fullsized":"2024-02-05T08:18:38.896739+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'35","last_scrub_stamp":"2024-02-05T08:17:54.291742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.291742+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:09.781739+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"298'16544","reported_seq":16749,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:40.788976+0000","last_change":"2024-02-05T08:17:06.041806+0000","last_active":"2024-02-05T08:18:40.788976+0000","last_peered":"2024-02-05T08:18:40.788976+0000","last_clean":"2024-02-05T08:18:40.788976+0000","last_became_active":"2024-02-05T08:14:57.761156+0000","last_became_peered":"2024-02-05T08:14:57.761156+0000","last_unstale":"2024-02-05T08:18:40.788976+0000","last_undegraded":"2024-02-05T08:18:40.788976+0000","last_fullsized":"2024-02-05T08:18:40.788976+0000","mapping_epoch":239,"log_start":"281'11600","ondisk_log_start":"281'11600","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'9937","last_scrub_stamp":"2024-02-05T08:17:06.041540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:06.041540+0000","objects_scrubbed":6705,"log_size":4944,"log_dups_size":0,"ondisk_log_size":4944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2521,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16544,"num_write_kb":529408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":503,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.823486+0000","last_change":"2024-02-05T08:16:49.867423+0000","last_active":"2024-02-05T08:18:37.823486+0000","last_peered":"2024-02-05T08:18:37.823486+0000","last_clean":"2024-02-05T08:18:37.823486+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:18:37.823486+0000","last_undegraded":"2024-02-05T08:18:37.823486+0000","last_fullsized":"2024-02-05T08:18:37.823486+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:49.867301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:49.867301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":546,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:41.709897+0000","last_change":"2024-02-05T08:18:41.709917+0000","last_active":"2024-02-05T08:18:41.709897+0000","last_peered":"2024-02-05T08:18:41.709897+0000","last_clean":"2024-02-05T08:18:41.709897+0000","last_became_active":"2024-02-05T08:18:34.645830+0000","last_became_peered":"2024-02-05T08:18:34.645830+0000","last_unstale":"2024-02-05T08:18:41.709897+0000","last_undegraded":"2024-02-05T08:18:41.709897+0000","last_fullsized":"2024-02-05T08:18:41.709897+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:41.709832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:41.709832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:50.117637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":550,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825874+0000","last_change":"2024-02-05T08:18:34.633615+0000","last_active":"2024-02-05T08:18:38.825874+0000","last_peered":"2024-02-05T08:18:38.825874+0000","last_clean":"2024-02-05T08:18:38.825874+0000","last_became_active":"2024-02-05T08:18:34.633214+0000","last_became_peered":"2024-02-05T08:18:34.633214+0000","last_unstale":"2024-02-05T08:18:38.825874+0000","last_undegraded":"2024-02-05T08:18:38.825874+0000","last_fullsized":"2024-02-05T08:18:38.825874+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:53.388314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:53.388314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:12.043332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"262'19","reported_seq":540,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.823834+0000","last_change":"2024-02-05T08:16:04.837042+0000","last_active":"2024-02-05T08:18:37.823834+0000","last_peered":"2024-02-05T08:18:37.823834+0000","last_clean":"2024-02-05T08:18:37.823834+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:18:37.823834+0000","last_undegraded":"2024-02-05T08:18:37.823834+0000","last_fullsized":"2024-02-05T08:18:37.823834+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'19","last_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:16:04.836768+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"297'16604","reported_seq":16777,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.932395+0000","last_change":"2024-02-05T08:17:51.835234+0000","last_active":"2024-02-05T08:18:37.932395+0000","last_peered":"2024-02-05T08:18:37.932395+0000","last_clean":"2024-02-05T08:18:37.932395+0000","last_became_active":"2024-02-05T08:14:57.765694+0000","last_became_peered":"2024-02-05T08:14:57.765694+0000","last_unstale":"2024-02-05T08:18:37.932395+0000","last_undegraded":"2024-02-05T08:18:37.932395+0000","last_fullsized":"2024-02-05T08:18:37.932395+0000","mapping_epoch":239,"log_start":"279'11200","ondisk_log_start":"279'11200","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'13220","last_scrub_stamp":"2024-02-05T08:17:51.835033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:51.835033+0000","objects_scrubbed":3578,"log_size":5404,"log_dups_size":0,"ondisk_log_size":5404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:59.083208+0000","scrub_duration":1459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9306112,"num_objects":142,"num_object_clones":0,"num_object_copies":426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16604,"num_write_kb":535872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":244,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823369+0000","last_change":"2024-02-05T08:18:34.638135+0000","last_active":"2024-02-05T08:18:38.823369+0000","last_peered":"2024-02-05T08:18:38.823369+0000","last_clean":"2024-02-05T08:18:38.823369+0000","last_became_active":"2024-02-05T08:18:34.637642+0000","last_became_peered":"2024-02-05T08:18:34.637642+0000","last_unstale":"2024-02-05T08:18:38.823369+0000","last_undegraded":"2024-02-05T08:18:38.823369+0000","last_fullsized":"2024-02-05T08:18:38.823369+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:54.817641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.817641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:58.762109+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":558,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825909+0000","last_change":"2024-02-05T08:16:48.614737+0000","last_active":"2024-02-05T08:18:38.825909+0000","last_peered":"2024-02-05T08:18:38.825909+0000","last_clean":"2024-02-05T08:18:38.825909+0000","last_became_active":"2024-02-05T08:12:29.679395+0000","last_became_peered":"2024-02-05T08:12:29.679395+0000","last_unstale":"2024-02-05T08:18:38.825909+0000","last_undegraded":"2024-02-05T08:18:38.825909+0000","last_fullsized":"2024-02-05T08:18:38.825909+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:16:48.614659+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:16:48.614659+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":538,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:41.490561+0000","last_change":"2024-02-05T08:18:41.490580+0000","last_active":"2024-02-05T08:18:41.490561+0000","last_peered":"2024-02-05T08:18:41.490561+0000","last_clean":"2024-02-05T08:18:41.490561+0000","last_became_active":"2024-02-05T08:12:29.670859+0000","last_became_peered":"2024-02-05T08:12:29.670859+0000","last_unstale":"2024-02-05T08:18:41.490561+0000","last_undegraded":"2024-02-05T08:18:41.490561+0000","last_fullsized":"2024-02-05T08:18:41.490561+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:41.490503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:18:41.490503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:04.724327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"295'52","reported_seq":825,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.824812+0000","last_change":"2024-02-05T08:18:34.743046+0000","last_active":"2024-02-05T08:18:38.824812+0000","last_peered":"2024-02-05T08:18:38.824812+0000","last_clean":"2024-02-05T08:18:38.824812+0000","last_became_active":"2024-02-05T08:18:34.666477+0000","last_became_peered":"2024-02-05T08:18:34.666477+0000","last_unstale":"2024-02-05T08:18:38.824812+0000","last_undegraded":"2024-02-05T08:18:38.824812+0000","last_fullsized":"2024-02-05T08:18:38.824812+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'51","last_scrub_stamp":"2024-02-05T08:17:56.019516+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:17:56.019516+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:12.185192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":499,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.825260+0000","last_change":"2024-02-05T08:16:09.774039+0000","last_active":"2024-02-05T08:18:37.825260+0000","last_peered":"2024-02-05T08:18:37.825260+0000","last_clean":"2024-02-05T08:18:37.825260+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:18:37.825260+0000","last_undegraded":"2024-02-05T08:18:37.825260+0000","last_fullsized":"2024-02-05T08:18:37.825260+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:09.773910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:09.773910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"298'16498","reported_seq":16709,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:40.759269+0000","last_change":"2024-02-05T08:16:58.188469+0000","last_active":"2024-02-05T08:18:40.759269+0000","last_peered":"2024-02-05T08:18:40.759269+0000","last_clean":"2024-02-05T08:18:40.759269+0000","last_became_active":"2024-02-05T08:14:57.763184+0000","last_became_peered":"2024-02-05T08:14:57.763184+0000","last_unstale":"2024-02-05T08:18:40.759269+0000","last_undegraded":"2024-02-05T08:18:40.759269+0000","last_fullsized":"2024-02-05T08:18:40.759269+0000","mapping_epoch":239,"log_start":"281'11600","ondisk_log_start":"281'11600","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'9334","last_scrub_stamp":"2024-02-05T08:16:58.188319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:16:58.188319+0000","objects_scrubbed":7257,"log_size":4898,"log_dups_size":0,"ondisk_log_size":4898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16498,"num_write_kb":527936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":539,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.861634+0000","last_change":"2024-02-05T08:18:35.074566+0000","last_active":"2024-02-05T08:18:38.861634+0000","last_peered":"2024-02-05T08:18:38.861634+0000","last_clean":"2024-02-05T08:18:38.861634+0000","last_became_active":"2024-02-05T08:18:34.643109+0000","last_became_peered":"2024-02-05T08:18:34.643109+0000","last_unstale":"2024-02-05T08:18:38.861634+0000","last_undegraded":"2024-02-05T08:18:38.861634+0000","last_fullsized":"2024-02-05T08:18:38.861634+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:35.074467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:35.074467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:02.738828+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":554,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:43.341914+0000","last_change":"2024-02-05T08:18:43.341960+0000","last_active":"2024-02-05T08:18:43.341914+0000","last_peered":"2024-02-05T08:18:43.341914+0000","last_clean":"2024-02-05T08:18:43.341914+0000","last_became_active":"2024-02-05T08:12:29.673554+0000","last_became_peered":"2024-02-05T08:12:29.673554+0000","last_unstale":"2024-02-05T08:18:43.341914+0000","last_undegraded":"2024-02-05T08:18:43.341914+0000","last_fullsized":"2024-02-05T08:18:43.341914+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:18:43.341831+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:18:43.341831+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:02.149423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":563,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:39.760785+0000","last_change":"2024-02-05T08:18:39.760847+0000","last_active":"2024-02-05T08:18:39.760785+0000","last_peered":"2024-02-05T08:18:39.760785+0000","last_clean":"2024-02-05T08:18:39.760785+0000","last_became_active":"2024-02-05T08:18:34.644683+0000","last_became_peered":"2024-02-05T08:18:34.644683+0000","last_unstale":"2024-02-05T08:18:39.760785+0000","last_undegraded":"2024-02-05T08:18:39.760785+0000","last_fullsized":"2024-02-05T08:18:39.760785+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:39.760577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:39.760577+0000","last_clean_scrub_stamp":"2024-02-05T08:18:39.760577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:56.899445+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'17","reported_seq":540,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825561+0000","last_change":"2024-02-05T08:17:47.561079+0000","last_active":"2024-02-05T08:18:38.825561+0000","last_peered":"2024-02-05T08:18:38.825561+0000","last_clean":"2024-02-05T08:18:38.825561+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:18:38.825561+0000","last_undegraded":"2024-02-05T08:18:38.825561+0000","last_fullsized":"2024-02-05T08:18:38.825561+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'17","last_scrub_stamp":"2024-02-05T08:17:47.560963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:47.560963+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:18:53.181518+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":534,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825277+0000","last_change":"2024-02-05T08:18:08.651235+0000","last_active":"2024-02-05T08:18:38.825277+0000","last_peered":"2024-02-05T08:18:38.825277+0000","last_clean":"2024-02-05T08:18:38.825277+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:18:38.825277+0000","last_undegraded":"2024-02-05T08:18:38.825277+0000","last_fullsized":"2024-02-05T08:18:38.825277+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:08.650666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:08.650666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:17.292283+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"298'16302","reported_seq":16861,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:40.750732+0000","last_change":"2024-02-05T08:18:38.883552+0000","last_active":"2024-02-05T08:18:40.750732+0000","last_peered":"2024-02-05T08:18:40.750732+0000","last_clean":"2024-02-05T08:18:40.750732+0000","last_became_active":"2024-02-05T08:18:38.829013+0000","last_became_peered":"2024-02-05T08:18:38.829013+0000","last_unstale":"2024-02-05T08:18:40.750732+0000","last_undegraded":"2024-02-05T08:18:40.750732+0000","last_fullsized":"2024-02-05T08:18:40.750732+0000","mapping_epoch":297,"log_start":"282'11595","ondisk_log_start":"282'11595","created":239,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'12548","last_scrub_stamp":"2024-02-05T08:17:48.702005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:48.702005+0000","objects_scrubbed":3805,"log_size":4707,"log_dups_size":0,"ondisk_log_size":4707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:04.542941+0000","scrub_duration":1352,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16302,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":11927552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":526,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826641+0000","last_change":"2024-02-05T08:18:34.634053+0000","last_active":"2024-02-05T08:18:38.826641+0000","last_peered":"2024-02-05T08:18:38.826641+0000","last_clean":"2024-02-05T08:18:38.826641+0000","last_became_active":"2024-02-05T08:18:34.633450+0000","last_became_peered":"2024-02-05T08:18:34.633450+0000","last_unstale":"2024-02-05T08:18:38.826641+0000","last_undegraded":"2024-02-05T08:18:38.826641+0000","last_fullsized":"2024-02-05T08:18:38.826641+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:05.496639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:05.496639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:15.739330+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":808,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827222+0000","last_change":"2024-02-05T08:16:15.047572+0000","last_active":"2024-02-05T08:18:38.827222+0000","last_peered":"2024-02-05T08:18:38.827222+0000","last_clean":"2024-02-05T08:18:38.827222+0000","last_became_active":"2024-02-05T08:12:32.677426+0000","last_became_peered":"2024-02-05T08:12:32.677426+0000","last_unstale":"2024-02-05T08:18:38.827222+0000","last_undegraded":"2024-02-05T08:18:38.827222+0000","last_fullsized":"2024-02-05T08:18:38.827222+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:16:15.047436+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:16:15.047436+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826307+0000","last_change":"2024-02-05T08:18:34.632300+0000","last_active":"2024-02-05T08:18:38.826307+0000","last_peered":"2024-02-05T08:18:38.826307+0000","last_clean":"2024-02-05T08:18:38.826307+0000","last_became_active":"2024-02-05T08:18:34.631974+0000","last_became_peered":"2024-02-05T08:18:34.631974+0000","last_unstale":"2024-02-05T08:18:38.826307+0000","last_undegraded":"2024-02-05T08:18:38.826307+0000","last_fullsized":"2024-02-05T08:18:38.826307+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:13.476308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:13.476308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"236'30","reported_seq":366,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.897142+0000","last_change":"2024-02-05T08:18:37.284687+0000","last_active":"2024-02-05T08:18:38.897142+0000","last_peered":"2024-02-05T08:18:38.897142+0000","last_clean":"2024-02-05T08:18:38.897142+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:18:38.897142+0000","last_undegraded":"2024-02-05T08:18:38.897142+0000","last_fullsized":"2024-02-05T08:18:38.897142+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:18:37.284559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.284559+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:07.016040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":519,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:42.996752+0000","last_change":"2024-02-05T08:18:42.996770+0000","last_active":"2024-02-05T08:18:42.996752+0000","last_peered":"2024-02-05T08:18:42.996752+0000","last_clean":"2024-02-05T08:18:42.996752+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:18:42.996752+0000","last_undegraded":"2024-02-05T08:18:42.996752+0000","last_fullsized":"2024-02-05T08:18:42.996752+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_clean_scrub_stamp":"2024-02-05T08:18:42.996695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:55.044771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"297'16326","reported_seq":16520,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.958837+0000","last_change":"2024-02-05T08:17:15.291706+0000","last_active":"2024-02-05T08:18:37.958837+0000","last_peered":"2024-02-05T08:18:37.958837+0000","last_clean":"2024-02-05T08:18:37.958837+0000","last_became_active":"2024-02-05T08:14:57.761382+0000","last_became_peered":"2024-02-05T08:14:57.761382+0000","last_unstale":"2024-02-05T08:18:37.958837+0000","last_undegraded":"2024-02-05T08:18:37.958837+0000","last_fullsized":"2024-02-05T08:18:37.958837+0000","mapping_epoch":239,"log_start":"279'11000","ondisk_log_start":"279'11000","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'10631","last_scrub_stamp":"2024-02-05T08:17:15.291551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:15.291551+0000","objects_scrubbed":5855,"log_size":5326,"log_dups_size":0,"ondisk_log_size":5326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8519680,"num_objects":130,"num_object_clones":0,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16326,"num_write_kb":526592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":527,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.824269+0000","last_change":"2024-02-05T08:18:34.660125+0000","last_active":"2024-02-05T08:18:38.824269+0000","last_peered":"2024-02-05T08:18:38.824269+0000","last_clean":"2024-02-05T08:18:38.824269+0000","last_became_active":"2024-02-05T08:18:34.659712+0000","last_became_peered":"2024-02-05T08:18:34.659712+0000","last_unstale":"2024-02-05T08:18:38.824269+0000","last_undegraded":"2024-02-05T08:18:38.824269+0000","last_fullsized":"2024-02-05T08:18:38.824269+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:01.934845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:01.934845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:13.833617+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":527,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.823229+0000","last_change":"2024-02-05T08:18:37.562478+0000","last_active":"2024-02-05T08:18:37.823229+0000","last_peered":"2024-02-05T08:18:37.823229+0000","last_clean":"2024-02-05T08:18:37.823229+0000","last_became_active":"2024-02-05T08:18:34.635954+0000","last_became_peered":"2024-02-05T08:18:34.635954+0000","last_unstale":"2024-02-05T08:18:37.823229+0000","last_undegraded":"2024-02-05T08:18:37.823229+0000","last_fullsized":"2024-02-05T08:18:37.823229+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:37.562368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.562368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:04.501373+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":1374,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:41.306370+0000","last_change":"2024-02-05T08:16:53.018165+0000","last_active":"2024-02-05T08:18:41.306370+0000","last_peered":"2024-02-05T08:18:41.306370+0000","last_clean":"2024-02-05T08:18:41.306370+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:18:41.306370+0000","last_undegraded":"2024-02-05T08:18:41.306370+0000","last_fullsized":"2024-02-05T08:18:41.306370+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:16:53.018039+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:16:53.018039+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":834,"num_read_kb":829,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":559,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.822124+0000","last_change":"2024-02-05T08:16:01.857925+0000","last_active":"2024-02-05T08:18:37.822124+0000","last_peered":"2024-02-05T08:18:37.822124+0000","last_clean":"2024-02-05T08:18:37.822124+0000","last_became_active":"2024-02-05T08:14:24.992820+0000","last_became_peered":"2024-02-05T08:14:24.992820+0000","last_unstale":"2024-02-05T08:18:37.822124+0000","last_undegraded":"2024-02-05T08:18:37.822124+0000","last_fullsized":"2024-02-05T08:18:37.822124+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:16:01.857750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:01.857750+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":518,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827041+0000","last_change":"2024-02-05T08:18:03.284339+0000","last_active":"2024-02-05T08:18:38.827041+0000","last_peered":"2024-02-05T08:18:38.827041+0000","last_clean":"2024-02-05T08:18:38.827041+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:18:38.827041+0000","last_undegraded":"2024-02-05T08:18:38.827041+0000","last_fullsized":"2024-02-05T08:18:38.827041+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:03.282010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:18:03.282010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:29.445541+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"298'16340","reported_seq":16536,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:40.754472+0000","last_change":"2024-02-05T08:17:17.506968+0000","last_active":"2024-02-05T08:18:40.754472+0000","last_peered":"2024-02-05T08:18:40.754472+0000","last_clean":"2024-02-05T08:18:40.754472+0000","last_became_active":"2024-02-05T08:14:57.764021+0000","last_became_peered":"2024-02-05T08:14:57.764021+0000","last_unstale":"2024-02-05T08:18:40.754472+0000","last_undegraded":"2024-02-05T08:18:40.754472+0000","last_fullsized":"2024-02-05T08:18:40.754472+0000","mapping_epoch":239,"log_start":"281'11600","ondisk_log_start":"281'11600","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'10561","last_scrub_stamp":"2024-02-05T08:17:17.506587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:17.506587+0000","objects_scrubbed":5822,"log_size":4740,"log_dups_size":0,"ondisk_log_size":4740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1470,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16340,"num_write_kb":522880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":527,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.822633+0000","last_change":"2024-02-05T08:18:00.957510+0000","last_active":"2024-02-05T08:18:38.822633+0000","last_peered":"2024-02-05T08:18:38.822633+0000","last_clean":"2024-02-05T08:18:38.822633+0000","last_became_active":"2024-02-05T08:12:29.672973+0000","last_became_peered":"2024-02-05T08:12:29.672973+0000","last_unstale":"2024-02-05T08:18:38.822633+0000","last_undegraded":"2024-02-05T08:18:38.822633+0000","last_fullsized":"2024-02-05T08:18:38.822633+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:00.957388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:00.957388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:22.336285+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":535,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.824671+0000","last_change":"2024-02-05T08:18:37.350834+0000","last_active":"2024-02-05T08:18:38.824671+0000","last_peered":"2024-02-05T08:18:38.824671+0000","last_clean":"2024-02-05T08:18:38.824671+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:18:38.824671+0000","last_undegraded":"2024-02-05T08:18:38.824671+0000","last_fullsized":"2024-02-05T08:18:38.824671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:37.350622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.350622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:04.705920+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":562,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.823690+0000","last_change":"2024-02-05T08:15:56.779910+0000","last_active":"2024-02-05T08:18:37.823690+0000","last_peered":"2024-02-05T08:18:37.823690+0000","last_clean":"2024-02-05T08:18:37.823690+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:18:37.823690+0000","last_undegraded":"2024-02-05T08:18:37.823690+0000","last_fullsized":"2024-02-05T08:18:37.823690+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-02-05T08:15:56.779551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:15:56.779551+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":255,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.896660+0000","last_change":"2024-02-05T08:18:01.301193+0000","last_active":"2024-02-05T08:18:38.896660+0000","last_peered":"2024-02-05T08:18:38.896660+0000","last_clean":"2024-02-05T08:18:38.896660+0000","last_became_active":"2024-02-05T08:12:29.688534+0000","last_became_peered":"2024-02-05T08:12:29.688534+0000","last_unstale":"2024-02-05T08:18:38.896660+0000","last_undegraded":"2024-02-05T08:18:38.896660+0000","last_fullsized":"2024-02-05T08:18:38.896660+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:01.301064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:18:01.301064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:02.278101+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"298'16164","reported_seq":16381,"reported_epoch":298,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T08:18:40.803340+0000","last_change":"2024-02-05T08:18:36.007367+0000","last_active":"2024-02-05T08:18:40.803340+0000","last_peered":"2024-02-05T08:18:40.803340+0000","last_clean":"2024-02-05T08:18:33.590778+0000","last_became_active":"2024-02-05T08:18:35.935838+0000","last_became_peered":"2024-02-05T08:18:35.935838+0000","last_unstale":"2024-02-05T08:18:40.803340+0000","last_undegraded":"2024-02-05T08:18:40.803340+0000","last_fullsized":"2024-02-05T08:18:40.803340+0000","mapping_epoch":295,"log_start":"281'11300","ondisk_log_start":"281'11300","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'9398","last_scrub_stamp":"2024-02-05T08:17:01.762104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:01.762104+0000","objects_scrubbed":6842,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16164,"num_write_kb":517248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,2],"avail_no_missing":["5","1","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":515,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.824465+0000","last_change":"2024-02-05T08:18:34.637157+0000","last_active":"2024-02-05T08:18:37.824465+0000","last_peered":"2024-02-05T08:18:37.824465+0000","last_clean":"2024-02-05T08:18:37.824465+0000","last_became_active":"2024-02-05T08:18:34.636693+0000","last_became_peered":"2024-02-05T08:18:34.636693+0000","last_unstale":"2024-02-05T08:18:37.824465+0000","last_undegraded":"2024-02-05T08:18:37.824465+0000","last_fullsized":"2024-02-05T08:18:37.824465+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:00.862633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:16:00.862633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":538,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826600+0000","last_change":"2024-02-05T08:18:34.634300+0000","last_active":"2024-02-05T08:18:38.826600+0000","last_peered":"2024-02-05T08:18:38.826600+0000","last_clean":"2024-02-05T08:18:38.826600+0000","last_became_active":"2024-02-05T08:18:34.633263+0000","last_became_peered":"2024-02-05T08:18:34.633263+0000","last_unstale":"2024-02-05T08:18:38.826600+0000","last_undegraded":"2024-02-05T08:18:38.826600+0000","last_fullsized":"2024-02-05T08:18:38.826600+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:50.387417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:50.387417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:13.838449+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'20","reported_seq":553,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823613+0000","last_change":"2024-02-05T08:17:52.948101+0000","last_active":"2024-02-05T08:18:38.823613+0000","last_peered":"2024-02-05T08:18:38.823613+0000","last_clean":"2024-02-05T08:18:38.823613+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:18:38.823613+0000","last_undegraded":"2024-02-05T08:18:38.823613+0000","last_fullsized":"2024-02-05T08:18:38.823613+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:17:52.947966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:52.947966+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:06.320713+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"277'11","reported_seq":704,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.823927+0000","last_change":"2024-02-05T08:17:07.973477+0000","last_active":"2024-02-05T08:18:37.823927+0000","last_peered":"2024-02-05T08:18:37.823927+0000","last_clean":"2024-02-05T08:18:37.823927+0000","last_became_active":"2024-02-05T08:12:29.683091+0000","last_became_peered":"2024-02-05T08:12:29.683091+0000","last_unstale":"2024-02-05T08:18:37.823927+0000","last_undegraded":"2024-02-05T08:18:37.823927+0000","last_fullsized":"2024-02-05T08:18:37.823927+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_clean_scrub_stamp":"2024-02-05T08:17:07.973171+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"298'20586","reported_seq":21357,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:40.781937+0000","last_change":"2024-02-05T08:18:38.954272+0000","last_active":"2024-02-05T08:18:40.781937+0000","last_peered":"2024-02-05T08:18:40.781937+0000","last_clean":"2024-02-05T08:18:40.781937+0000","last_became_active":"2024-02-05T08:18:38.871288+0000","last_became_peered":"2024-02-05T08:18:38.871288+0000","last_unstale":"2024-02-05T08:18:40.781937+0000","last_undegraded":"2024-02-05T08:18:40.781937+0000","last_fullsized":"2024-02-05T08:18:40.781937+0000","mapping_epoch":297,"log_start":"281'15802","ondisk_log_start":"281'15802","created":239,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'16825","last_scrub_stamp":"2024-02-05T08:17:48.136083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:48.136083+0000","objects_scrubbed":3810,"log_size":4784,"log_dups_size":0,"ondisk_log_size":4784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:12.507363+0000","scrub_duration":1270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16282,"num_write_kb":521024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":13041664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":527,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.861104+0000","last_change":"2024-02-05T08:18:36.117092+0000","last_active":"2024-02-05T08:18:38.861104+0000","last_peered":"2024-02-05T08:18:38.861104+0000","last_clean":"2024-02-05T08:18:38.861104+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:18:38.861104+0000","last_undegraded":"2024-02-05T08:18:38.861104+0000","last_fullsized":"2024-02-05T08:18:38.861104+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:36.116992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:36.116992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:49.466356+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":526,"reported_epoch":297,"state":"active+clean","last_fresh":"2024-02-05T08:18:37.822232+0000","last_change":"2024-02-05T08:16:08.716324+0000","last_active":"2024-02-05T08:18:37.822232+0000","last_peered":"2024-02-05T08:18:37.822232+0000","last_clean":"2024-02-05T08:18:37.822232+0000","last_became_active":"2024-02-05T08:11:23.200753+0000","last_became_peered":"2024-02-05T08:11:23.200753+0000","last_unstale":"2024-02-05T08:18:37.822232+0000","last_undegraded":"2024-02-05T08:18:37.822232+0000","last_fullsized":"2024-02-05T08:18:37.822232+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:16:08.716199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:16:08.716199+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"236'28","reported_seq":600,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823073+0000","last_change":"2024-02-05T08:17:12.688139+0000","last_active":"2024-02-05T08:18:38.823073+0000","last_peered":"2024-02-05T08:18:38.823073+0000","last_clean":"2024-02-05T08:18:38.823073+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:18:38.823073+0000","last_undegraded":"2024-02-05T08:18:38.823073+0000","last_fullsized":"2024-02-05T08:18:38.823073+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:17:12.687898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:12.687898+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"295'11","reported_seq":24,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826115+0000","last_change":"2024-02-05T08:18:34.760490+0000","last_active":"2024-02-05T08:18:38.826115+0000","last_peered":"2024-02-05T08:18:38.826115+0000","last_clean":"2024-02-05T08:18:38.826115+0000","last_became_active":"2024-02-05T08:18:34.664016+0000","last_became_peered":"2024-02-05T08:18:34.664016+0000","last_unstale":"2024-02-05T08:18:38.826115+0000","last_undegraded":"2024-02-05T08:18:38.826115+0000","last_fullsized":"2024-02-05T08:18:38.826115+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'10","last_scrub_stamp":"2024-02-05T08:17:14.040769+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:17:14.040769+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"298'16462","reported_seq":16644,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:40.797604+0000","last_change":"2024-02-05T08:17:45.599157+0000","last_active":"2024-02-05T08:18:40.797604+0000","last_peered":"2024-02-05T08:18:40.797604+0000","last_clean":"2024-02-05T08:18:40.797604+0000","last_became_active":"2024-02-05T08:14:57.765609+0000","last_became_peered":"2024-02-05T08:14:57.765609+0000","last_unstale":"2024-02-05T08:18:40.797604+0000","last_undegraded":"2024-02-05T08:18:40.797604+0000","last_fullsized":"2024-02-05T08:18:40.797604+0000","mapping_epoch":239,"log_start":"282'11800","ondisk_log_start":"282'11800","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"283'12489","last_scrub_stamp":"2024-02-05T08:17:45.598921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:45.598921+0000","objects_scrubbed":4027,"log_size":4662,"log_dups_size":0,"ondisk_log_size":4662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:13.318391+0000","scrub_duration":1561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16462,"num_write_kb":526721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":526,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.822941+0000","last_change":"2024-02-05T08:18:34.636227+0000","last_active":"2024-02-05T08:18:38.822941+0000","last_peered":"2024-02-05T08:18:38.822941+0000","last_clean":"2024-02-05T08:18:38.822941+0000","last_became_active":"2024-02-05T08:18:34.635908+0000","last_became_peered":"2024-02-05T08:18:34.635908+0000","last_unstale":"2024-02-05T08:18:38.822941+0000","last_undegraded":"2024-02-05T08:18:38.822941+0000","last_fullsized":"2024-02-05T08:18:38.822941+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:10.764008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:10.764008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":21,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827238+0000","last_change":"2024-02-05T08:18:34.735709+0000","last_active":"2024-02-05T08:18:38.827238+0000","last_peered":"2024-02-05T08:18:38.827238+0000","last_clean":"2024-02-05T08:18:38.827238+0000","last_became_active":"2024-02-05T08:18:34.649808+0000","last_became_peered":"2024-02-05T08:18:34.649808+0000","last_unstale":"2024-02-05T08:18:38.827238+0000","last_undegraded":"2024-02-05T08:18:38.827238+0000","last_fullsized":"2024-02-05T08:18:38.827238+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:17:52.089740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:52.089740+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:05.694192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"236'38","reported_seq":35,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825665+0000","last_change":"2024-02-05T08:18:34.724877+0000","last_active":"2024-02-05T08:18:38.825665+0000","last_peered":"2024-02-05T08:18:38.825665+0000","last_clean":"2024-02-05T08:18:38.825665+0000","last_became_active":"2024-02-05T08:18:34.653505+0000","last_became_peered":"2024-02-05T08:18:34.653505+0000","last_unstale":"2024-02-05T08:18:38.825665+0000","last_undegraded":"2024-02-05T08:18:38.825665+0000","last_fullsized":"2024-02-05T08:18:38.825665+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'38","last_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:17:05.909305+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":530,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:42.514603+0000","last_change":"2024-02-05T08:18:42.514623+0000","last_active":"2024-02-05T08:18:42.514603+0000","last_peered":"2024-02-05T08:18:42.514603+0000","last_clean":"2024-02-05T08:18:42.514603+0000","last_became_active":"2024-02-05T08:18:34.636667+0000","last_became_peered":"2024-02-05T08:18:34.636667+0000","last_unstale":"2024-02-05T08:18:42.514603+0000","last_undegraded":"2024-02-05T08:18:42.514603+0000","last_fullsized":"2024-02-05T08:18:42.514603+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:42.514544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:42.514544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:08.965745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"298'16418","reported_seq":16635,"reported_epoch":298,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T08:18:40.779490+0000","last_change":"2024-02-05T08:18:35.959427+0000","last_active":"2024-02-05T08:18:40.779490+0000","last_peered":"2024-02-05T08:18:40.779490+0000","last_clean":"2024-02-05T08:18:33.624954+0000","last_became_active":"2024-02-05T08:18:35.921598+0000","last_became_peered":"2024-02-05T08:18:35.921598+0000","last_unstale":"2024-02-05T08:18:40.779490+0000","last_undegraded":"2024-02-05T08:18:40.779490+0000","last_fullsized":"2024-02-05T08:18:40.779490+0000","mapping_epoch":295,"log_start":"282'12000","ondisk_log_start":"282'12000","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'9088","last_scrub_stamp":"2024-02-05T08:16:55.072580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:16:55.072580+0000","objects_scrubbed":7413,"log_size":4418,"log_dups_size":0,"ondisk_log_size":4418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16418,"num_write_kb":525376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2,7],"avail_no_missing":["1","2","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":522,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826135+0000","last_change":"2024-02-05T08:18:34.613605+0000","last_active":"2024-02-05T08:18:38.826135+0000","last_peered":"2024-02-05T08:18:38.826135+0000","last_clean":"2024-02-05T08:18:38.826135+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:18:38.826135+0000","last_undegraded":"2024-02-05T08:18:38.826135+0000","last_fullsized":"2024-02-05T08:18:38.826135+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.613476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.613476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:54.411772+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":687,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823801+0000","last_change":"2024-02-05T08:16:49.737238+0000","last_active":"2024-02-05T08:18:38.823801+0000","last_peered":"2024-02-05T08:18:38.823801+0000","last_clean":"2024-02-05T08:18:38.823801+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:18:38.823801+0000","last_undegraded":"2024-02-05T08:18:38.823801+0000","last_fullsized":"2024-02-05T08:18:38.823801+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:16:49.733743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:49.733743+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825756+0000","last_change":"2024-02-05T08:18:34.653276+0000","last_active":"2024-02-05T08:18:38.825756+0000","last_peered":"2024-02-05T08:18:38.825756+0000","last_clean":"2024-02-05T08:18:38.825756+0000","last_became_active":"2024-02-05T08:18:34.652764+0000","last_became_peered":"2024-02-05T08:18:34.652764+0000","last_unstale":"2024-02-05T08:18:38.825756+0000","last_undegraded":"2024-02-05T08:18:38.825756+0000","last_fullsized":"2024-02-05T08:18:38.825756+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:50.018829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:50.018829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:00.687115+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"236'27","reported_seq":618,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:40.384217+0000","last_change":"2024-02-05T08:18:40.384275+0000","last_active":"2024-02-05T08:18:40.384217+0000","last_peered":"2024-02-05T08:18:40.384217+0000","last_clean":"2024-02-05T08:18:40.384217+0000","last_became_active":"2024-02-05T08:18:34.658318+0000","last_became_peered":"2024-02-05T08:18:34.658318+0000","last_unstale":"2024-02-05T08:18:40.384217+0000","last_undegraded":"2024-02-05T08:18:40.384217+0000","last_fullsized":"2024-02-05T08:18:40.384217+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:18:40.384115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:40.384115+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:45.631489+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"298'16620","reported_seq":16807,"reported_epoch":298,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T08:18:40.788143+0000","last_change":"2024-02-05T08:18:35.989979+0000","last_active":"2024-02-05T08:18:40.788143+0000","last_peered":"2024-02-05T08:18:40.788143+0000","last_clean":"2024-02-05T08:18:33.618976+0000","last_became_active":"2024-02-05T08:18:35.934401+0000","last_became_peered":"2024-02-05T08:18:35.934401+0000","last_unstale":"2024-02-05T08:18:40.788143+0000","last_undegraded":"2024-02-05T08:18:40.788143+0000","last_fullsized":"2024-02-05T08:18:40.788143+0000","mapping_epoch":295,"log_start":"282'11900","ondisk_log_start":"282'11900","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"284'13143","last_scrub_stamp":"2024-02-05T08:17:52.291706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:52.291706+0000","objects_scrubbed":3510,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16620,"num_write_kb":531840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,1,5],"avail_no_missing":["7","1","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":17,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.833525+0000","last_change":"2024-02-05T08:18:34.627571+0000","last_active":"2024-02-05T08:18:38.833525+0000","last_peered":"2024-02-05T08:18:38.833525+0000","last_clean":"2024-02-05T08:18:38.833525+0000","last_became_active":"2024-02-05T08:18:34.627261+0000","last_became_peered":"2024-02-05T08:18:34.627261+0000","last_unstale":"2024-02-05T08:18:38.833525+0000","last_undegraded":"2024-02-05T08:18:38.833525+0000","last_fullsized":"2024-02-05T08:18:38.833525+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:50.260123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:16:50.260123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":530,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.826962+0000","last_change":"2024-02-05T08:16:49.669406+0000","last_active":"2024-02-05T08:18:38.826962+0000","last_peered":"2024-02-05T08:18:38.826962+0000","last_clean":"2024-02-05T08:18:38.826962+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:18:38.826962+0000","last_undegraded":"2024-02-05T08:18:38.826962+0000","last_fullsized":"2024-02-05T08:18:38.826962+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:49.669303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:16:49.669303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":526,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.822925+0000","last_change":"2024-02-05T08:16:55.778045+0000","last_active":"2024-02-05T08:18:38.822925+0000","last_peered":"2024-02-05T08:18:38.822925+0000","last_clean":"2024-02-05T08:18:38.822925+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:18:38.822925+0000","last_undegraded":"2024-02-05T08:18:38.822925+0000","last_fullsized":"2024-02-05T08:18:38.822925+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:55.777839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:16:55.777839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"236'19","reported_seq":589,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823524+0000","last_change":"2024-02-05T08:18:36.561959+0000","last_active":"2024-02-05T08:18:38.823524+0000","last_peered":"2024-02-05T08:18:38.823524+0000","last_clean":"2024-02-05T08:18:38.823524+0000","last_became_active":"2024-02-05T08:12:29.666284+0000","last_became_peered":"2024-02-05T08:12:29.666284+0000","last_unstale":"2024-02-05T08:18:38.823524+0000","last_undegraded":"2024-02-05T08:18:38.823524+0000","last_fullsized":"2024-02-05T08:18:38.823524+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:36.561837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:36.561837+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:42.609439+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"298'16640","reported_seq":16862,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:40.791872+0000","last_change":"2024-02-05T08:18:38.395942+0000","last_active":"2024-02-05T08:18:40.791872+0000","last_peered":"2024-02-05T08:18:40.791872+0000","last_clean":"2024-02-05T08:18:40.791872+0000","last_became_active":"2024-02-05T08:14:57.764870+0000","last_became_peered":"2024-02-05T08:14:57.764870+0000","last_unstale":"2024-02-05T08:18:40.791872+0000","last_undegraded":"2024-02-05T08:18:40.791872+0000","last_fullsized":"2024-02-05T08:18:40.791872+0000","mapping_epoch":239,"log_start":"282'12200","ondisk_log_start":"282'12200","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'16508","last_scrub_stamp":"2024-02-05T08:18:38.395801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:18:38.395801+0000","objects_scrubbed":132,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:52.305087+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16640,"num_write_kb":532480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":551,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.825710+0000","last_change":"2024-02-05T08:18:36.936749+0000","last_active":"2024-02-05T08:18:38.825710+0000","last_peered":"2024-02-05T08:18:38.825710+0000","last_clean":"2024-02-05T08:18:38.825710+0000","last_became_active":"2024-02-05T08:18:34.645067+0000","last_became_peered":"2024-02-05T08:18:34.645067+0000","last_unstale":"2024-02-05T08:18:38.825710+0000","last_undegraded":"2024-02-05T08:18:38.825710+0000","last_fullsized":"2024-02-05T08:18:38.825710+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:18:36.936635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:36.936635+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:48.454947+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":511,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.824844+0000","last_change":"2024-02-05T08:17:12.127732+0000","last_active":"2024-02-05T08:18:38.824844+0000","last_peered":"2024-02-05T08:18:38.824844+0000","last_clean":"2024-02-05T08:18:38.824844+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:18:38.824844+0000","last_undegraded":"2024-02-05T08:18:38.824844+0000","last_fullsized":"2024-02-05T08:18:38.824844+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:12.127605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:12.127605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":255,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.896824+0000","last_change":"2024-02-05T08:17:14.087240+0000","last_active":"2024-02-05T08:18:38.896824+0000","last_peered":"2024-02-05T08:18:38.896824+0000","last_clean":"2024-02-05T08:18:38.896824+0000","last_became_active":"2024-02-05T08:12:29.678501+0000","last_became_peered":"2024-02-05T08:12:29.678501+0000","last_unstale":"2024-02-05T08:18:38.896824+0000","last_undegraded":"2024-02-05T08:18:38.896824+0000","last_fullsized":"2024-02-05T08:18:38.896824+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:14.087110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:17:14.087110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"236'31","reported_seq":563,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823309+0000","last_change":"2024-02-05T08:16:58.775056+0000","last_active":"2024-02-05T08:18:38.823309+0000","last_peered":"2024-02-05T08:18:38.823309+0000","last_clean":"2024-02-05T08:18:38.823309+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:18:38.823309+0000","last_undegraded":"2024-02-05T08:18:38.823309+0000","last_fullsized":"2024-02-05T08:18:38.823309+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'31","last_scrub_stamp":"2024-02-05T08:16:58.774921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:16:58.774921+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"298'16434","reported_seq":16675,"reported_epoch":298,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T08:18:40.787647+0000","last_change":"2024-02-05T08:18:36.013595+0000","last_active":"2024-02-05T08:18:40.787647+0000","last_peered":"2024-02-05T08:18:40.787647+0000","last_clean":"2024-02-05T08:18:33.608376+0000","last_became_active":"2024-02-05T08:18:35.947792+0000","last_became_peered":"2024-02-05T08:18:35.947792+0000","last_unstale":"2024-02-05T08:18:40.787647+0000","last_undegraded":"2024-02-05T08:18:40.787647+0000","last_fullsized":"2024-02-05T08:18:40.787647+0000","mapping_epoch":295,"log_start":"281'11542","ondisk_log_start":"281'11542","created":239,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"285'13513","last_scrub_stamp":"2024-02-05T08:17:59.290228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:56.736469+0000","last_clean_scrub_stamp":"2024-02-05T08:17:59.290228+0000","objects_scrubbed":2946,"log_size":4892,"log_dups_size":0,"ondisk_log_size":4892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16434,"num_write_kb":525888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[4,6,7],"avail_no_missing":["4","3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":506,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823816+0000","last_change":"2024-02-05T08:16:52.875421+0000","last_active":"2024-02-05T08:18:38.823816+0000","last_peered":"2024-02-05T08:18:38.823816+0000","last_clean":"2024-02-05T08:18:38.823816+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:18:38.823816+0000","last_undegraded":"2024-02-05T08:18:38.823816+0000","last_fullsized":"2024-02-05T08:18:38.823816+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:52.874790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:16:52.874790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":712,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.824916+0000","last_change":"2024-02-05T08:18:29.716104+0000","last_active":"2024-02-05T08:18:38.824916+0000","last_peered":"2024-02-05T08:18:38.824916+0000","last_clean":"2024-02-05T08:18:38.824916+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:18:38.824916+0000","last_undegraded":"2024-02-05T08:18:38.824916+0000","last_fullsized":"2024-02-05T08:18:38.824916+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:18:29.715977+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:18:29.715977+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:40.647242+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.827338+0000","last_change":"2024-02-05T08:18:34.931257+0000","last_active":"2024-02-05T08:18:38.827338+0000","last_peered":"2024-02-05T08:18:38.827338+0000","last_clean":"2024-02-05T08:18:38.827338+0000","last_became_active":"2024-02-05T08:18:34.663738+0000","last_became_peered":"2024-02-05T08:18:34.663738+0000","last_unstale":"2024-02-05T08:18:38.827338+0000","last_undegraded":"2024-02-05T08:18:38.827338+0000","last_fullsized":"2024-02-05T08:18:38.827338+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.931138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.931138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:47.724113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"236'20","reported_seq":624,"reported_epoch":298,"state":"active+clean","last_fresh":"2024-02-05T08:18:38.823763+0000","last_change":"2024-02-05T08:18:34.732210+0000","last_active":"2024-02-05T08:18:38.823763+0000","last_peered":"2024-02-05T08:18:38.823763+0000","last_clean":"2024-02-05T08:18:38.823763+0000","last_became_active":"2024-02-05T08:18:34.637413+0000","last_became_peered":"2024-02-05T08:18:34.637413+0000","last_unstale":"2024-02-05T08:18:38.823763+0000","last_undegraded":"2024-02-05T08:18:38.823763+0000","last_fullsized":"2024-02-05T08:18:38.823763+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:18:00.684227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:00.684227+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:09.647929+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":26411008,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":263073,"num_write_kb":8431169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":576,"num_bytes_recovered":37421056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":50069504,"data_stored":50069504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77383,"ondisk_log_size":77383,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":834,"num_read_kb":829,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":311,"num_bytes_recovered":5930,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":765,"ondisk_log_size":765,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037933,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":505948,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93239204,"statfs":{"total":95995035648,"available":95476944896,"internally_reserved":0,"allocated":1396736,"data_stored":1075854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":516684234},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":2.109,"5min":1.855,"15min":1.4950000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":27.376000000000001,"5min":52.488999999999997,"15min":60.127000000000002},"last":0.51100000000000001},{"interface":"front","average":{"1min":2.782,"5min":2.0870000000000002,"15min":1.5640000000000001},"min":{"1min":0.27100000000000002,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":27.456,"5min":52.335000000000001,"15min":58.283000000000001},"last":0.499}]},{"osd":1,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":2.3069999999999999,"5min":1.617,"15min":1.476},"min":{"1min":0.252,"5min":0.23400000000000001,"15min":0.22900000000000001},"max":{"1min":27.518999999999998,"5min":27.518999999999998,"15min":38.399999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":2.2759999999999998,"5min":1.4890000000000001,"15min":1.333},"min":{"1min":0.27600000000000002,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":27.503,"5min":27.503,"15min":38.195999999999998},"last":0.48099999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.1559999999999999,"15min":0.95599999999999996},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.24099999999999999},"max":{"1min":3.238,"5min":7.5709999999999997,"15min":7.6779999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.226,"15min":1.0740000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.24299999999999999},"max":{"1min":3.9820000000000002,"5min":5.9400000000000004,"15min":8.9100000000000001},"last":0.45600000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.117,"15min":1.284},"min":{"1min":0.29699999999999999,"5min":0.28199999999999997,"15min":0.249},"max":{"1min":4.0209999999999999,"5min":5.984,"15min":58.515000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.0880000000000001,"15min":1.2609999999999999},"min":{"1min":0.35799999999999998,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":4.0960000000000001,"5min":5.391,"15min":58.225999999999999},"last":0.442}]},{"osd":4,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":1.571,"15min":0.876},"min":{"1min":0.36799999999999999,"5min":0.26500000000000001,"15min":0.22},"max":{"1min":2.9670000000000001,"5min":49.601999999999997,"15min":49.601999999999997},"last":0.36099999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.617,"15min":0.90400000000000003},"min":{"1min":0.28199999999999997,"5min":0.28100000000000003,"15min":0.249},"max":{"1min":4.0510000000000002,"5min":49.795999999999999,"15min":49.795999999999999},"last":0.40600000000000003}]},{"osd":5,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.9570000000000001,"15min":0.95899999999999996},"min":{"1min":0.32100000000000001,"5min":0.25800000000000001,"15min":0.24299999999999999},"max":{"1min":5.1769999999999996,"5min":49.914000000000001,"15min":49.914000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.617,"5min":1.879,"15min":0.94899999999999995},"min":{"1min":0.432,"5min":0.314,"15min":0.27000000000000002},"max":{"1min":7.2350000000000003,"5min":52.256,"15min":52.256},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Feb 5 08:17:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.296,"15min":0.76000000000000001},"min":{"1min":0.28799999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.1840000000000002,"5min":8.0050000000000008,"15min":8.0050000000000008},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.2210000000000001,"15min":0.73499999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.27300000000000002},"max":{"1min":3.9420000000000002,"5min":7.8029999999999999,"15min":7.8029999999999999},"last":0.48799999999999999}]}]},{"osd":6,"up_from":51,"seq":219043332275,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311560,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":93433592,"statfs":{"total":95995035648,"available":95675998208,"internally_reserved":0,"allocated":1376256,"data_stored":1074327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":317651398},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:18:26 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":3.2879999999999998,"15min":1.706},"min":{"1min":0.22500000000000001,"5min":0.215,"15min":0.215},"max":{"1min":5.7140000000000004,"5min":102.819,"15min":102.819},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.302,"5min":3.109,"15min":1.7729999999999999},"min":{"1min":0.22600000000000001,"5min":0.214,"15min":0.214},"max":{"1min":3.6240000000000001,"5min":102.75,"15min":102.75},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:18:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.611,"15min":1.1719999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.22700000000000001},"max":{"1min":6.0229999999999997,"5min":23.773,"15min":23.773},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.88500000000000001,"5min":1.4650000000000001,"15min":1.115},"min":{"1min":0.26300000000000001,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":2.5649999999999999,"5min":23.719000000000001,"15min":23.719000000000001},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:18:26 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.095,"15min":1.0389999999999999},"min":{"1min":0.30099999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.6580000000000004,"5min":22.503,"15min":22.503},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.1659999999999999,"15min":0.98799999999999999},"min":{"1min":0.33100000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.5859999999999999,"5min":22.466000000000001,"15min":22.466000000000001},"last":0.52900000000000003}]},{"osd":3,"last update":"Mon Feb 5 08:18:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":1.9530000000000001,"15min":1.323},"min":{"1min":0.34799999999999998,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":2.7240000000000002,"5min":102.438,"15min":102.438},"last":0.442},{"interface":"front","average":{"1min":0.86599999999999999,"5min":1.9690000000000001,"15min":1.327},"min":{"1min":0.34799999999999998,"5min":0.32200000000000001,"15min":0.29099999999999998},"max":{"1min":2.423,"5min":102.288,"15min":102.288},"last":0.45400000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:18:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.1080000000000001,"15min":0.66800000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.223},"max":{"1min":3.5499999999999998,"5min":9.8970000000000002,"15min":9.8970000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.482,"5min":1.1080000000000001,"15min":0.68999999999999995},"min":{"1min":0.30299999999999999,"5min":0.28599999999999998,"15min":0.26600000000000001},"max":{"1min":4.8579999999999997,"5min":4.9589999999999996,"15min":4.9589999999999996},"last":0.47099999999999997}]},{"osd":5,"last update":"Mon Feb 5 08:18:26 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.5529999999999999,"15min":1.0089999999999999},"min":{"1min":0.378,"5min":0.29299999999999998,"15min":0.24199999999999999},"max":{"1min":5.4340000000000002,"5min":24.675999999999998,"15min":43.643000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.806,"5min":1.4690000000000001,"15min":1.0009999999999999},"min":{"1min":0.32300000000000001,"5min":0.31900000000000001,"15min":0.30399999999999999},"max":{"1min":4.8979999999999997,"5min":19.132000000000001,"15min":43.893000000000001},"last":0.56299999999999994}]},{"osd":7,"last update":"Mon Feb 5 08:18:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.415,"15min":0.82299999999999995},"min":{"1min":0.40000000000000002,"5min":0.32600000000000001,"15min":0.29999999999999999},"max":{"1min":4.1900000000000004,"5min":24.728999999999999,"15min":24.728999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.905,"5min":1.4830000000000001,"15min":0.84899999999999998},"min":{"1min":0.40799999999999997,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":4.5979999999999999,"5min":21.91,"15min":21.91},"last":0.498}]}]},{"osd":1,"up_from":13,"seq":55834575052,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":533164,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":93211988,"statfs":{"total":95995035648,"available":95449075712,"internally_reserved":0,"allocated":823296,"data_stored":485084,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":1.482,"15min":1.619},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":2.9420000000000002,"5min":36.462000000000003,"15min":86.105000000000004},"last":0.221},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.6299999999999999,"15min":1.863},"min":{"1min":0.23899999999999999,"5min":0.21099999999999999,"15min":0.20699999999999999},"max":{"1min":2.7000000000000002,"5min":35.878,"15min":85.665000000000006},"last":0.40999999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:17:50 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":1.298,"15min":1.3620000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.17799999999999999},"max":{"1min":3.5299999999999998,"5min":19.016999999999999,"15min":49.030999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.5249999999999999,"15min":1.544},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":2.351,"5min":20.434000000000001,"15min":48.984000000000002},"last":0.255}]},{"osd":3,"last update":"Mon Feb 5 08:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.2330000000000001,"15min":1.5129999999999999},"min":{"1min":0.311,"5min":0.222,"15min":0.20599999999999999},"max":{"1min":2.9159999999999999,"5min":31.896000000000001,"15min":49.067999999999998},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.84199999999999997,"5min":1.2010000000000001,"15min":1.4530000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.24199999999999999},"max":{"1min":4.2279999999999998,"5min":32.018999999999998,"15min":48.734999999999999},"last":0.39600000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:18:37 2024","interfaces":[{"interface":"back","average":{"1min":2.5779999999999998,"5min":1.5600000000000001,"15min":1.268},"min":{"1min":0.371,"5min":0.33100000000000002,"15min":0.26000000000000001},"max":{"1min":24.196999999999999,"5min":24.196999999999999,"15min":25.952000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":2.4510000000000001,"5min":1.5089999999999999,"15min":1.3560000000000001},"min":{"1min":0.314,"5min":0.31,"15min":0.27100000000000002},"max":{"1min":24.215,"5min":24.215,"15min":25.922000000000001},"last":0.33700000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:17:56 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":2.2050000000000001,"15min":1.361},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.28699999999999998},"max":{"1min":4.8200000000000003,"5min":54.552999999999997,"15min":54.552999999999997},"last":0.436},{"interface":"front","average":{"1min":1.298,"5min":2.0840000000000001,"15min":1.387},"min":{"1min":0.311,"5min":0.311,"15min":0.28999999999999998},"max":{"1min":2.972,"5min":55.811,"15min":55.811},"last":0.32000000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:18:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.528,"15min":1.2330000000000001},"min":{"1min":0.41399999999999998,"5min":0.35199999999999998,"15min":0.33100000000000002},"max":{"1min":2.968,"5min":54.384,"15min":54.384},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.615,"15min":1.244},"min":{"1min":0.37,"5min":0.36899999999999999,"15min":0.34899999999999998},"max":{"1min":2.8769999999999998,"5min":55.210999999999999,"15min":55.210999999999999},"last":0.45700000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:17:50 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.5329999999999999,"15min":1.2470000000000001},"min":{"1min":0.34300000000000003,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":4.2400000000000002,"5min":7.9390000000000001,"15min":13.749000000000001},"last":0.37},{"interface":"front","average":{"1min":1.444,"5min":1.474,"15min":1.2110000000000001},"min":{"1min":0.44800000000000001,"5min":0.379,"15min":0.379},"max":{"1min":3.601,"5min":7.9640000000000004,"15min":7.9640000000000004},"last":0.42399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738576,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":577048,"kb_used_data":50252,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":93168104,"statfs":{"total":95995035648,"available":95404138496,"internally_reserved":0,"allocated":51458048,"data_stored":51136694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:18:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.86899999999999999,"15min":1.7909999999999999},"min":{"1min":0.20499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":2.2999999999999998,"5min":7.9580000000000002,"15min":66.772999999999996},"last":0.36799999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.0389999999999999,"15min":1.53},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":3.7050000000000001,"5min":7.4850000000000003,"15min":46.198999999999998},"last":0.60899999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:17:49 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.4590000000000001,"15min":1.496},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.649,"5min":12.074,"15min":46.170999999999999},"last":1.758},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.3500000000000001,"15min":1.625},"min":{"1min":0.26100000000000001,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":2.2890000000000001,"5min":15.569000000000001,"15min":46.543999999999997},"last":0.63}]},{"osd":3,"last update":"Mon Feb 5 08:18:24 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.0489999999999999,"15min":1.6850000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.21099999999999999},"max":{"1min":2.3639999999999999,"5min":7.3929999999999998,"15min":67.888000000000005},"last":3.758},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.94599999999999995,"15min":1.984},"min":{"1min":0.36599999999999999,"5min":0.23999999999999999,"15min":0.22600000000000001},"max":{"1min":2.6349999999999998,"5min":8.0239999999999991,"15min":66.891999999999996},"last":0.58799999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:17:42 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":1.667,"15min":1.2190000000000001},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":2.6739999999999999,"5min":22.920000000000002,"15min":22.920000000000002},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.4750000000000001,"15min":1.1930000000000001},"min":{"1min":0.39400000000000002,"5min":0.33800000000000002,"15min":0.26900000000000002},"max":{"1min":2.7610000000000001,"5min":22.983000000000001,"15min":22.983000000000001},"last":1.8069999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:18:07 2024","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.7450000000000001,"15min":1.3420000000000001},"min":{"1min":0.437,"5min":0.34799999999999998,"15min":0.28299999999999997},"max":{"1min":4.8559999999999999,"5min":22.800999999999998,"15min":22.800999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.627,"15min":1.266},"min":{"1min":0.35999999999999999,"5min":0.31900000000000001,"15min":0.29599999999999999},"max":{"1min":4.8099999999999996,"5min":22.744,"15min":22.744},"last":3.722}]},{"osd":6,"last update":"Mon Feb 5 08:18:31 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.1479999999999999,"15min":1.167},"min":{"1min":0.35299999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.113,"5min":7.8220000000000001,"15min":12.967000000000001},"last":1.8620000000000001},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1899999999999999,"15min":1.266},"min":{"1min":0.50700000000000001,"5min":0.34200000000000003,"15min":0.32300000000000001},"max":{"1min":3.6619999999999999,"5min":4.1210000000000004,"15min":15.721},"last":0.68899999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:17:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":2.552,"15min":1.54},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.9820000000000002,"5min":68.241,"15min":68.241},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.3839999999999999,"5min":2.395,"15min":1.577},"min":{"1min":0.40400000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":3.1339999999999999,"5min":68.980000000000004,"15min":68.980000000000004},"last":1.839}]}]},{"osd":2,"up_from":18,"seq":77309411527,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":501212,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93243940,"statfs":{"total":95995035648,"available":95481794560,"internally_reserved":0,"allocated":806912,"data_stored":484682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":512424390},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":2.3450000000000002,"15min":2.024},"min":{"1min":0.20499999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":13.638,"5min":107.155,"15min":107.155},"last":0.29699999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":2.6360000000000001,"15min":2.173},"min":{"1min":0.22600000000000001,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":10.782999999999999,"5min":107.221,"15min":107.221},"last":0.373}]},{"osd":1,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.054,"15min":1.446},"min":{"1min":0.23599999999999999,"5min":0.19400000000000001,"15min":0.17999999999999999},"max":{"1min":12.394,"5min":12.394,"15min":55.942},"last":0.26600000000000001},{"interface":"front","average":{"1min":1.272,"5min":1.3029999999999999,"15min":1.5820000000000001},"min":{"1min":0.28799999999999998,"5min":0.23400000000000001,"15min":0.193},"max":{"1min":10.752000000000001,"5min":10.871,"15min":55.960999999999999},"last":0.40799999999999997}]},{"osd":3,"last update":"Mon Feb 5 08:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.96699999999999997,"15min":1.764},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.20499999999999999},"max":{"1min":4.0570000000000004,"5min":8.6590000000000007,"15min":68.108000000000004},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.72999999999999998,"15min":1.5609999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":1.351,"5min":3.98,"15min":67.945999999999998},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:18:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.3089999999999999,"15min":1.385},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.27400000000000002},"max":{"1min":4.101,"5min":8.2620000000000005,"15min":31.596},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.516,"5min":1.1910000000000001,"15min":1.21},"min":{"1min":0.32800000000000001,"5min":0.30299999999999999,"15min":0.252},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":31.57},"last":0.53300000000000003}]},{"osd":5,"last update":"Mon Feb 5 08:18:03 2024","interfaces":[{"interface":"back","average":{"1min":2.8300000000000001,"5min":1.736,"15min":1.4870000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":26.373000000000001,"5min":26.373000000000001,"15min":58.799999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":2.4140000000000001,"5min":1.393,"15min":1.4119999999999999},"min":{"1min":0.33800000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":27.968,"5min":27.968,"15min":58.615000000000002},"last":0.56100000000000005}]},{"osd":6,"last update":"Mon Feb 5 08:18:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1830000000000001,"15min":1.2250000000000001},"min":{"1min":0.37,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.202,"5min":8.3089999999999993,"15min":15.859},"last":0.51400000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.127,"15min":1.089},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":9.1739999999999995,"5min":9.1739999999999995,"15min":9.1739999999999995},"last":0.47899999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.2270000000000001,"15min":1.194},"min":{"1min":0.34699999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":7.7530000000000001,"5min":7.7530000000000001,"15min":15.986000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.649,"5min":1.587,"15min":1.3580000000000001},"min":{"1min":0.503,"5min":0.41699999999999998,"15min":0.35299999999999998},"max":{"1min":4.2160000000000002,"5min":8.3369999999999997,"15min":15.951000000000001},"last":0.58999999999999997}]}]},{"osd":3,"up_from":27,"seq":115964117186,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":397100,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":93348052,"statfs":{"total":95995035648,"available":95588405248,"internally_reserved":0,"allocated":823296,"data_stored":485339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":405797321},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:18:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.5049999999999999,"15min":1.75},"min":{"1min":0.20799999999999999,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":2.4540000000000002,"5min":52.831000000000003,"15min":63.43},"last":1.034},{"interface":"front","average":{"1min":0.70999999999999996,"5min":1.556,"15min":1.8620000000000001},"min":{"1min":0.22700000000000001,"5min":0.184,"15min":0.184},"max":{"1min":2.6019999999999999,"5min":53.859999999999999,"15min":63.048999999999999},"last":6.6779999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:18:15 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":1.137,"15min":1.5049999999999999},"min":{"1min":0.248,"5min":0.23200000000000001,"15min":0.217},"max":{"1min":1.0149999999999999,"5min":23.154,"15min":51.405999999999999},"last":3.3119999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.161,"15min":1.8149999999999999},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.17399999999999999},"max":{"1min":3.3370000000000002,"5min":23.207000000000001,"15min":51.189999999999998},"last":3.54}]},{"osd":2,"last update":"Mon Feb 5 08:18:15 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.84299999999999997,"15min":1.5169999999999999},"min":{"1min":0.25800000000000001,"5min":0.251,"15min":0.251},"max":{"1min":3.1179999999999999,"5min":6.2709999999999999,"15min":62.902999999999999},"last":3.5649999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.81799999999999995,"15min":1.3440000000000001},"min":{"1min":0.26900000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.208,"5min":4.8179999999999996,"15min":63.468000000000004},"last":0.83499999999999996}]},{"osd":4,"last update":"Mon Feb 5 08:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.997,"15min":1.056},"min":{"1min":0.33200000000000002,"5min":0.30499999999999999,"15min":0.28799999999999998},"max":{"1min":3.3769999999999998,"5min":4.9889999999999999,"15min":7.7229999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.111,"5min":0.88400000000000001,"15min":1.1080000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.29599999999999999},"max":{"1min":5.8159999999999998,"5min":5.8159999999999998,"15min":10.539999999999999},"last":3.452}]},{"osd":5,"last update":"Mon Feb 5 08:18:03 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":2.8740000000000001,"15min":1.716},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.255},"max":{"1min":2.5470000000000002,"5min":117.432,"15min":117.432},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.2350000000000001,"5min":2.706,"15min":1.752},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.30299999999999999},"max":{"1min":3.9390000000000001,"5min":114.828,"15min":114.828},"last":3.3660000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.93100000000000005,"15min":1.1559999999999999},"min":{"1min":0.373,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":2.855,"5min":3.6360000000000001,"15min":10.622},"last":3.343},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.94399999999999995,"15min":1.1240000000000001},"min":{"1min":0.36299999999999999,"5min":0.32500000000000001,"15min":0.308},"max":{"1min":2.6099999999999999,"5min":5.9779999999999998,"15min":7.8639999999999999},"last":3.5910000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:17:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":2.8730000000000002,"15min":1.6679999999999999},"min":{"1min":0.42699999999999999,"5min":0.36799999999999999,"15min":0.35599999999999998},"max":{"1min":6.0990000000000002,"5min":114.84999999999999,"15min":114.84999999999999},"last":1.5880000000000001},{"interface":"front","average":{"1min":0.92200000000000004,"5min":2.7290000000000001,"15min":1.724},"min":{"1min":0.40200000000000002,"5min":0.38100000000000001,"15min":0.33600000000000002},"max":{"1min":2.27,"5min":114.871,"15min":114.871},"last":3.496}]}]},{"osd":4,"up_from":32,"seq":137438953661,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":410972,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93334180,"statfs":{"total":95995035648,"available":95574200320,"internally_reserved":0,"allocated":806912,"data_stored":484133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:18:27 2024","interfaces":[{"interface":"back","average":{"1min":2.0739999999999998,"5min":1.643,"15min":1.6379999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":25.600999999999999,"5min":39.552,"15min":73.078999999999994},"last":0.38800000000000001},{"interface":"front","average":{"1min":2.4249999999999998,"5min":1.893,"15min":1.6599999999999999},"min":{"1min":0.23599999999999999,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":23.030999999999999,"5min":39.350999999999999,"15min":73.412000000000006},"last":0.32500000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:18:27 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":2.6539999999999999,"15min":1.484},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.21099999999999999},"max":{"1min":3.3690000000000002,"5min":125.054,"15min":125.054},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":2.5920000000000001,"15min":1.387},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.23499999999999999},"max":{"1min":3.044,"5min":125.08799999999999,"15min":125.08799999999999},"last":0.309}]},{"osd":2,"last update":"Mon Feb 5 08:18:27 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.0529999999999999,"15min":1.248},"min":{"1min":0.32300000000000001,"5min":0.247,"15min":0.24199999999999999},"max":{"1min":4.8159999999999998,"5min":4.8159999999999998,"15min":96.849999999999994},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1759999999999999,"15min":1.325},"min":{"1min":0.27100000000000002,"5min":0.26700000000000002,"15min":0.26300000000000001},"max":{"1min":4.0940000000000003,"5min":5.6879999999999997,"15min":96.143000000000001},"last":0.374}]},{"osd":3,"last update":"Mon Feb 5 08:18:27 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.613,"15min":1.8280000000000001},"min":{"1min":0.33100000000000002,"5min":0.31900000000000001,"15min":0.24399999999999999},"max":{"1min":4.0670000000000002,"5min":38.985999999999997,"15min":73.602999999999994},"last":0.44600000000000001},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.4810000000000001,"15min":1.8240000000000001},"min":{"1min":0.39300000000000002,"5min":0.309,"15min":0.27200000000000002},"max":{"1min":3.4350000000000001,"5min":39.290999999999997,"15min":74.817999999999998},"last":0.43099999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:18:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.5620000000000001,"15min":0.81799999999999995},"min":{"1min":0.32900000000000001,"5min":0.29399999999999998,"15min":0.224},"max":{"1min":3.754,"5min":34.167000000000002,"15min":34.167000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.7330000000000001,"15min":0.89100000000000001},"min":{"1min":0.28899999999999998,"5min":0.254,"15min":0.254},"max":{"1min":3.9990000000000001,"5min":36.051000000000002,"15min":36.051000000000002},"last":0.41799999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:18:27 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.1830000000000001,"15min":0.71699999999999997},"min":{"1min":0.38100000000000001,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":4.0380000000000003,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.403,"5min":1.0640000000000001,"15min":0.67800000000000005},"min":{"1min":0.373,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.577,"5min":5.5049999999999999,"15min":5.5049999999999999},"last":0.29299999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:17:43 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.1950000000000001,"15min":0.72099999999999997},"min":{"1min":0.38,"5min":0.314,"15min":0.27000000000000002},"max":{"1min":4.8129999999999997,"5min":4.8129999999999997,"15min":4.8129999999999997},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.246,"5min":1.0820000000000001,"15min":0.70199999999999996},"min":{"1min":0.40500000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.3940000000000001,"5min":5.7450000000000001,"15min":5.7450000000000001},"last":0.33900000000000002}]}]},{"osd":5,"up_from":42,"seq":180388626615,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":549768,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":548990,"kb_avail":93195384,"statfs":{"total":95995035648,"available":95432073216,"internally_reserved":0,"allocated":786432,"data_stored":483996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":562166213},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":3.0779999999999998,"5min":2.0630000000000002,"15min":1.9119999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.19400000000000001},"max":{"1min":30.065999999999999,"5min":30.065999999999999,"15min":143.06299999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":2.831,"5min":1.806,"15min":1.8340000000000001},"min":{"1min":0.308,"5min":0.24299999999999999,"15min":0.20499999999999999},"max":{"1min":30.173999999999999,"5min":30.173999999999999,"15min":143.09700000000001},"last":0.52000000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.889,"5min":1.4359999999999999,"15min":1.1299999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.215},"max":{"1min":4.141,"5min":5.9000000000000004,"15min":46.200000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.675,"5min":1.1799999999999999,"15min":1.196},"min":{"1min":0.28699999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":5.2210000000000001,"5min":5.2210000000000001,"15min":46.343000000000004},"last":0.48499999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.2769999999999999,"15min":1.0449999999999999},"min":{"1min":0.34100000000000003,"5min":0.28999999999999998,"15min":0.246},"max":{"1min":7.1159999999999997,"5min":7.1159999999999997,"15min":10.733000000000001},"last":0.496},{"interface":"front","average":{"1min":2.008,"5min":1.476,"15min":1.143},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.79,"5min":5.8250000000000002,"15min":8.0220000000000002},"last":0.38100000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.6950000000000001,"5min":1.1990000000000001,"15min":1.6839999999999999},"min":{"1min":0.34699999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":7.165,"5min":7.165,"15min":143.03399999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.748,"5min":1.581,"15min":1.806},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.28100000000000003},"max":{"1min":4.1929999999999996,"5min":7.0309999999999997,"15min":143.46799999999999},"last":0.53000000000000003}]},{"osd":4,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.3640000000000001,"15min":0.749},"min":{"1min":0.40000000000000002,"5min":0.24399999999999999,"15min":0.23699999999999999},"max":{"1min":5.0819999999999999,"5min":5.3769999999999998,"15min":5.3769999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.6220000000000001,"5min":1.248,"15min":0.71099999999999997},"min":{"1min":0.29899999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":7.2380000000000004,"5min":7.2380000000000004,"15min":7.2380000000000004},"last":0.312}]},{"osd":6,"last update":"Mon Feb 5 08:18:31 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.4139999999999999,"15min":0.79300000000000004},"min":{"1min":0.432,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.952,"5min":7.5439999999999996,"15min":7.5439999999999996},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.869,"5min":1.4299999999999999,"15min":0.82599999999999996},"min":{"1min":0.41899999999999998,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":4.8170000000000002,"5min":7.194,"15min":7.194},"last":0.35799999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:17:52 2024","interfaces":[{"interface":"back","average":{"1min":2.371,"5min":1.6819999999999999,"15min":0.89800000000000002},"min":{"1min":0.41399999999999998,"5min":0.30399999999999999,"15min":0.25900000000000001},"max":{"1min":7.0330000000000004,"5min":9.6940000000000008,"15min":9.6940000000000008},"last":0.436},{"interface":"front","average":{"1min":2.274,"5min":1.5149999999999999,"15min":0.84699999999999998},"min":{"1min":0.65000000000000002,"5min":0.318,"15min":0.28199999999999997},"max":{"1min":5.2670000000000003,"5min":8.0289999999999999,"15min":8.0289999999999999},"last":0.40100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":50069504,"data_stored":50069504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:18:45.848 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T08:18:45.848 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-02-05T08:18:45.861 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:45.915 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:45.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:45.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:45.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:45 smithi196 ceph-mon[33018]: 6.a scrub starts 2024-02-05T08:18:45.932 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:45 smithi196 ceph-mon[33018]: 6.a scrub ok 2024-02-05T08:18:45.933 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:45 smithi196 ceph-mon[33018]: 4.6 scrub starts 2024-02-05T08:18:45.933 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:45 smithi196 ceph-mon[33018]: 4.6 scrub ok 2024-02-05T08:18:46.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:46.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:46.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:46.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:46.088 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:45 smithi134 ceph-mon[30086]: 6.a scrub starts 2024-02-05T08:18:46.088 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:45 smithi134 ceph-mon[30086]: 6.a scrub ok 2024-02-05T08:18:46.088 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:45 smithi134 ceph-mon[30086]: 4.6 scrub starts 2024-02-05T08:18:46.088 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:45 smithi134 ceph-mon[30086]: 4.6 scrub ok 2024-02-05T08:18:46.089 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:45 smithi134 ceph-mon[39278]: 6.a scrub starts 2024-02-05T08:18:46.089 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:45 smithi134 ceph-mon[39278]: 6.a scrub ok 2024-02-05T08:18:46.089 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:45 smithi134 ceph-mon[39278]: 4.6 scrub starts 2024-02-05T08:18:46.089 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:45 smithi134 ceph-mon[39278]: 4.6 scrub ok 2024-02-05T08:18:46.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:46.169 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:46.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:46.203 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:18:46.203 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:18:46.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:18:46.203 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:18:46.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.203 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136491 8.0 8:00352e96:::benchmark_data_smithi134_115836_object19831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:18:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.657285+0000", 2024-02-05T08:18:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.54365830199998, 2024-02-05T08:18:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.037191507, 2024-02-05T08:18:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:18:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136491 2024-02-05T08:18:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657285+0000", 2024-02-05T08:18:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657285+0000", 2024-02-05T08:18:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657286+0000", 2024-02-05T08:18:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4200000000000005e-07 2024-02-05T08:18:46.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657290+0000", 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1289999999999998e-06 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657303+0000", 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2362e-05 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657309+0000", 2024-02-05T08:18:46.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9159999999999998e-06 2024-02-05T08:18:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657364+0000", 2024-02-05T08:18:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5350999999999999e-05 2024-02-05T08:18:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657459+0000", 2024-02-05T08:18:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5309999999999994e-05 2024-02-05T08:18:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657540+0000", 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0322999999999997e-05 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.659672+0000", 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021321059999999999 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.659795+0000", 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000123063 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.664541+0000", 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047457879999999999 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.678869+0000", 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014328278999999999 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.678879+0000", 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0196e-05 2024-02-05T08:18:46.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.694389+0000", 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.015510425 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.694477+0000", 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7416999999999993e-05 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136499 8.0 8:0b745435:::benchmark_data_smithi134_115836_object19839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:18:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.669853+0000", 2024-02-05T08:18:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.53109104399999, 2024-02-05T08:18:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.025200141, 2024-02-05T08:18:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:18:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136499 2024-02-05T08:18:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.211 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669853+0000", 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669853+0000", 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669854+0000", 2024-02-05T08:18:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8000000000000004e-07 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669858+0000", 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1649999999999999e-06 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669872+0000", 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4574e-05 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669878+0000", 2024-02-05T08:18:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5770000000000001e-06 2024-02-05T08:18:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669926+0000", 2024-02-05T08:18:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8191e-05 2024-02-05T08:18:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.670025+0000", 2024-02-05T08:18:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8585999999999994e-05 2024-02-05T08:18:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:18:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.670114+0000", 2024-02-05T08:18:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9820000000000004e-05 2024-02-05T08:18:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.672248+0000", 2024-02-05T08:18:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021340370000000001 2024-02-05T08:18:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.672356+0000", 2024-02-05T08:18:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107456 2024-02-05T08:18:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.678926+0000", 2024-02-05T08:18:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065703549999999999 2024-02-05T08:18:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.678934+0000", 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3599999999999998e-06 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.694985+0000", 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0160511160000001 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.695053+0000", 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8023999999999998e-05 2024-02-05T08:18:46.217 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.218 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.218 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136506 8.0 8:02587487:::benchmark_data_smithi134_115836_object19846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.680322+0000", 2024-02-05T08:18:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.52062183700002, 2024-02-05T08:18:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.024271468, 2024-02-05T08:18:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136506 2024-02-05T08:18:46.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680322+0000", 2024-02-05T08:18:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680322+0000", 2024-02-05T08:18:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680323+0000", 2024-02-05T08:18:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5859999999999999e-06 2024-02-05T08:18:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680336+0000", 2024-02-05T08:18:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2333e-05 2024-02-05T08:18:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680350+0000", 2024-02-05T08:18:46.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4080000000000001e-05 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680359+0000", 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0569999999999999e-06 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680413+0000", 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4431000000000001e-05 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680421+0000", 2024-02-05T08:18:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2570000000000001e-06 2024-02-05T08:18:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.695096+0000", 2024-02-05T08:18:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.014675564 2024-02-05T08:18:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.695221+0000", 2024-02-05T08:18:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000125186 2024-02-05T08:18:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.695317+0000", 2024-02-05T08:18:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5402000000000006e-05 2024-02-05T08:18:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704478+0000", 2024-02-05T08:18:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0091610249999999997 2024-02-05T08:18:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704494+0000", 2024-02-05T08:18:46.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5965999999999999e-05 2024-02-05T08:18:46.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704537+0000", 2024-02-05T08:18:46.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2824999999999997e-05 2024-02-05T08:18:46.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704571+0000", 2024-02-05T08:18:46.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4183999999999997e-05 2024-02-05T08:18:46.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704593+0000", 2024-02-05T08:18:46.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2572e-05 2024-02-05T08:18:46.227 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.227 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.227 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.227 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136519 8.0 8:0144bb8f:::benchmark_data_smithi134_115836_object19859:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.227 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.697455+0000", 2024-02-05T08:18:46.227 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.50348847999999, 2024-02-05T08:18:46.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0072622899999999, 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136519 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.697455+0000", 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.697455+0000", 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.697456+0000", 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6199999999999997e-07 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.697460+0000", 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3649999999999997e-06 2024-02-05T08:18:46.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.697471+0000", 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0987e-05 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.697480+0000", 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8759999999999994e-06 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.700194+0000", 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002713615 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.700232+0000", 2024-02-05T08:18:46.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8225000000000001e-05 2024-02-05T08:18:46.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.698708+0000", 2024-02-05T08:18:46.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99847636799999995 2024-02-05T08:18:46.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.698833+0000", 2024-02-05T08:18:46.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000124256 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.698962+0000", 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012896000000000001 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704500+0000", 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055384709999999997 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704635+0000", 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013537299999999999 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704670+0000", 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4434000000000003e-05 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704699+0000", 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8796e-05 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704717+0000", 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8802000000000001e-05 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136524 8.0 8:003515e7:::benchmark_data_smithi134_115836_object19864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.706080+0000", 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.494863404, 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0065054609999999, 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.234 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136524 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706080+0000", 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706080+0000", 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706081+0000", 2024-02-05T08:18:46.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5499999999999996e-07 2024-02-05T08:18:46.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706086+0000", 2024-02-05T08:18:46.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1089999999999997e-06 2024-02-05T08:18:46.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706102+0000", 2024-02-05T08:18:46.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5302e-05 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706110+0000", 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7029999999999995e-06 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706277+0000", 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016661900000000001 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706286+0000", 2024-02-05T08:18:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9299999999999992e-06 2024-02-05T08:18:46.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.701831+0000", 2024-02-05T08:18:46.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99554492900000002 2024-02-05T08:18:46.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.701954+0000", 2024-02-05T08:18:46.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012360800000000001 2024-02-05T08:18:46.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.702061+0000", 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106554 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704762+0000", 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002701331 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704776+0000", 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3923e-05 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712452+0000", 2024-02-05T08:18:46.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.007675734 2024-02-05T08:18:46.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712534+0000", 2024-02-05T08:18:46.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2359000000000002e-05 2024-02-05T08:18:46.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712586+0000", 2024-02-05T08:18:46.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1405e-05 2024-02-05T08:18:46.241 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.241 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.241 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.241 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136525 8.0 8:0ae77d71:::benchmark_data_smithi134_115836_object19865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.241 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.706251+0000", 2024-02-05T08:18:46.241 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.49469221800001, 2024-02-05T08:18:46.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.012386306, 2024-02-05T08:18:46.241 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.241 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.241 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.241 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.242 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.242 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136525 2024-02-05T08:18:46.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.242 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706251+0000", 2024-02-05T08:18:46.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706251+0000", 2024-02-05T08:18:46.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706252+0000", 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3500000000000005e-07 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706257+0000", 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6079999999999998e-06 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706271+0000", 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3642000000000001e-05 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706279+0000", 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1359999999999997e-06 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706314+0000", 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5213000000000003e-05 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706319+0000", 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4979999999999997e-06 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704790+0000", 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99847085499999999 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704864+0000", 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3928000000000001e-05 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704940+0000", 2024-02-05T08:18:46.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5338000000000003e-05 2024-02-05T08:18:46.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712682+0000", 2024-02-05T08:18:46.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077425690000000004 2024-02-05T08:18:46.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712699+0000", 2024-02-05T08:18:46.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6458999999999999e-05 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718557+0000", 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058581809999999996 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718600+0000", 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3421999999999999e-05 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718638+0000", 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7622e-05 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136530 8.0 8:0397fdba:::benchmark_data_smithi134_115836_object19870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.712771+0000", 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.48817249500001, 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0059719009999999, 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.248 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136530 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712771+0000", 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712771+0000", 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712772+0000", 2024-02-05T08:18:46.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9599999999999999e-07 2024-02-05T08:18:46.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712779+0000", 2024-02-05T08:18:46.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9399999999999996e-06 2024-02-05T08:18:46.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712790+0000", 2024-02-05T08:18:46.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1144e-05 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712797+0000", 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.34e-06 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712880+0000", 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2985000000000006e-05 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712890+0000", 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7729999999999994e-06 2024-02-05T08:18:46.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.707451+0000", 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99456112200000002 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.707568+0000", 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117034 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.707678+0000", 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109516 2024-02-05T08:18:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712707+0000", 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050295879999999998 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712761+0000", 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3804000000000002e-05 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718673+0000", 2024-02-05T08:18:46.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059123350000000003 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718715+0000", 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1248999999999998e-05 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718743+0000", 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8374999999999999e-05 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136533 8.0 8:0b2b077c:::benchmark_data_smithi134_115836_object19873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.716799+0000", 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.48414423499997, 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0020862020000001, 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136533 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716799+0000", 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716799+0000", 2024-02-05T08:18:46.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716800+0000", 2024-02-05T08:18:46.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.85e-07 2024-02-05T08:18:46.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716804+0000", 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8829999999999999e-06 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716817+0000", 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3006e-05 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716824+0000", 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.328e-06 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716868+0000", 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3074999999999997e-05 2024-02-05T08:18:46.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716874+0000", 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0179999999999998e-06 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.710115+0000", 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99324165499999995 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.710199+0000", 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3902999999999999e-05 2024-02-05T08:18:46.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.710270+0000", 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1204999999999993e-05 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712836+0000", 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025652869999999999 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712850+0000", 2024-02-05T08:18:46.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4688999999999999e-05 2024-02-05T08:18:46.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718799+0000", 2024-02-05T08:18:46.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059487899999999998 2024-02-05T08:18:46.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718848+0000", 2024-02-05T08:18:46.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9363999999999997e-05 2024-02-05T08:18:46.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718886+0000", 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7214e-05 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136552 8.0 8:0d51f24e:::benchmark_data_smithi134_115836_object19892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.737390+0000", 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.46355326700001, 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98161826200000002, 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.261 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136552 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737390+0000", 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737390+0000", 2024-02-05T08:18:46.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737391+0000", 2024-02-05T08:18:46.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8999999999999995e-07 2024-02-05T08:18:46.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737396+0000", 2024-02-05T08:18:46.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8799999999999999e-06 2024-02-05T08:18:46.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737410+0000", 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3597e-05 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737418+0000", 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.242e-06 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737472+0000", 2024-02-05T08:18:46.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3678999999999999e-05 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737479+0000", 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5780000000000001e-06 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712881+0000", 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97540128100000001 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.713006+0000", 2024-02-05T08:18:46.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012547 2024-02-05T08:18:46.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.713129+0000", 2024-02-05T08:18:46.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000122784 2024-02-05T08:18:46.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718514+0000", 2024-02-05T08:18:46.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053848250000000002 2024-02-05T08:18:46.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718528+0000", 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4289e-05 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718931+0000", 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040299800000000001 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718976+0000", 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4570000000000002e-05 2024-02-05T08:18:46.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.719009+0000", 2024-02-05T08:18:46.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3179000000000002e-05 2024-02-05T08:18:46.268 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.268 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.268 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.268 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136555 8.0 8:068b584f:::benchmark_data_smithi134_115836_object19895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.268 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.739036+0000", 2024-02-05T08:18:46.268 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.461908118, 2024-02-05T08:18:46.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98997460400000004, 2024-02-05T08:18:46.269 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.269 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.269 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.269 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.269 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.269 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136555 2024-02-05T08:18:46.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.269 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739036+0000", 2024-02-05T08:18:46.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739036+0000", 2024-02-05T08:18:46.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739036+0000", 2024-02-05T08:18:46.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8100000000000001e-07 2024-02-05T08:18:46.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739041+0000", 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4689999999999999e-06 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739056+0000", 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4739000000000001e-05 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739064+0000", 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5299999999999996e-06 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739122+0000", 2024-02-05T08:18:46.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7986999999999999e-05 2024-02-05T08:18:46.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739131+0000", 2024-02-05T08:18:46.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5699999999999993e-06 2024-02-05T08:18:46.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.715762+0000", 2024-02-05T08:18:46.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97663171199999999 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.715882+0000", 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119974 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.716003+0000", 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120401 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.719060+0000", 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030575480000000002 2024-02-05T08:18:46.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.728903+0000", 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0098422530000000005 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.728924+0000", 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0871000000000001e-05 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.728972+0000", 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8936e-05 2024-02-05T08:18:46.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:46.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729010+0000", 2024-02-05T08:18:46.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7732999999999999e-05 2024-02-05T08:18:46.276 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.276 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136557 8.0 8:08e98751:::benchmark_data_smithi134_115836_object19897:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.742226+0000", 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.45871729700002, 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99639465800000004, 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136557 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742226+0000", 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742226+0000", 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742228+0000", 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.113e-06 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742232+0000", 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3309999999999997e-06 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742248+0000", 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6107e-05 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742256+0000", 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4330000000000004e-06 2024-02-05T08:18:46.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742341+0000", 2024-02-05T08:18:46.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4172999999999995e-05 2024-02-05T08:18:46.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742349+0000", 2024-02-05T08:18:46.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3799999999999994e-06 2024-02-05T08:18:46.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.719097+0000", 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97674827900000005 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.719221+0000", 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000124084 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.719387+0000", 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016600999999999999 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729020+0000", 2024-02-05T08:18:46.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0096322219999999993 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729069+0000", 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9456999999999998e-05 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738523+0000", 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.009454278 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738579+0000", 2024-02-05T08:18:46.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5782999999999998e-05 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738621+0000", 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2008000000000002e-05 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136562 8.0 8:0611ba46:::benchmark_data_smithi134_115836_object19902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.745425+0000", 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.45551871800001, 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99331518600000002, 2024-02-05T08:18:46.283 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136562 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745425+0000", 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745425+0000", 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745426+0000", 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8899999999999998e-07 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745431+0000", 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6870000000000002e-06 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745458+0000", 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7885000000000002e-05 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745469+0000", 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0129000000000001e-05 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745567+0000", 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8684999999999995e-05 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745578+0000", 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0891e-05 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.722102+0000", 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97652399700000003 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.722202+0000", 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100271 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.722420+0000", 2024-02-05T08:18:46.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021719600000000001 2024-02-05T08:18:46.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729025+0000", 2024-02-05T08:18:46.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066054420000000004 2024-02-05T08:18:46.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729107+0000", 2024-02-05T08:18:46.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1979999999999998e-05 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738665+0000", 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0095575179999999992 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738708+0000", 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.354e-05 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738740+0000", 2024-02-05T08:18:46.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2076000000000003e-05 2024-02-05T08:18:46.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.290 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.290 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.290 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136567 8.0 8:09cb9fd1:::benchmark_data_smithi134_115836_object19907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.290 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.757390+0000", 2024-02-05T08:18:46.290 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.44355393299998, 2024-02-05T08:18:46.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99000443400000004, 2024-02-05T08:18:46.290 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.290 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.290 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.290 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.291 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.291 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136567 2024-02-05T08:18:46.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.291 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757390+0000", 2024-02-05T08:18:46.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757390+0000", 2024-02-05T08:18:46.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757391+0000", 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.61e-07 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757394+0000", 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5899999999999999e-06 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757406+0000", 2024-02-05T08:18:46.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1481000000000001e-05 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757413+0000", 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4059999999999999e-06 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757473+0000", 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9667999999999999e-05 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757481+0000", 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4530000000000002e-06 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.725852+0000", 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96837125899999998 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.726036+0000", 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018403000000000001 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.726148+0000", 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011178799999999999 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729141+0000", 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029930619999999999 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738471+0000", 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0093298789999999993 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.747294+0000", 2024-02-05T08:18:46.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0088230219999999998 2024-02-05T08:18:46.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.747363+0000", 2024-02-05T08:18:46.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8349999999999994e-05 2024-02-05T08:18:46.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.747394+0000", 2024-02-05T08:18:46.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1684999999999999e-05 2024-02-05T08:18:46.297 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.297 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.297 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.297 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136569 8.0 8:020c7c43:::benchmark_data_smithi134_115836_object19909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.297 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.761088+0000", 2024-02-05T08:18:46.297 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.43985522600002, 2024-02-05T08:18:46.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98660557500000001, 2024-02-05T08:18:46.297 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.297 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.297 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136569 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761088+0000", 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761088+0000", 2024-02-05T08:18:46.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761089+0000", 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7900000000000007e-07 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761093+0000", 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8079999999999998e-06 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761103+0000", 2024-02-05T08:18:46.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0249e-05 2024-02-05T08:18:46.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761110+0000", 2024-02-05T08:18:46.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0599999999999996e-06 2024-02-05T08:18:46.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761153+0000", 2024-02-05T08:18:46.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3733999999999998e-05 2024-02-05T08:18:46.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761161+0000", 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6639999999999998e-06 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729171+0000", 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96800960300000005 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729280+0000", 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010945100000000001 2024-02-05T08:18:46.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729396+0000", 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116348 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738459+0000", 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0090630169999999996 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738476+0000", 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6878e-05 2024-02-05T08:18:46.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.747607+0000", 2024-02-05T08:18:46.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0091306809999999999 2024-02-05T08:18:46.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.747659+0000", 2024-02-05T08:18:46.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2420000000000003e-05 2024-02-05T08:18:46.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.747694+0000", 2024-02-05T08:18:46.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4683e-05 2024-02-05T08:18:46.304 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.304 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.304 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.304 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136574 8.0 8:0b467986:::benchmark_data_smithi134_115836_object19914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.304 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.774612+0000", 2024-02-05T08:18:46.304 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.42633124999998, 2024-02-05T08:18:46.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97985992499999996, 2024-02-05T08:18:46.304 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.304 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.305 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.305 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.305 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.305 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136574 2024-02-05T08:18:46.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.305 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774612+0000", 2024-02-05T08:18:46.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774612+0000", 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774614+0000", 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.629e-06 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774620+0000", 2024-02-05T08:18:46.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1920000000000002e-06 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774637+0000", 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6438000000000001e-05 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774647+0000", 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0020000000000001e-05 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774713+0000", 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5846000000000006e-05 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774725+0000", 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2101e-05 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.731889+0000", 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.957164601 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.731990+0000", 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100874 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.732097+0000", 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107283 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738480+0000", 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063826600000000001 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738866+0000", 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00038624999999999998 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.754413+0000", 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015546739 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.754449+0000", 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5766999999999998e-05 2024-02-05T08:18:46.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.754472+0000", 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3524999999999999e-05 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136593 8.0 8:0c9c3211:::benchmark_data_smithi134_115836_object19933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.850131+0000", 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 341.35081279399998, 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.90448195099999995, 2024-02-05T08:18:46.311 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.312 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.312 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.312 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.312 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.312 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136593 2024-02-05T08:18:46.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.312 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850131+0000", 2024-02-05T08:18:46.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850131+0000", 2024-02-05T08:18:46.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850132+0000", 2024-02-05T08:18:46.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.432e-06 2024-02-05T08:18:46.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850137+0000", 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.301e-06 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850148+0000", 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1627000000000001e-05 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850155+0000", 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2879999999999998e-06 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850215+0000", 2024-02-05T08:18:46.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0081000000000003e-05 2024-02-05T08:18:46.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850226+0000", 2024-02-05T08:18:46.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1097999999999999e-05 2024-02-05T08:18:46.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.735002+0000", 2024-02-05T08:18:46.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88477587000000002 2024-02-05T08:18:46.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.735508+0000", 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050598699999999995 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.735660+0000", 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000152409 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.742042+0000", 2024-02-05T08:18:46.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063815520000000004 2024-02-05T08:18:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.742099+0000", 2024-02-05T08:18:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7327000000000002e-05 2024-02-05T08:18:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.754523+0000", 2024-02-05T08:18:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012424117 2024-02-05T08:18:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.754575+0000", 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2299999999999997e-05 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.754613+0000", 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7561999999999997e-05 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170401 8.8 8:15576685:::benchmark_data_smithi134_115836_object53741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:18:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.792841+0000", 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 300.40810251099998, 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94320472300000002, 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170401 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792841+0000", 2024-02-05T08:18:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792841+0000", 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792842+0000", 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2109999999999999e-06 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792846+0000", 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5260000000000002e-06 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792857+0000", 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.135e-05 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792865+0000", 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8930000000000005e-06 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792926+0000", 2024-02-05T08:18:46.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0949000000000002e-05 2024-02-05T08:18:46.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792935+0000", 2024-02-05T08:18:46.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7050000000000005e-06 2024-02-05T08:18:46.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.733064+0000", 2024-02-05T08:18:46.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94012932299999996 2024-02-05T08:18:46.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.733194+0000", 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012970500000000001 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.733276+0000", 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2328000000000002e-05 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.735903+0000", 2024-02-05T08:18:46.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026266029999999999 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.735915+0000", 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1817999999999999e-05 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.735956+0000", 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1635999999999999e-05 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.736003+0000", 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7089000000000003e-05 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.736046+0000", 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2586999999999997e-05 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170446 8.8 8:1cf91133:::benchmark_data_smithi134_115836_object53786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.854446+0000", 2024-02-05T08:18:46.325 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 300.34649787199999, 2024-02-05T08:18:46.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88840418799999998, 2024-02-05T08:18:46.326 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.326 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.326 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.326 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.326 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.326 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170446 2024-02-05T08:18:46.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.326 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854446+0000", 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854446+0000", 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854447+0000", 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9099999999999991e-07 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854450+0000", 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6179999999999999e-06 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854462+0000", 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1548e-05 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854468+0000", 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7300000000000002e-06 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854532+0000", 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3870999999999999e-05 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854541+0000", 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2080000000000006e-06 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.736085+0000", 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88154407899999998 2024-02-05T08:18:46.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.736150+0000", 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5520999999999998e-05 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.736211+0000", 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0817999999999999e-05 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.740393+0000", 2024-02-05T08:18:46.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041813190000000002 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.742713+0000", 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023207359999999999 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.742773+0000", 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9789e-05 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.742813+0000", 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9972999999999998e-05 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.742850+0000", 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6986999999999997e-05 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170458 8.8 8:11a83214:::benchmark_data_smithi134_115836_object53798:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.863750+0000", 2024-02-05T08:18:46.332 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 300.33719389800001, 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88835640999999999, 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170458 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863750+0000", 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863750+0000", 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863751+0000", 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2999999999999999e-07 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863754+0000", 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5310000000000002e-06 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863764+0000", 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8770000000000005e-06 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863771+0000", 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9979999999999997e-06 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863806+0000", 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4671000000000001e-05 2024-02-05T08:18:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863812+0000", 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2790000000000004e-06 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.740517+0000", 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.876704539 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.740625+0000", 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108942 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.740705+0000", 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9560000000000004e-05 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.745209+0000", 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045041869999999998 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.745218+0000", 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.405e-06 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.751999+0000", 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067812380000000002 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.752057+0000", 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7773999999999999e-05 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.752106+0000", 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9579000000000001e-05 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170464 8.8 8:17746572:::benchmark_data_smithi134_115836_object53804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.870050+0000", 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 300.33089375999998, 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88894854099999998, 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170464 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.339 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870050+0000", 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870050+0000", 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870051+0000", 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7400000000000002e-07 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870055+0000", 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.561e-06 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870068+0000", 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2458e-05 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870075+0000", 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1890000000000001e-06 2024-02-05T08:18:46.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870130+0000", 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5297999999999998e-05 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870140+0000", 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2990000000000002e-06 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.745256+0000", 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.875116271 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.745380+0000", 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012443 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.745491+0000", 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110257 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.752163+0000", 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066722159999999999 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:18:46.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.752182+0000", 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9623000000000002e-05 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.758912+0000", 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067298669999999996 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.758962+0000", 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0207999999999998e-05 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.758998+0000", 2024-02-05T08:18:46.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5989999999999999e-05 2024-02-05T08:18:46.345 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.345 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.345 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.345 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:18:46.345 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:18:46.345 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:18:46.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:46.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:46.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:46.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:46.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:46.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:46.594 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:46.618 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:46.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:46.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:46.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:18:46.719 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:46.742 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:46.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:46.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:46.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:46.867 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:46.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:46.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:46.960 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:46.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:46.995 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:46.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:46.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:47.094 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:46 smithi134 ceph-mon[30086]: 5.c scrub starts 2024-02-05T08:18:47.094 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:46 smithi134 ceph-mon[30086]: 5.c scrub ok 2024-02-05T08:18:47.094 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:46 smithi134 ceph-mon[30086]: 5.f scrub starts 2024-02-05T08:18:47.094 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:46 smithi134 ceph-mon[30086]: 5.f scrub ok 2024-02-05T08:18:47.094 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:46 smithi134 ceph-mon[30086]: pgmap v621: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 617 op/s; 4.5 MiB/s, 83 objects/s recovering 2024-02-05T08:18:47.095 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:46 smithi134 ceph-mon[39278]: 5.c scrub starts 2024-02-05T08:18:47.095 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:46 smithi134 ceph-mon[39278]: 5.c scrub ok 2024-02-05T08:18:47.095 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:46 smithi134 ceph-mon[39278]: 5.f scrub starts 2024-02-05T08:18:47.095 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:46 smithi134 ceph-mon[39278]: 5.f scrub ok 2024-02-05T08:18:47.095 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:46 smithi134 ceph-mon[39278]: pgmap v621: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 617 op/s; 4.5 MiB/s, 83 objects/s recovering 2024-02-05T08:18:47.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:47.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:47.121 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:47.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:47.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:46 smithi196 ceph-mon[33018]: 5.c scrub starts 2024-02-05T08:18:47.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:46 smithi196 ceph-mon[33018]: 5.c scrub ok 2024-02-05T08:18:47.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:46 smithi196 ceph-mon[33018]: 5.f scrub starts 2024-02-05T08:18:47.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:46 smithi196 ceph-mon[33018]: 5.f scrub ok 2024-02-05T08:18:47.164 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:46 smithi196 ceph-mon[33018]: pgmap v621: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 617 op/s; 4.5 MiB/s, 83 objects/s recovering 2024-02-05T08:18:47.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T08:18:47.222 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:47.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:47.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:47.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:47.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:47.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:47.387 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:47.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:47.438 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:18:47.488 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:47.512 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:47.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:47.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:47.614 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:47.638 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:47.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:47.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:47.885 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:47.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:47.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:48.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:48.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:48.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:18:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:18:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:18:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:47 smithi134 ceph-mon[30086]: 6.1d scrub starts 2024-02-05T08:18:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:47 smithi134 ceph-mon[30086]: 4.15 scrub starts 2024-02-05T08:18:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:47 smithi134 ceph-mon[30086]: 4.15 scrub ok 2024-02-05T08:18:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:47 smithi134 ceph-mon[30086]: 6.1d scrub ok 2024-02-05T08:18:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:47 smithi134 ceph-mon[30086]: 3.1e scrub starts 2024-02-05T08:18:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:47 smithi134 ceph-mon[30086]: 3.1e scrub ok 2024-02-05T08:18:48.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:47 smithi134 ceph-mon[39278]: 6.1d scrub starts 2024-02-05T08:18:48.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:47 smithi134 ceph-mon[39278]: 4.15 scrub starts 2024-02-05T08:18:48.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:47 smithi134 ceph-mon[39278]: 4.15 scrub ok 2024-02-05T08:18:48.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:47 smithi134 ceph-mon[39278]: 6.1d scrub ok 2024-02-05T08:18:48.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:47 smithi134 ceph-mon[39278]: 3.1e scrub starts 2024-02-05T08:18:48.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:47 smithi134 ceph-mon[39278]: 3.1e scrub ok 2024-02-05T08:18:48.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:48.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:48.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:48.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:48.165 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:48.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:47 smithi196 ceph-mon[33018]: 6.1d scrub starts 2024-02-05T08:18:48.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:47 smithi196 ceph-mon[33018]: 4.15 scrub starts 2024-02-05T08:18:48.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:47 smithi196 ceph-mon[33018]: 4.15 scrub ok 2024-02-05T08:18:48.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:47 smithi196 ceph-mon[33018]: 6.1d scrub ok 2024-02-05T08:18:48.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:47 smithi196 ceph-mon[33018]: 3.1e scrub starts 2024-02-05T08:18:48.217 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:47 smithi196 ceph-mon[33018]: 3.1e scrub ok 2024-02-05T08:18:48.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:48.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:48.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:48.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:48.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:48.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:48.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:48.499 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:48.524 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:48.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:48.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:48.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:48.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:48.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:48.751 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:48.775 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:48.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:48.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:48.845 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:18:48.877 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:48.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:48.901 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:48.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:49.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:49.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:49.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:49.072 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:48 smithi196 ceph-mon[33018]: 9.6 scrub starts 2024-02-05T08:18:49.072 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:48 smithi196 ceph-mon[33018]: 4.1c scrub starts 2024-02-05T08:18:49.072 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:48 smithi196 ceph-mon[33018]: 4.1c scrub ok 2024-02-05T08:18:49.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:48 smithi196 ceph-mon[33018]: 9.6 scrub ok 2024-02-05T08:18:49.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:48 smithi196 ceph-mon[33018]: pgmap v622: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 617 B/s rd, 0 B/s wr, 496 op/s; 3.6 MiB/s, 66 objects/s recovering 2024-02-05T08:18:49.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:48 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2012939964' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:18:49.073 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:48 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:18:49.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:49.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:49.152 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:49.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:49.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:49.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[30086]: 9.6 scrub starts 2024-02-05T08:18:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[30086]: 4.1c scrub starts 2024-02-05T08:18:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[30086]: 4.1c scrub ok 2024-02-05T08:18:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[30086]: 9.6 scrub ok 2024-02-05T08:18:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[30086]: pgmap v622: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 617 B/s rd, 0 B/s wr, 496 op/s; 3.6 MiB/s, 66 objects/s recovering 2024-02-05T08:18:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2012939964' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:18:49.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:18:49.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[39278]: 9.6 scrub starts 2024-02-05T08:18:49.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[39278]: 4.1c scrub starts 2024-02-05T08:18:49.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[39278]: 4.1c scrub ok 2024-02-05T08:18:49.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[39278]: 9.6 scrub ok 2024-02-05T08:18:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[39278]: pgmap v622: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 617 B/s rd, 0 B/s wr, 496 op/s; 3.6 MiB/s, 66 objects/s recovering 2024-02-05T08:18:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2012939964' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:18:49.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:48 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:18:49.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:49.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:49.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:49.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:49.406 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:49.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:49.507 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:49.531 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:49.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:49.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:49.619 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:18:49.619 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:18:49.619 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T08:18:49.619 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:18:49.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:49.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:49.658 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:49.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:49.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:49.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:49.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:49.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:49.885 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:49.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:49.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:49.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:49.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:49 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:18:49.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:49 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:18:49.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:49 smithi196 ceph-mon[33018]: 2.2 scrub starts 2024-02-05T08:18:49.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:49 smithi196 ceph-mon[33018]: 2.2 scrub ok 2024-02-05T08:18:49.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:49 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:49.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:49 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:18:49.945 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:49 smithi196 ceph-mon[33018]: osdmap e299: 8 total, 8 up, 8 in 2024-02-05T08:18:50.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:50.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:50.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:50.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:50.136 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:50.143 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:18:50.143 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:18:50.144 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[30086]: 2.2 scrub starts 2024-02-05T08:18:50.144 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[30086]: 2.2 scrub ok 2024-02-05T08:18:50.144 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:50.144 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:18:50.144 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[30086]: osdmap e299: 8 total, 8 up, 8 in 2024-02-05T08:18:50.144 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:18:50.144 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:18:50.144 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[39278]: 2.2 scrub starts 2024-02-05T08:18:50.144 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[39278]: 2.2 scrub ok 2024-02-05T08:18:50.144 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:50.144 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:18:50.144 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:49 smithi134 ceph-mon[39278]: osdmap e299: 8 total, 8 up, 8 in 2024-02-05T08:18:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:50.162 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:50.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:50.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:50.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T08:18:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:50.291 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:50.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:50.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:50.415 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:50.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:50.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:50.529 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:18:50.541 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:50.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:50.642 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:50.668 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:50.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:50.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:50.796 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:50.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:50.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:50.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:50.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:50.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:50.940 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:50 smithi196 ceph-mon[33018]: 5.11 scrub starts 2024-02-05T08:18:50.940 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:50 smithi196 ceph-mon[33018]: pgmap v624: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 205 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:18:51.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:51.048 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:51.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:51.147 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:50 smithi134 ceph-mon[30086]: 5.11 scrub starts 2024-02-05T08:18:51.147 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:50 smithi134 ceph-mon[30086]: pgmap v624: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 205 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:18:51.148 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:50 smithi134 ceph-mon[39278]: 5.11 scrub starts 2024-02-05T08:18:51.148 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:50 smithi134 ceph-mon[39278]: pgmap v624: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 205 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:18:51.149 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:51.174 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:51.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:51.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:51.276 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:51.301 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:51.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:51.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:51.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:51.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:51.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:51.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:51.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:51.552 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:51.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:51.589 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:18:51.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:51.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:51.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:51.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:51.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:51.803 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:51.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:51.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:51.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:51.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:51.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:52.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:52.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:52.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:52.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:52.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:52.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:52.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:52.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:52.199 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:18:52.220 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:18:52.239 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:18:52.258 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:18:52.278 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:18:52.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:52.298 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:18:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:52.308 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:52.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:52.319 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:18:52.339 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:18:52.340 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:18:52.340 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:18:52.340 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:18:52.340 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:18:52.340 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:18:52.340 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:18:52.341 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:18:52.341 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:18:52.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:52.434 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:52.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:52.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:52.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:52.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:52.559 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:52.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:52.565 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:18:52.565 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:18:52.660 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:52.684 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:52.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:52.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:52.786 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:52.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:52.811 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:52.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:52.912 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:52.936 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:52.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:52.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:53.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:53.061 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:53.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:53.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:52 smithi134 ceph-mon[30086]: pgmap v625: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 205 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:18:53.151 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:52 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3903424670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:53.151 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:52 smithi134 ceph-mon[39278]: pgmap v625: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 205 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:18:53.151 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:52 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3903424670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:53.163 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:53.186 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:53.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:53.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:52 smithi196 ceph-mon[33018]: pgmap v625: 148 pgs: 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 205 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:18:53.218 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:52 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3903424670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:18:53.243 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T08:18:53.288 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:53.314 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:53.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:53.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:53.415 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:53.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:53.441 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:53.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:53.520 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:18:53.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:53.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:53.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:53.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:53.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:53.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:53.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:53.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:53.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:53.819 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:53.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:53.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:53.849 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:53.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:53.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:18:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:18:53.945 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:53.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:53.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:54.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:54.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:54.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:54.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:54.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:54.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:54.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:54.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:54.269 INFO:teuthology.orchestra.run.smithi134.stderr:set pool 1 pg_num to 11 2024-02-05T08:18:54.298 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:54.322 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:54.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:54.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:54.446 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:54.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:54.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:54.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:18:54.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:18:54.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:54 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/143404732' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T08:18:54.509 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:54 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T08:18:54.509 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:18:54.509 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:18:54.509 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:54 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/143404732' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T08:18:54.509 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:54 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T08:18:54.549 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:54.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:18:54.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:18:54.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:54 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/143404732' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T08:18:54.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:54 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T08:18:54.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:54.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:54.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:54.699 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:54.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:54.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:54.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:54.825 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:54.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:54.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:54.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:54.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:54.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:55.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:55.076 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:55.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:55.150 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-02-05T08:18:55.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-02-05T08:18:55.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:55.201 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:55.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:55.256 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:18:55.256 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:18:55.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:55.328 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:55.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:55.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:55.455 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:55.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:55.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:55 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:55.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:55 smithi134 ceph-mon[30086]: pgmap v626: 148 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfill_toofull, 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 41 op/s 2024-02-05T08:18:55.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:55 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-05T08:18:55.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:55 smithi134 ceph-mon[30086]: osdmap e300: 8 total, 8 up, 8 in 2024-02-05T08:18:55.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:55 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:55.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:55 smithi134 ceph-mon[39278]: pgmap v626: 148 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfill_toofull, 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 41 op/s 2024-02-05T08:18:55.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:55 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-05T08:18:55.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:55 smithi134 ceph-mon[39278]: osdmap e300: 8 total, 8 up, 8 in 2024-02-05T08:18:55.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:55.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:55 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:18:55.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:55 smithi196 ceph-mon[33018]: pgmap v626: 148 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfill_toofull, 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 41 op/s 2024-02-05T08:18:55.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:55 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-05T08:18:55.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:55 smithi196 ceph-mon[33018]: osdmap e300: 8 total, 8 up, 8 in 2024-02-05T08:18:55.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:55.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:55.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:55.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:55.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:55.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:55.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:55.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:55.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:55.831 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:55.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:55.832 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:18:55.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T08:18:55.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:55.974 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:55.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:56.076 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:56.100 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:56.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:56.148 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:18:56.202 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:56.227 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:56.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:56.259 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:56.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:56.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:56.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:56.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:56.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:56.478 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:56.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:56.515 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:56 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T08:18:56.515 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:56 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T08:18:56.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:56 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T08:18:56.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:56.606 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:56.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:56.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:56.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:56.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:56.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:56.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:56.834 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:56.858 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:56.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:56.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:56.957 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:18:56.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:56.984 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:56.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:56.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:57.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:57.109 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:57.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:57.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:57.210 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:57.234 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:57.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:57.282 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:18:57.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:57 smithi196 ceph-mon[33018]: pgmap v628: 148 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfill_toofull, 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:18:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:57 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-05T08:18:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:57 smithi196 ceph-mon[33018]: osdmap e301: 8 total, 8 up, 8 in 2024-02-05T08:18:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:57 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2910511542' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:57.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:57 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:57.359 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:57.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:57.422 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:57 smithi134 ceph-mon[30086]: pgmap v628: 148 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfill_toofull, 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:18:57.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:57 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-05T08:18:57.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:57 smithi134 ceph-mon[30086]: osdmap e301: 8 total, 8 up, 8 in 2024-02-05T08:18:57.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:57 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2910511542' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:57.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:57 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:57.423 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:57 smithi134 ceph-mon[39278]: pgmap v628: 148 pgs: 1 active+clean+scrubbing, 5 active+remapped+backfill_toofull, 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:18:57.423 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:57 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-05T08:18:57.423 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:57 smithi134 ceph-mon[39278]: osdmap e301: 8 total, 8 up, 8 in 2024-02-05T08:18:57.423 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:57 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2910511542' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:57.423 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:57 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:18:57.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:18:57.484 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:57.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:57.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:18:57.586 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:57.609 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:57.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:57.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:57.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:57.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:57.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:57.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:57.862 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:57.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:57.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:57.884 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:18:57.884 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:18:57.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:57.987 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:57.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:58.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:18:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:18:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:18:58.089 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:58.114 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:58.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:58.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:58.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:58.240 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:58.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:58.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:58.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:58.366 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:58.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:58.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:58.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:58 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:58.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:58 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:18:58.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:58 smithi196 ceph-mon[33018]: osdmap e302: 8 total, 8 up, 8 in 2024-02-05T08:18:58.488 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:58 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T08:18:58.490 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:58.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:58.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:58.513 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T08:18:58.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:58 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:58.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:58 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:18:58.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:58 smithi134 ceph-mon[30086]: osdmap e302: 8 total, 8 up, 8 in 2024-02-05T08:18:58.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:58 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T08:18:58.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:58 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:18:58.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:58 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:18:58.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:58 smithi134 ceph-mon[39278]: osdmap e302: 8 total, 8 up, 8 in 2024-02-05T08:18:58.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:58 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T08:18:58.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:18:58.619 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:58.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:18:58.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:58.746 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:58.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:58.795 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:18:58.848 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:58.872 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:58.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:58.974 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:18:59.002 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:59.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:18:59.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:59.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:59.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:59.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:59.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:18:59.258 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:59.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:18:59.360 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:59.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:59.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:59.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:59.468 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:59 smithi134 ceph-mon[30086]: pgmap v631: 158 pgs: 10 unknown, 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:18:59.468 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:59 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-05T08:18:59.468 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:18:59 smithi134 ceph-mon[30086]: osdmap e303: 8 total, 8 up, 8 in 2024-02-05T08:18:59.469 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:59 smithi134 ceph-mon[39278]: pgmap v631: 158 pgs: 10 unknown, 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:18:59.469 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:59 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-05T08:18:59.469 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:18:59 smithi134 ceph-mon[39278]: osdmap e303: 8 total, 8 up, 8 in 2024-02-05T08:18:59.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:18:59.509 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:59.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:18:59.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:59 smithi196 ceph-mon[33018]: pgmap v631: 158 pgs: 10 unknown, 5 active+remapped+backfill_toofull, 143 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:18:59.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:59 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-05T08:18:59.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:18:59 smithi196 ceph-mon[33018]: osdmap e303: 8 total, 8 up, 8 in 2024-02-05T08:18:59.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:18:59.633 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:59.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:18:59.735 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:18:59.759 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:18:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:59.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:18:59.806 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'unique_pool_2' removed 2024-02-05T08:18:59.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:18:59.894 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:18:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:18:59.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:18:59.996 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:00.025 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:00.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:00.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:00.151 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:19:00.151 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:19:00.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:00.152 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:00.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:00.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:00.276 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:00.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:00.378 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:00.401 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:00.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:00.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:00.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:00.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:00 smithi134 ceph-mon[30086]: osdmap e304: 8 total, 8 up, 8 in 2024-02-05T08:19:00.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:00 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1978046799' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:19:00.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:00 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1978046799' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:19:00.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:00 smithi134 ceph-mon[30086]: osdmap e305: 8 total, 8 up, 8 in 2024-02-05T08:19:00.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:00 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:19:00.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:00 smithi134 ceph-mon[39278]: osdmap e304: 8 total, 8 up, 8 in 2024-02-05T08:19:00.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:00 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1978046799' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:19:00.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:00 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1978046799' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:19:00.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:00 smithi134 ceph-mon[39278]: osdmap e305: 8 total, 8 up, 8 in 2024-02-05T08:19:00.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:00 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:19:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:00.528 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:00.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:00.568 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:00 smithi196 ceph-mon[33018]: osdmap e304: 8 total, 8 up, 8 in 2024-02-05T08:19:00.568 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:00 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1978046799' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:19:00.568 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:00 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1978046799' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:19:00.568 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:00 smithi196 ceph-mon[33018]: osdmap e305: 8 total, 8 up, 8 in 2024-02-05T08:19:00.568 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:00 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:19:00.591 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:19:00.591 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:19:00.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:00.653 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:00.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:00.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:00.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:00.777 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:00.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:00.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:00.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:00.902 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:00.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:00.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:00.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:19:01.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:01.027 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:01.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:01.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:01.065 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T08:19:01.066 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T08:19:01.066 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-02-05T08:19:01.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-02-05T08:19:01.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:01.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:01.154 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:01.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:01.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T08:19:01.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:01.278 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:01.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:01.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:01.405 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:01.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:01.441 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:19:01.507 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:01.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:01 smithi134 ceph-mon[30086]: pgmap v634: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:19:01.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:01 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:19:01.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:01 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:19:01.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:01 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:19:01.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:01 smithi134 ceph-mon[30086]: osdmap e306: 8 total, 8 up, 8 in 2024-02-05T08:19:01.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:01 smithi134 ceph-mon[39278]: pgmap v634: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:19:01.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:01 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:19:01.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:01 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:19:01.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:01 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:19:01.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:01 smithi134 ceph-mon[39278]: osdmap e306: 8 total, 8 up, 8 in 2024-02-05T08:19:01.531 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:01.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:01.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:01.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:01 smithi196 ceph-mon[33018]: pgmap v634: 158 pgs: 5 active+remapped+backfill_toofull, 153 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:19:01.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:01 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:19:01.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:01 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:19:01.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:01 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:19:01.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:01 smithi196 ceph-mon[33018]: osdmap e306: 8 total, 8 up, 8 in 2024-02-05T08:19:01.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:01.657 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:01.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:01.745 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:19:01.759 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:01.768 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:19:01.775 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:01.779 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:01.785 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:01.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:01.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:01.794 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:19:01.824 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:19:01.844 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:19:01.864 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:19:01.883 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:19:01.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:01.903 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:19:01.905 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:19:01.905 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:19:01.905 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:19:01.905 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:19:01.905 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:19:01.905 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:19:01.905 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:19:01.905 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:19:01.911 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:01.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:01.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:02.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:02.037 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:02.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:02.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:02.162 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:02.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:02.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:02.264 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:02.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:02.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:02.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:02.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:02.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:02.514 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:02.538 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:02.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:02.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:02.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:02 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1528476119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:02.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:02 smithi196 ceph-mon[33018]: osdmap e307: 8 total, 8 up, 8 in 2024-02-05T08:19:02.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:02.664 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:02.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:02.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:02.751 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:02 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1528476119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:02.751 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:02 smithi134 ceph-mon[30086]: osdmap e307: 8 total, 8 up, 8 in 2024-02-05T08:19:02.752 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:02 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1528476119' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:02.752 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:02 smithi134 ceph-mon[39278]: osdmap e307: 8 total, 8 up, 8 in 2024-02-05T08:19:02.766 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:02.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:02.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:02.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:02.916 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:02.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:03.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:03.041 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:03.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:03.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:03.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:03.162 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:19:03.162 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:19:03.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:19:03.162 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:19:03.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.162 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:27898 9.e 9:752e73bf:::benchmark_data_smithi134_144349_object27897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.030994+0000", 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 218.13013437699999, 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17265594500000001, 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 27898 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.030994+0000", 2024-02-05T08:19:03.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.030994+0000", 2024-02-05T08:19:03.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.030996+0000", 2024-02-05T08:19:03.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.643e-06 2024-02-05T08:19:03.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031038+0000", 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2579e-05 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031051+0000", 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2975e-05 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031057+0000", 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8409999999999996e-06 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031115+0000", 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7326e-05 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031214+0000", 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9677000000000001e-05 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031358+0000", 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000143454 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032987+0000", 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001629098 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033183+0000", 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019666400000000001 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.203561+0000", 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.170377946 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.203615+0000", 2024-02-05T08:19:03.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3207000000000001e-05 2024-02-05T08:19:03.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.168 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.168 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.203650+0000", 2024-02-05T08:19:03.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5534999999999998e-05 2024-02-05T08:19:03.168 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.168 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.168 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.168 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:27895 9.5 e249/239 9:a349bc50:::benchmark_data_smithi134_144349_object27894:head v 249'1711, mlcod=249'1707)", 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.032312+0000", 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 218.128816215, 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16973406999999999, 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032312+0000", 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032312+0000", 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032313+0000", 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5400000000000002e-07 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032354+0000", 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1622999999999998e-05 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032365+0000", 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0076e-05 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032371+0000", 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2360000000000001e-06 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032515+0000", 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000144141 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032533+0000", 2024-02-05T08:19:03.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8264e-05 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033555+0000", 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001022226 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202046+0000", 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16849105 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:27899 9.9 e249/239 9:9d77ddc3:::benchmark_data_smithi134_144349_object27898:head v 249'1791, mlcod=249'1789)", 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.032390+0000", 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 218.12873870799999, 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.169799125, 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032390+0000", 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.173 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032390+0000", 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032390+0000", 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0600000000000001e-07 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032423+0000", 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2428999999999997e-05 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.174 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032430+0000", 2024-02-05T08:19:03.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6629999999999993e-06 2024-02-05T08:19:03.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032436+0000", 2024-02-05T08:19:03.175 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.02e-06 2024-02-05T08:19:03.175 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.175 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.175 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.175 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032599+0000", 2024-02-05T08:19:03.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016209300000000001 2024-02-05T08:19:03.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032618+0000", 2024-02-05T08:19:03.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9505e-05 2024-02-05T08:19:03.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.176 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.176 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202151+0000", 2024-02-05T08:19:03.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.169532718 2024-02-05T08:19:03.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202189+0000", 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8291000000000003e-05 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:27902 9.d 9:b50a7d0d:::benchmark_data_smithi134_144349_object27901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.032502+0000", 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 218.12862638799999, 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17749103499999999, 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.177 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 27902 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032502+0000", 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032502+0000", 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032503+0000", 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0999999999999997e-07 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032531+0000", 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7786000000000001e-05 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032541+0000", 2024-02-05T08:19:03.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0200999999999999e-05 2024-02-05T08:19:03.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032548+0000", 2024-02-05T08:19:03.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9439999999999999e-06 2024-02-05T08:19:03.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032710+0000", 2024-02-05T08:19:03.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016212600000000001 2024-02-05T08:19:03.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032854+0000", 2024-02-05T08:19:03.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014396499999999999 2024-02-05T08:19:03.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-02-05T08:19:03.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032987+0000", 2024-02-05T08:19:03.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000132447 2024-02-05T08:19:03.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202061+0000", 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16907446300000001 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202865+0000", 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080343400000000001 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.209910+0000", 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0070456900000000003 2024-02-05T08:19:03.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.209960+0000", 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0050999999999999e-05 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.209993+0000", 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3018000000000001e-05 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:27907 9.e 9:74a14f65:::benchmark_data_smithi134_144349_object27906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.033274+0000", 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 218.12785480400001, 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17237598000000001, 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 27907 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033274+0000", 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033274+0000", 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033275+0000", 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8199999999999998e-07 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.187 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033344+0000", 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9425999999999996e-05 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033355+0000", 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0975000000000001e-05 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033362+0000", 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0539999999999997e-06 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033400+0000", 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7889000000000003e-05 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.201908+0000", 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.168507921 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202081+0000", 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017288399999999999 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.204290+0000", 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022091649999999999 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.204499+0000", 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000208996 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205559+0000", 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010599380000000001 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205615+0000", 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5642e-05 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205650+0000", 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5207999999999999e-05 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:27903 9.8 e249/239 9:1ece6453:::benchmark_data_smithi134_144349_object27902:head v 249'1662, mlcod=249'1659)", 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.033685+0000", 2024-02-05T08:19:03.191 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 218.127443828, 2024-02-05T08:19:03.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17151197500000001, 2024-02-05T08:19:03.192 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.192 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.192 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033685+0000", 2024-02-05T08:19:03.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033685+0000", 2024-02-05T08:19:03.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033685+0000", 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4999999999999998e-07 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033745+0000", 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9487000000000002e-05 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.201990+0000", 2024-02-05T08:19:03.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16824555199999999 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202001+0000", 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0828e-05 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202060+0000", 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8529999999999997e-05 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202082+0000", 2024-02-05T08:19:03.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2340999999999998e-05 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205157+0000", 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030755219999999998 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205197+0000", 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9264999999999998e-05 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:35467 9.2 e249/239 9:4b974889:::benchmark_data_smithi134_144349_object35466:head v 249'2223, mlcod=249'2220)", 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:29.630509+0000", 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 213.53061961200001, 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.154489985, 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.630509+0000", 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.630509+0000", 2024-02-05T08:19:03.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.630509+0000", 2024-02-05T08:19:03.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9699999999999996e-07 2024-02-05T08:19:03.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.630570+0000", 2024-02-05T08:19:03.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0089e-05 2024-02-05T08:19:03.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.765054+0000", 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.134484564 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.765061+0000", 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8900000000000001e-06 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.765153+0000", 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2193000000000003e-05 2024-02-05T08:19:03.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.765228+0000", 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4952999999999999e-05 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.784968+0000", 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019740159 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.784999+0000", 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0639999999999998e-05 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.199 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35875 9.e 9:7b26ffef:::benchmark_data_smithi134_144349_object35874:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.104276+0000", 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 213.056853069, 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19477491999999999, 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35875 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.200 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104276+0000", 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104276+0000", 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104277+0000", 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0690000000000001e-06 2024-02-05T08:19:03.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104323+0000", 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5961999999999998e-05 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104342+0000", 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9038e-05 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104350+0000", 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3189999999999995e-06 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104571+0000", 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022128600000000001 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104577+0000", 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4469999999999997e-06 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.112613+0000", 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0080362550000000008 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.112736+0000", 2024-02-05T08:19:03.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000123087 2024-02-05T08:19:03.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:19:03.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:03.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:03.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:03.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.112906+0000", 2024-02-05T08:19:03.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000169772 2024-02-05T08:19:03.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.117780+0000", 2024-02-05T08:19:03.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048745769999999997 2024-02-05T08:19:03.206 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.206 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.117817+0000", 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6451e-05 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.298971+0000", 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18115403999999999 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299019+0000", 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8164000000000002e-05 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.207 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299050+0000", 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1452999999999999e-05 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35882 9.e 9:79e542b7:::benchmark_data_smithi134_144349_object35881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.111606+0000", 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 213.04952294399999, 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18759089300000001, 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.208 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.209 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35882 2024-02-05T08:19:03.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.209 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111606+0000", 2024-02-05T08:19:03.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.209 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.209 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.209 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111606+0000", 2024-02-05T08:19:03.209 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.209 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111607+0000", 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4810000000000001e-06 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111642+0000", 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4699000000000001e-05 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111652+0000", 2024-02-05T08:19:03.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0574000000000001e-05 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111660+0000", 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2350000000000002e-06 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.114371+0000", 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027112450000000001 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.114476+0000", 2024-02-05T08:19:03.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105139 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.114563+0000", 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7253e-05 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.117828+0000", 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032642650000000001 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.118007+0000", 2024-02-05T08:19:03.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017978199999999999 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299094+0000", 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18108685299999999 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299148+0000", 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3956999999999999e-05 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299197+0000", 2024-02-05T08:19:03.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8409999999999999e-05 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35887 9.e 9:724dd1b7:::benchmark_data_smithi134_144349_object35886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.113626+0000", 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 213.04750305600001, 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18572786199999999, 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.214 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35887 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113626+0000", 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113626+0000", 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113627+0000", 2024-02-05T08:19:03.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2300000000000001e-06 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113653+0000", 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6208e-05 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113664+0000", 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0523000000000001e-05 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113671+0000", 2024-02-05T08:19:03.216 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4510000000000003e-06 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.116160+0000", 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024895170000000001 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.116289+0000", 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000128355 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.116439+0000", 2024-02-05T08:19:03.217 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015028799999999999 2024-02-05T08:19:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.119562+0000", 2024-02-05T08:19:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0031226969999999998 2024-02-05T08:19:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.119795+0000", 2024-02-05T08:19:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000233551 2024-02-05T08:19:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.218 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299259+0000", 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17946410800000001 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299310+0000", 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0439000000000003e-05 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299353+0000", 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3495000000000003e-05 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.219 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35906 9.e 9:7cbe34f1:::benchmark_data_smithi134_144349_object35905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:19:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.125518+0000", 2024-02-05T08:19:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 213.03561098599999, 2024-02-05T08:19:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17395559999999999, 2024-02-05T08:19:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35906 2024-02-05T08:19:03.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125518+0000", 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125518+0000", 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.221 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125519+0000", 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3960000000000001e-06 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125570+0000", 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1084000000000001e-05 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125586+0000", 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5903999999999999e-05 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125594+0000", 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9899999999999997e-06 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125651+0000", 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7179000000000003e-05 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125776+0000", 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012447200000000001 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125916+0000", 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000140013 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.127316+0000", 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014006859999999999 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.127928+0000", 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00061120399999999996 2024-02-05T08:19:03.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299393+0000", 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17146588300000001 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299440+0000", 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6684999999999998e-05 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299473+0000", 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3104e-05 2024-02-05T08:19:03.225 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35911 9.e 9:7aaf23ea:::benchmark_data_smithi134_144349_object35910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.129684+0000", 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 213.03144503300001, 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16995434200000001, 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35911 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.226 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129684+0000", 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129684+0000", 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129685+0000", 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8589999999999999e-06 2024-02-05T08:19:03.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129749+0000", 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3529999999999997e-05 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129764+0000", 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4793999999999999e-05 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129772+0000", 2024-02-05T08:19:03.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2109999999999998e-06 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129822+0000", 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9974e-05 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129921+0000", 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8887000000000004e-05 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.130017+0000", 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6364000000000001e-05 2024-02-05T08:19:03.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.131511+0000", 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014936540000000001 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.132132+0000", 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00062069600000000001 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299515+0000", 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16738375999999999 2024-02-05T08:19:03.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299604+0000", 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8363999999999996e-05 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299638+0000", 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4248999999999997e-05 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35921 9.e 9:776a85fb:::benchmark_data_smithi134_144349_object35920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.138253+0000", 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 213.02287511599999, 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.161497483, 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35921 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138253+0000", 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138253+0000", 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138255+0000", 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.576e-06 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138299+0000", 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4409999999999997e-05 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138310+0000", 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0794e-05 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138318+0000", 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7500000000000003e-06 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138376+0000", 2024-02-05T08:19:03.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8372999999999998e-05 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138498+0000", 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012176900000000001 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138626+0000", 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012799 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.139830+0000", 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001204199 2024-02-05T08:19:03.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140009+0000", 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017819299999999999 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299676+0000", 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.159667061 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299722+0000", 2024-02-05T08:19:03.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6164e-05 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299751+0000", 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9204e-05 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35922 9.e 9:77e64ba8:::benchmark_data_smithi134_144349_object35921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.140433+0000", 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 213.02069564199999, 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15941660299999999, 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.237 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35922 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140433+0000", 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140433+0000", 2024-02-05T08:19:03.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140434+0000", 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3099999999999996e-07 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140488+0000", 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4243000000000002e-05 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140498+0000", 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0015e-05 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140503+0000", 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1390000000000003e-06 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140550+0000", 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6978999999999999e-05 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140649+0000", 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9208000000000003e-05 2024-02-05T08:19:03.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140739+0000", 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9967000000000001e-05 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.141731+0000", 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00099119300000000002 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.141745+0000", 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4671000000000001e-05 2024-02-05T08:19:03.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299783+0000", 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15803773400000001 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299822+0000", 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9428999999999998e-05 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299850+0000", 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7093999999999998e-05 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.242 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58423 9.d 9:b730c3b2:::benchmark_data_smithi134_144349_object58422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.759126+0000", 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 198.402002568, 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14701034599999999, 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58423 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759126+0000", 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759126+0000", 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759128+0000", 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.601e-06 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759180+0000", 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2172999999999997e-05 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759194+0000", 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4131000000000001e-05 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759368+0000", 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017358299999999999 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759575+0000", 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0002073 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763082+0000", 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003507058 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.766845+0000", 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003762933 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776323+0000", 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094780709999999994 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776344+0000", 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0774000000000001e-05 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.906047+0000", 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.129703771 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.906098+0000", 2024-02-05T08:19:03.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0495999999999999e-05 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.906136+0000", 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8454999999999997e-05 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58429 9.e 9:74deef27:::benchmark_data_smithi134_144349_object58428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.763196+0000", 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 198.397932492, 2024-02-05T08:19:03.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14953992599999999, 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58429 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763196+0000", 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763196+0000", 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763198+0000", 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5960000000000001e-06 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763237+0000", 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9320999999999998e-05 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763247+0000", 2024-02-05T08:19:03.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6909999999999992e-06 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763254+0000", 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8580000000000002e-06 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.766437+0000", 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003183909 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.766559+0000", 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121561 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.766696+0000", 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013721 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.778872+0000", 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012176200999999999 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.784715+0000", 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058425609999999996 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912622+0000", 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.127907411 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912685+0000", 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2501000000000003e-05 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912736+0000", 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1106000000000002e-05 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58435 9.e 9:7bef458b:::benchmark_data_smithi134_144349_object58434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.765269+0000", 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 198.39585926500001, 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14758520999999999, 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58435 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765269+0000", 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765269+0000", 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765271+0000", 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.638e-06 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765301+0000", 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0369e-05 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765313+0000", 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1481000000000001e-05 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765321+0000", 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6720000000000004e-06 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776013+0000", 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010692657 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776149+0000", 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000135909 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776445+0000", 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00029550300000000002 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.778879+0000", 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024342790000000001 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.784792+0000", 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059131610000000001 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912788+0000", 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12799622299999999 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912832+0000", 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3402000000000002e-05 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912855+0000", 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2915999999999999e-05 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58444 9.3 9:c881298d:::benchmark_data_smithi134_144349_object58443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:19:03.259 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.777171+0000", 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 198.38395774599999, 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14967462100000001, 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58444 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777171+0000", 2024-02-05T08:19:03.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777171+0000", 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777172+0000", 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.155e-06 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777202+0000", 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0267e-05 2024-02-05T08:19:03.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777211+0000", 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9730000000000003e-06 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777218+0000", 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2940000000000002e-06 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.779461+0000", 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002243423 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.779586+0000", 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000125325 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.779714+0000", 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012767599999999999 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.785089+0000", 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053749059999999996 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.787028+0000", 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019386290000000001 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.925411+0000", 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13838302199999999 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.926806+0000", 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013959249999999999 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.926845+0000", 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9026000000000002e-05 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58447 9.3 9:ce75229c:::benchmark_data_smithi134_144349_object58446:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.780114+0000", 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 198.38101461799999, 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14685973299999999, 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.265 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58447 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.780114+0000", 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.780114+0000", 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.780115+0000", 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5009999999999999e-06 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.782413+0000", 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022973640000000001 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.782431+0000", 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7779999999999999e-05 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.782440+0000", 2024-02-05T08:19:03.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0759999999999992e-06 2024-02-05T08:19:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.782633+0000", 2024-02-05T08:19:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000193487 2024-02-05T08:19:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.782778+0000", 2024-02-05T08:19:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014436599999999999 2024-02-05T08:19:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.782922+0000", 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014404700000000001 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.785359+0000", 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002437874 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.787065+0000", 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017058349999999999 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.926892+0000", 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.139826541 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.926939+0000", 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7029e-05 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.926974+0000", 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4832999999999997e-05 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:179097 9.d 9:b97b4d8e:::benchmark_data_smithi134_144349_object47357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e280)", 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:17:28.532699+0000", 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 94.628430039999998, 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13586507, 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 179097 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.532699+0000", 2024-02-05T08:19:03.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.532699+0000", 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.532699+0000", 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.13e-07 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:19:03.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.532702+0000", 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4490000000000002e-06 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.532708+0000", 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1970000000000001e-06 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.532713+0000", 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4210000000000003e-06 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.631983+0000", 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.099269410000000002 2024-02-05T08:19:03.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.632068+0000", 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5277000000000004e-05 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.632138+0000", 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0277000000000005e-05 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.637729+0000", 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055905429999999999 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.637762+0000", 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3068000000000003e-05 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.668484+0000", 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.030722070000000001 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.668530+0000", 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5958999999999998e-05 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:19:03.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:19:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:28.668564+0000", 2024-02-05T08:19:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3886e-05 2024-02-05T08:19:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:19:03.276 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:19:03.276 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:19:03.305 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:03.329 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:03.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:03.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:03.331 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:03 smithi134 ceph-mon[30086]: pgmap v637: 142 pgs: 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:19:03.332 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:03 smithi134 ceph-mon[30086]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull) 2024-02-05T08:19:03.332 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:03 smithi134 ceph-mon[39278]: pgmap v637: 142 pgs: 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:19:03.332 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:03 smithi134 ceph-mon[39278]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull) 2024-02-05T08:19:03.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:03.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:03.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:03.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:03.563 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:03 smithi196 ceph-mon[33018]: pgmap v637: 142 pgs: 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:19:03.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:03 smithi196 ceph-mon[33018]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull) 2024-02-05T08:19:03.578 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:03.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:03.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:03.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:03.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:03.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:03.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:03.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:03.830 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:03.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:03.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:03.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:03.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:03.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:04.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:04.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:19:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:19:04.081 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:04.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:04.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:04.205 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:04.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:04.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T08:19:04.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:04.330 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:04.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:04.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:04.432 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:04.456 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:04.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:04.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:04.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:04.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:04.683 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:04.707 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:04.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:04.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:04.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:04.833 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:04.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:04.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:04.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:04.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:04.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:05.059 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:05.083 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:05.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:05.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:05.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:19:05.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:05.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:05.209 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:05.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:05.310 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:05.339 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:05.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:05.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:05.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:05.461 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:05 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:05.461 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:05 smithi134 ceph-mon[30086]: pgmap v639: 142 pgs: 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:05.461 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:05 smithi134 ceph-mon[30086]: osdmap e308: 8 total, 8 up, 8 in 2024-02-05T08:19:05.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:05 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:05.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:05 smithi134 ceph-mon[39278]: pgmap v639: 142 pgs: 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:05.461 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:05 smithi134 ceph-mon[39278]: osdmap e308: 8 total, 8 up, 8 in 2024-02-05T08:19:05.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:05.466 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:05.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:05 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:05 smithi196 ceph-mon[33018]: pgmap v639: 142 pgs: 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:05 smithi196 ceph-mon[33018]: osdmap e308: 8 total, 8 up, 8 in 2024-02-05T08:19:05.566 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:05.590 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:05.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:05.691 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:05.714 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:05.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:05.816 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:05.841 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:05.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:05.847 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:05.851 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:05.943 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:05.966 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:05.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:05.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:06.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:06.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:06.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:06.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:06.198 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:06.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:06.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:06.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:06.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:06.325 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'unique_pool_3' created 2024-02-05T08:19:06.328 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:19:06.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:06.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:06.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:06.359 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:06 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4112990278' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:19:06.359 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:06 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:19:06.359 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:06 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/899655277' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-05T08:19:06.360 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:06 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4112990278' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:19:06.360 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:06 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:19:06.360 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:06 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/899655277' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-05T08:19:06.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:06.482 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:06.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:06.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:06 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4112990278' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:19:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:06 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:19:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:06 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/899655277' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-05T08:19:06.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:06.611 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:06.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:06.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:06.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:06.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:06.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:06.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:06.867 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:06.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:06.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:06.969 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:06.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:06.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:07.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:07.118 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:07.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:07.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:07.220 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:07.248 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:07.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:07.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:07.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:07.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:07.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:07.501 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:07.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:07 smithi196 ceph-mon[33018]: pgmap v641: 142 pgs: 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:07 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:19:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:07 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:19:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:07 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/899655277' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-05T08:19:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:07 smithi196 ceph-mon[33018]: osdmap e309: 8 total, 8 up, 8 in 2024-02-05T08:19:07.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:07.628 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:07.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:07.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:07.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:07.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:07 smithi134 ceph-mon[30086]: pgmap v641: 142 pgs: 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:07.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:07 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:19:07.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:07 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:19:07.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:07 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/899655277' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-05T08:19:07.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:07 smithi134 ceph-mon[30086]: osdmap e309: 8 total, 8 up, 8 in 2024-02-05T08:19:07.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:07 smithi134 ceph-mon[39278]: pgmap v641: 142 pgs: 142 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:07.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:07 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:19:07.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:07 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:19:07.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:07 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/899655277' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-05T08:19:07.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:07 smithi134 ceph-mon[39278]: osdmap e309: 8 total, 8 up, 8 in 2024-02-05T08:19:07.754 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:07.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:07.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:07.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:07.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:07.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:08.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:08.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:08.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:08.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:19:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:19:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:19:08.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:08.137 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:08.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:08.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:08.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:08.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:08.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:08.391 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:08.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:08.493 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:08.517 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:08.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:08.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[30086]: 3.8 scrub starts 2024-02-05T08:19:08.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[30086]: 3.8 scrub ok 2024-02-05T08:19:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[30086]: osdmap e310: 8 total, 8 up, 8 in 2024-02-05T08:19:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[30086]: 1.7 scrub starts 2024-02-05T08:19:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[30086]: 1.7 scrub ok 2024-02-05T08:19:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[30086]: 1.3 scrub starts 2024-02-05T08:19:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[30086]: 1.3 scrub ok 2024-02-05T08:19:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:19:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:19:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[39278]: 3.8 scrub starts 2024-02-05T08:19:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[39278]: 3.8 scrub ok 2024-02-05T08:19:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[39278]: osdmap e310: 8 total, 8 up, 8 in 2024-02-05T08:19:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[39278]: 1.7 scrub starts 2024-02-05T08:19:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[39278]: 1.7 scrub ok 2024-02-05T08:19:08.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[39278]: 1.3 scrub starts 2024-02-05T08:19:08.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[39278]: 1.3 scrub ok 2024-02-05T08:19:08.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:19:08.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:19:08.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-02-05T08:19:08.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:08.643 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:08.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:08.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:08.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:08.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:08.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:08 smithi196 ceph-mon[33018]: 3.8 scrub starts 2024-02-05T08:19:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:08 smithi196 ceph-mon[33018]: 3.8 scrub ok 2024-02-05T08:19:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:08 smithi196 ceph-mon[33018]: osdmap e310: 8 total, 8 up, 8 in 2024-02-05T08:19:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:08 smithi196 ceph-mon[33018]: 1.7 scrub starts 2024-02-05T08:19:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:08 smithi196 ceph-mon[33018]: 1.7 scrub ok 2024-02-05T08:19:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:08 smithi196 ceph-mon[33018]: 1.3 scrub starts 2024-02-05T08:19:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:08 smithi196 ceph-mon[33018]: 1.3 scrub ok 2024-02-05T08:19:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:19:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:19:08.871 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:08.894 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:08.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:08.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:09.019 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:09.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:09.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:09.144 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:09.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:09.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:09.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:09.270 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:09.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:09.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:09.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:09.395 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:09.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:09.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:09.497 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:09.522 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:09.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:09.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[39278]: 5.b scrub starts 2024-02-05T08:19:09.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[39278]: 5.b scrub ok 2024-02-05T08:19:09.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[39278]: pgmap v644: 158 pgs: 10 unknown, 148 active+clean; 530 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[39278]: 1.4 scrub starts 2024-02-05T08:19:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[39278]: 1.4 scrub ok 2024-02-05T08:19:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[39278]: 1.2 scrub starts 2024-02-05T08:19:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[39278]: 1.2 scrub ok 2024-02-05T08:19:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[39278]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:19:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[39278]: osdmap e311: 8 total, 8 up, 8 in 2024-02-05T08:19:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[39278]: 1.6 scrub starts 2024-02-05T08:19:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[39278]: 1.6 scrub ok 2024-02-05T08:19:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[39278]: 1.5 scrub starts 2024-02-05T08:19:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[39278]: 1.5 scrub ok 2024-02-05T08:19:09.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[30086]: 5.b scrub starts 2024-02-05T08:19:09.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[30086]: 5.b scrub ok 2024-02-05T08:19:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[30086]: pgmap v644: 158 pgs: 10 unknown, 148 active+clean; 530 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[30086]: 1.4 scrub starts 2024-02-05T08:19:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[30086]: 1.4 scrub ok 2024-02-05T08:19:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[30086]: 1.2 scrub starts 2024-02-05T08:19:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[30086]: 1.2 scrub ok 2024-02-05T08:19:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[30086]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:19:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[30086]: osdmap e311: 8 total, 8 up, 8 in 2024-02-05T08:19:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[30086]: 1.6 scrub starts 2024-02-05T08:19:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[30086]: 1.6 scrub ok 2024-02-05T08:19:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[30086]: 1.5 scrub starts 2024-02-05T08:19:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:09 smithi134 ceph-mon[30086]: 1.5 scrub ok 2024-02-05T08:19:09.624 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:09.648 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:09.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:09.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:09.724 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:19:09.724 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:19:09.724 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:19:09.724 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:19:09.724 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:19:09.728 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:19:09.747 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:19:09.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:09.767 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:19:09.768 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:19:09.768 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:19:09.768 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:19:09.768 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:19:09.768 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:19:09.769 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:19:09.769 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:19:09.769 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:19:09.773 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:09.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:09 smithi196 ceph-mon[33018]: 5.b scrub starts 2024-02-05T08:19:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:09 smithi196 ceph-mon[33018]: 5.b scrub ok 2024-02-05T08:19:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:09 smithi196 ceph-mon[33018]: pgmap v644: 158 pgs: 10 unknown, 148 active+clean; 530 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:09 smithi196 ceph-mon[33018]: 1.4 scrub starts 2024-02-05T08:19:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:09 smithi196 ceph-mon[33018]: 1.4 scrub ok 2024-02-05T08:19:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:09 smithi196 ceph-mon[33018]: 1.2 scrub starts 2024-02-05T08:19:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:09 smithi196 ceph-mon[33018]: 1.2 scrub ok 2024-02-05T08:19:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:09 smithi196 ceph-mon[33018]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:19:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:09 smithi196 ceph-mon[33018]: osdmap e311: 8 total, 8 up, 8 in 2024-02-05T08:19:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:09 smithi196 ceph-mon[33018]: 1.6 scrub starts 2024-02-05T08:19:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:09 smithi196 ceph-mon[33018]: 1.6 scrub ok 2024-02-05T08:19:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:09 smithi196 ceph-mon[33018]: 1.5 scrub starts 2024-02-05T08:19:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:09 smithi196 ceph-mon[33018]: 1.5 scrub ok 2024-02-05T08:19:09.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:09.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:09.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:09.968 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:19:09.968 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:19:10.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:10.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:10.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:10.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:10.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:10.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:10.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:10.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:10.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:10.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:10.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:10.378 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:10.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:10.403 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:10.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:10.503 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:10.526 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:10.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:10.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:10.628 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:10.652 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:10.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:10.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:10.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:10.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[30086]: 1.1 scrub starts 2024-02-05T08:19:10.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[30086]: 1.1 scrub ok 2024-02-05T08:19:10.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[30086]: 1.9 scrub starts 2024-02-05T08:19:10.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[30086]: 1.9 scrub ok 2024-02-05T08:19:10.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[30086]: 5.11 scrub starts 2024-02-05T08:19:10.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[30086]: 5.11 scrub ok 2024-02-05T08:19:10.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[30086]: 4.2 scrub starts 2024-02-05T08:19:10.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[30086]: 4.2 scrub ok 2024-02-05T08:19:10.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3538606185' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:10.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[30086]: osdmap e312: 8 total, 8 up, 8 in 2024-02-05T08:19:10.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[39278]: 1.1 scrub starts 2024-02-05T08:19:10.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[39278]: 1.1 scrub ok 2024-02-05T08:19:10.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[39278]: 1.9 scrub starts 2024-02-05T08:19:10.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[39278]: 1.9 scrub ok 2024-02-05T08:19:10.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[39278]: 5.11 scrub starts 2024-02-05T08:19:10.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[39278]: 5.11 scrub ok 2024-02-05T08:19:10.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[39278]: 4.2 scrub starts 2024-02-05T08:19:10.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[39278]: 4.2 scrub ok 2024-02-05T08:19:10.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3538606185' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:10.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:10 smithi134 ceph-mon[39278]: osdmap e312: 8 total, 8 up, 8 in 2024-02-05T08:19:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:10.779 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:10.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:10 smithi196 ceph-mon[33018]: 1.1 scrub starts 2024-02-05T08:19:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:10 smithi196 ceph-mon[33018]: 1.1 scrub ok 2024-02-05T08:19:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:10 smithi196 ceph-mon[33018]: 1.9 scrub starts 2024-02-05T08:19:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:10 smithi196 ceph-mon[33018]: 1.9 scrub ok 2024-02-05T08:19:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:10 smithi196 ceph-mon[33018]: 5.11 scrub starts 2024-02-05T08:19:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:10 smithi196 ceph-mon[33018]: 5.11 scrub ok 2024-02-05T08:19:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:10 smithi196 ceph-mon[33018]: 4.2 scrub starts 2024-02-05T08:19:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:10 smithi196 ceph-mon[33018]: 4.2 scrub ok 2024-02-05T08:19:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:10 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3538606185' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:10 smithi196 ceph-mon[33018]: osdmap e312: 8 total, 8 up, 8 in 2024-02-05T08:19:10.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:10.904 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:10.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:10.952 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T08:19:11.006 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:11.030 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:11.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:11.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:11.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:11.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:11.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:11.258 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:11.282 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:11.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:11.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:11.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:11.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:11.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:11.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:11.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:11.533 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:11.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:11.635 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:11.659 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:11.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:11.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:11.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:11.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[30086]: 1.a scrub starts 2024-02-05T08:19:11.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[30086]: 1.a scrub ok 2024-02-05T08:19:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[30086]: pgmap v646: 158 pgs: 158 active+clean; 693 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[30086]: 6.c scrub starts 2024-02-05T08:19:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[30086]: 6.c scrub ok 2024-02-05T08:19:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[30086]: 1.0 scrub starts 2024-02-05T08:19:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[30086]: 1.0 scrub ok 2024-02-05T08:19:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[30086]: 3.a scrub starts 2024-02-05T08:19:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[30086]: 3.a scrub ok 2024-02-05T08:19:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[30086]: 6.0 scrub starts 2024-02-05T08:19:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[30086]: 6.0 scrub ok 2024-02-05T08:19:11.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[39278]: 1.a scrub starts 2024-02-05T08:19:11.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[39278]: 1.a scrub ok 2024-02-05T08:19:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[39278]: pgmap v646: 158 pgs: 158 active+clean; 693 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[39278]: 6.c scrub starts 2024-02-05T08:19:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[39278]: 6.c scrub ok 2024-02-05T08:19:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[39278]: 1.0 scrub starts 2024-02-05T08:19:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[39278]: 1.0 scrub ok 2024-02-05T08:19:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[39278]: 3.a scrub starts 2024-02-05T08:19:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[39278]: 3.a scrub ok 2024-02-05T08:19:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[39278]: 6.0 scrub starts 2024-02-05T08:19:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:11 smithi134 ceph-mon[39278]: 6.0 scrub ok 2024-02-05T08:19:11.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:11.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:11.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:11 smithi196 ceph-mon[33018]: 1.a scrub starts 2024-02-05T08:19:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:11 smithi196 ceph-mon[33018]: 1.a scrub ok 2024-02-05T08:19:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:11 smithi196 ceph-mon[33018]: pgmap v646: 158 pgs: 158 active+clean; 693 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:11 smithi196 ceph-mon[33018]: 6.c scrub starts 2024-02-05T08:19:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:11 smithi196 ceph-mon[33018]: 6.c scrub ok 2024-02-05T08:19:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:11 smithi196 ceph-mon[33018]: 1.0 scrub starts 2024-02-05T08:19:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:11 smithi196 ceph-mon[33018]: 1.0 scrub ok 2024-02-05T08:19:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:11 smithi196 ceph-mon[33018]: 3.a scrub starts 2024-02-05T08:19:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:11 smithi196 ceph-mon[33018]: 3.a scrub ok 2024-02-05T08:19:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:11 smithi196 ceph-mon[33018]: 6.0 scrub starts 2024-02-05T08:19:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:11 smithi196 ceph-mon[33018]: 6.0 scrub ok 2024-02-05T08:19:11.819 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:11.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:11.914 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:11.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:11.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:12.015 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:12.039 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:12.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:12.140 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:12.167 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:12.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:12.269 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:12.292 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:12.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:12.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:12.422 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:12.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:12.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:12.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:12.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:12.562 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:12.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:12.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:12.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:12.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[30086]: 1.8 scrub starts 2024-02-05T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[30086]: 1.8 scrub ok 2024-02-05T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[30086]: 3.18 scrub starts 2024-02-05T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[30086]: 3.18 scrub ok 2024-02-05T08:19:12.685 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:19:12.686 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:19:12.686 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[30086]: osdmap e313: 8 total, 8 up, 8 in 2024-02-05T08:19:12.686 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[30086]: 3.6 scrub starts 2024-02-05T08:19:12.686 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[30086]: 3.6 scrub ok 2024-02-05T08:19:12.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[39278]: 1.8 scrub starts 2024-02-05T08:19:12.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[39278]: 1.8 scrub ok 2024-02-05T08:19:12.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:19:12.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:19:12.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[39278]: 3.18 scrub starts 2024-02-05T08:19:12.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[39278]: 3.18 scrub ok 2024-02-05T08:19:12.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:19:12.686 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:19:12.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[39278]: osdmap e313: 8 total, 8 up, 8 in 2024-02-05T08:19:12.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[39278]: 3.6 scrub starts 2024-02-05T08:19:12.687 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:12 smithi134 ceph-mon[39278]: 3.6 scrub ok 2024-02-05T08:19:12.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:12.802 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:12.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:12 smithi196 ceph-mon[33018]: 1.8 scrub starts 2024-02-05T08:19:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:12 smithi196 ceph-mon[33018]: 1.8 scrub ok 2024-02-05T08:19:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:12 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:19:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:12 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:19:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:12 smithi196 ceph-mon[33018]: 3.18 scrub starts 2024-02-05T08:19:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:12 smithi196 ceph-mon[33018]: 3.18 scrub ok 2024-02-05T08:19:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:12 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:19:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:12 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:19:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:12 smithi196 ceph-mon[33018]: osdmap e313: 8 total, 8 up, 8 in 2024-02-05T08:19:12.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:12 smithi196 ceph-mon[33018]: 3.6 scrub starts 2024-02-05T08:19:12.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:12 smithi196 ceph-mon[33018]: 3.6 scrub ok 2024-02-05T08:19:12.904 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:12.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:12.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:12.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:13.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:13.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:13.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:13.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:13.161 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:13.187 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:13.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:13.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:13.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:13.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:13.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:13.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:13.353 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:19:13.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:13.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:13.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:13.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:13.554 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[30086]: 4.7 scrub starts 2024-02-05T08:19:13.554 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[30086]: 4.7 scrub ok 2024-02-05T08:19:13.554 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[30086]: pgmap v649: 158 pgs: 158 active+clean; 693 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:13.554 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[30086]: 5.1c scrub starts 2024-02-05T08:19:13.554 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[30086]: 5.1c scrub ok 2024-02-05T08:19:13.554 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[30086]: 5.2 scrub starts 2024-02-05T08:19:13.555 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[30086]: 5.2 scrub ok 2024-02-05T08:19:13.555 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[39278]: 4.7 scrub starts 2024-02-05T08:19:13.555 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[39278]: 4.7 scrub ok 2024-02-05T08:19:13.555 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[39278]: pgmap v649: 158 pgs: 158 active+clean; 693 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:13.555 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[39278]: 5.1c scrub starts 2024-02-05T08:19:13.555 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[39278]: 5.1c scrub ok 2024-02-05T08:19:13.555 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[39278]: 5.2 scrub starts 2024-02-05T08:19:13.555 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:13 smithi134 ceph-mon[39278]: 5.2 scrub ok 2024-02-05T08:19:13.569 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:13.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:13.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:13.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:13.699 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:13.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:13.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:13.710 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:13 smithi196 ceph-mon[33018]: 4.7 scrub starts 2024-02-05T08:19:13.710 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:13 smithi196 ceph-mon[33018]: 4.7 scrub ok 2024-02-05T08:19:13.710 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:13 smithi196 ceph-mon[33018]: pgmap v649: 158 pgs: 158 active+clean; 693 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:13.710 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:13 smithi196 ceph-mon[33018]: 5.1c scrub starts 2024-02-05T08:19:13.710 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:13 smithi196 ceph-mon[33018]: 5.1c scrub ok 2024-02-05T08:19:13.710 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:13 smithi196 ceph-mon[33018]: 5.2 scrub starts 2024-02-05T08:19:13.710 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:13 smithi196 ceph-mon[33018]: 5.2 scrub ok 2024-02-05T08:19:13.803 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:13.812 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:19:13.829 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:13.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:13.931 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:13.954 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:13.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:13.980 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:19:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:19:14.056 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:14.079 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:14.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:14.181 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:14.204 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:14.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:14.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:14.232 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:19:14.232 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:19:14.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:14.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:14.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:14.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:14.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:14.456 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:14.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:14.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:14.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:14 smithi134 ceph-mon[30086]: 5.15 scrub starts 2024-02-05T08:19:14.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:14 smithi134 ceph-mon[30086]: 5.15 scrub ok 2024-02-05T08:19:14.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:14 smithi134 ceph-mon[30086]: 3.9 scrub starts 2024-02-05T08:19:14.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:14 smithi134 ceph-mon[30086]: 3.9 scrub ok 2024-02-05T08:19:14.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:14 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:19:14.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:14 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:19:14.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:14 smithi134 ceph-mon[39278]: 5.15 scrub starts 2024-02-05T08:19:14.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:14 smithi134 ceph-mon[39278]: 5.15 scrub ok 2024-02-05T08:19:14.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:14 smithi134 ceph-mon[39278]: 3.9 scrub starts 2024-02-05T08:19:14.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:14 smithi134 ceph-mon[39278]: 3.9 scrub ok 2024-02-05T08:19:14.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:14 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:19:14.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:14 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:19:14.558 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:14.582 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:14.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:14.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:14.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:14.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:14.709 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:14.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:14.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:14.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:14 smithi196 ceph-mon[33018]: 5.15 scrub starts 2024-02-05T08:19:14.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:14 smithi196 ceph-mon[33018]: 5.15 scrub ok 2024-02-05T08:19:14.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:14 smithi196 ceph-mon[33018]: 3.9 scrub starts 2024-02-05T08:19:14.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:14 smithi196 ceph-mon[33018]: 3.9 scrub ok 2024-02-05T08:19:14.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:14 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:19:14.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:14 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:19:14.835 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:14.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:14.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:14.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:14.961 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:14.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:14.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:15.001 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:15.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:15.087 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:15.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:15.188 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:15.212 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:15.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:15.314 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:15.338 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:15.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:15.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:15.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:15.464 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:15.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:15.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:15.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T08:19:15.565 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:15.592 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:15.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:15.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:15.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:15.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[30086]: pgmap v650: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:15.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[30086]: 2.0 scrub starts 2024-02-05T08:19:15.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[30086]: 2.0 scrub ok 2024-02-05T08:19:15.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[30086]: 3.1c scrub starts 2024-02-05T08:19:15.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[30086]: 3.1c scrub ok 2024-02-05T08:19:15.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[30086]: 5.5 scrub starts 2024-02-05T08:19:15.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[30086]: 5.5 scrub ok 2024-02-05T08:19:15.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[30086]: osdmap e314: 8 total, 8 up, 8 in 2024-02-05T08:19:15.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[30086]: 4.0 scrub starts 2024-02-05T08:19:15.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[30086]: 4.0 scrub ok 2024-02-05T08:19:15.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:15.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[39278]: pgmap v650: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:15.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[39278]: 2.0 scrub starts 2024-02-05T08:19:15.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[39278]: 2.0 scrub ok 2024-02-05T08:19:15.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[39278]: 3.1c scrub starts 2024-02-05T08:19:15.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[39278]: 3.1c scrub ok 2024-02-05T08:19:15.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[39278]: 5.5 scrub starts 2024-02-05T08:19:15.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[39278]: 5.5 scrub ok 2024-02-05T08:19:15.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[39278]: osdmap e314: 8 total, 8 up, 8 in 2024-02-05T08:19:15.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[39278]: 4.0 scrub starts 2024-02-05T08:19:15.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:15 smithi134 ceph-mon[39278]: 4.0 scrub ok 2024-02-05T08:19:15.694 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:15.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:15.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:15 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:15 smithi196 ceph-mon[33018]: pgmap v650: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:15 smithi196 ceph-mon[33018]: 2.0 scrub starts 2024-02-05T08:19:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:15 smithi196 ceph-mon[33018]: 2.0 scrub ok 2024-02-05T08:19:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:15 smithi196 ceph-mon[33018]: 3.1c scrub starts 2024-02-05T08:19:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:15 smithi196 ceph-mon[33018]: 3.1c scrub ok 2024-02-05T08:19:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:15 smithi196 ceph-mon[33018]: 5.5 scrub starts 2024-02-05T08:19:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:15 smithi196 ceph-mon[33018]: 5.5 scrub ok 2024-02-05T08:19:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:15 smithi196 ceph-mon[33018]: osdmap e314: 8 total, 8 up, 8 in 2024-02-05T08:19:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:15 smithi196 ceph-mon[33018]: 4.0 scrub starts 2024-02-05T08:19:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:15 smithi196 ceph-mon[33018]: 4.0 scrub ok 2024-02-05T08:19:15.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:15.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:15.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:15.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:15.971 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:15.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:15.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:16.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:16.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:16.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:16.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:16.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:16.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:16.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:16.326 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:16.350 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:16.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:16.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:16.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:16.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:16.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:16.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[39278]: 6.15 scrub starts 2024-02-05T08:19:16.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[39278]: 5.e scrub starts 2024-02-05T08:19:16.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[39278]: 5.e scrub ok 2024-02-05T08:19:16.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[39278]: 6.15 scrub ok 2024-02-05T08:19:16.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[39278]: 4.3 scrub starts 2024-02-05T08:19:16.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[39278]: 4.3 scrub ok 2024-02-05T08:19:16.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[39278]: 4.4 scrub starts 2024-02-05T08:19:16.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[39278]: 4.4 scrub ok 2024-02-05T08:19:16.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[39278]: 4.19 scrub starts 2024-02-05T08:19:16.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[39278]: 4.19 scrub ok 2024-02-05T08:19:16.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[30086]: 6.15 scrub starts 2024-02-05T08:19:16.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[30086]: 5.e scrub starts 2024-02-05T08:19:16.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[30086]: 5.e scrub ok 2024-02-05T08:19:16.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[30086]: 6.15 scrub ok 2024-02-05T08:19:16.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[30086]: 4.3 scrub starts 2024-02-05T08:19:16.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[30086]: 4.3 scrub ok 2024-02-05T08:19:16.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[30086]: 4.4 scrub starts 2024-02-05T08:19:16.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[30086]: 4.4 scrub ok 2024-02-05T08:19:16.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[30086]: 4.19 scrub starts 2024-02-05T08:19:16.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:16 smithi134 ceph-mon[30086]: 4.19 scrub ok 2024-02-05T08:19:16.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:16.602 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:16.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:16.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:16.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:16.729 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:16.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:16.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:16 smithi196 ceph-mon[33018]: 6.15 scrub starts 2024-02-05T08:19:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:16 smithi196 ceph-mon[33018]: 5.e scrub starts 2024-02-05T08:19:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:16 smithi196 ceph-mon[33018]: 5.e scrub ok 2024-02-05T08:19:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:16 smithi196 ceph-mon[33018]: 6.15 scrub ok 2024-02-05T08:19:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:16 smithi196 ceph-mon[33018]: 4.3 scrub starts 2024-02-05T08:19:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:16 smithi196 ceph-mon[33018]: 4.3 scrub ok 2024-02-05T08:19:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:16 smithi196 ceph-mon[33018]: 4.4 scrub starts 2024-02-05T08:19:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:16 smithi196 ceph-mon[33018]: 4.4 scrub ok 2024-02-05T08:19:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:16 smithi196 ceph-mon[33018]: 4.19 scrub starts 2024-02-05T08:19:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:16 smithi196 ceph-mon[33018]: 4.19 scrub ok 2024-02-05T08:19:16.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:16.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:16.856 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:16.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:16.957 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:16.981 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:16.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:17.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:17.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:17.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:17.159 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:17.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:17.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:17.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:17.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:17.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:17.361 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-02-05T08:19:17.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:17.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:17.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:17.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:17.495 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:17.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:17.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:17.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:17.624 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:17.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:17.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:17.726 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:17.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[30086]: pgmap v652: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:17.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[30086]: 6.e scrub starts 2024-02-05T08:19:17.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[30086]: 6.e scrub ok 2024-02-05T08:19:17.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[30086]: 5.16 scrub starts 2024-02-05T08:19:17.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[30086]: 5.16 scrub ok 2024-02-05T08:19:17.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[30086]: 3.10 scrub starts 2024-02-05T08:19:17.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[30086]: 3.10 scrub ok 2024-02-05T08:19:17.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[30086]: 6.4 scrub starts 2024-02-05T08:19:17.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[30086]: 6.4 scrub ok 2024-02-05T08:19:17.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3353293662' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:19:17.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[39278]: pgmap v652: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:17.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[39278]: 6.e scrub starts 2024-02-05T08:19:17.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[39278]: 6.e scrub ok 2024-02-05T08:19:17.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[39278]: 5.16 scrub starts 2024-02-05T08:19:17.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[39278]: 5.16 scrub ok 2024-02-05T08:19:17.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[39278]: 3.10 scrub starts 2024-02-05T08:19:17.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[39278]: 3.10 scrub ok 2024-02-05T08:19:17.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[39278]: 6.4 scrub starts 2024-02-05T08:19:17.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[39278]: 6.4 scrub ok 2024-02-05T08:19:17.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3353293662' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:19:17.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:17.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:17 smithi196 ceph-mon[33018]: pgmap v652: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:17 smithi196 ceph-mon[33018]: 6.e scrub starts 2024-02-05T08:19:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:17 smithi196 ceph-mon[33018]: 6.e scrub ok 2024-02-05T08:19:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:17 smithi196 ceph-mon[33018]: 5.16 scrub starts 2024-02-05T08:19:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:17 smithi196 ceph-mon[33018]: 5.16 scrub ok 2024-02-05T08:19:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:17 smithi196 ceph-mon[33018]: 3.10 scrub starts 2024-02-05T08:19:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:17 smithi196 ceph-mon[33018]: 3.10 scrub ok 2024-02-05T08:19:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:17 smithi196 ceph-mon[33018]: 6.4 scrub starts 2024-02-05T08:19:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:17 smithi196 ceph-mon[33018]: 6.4 scrub ok 2024-02-05T08:19:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:17 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3353293662' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:19:17.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:17.871 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:17.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:17.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:17.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:17.884 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:19:17.906 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:19:17.925 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:19:17.944 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:19:17.964 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:19:17.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:17.984 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:19:17.994 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:19:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:19:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:19:18.004 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:19:18.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:18.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:18.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:18.024 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:19:18.025 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:19:18.025 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:19:18.025 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:19:18.025 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:19:18.025 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:19:18.026 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:19:18.026 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:19:18.026 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:19:18.106 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:18.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:18.132 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:18.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:18.233 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:18.257 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:18.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:18.358 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:18.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:18.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:18.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:18.486 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:18.578 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3353293662' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:19:18.578 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[30086]: osdmap e315: 8 total, 8 up, 8 in 2024-02-05T08:19:18.578 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[30086]: 3.c scrub starts 2024-02-05T08:19:18.579 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[30086]: 3.c scrub ok 2024-02-05T08:19:18.579 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[30086]: 4.b scrub starts 2024-02-05T08:19:18.579 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[30086]: 4.b scrub ok 2024-02-05T08:19:18.579 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[30086]: 5.12 scrub starts 2024-02-05T08:19:18.579 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[30086]: 5.12 scrub ok 2024-02-05T08:19:18.579 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1265528314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:18.579 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:19:18.579 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:19:18.580 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3353293662' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:19:18.580 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[39278]: osdmap e315: 8 total, 8 up, 8 in 2024-02-05T08:19:18.580 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[39278]: 3.c scrub starts 2024-02-05T08:19:18.580 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[39278]: 3.c scrub ok 2024-02-05T08:19:18.580 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[39278]: 4.b scrub starts 2024-02-05T08:19:18.580 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[39278]: 4.b scrub ok 2024-02-05T08:19:18.580 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[39278]: 5.12 scrub starts 2024-02-05T08:19:18.580 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[39278]: 5.12 scrub ok 2024-02-05T08:19:18.580 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1265528314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:18.580 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:19:18.581 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:18 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:19:18.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:18.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:18.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:18.742 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:18.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:18 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3353293662' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:19:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:18 smithi196 ceph-mon[33018]: osdmap e315: 8 total, 8 up, 8 in 2024-02-05T08:19:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:18 smithi196 ceph-mon[33018]: 3.c scrub starts 2024-02-05T08:19:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:18 smithi196 ceph-mon[33018]: 3.c scrub ok 2024-02-05T08:19:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:18 smithi196 ceph-mon[33018]: 4.b scrub starts 2024-02-05T08:19:18.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:18 smithi196 ceph-mon[33018]: 4.b scrub ok 2024-02-05T08:19:18.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:18 smithi196 ceph-mon[33018]: 5.12 scrub starts 2024-02-05T08:19:18.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:18 smithi196 ceph-mon[33018]: 5.12 scrub ok 2024-02-05T08:19:18.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:18 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1265528314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:18.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:18 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:19:18.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:18 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:19:18.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:18.870 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:18.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:18.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:18.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:19.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:19.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:19.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:19.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:19.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:19.258 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:19.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:19.360 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:19.384 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:19.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:19.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:19.486 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:19.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:19.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:19.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:19.614 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:19.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:19.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:19.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[30086]: pgmap v654: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:19.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[30086]: 3.f scrub starts 2024-02-05T08:19:19.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[30086]: 3.f scrub ok 2024-02-05T08:19:19.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[30086]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:19:19.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[30086]: osdmap e316: 8 total, 8 up, 8 in 2024-02-05T08:19:19.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[30086]: 3.5 scrub starts 2024-02-05T08:19:19.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[30086]: 3.5 scrub ok 2024-02-05T08:19:19.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[39278]: pgmap v654: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:19.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[39278]: 3.f scrub starts 2024-02-05T08:19:19.706 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[39278]: 3.f scrub ok 2024-02-05T08:19:19.706 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[39278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:19:19.706 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[39278]: osdmap e316: 8 total, 8 up, 8 in 2024-02-05T08:19:19.706 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[39278]: 3.5 scrub starts 2024-02-05T08:19:19.706 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:19 smithi134 ceph-mon[39278]: 3.5 scrub ok 2024-02-05T08:19:19.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:19.764 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:19.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:19.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:19 smithi196 ceph-mon[33018]: pgmap v654: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-02-05T08:19:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:19 smithi196 ceph-mon[33018]: 3.f scrub starts 2024-02-05T08:19:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:19 smithi196 ceph-mon[33018]: 3.f scrub ok 2024-02-05T08:19:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:19 smithi196 ceph-mon[33018]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:19:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:19 smithi196 ceph-mon[33018]: osdmap e316: 8 total, 8 up, 8 in 2024-02-05T08:19:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:19 smithi196 ceph-mon[33018]: 3.5 scrub starts 2024-02-05T08:19:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:19 smithi196 ceph-mon[33018]: 3.5 scrub ok 2024-02-05T08:19:19.865 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:19.888 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:19.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:19.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:19.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:19.995 DEBUG:teuthology.orchestra.run.smithi134:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-02-05T08:19:19.996 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-02-05T08:19:19.996 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-02-05T08:19:19.996 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:19:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:20.015 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:20.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:20.015 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T08:19:20.016 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T08:19:20.116 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:20.125 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:hints = 1 2024-02-05T08:19:20.125 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T08:19:20.125 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_174861 2024-02-05T08:19:20.125 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:19:20.125 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-02-05T08:19:20.152 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:20.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:20.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:20.295 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:20.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:20.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:20.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:20.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:20.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:20.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:20.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:20.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:20.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:20.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:20.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:20.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:20.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:19:20.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:19:20.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[30086]: 3.1a scrub starts 2024-02-05T08:19:20.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[30086]: 3.1a scrub ok 2024-02-05T08:19:20.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[30086]: 6.7 scrub starts 2024-02-05T08:19:20.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[30086]: 6.7 scrub ok 2024-02-05T08:19:20.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[30086]: 3.2 scrub starts 2024-02-05T08:19:20.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[30086]: 3.2 scrub ok 2024-02-05T08:19:20.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:19:20.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:19:20.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[39278]: 3.1a scrub starts 2024-02-05T08:19:20.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[39278]: 3.1a scrub ok 2024-02-05T08:19:20.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[39278]: 6.7 scrub starts 2024-02-05T08:19:20.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[39278]: 6.7 scrub ok 2024-02-05T08:19:20.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[39278]: 3.2 scrub starts 2024-02-05T08:19:20.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:20 smithi134 ceph-mon[39278]: 3.2 scrub ok 2024-02-05T08:19:20.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:20 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:19:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:20 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:19:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:20 smithi196 ceph-mon[33018]: 3.1a scrub starts 2024-02-05T08:19:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:20 smithi196 ceph-mon[33018]: 3.1a scrub ok 2024-02-05T08:19:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:20 smithi196 ceph-mon[33018]: 6.7 scrub starts 2024-02-05T08:19:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:20 smithi196 ceph-mon[33018]: 6.7 scrub ok 2024-02-05T08:19:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:20 smithi196 ceph-mon[33018]: 3.2 scrub starts 2024-02-05T08:19:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:20 smithi196 ceph-mon[33018]: 3.2 scrub ok 2024-02-05T08:19:20.854 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:20.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:20.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:20.995 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:20.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:20.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:21.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:21.125 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 1 15 3657 3642 227.612 227.625 0.00306185 0.00437885 2024-02-05T08:19:21.133 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:21.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:21.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:21.272 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:21.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:21.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:21.384 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:21 smithi134 ceph-mon[30086]: pgmap v656: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:19:21.384 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:21 smithi134 ceph-mon[30086]: 6.1c scrub starts 2024-02-05T08:19:21.385 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:21 smithi134 ceph-mon[30086]: 6.1c scrub ok 2024-02-05T08:19:21.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:21.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:21.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:21.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:21.553 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:21.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:21.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:19:21.644 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:21 smithi134 ceph-mon[39278]: pgmap v656: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:19:21.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:21 smithi134 ceph-mon[39278]: 6.1c scrub starts 2024-02-05T08:19:21.645 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:21 smithi134 ceph-mon[39278]: 6.1c scrub ok 2024-02-05T08:19:21.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:21.698 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:21.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:21.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:21 smithi196 ceph-mon[33018]: pgmap v656: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:19:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:21 smithi196 ceph-mon[33018]: 6.1c scrub starts 2024-02-05T08:19:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:21 smithi196 ceph-mon[33018]: 6.1c scrub ok 2024-02-05T08:19:21.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:21.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:21.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:21.972 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:21.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:22.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:22.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:22.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:22.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:22.125 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 2 16 6964 6948 217.074 206.625 0.00481173 0.004594 2024-02-05T08:19:22.212 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:22.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:22.258 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:22.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:22.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:22.423 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:22.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:22.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:22.565 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:22.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:22.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:22.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:22.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:22.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:22.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:22.751 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:19:22.751 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:19:22.752 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:19:22.752 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:19:22.752 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.752 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108936 8.6 8:6ce9955c:::benchmark_data_smithi134_115836_object108935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-05T08:19:22.752 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:28.999547+0000", 2024-02-05T08:19:22.752 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 413.35241236899998, 2024-02-05T08:19:22.752 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.767948115, 2024-02-05T08:19:22.752 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.752 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:19:22.752 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.752 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.752 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.752 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108936 2024-02-05T08:19:22.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.753 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.753 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.753 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999547+0000", 2024-02-05T08:19:22.753 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.753 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.753 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999547+0000", 2024-02-05T08:19:22.753 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.753 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.753 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.754 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.754 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999548+0000", 2024-02-05T08:19:22.754 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.195e-06 2024-02-05T08:19:22.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.754 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.754 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999572+0000", 2024-02-05T08:19:22.754 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3781000000000002e-05 2024-02-05T08:19:22.754 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.754 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.754 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.754 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999583+0000", 2024-02-05T08:19:22.754 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1143e-05 2024-02-05T08:19:22.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.755 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.755 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999591+0000", 2024-02-05T08:19:22.755 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1140000000000006e-06 2024-02-05T08:19:22.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.755 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.755 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.005098+0000", 2024-02-05T08:19:22.755 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0055070800000000001 2024-02-05T08:19:22.755 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.755 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.755 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.005332+0000", 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023392899999999999 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,7", 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.005485+0000", 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015303800000000001 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.015634+0000", 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010149004 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.756 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.757 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.015665+0000", 2024-02-05T08:19:22.757 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0966000000000001e-05 2024-02-05T08:19:22.757 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.757 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.757 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.757 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.041481+0000", 2024-02-05T08:19:22.757 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.025815647000000001 2024-02-05T08:19:22.757 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.757 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.757 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.757 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.041495+0000", 2024-02-05T08:19:22.757 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4079e-05 2024-02-05T08:19:22.757 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.758 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.758 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.758 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666718+0000", 2024-02-05T08:19:22.758 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.62522305499999997 2024-02-05T08:19:22.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.758 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.758 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.758 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666730+0000", 2024-02-05T08:19:22.758 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2157000000000001e-05 2024-02-05T08:19:22.758 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.758 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.758 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.758 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.767398+0000", 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1006681920000001 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.767495+0000", 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6735000000000001e-05 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108967 8.6 8:6b4bec94:::benchmark_data_smithi134_115836_object108966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.046490+0000", 2024-02-05T08:19:22.759 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 413.3054692, 2024-02-05T08:19:22.760 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.731470512, 2024-02-05T08:19:22.760 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.760 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.760 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.760 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.760 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.760 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108967 2024-02-05T08:19:22.760 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.760 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.760 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.760 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.760 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046490+0000", 2024-02-05T08:19:22.760 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046490+0000", 2024-02-05T08:19:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046492+0000", 2024-02-05T08:19:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7060000000000001e-06 2024-02-05T08:19:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.761 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046518+0000", 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5791000000000001e-05 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046527+0000", 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3980000000000005e-06 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046537+0000", 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0380999999999999e-05 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.762 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.053715+0000", 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0071773959999999999 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.053726+0000", 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1117999999999999e-05 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666815+0000", 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.61308943100000002 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.763 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666824+0000", 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0059999999999998e-06 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.767788+0000", 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1009632439999999 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.767916+0000", 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012850399999999999 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.764 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.768062+0000", 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014636200000000001 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777461+0000", 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0093981900000000007 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777525+0000", 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4302999999999997e-05 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.765 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777717+0000", 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019194500000000001 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777765+0000", 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8143999999999998e-05 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777864+0000", 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9068999999999996e-05 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.766 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777961+0000", 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6523999999999999e-05 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108968 8.6 8:680b4c21:::benchmark_data_smithi134_115836_object108967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.048233+0000", 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 413.303726697, 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7300381819999999, 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.767 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108968 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048233+0000", 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048233+0000", 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.768 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048235+0000", 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.018e-06 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048260+0000", 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5247000000000001e-05 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048272+0000", 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2483000000000001e-05 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.769 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048345+0000", 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2766000000000001e-05 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.053751+0000", 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0054057139999999998 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.053759+0000", 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0479999999999999e-06 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.770 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666852+0000", 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.61309340899999998 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666861+0000", 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5750000000000001e-06 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.770884+0000", 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1040229189999999 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.771 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.771021+0000", 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000137353 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.771230+0000", 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020888999999999999 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777723+0000", 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00649338 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.772 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778025+0000", 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030212499999999998 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778073+0000", 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7438000000000002e-05 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778116+0000", 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3133000000000002e-05 2024-02-05T08:19:22.773 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778154+0000", 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7744999999999999e-05 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778271+0000", 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000116939 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108988 8.6 8:6273284f:::benchmark_data_smithi134_115836_object108987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:19:22.774 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.068824+0000", 2024-02-05T08:19:22.775 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 413.28313486799999, 2024-02-05T08:19:22.775 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7214295480000001, 2024-02-05T08:19:22.775 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.775 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.775 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.775 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.775 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.775 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108988 2024-02-05T08:19:22.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.775 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.775 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.775 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.775 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068824+0000", 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068824+0000", 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068826+0000", 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.35e-06 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.776 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068848+0000", 2024-02-05T08:19:22.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2093e-05 2024-02-05T08:19:22.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068861+0000", 2024-02-05T08:19:22.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2918000000000001e-05 2024-02-05T08:19:22.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.777 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.777 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.777 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068871+0000", 2024-02-05T08:19:22.777 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.009e-05 2024-02-05T08:19:22.777 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068950+0000", 2024-02-05T08:19:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8747999999999999e-05 2024-02-05T08:19:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068962+0000", 2024-02-05T08:19:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2023e-05 2024-02-05T08:19:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.778 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666889+0000", 2024-02-05T08:19:22.779 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.59792759500000003 2024-02-05T08:19:22.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.779 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.779 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666898+0000", 2024-02-05T08:19:22.779 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4479999999999994e-06 2024-02-05T08:19:22.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.779 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.779 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778301+0000", 2024-02-05T08:19:22.779 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.111403232 2024-02-05T08:19:22.779 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.779 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.780 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.780 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778649+0000", 2024-02-05T08:19:22.780 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00034766799999999998 2024-02-05T08:19:22.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.780 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:19:22.780 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778800+0000", 2024-02-05T08:19:22.780 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000151101 2024-02-05T08:19:22.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.780 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.780 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.780 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788785+0000", 2024-02-05T08:19:22.781 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0099855810000000003 2024-02-05T08:19:22.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.781 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.781 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.781 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788832+0000", 2024-02-05T08:19:22.781 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6394000000000003e-05 2024-02-05T08:19:22.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.781 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.782 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.782 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788885+0000", 2024-02-05T08:19:22.782 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3412999999999998e-05 2024-02-05T08:19:22.782 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.782 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.782 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.782 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788973+0000", 2024-02-05T08:19:22.782 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7843000000000006e-05 2024-02-05T08:19:22.782 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.782 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.782 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.782 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.789172+0000", 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019889199999999999 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790254+0000", 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010821590000000001 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108994 8.6 8:69b35254:::benchmark_data_smithi134_115836_object108993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.072494+0000", 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 413.279465555, 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7180740800000001, 2024-02-05T08:19:22.783 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108994 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072494+0000", 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.784 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072494+0000", 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072496+0000", 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.815e-06 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072522+0000", 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6954e-05 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.785 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072539+0000", 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6602000000000001e-05 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072548+0000", 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3070000000000008e-06 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072616+0000", 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7382e-05 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.786 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072628+0000", 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1772e-05 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666926+0000", 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.59429891000000001 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666935+0000", 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0989999999999999e-06 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.782050+0000", 2024-02-05T08:19:22.787 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1151151159999999 2024-02-05T08:19:22.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.782294+0000", 2024-02-05T08:19:22.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000243849 2024-02-05T08:19:22.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:19:22.788 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.782456+0000", 2024-02-05T08:19:22.788 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016257700000000001 2024-02-05T08:19:22.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.788 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788847+0000", 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.006391112 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788929+0000", 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1520000000000006e-05 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790307+0000", 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0013783059999999999 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.789 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790406+0000", 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9007999999999998e-05 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790467+0000", 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1057999999999997e-05 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790568+0000", 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010069300000000001 2024-02-05T08:19:22.790 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.791 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.791 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.791 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.791 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.791 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108995 8.6 8:6351071d:::benchmark_data_smithi134_115836_object108994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:19:22.791 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.072933+0000", 2024-02-05T08:19:22.791 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 413.27902661799999, 2024-02-05T08:19:22.791 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7266677289999999, 2024-02-05T08:19:22.791 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.791 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.791 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.791 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.791 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.792 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108995 2024-02-05T08:19:22.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.792 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.792 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.792 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.792 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072933+0000", 2024-02-05T08:19:22.792 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.792 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.792 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.792 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072933+0000", 2024-02-05T08:19:22.792 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072934+0000", 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0410000000000001e-06 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072956+0000", 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2197000000000001e-05 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072971+0000", 2024-02-05T08:19:22.793 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5152e-05 2024-02-05T08:19:22.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.794 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.794 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.794 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072979+0000", 2024-02-05T08:19:22.794 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9689999999999994e-06 2024-02-05T08:19:22.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.794 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.794 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.794 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.073032+0000", 2024-02-05T08:19:22.794 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3375000000000003e-05 2024-02-05T08:19:22.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.794 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.794 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.073041+0000", 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0480000000000004e-06 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666963+0000", 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.59392174399999997 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666972+0000", 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4500000000000004e-06 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.795 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.785402+0000", 2024-02-05T08:19:22.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1184303840000001 2024-02-05T08:19:22.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.796 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.785534+0000", 2024-02-05T08:19:22.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013186 2024-02-05T08:19:22.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.796 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.796 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:19:22.796 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.785689+0000", 2024-02-05T08:19:22.796 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000155487 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790323+0000", 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0046338509999999996 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790357+0000", 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3657e-05 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.797 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790612+0000", 2024-02-05T08:19:22.798 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00025531600000000001 2024-02-05T08:19:22.798 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.798 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.798 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.798 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.799422+0000", 2024-02-05T08:19:22.798 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0088102289999999993 2024-02-05T08:19:22.798 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.798 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.798 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.798 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.799489+0000", 2024-02-05T08:19:22.798 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6593000000000004e-05 2024-02-05T08:19:22.798 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.798 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.799 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.799 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.799600+0000", 2024-02-05T08:19:22.799 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111376 2024-02-05T08:19:22.799 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.799 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.799 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.799 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.799 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108980 8.5 8:ad067b35:::benchmark_data_smithi134_115836_object108979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e188)", 2024-02-05T08:19:22.799 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.663590+0000", 2024-02-05T08:19:22.799 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 412.68836954099999, 2024-02-05T08:19:22.799 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1061829890000001, 2024-02-05T08:19:22.799 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108980 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.663590+0000", 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.800 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.663590+0000", 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.663591+0000", 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7680000000000001e-06 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.663619+0000", 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7832999999999999e-05 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.801 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.665311+0000", 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0016921149999999999 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.665334+0000", 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2626e-05 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666524+0000", 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011904350000000001 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.802 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666534+0000", 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3009999999999995e-06 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.761805+0000", 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.095271168 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.763439+0000", 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001634251 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,6,7", 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.763616+0000", 2024-02-05T08:19:22.803 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000176711 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769220+0000", 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0056040200000000004 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769258+0000", 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8101e-05 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.804 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769293+0000", 2024-02-05T08:19:22.805 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4687999999999997e-05 2024-02-05T08:19:22.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.805 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.805 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.805 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769610+0000", 2024-02-05T08:19:22.805 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031678300000000001 2024-02-05T08:19:22.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.805 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.805 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.805 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769668+0000", 2024-02-05T08:19:22.805 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8451e-05 2024-02-05T08:19:22.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.805 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769773+0000", 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104738 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:134904 8.e 8:7d534b82:::benchmark_data_smithi134_115836_object18244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.506668+0000", 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 379.84529108800001, 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.19134764500000001, 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.806 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 134904 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506668+0000", 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506668+0000", 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.807 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506669+0000", 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4900000000000004e-07 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506674+0000", 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0529999999999997e-06 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506688+0000", 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4287999999999999e-05 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506696+0000", 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1709999999999996e-06 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.516764+0000", 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010068344999999999 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.516879+0000", 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114751 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.516995+0000", 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115844 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.521656+0000", 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0046613660000000001 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.528669+0000", 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0070134309999999997 2024-02-05T08:19:22.810 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.697929+0000", 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16926000699999999 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.697981+0000", 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1521999999999999e-05 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.698016+0000", 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4918e-05 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.811 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:134932 8.e 8:747554f1:::benchmark_data_smithi134_115836_object18272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.538807+0000", 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 379.81315214900002, 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.18031301799999999, 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 134932 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538807+0000", 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538807+0000", 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538808+0000", 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.33e-06 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538813+0000", 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9429999999999999e-06 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.694877+0000", 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15606357700000001 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.694895+0000", 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8104000000000001e-05 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.814 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.698658+0000", 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0037627429999999998 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.698821+0000", 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016332799999999999 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.698943+0000", 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012196 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.815 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.708664+0000", 2024-02-05T08:19:22.816 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0097209140000000006 2024-02-05T08:19:22.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.816 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.816 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.816 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.708709+0000", 2024-02-05T08:19:22.816 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5241999999999998e-05 2024-02-05T08:19:22.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.816 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.816 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.816 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.718868+0000", 2024-02-05T08:19:22.816 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010159062999999999 2024-02-05T08:19:22.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.719076+0000", 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020797 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.719120+0000", 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3844000000000002e-05 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.818 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15087.0:162972 8.8 e214/194 8:14305b79:::benchmark_data_smithi134_115836_object46312:head v 214'10137, mlcod=214'10135)", 2024-02-05T08:19:22.818 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:37.286547+0000", 2024-02-05T08:19:22.818 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 345.065412477, 2024-02-05T08:19:22.818 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.187515187, 2024-02-05T08:19:22.818 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.818 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.818 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.818 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.818 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.818 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286547+0000", 2024-02-05T08:19:22.818 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286547+0000", 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286547+0000", 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8800000000000002e-07 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286553+0000", 2024-02-05T08:19:22.819 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9819999999999996e-06 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286560+0000", 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1380000000000001e-06 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286567+0000", 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3600000000000001e-06 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286636+0000", 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8868999999999994e-05 2024-02-05T08:19:22.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286659+0000", 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.304e-05 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.474023+0000", 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.187364261 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.474062+0000", 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8949000000000003e-05 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.821 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDPGPush(8.8 214/194 [PushOp(8:1b8b759a:::benchmark_data_smithi134_115836_object58675:head, version: 173'3646, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:1b8b759a:::benchmark_data_smithi134_115836_object58675:head@173'3646, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:37.291477+0000", 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 345.06048191100001, 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.18357304299999999, 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.291477+0000", 2024-02-05T08:19:22.822 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.291477+0000", 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.291478+0000", 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0800000000000004e-07 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.291523+0000", 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5318999999999998e-05 2024-02-05T08:19:22.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.464807+0000", 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.17328333600000001 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.464836+0000", 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9229000000000001e-05 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.474831+0000", 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0099948740000000008 2024-02-05T08:19:22.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.474844+0000", 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3338e-05 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.475050+0000", 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020613900000000001 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.825 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170535 8.c 8:30bf855a:::benchmark_data_smithi134_115836_object53875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.749814+0000", 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 335.60214503499998, 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.99481441999999998, 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170535 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749814+0000", 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749814+0000", 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749816+0000", 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.547e-06 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749821+0000", 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8679999999999998e-06 2024-02-05T08:19:22.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749834+0000", 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3169999999999999e-05 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749842+0000", 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3229999999999998e-06 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749901+0000", 2024-02-05T08:19:22.828 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8464000000000001e-05 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749909+0000", 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8550000000000008e-06 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.733527+0000", 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.98361745899999997 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.829 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.733637+0000", 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109769 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.733763+0000", 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000126191 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744320+0000", 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010556640000000001 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.830 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744459+0000", 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013946999999999999 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744506+0000", 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7512000000000002e-05 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744579+0000", 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2857999999999999e-05 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.831 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744629+0000", 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9293999999999999e-05 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170561 8.c 8:320abcbc:::benchmark_data_smithi134_115836_object53901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.770164+0000", 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 335.58179494299998, 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.97474936999999995, 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.832 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170561 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770164+0000", 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770164+0000", 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.833 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770166+0000", 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6750000000000001e-06 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770187+0000", 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1461999999999999e-05 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770200+0000", 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2884000000000001e-05 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.834 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770208+0000", 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6839999999999996e-06 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770266+0000", 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8081000000000002e-05 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770275+0000", 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4579999999999993e-06 2024-02-05T08:19:22.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.836 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.836 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.836 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.736290+0000", 2024-02-05T08:19:22.836 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.96601568100000001 2024-02-05T08:19:22.836 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.836 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.836 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.836 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.736375+0000", 2024-02-05T08:19:22.836 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5116000000000003e-05 2024-02-05T08:19:22.836 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.836 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.836 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.736487+0000", 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111376 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744402+0000", 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0079155340000000001 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744684+0000", 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00028183599999999999 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.837 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744761+0000", 2024-02-05T08:19:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7120999999999996e-05 2024-02-05T08:19:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744873+0000", 2024-02-05T08:19:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011210299999999999 2024-02-05T08:19:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744914+0000", 2024-02-05T08:19:22.838 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0358999999999998e-05 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170591 8.c 8:3e90cc49:::benchmark_data_smithi134_115836_object53931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.795150+0000", 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 335.556809391, 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.949924875, 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.839 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170591 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795150+0000", 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795150+0000", 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.840 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795151+0000", 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2049999999999999e-06 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795156+0000", 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.634e-06 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795168+0000", 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2119999999999999e-05 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.841 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795181+0000", 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3655999999999999e-05 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795293+0000", 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011196600000000001 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795312+0000", 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8717000000000001e-05 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.842 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.738715+0000", 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94340282600000003 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.738798+0000", 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3123999999999999e-05 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.738868+0000", 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9904e-05 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.843 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744471+0000", 2024-02-05T08:19:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0056029870000000002 2024-02-05T08:19:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744956+0000", 2024-02-05T08:19:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00048495400000000001 2024-02-05T08:19:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744999+0000", 2024-02-05T08:19:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3164999999999998e-05 2024-02-05T08:19:22.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745044+0000", 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4780999999999997e-05 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745075+0000", 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0836000000000001e-05 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.845 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170592 8.c 8:3f22b68d:::benchmark_data_smithi134_115836_object53932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:19:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.795212+0000", 2024-02-05T08:19:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 335.55674686999998, 2024-02-05T08:19:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95563532299999998, 2024-02-05T08:19:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170592 2024-02-05T08:19:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.846 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795212+0000", 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795212+0000", 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795213+0000", 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3200000000000005e-07 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.847 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795216+0000", 2024-02-05T08:19:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7470000000000001e-06 2024-02-05T08:19:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795223+0000", 2024-02-05T08:19:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9240000000000001e-06 2024-02-05T08:19:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.848 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795228+0000", 2024-02-05T08:19:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7729999999999996e-06 2024-02-05T08:19:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795334+0000", 2024-02-05T08:19:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000105321 2024-02-05T08:19:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795339+0000", 2024-02-05T08:19:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7440000000000004e-06 2024-02-05T08:19:22.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.741396+0000", 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94605645199999999 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.741504+0000", 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107666 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.741620+0000", 2024-02-05T08:19:22.850 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011629900000000001 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744703+0000", 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030836259999999999 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750711+0000", 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0060076019999999999 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750749+0000", 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8243e-05 2024-02-05T08:19:22.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750814+0000", 2024-02-05T08:19:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4344999999999995e-05 2024-02-05T08:19:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750848+0000", 2024-02-05T08:19:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4048999999999999e-05 2024-02-05T08:19:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170598 8.c 8:396b7832:::benchmark_data_smithi134_115836_object53938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:19:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.801793+0000", 2024-02-05T08:19:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 335.55016594699998, 2024-02-05T08:19:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95466330200000005, 2024-02-05T08:19:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170598 2024-02-05T08:19:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.853 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801793+0000", 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801793+0000", 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801794+0000", 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0699999999999999e-06 2024-02-05T08:19:22.854 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801798+0000", 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9999999999999998e-06 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801812+0000", 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3257e-05 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801820+0000", 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5380000000000002e-06 2024-02-05T08:19:22.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801881+0000", 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0494000000000001e-05 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801890+0000", 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5859999999999994e-06 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745103+0000", 2024-02-05T08:19:22.856 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94321233000000004 2024-02-05T08:19:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745242+0000", 2024-02-05T08:19:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000139357 2024-02-05T08:19:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.857 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.857 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:22 smithi134 ceph-mon[30086]: 6.2 scrub starts 2024-02-05T08:19:22.857 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:22 smithi134 ceph-mon[30086]: 6.2 scrub ok 2024-02-05T08:19:22.857 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:22 smithi134 ceph-mon[30086]: 5.1b scrub starts 2024-02-05T08:19:22.857 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:22 smithi134 ceph-mon[30086]: 5.1b scrub ok 2024-02-05T08:19:22.857 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:22 smithi134 ceph-mon[30086]: 6.18 scrub starts 2024-02-05T08:19:22.858 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:22 smithi134 ceph-mon[30086]: 6.18 scrub ok 2024-02-05T08:19:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:19:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745347+0000", 2024-02-05T08:19:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000105308 2024-02-05T08:19:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.858 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750885+0000", 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0055373130000000003 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756344+0000", 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.005459518 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756359+0000", 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5268999999999999e-05 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.859 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756414+0000", 2024-02-05T08:19:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4469999999999999e-05 2024-02-05T08:19:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756457+0000", 2024-02-05T08:19:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2792e-05 2024-02-05T08:19:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.860 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170600 8.c 8:3e055f0a:::benchmark_data_smithi134_115836_object53940:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.801861+0000", 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 335.55009875000002, 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95753421999999999, 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170600 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.861 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801861+0000", 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801861+0000", 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801861+0000", 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7099999999999997e-07 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801863+0000", 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4109999999999999e-06 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801869+0000", 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.789e-06 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801874+0000", 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3310000000000003e-06 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801917+0000", 2024-02-05T08:19:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2317000000000002e-05 2024-02-05T08:19:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801924+0000", 2024-02-05T08:19:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4080000000000001e-06 2024-02-05T08:19:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.748074+0000", 2024-02-05T08:19:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94614989999999999 2024-02-05T08:19:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.864 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.748198+0000", 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012395200000000001 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.748296+0000", 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8288000000000005e-05 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756466+0000", 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0081697049999999993 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.865 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756527+0000", 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1192000000000006e-05 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759307+0000", 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027796460000000002 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759358+0000", 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0918000000000003e-05 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.866 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759395+0000", 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6992000000000001e-05 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170603 8.c 8:335f5460:::benchmark_data_smithi134_115836_object53943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.805707+0000", 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 335.54625208800002, 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95838726600000002, 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.867 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170603 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805707+0000", 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805707+0000", 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.868 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805709+0000", 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.415e-06 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805713+0000", 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6020000000000002e-06 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805725+0000", 2024-02-05T08:19:22.869 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1729000000000001e-05 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805732+0000", 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4839999999999998e-06 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805784+0000", 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1134000000000002e-05 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805792+0000", 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6440000000000006e-06 2024-02-05T08:19:22.870 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750912+0000", 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94512030199999997 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.751005+0000", 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2283999999999999e-05 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.751107+0000", 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000101828 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.871 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756471+0000", 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0053649029999999999 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759438+0000", 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002966266 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764000+0000", 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0045619750000000002 2024-02-05T08:19:22.872 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764055+0000", 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5158000000000003e-05 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764094+0000", 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9542000000000002e-05 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.873 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170608 8.c 8:33618a8a:::benchmark_data_smithi134_115836_object53948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.807038+0000", 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 335.54492085800001, 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95716985099999996, 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170608 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.874 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807038+0000", 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807038+0000", 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807039+0000", 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4e-07 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.875 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807044+0000", 2024-02-05T08:19:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9660000000000004e-06 2024-02-05T08:19:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807056+0000", 2024-02-05T08:19:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1437000000000001e-05 2024-02-05T08:19:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807064+0000", 2024-02-05T08:19:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2779999999999993e-06 2024-02-05T08:19:22.876 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807115+0000", 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0725e-05 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807122+0000", 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4449999999999999e-06 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.753804+0000", 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94668188499999995 2024-02-05T08:19:22.877 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.753906+0000", 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000102166 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.753985+0000", 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8432999999999999e-05 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756543+0000", 2024-02-05T08:19:22.878 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0025579399999999999 2024-02-05T08:19:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759479+0000", 2024-02-05T08:19:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0029361249999999999 2024-02-05T08:19:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764128+0000", 2024-02-05T08:19:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0046493369999999999 2024-02-05T08:19:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.879 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764168+0000", 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9558999999999999e-05 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764208+0000", 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0714999999999999e-05 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170625 8.c 8:322ad108:::benchmark_data_smithi134_115836_object53965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.834138+0000", 2024-02-05T08:19:22.880 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 335.51782163899998, 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.93039029699999998, 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170625 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834138+0000", 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.881 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834138+0000", 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834138+0000", 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8100000000000001e-07 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834143+0000", 2024-02-05T08:19:22.882 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0929999999999996e-06 2024-02-05T08:19:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834158+0000", 2024-02-05T08:19:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5057e-05 2024-02-05T08:19:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834166+0000", 2024-02-05T08:19:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2390000000000002e-06 2024-02-05T08:19:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.883 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834213+0000", 2024-02-05T08:19:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7029e-05 2024-02-05T08:19:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:19:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834219+0000", 2024-02-05T08:19:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3319999999999997e-06 2024-02-05T08:19:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756569+0000", 2024-02-05T08:19:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.92234994100000001 2024-02-05T08:19:22.884 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756684+0000", 2024-02-05T08:19:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115053 2024-02-05T08:19:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:19:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756836+0000", 2024-02-05T08:19:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015174599999999999 2024-02-05T08:19:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:19:22.885 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759502+0000", 2024-02-05T08:19:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0026656649999999998 2024-02-05T08:19:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759930+0000", 2024-02-05T08:19:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00042810999999999999 2024-02-05T08:19:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:19:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764311+0000", 2024-02-05T08:19:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0043816990000000002 2024-02-05T08:19:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.886 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764492+0000", 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018032500000000001 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764528+0000", 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6127000000000002e-05 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:22.887 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:19:22.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:22 smithi134 ceph-mon[39278]: 6.2 scrub starts 2024-02-05T08:19:22.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:22 smithi134 ceph-mon[39278]: 6.2 scrub ok 2024-02-05T08:19:22.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:22 smithi134 ceph-mon[39278]: 5.1b scrub starts 2024-02-05T08:19:22.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:22 smithi134 ceph-mon[39278]: 5.1b scrub ok 2024-02-05T08:19:22.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:22 smithi134 ceph-mon[39278]: 6.18 scrub starts 2024-02-05T08:19:22.888 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:22 smithi134 ceph-mon[39278]: 6.18 scrub ok 2024-02-05T08:19:22.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:22 smithi196 ceph-mon[33018]: 6.2 scrub starts 2024-02-05T08:19:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:22 smithi196 ceph-mon[33018]: 6.2 scrub ok 2024-02-05T08:19:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:22 smithi196 ceph-mon[33018]: 5.1b scrub starts 2024-02-05T08:19:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:22 smithi196 ceph-mon[33018]: 5.1b scrub ok 2024-02-05T08:19:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:22 smithi196 ceph-mon[33018]: 6.18 scrub starts 2024-02-05T08:19:22.889 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:22 smithi196 ceph-mon[33018]: 6.18 scrub ok 2024-02-05T08:19:22.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:22.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:22.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:23.027 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:23.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:23.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:23.126 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 3 16 9955 9939 207.024 186.938 0.00600389 0.00481901 2024-02-05T08:19:23.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:23.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:23.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:23.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:23.346 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:23.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:23.415 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:19:23.448 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:23.487 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:23.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:23.589 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:23.624 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:23.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:23.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:23.721 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:23 smithi196 ceph-mon[33018]: pgmap v657: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-02-05T08:19:23.721 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:23 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1772917734' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:19:23.721 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:23 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:19:23.721 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:23 smithi196 ceph-mon[33018]: 4.10 scrub starts 2024-02-05T08:19:23.721 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:23 smithi196 ceph-mon[33018]: 4.10 scrub ok 2024-02-05T08:19:23.721 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:19:23.726 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:23.763 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:23.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:23.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:23.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:23 smithi134 ceph-mon[30086]: pgmap v657: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-02-05T08:19:23.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:23 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1772917734' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:19:23.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:23 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:19:23.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:23 smithi134 ceph-mon[30086]: 4.10 scrub starts 2024-02-05T08:19:23.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:23 smithi134 ceph-mon[30086]: 4.10 scrub ok 2024-02-05T08:19:23.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:19:23.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:23 smithi134 ceph-mon[39278]: pgmap v657: 158 pgs: 158 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 734 B/s rd, 0 op/s 2024-02-05T08:19:23.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:23 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1772917734' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:19:23.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:23 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:19:23.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:23 smithi134 ceph-mon[39278]: 4.10 scrub starts 2024-02-05T08:19:23.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:23 smithi134 ceph-mon[39278]: 4.10 scrub ok 2024-02-05T08:19:23.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:19:23.865 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:23.906 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:23.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:23.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:24.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:24.009 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T08:19:24.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:24.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:24.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:19:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:19:24.126 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 4 16 12966 12950 202.311 188.188 0.00422691 0.00493401 2024-02-05T08:19:24.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:24.183 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:24.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:24.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:24.321 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:24.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:24.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:24.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:24.467 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:24.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:24.509 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:24.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:24.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:24.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:24.709 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:24.749 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:24.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:24.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:24.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:24 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:19:24.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:24 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:19:24.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:24 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:19:24.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:24 smithi134 ceph-mon[30086]: osdmap e317: 8 total, 8 up, 8 in 2024-02-05T08:19:24.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:24 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:19:24.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:24 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:19:24.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:24 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:19:24.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:24 smithi134 ceph-mon[39278]: osdmap e317: 8 total, 8 up, 8 in 2024-02-05T08:19:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:24 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:19:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:24 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:19:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:24 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:19:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:24 smithi196 ceph-mon[33018]: osdmap e317: 8 total, 8 up, 8 in 2024-02-05T08:19:24.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:24.891 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:24.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:24.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:24.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:25.031 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:25.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:25.126 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 5 16 15631 15615 195.159 166.562 0.00490067 0.005116 2024-02-05T08:19:25.132 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:25.155 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:25.178 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:25.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:25.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:25.280 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:25.318 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:25.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:25.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:25.420 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:25.460 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:25.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:25.561 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:25.600 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:25.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:25.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:25.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:25.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:25.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:25.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:25.766 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:25 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:25.767 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:25 smithi134 ceph-mon[30086]: pgmap v659: 158 pgs: 158 active+clean; 509 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-02-05T08:19:25.767 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:25 smithi134 ceph-mon[30086]: 4.d scrub starts 2024-02-05T08:19:25.767 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:25 smithi134 ceph-mon[30086]: 4.d scrub ok 2024-02-05T08:19:25.767 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:25 smithi134 ceph-mon[30086]: 6.9 deep-scrub starts 2024-02-05T08:19:25.767 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:25 smithi134 ceph-mon[30086]: osdmap e318: 8 total, 8 up, 8 in 2024-02-05T08:19:25.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:25 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:25.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:25 smithi134 ceph-mon[39278]: pgmap v659: 158 pgs: 158 active+clean; 509 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-02-05T08:19:25.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:25 smithi134 ceph-mon[39278]: 4.d scrub starts 2024-02-05T08:19:25.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:25 smithi134 ceph-mon[39278]: 4.d scrub ok 2024-02-05T08:19:25.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:25 smithi134 ceph-mon[39278]: 6.9 deep-scrub starts 2024-02-05T08:19:25.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:25 smithi134 ceph-mon[39278]: osdmap e318: 8 total, 8 up, 8 in 2024-02-05T08:19:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:25 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:25 smithi196 ceph-mon[33018]: pgmap v659: 158 pgs: 158 active+clean; 509 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-02-05T08:19:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:25 smithi196 ceph-mon[33018]: 4.d scrub starts 2024-02-05T08:19:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:25 smithi196 ceph-mon[33018]: 4.d scrub ok 2024-02-05T08:19:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:25 smithi196 ceph-mon[33018]: 6.9 deep-scrub starts 2024-02-05T08:19:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:25 smithi196 ceph-mon[33018]: osdmap e318: 8 total, 8 up, 8 in 2024-02-05T08:19:25.840 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:25.877 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:25.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:25.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:25.994 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:26.013 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:26.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:26.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:26.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:26.126 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 6 16 18351 18335 190.965 170 0.00532877 0.0052313 2024-02-05T08:19:26.156 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:26.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:26.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:26.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:26.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:26.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:26.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:26.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:26.430 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:26.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:26.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:26.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:26.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:26.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:26.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:26.706 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:26.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:26.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:26.709 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:26 smithi134 ceph-mon[30086]: 6.9 deep-scrub ok 2024-02-05T08:19:26.709 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:26 smithi134 ceph-mon[30086]: 3.13 scrub starts 2024-02-05T08:19:26.709 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:26 smithi134 ceph-mon[30086]: 3.13 scrub ok 2024-02-05T08:19:26.709 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:26 smithi134 ceph-mon[30086]: 3.1f scrub starts 2024-02-05T08:19:26.709 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:26 smithi134 ceph-mon[30086]: 3.1f scrub ok 2024-02-05T08:19:26.710 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:26 smithi134 ceph-mon[39278]: 6.9 deep-scrub ok 2024-02-05T08:19:26.710 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:26 smithi134 ceph-mon[39278]: 3.13 scrub starts 2024-02-05T08:19:26.710 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:26 smithi134 ceph-mon[39278]: 3.13 scrub ok 2024-02-05T08:19:26.710 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:26 smithi134 ceph-mon[39278]: 3.1f scrub starts 2024-02-05T08:19:26.710 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:26 smithi134 ceph-mon[39278]: 3.1f scrub ok 2024-02-05T08:19:26.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:26 smithi196 ceph-mon[33018]: 6.9 deep-scrub ok 2024-02-05T08:19:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:26 smithi196 ceph-mon[33018]: 3.13 scrub starts 2024-02-05T08:19:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:26 smithi196 ceph-mon[33018]: 3.13 scrub ok 2024-02-05T08:19:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:26 smithi196 ceph-mon[33018]: 3.1f scrub starts 2024-02-05T08:19:26.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:26 smithi196 ceph-mon[33018]: 3.1f scrub ok 2024-02-05T08:19:26.847 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:26.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:26.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:26.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:26.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:26.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:27.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:27.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:27.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:27.126 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 7 15 20954 20939 186.932 162.75 0.00661556 0.00534259 2024-02-05T08:19:27.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:19:27.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:27.340 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:27.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:27.441 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:27.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:27.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:27.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:27 smithi134 ceph-mon[30086]: pgmap v661: 158 pgs: 158 active+clean; 602 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 690 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-02-05T08:19:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:27 smithi134 ceph-mon[30086]: 6.12 scrub starts 2024-02-05T08:19:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:27 smithi134 ceph-mon[30086]: 6.12 scrub ok 2024-02-05T08:19:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:27 smithi134 ceph-mon[30086]: 6.10 scrub starts 2024-02-05T08:19:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:27 smithi134 ceph-mon[30086]: 6.10 scrub ok 2024-02-05T08:19:27.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:27 smithi134 ceph-mon[39278]: pgmap v661: 158 pgs: 158 active+clean; 602 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 690 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-02-05T08:19:27.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:27 smithi134 ceph-mon[39278]: 6.12 scrub starts 2024-02-05T08:19:27.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:27 smithi134 ceph-mon[39278]: 6.12 scrub ok 2024-02-05T08:19:27.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:27 smithi134 ceph-mon[39278]: 6.10 scrub starts 2024-02-05T08:19:27.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:27 smithi134 ceph-mon[39278]: 6.10 scrub ok 2024-02-05T08:19:27.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:27.616 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:27.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:27.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:27.752 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:27.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:27.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:27 smithi196 ceph-mon[33018]: pgmap v661: 158 pgs: 158 active+clean; 602 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 690 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-02-05T08:19:27.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:27 smithi196 ceph-mon[33018]: 6.12 scrub starts 2024-02-05T08:19:27.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:27 smithi196 ceph-mon[33018]: 6.12 scrub ok 2024-02-05T08:19:27.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:27 smithi196 ceph-mon[33018]: 6.10 scrub starts 2024-02-05T08:19:27.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:27 smithi196 ceph-mon[33018]: 6.10 scrub ok 2024-02-05T08:19:27.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:27.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:27.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:27.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:27.992 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:28.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:28.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:19:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:19:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:19:28.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:28.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:28.126 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 8 16 23385 23369 182.545 151.875 0.00495554 0.00547239 2024-02-05T08:19:28.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:28.176 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:28.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:28.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:28.278 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:28.312 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:28.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:28.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:28.414 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:28.450 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:28.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:28.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:28.590 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:28.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:28.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:28.727 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:28.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:28.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:28 smithi134 ceph-mon[30086]: 4.13 scrub starts 2024-02-05T08:19:28.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:28 smithi134 ceph-mon[30086]: 4.13 scrub ok 2024-02-05T08:19:28.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:28 smithi134 ceph-mon[30086]: 6.1 scrub starts 2024-02-05T08:19:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:28 smithi134 ceph-mon[39278]: 4.13 scrub starts 2024-02-05T08:19:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:28 smithi134 ceph-mon[39278]: 4.13 scrub ok 2024-02-05T08:19:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:28 smithi134 ceph-mon[39278]: 6.1 scrub starts 2024-02-05T08:19:28.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:28 smithi196 ceph-mon[33018]: 4.13 scrub starts 2024-02-05T08:19:28.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:28 smithi196 ceph-mon[33018]: 4.13 scrub ok 2024-02-05T08:19:28.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:28 smithi196 ceph-mon[33018]: 6.1 scrub starts 2024-02-05T08:19:28.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:28.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:28.870 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:28.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:28.971 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:29.017 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:29.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:29.119 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:29.127 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 9 16 25803 25787 179.042 151.125 0.00410781 0.00557887 2024-02-05T08:19:29.158 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:29.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:29.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:29.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:29.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:29.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:29.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:29.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:29.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:29.577 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:29.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:29.679 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:29.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:29.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:29.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:29 smithi134 ceph-mon[30086]: pgmap v662: 158 pgs: 158 active+clean; 928 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-02-05T08:19:29.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:29 smithi134 ceph-mon[30086]: 6.1 scrub ok 2024-02-05T08:19:29.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:29 smithi134 ceph-mon[30086]: 6.17 deep-scrub starts 2024-02-05T08:19:29.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:29 smithi134 ceph-mon[30086]: 6.17 deep-scrub ok 2024-02-05T08:19:29.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:29 smithi134 ceph-mon[30086]: 3.19 scrub starts 2024-02-05T08:19:29.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:29 smithi134 ceph-mon[30086]: 3.19 scrub ok 2024-02-05T08:19:29.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:29 smithi134 ceph-mon[39278]: pgmap v662: 158 pgs: 158 active+clean; 928 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-02-05T08:19:29.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:29 smithi134 ceph-mon[39278]: 6.1 scrub ok 2024-02-05T08:19:29.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:29 smithi134 ceph-mon[39278]: 6.17 deep-scrub starts 2024-02-05T08:19:29.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:29 smithi134 ceph-mon[39278]: 6.17 deep-scrub ok 2024-02-05T08:19:29.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:29 smithi134 ceph-mon[39278]: 3.19 scrub starts 2024-02-05T08:19:29.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:29 smithi134 ceph-mon[39278]: 3.19 scrub ok 2024-02-05T08:19:29.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:29 smithi196 ceph-mon[33018]: pgmap v662: 158 pgs: 158 active+clean; 928 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-02-05T08:19:29.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:29 smithi196 ceph-mon[33018]: 6.1 scrub ok 2024-02-05T08:19:29.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:29 smithi196 ceph-mon[33018]: 6.17 deep-scrub starts 2024-02-05T08:19:29.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:29 smithi196 ceph-mon[33018]: 6.17 deep-scrub ok 2024-02-05T08:19:29.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:29 smithi196 ceph-mon[33018]: 3.19 scrub starts 2024-02-05T08:19:29.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:29 smithi196 ceph-mon[33018]: 3.19 scrub ok 2024-02-05T08:19:29.819 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:29.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:29.851 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:29.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:29.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:29.990 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:29.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:29.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:30.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:30.127 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 10 15 28126 28111 175.662 145.25 0.00675704 0.00568747 2024-02-05T08:19:30.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:30.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:30.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:30.267 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:30.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:30.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:30.407 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:30.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:30.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:30.539 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:30.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:30.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:30.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:30.679 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:30.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:30.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:30.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:30.820 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:30.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:30.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:30.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:30.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:30.953 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:30.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:31.054 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:31.092 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:31.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:31.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:31.127 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 11 15 30283 30268 171.948 134.812 0.00858779 0.00581086 2024-02-05T08:19:31.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:31.220 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:31.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:31.321 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:31.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:31.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:31.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:31.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:31.495 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:31.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:31.599 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:31.637 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:31.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:31.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:31.739 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:31.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:31 smithi134 ceph-mon[30086]: pgmap v663: 158 pgs: 158 active+clean; 1.4 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 181 MiB/s wr, 2.90k op/s 2024-02-05T08:19:31.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:31 smithi134 ceph-mon[30086]: 3.d deep-scrub starts 2024-02-05T08:19:31.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:31 smithi134 ceph-mon[30086]: 3.d deep-scrub ok 2024-02-05T08:19:31.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:31 smithi134 ceph-mon[30086]: 4.1 scrub starts 2024-02-05T08:19:31.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:31 smithi134 ceph-mon[30086]: 4.1 scrub ok 2024-02-05T08:19:31.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:31 smithi134 ceph-mon[39278]: pgmap v663: 158 pgs: 158 active+clean; 1.4 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 181 MiB/s wr, 2.90k op/s 2024-02-05T08:19:31.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:31 smithi134 ceph-mon[39278]: 3.d deep-scrub starts 2024-02-05T08:19:31.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:31 smithi134 ceph-mon[39278]: 3.d deep-scrub ok 2024-02-05T08:19:31.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:31 smithi134 ceph-mon[39278]: 4.1 scrub starts 2024-02-05T08:19:31.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:31 smithi134 ceph-mon[39278]: 4.1 scrub ok 2024-02-05T08:19:31.781 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:31.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:31 smithi196 ceph-mon[33018]: pgmap v663: 158 pgs: 158 active+clean; 1.4 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 181 MiB/s wr, 2.90k op/s 2024-02-05T08:19:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:31 smithi196 ceph-mon[33018]: 3.d deep-scrub starts 2024-02-05T08:19:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:31 smithi196 ceph-mon[33018]: 3.d deep-scrub ok 2024-02-05T08:19:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:31 smithi196 ceph-mon[33018]: 4.1 scrub starts 2024-02-05T08:19:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:31 smithi196 ceph-mon[33018]: 4.1 scrub ok 2024-02-05T08:19:31.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:31.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:31.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:32.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:32.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:32.061 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:32.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:32.127 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 12 16 32058 32042 166.858 110.875 0.00545105 0.00598843 2024-02-05T08:19:32.162 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:32.196 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:32.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:32.297 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:32.334 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:32.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:32.436 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:32.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:32.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:32.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:32.514 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:19:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:19:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15714.0:31201 10.2 10:4df2fc5f:::benchmark_data_smithi134_174861_object31200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-02-05T08:19:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:19:31.648709+0000", 2024-02-05T08:19:32.514 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.002670202, 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027055540000000002, 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15714", 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 31201 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.648709+0000", 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:32.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.648709+0000", 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.648711+0000", 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.328e-06 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.648789+0000", 2024-02-05T08:19:32.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8029999999999997e-05 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.648815+0000", 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6228e-05 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.648823+0000", 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0639999999999994e-06 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:19:32.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.649026+0000", 2024-02-05T08:19:32.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020326200000000001 2024-02-05T08:19:32.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.518 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:19:32.518 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.649228+0000", 2024-02-05T08:19:32.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020202800000000001 2024-02-05T08:19:32.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.518 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,4", 2024-02-05T08:19:32.518 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.649365+0000", 2024-02-05T08:19:32.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013633499999999999 2024-02-05T08:19:32.518 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15714.0:31206 10.2 10.5ecaf9e2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:19:31.650090+0000", 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.001289311, 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001403938, 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15714", 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:19:32.519 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 31206 2024-02-05T08:19:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.650090+0000", 2024-02-05T08:19:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.650090+0000", 2024-02-05T08:19:32.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.650091+0000", 2024-02-05T08:19:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1620000000000001e-06 2024-02-05T08:19:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.650143+0000", 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1752999999999997e-05 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.650161+0000", 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8159e-05 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.650187+0000", 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5378999999999999e-05 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:32.522 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.15714.0:31201 10.2 e319/309)", 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:19:31.651186+0000", 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.00019372200000000001, 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00036585799999999999, 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.651186+0000", 2024-02-05T08:19:32.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.651186+0000", 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.651187+0000", 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9899999999999999e-07 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:19:32.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.651192+0000", 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.378e-06 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.651208+0000", 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5557e-05 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:19:31.651216+0000", 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1410000000000005e-06 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:32.525 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:19:32.526 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:19:32.526 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 3 2024-02-05T08:19:32.526 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:19:32.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:32.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:32.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:32.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:32.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:32.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:32.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:32.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:19:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:19:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[30086]: 5.9 deep-scrub starts 2024-02-05T08:19:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[30086]: 5.9 deep-scrub ok 2024-02-05T08:19:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[30086]: osdmap e319: 8 total, 8 up, 8 in 2024-02-05T08:19:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[30086]: 4.16 scrub starts 2024-02-05T08:19:32.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[30086]: 4.16 scrub ok 2024-02-05T08:19:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:19:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:19:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[39278]: 5.9 deep-scrub starts 2024-02-05T08:19:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[39278]: 5.9 deep-scrub ok 2024-02-05T08:19:32.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[39278]: osdmap e319: 8 total, 8 up, 8 in 2024-02-05T08:19:32.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[39278]: 4.16 scrub starts 2024-02-05T08:19:32.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:32 smithi134 ceph-mon[39278]: 4.16 scrub ok 2024-02-05T08:19:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:32 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:19:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:32 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:19:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:32 smithi196 ceph-mon[33018]: 5.9 deep-scrub starts 2024-02-05T08:19:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:32 smithi196 ceph-mon[33018]: 5.9 deep-scrub ok 2024-02-05T08:19:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:32 smithi196 ceph-mon[33018]: osdmap e319: 8 total, 8 up, 8 in 2024-02-05T08:19:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:32 smithi196 ceph-mon[33018]: 4.16 scrub starts 2024-02-05T08:19:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:32 smithi196 ceph-mon[33018]: 4.16 scrub ok 2024-02-05T08:19:32.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:32.891 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:32.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:32.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:32.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:33.029 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:33.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:33.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:33.127 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 13 16 34151 34135 164.084 130.812 0.0089238 0.00608875 2024-02-05T08:19:33.132 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:33.170 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:33.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:33.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:33.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:33.307 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:33.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:33.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:33.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:33.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:33.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:33.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:33.593 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:33.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:33.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:33.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:33.715 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[30086]: pgmap v665: 158 pgs: 158 active+clean; 1.4 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-02-05T08:19:33.715 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[30086]: 3.1d deep-scrub starts 2024-02-05T08:19:33.716 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[30086]: 3.1d deep-scrub ok 2024-02-05T08:19:33.716 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[30086]: 5.1e scrub starts 2024-02-05T08:19:33.716 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[30086]: 5.1e scrub ok 2024-02-05T08:19:33.716 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:19:33.716 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:19:33.716 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:19:33.716 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:19:33.716 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[39278]: pgmap v665: 158 pgs: 158 active+clean; 1.4 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-02-05T08:19:33.716 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[39278]: 3.1d deep-scrub starts 2024-02-05T08:19:33.716 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[39278]: 3.1d deep-scrub ok 2024-02-05T08:19:33.716 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[39278]: 5.1e scrub starts 2024-02-05T08:19:33.716 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[39278]: 5.1e scrub ok 2024-02-05T08:19:33.717 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:19:33.717 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:19:33.717 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:19:33.717 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:33 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:19:33.728 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:33.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:33.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:33.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:19:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:19:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:33 smithi196 ceph-mon[33018]: pgmap v665: 158 pgs: 158 active+clean; 1.4 GiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 117 MiB/s wr, 1.88k op/s 2024-02-05T08:19:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:33 smithi196 ceph-mon[33018]: 3.1d deep-scrub starts 2024-02-05T08:19:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:33 smithi196 ceph-mon[33018]: 3.1d deep-scrub ok 2024-02-05T08:19:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:33 smithi196 ceph-mon[33018]: 5.1e scrub starts 2024-02-05T08:19:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:33 smithi196 ceph-mon[33018]: 5.1e scrub ok 2024-02-05T08:19:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:33 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:19:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:33 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:19:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:33 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:19:33.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:33 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:19:33.830 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:33.867 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:33.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:33.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:33.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:34.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:34.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:34.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:34.105 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:34.127 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 14 16 36097 36081 161.048 121.625 0.00555782 0.00620293 2024-02-05T08:19:34.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:34.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:34.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:34.225 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:19:34.234 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:19:34.251 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:34.294 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:34.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:34.396 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:34.434 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:34.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:34.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:34.572 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T08:19:34.586 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:34.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:34.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:34.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:34.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:34.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:34.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:34.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:34 smithi134 ceph-mon[30086]: osdmap e320: 8 total, 8 up, 8 in 2024-02-05T08:19:34.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:34 smithi134 ceph-mon[30086]: 5.3 scrub starts 2024-02-05T08:19:34.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:34 smithi134 ceph-mon[30086]: 5.3 scrub ok 2024-02-05T08:19:34.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:34 smithi134 ceph-mon[39278]: osdmap e320: 8 total, 8 up, 8 in 2024-02-05T08:19:34.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:34 smithi134 ceph-mon[39278]: 5.3 scrub starts 2024-02-05T08:19:34.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:34 smithi134 ceph-mon[39278]: 5.3 scrub ok 2024-02-05T08:19:34.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:34 smithi196 ceph-mon[33018]: osdmap e320: 8 total, 8 up, 8 in 2024-02-05T08:19:34.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:34 smithi196 ceph-mon[33018]: 5.3 scrub starts 2024-02-05T08:19:34.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:34 smithi196 ceph-mon[33018]: 5.3 scrub ok 2024-02-05T08:19:34.827 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:34.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:34.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:34.953 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:19:34.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:34.997 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:19:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:35.018 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:35.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:35.040 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:19:35.083 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:19:35.098 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:35.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:35.127 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:19:35.128 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 15 16 37923 37907 157.915 114.125 0.0129033 0.00632661 2024-02-05T08:19:35.151 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:35.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:35.166 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:19:35.214 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:19:35.252 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:19:35.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:35.256 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:19:35.256 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:19:35.256 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:19:35.256 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:19:35.256 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:19:35.256 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:19:35.256 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:19:35.256 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:19:35.290 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:35.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:35.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:35.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:35.431 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:35.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:35.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:35.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:35.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:35.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:35.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:35.667 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:35.706 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:35.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:35.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:35.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[30086]: pgmap v667: 158 pgs: 158 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 180 MiB/s wr, 2.89k op/s 2024-02-05T08:19:35.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[30086]: from='client.25568 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:19:35.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[30086]: 4.8 deep-scrub starts 2024-02-05T08:19:35.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[30086]: 4.8 deep-scrub ok 2024-02-05T08:19:35.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[30086]: 4.1f scrub starts 2024-02-05T08:19:35.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[30086]: 4.1f scrub ok 2024-02-05T08:19:35.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[30086]: 3.17 deep-scrub starts 2024-02-05T08:19:35.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[30086]: 3.17 deep-scrub ok 2024-02-05T08:19:35.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2489523606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:35.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[30086]: 5.0 scrub starts 2024-02-05T08:19:35.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[30086]: 5.0 scrub ok 2024-02-05T08:19:35.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:35.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[39278]: pgmap v667: 158 pgs: 158 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 180 MiB/s wr, 2.89k op/s 2024-02-05T08:19:35.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[39278]: from='client.25568 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:19:35.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[39278]: 4.8 deep-scrub starts 2024-02-05T08:19:35.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[39278]: 4.8 deep-scrub ok 2024-02-05T08:19:35.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[39278]: 4.1f scrub starts 2024-02-05T08:19:35.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[39278]: 4.1f scrub ok 2024-02-05T08:19:35.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[39278]: 3.17 deep-scrub starts 2024-02-05T08:19:35.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[39278]: 3.17 deep-scrub ok 2024-02-05T08:19:35.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2489523606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:35.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[39278]: 5.0 scrub starts 2024-02-05T08:19:35.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:35 smithi134 ceph-mon[39278]: 5.0 scrub ok 2024-02-05T08:19:35.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:35 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:35 smithi196 ceph-mon[33018]: pgmap v667: 158 pgs: 158 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 180 MiB/s wr, 2.89k op/s 2024-02-05T08:19:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:35 smithi196 ceph-mon[33018]: from='client.25568 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:19:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:35 smithi196 ceph-mon[33018]: 4.8 deep-scrub starts 2024-02-05T08:19:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:35 smithi196 ceph-mon[33018]: 4.8 deep-scrub ok 2024-02-05T08:19:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:35 smithi196 ceph-mon[33018]: 4.1f scrub starts 2024-02-05T08:19:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:35 smithi196 ceph-mon[33018]: 4.1f scrub ok 2024-02-05T08:19:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:35 smithi196 ceph-mon[33018]: 3.17 deep-scrub starts 2024-02-05T08:19:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:35 smithi196 ceph-mon[33018]: 3.17 deep-scrub ok 2024-02-05T08:19:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:35 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2489523606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:35 smithi196 ceph-mon[33018]: 5.0 scrub starts 2024-02-05T08:19:35.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:35 smithi196 ceph-mon[33018]: 5.0 scrub ok 2024-02-05T08:19:35.844 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:35.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:35.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:35.982 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:35.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:36.047 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":667,"stamp":"2024-02-05T08:19:33.781108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2146113884,"num_objects":32936,"num_object_clones":0,"num_object_copies":98808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32936,"num_whiteouts":0,"num_read":2020,"num_read_kb":1772,"num_write":33460,"num_write_kb":2095277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":7802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35030,"ondisk_log_size":35030,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9863688,"kb_used_data":6017200,"kb_used_omap":12,"kb_used_meta":3846387,"kb_avail":740097528,"statfs":{"total":767960285184,"available":757859868672,"internally_reserved":0,"allocated":6161612800,"data_stored":6158325800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":3938700851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,8,11,7,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1514864640,"num_objects":23115,"num_object_clones":0,"num_object_copies":69345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23115,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":23115,"num_write_kb":1479360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004260"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":555,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.485626+0000","last_change":"2024-02-05T08:17:14.644253+0000","last_active":"2024-02-05T08:19:31.485626+0000","last_peered":"2024-02-05T08:19:31.485626+0000","last_clean":"2024-02-05T08:19:31.485626+0000","last_became_active":"2024-02-05T08:04:47.578602+0000","last_became_peered":"2024-02-05T08:04:47.578602+0000","last_unstale":"2024-02-05T08:19:31.485626+0000","last_undegraded":"2024-02-05T08:19:31.485626+0000","last_fullsized":"2024-02-05T08:19:31.485626+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:14.644144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:14.644144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":569,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.480770+0000","last_change":"2024-02-05T08:18:34.301578+0000","last_active":"2024-02-05T08:19:31.480770+0000","last_peered":"2024-02-05T08:19:31.480770+0000","last_clean":"2024-02-05T08:19:31.480770+0000","last_became_active":"2024-02-05T08:04:45.560363+0000","last_became_peered":"2024-02-05T08:04:45.560363+0000","last_unstale":"2024-02-05T08:19:31.480770+0000","last_undegraded":"2024-02-05T08:19:31.480770+0000","last_fullsized":"2024-02-05T08:19:31.480770+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.301502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.301502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:37.136866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":566,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:25.569580+0000","last_change":"2024-02-05T08:19:25.569614+0000","last_active":"2024-02-05T08:19:25.569580+0000","last_peered":"2024-02-05T08:19:25.569580+0000","last_clean":"2024-02-05T08:19:25.569580+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:19:25.569580+0000","last_undegraded":"2024-02-05T08:19:25.569580+0000","last_fullsized":"2024-02-05T08:19:25.569580+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:25.569504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:25.569504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:27.084963+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"236'19","reported_seq":625,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.485376+0000","last_change":"2024-02-05T08:18:01.603030+0000","last_active":"2024-02-05T08:19:31.485376+0000","last_peered":"2024-02-05T08:19:31.485376+0000","last_clean":"2024-02-05T08:19:31.485376+0000","last_became_active":"2024-02-05T08:12:29.675863+0000","last_became_peered":"2024-02-05T08:12:29.675863+0000","last_unstale":"2024-02-05T08:19:31.485376+0000","last_undegraded":"2024-02-05T08:19:31.485376+0000","last_fullsized":"2024-02-05T08:19:31.485376+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:01.602887+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:18:01.602887+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":59,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.492172+0000","last_change":"2024-02-05T08:18:34.633406+0000","last_active":"2024-02-05T08:19:31.492172+0000","last_peered":"2024-02-05T08:19:31.492172+0000","last_clean":"2024-02-05T08:19:31.492172+0000","last_became_active":"2024-02-05T08:18:34.633083+0000","last_became_peered":"2024-02-05T08:18:34.633083+0000","last_unstale":"2024-02-05T08:19:31.492172+0000","last_undegraded":"2024-02-05T08:19:31.492172+0000","last_fullsized":"2024-02-05T08:19:31.492172+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:07.163073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:18:07.163073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":572,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.487810+0000","last_change":"2024-02-05T08:18:39.536133+0000","last_active":"2024-02-05T08:19:31.487810+0000","last_peered":"2024-02-05T08:19:31.487810+0000","last_clean":"2024-02-05T08:19:31.487810+0000","last_became_active":"2024-02-05T08:12:29.676482+0000","last_became_peered":"2024-02-05T08:12:29.676482+0000","last_unstale":"2024-02-05T08:19:31.487810+0000","last_undegraded":"2024-02-05T08:19:31.487810+0000","last_fullsized":"2024-02-05T08:19:31.487810+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:39.536022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:18:39.536022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:45.419854+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":67,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.492317+0000","last_change":"2024-02-05T08:18:47.184065+0000","last_active":"2024-02-05T08:19:31.492317+0000","last_peered":"2024-02-05T08:19:31.492317+0000","last_clean":"2024-02-05T08:19:31.492317+0000","last_became_active":"2024-02-05T08:18:34.667678+0000","last_became_peered":"2024-02-05T08:18:34.667678+0000","last_unstale":"2024-02-05T08:19:31.492317+0000","last_undegraded":"2024-02-05T08:19:31.492317+0000","last_fullsized":"2024-02-05T08:19:31.492317+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:47.183986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:18:47.183986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:04.700552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"236'34","reported_seq":83,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491294+0000","last_change":"2024-02-05T08:19:16.092816+0000","last_active":"2024-02-05T08:19:31.491294+0000","last_peered":"2024-02-05T08:19:31.491294+0000","last_clean":"2024-02-05T08:19:31.491294+0000","last_became_active":"2024-02-05T08:18:34.649433+0000","last_became_peered":"2024-02-05T08:18:34.649433+0000","last_unstale":"2024-02-05T08:19:31.491294+0000","last_undegraded":"2024-02-05T08:19:31.491294+0000","last_fullsized":"2024-02-05T08:19:31.491294+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:19:16.092725+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.092725+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:24.892477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":553,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.809269+0000","last_change":"2024-02-05T08:19:21.662047+0000","last_active":"2024-02-05T08:19:24.809269+0000","last_peered":"2024-02-05T08:19:24.809269+0000","last_clean":"2024-02-05T08:19:24.809269+0000","last_became_active":"2024-02-05T08:04:47.581312+0000","last_became_peered":"2024-02-05T08:04:47.581312+0000","last_unstale":"2024-02-05T08:19:24.809269+0000","last_undegraded":"2024-02-05T08:19:24.809269+0000","last_fullsized":"2024-02-05T08:19:24.809269+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:21.661930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:21.661930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:36.451090+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":567,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.494614+0000","last_change":"2024-02-05T08:19:21.500610+0000","last_active":"2024-02-05T08:19:31.494614+0000","last_peered":"2024-02-05T08:19:31.494614+0000","last_clean":"2024-02-05T08:19:31.494614+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:19:31.494614+0000","last_undegraded":"2024-02-05T08:19:31.494614+0000","last_fullsized":"2024-02-05T08:19:31.494614+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:21.500492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:19:21.500492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:37.507536+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":595,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:32.421471+0000","last_change":"2024-02-05T08:19:32.421512+0000","last_active":"2024-02-05T08:19:32.421471+0000","last_peered":"2024-02-05T08:19:32.421471+0000","last_clean":"2024-02-05T08:19:32.421471+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:19:32.421471+0000","last_undegraded":"2024-02-05T08:19:32.421471+0000","last_fullsized":"2024-02-05T08:19:32.421471+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_clean_scrub_stamp":"2024-02-05T08:19:32.421388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:42.025451+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"263'19","reported_seq":630,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.485894+0000","last_change":"2024-02-05T08:18:34.732396+0000","last_active":"2024-02-05T08:19:31.485894+0000","last_peered":"2024-02-05T08:19:31.485894+0000","last_clean":"2024-02-05T08:19:31.485894+0000","last_became_active":"2024-02-05T08:18:34.636165+0000","last_became_peered":"2024-02-05T08:18:34.636165+0000","last_unstale":"2024-02-05T08:19:31.485894+0000","last_undegraded":"2024-02-05T08:19:31.485894+0000","last_fullsized":"2024-02-05T08:19:31.485894+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:18:06.570361+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:18:06.570361+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:33.261052+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":568,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491001+0000","last_change":"2024-02-05T08:18:34.635235+0000","last_active":"2024-02-05T08:19:31.491001+0000","last_peered":"2024-02-05T08:19:31.491001+0000","last_clean":"2024-02-05T08:19:31.491001+0000","last_became_active":"2024-02-05T08:18:34.634934+0000","last_became_peered":"2024-02-05T08:18:34.634934+0000","last_unstale":"2024-02-05T08:19:31.491001+0000","last_undegraded":"2024-02-05T08:19:31.491001+0000","last_fullsized":"2024-02-05T08:19:31.491001+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:32.611856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:32.611856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:48.401031+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":560,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.482173+0000","last_change":"2024-02-05T08:18:04.094625+0000","last_active":"2024-02-05T08:19:31.482173+0000","last_peered":"2024-02-05T08:19:31.482173+0000","last_clean":"2024-02-05T08:19:31.482173+0000","last_became_active":"2024-02-05T08:04:45.562510+0000","last_became_peered":"2024-02-05T08:04:45.562510+0000","last_unstale":"2024-02-05T08:19:31.482173+0000","last_undegraded":"2024-02-05T08:19:31.482173+0000","last_fullsized":"2024-02-05T08:19:31.482173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:04.094513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:04.094513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":575,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490442+0000","last_change":"2024-02-05T08:19:14.458299+0000","last_active":"2024-02-05T08:19:31.490442+0000","last_peered":"2024-02-05T08:19:31.490442+0000","last_clean":"2024-02-05T08:19:31.490442+0000","last_became_active":"2024-02-05T08:04:41.540561+0000","last_became_peered":"2024-02-05T08:04:41.540561+0000","last_unstale":"2024-02-05T08:19:31.490442+0000","last_undegraded":"2024-02-05T08:19:31.490442+0000","last_fullsized":"2024-02-05T08:19:31.490442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:14.458198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.458198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:25.111108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":600,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.486190+0000","last_change":"2024-02-05T08:18:34.722007+0000","last_active":"2024-02-05T08:19:31.486190+0000","last_peered":"2024-02-05T08:19:31.486190+0000","last_clean":"2024-02-05T08:19:31.486190+0000","last_became_active":"2024-02-05T08:18:34.636037+0000","last_became_peered":"2024-02-05T08:18:34.636037+0000","last_unstale":"2024-02-05T08:19:31.486190+0000","last_undegraded":"2024-02-05T08:19:31.486190+0000","last_fullsized":"2024-02-05T08:19:31.486190+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-02-05T08:18:05.582303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:05.582303+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":569,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.489111+0000","last_change":"2024-02-05T08:17:51.825665+0000","last_active":"2024-02-05T08:19:31.489111+0000","last_peered":"2024-02-05T08:19:31.489111+0000","last_clean":"2024-02-05T08:19:31.489111+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:19:31.489111+0000","last_undegraded":"2024-02-05T08:19:31.489111+0000","last_fullsized":"2024-02-05T08:19:31.489111+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:51.825345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:17:51.825345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":560,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.483326+0000","last_change":"2024-02-05T08:18:35.317828+0000","last_active":"2024-02-05T08:19:31.483326+0000","last_peered":"2024-02-05T08:19:31.483326+0000","last_clean":"2024-02-05T08:19:31.483326+0000","last_became_active":"2024-02-05T08:04:45.559572+0000","last_became_peered":"2024-02-05T08:04:45.559572+0000","last_unstale":"2024-02-05T08:19:31.483326+0000","last_undegraded":"2024-02-05T08:19:31.483326+0000","last_fullsized":"2024-02-05T08:19:31.483326+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:35.317741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:48.942873+0000","last_clean_scrub_stamp":"2024-02-05T08:18:35.317741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:35.854280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":556,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.809456+0000","last_change":"2024-02-05T08:18:44.443095+0000","last_active":"2024-02-05T08:19:24.809456+0000","last_peered":"2024-02-05T08:19:24.809456+0000","last_clean":"2024-02-05T08:19:24.809456+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:19:24.809456+0000","last_undegraded":"2024-02-05T08:19:24.809456+0000","last_fullsized":"2024-02-05T08:19:24.809456+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:44.443012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:18:44.443012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:13.155368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"236'34","reported_seq":681,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490253+0000","last_change":"2024-02-05T08:18:47.576343+0000","last_active":"2024-02-05T08:19:31.490253+0000","last_peered":"2024-02-05T08:19:31.490253+0000","last_clean":"2024-02-05T08:19:31.490253+0000","last_became_active":"2024-02-05T08:18:34.669030+0000","last_became_peered":"2024-02-05T08:18:34.669030+0000","last_unstale":"2024-02-05T08:19:31.490253+0000","last_undegraded":"2024-02-05T08:19:31.490253+0000","last_fullsized":"2024-02-05T08:19:31.490253+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:18:47.576225+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:18:47.576225+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:54.384107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":59,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.494060+0000","last_change":"2024-02-05T08:18:34.642751+0000","last_active":"2024-02-05T08:19:31.494060+0000","last_peered":"2024-02-05T08:19:31.494060+0000","last_clean":"2024-02-05T08:19:31.494060+0000","last_became_active":"2024-02-05T08:18:34.642494+0000","last_became_peered":"2024-02-05T08:18:34.642494+0000","last_unstale":"2024-02-05T08:19:31.494060+0000","last_undegraded":"2024-02-05T08:19:31.494060+0000","last_fullsized":"2024-02-05T08:19:31.494060+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_clean_scrub_stamp":"2024-02-05T08:18:04.435264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":579,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.487966+0000","last_change":"2024-02-05T08:19:12.320431+0000","last_active":"2024-02-05T08:19:31.487966+0000","last_peered":"2024-02-05T08:19:31.487966+0000","last_clean":"2024-02-05T08:19:31.487966+0000","last_became_active":"2024-02-05T08:18:34.638652+0000","last_became_peered":"2024-02-05T08:18:34.638652+0000","last_unstale":"2024-02-05T08:19:31.487966+0000","last_undegraded":"2024-02-05T08:19:31.487966+0000","last_fullsized":"2024-02-05T08:19:31.487966+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:12.320317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:19:12.320317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:41.298516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":567,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.486041+0000","last_change":"2024-02-05T08:19:19.338812+0000","last_active":"2024-02-05T08:19:31.486041+0000","last_peered":"2024-02-05T08:19:31.486041+0000","last_clean":"2024-02-05T08:19:31.486041+0000","last_became_active":"2024-02-05T08:04:41.533994+0000","last_became_peered":"2024-02-05T08:04:41.533994+0000","last_unstale":"2024-02-05T08:19:31.486041+0000","last_undegraded":"2024-02-05T08:19:31.486041+0000","last_fullsized":"2024-02-05T08:19:31.486041+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:19.338699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:19:19.338699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:37.639968+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"236'23","reported_seq":71,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.493449+0000","last_change":"2024-02-05T08:18:34.763568+0000","last_active":"2024-02-05T08:19:31.493449+0000","last_peered":"2024-02-05T08:19:31.493449+0000","last_clean":"2024-02-05T08:19:31.493449+0000","last_became_active":"2024-02-05T08:18:34.663801+0000","last_became_peered":"2024-02-05T08:18:34.663801+0000","last_unstale":"2024-02-05T08:19:31.493449+0000","last_undegraded":"2024-02-05T08:19:31.493449+0000","last_fullsized":"2024-02-05T08:19:31.493449+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'23","last_scrub_stamp":"2024-02-05T08:17:58.012827+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:17:58.012827+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":574,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.487243+0000","last_change":"2024-02-05T08:19:20.247539+0000","last_active":"2024-02-05T08:19:31.487243+0000","last_peered":"2024-02-05T08:19:31.487243+0000","last_clean":"2024-02-05T08:19:31.487243+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:19:31.487243+0000","last_undegraded":"2024-02-05T08:19:31.487243+0000","last_fullsized":"2024-02-05T08:19:31.487243+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:19:20.247396+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:19:20.247396+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:23.096114+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":395,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.488444+0000","last_change":"2024-02-05T08:18:48.334093+0000","last_active":"2024-02-05T08:19:31.488444+0000","last_peered":"2024-02-05T08:19:31.488444+0000","last_clean":"2024-02-05T08:19:31.488444+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:19:31.488444+0000","last_undegraded":"2024-02-05T08:19:31.488444+0000","last_fullsized":"2024-02-05T08:19:31.488444+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:18:48.334005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:48.334005+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:58.800318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":598,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.480994+0000","last_change":"2024-02-05T08:19:28.743197+0000","last_active":"2024-02-05T08:19:31.480994+0000","last_peered":"2024-02-05T08:19:31.480994+0000","last_clean":"2024-02-05T08:19:31.480994+0000","last_became_active":"2024-02-05T08:18:34.657282+0000","last_became_peered":"2024-02-05T08:18:34.657282+0000","last_unstale":"2024-02-05T08:19:31.480994+0000","last_undegraded":"2024-02-05T08:19:31.480994+0000","last_fullsized":"2024-02-05T08:19:31.480994+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:19:28.743054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:28.743054+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:49.165986+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"236'20","reported_seq":656,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.811350+0000","last_change":"2024-02-05T08:18:34.714282+0000","last_active":"2024-02-05T08:19:24.811350+0000","last_peered":"2024-02-05T08:19:24.811350+0000","last_clean":"2024-02-05T08:19:24.811350+0000","last_became_active":"2024-02-05T08:18:34.645441+0000","last_became_peered":"2024-02-05T08:18:34.645441+0000","last_unstale":"2024-02-05T08:19:24.811350+0000","last_undegraded":"2024-02-05T08:19:24.811350+0000","last_fullsized":"2024-02-05T08:19:24.811350+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:17:52.347554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:17:52.347554+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":564,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.483464+0000","last_change":"2024-02-05T08:18:46.551083+0000","last_active":"2024-02-05T08:19:31.483464+0000","last_peered":"2024-02-05T08:19:31.483464+0000","last_clean":"2024-02-05T08:19:31.483464+0000","last_became_active":"2024-02-05T08:04:47.577895+0000","last_became_peered":"2024-02-05T08:04:47.577895+0000","last_unstale":"2024-02-05T08:19:31.483464+0000","last_undegraded":"2024-02-05T08:19:31.483464+0000","last_fullsized":"2024-02-05T08:19:31.483464+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:46.550985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:46.550985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:00.817188+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":550,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.811581+0000","last_change":"2024-02-05T08:16:57.872130+0000","last_active":"2024-02-05T08:19:24.811581+0000","last_peered":"2024-02-05T08:19:24.811581+0000","last_clean":"2024-02-05T08:19:24.811581+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:19:24.811581+0000","last_undegraded":"2024-02-05T08:19:24.811581+0000","last_fullsized":"2024-02-05T08:19:24.811581+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:16:57.871993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:32.231861+0000","last_clean_scrub_stamp":"2024-02-05T08:16:57.871993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":67,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490760+0000","last_change":"2024-02-05T08:19:11.049585+0000","last_active":"2024-02-05T08:19:31.490760+0000","last_peered":"2024-02-05T08:19:31.490760+0000","last_clean":"2024-02-05T08:19:31.490760+0000","last_became_active":"2024-02-05T08:18:34.653773+0000","last_became_peered":"2024-02-05T08:18:34.653773+0000","last_unstale":"2024-02-05T08:19:31.490760+0000","last_undegraded":"2024-02-05T08:19:31.490760+0000","last_fullsized":"2024-02-05T08:19:31.490760+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:11.049460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:19:11.049460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:16.817492+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"236'19","reported_seq":322,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.488238+0000","last_change":"2024-02-05T08:18:08.410519+0000","last_active":"2024-02-05T08:19:31.488238+0000","last_peered":"2024-02-05T08:19:31.488238+0000","last_clean":"2024-02-05T08:19:31.488238+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:19:31.488238+0000","last_undegraded":"2024-02-05T08:19:31.488238+0000","last_fullsized":"2024-02-05T08:19:31.488238+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:08.410382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:08.410382+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:34.374145+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":557,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.482352+0000","last_change":"2024-02-05T08:19:26.157804+0000","last_active":"2024-02-05T08:19:31.482352+0000","last_peered":"2024-02-05T08:19:31.482352+0000","last_clean":"2024-02-05T08:19:31.482352+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:19:31.482352+0000","last_undegraded":"2024-02-05T08:19:31.482352+0000","last_fullsized":"2024-02-05T08:19:31.482352+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:26.157698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:26.157698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:30.664394+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":301,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.487024+0000","last_change":"2024-02-05T08:19:09.371059+0000","last_active":"2024-02-05T08:19:31.487024+0000","last_peered":"2024-02-05T08:19:31.487024+0000","last_clean":"2024-02-05T08:19:31.487024+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:19:31.487024+0000","last_undegraded":"2024-02-05T08:19:31.487024+0000","last_fullsized":"2024-02-05T08:19:31.487024+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:09.370984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:19:09.370984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:24.611202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":569,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.810168+0000","last_change":"2024-02-05T08:18:34.644923+0000","last_active":"2024-02-05T08:19:24.810168+0000","last_peered":"2024-02-05T08:19:24.810168+0000","last_clean":"2024-02-05T08:19:24.810168+0000","last_became_active":"2024-02-05T08:18:34.644537+0000","last_became_peered":"2024-02-05T08:18:34.644537+0000","last_unstale":"2024-02-05T08:19:24.810168+0000","last_undegraded":"2024-02-05T08:19:24.810168+0000","last_fullsized":"2024-02-05T08:19:24.810168+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:48.387323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:48.387323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":73,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491022+0000","last_change":"2024-02-05T08:19:23.023668+0000","last_active":"2024-02-05T08:19:31.491022+0000","last_peered":"2024-02-05T08:19:31.491022+0000","last_clean":"2024-02-05T08:19:31.491022+0000","last_became_active":"2024-02-05T08:18:34.654438+0000","last_became_peered":"2024-02-05T08:18:34.654438+0000","last_unstale":"2024-02-05T08:19:31.491022+0000","last_undegraded":"2024-02-05T08:19:31.491022+0000","last_fullsized":"2024-02-05T08:19:31.491022+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:19:23.023541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:23.023541+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:35.700062+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":67,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.492089+0000","last_change":"2024-02-05T08:18:40.134914+0000","last_active":"2024-02-05T08:19:31.492089+0000","last_peered":"2024-02-05T08:19:31.492089+0000","last_clean":"2024-02-05T08:19:31.492089+0000","last_became_active":"2024-02-05T08:18:34.651979+0000","last_became_peered":"2024-02-05T08:18:34.651979+0000","last_unstale":"2024-02-05T08:19:31.492089+0000","last_undegraded":"2024-02-05T08:19:31.492089+0000","last_fullsized":"2024-02-05T08:19:31.492089+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:40.134574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:03.937759+0000","last_clean_scrub_stamp":"2024-02-05T08:18:40.134574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:55.004963+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":581,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.482989+0000","last_change":"2024-02-05T08:18:38.014240+0000","last_active":"2024-02-05T08:19:31.482989+0000","last_peered":"2024-02-05T08:19:31.482989+0000","last_clean":"2024-02-05T08:19:31.482989+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:19:31.482989+0000","last_undegraded":"2024-02-05T08:19:31.482989+0000","last_fullsized":"2024-02-05T08:19:31.482989+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:38.014162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:18:38.014162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:40.648191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":576,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.492019+0000","last_change":"2024-02-05T08:18:41.576956+0000","last_active":"2024-02-05T08:19:31.492019+0000","last_peered":"2024-02-05T08:19:31.492019+0000","last_clean":"2024-02-05T08:19:31.492019+0000","last_became_active":"2024-02-05T08:04:41.541871+0000","last_became_peered":"2024-02-05T08:04:41.541871+0000","last_unstale":"2024-02-05T08:19:31.492019+0000","last_undegraded":"2024-02-05T08:19:31.492019+0000","last_fullsized":"2024-02-05T08:19:31.492019+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:41.576882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:18:41.576882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:00.332588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"236'24","reported_seq":71,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491587+0000","last_change":"2024-02-05T08:18:34.675763+0000","last_active":"2024-02-05T08:19:31.491587+0000","last_peered":"2024-02-05T08:19:31.491587+0000","last_clean":"2024-02-05T08:19:31.491587+0000","last_became_active":"2024-02-05T08:18:34.631912+0000","last_became_peered":"2024-02-05T08:18:34.631912+0000","last_unstale":"2024-02-05T08:19:31.491587+0000","last_undegraded":"2024-02-05T08:19:31.491587+0000","last_fullsized":"2024-02-05T08:19:31.491587+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:18:33.096912+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:18:33.096912+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:49.851211+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":553,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:26.593445+0000","last_change":"2024-02-05T08:19:26.593483+0000","last_active":"2024-02-05T08:19:26.593445+0000","last_peered":"2024-02-05T08:19:26.593445+0000","last_clean":"2024-02-05T08:19:26.593445+0000","last_became_active":"2024-02-05T08:04:47.582080+0000","last_became_peered":"2024-02-05T08:04:47.582080+0000","last_unstale":"2024-02-05T08:19:26.593445+0000","last_undegraded":"2024-02-05T08:19:26.593445+0000","last_fullsized":"2024-02-05T08:19:26.593445+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:26.593364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:19:26.593364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:29.325065+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":67,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.492008+0000","last_change":"2024-02-05T08:18:44.134006+0000","last_active":"2024-02-05T08:19:31.492008+0000","last_peered":"2024-02-05T08:19:31.492008+0000","last_clean":"2024-02-05T08:19:31.492008+0000","last_became_active":"2024-02-05T08:18:34.652065+0000","last_became_peered":"2024-02-05T08:18:34.652065+0000","last_unstale":"2024-02-05T08:19:31.492008+0000","last_undegraded":"2024-02-05T08:19:31.492008+0000","last_fullsized":"2024-02-05T08:19:31.492008+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:44.133926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:44.133926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:45.216555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":581,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.484031+0000","last_change":"2024-02-05T08:18:34.642703+0000","last_active":"2024-02-05T08:19:31.484031+0000","last_peered":"2024-02-05T08:19:31.484031+0000","last_clean":"2024-02-05T08:19:31.484031+0000","last_became_active":"2024-02-05T08:18:34.642387+0000","last_became_peered":"2024-02-05T08:18:34.642387+0000","last_unstale":"2024-02-05T08:19:31.484031+0000","last_undegraded":"2024-02-05T08:19:31.484031+0000","last_fullsized":"2024-02-05T08:19:31.484031+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:58.070086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:58.070086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"263'19","reported_seq":236,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491251+0000","last_change":"2024-02-05T08:18:42.566630+0000","last_active":"2024-02-05T08:19:31.491251+0000","last_peered":"2024-02-05T08:19:31.491251+0000","last_clean":"2024-02-05T08:19:31.491251+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:19:31.491251+0000","last_undegraded":"2024-02-05T08:19:31.491251+0000","last_fullsized":"2024-02-05T08:19:31.491251+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_clean_scrub_stamp":"2024-02-05T08:18:42.566537+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:48.083453+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":67,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491432+0000","last_change":"2024-02-05T08:18:37.946586+0000","last_active":"2024-02-05T08:19:31.491432+0000","last_peered":"2024-02-05T08:19:31.491432+0000","last_clean":"2024-02-05T08:19:31.491432+0000","last_became_active":"2024-02-05T08:18:34.650359+0000","last_became_peered":"2024-02-05T08:18:34.650359+0000","last_unstale":"2024-02-05T08:19:31.491432+0000","last_undegraded":"2024-02-05T08:19:31.491432+0000","last_fullsized":"2024-02-05T08:19:31.491432+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:37.946440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.946440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:54.089385+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":581,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.481423+0000","last_change":"2024-02-05T08:19:17.633721+0000","last_active":"2024-02-05T08:19:31.481423+0000","last_peered":"2024-02-05T08:19:31.481423+0000","last_clean":"2024-02-05T08:19:31.481423+0000","last_became_active":"2024-02-05T08:18:34.660727+0000","last_became_peered":"2024-02-05T08:18:34.660727+0000","last_unstale":"2024-02-05T08:19:31.481423+0000","last_undegraded":"2024-02-05T08:19:31.481423+0000","last_fullsized":"2024-02-05T08:19:31.481423+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:17.633650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.633650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:44.201663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":588,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.486530+0000","last_change":"2024-02-05T08:18:37.567025+0000","last_active":"2024-02-05T08:19:31.486530+0000","last_peered":"2024-02-05T08:19:31.486530+0000","last_clean":"2024-02-05T08:19:31.486530+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:19:31.486530+0000","last_undegraded":"2024-02-05T08:19:31.486530+0000","last_fullsized":"2024-02-05T08:19:31.486530+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:37.566764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.566764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:59.493231+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"249'24","reported_seq":644,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.486296+0000","last_change":"2024-02-05T08:19:27.349717+0000","last_active":"2024-02-05T08:19:31.486296+0000","last_peered":"2024-02-05T08:19:31.486296+0000","last_clean":"2024-02-05T08:19:31.486296+0000","last_became_active":"2024-02-05T08:12:29.675657+0000","last_became_peered":"2024-02-05T08:12:29.675657+0000","last_unstale":"2024-02-05T08:19:31.486296+0000","last_undegraded":"2024-02-05T08:19:31.486296+0000","last_fullsized":"2024-02-05T08:19:31.486296+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:19:27.349541+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:19:27.349541+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:46.358486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":557,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.811054+0000","last_change":"2024-02-05T08:18:40.590701+0000","last_active":"2024-02-05T08:19:24.811054+0000","last_peered":"2024-02-05T08:19:24.811054+0000","last_clean":"2024-02-05T08:19:24.811054+0000","last_became_active":"2024-02-05T08:18:34.645027+0000","last_became_peered":"2024-02-05T08:18:34.645027+0000","last_unstale":"2024-02-05T08:19:24.811054+0000","last_undegraded":"2024-02-05T08:19:24.811054+0000","last_fullsized":"2024-02-05T08:19:24.811054+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:40.590596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:40.590596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:06.745385+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":755,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491043+0000","last_change":"2024-02-05T08:19:12.452355+0000","last_active":"2024-02-05T08:19:31.491043+0000","last_peered":"2024-02-05T08:19:31.491043+0000","last_clean":"2024-02-05T08:19:31.491043+0000","last_became_active":"2024-02-05T08:04:45.559514+0000","last_became_peered":"2024-02-05T08:04:45.559514+0000","last_unstale":"2024-02-05T08:19:31.491043+0000","last_undegraded":"2024-02-05T08:19:31.491043+0000","last_fullsized":"2024-02-05T08:19:31.491043+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:19:12.452237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:12.452237+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:27.261806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":577,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.482511+0000","last_change":"2024-02-05T08:19:25.118455+0000","last_active":"2024-02-05T08:19:31.482511+0000","last_peered":"2024-02-05T08:19:31.482511+0000","last_clean":"2024-02-05T08:19:31.482511+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:19:31.482511+0000","last_undegraded":"2024-02-05T08:19:31.482511+0000","last_fullsized":"2024-02-05T08:19:31.482511+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:25.118336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:25.118336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:25.631415+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":71,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.492205+0000","last_change":"2024-02-05T08:18:34.765274+0000","last_active":"2024-02-05T08:19:31.492205+0000","last_peered":"2024-02-05T08:19:31.492205+0000","last_clean":"2024-02-05T08:19:31.492205+0000","last_became_active":"2024-02-05T08:18:34.655959+0000","last_became_peered":"2024-02-05T08:18:34.655959+0000","last_unstale":"2024-02-05T08:19:31.492205+0000","last_undegraded":"2024-02-05T08:19:31.492205+0000","last_fullsized":"2024-02-05T08:19:31.492205+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-02-05T08:17:17.993472+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:17:17.993472+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.486611+0000","last_change":"2024-02-05T08:19:28.519254+0000","last_active":"2024-02-05T08:19:31.486611+0000","last_peered":"2024-02-05T08:19:31.486611+0000","last_clean":"2024-02-05T08:19:31.486611+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:19:31.486611+0000","last_undegraded":"2024-02-05T08:19:31.486611+0000","last_fullsized":"2024-02-05T08:19:31.486611+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:28.519149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:28.519149+0000","last_clean_scrub_stamp":"2024-02-05T08:19:28.519149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:43.220985+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"295'10","reported_seq":76,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.492496+0000","last_change":"2024-02-05T08:18:34.681440+0000","last_active":"2024-02-05T08:19:31.492496+0000","last_peered":"2024-02-05T08:19:31.492496+0000","last_clean":"2024-02-05T08:19:31.492496+0000","last_became_active":"2024-02-05T08:18:34.644517+0000","last_became_peered":"2024-02-05T08:18:34.644517+0000","last_unstale":"2024-02-05T08:19:31.492496+0000","last_undegraded":"2024-02-05T08:19:31.492496+0000","last_fullsized":"2024-02-05T08:19:31.492496+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'9","last_scrub_stamp":"2024-02-05T08:18:03.113117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:03.113117+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":569,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.811524+0000","last_change":"2024-02-05T08:18:34.636065+0000","last_active":"2024-02-05T08:19:24.811524+0000","last_peered":"2024-02-05T08:19:24.811524+0000","last_clean":"2024-02-05T08:19:24.811524+0000","last_became_active":"2024-02-05T08:18:34.635793+0000","last_became_peered":"2024-02-05T08:18:34.635793+0000","last_unstale":"2024-02-05T08:19:24.811524+0000","last_undegraded":"2024-02-05T08:19:24.811524+0000","last_fullsized":"2024-02-05T08:19:24.811524+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:03.472950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:18:03.472950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":629,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491189+0000","last_change":"2024-02-05T08:18:46.512784+0000","last_active":"2024-02-05T08:19:31.491189+0000","last_peered":"2024-02-05T08:19:31.491189+0000","last_clean":"2024-02-05T08:19:31.491189+0000","last_became_active":"2024-02-05T08:18:34.640741+0000","last_became_peered":"2024-02-05T08:18:34.640741+0000","last_unstale":"2024-02-05T08:19:31.491189+0000","last_undegraded":"2024-02-05T08:19:31.491189+0000","last_fullsized":"2024-02-05T08:19:31.491189+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:18:46.512698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:46.512698+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:49.244601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":572,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490603+0000","last_change":"2024-02-05T08:18:32.704540+0000","last_active":"2024-02-05T08:19:31.490603+0000","last_peered":"2024-02-05T08:19:31.490603+0000","last_clean":"2024-02-05T08:19:31.490603+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:19:31.490603+0000","last_undegraded":"2024-02-05T08:19:31.490603+0000","last_fullsized":"2024-02-05T08:19:31.490603+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:32.704432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:32.704432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:51.640587+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":59,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.494138+0000","last_change":"2024-02-05T08:18:34.656052+0000","last_active":"2024-02-05T08:19:31.494138+0000","last_peered":"2024-02-05T08:19:31.494138+0000","last_clean":"2024-02-05T08:19:31.494138+0000","last_became_active":"2024-02-05T08:18:34.655760+0000","last_became_peered":"2024-02-05T08:18:34.655760+0000","last_unstale":"2024-02-05T08:19:31.494138+0000","last_undegraded":"2024-02-05T08:19:31.494138+0000","last_fullsized":"2024-02-05T08:19:31.494138+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:51.982489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:17:51.982489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":581,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.482463+0000","last_change":"2024-02-05T08:18:09.089321+0000","last_active":"2024-02-05T08:19:31.482463+0000","last_peered":"2024-02-05T08:19:31.482463+0000","last_clean":"2024-02-05T08:19:31.482463+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:19:31.482463+0000","last_undegraded":"2024-02-05T08:19:31.482463+0000","last_fullsized":"2024-02-05T08:19:31.482463+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:09.089099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:18:09.089099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"278'19","reported_seq":614,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.811437+0000","last_change":"2024-02-05T08:18:34.709290+0000","last_active":"2024-02-05T08:19:24.811437+0000","last_peered":"2024-02-05T08:19:24.811437+0000","last_clean":"2024-02-05T08:19:24.811437+0000","last_became_active":"2024-02-05T08:18:34.645881+0000","last_became_peered":"2024-02-05T08:18:34.645881+0000","last_unstale":"2024-02-05T08:19:24.811437+0000","last_undegraded":"2024-02-05T08:19:24.811437+0000","last_fullsized":"2024-02-05T08:19:24.811437+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:17:11.006333+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":577,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.482591+0000","last_change":"2024-02-05T08:19:15.271802+0000","last_active":"2024-02-05T08:19:31.482591+0000","last_peered":"2024-02-05T08:19:31.482591+0000","last_clean":"2024-02-05T08:19:31.482591+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:19:31.482591+0000","last_undegraded":"2024-02-05T08:19:31.482591+0000","last_fullsized":"2024-02-05T08:19:31.482591+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:15.271671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.271671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:41.325937+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":580,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.492266+0000","last_change":"2024-02-05T08:19:16.486982+0000","last_active":"2024-02-05T08:19:31.492266+0000","last_peered":"2024-02-05T08:19:31.492266+0000","last_clean":"2024-02-05T08:19:31.492266+0000","last_became_active":"2024-02-05T08:18:34.640731+0000","last_became_peered":"2024-02-05T08:18:34.640731+0000","last_unstale":"2024-02-05T08:19:31.492266+0000","last_undegraded":"2024-02-05T08:19:31.492266+0000","last_fullsized":"2024-02-05T08:19:31.492266+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.486901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.486901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:33.515220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":306,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.487631+0000","last_change":"2024-02-05T08:19:16.518726+0000","last_active":"2024-02-05T08:19:31.487631+0000","last_peered":"2024-02-05T08:19:31.487631+0000","last_clean":"2024-02-05T08:19:31.487631+0000","last_became_active":"2024-02-05T08:12:29.684444+0000","last_became_peered":"2024-02-05T08:12:29.684444+0000","last_unstale":"2024-02-05T08:19:31.487631+0000","last_undegraded":"2024-02-05T08:19:31.487631+0000","last_fullsized":"2024-02-05T08:19:31.487631+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.518655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.518655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:42.682400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'15","reported_seq":67,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.493856+0000","last_change":"2024-02-05T08:18:34.759559+0000","last_active":"2024-02-05T08:19:31.493856+0000","last_peered":"2024-02-05T08:19:31.493856+0000","last_clean":"2024-02-05T08:19:31.493856+0000","last_became_active":"2024-02-05T08:18:34.655119+0000","last_became_peered":"2024-02-05T08:18:34.655119+0000","last_unstale":"2024-02-05T08:19:31.493856+0000","last_undegraded":"2024-02-05T08:19:31.493856+0000","last_fullsized":"2024-02-05T08:19:31.493856+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:17:54.101749+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.101749+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"319'2143","reported_seq":2173,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:33.204915+0000","last_change":"2024-02-05T08:19:07.333147+0000","last_active":"2024-02-05T08:19:33.204915+0000","last_peered":"2024-02-05T08:19:33.204915+0000","last_clean":"2024-02-05T08:19:33.204915+0000","last_became_active":"2024-02-05T08:19:07.332783+0000","last_became_peered":"2024-02-05T08:19:07.332783+0000","last_unstale":"2024-02-05T08:19:33.204915+0000","last_undegraded":"2024-02-05T08:19:33.204915+0000","last_fullsized":"2024-02-05T08:19:33.204915+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2143,"log_dups_size":0,"ondisk_log_size":2143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:27.089978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140443648,"num_objects":2143,"num_object_clones":0,"num_object_copies":6429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2143,"num_write_kb":137152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":576,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491705+0000","last_change":"2024-02-05T08:18:44.548667+0000","last_active":"2024-02-05T08:19:31.491705+0000","last_peered":"2024-02-05T08:19:31.491705+0000","last_clean":"2024-02-05T08:19:31.491705+0000","last_became_active":"2024-02-05T08:12:29.679849+0000","last_became_peered":"2024-02-05T08:12:29.679849+0000","last_unstale":"2024-02-05T08:19:31.491705+0000","last_undegraded":"2024-02-05T08:19:31.491705+0000","last_fullsized":"2024-02-05T08:19:31.491705+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:44.548575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:44.548575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:57.417260+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":589,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.484088+0000","last_change":"2024-02-05T08:19:18.255247+0000","last_active":"2024-02-05T08:19:31.484088+0000","last_peered":"2024-02-05T08:19:31.484088+0000","last_clean":"2024-02-05T08:19:31.484088+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:19:31.484088+0000","last_undegraded":"2024-02-05T08:19:31.484088+0000","last_fullsized":"2024-02-05T08:19:31.484088+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:19:18.255099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:18.255099+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:21.321010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"262'34","reported_seq":658,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490042+0000","last_change":"2024-02-05T08:18:09.398357+0000","last_active":"2024-02-05T08:19:31.490042+0000","last_peered":"2024-02-05T08:19:31.490042+0000","last_clean":"2024-02-05T08:19:31.490042+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:19:31.490042+0000","last_undegraded":"2024-02-05T08:19:31.490042+0000","last_fullsized":"2024-02-05T08:19:31.490042+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'34","last_scrub_stamp":"2024-02-05T08:18:09.398226+0000","last_deep_scrub":"262'34","last_deep_scrub_stamp":"2024-02-05T08:16:06.848964+0000","last_clean_scrub_stamp":"2024-02-05T08:18:09.398226+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":777,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.486183+0000","last_change":"2024-02-05T08:19:31.117664+0000","last_active":"2024-02-05T08:19:31.486183+0000","last_peered":"2024-02-05T08:19:31.486183+0000","last_clean":"2024-02-05T08:19:31.486183+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:19:31.486183+0000","last_undegraded":"2024-02-05T08:19:31.486183+0000","last_fullsized":"2024-02-05T08:19:31.486183+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_clean_scrub_stamp":"2024-02-05T08:19:31.117523+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:58.253463+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"319'2031","reported_seq":2061,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:32.655973+0000","last_change":"2024-02-05T08:19:07.346353+0000","last_active":"2024-02-05T08:19:32.655973+0000","last_peered":"2024-02-05T08:19:32.655973+0000","last_clean":"2024-02-05T08:19:32.655973+0000","last_became_active":"2024-02-05T08:19:07.346046+0000","last_became_peered":"2024-02-05T08:19:07.346046+0000","last_unstale":"2024-02-05T08:19:32.655973+0000","last_undegraded":"2024-02-05T08:19:32.655973+0000","last_fullsized":"2024-02-05T08:19:32.655973+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2031,"log_dups_size":0,"ondisk_log_size":2031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:35.076716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133103616,"num_objects":2031,"num_object_clones":0,"num_object_copies":6093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2031,"num_write_kb":129984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":59,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.492258+0000","last_change":"2024-02-05T08:18:34.654626+0000","last_active":"2024-02-05T08:19:31.492258+0000","last_peered":"2024-02-05T08:19:31.492258+0000","last_clean":"2024-02-05T08:19:31.492258+0000","last_became_active":"2024-02-05T08:18:34.653669+0000","last_became_peered":"2024-02-05T08:18:34.653669+0000","last_unstale":"2024-02-05T08:19:31.492258+0000","last_undegraded":"2024-02-05T08:19:31.492258+0000","last_fullsized":"2024-02-05T08:19:31.492258+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:31.245736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:18:31.245736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:44.428368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":569,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.483113+0000","last_change":"2024-02-05T08:17:12.948132+0000","last_active":"2024-02-05T08:19:31.483113+0000","last_peered":"2024-02-05T08:19:31.483113+0000","last_clean":"2024-02-05T08:19:31.483113+0000","last_became_active":"2024-02-05T08:04:41.542243+0000","last_became_peered":"2024-02-05T08:04:41.542243+0000","last_unstale":"2024-02-05T08:19:31.483113+0000","last_undegraded":"2024-02-05T08:19:31.483113+0000","last_fullsized":"2024-02-05T08:19:31.483113+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:12.948033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:17:12.948033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":632,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.488279+0000","last_change":"2024-02-05T08:18:34.747769+0000","last_active":"2024-02-05T08:19:31.488279+0000","last_peered":"2024-02-05T08:19:31.488279+0000","last_clean":"2024-02-05T08:19:31.488279+0000","last_became_active":"2024-02-05T08:18:34.644717+0000","last_became_peered":"2024-02-05T08:18:34.644717+0000","last_unstale":"2024-02-05T08:19:31.488279+0000","last_undegraded":"2024-02-05T08:19:31.488279+0000","last_fullsized":"2024-02-05T08:19:31.488279+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-02-05T08:18:02.629366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:02.629366+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":576,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491501+0000","last_change":"2024-02-05T08:18:33.725725+0000","last_active":"2024-02-05T08:19:31.491501+0000","last_peered":"2024-02-05T08:19:31.491501+0000","last_clean":"2024-02-05T08:19:31.491501+0000","last_became_active":"2024-02-05T08:04:45.569068+0000","last_became_peered":"2024-02-05T08:04:45.569068+0000","last_unstale":"2024-02-05T08:19:31.491501+0000","last_undegraded":"2024-02-05T08:19:31.491501+0000","last_fullsized":"2024-02-05T08:19:31.491501+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:33.725647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:18:33.725647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:53.197171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"318'1586","reported_seq":1614,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:28.792874+0000","last_change":"2024-02-05T08:19:07.345567+0000","last_active":"2024-02-05T08:19:28.792874+0000","last_peered":"2024-02-05T08:19:28.792874+0000","last_clean":"2024-02-05T08:19:28.792874+0000","last_became_active":"2024-02-05T08:19:07.345271+0000","last_became_peered":"2024-02-05T08:19:07.345271+0000","last_unstale":"2024-02-05T08:19:28.792874+0000","last_undegraded":"2024-02-05T08:19:28.792874+0000","last_fullsized":"2024-02-05T08:19:28.792874+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":1586,"log_dups_size":0,"ondisk_log_size":1586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:13.627916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103940096,"num_objects":1586,"num_object_clones":0,"num_object_copies":4758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1586,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1586,"num_write_kb":101504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":569,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.482443+0000","last_change":"2024-02-05T08:18:34.642584+0000","last_active":"2024-02-05T08:19:31.482443+0000","last_peered":"2024-02-05T08:19:31.482443+0000","last_clean":"2024-02-05T08:19:31.482443+0000","last_became_active":"2024-02-05T08:18:34.642239+0000","last_became_peered":"2024-02-05T08:18:34.642239+0000","last_unstale":"2024-02-05T08:19:31.482443+0000","last_undegraded":"2024-02-05T08:19:31.482443+0000","last_fullsized":"2024-02-05T08:19:31.482443+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:00.084681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:00.084681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":576,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490879+0000","last_change":"2024-02-05T08:19:07.639820+0000","last_active":"2024-02-05T08:19:31.490879+0000","last_peered":"2024-02-05T08:19:31.490879+0000","last_clean":"2024-02-05T08:19:31.490879+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:19:31.490879+0000","last_undegraded":"2024-02-05T08:19:31.490879+0000","last_fullsized":"2024-02-05T08:19:31.490879+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:07.639746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:19:07.639746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:35.193266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":590,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.482812+0000","last_change":"2024-02-05T08:19:30.082858+0000","last_active":"2024-02-05T08:19:31.482812+0000","last_peered":"2024-02-05T08:19:31.482812+0000","last_clean":"2024-02-05T08:19:31.482812+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:19:31.482812+0000","last_undegraded":"2024-02-05T08:19:31.482812+0000","last_fullsized":"2024-02-05T08:19:31.482812+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_clean_scrub_stamp":"2024-02-05T08:19:30.082733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:43.747361+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"236'35","reported_seq":350,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.489321+0000","last_change":"2024-02-05T08:19:13.499434+0000","last_active":"2024-02-05T08:19:31.489321+0000","last_peered":"2024-02-05T08:19:31.489321+0000","last_clean":"2024-02-05T08:19:31.489321+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:19:31.489321+0000","last_undegraded":"2024-02-05T08:19:31.489321+0000","last_fullsized":"2024-02-05T08:19:31.489321+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'35","last_scrub_stamp":"2024-02-05T08:19:13.499309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:13.499309+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:16.241570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"319'2101","reported_seq":2130,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:33.110759+0000","last_change":"2024-02-05T08:19:07.345502+0000","last_active":"2024-02-05T08:19:33.110759+0000","last_peered":"2024-02-05T08:19:33.110759+0000","last_clean":"2024-02-05T08:19:33.110759+0000","last_became_active":"2024-02-05T08:19:07.345197+0000","last_became_peered":"2024-02-05T08:19:07.345197+0000","last_unstale":"2024-02-05T08:19:33.110759+0000","last_undegraded":"2024-02-05T08:19:33.110759+0000","last_fullsized":"2024-02-05T08:19:33.110759+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2101,"log_dups_size":0,"ondisk_log_size":2101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:33.603880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137691136,"num_objects":2101,"num_object_clones":0,"num_object_copies":6303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2101,"num_write_kb":134464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":553,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.810284+0000","last_change":"2024-02-05T08:19:24.708102+0000","last_active":"2024-02-05T08:19:24.810284+0000","last_peered":"2024-02-05T08:19:24.810284+0000","last_clean":"2024-02-05T08:19:24.810284+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:19:24.810284+0000","last_undegraded":"2024-02-05T08:19:24.810284+0000","last_fullsized":"2024-02-05T08:19:24.810284+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_clean_scrub_stamp":"2024-02-05T08:19:24.707985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:43.900668+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":588,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490165+0000","last_change":"2024-02-05T08:18:41.709917+0000","last_active":"2024-02-05T08:19:31.490165+0000","last_peered":"2024-02-05T08:19:31.490165+0000","last_clean":"2024-02-05T08:19:31.490165+0000","last_became_active":"2024-02-05T08:18:34.645830+0000","last_became_peered":"2024-02-05T08:18:34.645830+0000","last_unstale":"2024-02-05T08:19:31.490165+0000","last_undegraded":"2024-02-05T08:19:31.490165+0000","last_fullsized":"2024-02-05T08:19:31.490165+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:41.709832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:41.709832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:50.117637+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":600,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490522+0000","last_change":"2024-02-05T08:19:17.520776+0000","last_active":"2024-02-05T08:19:31.490522+0000","last_peered":"2024-02-05T08:19:31.490522+0000","last_clean":"2024-02-05T08:19:31.490522+0000","last_became_active":"2024-02-05T08:18:34.633214+0000","last_became_peered":"2024-02-05T08:18:34.633214+0000","last_unstale":"2024-02-05T08:19:31.490522+0000","last_undegraded":"2024-02-05T08:19:31.490522+0000","last_fullsized":"2024-02-05T08:19:31.490522+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:17.520613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.520613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:37.304945+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"262'19","reported_seq":590,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.809659+0000","last_change":"2024-02-05T08:19:17.554248+0000","last_active":"2024-02-05T08:19:24.809659+0000","last_peered":"2024-02-05T08:19:24.809659+0000","last_clean":"2024-02-05T08:19:24.809659+0000","last_became_active":"2024-02-05T08:04:43.552483+0000","last_became_peered":"2024-02-05T08:04:43.552483+0000","last_unstale":"2024-02-05T08:19:24.809659+0000","last_undegraded":"2024-02-05T08:19:24.809659+0000","last_fullsized":"2024-02-05T08:19:24.809659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'19","last_scrub_stamp":"2024-02-05T08:19:17.554161+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.554161+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:46.190354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"319'2171","reported_seq":2201,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:32.657951+0000","last_change":"2024-02-05T08:19:07.332201+0000","last_active":"2024-02-05T08:19:32.657951+0000","last_peered":"2024-02-05T08:19:32.657951+0000","last_clean":"2024-02-05T08:19:32.657951+0000","last_became_active":"2024-02-05T08:19:07.331718+0000","last_became_peered":"2024-02-05T08:19:07.331718+0000","last_unstale":"2024-02-05T08:19:32.657951+0000","last_undegraded":"2024-02-05T08:19:32.657951+0000","last_fullsized":"2024-02-05T08:19:32.657951+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2171,"log_dups_size":0,"ondisk_log_size":2171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:13.128925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142278656,"num_objects":2171,"num_object_clones":0,"num_object_copies":6513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":138944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'130","reported_seq":343,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490794+0000","last_change":"2024-02-05T08:19:08.488730+0000","last_active":"2024-02-05T08:19:31.490794+0000","last_peered":"2024-02-05T08:19:31.490794+0000","last_clean":"2024-02-05T08:19:31.490794+0000","last_became_active":"2024-02-05T08:19:01.816550+0000","last_became_peered":"2024-02-05T08:19:01.816550+0000","last_unstale":"2024-02-05T08:19:31.490794+0000","last_undegraded":"2024-02-05T08:19:31.490794+0000","last_fullsized":"2024-02-05T08:19:31.490794+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:08.488647+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:08.488647+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:09.049752+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":287,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.488912+0000","last_change":"2024-02-05T08:18:34.638135+0000","last_active":"2024-02-05T08:19:31.488912+0000","last_peered":"2024-02-05T08:19:31.488912+0000","last_clean":"2024-02-05T08:19:31.488912+0000","last_became_active":"2024-02-05T08:18:34.637642+0000","last_became_peered":"2024-02-05T08:18:34.637642+0000","last_unstale":"2024-02-05T08:19:31.488912+0000","last_undegraded":"2024-02-05T08:19:31.488912+0000","last_fullsized":"2024-02-05T08:19:31.488912+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:54.817641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.817641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":609,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491521+0000","last_change":"2024-02-05T08:18:48.543478+0000","last_active":"2024-02-05T08:19:31.491521+0000","last_peered":"2024-02-05T08:19:31.491521+0000","last_clean":"2024-02-05T08:19:31.491521+0000","last_became_active":"2024-02-05T08:12:29.679395+0000","last_became_peered":"2024-02-05T08:12:29.679395+0000","last_unstale":"2024-02-05T08:19:31.491521+0000","last_undegraded":"2024-02-05T08:19:31.491521+0000","last_fullsized":"2024-02-05T08:19:31.491521+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:18:48.543343+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:18:48.543343+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:59.710896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":581,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.486008+0000","last_change":"2024-02-05T08:18:41.490580+0000","last_active":"2024-02-05T08:19:31.486008+0000","last_peered":"2024-02-05T08:19:31.486008+0000","last_clean":"2024-02-05T08:19:31.486008+0000","last_became_active":"2024-02-05T08:12:29.670859+0000","last_became_peered":"2024-02-05T08:12:29.670859+0000","last_unstale":"2024-02-05T08:19:31.486008+0000","last_undegraded":"2024-02-05T08:19:31.486008+0000","last_fullsized":"2024-02-05T08:19:31.486008+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:41.490503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:18:41.490503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:04.724327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"295'52","reported_seq":886,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.481834+0000","last_change":"2024-02-05T08:19:15.601719+0000","last_active":"2024-02-05T08:19:31.481834+0000","last_peered":"2024-02-05T08:19:31.481834+0000","last_clean":"2024-02-05T08:19:31.481834+0000","last_became_active":"2024-02-05T08:18:34.666477+0000","last_became_peered":"2024-02-05T08:18:34.666477+0000","last_unstale":"2024-02-05T08:19:31.481834+0000","last_undegraded":"2024-02-05T08:19:31.481834+0000","last_fullsized":"2024-02-05T08:19:31.481834+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'52","last_scrub_stamp":"2024-02-05T08:19:15.601598+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.601598+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:26.003935+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":550,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.809848+0000","last_change":"2024-02-05T08:19:14.615122+0000","last_active":"2024-02-05T08:19:24.809848+0000","last_peered":"2024-02-05T08:19:24.809848+0000","last_clean":"2024-02-05T08:19:24.809848+0000","last_became_active":"2024-02-05T08:04:45.561782+0000","last_became_peered":"2024-02-05T08:04:45.561782+0000","last_unstale":"2024-02-05T08:19:24.809848+0000","last_undegraded":"2024-02-05T08:19:24.809848+0000","last_fullsized":"2024-02-05T08:19:24.809848+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:14.615047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.615047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:20.657059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"319'2190","reported_seq":2219,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:33.215039+0000","last_change":"2024-02-05T08:19:07.333803+0000","last_active":"2024-02-05T08:19:33.215039+0000","last_peered":"2024-02-05T08:19:33.215039+0000","last_clean":"2024-02-05T08:19:33.215039+0000","last_became_active":"2024-02-05T08:19:07.333532+0000","last_became_peered":"2024-02-05T08:19:07.333532+0000","last_unstale":"2024-02-05T08:19:33.215039+0000","last_undegraded":"2024-02-05T08:19:33.215039+0000","last_fullsized":"2024-02-05T08:19:33.215039+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2190,"log_dups_size":0,"ondisk_log_size":2190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:11.569410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143523840,"num_objects":2190,"num_object_clones":0,"num_object_copies":6570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2190,"num_write_kb":140160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'130","reported_seq":875,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.482227+0000","last_change":"2024-02-05T08:19:08.136074+0000","last_active":"2024-02-05T08:19:31.482227+0000","last_peered":"2024-02-05T08:19:31.482227+0000","last_clean":"2024-02-05T08:19:31.482227+0000","last_became_active":"2024-02-05T08:19:01.813292+0000","last_became_peered":"2024-02-05T08:19:01.813292+0000","last_unstale":"2024-02-05T08:19:31.482227+0000","last_undegraded":"2024-02-05T08:19:31.482227+0000","last_fullsized":"2024-02-05T08:19:31.482227+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:08.135992+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:08.135992+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:09.930222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":582,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.483894+0000","last_change":"2024-02-05T08:18:35.074566+0000","last_active":"2024-02-05T08:19:31.483894+0000","last_peered":"2024-02-05T08:19:31.483894+0000","last_clean":"2024-02-05T08:19:31.483894+0000","last_became_active":"2024-02-05T08:18:34.643109+0000","last_became_peered":"2024-02-05T08:18:34.643109+0000","last_unstale":"2024-02-05T08:19:31.483894+0000","last_undegraded":"2024-02-05T08:19:31.483894+0000","last_fullsized":"2024-02-05T08:19:31.483894+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:35.074467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:35.074467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:02.738828+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":597,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.480670+0000","last_change":"2024-02-05T08:18:43.341960+0000","last_active":"2024-02-05T08:19:31.480670+0000","last_peered":"2024-02-05T08:19:31.480670+0000","last_clean":"2024-02-05T08:19:31.480670+0000","last_became_active":"2024-02-05T08:12:29.673554+0000","last_became_peered":"2024-02-05T08:12:29.673554+0000","last_unstale":"2024-02-05T08:19:31.480670+0000","last_undegraded":"2024-02-05T08:19:31.480670+0000","last_fullsized":"2024-02-05T08:19:31.480670+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:18:43.341831+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:18:43.341831+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:02.149423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":606,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491247+0000","last_change":"2024-02-05T08:18:39.760847+0000","last_active":"2024-02-05T08:19:31.491247+0000","last_peered":"2024-02-05T08:19:31.491247+0000","last_clean":"2024-02-05T08:19:31.491247+0000","last_became_active":"2024-02-05T08:18:34.644683+0000","last_became_peered":"2024-02-05T08:18:34.644683+0000","last_unstale":"2024-02-05T08:19:31.491247+0000","last_undegraded":"2024-02-05T08:19:31.491247+0000","last_fullsized":"2024-02-05T08:19:31.491247+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:39.760577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:39.760577+0000","last_clean_scrub_stamp":"2024-02-05T08:18:39.760577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:56.899445+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'17","reported_seq":590,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491271+0000","last_change":"2024-02-05T08:19:11.615846+0000","last_active":"2024-02-05T08:19:31.491271+0000","last_peered":"2024-02-05T08:19:31.491271+0000","last_clean":"2024-02-05T08:19:31.491271+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:19:31.491271+0000","last_undegraded":"2024-02-05T08:19:31.491271+0000","last_fullsized":"2024-02-05T08:19:31.491271+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'17","last_scrub_stamp":"2024-02-05T08:19:11.615763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:11.615763+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:35.229330+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":584,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491222+0000","last_change":"2024-02-05T08:19:18.605539+0000","last_active":"2024-02-05T08:19:31.491222+0000","last_peered":"2024-02-05T08:19:31.491222+0000","last_clean":"2024-02-05T08:19:31.491222+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:19:31.491222+0000","last_undegraded":"2024-02-05T08:19:31.491222+0000","last_fullsized":"2024-02-05T08:19:31.491222+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:18.605462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:18.605462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:34.531727+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"319'2080","reported_seq":2110,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:32.617274+0000","last_change":"2024-02-05T08:19:07.330113+0000","last_active":"2024-02-05T08:19:32.617274+0000","last_peered":"2024-02-05T08:19:32.617274+0000","last_clean":"2024-02-05T08:19:32.617274+0000","last_became_active":"2024-02-05T08:19:07.329171+0000","last_became_peered":"2024-02-05T08:19:07.329171+0000","last_unstale":"2024-02-05T08:19:32.617274+0000","last_undegraded":"2024-02-05T08:19:32.617274+0000","last_fullsized":"2024-02-05T08:19:32.617274+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2080,"log_dups_size":0,"ondisk_log_size":2080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:31.381699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136314880,"num_objects":2080,"num_object_clones":0,"num_object_copies":6240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2080,"num_write_kb":133120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":576,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.492147+0000","last_change":"2024-02-05T08:19:19.557662+0000","last_active":"2024-02-05T08:19:31.492147+0000","last_peered":"2024-02-05T08:19:31.492147+0000","last_clean":"2024-02-05T08:19:31.492147+0000","last_became_active":"2024-02-05T08:18:34.633450+0000","last_became_peered":"2024-02-05T08:18:34.633450+0000","last_unstale":"2024-02-05T08:19:31.492147+0000","last_undegraded":"2024-02-05T08:19:31.492147+0000","last_fullsized":"2024-02-05T08:19:31.492147+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:19.557556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:19.557556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:42.073792+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":867,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.482721+0000","last_change":"2024-02-05T08:19:10.129563+0000","last_active":"2024-02-05T08:19:31.482721+0000","last_peered":"2024-02-05T08:19:31.482721+0000","last_clean":"2024-02-05T08:19:31.482721+0000","last_became_active":"2024-02-05T08:18:57.285362+0000","last_became_peered":"2024-02-05T08:18:57.285362+0000","last_unstale":"2024-02-05T08:19:31.482721+0000","last_undegraded":"2024-02-05T08:19:31.482721+0000","last_fullsized":"2024-02-05T08:19:31.482721+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:10.129494+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:10.129494+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:13.406460+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":67,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491718+0000","last_change":"2024-02-05T08:19:20.109021+0000","last_active":"2024-02-05T08:19:31.491718+0000","last_peered":"2024-02-05T08:19:31.491718+0000","last_clean":"2024-02-05T08:19:31.491718+0000","last_became_active":"2024-02-05T08:18:34.631974+0000","last_became_peered":"2024-02-05T08:18:34.631974+0000","last_unstale":"2024-02-05T08:19:31.491718+0000","last_undegraded":"2024-02-05T08:19:31.491718+0000","last_fullsized":"2024-02-05T08:19:31.491718+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:20.108944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:20.108944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:43.029476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"236'30","reported_seq":419,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.488175+0000","last_change":"2024-02-05T08:18:37.284687+0000","last_active":"2024-02-05T08:19:31.488175+0000","last_peered":"2024-02-05T08:19:31.488175+0000","last_clean":"2024-02-05T08:19:31.488175+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:19:31.488175+0000","last_undegraded":"2024-02-05T08:19:31.488175+0000","last_fullsized":"2024-02-05T08:19:31.488175+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:18:37.284559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.284559+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:07.016040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":562,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.487346+0000","last_change":"2024-02-05T08:18:42.996770+0000","last_active":"2024-02-05T08:19:31.487346+0000","last_peered":"2024-02-05T08:19:31.487346+0000","last_clean":"2024-02-05T08:19:31.487346+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:19:31.487346+0000","last_undegraded":"2024-02-05T08:19:31.487346+0000","last_fullsized":"2024-02-05T08:19:31.487346+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_clean_scrub_stamp":"2024-02-05T08:18:42.996695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:55.044771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"318'1484","reported_seq":1512,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:28.786649+0000","last_change":"2024-02-05T08:19:07.327455+0000","last_active":"2024-02-05T08:19:28.786649+0000","last_peered":"2024-02-05T08:19:28.786649+0000","last_clean":"2024-02-05T08:19:28.786649+0000","last_became_active":"2024-02-05T08:19:07.327007+0000","last_became_peered":"2024-02-05T08:19:07.327007+0000","last_unstale":"2024-02-05T08:19:28.786649+0000","last_undegraded":"2024-02-05T08:19:28.786649+0000","last_fullsized":"2024-02-05T08:19:28.786649+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":1484,"log_dups_size":0,"ondisk_log_size":1484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:16.019948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97255424,"num_objects":1484,"num_object_clones":0,"num_object_copies":4452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1484,"num_write_kb":94976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'130","reported_seq":343,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.481520+0000","last_change":"2024-02-05T08:19:07.594499+0000","last_active":"2024-02-05T08:19:31.481520+0000","last_peered":"2024-02-05T08:19:31.481520+0000","last_clean":"2024-02-05T08:19:31.481520+0000","last_became_active":"2024-02-05T08:19:01.817375+0000","last_became_peered":"2024-02-05T08:19:01.817375+0000","last_unstale":"2024-02-05T08:19:31.481520+0000","last_undegraded":"2024-02-05T08:19:31.481520+0000","last_fullsized":"2024-02-05T08:19:31.481520+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:07.594418+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:07.594418+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:17.880583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":578,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.481335+0000","last_change":"2024-02-05T08:19:16.646507+0000","last_active":"2024-02-05T08:19:31.481335+0000","last_peered":"2024-02-05T08:19:31.481335+0000","last_clean":"2024-02-05T08:19:31.481335+0000","last_became_active":"2024-02-05T08:18:34.659712+0000","last_became_peered":"2024-02-05T08:18:34.659712+0000","last_unstale":"2024-02-05T08:19:31.481335+0000","last_undegraded":"2024-02-05T08:19:31.481335+0000","last_fullsized":"2024-02-05T08:19:31.481335+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.646431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.646431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:27.384100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":570,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.809962+0000","last_change":"2024-02-05T08:18:37.562478+0000","last_active":"2024-02-05T08:19:24.809962+0000","last_peered":"2024-02-05T08:19:24.809962+0000","last_clean":"2024-02-05T08:19:24.809962+0000","last_became_active":"2024-02-05T08:18:34.635954+0000","last_became_peered":"2024-02-05T08:18:34.635954+0000","last_unstale":"2024-02-05T08:19:24.809962+0000","last_undegraded":"2024-02-05T08:19:24.809962+0000","last_fullsized":"2024-02-05T08:19:24.809962+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:37.562368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.562368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:04.501373+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":1475,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.482666+0000","last_change":"2024-02-05T08:19:14.179660+0000","last_active":"2024-02-05T08:19:31.482666+0000","last_peered":"2024-02-05T08:19:31.482666+0000","last_clean":"2024-02-05T08:19:31.482666+0000","last_became_active":"2024-02-05T08:04:40.526736+0000","last_became_peered":"2024-02-05T08:04:40.526736+0000","last_unstale":"2024-02-05T08:19:31.482666+0000","last_undegraded":"2024-02-05T08:19:31.482666+0000","last_fullsized":"2024-02-05T08:19:31.482666+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:19:14.179578+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.179578+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:22.868977+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":884,"num_read_kb":879,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":609,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.811160+0000","last_change":"2024-02-05T08:18:45.398016+0000","last_active":"2024-02-05T08:19:24.811160+0000","last_peered":"2024-02-05T08:19:24.811160+0000","last_clean":"2024-02-05T08:19:24.811160+0000","last_became_active":"2024-02-05T08:14:24.992820+0000","last_became_peered":"2024-02-05T08:14:24.992820+0000","last_unstale":"2024-02-05T08:19:24.811160+0000","last_undegraded":"2024-02-05T08:19:24.811160+0000","last_fullsized":"2024-02-05T08:19:24.811160+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:18:45.397922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:45.397922+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:54.624995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":560,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.493741+0000","last_change":"2024-02-05T08:18:03.284339+0000","last_active":"2024-02-05T08:19:31.493741+0000","last_peered":"2024-02-05T08:19:31.493741+0000","last_clean":"2024-02-05T08:19:31.493741+0000","last_became_active":"2024-02-05T08:04:45.558863+0000","last_became_peered":"2024-02-05T08:04:45.558863+0000","last_unstale":"2024-02-05T08:19:31.493741+0000","last_undegraded":"2024-02-05T08:19:31.493741+0000","last_fullsized":"2024-02-05T08:19:31.493741+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:03.282010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:18:03.282010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"319'2073","reported_seq":2103,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:32.662803+0000","last_change":"2024-02-05T08:19:07.333127+0000","last_active":"2024-02-05T08:19:32.662803+0000","last_peered":"2024-02-05T08:19:32.662803+0000","last_clean":"2024-02-05T08:19:32.662803+0000","last_became_active":"2024-02-05T08:19:07.332849+0000","last_became_peered":"2024-02-05T08:19:07.332849+0000","last_unstale":"2024-02-05T08:19:32.662803+0000","last_undegraded":"2024-02-05T08:19:32.662803+0000","last_fullsized":"2024-02-05T08:19:32.662803+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2073,"log_dups_size":0,"ondisk_log_size":2073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:34.840662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135856128,"num_objects":2073,"num_object_clones":0,"num_object_copies":6219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2073,"num_write_kb":132672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'130","reported_seq":604,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.487867+0000","last_change":"2024-02-05T08:19:08.369253+0000","last_active":"2024-02-05T08:19:31.487867+0000","last_peered":"2024-02-05T08:19:31.487867+0000","last_clean":"2024-02-05T08:19:31.487867+0000","last_became_active":"2024-02-05T08:19:01.817743+0000","last_became_peered":"2024-02-05T08:19:01.817743+0000","last_unstale":"2024-02-05T08:19:31.487867+0000","last_undegraded":"2024-02-05T08:19:31.487867+0000","last_fullsized":"2024-02-05T08:19:31.487867+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:08.369160+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:08.369160+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:08.459054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":578,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.480854+0000","last_change":"2024-02-05T08:19:27.862650+0000","last_active":"2024-02-05T08:19:31.480854+0000","last_peered":"2024-02-05T08:19:31.480854+0000","last_clean":"2024-02-05T08:19:31.480854+0000","last_became_active":"2024-02-05T08:12:29.672973+0000","last_became_peered":"2024-02-05T08:12:29.672973+0000","last_unstale":"2024-02-05T08:19:31.480854+0000","last_undegraded":"2024-02-05T08:19:31.480854+0000","last_fullsized":"2024-02-05T08:19:31.480854+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:27.862543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:27.862543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:53.442473+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":578,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.481497+0000","last_change":"2024-02-05T08:18:37.350834+0000","last_active":"2024-02-05T08:19:31.481497+0000","last_peered":"2024-02-05T08:19:31.481497+0000","last_clean":"2024-02-05T08:19:31.481497+0000","last_became_active":"2024-02-05T08:04:41.538412+0000","last_became_peered":"2024-02-05T08:04:41.538412+0000","last_unstale":"2024-02-05T08:19:31.481497+0000","last_undegraded":"2024-02-05T08:19:31.481497+0000","last_fullsized":"2024-02-05T08:19:31.481497+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:37.350622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.350622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:04.705920+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":613,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.809565+0000","last_change":"2024-02-05T08:19:15.574020+0000","last_active":"2024-02-05T08:19:24.809565+0000","last_peered":"2024-02-05T08:19:24.809565+0000","last_clean":"2024-02-05T08:19:24.809565+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:19:24.809565+0000","last_undegraded":"2024-02-05T08:19:24.809565+0000","last_fullsized":"2024-02-05T08:19:24.809565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-02-05T08:19:15.573897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.573897+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:18.656716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":306,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.487607+0000","last_change":"2024-02-05T08:19:12.455979+0000","last_active":"2024-02-05T08:19:31.487607+0000","last_peered":"2024-02-05T08:19:31.487607+0000","last_clean":"2024-02-05T08:19:31.487607+0000","last_became_active":"2024-02-05T08:12:29.688534+0000","last_became_peered":"2024-02-05T08:12:29.688534+0000","last_unstale":"2024-02-05T08:19:31.487607+0000","last_undegraded":"2024-02-05T08:19:31.487607+0000","last_fullsized":"2024-02-05T08:19:31.487607+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:12.455899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:19:12.455899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:38.552593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"319'2132","reported_seq":2162,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:33.215151+0000","last_change":"2024-02-05T08:19:07.332440+0000","last_active":"2024-02-05T08:19:33.215151+0000","last_peered":"2024-02-05T08:19:33.215151+0000","last_clean":"2024-02-05T08:19:33.215151+0000","last_became_active":"2024-02-05T08:19:07.332176+0000","last_became_peered":"2024-02-05T08:19:07.332176+0000","last_unstale":"2024-02-05T08:19:33.215151+0000","last_undegraded":"2024-02-05T08:19:33.215151+0000","last_fullsized":"2024-02-05T08:19:33.215151+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2132,"log_dups_size":0,"ondisk_log_size":2132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:13.216292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139722752,"num_objects":2132,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2132,"num_write_kb":136448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"82'130","reported_seq":604,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.488102+0000","last_change":"2024-02-05T08:19:07.349084+0000","last_active":"2024-02-05T08:19:31.488102+0000","last_peered":"2024-02-05T08:19:31.488102+0000","last_clean":"2024-02-05T08:19:31.488102+0000","last_became_active":"2024-02-05T08:19:01.815446+0000","last_became_peered":"2024-02-05T08:19:01.815446+0000","last_unstale":"2024-02-05T08:19:31.488102+0000","last_undegraded":"2024-02-05T08:19:31.488102+0000","last_fullsized":"2024-02-05T08:19:31.488102+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:07.349004+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:07.349004+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:30.588026+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":566,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.809008+0000","last_change":"2024-02-05T08:19:10.488490+0000","last_active":"2024-02-05T08:19:24.809008+0000","last_peered":"2024-02-05T08:19:24.809008+0000","last_clean":"2024-02-05T08:19:24.809008+0000","last_became_active":"2024-02-05T08:18:34.636693+0000","last_became_peered":"2024-02-05T08:18:34.636693+0000","last_unstale":"2024-02-05T08:19:24.809008+0000","last_undegraded":"2024-02-05T08:19:24.809008+0000","last_fullsized":"2024-02-05T08:19:24.809008+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:10.488395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:10.488395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:26.259113+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":589,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491091+0000","last_change":"2024-02-05T08:19:18.499832+0000","last_active":"2024-02-05T08:19:31.491091+0000","last_peered":"2024-02-05T08:19:31.491091+0000","last_clean":"2024-02-05T08:19:31.491091+0000","last_became_active":"2024-02-05T08:18:34.633263+0000","last_became_peered":"2024-02-05T08:18:34.633263+0000","last_unstale":"2024-02-05T08:19:31.491091+0000","last_undegraded":"2024-02-05T08:19:31.491091+0000","last_fullsized":"2024-02-05T08:19:31.491091+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:18.499743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:18.499743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:43.590827+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'20","reported_seq":604,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.481518+0000","last_change":"2024-02-05T08:19:09.511660+0000","last_active":"2024-02-05T08:19:31.481518+0000","last_peered":"2024-02-05T08:19:31.481518+0000","last_clean":"2024-02-05T08:19:31.481518+0000","last_became_active":"2024-02-05T08:04:43.552053+0000","last_became_peered":"2024-02-05T08:04:43.552053+0000","last_unstale":"2024-02-05T08:19:31.481518+0000","last_undegraded":"2024-02-05T08:19:31.481518+0000","last_fullsized":"2024-02-05T08:19:31.481518+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:19:09.511509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:09.511509+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:20.442695+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"277'11","reported_seq":757,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.809730+0000","last_change":"2024-02-05T08:17:07.973477+0000","last_active":"2024-02-05T08:19:24.809730+0000","last_peered":"2024-02-05T08:19:24.809730+0000","last_clean":"2024-02-05T08:19:24.809730+0000","last_became_active":"2024-02-05T08:12:29.683091+0000","last_became_peered":"2024-02-05T08:12:29.683091+0000","last_unstale":"2024-02-05T08:19:24.809730+0000","last_undegraded":"2024-02-05T08:19:24.809730+0000","last_fullsized":"2024-02-05T08:19:24.809730+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_clean_scrub_stamp":"2024-02-05T08:17:07.973171+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"319'2128","reported_seq":2158,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:32.653003+0000","last_change":"2024-02-05T08:19:07.346393+0000","last_active":"2024-02-05T08:19:32.653003+0000","last_peered":"2024-02-05T08:19:32.653003+0000","last_clean":"2024-02-05T08:19:32.653003+0000","last_became_active":"2024-02-05T08:19:07.346024+0000","last_became_peered":"2024-02-05T08:19:07.346024+0000","last_unstale":"2024-02-05T08:19:32.653003+0000","last_undegraded":"2024-02-05T08:19:32.653003+0000","last_fullsized":"2024-02-05T08:19:32.653003+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2128,"log_dups_size":0,"ondisk_log_size":2128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:30.760245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139460608,"num_objects":2128,"num_object_clones":0,"num_object_copies":6384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2128,"num_write_kb":136192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'130","reported_seq":343,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.492098+0000","last_change":"2024-02-05T08:19:08.051637+0000","last_active":"2024-02-05T08:19:31.492098+0000","last_peered":"2024-02-05T08:19:31.492098+0000","last_clean":"2024-02-05T08:19:31.492098+0000","last_became_active":"2024-02-05T08:19:01.819856+0000","last_became_peered":"2024-02-05T08:19:01.819856+0000","last_unstale":"2024-02-05T08:19:31.492098+0000","last_undegraded":"2024-02-05T08:19:31.492098+0000","last_fullsized":"2024-02-05T08:19:31.492098+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:08.051550+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:08.051550+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:23.498386+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":570,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.483689+0000","last_change":"2024-02-05T08:18:36.117092+0000","last_active":"2024-02-05T08:19:31.483689+0000","last_peered":"2024-02-05T08:19:31.483689+0000","last_clean":"2024-02-05T08:19:31.483689+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:19:31.483689+0000","last_undegraded":"2024-02-05T08:19:31.483689+0000","last_fullsized":"2024-02-05T08:19:31.483689+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:36.116992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:36.116992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:49.466356+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":576,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-02-05T08:19:24.811261+0000","last_change":"2024-02-05T08:19:11.483166+0000","last_active":"2024-02-05T08:19:24.811261+0000","last_peered":"2024-02-05T08:19:24.811261+0000","last_clean":"2024-02-05T08:19:24.811261+0000","last_became_active":"2024-02-05T08:11:23.200753+0000","last_became_peered":"2024-02-05T08:11:23.200753+0000","last_unstale":"2024-02-05T08:19:24.811261+0000","last_undegraded":"2024-02-05T08:19:24.811261+0000","last_fullsized":"2024-02-05T08:19:24.811261+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:19:11.483052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:11.483052+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:32.079993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"236'28","reported_seq":651,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.486477+0000","last_change":"2024-02-05T08:19:30.389683+0000","last_active":"2024-02-05T08:19:31.486477+0000","last_peered":"2024-02-05T08:19:31.486477+0000","last_clean":"2024-02-05T08:19:31.486477+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:19:31.486477+0000","last_undegraded":"2024-02-05T08:19:31.486477+0000","last_fullsized":"2024-02-05T08:19:31.486477+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:19:30.389554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:30.389554+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:52.595376+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"295'11","reported_seq":76,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491160+0000","last_change":"2024-02-05T08:18:34.760490+0000","last_active":"2024-02-05T08:19:31.491160+0000","last_peered":"2024-02-05T08:19:31.491160+0000","last_clean":"2024-02-05T08:19:31.491160+0000","last_became_active":"2024-02-05T08:18:34.664016+0000","last_became_peered":"2024-02-05T08:18:34.664016+0000","last_unstale":"2024-02-05T08:19:31.491160+0000","last_undegraded":"2024-02-05T08:19:31.491160+0000","last_fullsized":"2024-02-05T08:19:31.491160+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'10","last_scrub_stamp":"2024-02-05T08:17:14.040769+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:17:14.040769+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"319'2076","reported_seq":2106,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:32.872893+0000","last_change":"2024-02-05T08:19:07.331856+0000","last_active":"2024-02-05T08:19:32.872893+0000","last_peered":"2024-02-05T08:19:32.872893+0000","last_clean":"2024-02-05T08:19:32.872893+0000","last_became_active":"2024-02-05T08:19:07.331338+0000","last_became_peered":"2024-02-05T08:19:07.331338+0000","last_unstale":"2024-02-05T08:19:32.872893+0000","last_undegraded":"2024-02-05T08:19:32.872893+0000","last_fullsized":"2024-02-05T08:19:32.872893+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2076,"log_dups_size":0,"ondisk_log_size":2076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:25.737803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136052736,"num_objects":2076,"num_object_clones":0,"num_object_copies":6228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2076,"num_write_kb":132864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'130","reported_seq":343,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.486062+0000","last_change":"2024-02-05T08:19:08.403635+0000","last_active":"2024-02-05T08:19:31.486062+0000","last_peered":"2024-02-05T08:19:31.486062+0000","last_clean":"2024-02-05T08:19:31.486062+0000","last_became_active":"2024-02-05T08:19:01.816923+0000","last_became_peered":"2024-02-05T08:19:01.816923+0000","last_unstale":"2024-02-05T08:19:31.486062+0000","last_undegraded":"2024-02-05T08:19:31.486062+0000","last_fullsized":"2024-02-05T08:19:31.486062+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:08.403509+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:08.403509+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:32.649574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":577,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.485756+0000","last_change":"2024-02-05T08:19:21.335021+0000","last_active":"2024-02-05T08:19:31.485756+0000","last_peered":"2024-02-05T08:19:31.485756+0000","last_clean":"2024-02-05T08:19:31.485756+0000","last_became_active":"2024-02-05T08:18:34.635908+0000","last_became_peered":"2024-02-05T08:18:34.635908+0000","last_unstale":"2024-02-05T08:19:31.485756+0000","last_undegraded":"2024-02-05T08:19:31.485756+0000","last_fullsized":"2024-02-05T08:19:31.485756+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:21.334902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:21.334902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:25.566399+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":63,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.493778+0000","last_change":"2024-02-05T08:18:34.735709+0000","last_active":"2024-02-05T08:19:31.493778+0000","last_peered":"2024-02-05T08:19:31.493778+0000","last_clean":"2024-02-05T08:19:31.493778+0000","last_became_active":"2024-02-05T08:18:34.649808+0000","last_became_peered":"2024-02-05T08:18:34.649808+0000","last_unstale":"2024-02-05T08:19:31.493778+0000","last_undegraded":"2024-02-05T08:19:31.493778+0000","last_fullsized":"2024-02-05T08:19:31.493778+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:17:52.089740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:17:52.089740+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"236'38","reported_seq":85,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490847+0000","last_change":"2024-02-05T08:19:15.091633+0000","last_active":"2024-02-05T08:19:31.490847+0000","last_peered":"2024-02-05T08:19:31.490847+0000","last_clean":"2024-02-05T08:19:31.490847+0000","last_became_active":"2024-02-05T08:18:34.653505+0000","last_became_peered":"2024-02-05T08:18:34.653505+0000","last_unstale":"2024-02-05T08:19:31.490847+0000","last_undegraded":"2024-02-05T08:19:31.490847+0000","last_fullsized":"2024-02-05T08:19:31.490847+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'38","last_scrub_stamp":"2024-02-05T08:19:15.091527+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.091527+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:35.814778+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":573,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.486074+0000","last_change":"2024-02-05T08:18:42.514623+0000","last_active":"2024-02-05T08:19:31.486074+0000","last_peered":"2024-02-05T08:19:31.486074+0000","last_clean":"2024-02-05T08:19:31.486074+0000","last_became_active":"2024-02-05T08:18:34.636667+0000","last_became_peered":"2024-02-05T08:18:34.636667+0000","last_unstale":"2024-02-05T08:19:31.486074+0000","last_undegraded":"2024-02-05T08:19:31.486074+0000","last_fullsized":"2024-02-05T08:19:31.486074+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:42.514544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:42.514544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:08.965745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"319'2122","reported_seq":2152,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:32.892084+0000","last_change":"2024-02-05T08:19:07.330853+0000","last_active":"2024-02-05T08:19:32.892084+0000","last_peered":"2024-02-05T08:19:32.892084+0000","last_clean":"2024-02-05T08:19:32.892084+0000","last_became_active":"2024-02-05T08:19:07.330566+0000","last_became_peered":"2024-02-05T08:19:07.330566+0000","last_unstale":"2024-02-05T08:19:32.892084+0000","last_undegraded":"2024-02-05T08:19:32.892084+0000","last_fullsized":"2024-02-05T08:19:32.892084+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2122,"log_dups_size":0,"ondisk_log_size":2122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:11.046503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139067392,"num_objects":2122,"num_object_clones":0,"num_object_copies":6366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2122,"num_write_kb":135808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'130","reported_seq":343,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491359+0000","last_change":"2024-02-05T08:19:09.444525+0000","last_active":"2024-02-05T08:19:31.491359+0000","last_peered":"2024-02-05T08:19:31.491359+0000","last_clean":"2024-02-05T08:19:31.491359+0000","last_became_active":"2024-02-05T08:19:01.817169+0000","last_became_peered":"2024-02-05T08:19:01.817169+0000","last_unstale":"2024-02-05T08:19:31.491359+0000","last_undegraded":"2024-02-05T08:19:31.491359+0000","last_fullsized":"2024-02-05T08:19:31.491359+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:09.444410+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:09.444410+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:26.321250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":564,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490831+0000","last_change":"2024-02-05T08:18:34.613605+0000","last_active":"2024-02-05T08:19:31.490831+0000","last_peered":"2024-02-05T08:19:31.490831+0000","last_clean":"2024-02-05T08:19:31.490831+0000","last_became_active":"2024-02-05T08:04:47.584093+0000","last_became_peered":"2024-02-05T08:04:47.584093+0000","last_unstale":"2024-02-05T08:19:31.490831+0000","last_undegraded":"2024-02-05T08:19:31.490831+0000","last_fullsized":"2024-02-05T08:19:31.490831+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.613476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.613476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:54.411772+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":747,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.489227+0000","last_change":"2024-02-05T08:19:15.267565+0000","last_active":"2024-02-05T08:19:31.489227+0000","last_peered":"2024-02-05T08:19:31.489227+0000","last_clean":"2024-02-05T08:19:31.489227+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:19:31.489227+0000","last_undegraded":"2024-02-05T08:19:31.489227+0000","last_fullsized":"2024-02-05T08:19:31.489227+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:19:15.267486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.267486+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:30.495017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":67,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490937+0000","last_change":"2024-02-05T08:19:07.027597+0000","last_active":"2024-02-05T08:19:31.490937+0000","last_peered":"2024-02-05T08:19:31.490937+0000","last_clean":"2024-02-05T08:19:31.490937+0000","last_became_active":"2024-02-05T08:18:34.652764+0000","last_became_peered":"2024-02-05T08:18:34.652764+0000","last_unstale":"2024-02-05T08:19:31.490937+0000","last_undegraded":"2024-02-05T08:19:31.490937+0000","last_fullsized":"2024-02-05T08:19:31.490937+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:07.027521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:07.027521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:23.743840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"236'27","reported_seq":660,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.482801+0000","last_change":"2024-02-05T08:18:40.384275+0000","last_active":"2024-02-05T08:19:31.482801+0000","last_peered":"2024-02-05T08:19:31.482801+0000","last_clean":"2024-02-05T08:19:31.482801+0000","last_became_active":"2024-02-05T08:18:34.658318+0000","last_became_peered":"2024-02-05T08:18:34.658318+0000","last_unstale":"2024-02-05T08:19:31.482801+0000","last_undegraded":"2024-02-05T08:19:31.482801+0000","last_fullsized":"2024-02-05T08:19:31.482801+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:18:40.384115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:40.384115+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:45.631489+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"319'2088","reported_seq":2118,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:32.659393+0000","last_change":"2024-02-05T08:19:07.331816+0000","last_active":"2024-02-05T08:19:32.659393+0000","last_peered":"2024-02-05T08:19:32.659393+0000","last_clean":"2024-02-05T08:19:32.659393+0000","last_became_active":"2024-02-05T08:19:07.331439+0000","last_became_peered":"2024-02-05T08:19:07.331439+0000","last_unstale":"2024-02-05T08:19:32.659393+0000","last_undegraded":"2024-02-05T08:19:32.659393+0000","last_fullsized":"2024-02-05T08:19:32.659393+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2088,"log_dups_size":0,"ondisk_log_size":2088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:19.515541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136839168,"num_objects":2088,"num_object_clones":0,"num_object_copies":6264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":133632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":67,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.492306+0000","last_change":"2024-02-05T08:19:10.027572+0000","last_active":"2024-02-05T08:19:31.492306+0000","last_peered":"2024-02-05T08:19:31.492306+0000","last_clean":"2024-02-05T08:19:31.492306+0000","last_became_active":"2024-02-05T08:18:34.627261+0000","last_became_peered":"2024-02-05T08:18:34.627261+0000","last_unstale":"2024-02-05T08:19:31.492306+0000","last_undegraded":"2024-02-05T08:19:31.492306+0000","last_fullsized":"2024-02-05T08:19:31.492306+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:10.027469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:19:10.027469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:38.513356+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":580,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.493658+0000","last_change":"2024-02-05T08:18:45.518432+0000","last_active":"2024-02-05T08:19:31.493658+0000","last_peered":"2024-02-05T08:19:31.493658+0000","last_clean":"2024-02-05T08:19:31.493658+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:19:31.493658+0000","last_undegraded":"2024-02-05T08:19:31.493658+0000","last_fullsized":"2024-02-05T08:19:31.493658+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:45.518363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:45.518363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:13.740354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":576,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.486158+0000","last_change":"2024-02-05T08:19:13.275219+0000","last_active":"2024-02-05T08:19:31.486158+0000","last_peered":"2024-02-05T08:19:31.486158+0000","last_clean":"2024-02-05T08:19:31.486158+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:19:31.486158+0000","last_undegraded":"2024-02-05T08:19:31.486158+0000","last_fullsized":"2024-02-05T08:19:31.486158+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:13.275109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:19:13.275109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:41.604604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"236'19","reported_seq":631,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.489079+0000","last_change":"2024-02-05T08:18:36.561959+0000","last_active":"2024-02-05T08:19:31.489079+0000","last_peered":"2024-02-05T08:19:31.489079+0000","last_clean":"2024-02-05T08:19:31.489079+0000","last_became_active":"2024-02-05T08:12:29.666284+0000","last_became_peered":"2024-02-05T08:12:29.666284+0000","last_unstale":"2024-02-05T08:19:31.489079+0000","last_undegraded":"2024-02-05T08:19:31.489079+0000","last_fullsized":"2024-02-05T08:19:31.489079+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:36.561837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:36.561837+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:42.609439+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"319'2149","reported_seq":2179,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:33.140636+0000","last_change":"2024-02-05T08:19:07.346368+0000","last_active":"2024-02-05T08:19:33.140636+0000","last_peered":"2024-02-05T08:19:33.140636+0000","last_clean":"2024-02-05T08:19:33.140636+0000","last_became_active":"2024-02-05T08:19:07.346034+0000","last_became_peered":"2024-02-05T08:19:07.346034+0000","last_unstale":"2024-02-05T08:19:33.140636+0000","last_undegraded":"2024-02-05T08:19:33.140636+0000","last_fullsized":"2024-02-05T08:19:33.140636+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2149,"log_dups_size":0,"ondisk_log_size":2149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:24.457034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140836864,"num_objects":2149,"num_object_clones":0,"num_object_copies":6447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2149,"num_write_kb":137536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'130","reported_seq":343,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490041+0000","last_change":"2024-02-05T08:19:10.494455+0000","last_active":"2024-02-05T08:19:31.490041+0000","last_peered":"2024-02-05T08:19:31.490041+0000","last_clean":"2024-02-05T08:19:31.490041+0000","last_became_active":"2024-02-05T08:19:01.817126+0000","last_became_peered":"2024-02-05T08:19:01.817126+0000","last_unstale":"2024-02-05T08:19:31.490041+0000","last_undegraded":"2024-02-05T08:19:31.490041+0000","last_fullsized":"2024-02-05T08:19:31.490041+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:10.494376+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:10.494376+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:28.065263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":593,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.491431+0000","last_change":"2024-02-05T08:18:36.936749+0000","last_active":"2024-02-05T08:19:31.491431+0000","last_peered":"2024-02-05T08:19:31.491431+0000","last_clean":"2024-02-05T08:19:31.491431+0000","last_became_active":"2024-02-05T08:18:34.645067+0000","last_became_peered":"2024-02-05T08:18:34.645067+0000","last_unstale":"2024-02-05T08:19:31.491431+0000","last_undegraded":"2024-02-05T08:19:31.491431+0000","last_fullsized":"2024-02-05T08:19:31.491431+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:18:36.936635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:36.936635+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:48.454947+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":561,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.481712+0000","last_change":"2024-02-05T08:18:45.429376+0000","last_active":"2024-02-05T08:19:31.481712+0000","last_peered":"2024-02-05T08:19:31.481712+0000","last_clean":"2024-02-05T08:19:31.481712+0000","last_became_active":"2024-02-05T08:04:45.559298+0000","last_became_peered":"2024-02-05T08:04:45.559298+0000","last_unstale":"2024-02-05T08:19:31.481712+0000","last_undegraded":"2024-02-05T08:19:31.481712+0000","last_fullsized":"2024-02-05T08:19:31.481712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:45.429303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:45.429303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:08.219628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":306,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.487717+0000","last_change":"2024-02-05T08:19:10.372396+0000","last_active":"2024-02-05T08:19:31.487717+0000","last_peered":"2024-02-05T08:19:31.487717+0000","last_clean":"2024-02-05T08:19:31.487717+0000","last_became_active":"2024-02-05T08:12:29.678501+0000","last_became_peered":"2024-02-05T08:12:29.678501+0000","last_unstale":"2024-02-05T08:19:31.487717+0000","last_undegraded":"2024-02-05T08:19:31.487717+0000","last_fullsized":"2024-02-05T08:19:31.487717+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:10.372287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:19:10.372287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:36.737501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"236'31","reported_seq":613,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.488586+0000","last_change":"2024-02-05T08:19:24.342892+0000","last_active":"2024-02-05T08:19:31.488586+0000","last_peered":"2024-02-05T08:19:31.488586+0000","last_clean":"2024-02-05T08:19:31.488586+0000","last_became_active":"2024-02-05T08:04:43.554897+0000","last_became_peered":"2024-02-05T08:04:43.554897+0000","last_unstale":"2024-02-05T08:19:31.488586+0000","last_undegraded":"2024-02-05T08:19:31.488586+0000","last_fullsized":"2024-02-05T08:19:31.488586+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'31","last_scrub_stamp":"2024-02-05T08:19:24.342770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:24.342770+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:52.037793+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"319'2184","reported_seq":2213,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:33.424352+0000","last_change":"2024-02-05T08:19:07.325738+0000","last_active":"2024-02-05T08:19:33.424352+0000","last_peered":"2024-02-05T08:19:33.424352+0000","last_clean":"2024-02-05T08:19:33.424352+0000","last_became_active":"2024-02-05T08:19:07.325409+0000","last_became_peered":"2024-02-05T08:19:07.325409+0000","last_unstale":"2024-02-05T08:19:33.424352+0000","last_undegraded":"2024-02-05T08:19:33.424352+0000","last_fullsized":"2024-02-05T08:19:33.424352+0000","mapping_epoch":309,"log_start":"0'0","ondisk_log_start":"0'0","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":2184,"log_dups_size":0,"ondisk_log_size":2184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:33.552161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143130624,"num_objects":2184,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2184,"num_write_kb":139776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"82'130","reported_seq":875,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.482038+0000","last_change":"2024-02-05T08:19:09.153997+0000","last_active":"2024-02-05T08:19:31.482038+0000","last_peered":"2024-02-05T08:19:31.482038+0000","last_clean":"2024-02-05T08:19:31.482038+0000","last_became_active":"2024-02-05T08:19:01.816744+0000","last_became_peered":"2024-02-05T08:19:01.816744+0000","last_unstale":"2024-02-05T08:19:31.482038+0000","last_undegraded":"2024-02-05T08:19:31.482038+0000","last_fullsized":"2024-02-05T08:19:31.482038+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:09.153860+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:09.153860+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:31.128753+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":556,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.489204+0000","last_change":"2024-02-05T08:19:16.282756+0000","last_active":"2024-02-05T08:19:31.489204+0000","last_peered":"2024-02-05T08:19:31.489204+0000","last_clean":"2024-02-05T08:19:31.489204+0000","last_became_active":"2024-02-05T08:04:47.573359+0000","last_became_peered":"2024-02-05T08:04:47.573359+0000","last_unstale":"2024-02-05T08:19:31.489204+0000","last_undegraded":"2024-02-05T08:19:31.489204+0000","last_fullsized":"2024-02-05T08:19:31.489204+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.282641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.282641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:30.855865+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":764,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.490512+0000","last_change":"2024-02-05T08:18:29.716104+0000","last_active":"2024-02-05T08:19:31.490512+0000","last_peered":"2024-02-05T08:19:31.490512+0000","last_clean":"2024-02-05T08:19:31.490512+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:19:31.490512+0000","last_undegraded":"2024-02-05T08:19:31.490512+0000","last_fullsized":"2024-02-05T08:19:31.490512+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:18:29.715977+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:18:29.715977+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:40.647242+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":67,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.492335+0000","last_change":"2024-02-05T08:18:34.931257+0000","last_active":"2024-02-05T08:19:31.492335+0000","last_peered":"2024-02-05T08:19:31.492335+0000","last_clean":"2024-02-05T08:19:31.492335+0000","last_became_active":"2024-02-05T08:18:34.663738+0000","last_became_peered":"2024-02-05T08:18:34.663738+0000","last_unstale":"2024-02-05T08:19:31.492335+0000","last_undegraded":"2024-02-05T08:19:31.492335+0000","last_fullsized":"2024-02-05T08:19:31.492335+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.931138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.931138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:19:47.724113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"236'20","reported_seq":666,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-02-05T08:19:31.487881+0000","last_change":"2024-02-05T08:18:34.732210+0000","last_active":"2024-02-05T08:19:31.487881+0000","last_peered":"2024-02-05T08:19:31.487881+0000","last_clean":"2024-02-05T08:19:31.487881+0000","last_became_active":"2024-02-05T08:18:34.637413+0000","last_became_peered":"2024-02-05T08:18:34.637413+0000","last_unstale":"2024-02-05T08:19:31.487881+0000","last_undegraded":"2024-02-05T08:19:31.487881+0000","last_fullsized":"2024-02-05T08:19:31.487881+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:18:00.684227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:00.684227+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2145517568,"num_objects":32738,"num_object_clones":0,"num_object_copies":98214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32738,"num_write_kb":2095232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6315180032,"data_stored":6315180032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32738,"ondisk_log_size":32738,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":879,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":311,"num_bytes_recovered":5930,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":765,"ondisk_log_size":765,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037943,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1090464,"kb_used_data":545620,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92654688,"statfs":{"total":95995035648,"available":94878400512,"internally_reserved":0,"allocated":558714880,"data_stored":558307631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":557906378},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":1.2809999999999999,"15min":1.516},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.181,"5min":27.376000000000001,"15min":60.127000000000002},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.486,"15min":1.589},"min":{"1min":0.27600000000000002,"5min":0.23200000000000001,"15min":0.22500000000000001},"max":{"1min":2.4609999999999999,"5min":27.456,"15min":58.283000000000001},"last":1.2270000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.544,"15min":1.512},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.22900000000000001},"max":{"1min":2.7810000000000001,"5min":27.518999999999998,"15min":38.399999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.083,"5min":1.4330000000000001,"15min":1.3680000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.22500000000000001},"max":{"1min":4.2409999999999997,"5min":27.503,"15min":38.195999999999998},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.089,"15min":0.98799999999999999},"min":{"1min":0.314,"5min":0.26100000000000001,"15min":0.24099999999999999},"max":{"1min":4.2140000000000004,"5min":7.5709999999999997,"15min":7.6779999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":1.145,"15min":1.087},"min":{"1min":0.33000000000000002,"5min":0.26200000000000001,"15min":0.24299999999999999},"max":{"1min":2.5419999999999998,"5min":5.9400000000000004,"15min":8.9100000000000001},"last":1.2010000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.1259999999999999,"15min":1.3109999999999999},"min":{"1min":0.36299999999999999,"5min":0.29699999999999999,"15min":0.249},"max":{"1min":2.5830000000000002,"5min":5.984,"15min":58.515000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.119,"15min":1.296},"min":{"1min":0.32500000000000001,"5min":0.30199999999999999,"15min":0.27000000000000002},"max":{"1min":2.4630000000000001,"5min":5.391,"15min":58.225999999999999},"last":0.59499999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.1819999999999999,"15min":0.91600000000000004},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.22},"max":{"1min":4.1449999999999996,"5min":7.6340000000000003,"15min":49.601999999999997},"last":0.496},{"interface":"front","average":{"1min":0.85899999999999999,"5min":1.147,"15min":0.93200000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":2.9260000000000002,"5min":7.8399999999999999,"15min":49.795999999999999},"last":0.55300000000000005}]},{"osd":5,"last update":"Mon Feb 5 08:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.504,"15min":1.0029999999999999},"min":{"1min":0.31900000000000001,"5min":0.30099999999999999,"15min":0.24299999999999999},"max":{"1min":4.5579999999999998,"5min":8.0709999999999997,"15min":49.914000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.5509999999999999,"15min":1.026},"min":{"1min":0.308,"5min":0.308,"15min":0.27000000000000002},"max":{"1min":9.4459999999999997,"5min":9.4459999999999997,"15min":52.256},"last":0.52200000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:18:55 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":1.3180000000000001,"15min":0.79100000000000004},"min":{"1min":0.29199999999999998,"5min":0.28799999999999998,"15min":0.28599999999999998},"max":{"1min":2.488,"5min":8.0050000000000008,"15min":8.0050000000000008},"last":1.1259999999999999},{"interface":"front","average":{"1min":0.92100000000000004,"5min":1.28,"15min":0.76500000000000001},"min":{"1min":0.42799999999999999,"5min":0.32500000000000001,"15min":0.27300000000000002},"max":{"1min":2.871,"5min":7.8029999999999999,"15min":7.8029999999999999},"last":0.81599999999999995}]}]},{"osd":6,"up_from":51,"seq":219043332285,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1239816,"kb_used_data":925440,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":92505336,"statfs":{"total":95995035648,"available":94725464064,"internally_reserved":0,"allocated":947650560,"data_stored":947262264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":321911238},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.6459999999999999,"15min":1.724},"min":{"1min":0.26700000000000002,"5min":0.215,"15min":0.215},"max":{"1min":5.3390000000000004,"5min":76.123000000000005,"15min":102.819},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.6080000000000001,"15min":1.7869999999999999},"min":{"1min":0.30399999999999999,"5min":0.22600000000000001,"15min":0.214},"max":{"1min":4.9480000000000004,"5min":76.037000000000006,"15min":102.75},"last":0.307}]},{"osd":1,"last update":"Mon Feb 5 08:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.90900000000000003,"15min":1.169},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.22700000000000001},"max":{"1min":1.1970000000000001,"5min":6.0229999999999997,"15min":23.773},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.96299999999999997,"15min":1.1319999999999999},"min":{"1min":0.30299999999999999,"5min":0.26300000000000001,"15min":0.23400000000000001},"max":{"1min":3.0249999999999999,"5min":5.0209999999999999,"15min":23.719000000000001},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Feb 5 08:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.86599999999999999,"15min":1.0620000000000001},"min":{"1min":0.39500000000000002,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":3.0680000000000001,"5min":4.6580000000000004,"15min":22.503},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.93500000000000005,"15min":1.0089999999999999},"min":{"1min":0.41299999999999998,"5min":0.27300000000000002,"15min":0.23400000000000001},"max":{"1min":4.9980000000000002,"5min":4.9980000000000002,"15min":22.466000000000001},"last":0.41899999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.82899999999999996,"15min":1.327},"min":{"1min":0.35399999999999998,"5min":0.312,"15min":0.27100000000000002},"max":{"1min":1.4590000000000001,"5min":3.4689999999999999,"15min":102.438},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":1.3500000000000001},"min":{"1min":0.373,"5min":0.32200000000000001,"15min":0.29099999999999998},"max":{"1min":3.093,"5min":4.4939999999999998,"15min":102.288},"last":0.373}]},{"osd":4,"last update":"Mon Feb 5 08:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":1.02,"15min":0.68700000000000006},"min":{"1min":0.251,"5min":0.251,"15min":0.223},"max":{"1min":4.9740000000000002,"5min":4.9740000000000002,"15min":9.8970000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.95899999999999996,"15min":0.71099999999999997},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.26600000000000001},"max":{"1min":3.633,"5min":4.8579999999999997,"15min":4.9589999999999996},"last":0.48499999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:19:29 2024","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":1.494,"15min":1.0449999999999999},"min":{"1min":0.33900000000000002,"5min":0.29299999999999998,"15min":0.24199999999999999},"max":{"1min":5.0970000000000004,"5min":24.675999999999998,"15min":43.643000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.82899999999999996,"5min":1.421,"15min":1.026},"min":{"1min":0.35999999999999999,"5min":0.32300000000000001,"15min":0.30399999999999999},"max":{"1min":4.5759999999999996,"5min":19.132000000000001,"15min":43.893000000000001},"last":0.504}]},{"osd":7,"last update":"Mon Feb 5 08:19:04 2024","interfaces":[{"interface":"back","average":{"1min":1.8740000000000001,"5min":1.621,"15min":0.91300000000000003},"min":{"1min":0.38700000000000001,"5min":0.372,"15min":0.29999999999999999},"max":{"1min":9.0229999999999997,"5min":24.728999999999999,"15min":24.728999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.895,"5min":1.6759999999999999,"15min":0.93999999999999995},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.29299999999999998},"max":{"1min":14.244999999999999,"5min":21.91,"15min":21.91},"last":0.52200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575062,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":960884,"kb_used_data":402152,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92784268,"statfs":{"total":95995035648,"available":95011090432,"internally_reserved":0,"allocated":411803648,"data_stored":411375517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1659999999999999,"15min":1.625},"min":{"1min":0.221,"5min":0.183,"15min":0.183},"max":{"1min":1.0660000000000001,"5min":36.462000000000003,"15min":86.105000000000004},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.331,"15min":1.907},"min":{"1min":0.26300000000000001,"5min":0.21099999999999999,"15min":0.20699999999999999},"max":{"1min":8.0340000000000007,"5min":35.878,"15min":85.665000000000006},"last":0.754}]},{"osd":2,"last update":"Mon Feb 5 08:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.84299999999999997,"15min":1.373},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.191},"max":{"1min":1.4930000000000001,"5min":4.2859999999999996,"15min":49.030999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.2210000000000001,"5min":0.92100000000000004,"15min":1.599},"min":{"1min":0.255,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":7.298,"5min":7.298,"15min":48.984000000000002},"last":0.68100000000000005}]},{"osd":3,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.754,"15min":1.5189999999999999},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.20599999999999999},"max":{"1min":1.026,"5min":5.3289999999999997,"15min":49.067999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.81699999999999995,"15min":1.46},"min":{"1min":0.32100000000000001,"5min":0.254,"15min":0.24199999999999999},"max":{"1min":2.3119999999999998,"5min":4.3529999999999998,"15min":48.734999999999999},"last":1.091}]},{"osd":4,"last update":"Mon Feb 5 08:18:37 2024","interfaces":[{"interface":"back","average":{"1min":2.5779999999999998,"5min":1.5600000000000001,"15min":1.268},"min":{"1min":0.371,"5min":0.33100000000000002,"15min":0.26000000000000001},"max":{"1min":24.196999999999999,"5min":24.196999999999999,"15min":25.952000000000002},"last":1.2030000000000001},{"interface":"front","average":{"1min":2.4510000000000001,"5min":1.5089999999999999,"15min":1.3560000000000001},"min":{"1min":0.314,"5min":0.31,"15min":0.27100000000000002},"max":{"1min":24.215,"5min":24.215,"15min":25.922000000000001},"last":0.92100000000000004}]},{"osd":5,"last update":"Mon Feb 5 08:18:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.492,"15min":1.3999999999999999},"min":{"1min":0.40100000000000002,"5min":0.39400000000000002,"15min":0.28699999999999998},"max":{"1min":3.1699999999999999,"5min":9.6820000000000004,"15min":54.552999999999997},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.296,"15min":1.4379999999999999},"min":{"1min":0.32000000000000001,"5min":0.311,"15min":0.28999999999999998},"max":{"1min":4.1239999999999997,"5min":6.0259999999999998,"15min":55.811},"last":0.72099999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:18:37 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.528,"15min":1.2330000000000001},"min":{"1min":0.41399999999999998,"5min":0.35199999999999998,"15min":0.33100000000000002},"max":{"1min":2.968,"5min":54.384,"15min":54.384},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.615,"15min":1.244},"min":{"1min":0.37,"5min":0.36899999999999999,"15min":0.34899999999999998},"max":{"1min":2.8769999999999998,"5min":55.210999999999999,"15min":55.210999999999999},"last":1.2410000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:18:52 2024","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.45,"15min":1.3089999999999999},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":5.7640000000000002,"5min":6.0579999999999998,"15min":13.749000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.3280000000000001,"15min":1.242},"min":{"1min":0.42399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":2.4020000000000001,"5min":4.6180000000000003,"15min":7.9640000000000004},"last":1.167}]}]},{"osd":0,"up_from":8,"seq":34359738586,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1498900,"kb_used_data":976268,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92246252,"statfs":{"total":95995035648,"available":94460162048,"internally_reserved":0,"allocated":999698432,"data_stored":999290842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:18:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.86899999999999999,"15min":1.7909999999999999},"min":{"1min":0.20499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":2.2999999999999998,"5min":7.9580000000000002,"15min":66.772999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.0389999999999999,"15min":1.53},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":3.7050000000000001,"5min":7.4850000000000003,"15min":46.198999999999998},"last":0.69799999999999995}]},{"osd":2,"last update":"Mon Feb 5 08:18:50 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.94699999999999995,"15min":1.518},"min":{"1min":0.246,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.2410000000000001,"5min":8.75,"15min":46.170999999999999},"last":4.0149999999999997},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.78300000000000003,"15min":1.655},"min":{"1min":0.29199999999999998,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":2.4049999999999998,"5min":3.1360000000000001,"15min":46.543999999999997},"last":2.3290000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:19:27 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.79800000000000004,"15min":1.706},"min":{"1min":0.27900000000000003,"5min":0.24399999999999999,"15min":0.21099999999999999},"max":{"1min":3.8130000000000002,"5min":6.7439999999999998,"15min":67.888000000000005},"last":3.8130000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.70699999999999996,"15min":1.9890000000000001},"min":{"1min":0.32700000000000001,"5min":0.23999999999999999,"15min":0.22600000000000001},"max":{"1min":1.3939999999999999,"5min":3.1709999999999998,"15min":66.891999999999996},"last":0.57599999999999996}]},{"osd":4,"last update":"Mon Feb 5 08:18:46 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.2709999999999999,"15min":1.248},"min":{"1min":0.32400000000000001,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":3.629,"5min":22.920000000000002,"15min":22.920000000000002},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.014,"5min":1.3360000000000001,"15min":1.22},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.26900000000000002},"max":{"1min":3.081,"5min":22.983000000000001,"15min":22.983000000000001},"last":4.5979999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:19:08 2024","interfaces":[{"interface":"back","average":{"1min":2.573,"5min":1.8049999999999999,"15min":1.4730000000000001},"min":{"1min":0.40400000000000003,"5min":0.377,"15min":0.28299999999999997},"max":{"1min":38.979999999999997,"5min":38.979999999999997,"15min":38.979999999999997},"last":1.6970000000000001},{"interface":"front","average":{"1min":2.7320000000000002,"5min":1.8320000000000001,"15min":1.4059999999999999},"min":{"1min":0.42499999999999999,"5min":0.35999999999999999,"15min":0.29599999999999999},"max":{"1min":38.960999999999999,"5min":38.960999999999999,"15min":38.960999999999999},"last":3.9129999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:18:31 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.1479999999999999,"15min":1.167},"min":{"1min":0.35299999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.113,"5min":7.8220000000000001,"15min":12.967000000000001},"last":4.0970000000000004},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.1899999999999999,"15min":1.266},"min":{"1min":0.50700000000000001,"5min":0.34200000000000003,"15min":0.32300000000000001},"max":{"1min":3.6619999999999999,"5min":4.1210000000000004,"15min":15.721},"last":1.73}]},{"osd":7,"last update":"Mon Feb 5 08:18:50 2024","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.4870000000000001,"15min":1.6659999999999999},"min":{"1min":0.35199999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":25.18,"5min":68.241,"15min":68.241},"last":1.2849999999999999},{"interface":"front","average":{"1min":2.5510000000000002,"5min":2.5640000000000001,"15min":1.706},"min":{"1min":0.47199999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":25.373999999999999,"5min":68.980000000000004,"15min":68.980000000000004},"last":3.956}]}]},{"osd":2,"up_from":18,"seq":77309411537,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1301608,"kb_used_data":788700,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":92443544,"statfs":{"total":95995035648,"available":94662189056,"internally_reserved":0,"allocated":807628800,"data_stored":807213099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":525203910},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:18:54 2024","interfaces":[{"interface":"back","average":{"1min":2.238,"5min":2.3199999999999998,"15min":2.1480000000000001},"min":{"1min":0.222,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":30.731999999999999,"5min":107.155,"15min":107.155},"last":1.8879999999999999},{"interface":"front","average":{"1min":2.3919999999999999,"5min":2.528,"15min":2.3079999999999998},"min":{"1min":0.23100000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":30.838999999999999,"5min":107.221,"15min":107.221},"last":3.0990000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:18:54 2024","interfaces":[{"interface":"back","average":{"1min":2.4540000000000002,"5min":1.167,"15min":1.5860000000000001},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.17999999999999999},"max":{"1min":35.695999999999998,"5min":35.695999999999998,"15min":55.942},"last":2.7360000000000002},{"interface":"front","average":{"1min":2.9209999999999998,"5min":1.581,"15min":1.752},"min":{"1min":0.216,"5min":0.216,"15min":0.193},"max":{"1min":30.765000000000001,"5min":30.765000000000001,"15min":55.960999999999999},"last":1.49}]},{"osd":3,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.83199999999999996,"15min":1.792},"min":{"1min":0.222,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":3.6920000000000002,"5min":8.3629999999999995,"15min":68.108000000000004},"last":1.4430000000000001},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.63800000000000001,"15min":1.587},"min":{"1min":0.30299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":6.6920000000000002,"5min":6.6920000000000002,"15min":67.945999999999998},"last":1.8540000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:18:40 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.3089999999999999,"15min":1.385},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.27400000000000002},"max":{"1min":4.101,"5min":8.2620000000000005,"15min":31.596},"last":1.526},{"interface":"front","average":{"1min":1.516,"5min":1.1910000000000001,"15min":1.21},"min":{"1min":0.32800000000000001,"5min":0.30299999999999999,"15min":0.252},"max":{"1min":9.5060000000000002,"5min":9.5060000000000002,"15min":31.57},"last":1.7430000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:19:06 2024","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.7070000000000001,"15min":1.5600000000000001},"min":{"1min":0.33500000000000002,"5min":0.246,"15min":0.246},"max":{"1min":9.5500000000000007,"5min":26.373000000000001,"15min":58.799999999999997},"last":1.8260000000000001},{"interface":"front","average":{"1min":1.45,"5min":1.46,"15min":1.4710000000000001},"min":{"1min":0.32300000000000001,"5min":0.30399999999999999,"15min":0.26900000000000002},"max":{"1min":12.925000000000001,"5min":27.968,"15min":58.615000000000002},"last":2.6709999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:18:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1830000000000001,"15min":1.2250000000000001},"min":{"1min":0.37,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.202,"5min":8.3089999999999993,"15min":15.859},"last":1.6559999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.127,"15min":1.089},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.32900000000000001},"max":{"1min":9.1739999999999995,"5min":9.1739999999999995,"15min":9.1739999999999995},"last":1.7889999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:18:54 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.206,"15min":1.2350000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.5149999999999997,"5min":7.7530000000000001,"15min":15.986000000000001},"last":2.71},{"interface":"front","average":{"1min":1.905,"5min":1.633,"15min":1.4410000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":11.945,"5min":11.945,"15min":15.951000000000001},"last":1.599}]}]},{"osd":3,"up_from":27,"seq":115964117196,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1083192,"kb_used_data":677164,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92661960,"statfs":{"total":95995035648,"available":94885847040,"internally_reserved":0,"allocated":693415936,"data_stored":692984508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":415758793},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":1.1899999999999999,"15min":1.752},"min":{"1min":0.246,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":1.034,"5min":52.831000000000003,"15min":63.43},"last":11.747999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.3740000000000001,"15min":1.879},"min":{"1min":0.246,"5min":0.20499999999999999,"15min":0.184},"max":{"1min":6.6779999999999999,"5min":53.859999999999999,"15min":63.048999999999999},"last":11.618}]},{"osd":1,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.93500000000000005,"15min":1.5109999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.217},"max":{"1min":3.3119999999999998,"5min":23.154,"15min":51.405999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.89200000000000002,"15min":1.8260000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.17399999999999999},"max":{"1min":3.54,"5min":23.207000000000001,"15min":51.189999999999998},"last":0.34100000000000003}]},{"osd":2,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.625,"15min":1.526},"min":{"1min":0.27200000000000002,"5min":0.251,"15min":0.251},"max":{"1min":3.5649999999999999,"5min":4.0170000000000003,"15min":62.902999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.57399999999999995,"15min":1.339},"min":{"1min":0.249,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.83499999999999996,"5min":1.931,"15min":63.468000000000004},"last":0.309}]},{"osd":4,"last update":"Mon Feb 5 08:18:39 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.997,"15min":1.056},"min":{"1min":0.33200000000000002,"5min":0.30499999999999999,"15min":0.28799999999999998},"max":{"1min":3.3769999999999998,"5min":4.9889999999999999,"15min":7.7229999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.111,"5min":0.88400000000000001,"15min":1.1080000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.29599999999999999},"max":{"1min":5.8159999999999998,"5min":5.8159999999999998,"15min":10.539999999999999},"last":0.74199999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:19:05 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":2.8140000000000001,"15min":1.7769999999999999},"min":{"1min":0.36699999999999999,"5min":0.32500000000000001,"15min":0.255},"max":{"1min":11.519,"5min":117.432,"15min":117.432},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.661,"5min":2.7360000000000002,"15min":1.823},"min":{"1min":0.36899999999999999,"5min":0.32300000000000001,"15min":0.30299999999999999},"max":{"1min":11.071,"5min":114.828,"15min":114.828},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:19:26 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.82199999999999995,"15min":1.1659999999999999},"min":{"1min":0.36099999999999999,"5min":0.34999999999999998,"15min":0.34599999999999997},"max":{"1min":3.343,"5min":3.6360000000000001,"15min":10.622},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.878,"5min":0.81799999999999995,"15min":1.1439999999999999},"min":{"1min":0.38200000000000001,"5min":0.36299999999999999,"15min":0.308},"max":{"1min":3.5910000000000002,"5min":3.8929999999999998,"15min":7.8639999999999999},"last":0.754}]},{"osd":7,"last update":"Mon Feb 5 08:18:47 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":2.8010000000000002,"15min":1.7210000000000001},"min":{"1min":0.40899999999999997,"5min":0.36799999999999999,"15min":0.35599999999999998},"max":{"1min":4.3460000000000001,"5min":114.84999999999999,"15min":114.84999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.03,"5min":2.6219999999999999,"15min":1.7509999999999999},"min":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.33600000000000002},"max":{"1min":3.496,"5min":114.871,"15min":114.871},"last":0.71199999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953671,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1491172,"kb_used_data":1040728,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92253980,"statfs":{"total":95995035648,"available":94468075520,"internally_reserved":0,"allocated":1065705472,"data_stored":1065292774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.0720000000000001,"15min":1.651},"min":{"1min":0.27000000000000002,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":2.7090000000000001,"5min":25.600999999999999,"15min":73.078999999999994},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.69199999999999995,"5min":1.2689999999999999,"15min":1.6699999999999999},"min":{"1min":0.25,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":1.8180000000000001,"5min":23.030999999999999,"15min":73.412000000000006},"last":1.6479999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.97299999999999998,"15min":1.486},"min":{"1min":0.30399999999999999,"5min":0.23300000000000001,"15min":0.21099999999999999},"max":{"1min":1.4590000000000001,"5min":8.3320000000000007,"15min":125.054},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.89300000000000002,"15min":1.387},"min":{"1min":0.309,"5min":0.25700000000000001,"15min":0.23499999999999999},"max":{"1min":1.1910000000000001,"5min":6.2370000000000001,"15min":125.08799999999999},"last":0.95399999999999996}]},{"osd":2,"last update":"Mon Feb 5 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.91000000000000003,"15min":1.252},"min":{"1min":0.36499999999999999,"5min":0.26500000000000001,"15min":0.24199999999999999},"max":{"1min":1.2210000000000001,"5min":4.8159999999999998,"15min":96.849999999999994},"last":1.835},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.0309999999999999,"15min":1.333},"min":{"1min":0.374,"5min":0.27100000000000002,"15min":0.26300000000000001},"max":{"1min":1.9670000000000001,"5min":5.6879999999999997,"15min":96.143000000000001},"last":0.51900000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":1.0029999999999999,"15min":1.837},"min":{"1min":0.40999999999999998,"5min":0.33100000000000002,"15min":0.24399999999999999},"max":{"1min":1.7929999999999999,"5min":5.5460000000000003,"15min":73.602999999999994},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.89200000000000002,"15min":1.8320000000000001},"min":{"1min":0.33200000000000002,"5min":0.309,"15min":0.27200000000000002},"max":{"1min":1.78,"5min":3.4350000000000001,"15min":74.817999999999998},"last":1.8089999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:19:11 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.585,"15min":0.86399999999999999},"min":{"1min":0.32600000000000001,"5min":0.29399999999999998,"15min":0.224},"max":{"1min":3.653,"5min":34.167000000000002,"15min":34.167000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":1.641,"15min":0.92700000000000005},"min":{"1min":0.32600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":3.7040000000000002,"5min":36.051000000000002,"15min":36.051000000000002},"last":0.308}]},{"osd":6,"last update":"Mon Feb 5 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":1.085,"15min":0.73399999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.6970000000000001,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.67600000000000005,"5min":1.012,"15min":0.69499999999999995},"min":{"1min":0.29299999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.6659999999999999,"5min":5.5049999999999999,"15min":5.5049999999999999},"last":0.39100000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:18:44 2024","interfaces":[{"interface":"back","average":{"1min":1.228,"5min":1.2390000000000001,"15min":0.77100000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.27000000000000002},"max":{"1min":3.411,"5min":4.8129999999999997,"15min":4.8129999999999997},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.22,"15min":0.76900000000000002},"min":{"1min":0.33900000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":4.6529999999999996,"5min":5.7450000000000001,"15min":5.7450000000000001},"last":0.501}]}]},{"osd":5,"up_from":42,"seq":180388626625,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1197652,"kb_used_data":661128,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92547500,"statfs":{"total":95995035648,"available":94768640000,"internally_reserved":0,"allocated":676995072,"data_stored":676599165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:18:53 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.8779999999999999,"15min":1.9399999999999999},"min":{"1min":0.35599999999999998,"5min":0.216,"15min":0.19400000000000001},"max":{"1min":3.2200000000000002,"5min":30.065999999999999,"15min":143.06299999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.6879999999999999,"15min":1.861},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.20499999999999999},"max":{"1min":3.5640000000000001,"5min":30.173999999999999,"15min":143.09700000000001},"last":0.51900000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:18:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.321,"15min":1.163},"min":{"1min":0.42799999999999999,"5min":0.22500000000000001,"15min":0.215},"max":{"1min":2.8090000000000002,"5min":5.8940000000000001,"15min":46.200000000000003},"last":0.37},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.1859999999999999,"15min":1.23},"min":{"1min":0.29699999999999999,"5min":0.246,"15min":0.22900000000000001},"max":{"1min":4.7969999999999997,"5min":5.2210000000000001,"15min":46.343000000000004},"last":0.55200000000000005}]},{"osd":2,"last update":"Mon Feb 5 08:18:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.258,"15min":1.0920000000000001},"min":{"1min":0.315,"5min":0.28999999999999998,"15min":0.246},"max":{"1min":4.7519999999999998,"5min":7.1159999999999997,"15min":10.733000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":1.069,"5min":1.409,"15min":1.1759999999999999},"min":{"1min":0.38100000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.8620000000000001,"5min":5.8250000000000002,"15min":8.0220000000000002},"last":0.432}]},{"osd":3,"last update":"Mon Feb 5 08:18:53 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.28,"15min":1.732},"min":{"1min":0.36799999999999999,"5min":0.315,"15min":0.28799999999999998},"max":{"1min":4.7889999999999997,"5min":7.165,"15min":143.03399999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.89200000000000002,"5min":1.3560000000000001,"15min":1.8260000000000001},"min":{"1min":0.41599999999999998,"5min":0.33800000000000002,"15min":0.28100000000000003},"max":{"1min":2.786,"5min":5.8609999999999998,"15min":143.46799999999999},"last":1.226}]},{"osd":4,"last update":"Mon Feb 5 08:18:53 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.252,"15min":0.78200000000000003},"min":{"1min":0.26600000000000001,"5min":0.24399999999999999,"15min":0.23699999999999999},"max":{"1min":3.3610000000000002,"5min":5.3769999999999998,"15min":5.3769999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.573,"5min":1.3779999999999999,"15min":0.78800000000000003},"min":{"1min":0.312,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":6.4029999999999996,"5min":7.2380000000000004,"15min":7.2380000000000004},"last":0.84099999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:18:31 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.4139999999999999,"15min":0.79300000000000004},"min":{"1min":0.432,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.952,"5min":7.5439999999999996,"15min":7.5439999999999996},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.869,"5min":1.4299999999999999,"15min":0.82599999999999996},"min":{"1min":0.41899999999999998,"5min":0.316,"15min":0.29299999999999998},"max":{"1min":4.8170000000000002,"5min":7.194,"15min":7.194},"last":0.46300000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:18:53 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.7969999999999999,"15min":0.96399999999999997},"min":{"1min":0.41199999999999998,"5min":0.30399999999999999,"15min":0.25900000000000001},"max":{"1min":4.4370000000000003,"5min":9.6940000000000008,"15min":9.6940000000000008},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.647,"5min":1.587,"15min":0.92400000000000004},"min":{"1min":0.40000000000000002,"5min":0.318,"15min":0.28199999999999997},"max":{"1min":9.2360000000000007,"5min":9.2360000000000007,"15min":9.2360000000000007},"last":0.58299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1022099456,"data_stored":1022099456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":428998656,"data_stored":428998656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":837222400,"data_stored":837222400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":699203584,"data_stored":699203584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1101266944,"data_stored":1101266944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":708968448,"data_stored":708968448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":958660608,"data_stored":958660608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":558759936,"data_stored":558759936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:19:36.054 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-02-05T08:19:36.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:36.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:36.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:36.128 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 16 16 39919 39903 155.841 124.75 0.00739713 0.00641149 2024-02-05T08:19:36.222 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:36.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:36.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:36.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:36.413 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:36.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:36.515 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:36.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:36.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:36.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:36.650 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:36.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:36.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:36.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:36.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:36 smithi134 ceph-mon[30086]: 5.7 scrub starts 2024-02-05T08:19:36.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:36 smithi134 ceph-mon[30086]: 5.7 scrub ok 2024-02-05T08:19:36.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:36 smithi134 ceph-mon[30086]: 4.1e scrub starts 2024-02-05T08:19:36.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:36 smithi134 ceph-mon[30086]: 4.1e scrub ok 2024-02-05T08:19:36.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:36 smithi134 ceph-mon[30086]: 4.14 scrub starts 2024-02-05T08:19:36.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:36 smithi134 ceph-mon[30086]: 4.14 scrub ok 2024-02-05T08:19:36.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:36 smithi134 ceph-mon[39278]: 5.7 scrub starts 2024-02-05T08:19:36.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:36 smithi134 ceph-mon[39278]: 5.7 scrub ok 2024-02-05T08:19:36.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:36 smithi134 ceph-mon[39278]: 4.1e scrub starts 2024-02-05T08:19:36.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:36 smithi134 ceph-mon[39278]: 4.1e scrub ok 2024-02-05T08:19:36.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:36 smithi134 ceph-mon[39278]: 4.14 scrub starts 2024-02-05T08:19:36.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:36 smithi134 ceph-mon[39278]: 4.14 scrub ok 2024-02-05T08:19:36.790 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:36 smithi196 ceph-mon[33018]: 5.7 scrub starts 2024-02-05T08:19:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:36 smithi196 ceph-mon[33018]: 5.7 scrub ok 2024-02-05T08:19:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:36 smithi196 ceph-mon[33018]: 4.1e scrub starts 2024-02-05T08:19:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:36 smithi196 ceph-mon[33018]: 4.1e scrub ok 2024-02-05T08:19:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:36 smithi196 ceph-mon[33018]: 4.14 scrub starts 2024-02-05T08:19:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:36 smithi196 ceph-mon[33018]: 4.14 scrub ok 2024-02-05T08:19:36.826 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:36.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:36.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:36.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:36.963 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:36.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:36.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:37.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:37.105 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:37.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:37.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:37.128 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 17 16 41854 41838 153.788 120.938 0.00764031 0.00649735 2024-02-05T08:19:37.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:37.245 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:37.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:37.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:37.326 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:37.346 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:37.380 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:37.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:37.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:37.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:37.515 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:37.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:37.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:37.617 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:37.663 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:37.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:37.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[30086]: pgmap v668: 158 pgs: 158 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 150 MiB/s wr, 2.39k op/s 2024-02-05T08:19:37.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[30086]: 6.1a scrub starts 2024-02-05T08:19:37.743 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[30086]: 6.1a scrub ok 2024-02-05T08:19:37.743 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[30086]: 5.1d deep-scrub starts 2024-02-05T08:19:37.743 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[30086]: 5.1d deep-scrub ok 2024-02-05T08:19:37.743 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[30086]: 5.17 deep-scrub starts 2024-02-05T08:19:37.743 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[30086]: 5.17 deep-scrub ok 2024-02-05T08:19:37.743 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[39278]: pgmap v668: 158 pgs: 158 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 150 MiB/s wr, 2.39k op/s 2024-02-05T08:19:37.743 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[39278]: 6.1a scrub starts 2024-02-05T08:19:37.743 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[39278]: 6.1a scrub ok 2024-02-05T08:19:37.744 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[39278]: 5.1d deep-scrub starts 2024-02-05T08:19:37.744 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[39278]: 5.1d deep-scrub ok 2024-02-05T08:19:37.744 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[39278]: 5.17 deep-scrub starts 2024-02-05T08:19:37.744 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-mon[39278]: 5.17 deep-scrub ok 2024-02-05T08:19:37.764 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:37.803 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:37.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:37.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:37.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:37 smithi196 ceph-mon[33018]: pgmap v668: 158 pgs: 158 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 150 MiB/s wr, 2.39k op/s 2024-02-05T08:19:37.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:37 smithi196 ceph-mon[33018]: 6.1a scrub starts 2024-02-05T08:19:37.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:37 smithi196 ceph-mon[33018]: 6.1a scrub ok 2024-02-05T08:19:37.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:37 smithi196 ceph-mon[33018]: 5.1d deep-scrub starts 2024-02-05T08:19:37.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:37 smithi196 ceph-mon[33018]: 5.1d deep-scrub ok 2024-02-05T08:19:37.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:37 smithi196 ceph-mon[33018]: 5.17 deep-scrub starts 2024-02-05T08:19:37.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:37 smithi196 ceph-mon[33018]: 5.17 deep-scrub ok 2024-02-05T08:19:37.905 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:37.942 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:37.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:38.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:19:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:19:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:19:38.043 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:38.077 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:38.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:38.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:38.128 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 18 16 43769 43753 151.893 119.688 0.00554339 0.00657841 2024-02-05T08:19:38.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:38.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:38.232 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:38.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:38.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:38.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:38.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:38.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:38.509 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:38.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:38.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:38.644 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:38.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:38.746 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:38.787 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:38.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:38 smithi196 ceph-mon[33018]: 3.e scrub starts 2024-02-05T08:19:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:38 smithi196 ceph-mon[33018]: 3.e scrub ok 2024-02-05T08:19:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:19:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:19:38.889 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:38.967 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:38 smithi134 ceph-mon[30086]: 3.e scrub starts 2024-02-05T08:19:38.967 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:38 smithi134 ceph-mon[30086]: 3.e scrub ok 2024-02-05T08:19:38.967 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:19:38.967 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:19:38.967 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:38 smithi134 ceph-mon[39278]: 3.e scrub starts 2024-02-05T08:19:38.967 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:38 smithi134 ceph-mon[39278]: 3.e scrub ok 2024-02-05T08:19:38.967 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:19:38.968 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:19:38.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:19:39.006 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:39.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:39.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:39.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:39.128 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 19 16 45588 45572 149.882 113.688 0.00784079 0.00666694 2024-02-05T08:19:39.146 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:39.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:39.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:39.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:39.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:39.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:39.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:39.421 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:39.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:39.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:39.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:39.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:39.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:39.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:39.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:39.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:39.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:39.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[30086]: pgmap v669: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-02-05T08:19:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:19:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:19:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[30086]: 3.12 scrub starts 2024-02-05T08:19:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[30086]: 3.12 scrub ok 2024-02-05T08:19:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:19:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:19:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[30086]: 3.7 scrub starts 2024-02-05T08:19:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[30086]: 3.7 scrub ok 2024-02-05T08:19:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[30086]: osdmap e321: 8 total, 8 up, 8 in 2024-02-05T08:19:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[39278]: pgmap v669: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-02-05T08:19:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:19:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:19:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[39278]: 3.12 scrub starts 2024-02-05T08:19:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[39278]: 3.12 scrub ok 2024-02-05T08:19:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:19:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:19:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[39278]: 3.7 scrub starts 2024-02-05T08:19:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[39278]: 3.7 scrub ok 2024-02-05T08:19:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:39 smithi134 ceph-mon[39278]: osdmap e321: 8 total, 8 up, 8 in 2024-02-05T08:19:39.796 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:39 smithi196 ceph-mon[33018]: pgmap v669: 158 pgs: 158 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-02-05T08:19:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:39 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:19:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:39 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:19:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:39 smithi196 ceph-mon[33018]: 3.12 scrub starts 2024-02-05T08:19:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:39 smithi196 ceph-mon[33018]: 3.12 scrub ok 2024-02-05T08:19:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:39 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:19:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:39 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:19:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:39 smithi196 ceph-mon[33018]: 3.7 scrub starts 2024-02-05T08:19:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:39 smithi196 ceph-mon[33018]: 3.7 scrub ok 2024-02-05T08:19:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:39 smithi196 ceph-mon[33018]: osdmap e321: 8 total, 8 up, 8 in 2024-02-05T08:19:39.833 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:39.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:39.935 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:19:39.989 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:39.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:39.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:19:40.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:40.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:40.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:40.129 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:19:40.130112+0000 min lat: 0.00127789 max lat: 0.0705605 avg lat: 0.00676418 2024-02-05T08:19:40.129 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:19:40.129 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 20 16 47298 47282 147.729 106.875 0.00662875 0.00676418 2024-02-05T08:19:40.226 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:40.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:40.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:40.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:40.396 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:40.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:40.497 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:40.542 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:40.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:40.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:40.679 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:40.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:40.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:40.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:40.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:40.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:40.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:40.878 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:40.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:40.945 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:40.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:40.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:41.008 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:19:41.008 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:19:41.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:40 smithi134 ceph-mon[30086]: pgmap v671: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-02-05T08:19:41.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:40 smithi134 ceph-mon[30086]: osdmap e322: 8 total, 8 up, 8 in 2024-02-05T08:19:41.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:40 smithi134 ceph-mon[30086]: 5.1a scrub starts 2024-02-05T08:19:41.009 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:40 smithi134 ceph-mon[30086]: 5.1a scrub ok 2024-02-05T08:19:41.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:40 smithi134 ceph-mon[39278]: pgmap v671: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-02-05T08:19:41.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:40 smithi134 ceph-mon[39278]: osdmap e322: 8 total, 8 up, 8 in 2024-02-05T08:19:41.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:40 smithi134 ceph-mon[39278]: 5.1a scrub starts 2024-02-05T08:19:41.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:40 smithi134 ceph-mon[39278]: 5.1a scrub ok 2024-02-05T08:19:41.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:41.055 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:19:41.055 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:19:41.055 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T08:19:41.056 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-05T08:19:41.056 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.1 2024-02-05T08:19:41.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:40 smithi196 ceph-mon[33018]: pgmap v671: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-02-05T08:19:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:40 smithi196 ceph-mon[33018]: osdmap e322: 8 total, 8 up, 8 in 2024-02-05T08:19:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:40 smithi196 ceph-mon[33018]: 5.1a scrub starts 2024-02-05T08:19:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:40 smithi196 ceph-mon[33018]: 5.1a scrub ok 2024-02-05T08:19:41.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:41.092 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:41.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:41.129 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 21 16 48929 48913 145.547 101.938 0.0118144 0.00686519 2024-02-05T08:19:41.193 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:41.243 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:41.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:41.277 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:41 smithi134 systemd[1]: Stopping Ceph osd.1 for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:19:41.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:41.389 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:41.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:41.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:41.535 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:41.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:41.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:41.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:41.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:41.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:41.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:41.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:41.818 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:41.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:41.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:41.827 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:19:41.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:41.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:41.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:41.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:42.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:41 smithi134 ceph-mon[30086]: 6.8 scrub starts 2024-02-05T08:19:42.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:41 smithi134 ceph-mon[30086]: 6.8 scrub ok 2024-02-05T08:19:42.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:41 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2421931686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:19:42.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:41 smithi134 ceph-mon[39278]: 6.8 scrub starts 2024-02-05T08:19:42.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:41 smithi134 ceph-mon[39278]: 6.8 scrub ok 2024-02-05T08:19:42.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:41 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2421931686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:19:42.055 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:41 smithi196 ceph-mon[33018]: 6.8 scrub starts 2024-02-05T08:19:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:41 smithi196 ceph-mon[33018]: 6.8 scrub ok 2024-02-05T08:19:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:41 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2421931686' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:19:42.089 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:42.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:42.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:42.101 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T08:19:42.129 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 22 16 50551 50535 143.539 101.375 0.00480934 0.00696142 2024-02-05T08:19:42.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:42.220 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:42.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:42.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:42.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:42.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:42.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:42.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:42.455 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:42.494 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:42.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:42.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:42.632 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:42.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:42.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:42.734 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:42.772 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:42.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:42.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:42.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:42.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:42.922 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:42.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:43.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:43.062 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:42 smithi134 ceph-mon[30086]: pgmap v673: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-02-05T08:19:43.062 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:42 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:19:43.062 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:42 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2421931686' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:19:43.063 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:42 smithi134 ceph-mon[30086]: osdmap e323: 8 total, 8 up, 8 in 2024-02-05T08:19:43.063 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:42 smithi134 ceph-mon[39278]: pgmap v673: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-02-05T08:19:43.063 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:42 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:19:43.063 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:42 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2421931686' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:19:43.063 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:42 smithi134 ceph-mon[39278]: osdmap e323: 8 total, 8 up, 8 in 2024-02-05T08:19:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:42 smithi196 ceph-mon[33018]: pgmap v673: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-02-05T08:19:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:42 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:19:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:42 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2421931686' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:19:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:42 smithi196 ceph-mon[33018]: osdmap e323: 8 total, 8 up, 8 in 2024-02-05T08:19:43.069 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:43.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:43.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:43.129 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 23 16 52185 52169 141.738 102.125 0.0058792 0.00705015 2024-02-05T08:19:43.171 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:43.215 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:43.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:43.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:43.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:43.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:43.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:43.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:43.491 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:43.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:43.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:43.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:43.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:43.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:43.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:43.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:43.772 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:43.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:43.828 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:43.872 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:43.920 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:43.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:43.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:44.022 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:44.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:44.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:44.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:43 smithi196 ceph-mon[33018]: osd.1 marked itself down and dead 2024-02-05T08:19:44.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:19:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:19:44.130 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 24 16 53479 53463 139.201 80.875 0.00227021 0.00711074 2024-02-05T08:19:44.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:44.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:44.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:44.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:44.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:43 smithi134 ceph-mon[30086]: osd.1 marked itself down and dead 2024-02-05T08:19:44.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:43 smithi134 ceph-mon[39278]: osd.1 marked itself down and dead 2024-02-05T08:19:44.277 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:43 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1[49860]: 2024-02-05T08:19:43.808+0000 7f611cb85700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:19:44.277 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:43 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1[49860]: 2024-02-05T08:19:43.808+0000 7f611cb85700 -1 osd.1 323 *** Got signal Terminated *** 2024-02-05T08:19:44.277 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:43 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1[49860]: 2024-02-05T08:19:43.808+0000 7f611cb85700 -1 osd.1 323 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T08:19:44.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:44.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:44.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:44.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:44.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:44.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:44.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:44.524 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:44.548 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:44.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:44.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:44.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:44.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:44.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:44.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:44.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:44.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:44.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:44.935 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:19:44.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:44.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:44.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:45.043 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:45.077 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:45.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:45.130 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 25 16 53813 53797 134.469 20.875 0.00873569 0.00743168 2024-02-05T08:19:45.178 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:45.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:44 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:45.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:44 smithi134 ceph-mon[39278]: pgmap v675: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-02-05T08:19:45.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:44 smithi134 ceph-mon[39278]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-05T08:19:45.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:44 smithi134 ceph-mon[39278]: osdmap e324: 8 total, 7 up, 8 in 2024-02-05T08:19:45.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:44 smithi134 ceph-mon[39278]: osdmap e325: 8 total, 7 up, 8 in 2024-02-05T08:19:45.204 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:44 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:45.204 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:44 smithi134 ceph-mon[30086]: pgmap v675: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-02-05T08:19:45.204 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:44 smithi134 ceph-mon[30086]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-05T08:19:45.204 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:44 smithi134 ceph-mon[30086]: osdmap e324: 8 total, 7 up, 8 in 2024-02-05T08:19:45.204 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:44 smithi134 ceph-mon[30086]: osdmap e325: 8 total, 7 up, 8 in 2024-02-05T08:19:45.213 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:45.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:45.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:44 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:44 smithi196 ceph-mon[33018]: pgmap v675: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.7 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-02-05T08:19:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:44 smithi196 ceph-mon[33018]: Health check failed: 1 osds down (OSD_DOWN) 2024-02-05T08:19:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:44 smithi196 ceph-mon[33018]: osdmap e324: 8 total, 7 up, 8 in 2024-02-05T08:19:45.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:44 smithi196 ceph-mon[33018]: osdmap e325: 8 total, 7 up, 8 in 2024-02-05T08:19:45.314 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:45.357 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:45.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:45.458 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:45.474 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:45 smithi134 podman[177487]: 2024-02-05 08:19:45.415338921 +0000 UTC m=+2.496274213 container died 68d3f6faf38203d22c22325d2e4bd2978b3b1bc9fa23ec530699630a3ab89e53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131) 2024-02-05T08:19:45.501 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:45.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:45.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:45.603 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:45.639 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:45.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:45.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:45.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:45.778 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:45.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:45.858 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:19:45.859 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:19:45.859 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:19:45.859 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:19:45.859 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:19:45.859 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:19:45.859 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:19:45.859 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:19:45.859 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:19:45.859 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:19:45.859 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:19:45.859 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:19:45.859 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:19:45.859 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:19:45.860 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:19:45.879 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:45.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:45.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:46.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:46.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:45 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3335514785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:46.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:45 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3335514785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:46.051 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:46.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:46.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:46.129 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 26 16 55503 55487 133.359 105.625 0.00351157 0.00749304 2024-02-05T08:19:46.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:46.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:46.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:46.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:45 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3335514785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:46.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:46.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:46.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:46.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:46.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:46.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:46.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:46.612 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:46.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:46.664 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:19:46.665 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:19:46.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:46.753 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:46.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:46.854 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:46.889 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:46.890 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:46 smithi134 podman[177487]: 2024-02-05 08:19:46.790776133 +0000 UTC m=+3.871711417 container cleanup 68d3f6faf38203d22c22325d2e4bd2978b3b1bc9fa23ec530699630a3ab89e53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T08:19:46.890 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:46 smithi134 bash[177487]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1 2024-02-05T08:19:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:46.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:46.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:47.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:47.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:47.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:47.130 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 27 16 57189 57173 132.322 105.375 0.0106252 0.00755287 2024-02-05T08:19:47.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:47.173 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:47.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:47.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:47.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:46 smithi134 ceph-mon[30086]: pgmap v678: 158 pgs: 15 stale+active+clean, 143 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 856 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-05T08:19:47.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:46 smithi134 ceph-mon[39278]: pgmap v678: 158 pgs: 15 stale+active+clean, 143 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 856 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-05T08:19:47.310 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:47.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:47.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:46 smithi196 ceph-mon[33018]: pgmap v678: 158 pgs: 15 stale+active+clean, 143 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 856 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-05T08:19:47.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:47.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:47.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:47.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:47.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:47.589 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:47.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:47.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:47.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:47.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:47.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:47.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:47.739 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:47 smithi134 podman[177666]: 2024-02-05 08:19:47.284167349 +0000 UTC m=+2.760501942 container remove 68d3f6faf38203d22c22325d2e4bd2978b3b1bc9fa23ec530699630a3ab89e53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T08:19:47.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:47.865 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:47.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:47.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:47.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:47.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:47.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:48.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:48.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:47 smithi134 ceph-mon[30086]: Health check failed: Degraded data redundancy: 7188/164577 objects degraded (4.368%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T08:19:48.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:19:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:19:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:19:48.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:47 smithi134 ceph-mon[39278]: Health check failed: Degraded data redundancy: 7188/164577 objects degraded (4.368%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T08:19:48.044 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:48.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:48.130 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 28 16 58816 58800 131.228 101.688 0.00950844 0.00761572 2024-02-05T08:19:48.146 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:48.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:48.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:48.281 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:48.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:48.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:48.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:47 smithi196 ceph-mon[33018]: Health check failed: Degraded data redundancy: 7188/164577 objects degraded (4.368%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T08:19:48.383 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:48.424 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:48.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:48.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:48.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:48.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:48.665 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:19:48.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:48.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:48.753 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:48.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:48.855 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:48.896 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:48.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:48.997 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:49.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:48 smithi134 ceph-mon[30086]: pgmap v679: 158 pgs: 2 active+undersized+degraded, 13 stale+active+clean, 3 active+undersized, 140 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 5.5 KiB/s rd, 116 MiB/s wr, 1.85k op/s; 7188/164577 objects degraded (4.368%); 35 KiB/s, 0 objects/s recovering 2024-02-05T08:19:49.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:48 smithi134 ceph-mon[39278]: pgmap v679: 158 pgs: 2 active+undersized+degraded, 13 stale+active+clean, 3 active+undersized, 140 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 5.5 KiB/s rd, 116 MiB/s wr, 1.85k op/s; 7188/164577 objects degraded (4.368%); 35 KiB/s, 0 objects/s recovering 2024-02-05T08:19:49.036 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:49.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:49.130 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 29 16 60385 60369 130.084 98.0625 0.00708171 0.00768238 2024-02-05T08:19:49.138 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:49.174 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:49.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:49.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:49.276 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:49.348 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:48 smithi196 ceph-mon[33018]: pgmap v679: 158 pgs: 2 active+undersized+degraded, 13 stale+active+clean, 3 active+undersized, 140 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 5.5 KiB/s rd, 116 MiB/s wr, 1.85k op/s; 7188/164577 objects degraded (4.368%); 35 KiB/s, 0 objects/s recovering 2024-02-05T08:19:49.390 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:49.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:49.492 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:49.520 INFO:teuthology.orchestra.run.smithi134.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-02-05T08:19:49.528 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:49.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:49.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:49.667 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:49.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:49.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:49.804 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:49.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:49.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:49.906 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:49.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:49.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:49.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:50.045 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:50.082 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:50.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:50.130 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 30 16 61995 61979 129.102 100.625 0.00709773 0.00774106 2024-02-05T08:19:50.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:50.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:50.217 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:50.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:50.318 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:50.355 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:50.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:50.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:50.456 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:50.495 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:50.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:50.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:50.631 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:50.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:50.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:50.676 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T08:19:50.676 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T08:19:50.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:50.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:50.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:50.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:50.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:50.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:51.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:51.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:51.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:51.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:51.130 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 31 16 63530 63514 128.032 95.9375 0.00703882 0.00780636 2024-02-05T08:19:51.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:51.179 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:51.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:51.210 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:51.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:51.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:50 smithi134 ceph-mon[30086]: pgmap v680: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 4.3 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 10941/176652 objects degraded (6.194%); 26 KiB/s, 0 objects/s recovering 2024-02-05T08:19:51.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:50 smithi134 ceph-mon[39278]: pgmap v680: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 4.3 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 10941/176652 objects degraded (6.194%); 26 KiB/s, 0 objects/s recovering 2024-02-05T08:19:51.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:50 smithi196 ceph-mon[33018]: pgmap v680: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 4.3 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 10941/176652 objects degraded (6.194%); 26 KiB/s, 0 objects/s recovering 2024-02-05T08:19:51.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:51.349 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:51.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:51.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:51.486 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:51.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:51.495 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:19:51.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:51.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:51.624 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:51.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:51.636 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:51 smithi134 podman[178536]: 2024-02-05 08:19:51.258230265 +0000 UTC m=+0.040261362 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:19:51.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:51.761 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:51.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:51.861 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:51.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:51.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:51.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:51.964 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:51 smithi134 podman[178536]: 2024-02-05 08:19:51.636117279 +0000 UTC m=+0.418148403 container create ac194198d28dafad46d010ee6ae3388c0c9a2e015bd276180b9a7e20c7f0cae9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-deactivate, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-02-05T08:19:51.964 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:51 smithi134 podman[178536]: 2024-02-05 08:19:51.964835619 +0000 UTC m=+0.746866695 container init ac194198d28dafad46d010ee6ae3388c0c9a2e015bd276180b9a7e20c7f0cae9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True) 2024-02-05T08:19:52.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:52.047 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:52.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:52.130 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 32 16 64886 64870 126.679 84.75 0.0123722 0.007889 2024-02-05T08:19:52.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:52.189 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:52.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:52.276 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:51 smithi134 podman[178536]: 2024-02-05 08:19:51.97443482 +0000 UTC m=+0.756465898 container start ac194198d28dafad46d010ee6ae3388c0c9a2e015bd276180b9a7e20c7f0cae9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True) 2024-02-05T08:19:52.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:52.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:52.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:52.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:52.460 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:52.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:52.561 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:52.603 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:52.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:52.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:52.739 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:52.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:52.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:52.841 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:52.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:52.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:52.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:52.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:53.013 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:53.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:53.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:53.130 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 33 16 66431 66415 125.766 96.5625 0.00989826 0.00794679 2024-02-05T08:19:53.149 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:53.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:53.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:53.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:52 smithi134 ceph-mon[30086]: pgmap v681: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 3.6 KiB/s rd, 60 MiB/s wr, 965 op/s; 10941/176652 objects degraded (6.194%); 26 KiB/s, 0 objects/s recovering 2024-02-05T08:19:53.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:52 smithi134 ceph-mon[39278]: pgmap v681: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 3.6 KiB/s rd, 60 MiB/s wr, 965 op/s; 10941/176652 objects degraded (6.194%); 26 KiB/s, 0 objects/s recovering 2024-02-05T08:19:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:53.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:53.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:53.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:52 smithi196 ceph-mon[33018]: pgmap v681: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 3.6 KiB/s rd, 60 MiB/s wr, 965 op/s; 10941/176652 objects degraded (6.194%); 26 KiB/s, 0 objects/s recovering 2024-02-05T08:19:53.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:53.418 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:53.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:53.520 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:53.545 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:53.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:53.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:53.647 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:53.683 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:53.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:53.784 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:53.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:53.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:53.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:53.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:53.964 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:53.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:53.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:54.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:19:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:19:54.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:54.102 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:54.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:54.130 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 34 16 67864 67848 124.701 89.5625 0.0164998 0.00801431 2024-02-05T08:19:54.204 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:54.240 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:54.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:54.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:54.342 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:54.365 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:54.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:54.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:54.490 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:54.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:54.592 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:54.630 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:54.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:54.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:54.732 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:54.755 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:54.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:54.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:19:54.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:19:54.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:19:54.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:19:54.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:19:54.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:19:54.857 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:54.891 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:54.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:54.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:54.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:55.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:55.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:55.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:55.122 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:55.131 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 35 16 68998 68982 123.163 70.875 0.00550041 0.00811493 2024-02-05T08:19:55.161 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:55.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:55.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:55.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:55.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:55.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:55.438 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:55.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:55.540 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:55.575 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:55.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:55.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:55.677 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:55.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:55.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:55.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:55.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:55 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:55.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:55 smithi134 ceph-mon[30086]: pgmap v682: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 3.4 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 12334/196671 objects degraded (6.271%); 21 KiB/s, 0 objects/s recovering 2024-02-05T08:19:55.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:55 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 12334/196671 objects degraded (6.271%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:19:55.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:55 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:55.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:55 smithi134 ceph-mon[39278]: pgmap v682: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 3.4 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 12334/196671 objects degraded (6.271%); 21 KiB/s, 0 objects/s recovering 2024-02-05T08:19:55.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:55 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 12334/196671 objects degraded (6.271%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:19:55.813 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:55 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:19:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:55 smithi196 ceph-mon[33018]: pgmap v682: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 3.4 KiB/s rd, 84 MiB/s wr, 1.34k op/s; 12334/196671 objects degraded (6.271%); 21 KiB/s, 0 objects/s recovering 2024-02-05T08:19:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:55 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 12334/196671 objects degraded (6.271%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:19:55.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:55.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:55.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:55.953 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:55.986 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:55.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:56.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:56.122 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:56.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:56.130 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 36 16 70412 70396 122.197 88.375 0.00672561 0.00817817 2024-02-05T08:19:56.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:56.257 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:56.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:56.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:56.391 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:56.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:56.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:56.493 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:56.517 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:56.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:56.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:56.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:56.652 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:56.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:56.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:56.793 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:56.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:56.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:56.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:56.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:56.948 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:56.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:57.049 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:57.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:57.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:57.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:57.131 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 37 16 71670 71654 121.019 78.625 0.0101036 0.00825811 2024-02-05T08:19:57.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:57.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:57.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:57.277 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:57 smithi134 podman[178536]: 2024-02-05 08:19:57.120341226 +0000 UTC m=+5.902372303 container attach ac194198d28dafad46d010ee6ae3388c0c9a2e015bd276180b9a7e20c7f0cae9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-05T08:19:57.277 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:19:57 smithi134 podman[178536]: 2024-02-05 08:19:57.122419185 +0000 UTC m=+5.904450268 container died ac194198d28dafad46d010ee6ae3388c0c9a2e015bd276180b9a7e20c7f0cae9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, CEPH_POINT_RELEASE=) 2024-02-05T08:19:57.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:57.366 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:57.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:57.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:57.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:57.507 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:57.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:57.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:57.608 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:57.647 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:57.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:57.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:57 smithi134 ceph-mon[30086]: pgmap v683: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 3.1 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 12334/199260 objects degraded (6.190%); 19 KiB/s, 0 objects/s recovering 2024-02-05T08:19:57.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:57 smithi134 ceph-mon[39278]: pgmap v683: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 3.1 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 12334/199260 objects degraded (6.190%); 19 KiB/s, 0 objects/s recovering 2024-02-05T08:19:57.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:57.786 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:57.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:57 smithi196 ceph-mon[33018]: pgmap v683: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 3.1 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 12334/199260 objects degraded (6.190%); 19 KiB/s, 0 objects/s recovering 2024-02-05T08:19:57.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:57.899 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.1.service' 2024-02-05T08:19:57.925 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:57.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:57.974 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:19:57.975 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-05T08:19:57.975 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-02-05T08:19:57.975 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi134.front.sepia.ceph.com 2024-02-05T08:19:57.975 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -n osd.1 --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-02-05T08:19:58.027 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:58.028 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:19:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:19:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:19:58.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:58.063 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:58.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:58.131 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 38 16 73098 73082 120.183 89.25 0.0154805 0.00831494 2024-02-05T08:19:58.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:19:58.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:58.218 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:58.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:19:58.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:19:58.362 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:58.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:19:58.364 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:19:58.464 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:58.505 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:58.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:58.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:58.636 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:58 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2243159056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:19:58.637 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:58 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2243159056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:19:58.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:58.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:58.740 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:19:58.740 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:19:58.740 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:19:58.740 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:19:58.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:58.753 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:19:58.779 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:58.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:58.798 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:19:58.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:58 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2243159056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:19:58.834 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:19:58.835 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:19:58.835 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:19:58.835 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:19:58.835 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:19:58.835 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:19:58.835 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:19:58.835 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:19:58.835 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:19:58.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:58.917 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:58.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:58.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:59.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:19:59.058 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:59.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:59.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:19:59.131 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 39 16 74503 74487 119.352 87.8125 0.00723796 0.0083734 2024-02-05T08:19:59.160 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:19:59.199 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:19:59.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:59.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:19:59.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:19:59.336 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:59.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:19:59.343 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/osd.1/config 2024-02-05T08:19:59.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:59.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:59.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:59.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:59.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:19:59.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:59.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:19:59.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:59.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:59 smithi134 ceph-mon[30086]: pgmap v684: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 2.8 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 13216/206811 objects degraded (6.390%); 17 KiB/s, 0 objects/s recovering 2024-02-05T08:19:59.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:59 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:19:59.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:59 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2243159056' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:19:59.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:59 smithi134 ceph-mon[30086]: osdmap e326: 8 total, 7 up, 8 in 2024-02-05T08:19:59.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:19:59 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1781167736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:59.720 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:59 smithi134 ceph-mon[39278]: pgmap v684: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 2.8 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 13216/206811 objects degraded (6.390%); 17 KiB/s, 0 objects/s recovering 2024-02-05T08:19:59.720 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:59 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:19:59.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:59 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2243159056' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:19:59.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:59 smithi134 ceph-mon[39278]: osdmap e326: 8 total, 7 up, 8 in 2024-02-05T08:19:59.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:19:59 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1781167736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:59.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:59.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:59.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:59 smithi196 ceph-mon[33018]: pgmap v684: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 2.8 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 13216/206811 objects degraded (6.390%); 17 KiB/s, 0 objects/s recovering 2024-02-05T08:19:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:59 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:19:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:59 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2243159056' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:19:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:59 smithi196 ceph-mon[33018]: osdmap e326: 8 total, 7 up, 8 in 2024-02-05T08:19:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:19:59 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1781167736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:19:59.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:19:59.895 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:19:59.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:19:59.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:19:59.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:00.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:00.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:00.131 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:20:00.132360+0000 min lat: 0.00127789 max lat: 1.05979 avg lat: 0.00842867 2024-02-05T08:20:00.131 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:20:00.131 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 40 16 75917 75901 118.578 88.375 0.0154177 0.00842867 2024-02-05T08:20:00.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:00.178 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:00.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:00.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:00.313 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:00.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:00.415 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:00.456 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:00.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:00.557 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:00.590 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:00.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:00.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:00.692 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:00.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:00.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:00.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 13216/206811 objects degraded (6.390%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:00.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 13216/206811 objects degraded (6.390%), 25 pgs degraded 2024-02-05T08:20:00.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T08:20:00.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: [WRN] OSD_DOWN: 1 osds down 2024-02-05T08:20:00.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: osd.1 (root=default,host=smithi134) is down 2024-02-05T08:20:00.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: [WRN] PG_DEGRADED: Degraded data redundancy: 13216/206811 objects degraded (6.390%), 25 pgs degraded 2024-02-05T08:20:00.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-02-05T08:20:00.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 2.2 is active+undersized+degraded, acting [5,6] 2024-02-05T08:20:00.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 3.6 is active+undersized+degraded, acting [0,2] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.2 is active+undersized+degraded, acting [5,4] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.4 is active+undersized+degraded, acting [2,3] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.9 is active+undersized+degraded, acting [4,3] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.b is active+undersized+degraded, acting [0,4] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.d is active+undersized+degraded, acting [4,2] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.e is active+undersized+degraded, acting [4,6] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.f is active+undersized+degraded, acting [3,4] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.10 is active+undersized+degraded, acting [3,6] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.13 is active+undersized+degraded, acting [4,6] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.17 is active+undersized+degraded, acting [3,7] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.18 is active+undersized+degraded, acting [4,6] 2024-02-05T08:20:00.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.1b is active+undersized+degraded, acting [4,3] 2024-02-05T08:20:00.779 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.1c is active+undersized+degraded, acting [2,3] 2024-02-05T08:20:00.779 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 4.1d is active+undersized+degraded, acting [3,4] 2024-02-05T08:20:00.779 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 5.0 is active+undersized+degraded, acting [3,4] 2024-02-05T08:20:00.779 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 5.15 is active+undersized+degraded, acting [5,0] 2024-02-05T08:20:00.779 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 10.0 is active+undersized+degraded, acting [6,4] 2024-02-05T08:20:00.779 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 10.2 is active+undersized+degraded, acting [4,0] 2024-02-05T08:20:00.779 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[30086]: pg 10.b is active+undersized+degraded, acting [6,0] 2024-02-05T08:20:00.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 13216/206811 objects degraded (6.390%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:00.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 13216/206811 objects degraded (6.390%), 25 pgs degraded 2024-02-05T08:20:00.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T08:20:00.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: [WRN] OSD_DOWN: 1 osds down 2024-02-05T08:20:00.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: osd.1 (root=default,host=smithi134) is down 2024-02-05T08:20:00.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: [WRN] PG_DEGRADED: Degraded data redundancy: 13216/206811 objects degraded (6.390%), 25 pgs degraded 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 2.2 is active+undersized+degraded, acting [5,6] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 3.6 is active+undersized+degraded, acting [0,2] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.2 is active+undersized+degraded, acting [5,4] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.4 is active+undersized+degraded, acting [2,3] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.9 is active+undersized+degraded, acting [4,3] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.b is active+undersized+degraded, acting [0,4] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.d is active+undersized+degraded, acting [4,2] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.e is active+undersized+degraded, acting [4,6] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.f is active+undersized+degraded, acting [3,4] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.10 is active+undersized+degraded, acting [3,6] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.13 is active+undersized+degraded, acting [4,6] 2024-02-05T08:20:00.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-02-05T08:20:00.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.17 is active+undersized+degraded, acting [3,7] 2024-02-05T08:20:00.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.18 is active+undersized+degraded, acting [4,6] 2024-02-05T08:20:00.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.1b is active+undersized+degraded, acting [4,3] 2024-02-05T08:20:00.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.1c is active+undersized+degraded, acting [2,3] 2024-02-05T08:20:00.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 4.1d is active+undersized+degraded, acting [3,4] 2024-02-05T08:20:00.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 5.0 is active+undersized+degraded, acting [3,4] 2024-02-05T08:20:00.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 5.15 is active+undersized+degraded, acting [5,0] 2024-02-05T08:20:00.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 10.0 is active+undersized+degraded, acting [6,4] 2024-02-05T08:20:00.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 10.2 is active+undersized+degraded, acting [4,0] 2024-02-05T08:20:00.781 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:00 smithi134 ceph-mon[39278]: pg 10.b is active+undersized+degraded, acting [6,0] 2024-02-05T08:20:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 13216/206811 objects degraded (6.390%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 13216/206811 objects degraded (6.390%), 25 pgs degraded 2024-02-05T08:20:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T08:20:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: [WRN] OSD_DOWN: 1 osds down 2024-02-05T08:20:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: osd.1 (root=default,host=smithi134) is down 2024-02-05T08:20:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: [WRN] PG_DEGRADED: Degraded data redundancy: 13216/206811 objects degraded (6.390%), 25 pgs degraded 2024-02-05T08:20:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-02-05T08:20:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 2.2 is active+undersized+degraded, acting [5,6] 2024-02-05T08:20:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-02-05T08:20:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 3.6 is active+undersized+degraded, acting [0,2] 2024-02-05T08:20:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-02-05T08:20:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.2 is active+undersized+degraded, acting [5,4] 2024-02-05T08:20:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.4 is active+undersized+degraded, acting [2,3] 2024-02-05T08:20:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.9 is active+undersized+degraded, acting [4,3] 2024-02-05T08:20:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.b is active+undersized+degraded, acting [0,4] 2024-02-05T08:20:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.d is active+undersized+degraded, acting [4,2] 2024-02-05T08:20:00.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.e is active+undersized+degraded, acting [4,6] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.f is active+undersized+degraded, acting [3,4] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.10 is active+undersized+degraded, acting [3,6] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.13 is active+undersized+degraded, acting [4,6] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.17 is active+undersized+degraded, acting [3,7] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.18 is active+undersized+degraded, acting [4,6] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.1b is active+undersized+degraded, acting [4,3] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.1c is active+undersized+degraded, acting [2,3] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 4.1d is active+undersized+degraded, acting [3,4] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 5.0 is active+undersized+degraded, acting [3,4] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 5.15 is active+undersized+degraded, acting [5,0] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 10.0 is active+undersized+degraded, acting [6,4] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 10.2 is active+undersized+degraded, acting [4,0] 2024-02-05T08:20:00.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:00 smithi196 ceph-mon[33018]: pg 10.b is active+undersized+degraded, acting [6,0] 2024-02-05T08:20:00.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:00.865 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:00.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:00.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:01.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:01.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:01.102 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:01.132 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 41 16 77299 77283 117.789 86.375 0.00928961 0.00848468 2024-02-05T08:20:01.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:01.146 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:01.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:01.188 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:20:01.188 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:20:01.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:01.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:01.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:01.367 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:20:01.394 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:01.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:01 smithi134 ceph-mon[30086]: pgmap v686: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.42k op/s; 13656/219348 objects degraded (6.226%) 2024-02-05T08:20:01.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:01 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:20:01.419 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:01 smithi134 ceph-mon[39278]: pgmap v686: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.42k op/s; 13656/219348 objects degraded (6.226%) 2024-02-05T08:20:01.419 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:01 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:20:01.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:01.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:01.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:01.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:01.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:01 smithi196 ceph-mon[33018]: pgmap v686: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.42k op/s; 13656/219348 objects degraded (6.226%) 2024-02-05T08:20:01.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:01 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:20:01.574 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:01.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:01.677 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:01.707 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:01.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:01.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:01.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:01.851 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:01.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:01.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:01.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:01.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:01.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:02.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:02.132 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 42 16 78515 78499 116.794 76 0.0155194 0.00855741 2024-02-05T08:20:02.141 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:02.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:02.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:02.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:02.275 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:02.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:02.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:02.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:02.422 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:02.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:02.524 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:02.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:02.565 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:02.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:02.666 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:02.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:02.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:02.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:02.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:02.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:02.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:02.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:02.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:03.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:03.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:03.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:03.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:03.133 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 43 16 79755 79739 115.879 77.5 0.0129616 0.00862492 2024-02-05T08:20:03.156 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:03.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:03.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:03.190 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:20:03.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:03.270 INFO:teuthology.orchestra.run.smithi134.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-05T08:20:03.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:03.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:03.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:03.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:03.448 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:03.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:03.475 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:03 smithi134 ceph-mon[30086]: pgmap v687: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.42k op/s; 13656/219348 objects degraded (6.226%) 2024-02-05T08:20:03.475 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:03 smithi134 ceph-mon[39278]: pgmap v687: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.42k op/s; 13656/219348 objects degraded (6.226%) 2024-02-05T08:20:03.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:03.580 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:03.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:03.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:03.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:03.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:03.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:03.745 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:03 smithi196 ceph-mon[33018]: pgmap v687: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.42k op/s; 13656/219348 objects degraded (6.226%) 2024-02-05T08:20:03.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:03.862 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:03.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:03.906 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:20:03.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:04.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:04.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:04.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:20:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:20:04.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:04.133 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 44 16 80961 80945 114.959 75.375 0.00994473 0.00869343 2024-02-05T08:20:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:04.151 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:04.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:04.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:04.289 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:04.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:04.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:04.391 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:04.416 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:04.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:04.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:04.477 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:20:04.518 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:04.557 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:04.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:04.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:04.659 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:04.699 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:04.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:04.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:04.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:04.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:04.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:04.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:04.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:04.976 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:04.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:04.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:05.078 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:05.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:05.124 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:05.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:05.133 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 45 16 82087 82071 113.968 70.375 0.00862917 0.00877011 2024-02-05T08:20:05.225 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:05.264 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:05.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:05.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:05.366 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:05.406 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:05.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:05.508 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:05.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:05.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:05.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:05 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:05.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:05 smithi134 ceph-mon[30086]: pgmap v688: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s; 15003/237093 objects degraded (6.328%) 2024-02-05T08:20:05.651 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:05 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 15003/237093 objects degraded (6.328%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:05.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:05.653 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:05 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:05.653 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:05 smithi134 ceph-mon[39278]: pgmap v688: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s; 15003/237093 objects degraded (6.328%) 2024-02-05T08:20:05.653 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:05 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 15003/237093 objects degraded (6.328%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:05.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:05 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:05.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:05 smithi196 ceph-mon[33018]: pgmap v688: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s; 15003/237093 objects degraded (6.328%) 2024-02-05T08:20:05.687 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:05 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 15003/237093 objects degraded (6.328%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:05.761 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:05.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:05.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:05.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:05.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:05.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:05.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:06.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:06.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:06.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:06.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:06.133 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 46 16 83253 83237 113.074 72.875 0.0148887 0.00883939 2024-02-05T08:20:06.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:06.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:06.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:06.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:06.271 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:06.306 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:06.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:06.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:06.408 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:06.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:06.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:06.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:06.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:06.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:06.682 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:06.729 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:06.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:06.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:06.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:06.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:06.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:07.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:07.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:07.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:07.034 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:20:07.034 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:20:07.106 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:07.133 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 47 16 84462 84446 112.275 75.5625 0.00671757 0.00890138 2024-02-05T08:20:07.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:07.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:07.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:07.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:07.311 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:07.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:07.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:07.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:07.455 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:07.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:07.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:07.596 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:07.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:07.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:07.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:07 smithi134 ceph-mon[30086]: pgmap v689: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s; 15003/239514 objects degraded (6.264%) 2024-02-05T08:20:07.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:07.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:07.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:07 smithi134 ceph-mon[39278]: pgmap v689: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s; 15003/239514 objects degraded (6.264%) 2024-02-05T08:20:07.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:07.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:07.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:07.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:07.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:07.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:07 smithi196 ceph-mon[33018]: pgmap v689: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s; 15003/239514 objects degraded (6.264%) 2024-02-05T08:20:07.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:07.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:07.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:07.871 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T08:20:07.887 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:07.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:07.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:07.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:08.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:20:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:20:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:20:08.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:08.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:08.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:08.134 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 48 16 85614 85598 111.436 72 0.0107082 0.00896867 2024-02-05T08:20:08.135 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:08.177 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:08.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:08.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:08.279 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:08.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:08.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:08.425 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:08.482 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:08.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:08.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:08.584 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:08.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:08.626 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:08.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:08.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:08.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:08.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:08.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:08.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:20:08.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:20:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:20:08.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:08.891 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:08.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:08.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:09.027 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:09.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:09.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:09.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:09.134 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 49 16 86827 86811 110.709 75.8125 0.0103286 0.00902659 2024-02-05T08:20:09.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:09.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:09.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:09.286 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:09.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:09.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:09.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:09.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:09.468 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:09.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:09.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:09.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:09.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:09.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:09.744 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:09.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:09.754 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:09 smithi134 ceph-mon[30086]: pgmap v690: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 15706/245985 objects degraded (6.385%) 2024-02-05T08:20:09.754 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:09 smithi134 ceph-mon[39278]: pgmap v690: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 15706/245985 objects degraded (6.385%) 2024-02-05T08:20:09.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:09 smithi196 ceph-mon[33018]: pgmap v690: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 15706/245985 objects degraded (6.385%) 2024-02-05T08:20:09.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:09.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:09.888 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:09.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:09.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:10.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:10.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:10.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:10.118 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:10.134 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 50 16 87928 87912 109.871 68.8125 0.0163012 0.00909427 2024-02-05T08:20:10.164 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:10.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:10.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:10.207 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:20:10.235 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:20:10.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:10.287 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:20:10.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:10.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:10.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:10.332 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:20:10.384 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:20:10.400 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:10.436 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:20:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:10.454 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:10.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:10.488 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:20:10.535 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:20:10.535 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:20:10.535 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:20:10.535 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:20:10.535 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:20:10.535 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:20:10.535 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:20:10.536 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:20:10.536 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:20:10.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:10.586 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:10.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:10.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:10.729 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:10.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:10.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:10.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:10 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 15706/245985 objects degraded (6.385%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:10.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:10 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3708803609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:10.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:10 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 15706/245985 objects degraded (6.385%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:10.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:10 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3708803609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:10 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 15706/245985 objects degraded (6.385%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:10 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3708803609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:10.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:10.873 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:10.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:10.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:10.975 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:11.011 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:11.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:11.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:11.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:11.134 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 51 16 88991 88975 109.019 66.4375 0.0183474 0.00916721 2024-02-05T08:20:11.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:11.151 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:11.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:11.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:11.288 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:11.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:11.390 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:11.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:11.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:11.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:11.460 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:20:11.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:11.560 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:11.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:11.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:11.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:11.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:11.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:11.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:11 smithi134 ceph-mon[30086]: pgmap v691: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 895 B/s rd, 73 MiB/s wr, 1.18k op/s; 16075/257235 objects degraded (6.249%) 2024-02-05T08:20:11.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:11 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/815849504' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:20:11.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:11 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:20:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:11 smithi134 ceph-mon[39278]: pgmap v691: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 895 B/s rd, 73 MiB/s wr, 1.18k op/s; 16075/257235 objects degraded (6.249%) 2024-02-05T08:20:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:11 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/815849504' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:20:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:11 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:20:11.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:11 smithi196 ceph-mon[33018]: pgmap v691: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 895 B/s rd, 73 MiB/s wr, 1.18k op/s; 16075/257235 objects degraded (6.249%) 2024-02-05T08:20:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:11 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/815849504' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:20:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:11 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:20:11.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:11.857 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:11.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:11.958 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:12.007 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:12.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:12.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:12.076 INFO:teuthology.orchestra.run.smithi134.stdout:6.e 2024-02-05T08:20:12.076 INFO:teuthology.orchestra.run.smithi134.stdout:6.d 2024-02-05T08:20:12.076 INFO:teuthology.orchestra.run.smithi134.stdout:6.b 2024-02-05T08:20:12.076 INFO:teuthology.orchestra.run.smithi134.stdout:6.6 2024-02-05T08:20:12.076 INFO:teuthology.orchestra.run.smithi134.stdout:6.4 2024-02-05T08:20:12.076 INFO:teuthology.orchestra.run.smithi134.stdout:6.1 2024-02-05T08:20:12.076 INFO:teuthology.orchestra.run.smithi134.stdout:6.10 2024-02-05T08:20:12.076 INFO:teuthology.orchestra.run.smithi134.stdout:5.c 2024-02-05T08:20:12.076 INFO:teuthology.orchestra.run.smithi134.stdout:5.7 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:5.5 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:5.1d 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:5.1a 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:5.19 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:5.18 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:5.15 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:3.4 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:3.1a 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:6.1a 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:3.6 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:1.3 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:3.18 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:3.16 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:3.8 2024-02-05T08:20:12.077 INFO:teuthology.orchestra.run.smithi134.stdout:3.0 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:6.18 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:10.0 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:3.1c 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:3.19 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:10.2 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:2.0 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:2.2 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:3.a 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:4.1c 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:3.1e 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:6.1d 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:4.17 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:4.14 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:10.b 2024-02-05T08:20:12.078 INFO:teuthology.orchestra.run.smithi134.stdout:3.e 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:4.10 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:4.13 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:5.8 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:4.e 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:6.19 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:4.18 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:5.17 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:4.1b 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:4.1d 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:4.2 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:4.4 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:4.9 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:5.16 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:5.13 2024-02-05T08:20:12.079 INFO:teuthology.orchestra.run.smithi134.stdout:4.b 2024-02-05T08:20:12.080 INFO:teuthology.orchestra.run.smithi134.stdout:4.d 2024-02-05T08:20:12.080 INFO:teuthology.orchestra.run.smithi134.stdout:4.f 2024-02-05T08:20:12.080 INFO:teuthology.orchestra.run.smithi134.stdout:5.0 2024-02-05T08:20:12.080 INFO:teuthology.orchestra.run.smithi134.stdout:5.12 2024-02-05T08:20:12.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:12.134 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 52 16 90174 90158 108.344 73.9375 0.0183988 0.00922541 2024-02-05T08:20:12.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:12.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:12.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:12.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:12.285 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:12.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:12.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:12.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:12.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:12.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:12.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:12.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:12.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:12.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:12.658 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:12.705 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:12.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:12.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:12 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:20:12.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:12 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:20:12.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:12 smithi134 ceph-mon[30086]: osdmap e327: 8 total, 7 up, 8 in 2024-02-05T08:20:12.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:12 smithi134 ceph-mon[30086]: 4.11 scrub starts 2024-02-05T08:20:12.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:12 smithi134 ceph-mon[30086]: 4.11 scrub ok 2024-02-05T08:20:12.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:12 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:20:12.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:12 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:20:12.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:12 smithi134 ceph-mon[39278]: osdmap e327: 8 total, 7 up, 8 in 2024-02-05T08:20:12.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:12 smithi134 ceph-mon[39278]: 4.11 scrub starts 2024-02-05T08:20:12.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:12 smithi134 ceph-mon[39278]: 4.11 scrub ok 2024-02-05T08:20:12.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:12 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:20:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:12 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:20:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:12 smithi196 ceph-mon[33018]: osdmap e327: 8 total, 7 up, 8 in 2024-02-05T08:20:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:12 smithi196 ceph-mon[33018]: 4.11 scrub starts 2024-02-05T08:20:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:12 smithi196 ceph-mon[33018]: 4.11 scrub ok 2024-02-05T08:20:12.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:12.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:12.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:12.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:12.982 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:12.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:12.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:13.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:13.127 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:13.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:13.134 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 53 16 91448 91432 107.802 79.625 0.0065514 0.00927144 2024-02-05T08:20:13.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:13.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:13.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:13.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:13.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:13.400 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:13.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:13.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:13.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -n osd.1 --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 5.c --file /var/log/ceph/exp.5.c.1 2024-02-05T08:20:13.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:13.535 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:13.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:13.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:13.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:13.681 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:13.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:13.714 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:13 smithi196 ceph-mon[33018]: pgmap v693: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s; 16075/257235 objects degraded (6.249%) 2024-02-05T08:20:13.715 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:13 smithi196 ceph-mon[33018]: 4.c scrub starts 2024-02-05T08:20:13.715 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:13 smithi196 ceph-mon[33018]: 4.c scrub ok 2024-02-05T08:20:13.715 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:13 smithi196 ceph-mon[33018]: 3.1 scrub starts 2024-02-05T08:20:13.715 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:13 smithi196 ceph-mon[33018]: 3.1 scrub ok 2024-02-05T08:20:13.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:13 smithi134 ceph-mon[30086]: pgmap v693: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s; 16075/257235 objects degraded (6.249%) 2024-02-05T08:20:13.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:13 smithi134 ceph-mon[30086]: 4.c scrub starts 2024-02-05T08:20:13.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:13 smithi134 ceph-mon[30086]: 4.c scrub ok 2024-02-05T08:20:13.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:13 smithi134 ceph-mon[30086]: 3.1 scrub starts 2024-02-05T08:20:13.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:13 smithi134 ceph-mon[30086]: 3.1 scrub ok 2024-02-05T08:20:13.775 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:13 smithi134 ceph-mon[39278]: pgmap v693: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s; 16075/257235 objects degraded (6.249%) 2024-02-05T08:20:13.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:13 smithi134 ceph-mon[39278]: 4.c scrub starts 2024-02-05T08:20:13.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:13 smithi134 ceph-mon[39278]: 4.c scrub ok 2024-02-05T08:20:13.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:13 smithi134 ceph-mon[39278]: 3.1 scrub starts 2024-02-05T08:20:13.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:13 smithi134 ceph-mon[39278]: 3.1 scrub ok 2024-02-05T08:20:13.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:13.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:13.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:13.918 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:13.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:13.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:13.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:13.991 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/osd.1/config 2024-02-05T08:20:14.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:14.065 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:20:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:20:14.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:14.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:14.135 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 54 16 92631 92615 107.175 73.9375 0.0124303 0.00932471 2024-02-05T08:20:14.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:14.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:14.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:14.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:14.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:14.400 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:14.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:14.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:14.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:14 smithi196 ceph-mon[33018]: 6.a scrub starts 2024-02-05T08:20:14.479 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:14 smithi196 ceph-mon[33018]: 6.a scrub ok 2024-02-05T08:20:14.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:14.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:14.545 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:14.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:14.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:14.685 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:14.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:14.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:14.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:14 smithi134 ceph-mon[30086]: 6.a scrub starts 2024-02-05T08:20:14.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:14 smithi134 ceph-mon[30086]: 6.a scrub ok 2024-02-05T08:20:14.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:14 smithi134 ceph-mon[39278]: 6.a scrub starts 2024-02-05T08:20:14.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:14 smithi134 ceph-mon[39278]: 6.a scrub ok 2024-02-05T08:20:14.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:20:14.828 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:20:14.828 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:14.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:14.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:14.941 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:20:14.969 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:14.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:15.071 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:15.134 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 55 16 93654 93638 106.389 63.9375 0.00903789 0.00939474 2024-02-05T08:20:15.137 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:15.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:15.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:15.239 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:15.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:15.280 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:15.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:15.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:15.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:15.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:15.470 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:20:15.470 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:20:15.470 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-02-05T08:20:15.470 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-02-05T08:20:15.470 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:20:15.521 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:15.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:15.564 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:15.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:15.665 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:15.670 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:15 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:15.670 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:15 smithi134 ceph-mon[30086]: pgmap v694: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s; 17108/272454 objects degraded (6.279%) 2024-02-05T08:20:15.670 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:15 smithi134 ceph-mon[30086]: 10.9 scrub starts 2024-02-05T08:20:15.670 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:15 smithi134 ceph-mon[30086]: 5.14 scrub starts 2024-02-05T08:20:15.670 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:15 smithi134 ceph-mon[30086]: 5.14 scrub ok 2024-02-05T08:20:15.670 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:15 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 17108/272454 objects degraded (6.279%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:15.670 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:15 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:15.670 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:15 smithi134 ceph-mon[39278]: pgmap v694: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s; 17108/272454 objects degraded (6.279%) 2024-02-05T08:20:15.670 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:15 smithi134 ceph-mon[39278]: 10.9 scrub starts 2024-02-05T08:20:15.670 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:15 smithi134 ceph-mon[39278]: 5.14 scrub starts 2024-02-05T08:20:15.671 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:15 smithi134 ceph-mon[39278]: 5.14 scrub ok 2024-02-05T08:20:15.671 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:15 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 17108/272454 objects degraded (6.279%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:15.699 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:15.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:15.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:15.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:15.801 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:15 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:15.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:15 smithi196 ceph-mon[33018]: pgmap v694: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s; 17108/272454 objects degraded (6.279%) 2024-02-05T08:20:15.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:15 smithi196 ceph-mon[33018]: 10.9 scrub starts 2024-02-05T08:20:15.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:15 smithi196 ceph-mon[33018]: 5.14 scrub starts 2024-02-05T08:20:15.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:15 smithi196 ceph-mon[33018]: 5.14 scrub ok 2024-02-05T08:20:15.802 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:15 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 17108/272454 objects degraded (6.279%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:15.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:15.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:15.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:15.946 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:15.977 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:15.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:15.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:16.079 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:16.115 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:16.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:16.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:16.134 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 56 15 94740 94725 105.702 67.9375 0.00982097 0.0094561 2024-02-05T08:20:16.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:16.256 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:16.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:16.358 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:16.375 INFO:teuthology.orchestra.run.smithi134.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-05T08:20:16.407 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:16.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:16.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:16.508 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:16.540 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:16.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:16.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:16.676 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:16.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:16.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:16.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:16.822 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:16.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:16.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:20:16.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:16.958 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:16.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:16.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:17.060 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:17.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:17.108 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:17.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:17.134 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 57 16 95879 95863 105.096 71.125 0.0151863 0.00951029 2024-02-05T08:20:17.209 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:17.251 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:17.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:17.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:17.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T08:20:17.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:17.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:17.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:17.493 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:17.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[30086]: pgmap v695: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s; 17108/274647 objects degraded (6.229%) 2024-02-05T08:20:17.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[30086]: 5.d scrub starts 2024-02-05T08:20:17.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[30086]: 5.d scrub ok 2024-02-05T08:20:17.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[30086]: 3.15 scrub starts 2024-02-05T08:20:17.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[30086]: 6.1e scrub starts 2024-02-05T08:20:17.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[30086]: 3.15 scrub ok 2024-02-05T08:20:17.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[30086]: 6.1e scrub ok 2024-02-05T08:20:17.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[30086]: 10.9 scrub ok 2024-02-05T08:20:17.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[30086]: 4.6 scrub starts 2024-02-05T08:20:17.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[30086]: 4.6 scrub ok 2024-02-05T08:20:17.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[30086]: 6.1f scrub starts 2024-02-05T08:20:17.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[30086]: 6.1f scrub ok 2024-02-05T08:20:17.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[39278]: pgmap v695: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s; 17108/274647 objects degraded (6.229%) 2024-02-05T08:20:17.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[39278]: 5.d scrub starts 2024-02-05T08:20:17.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[39278]: 5.d scrub ok 2024-02-05T08:20:17.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[39278]: 3.15 scrub starts 2024-02-05T08:20:17.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[39278]: 6.1e scrub starts 2024-02-05T08:20:17.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[39278]: 3.15 scrub ok 2024-02-05T08:20:17.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[39278]: 6.1e scrub ok 2024-02-05T08:20:17.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[39278]: 10.9 scrub ok 2024-02-05T08:20:17.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[39278]: 4.6 scrub starts 2024-02-05T08:20:17.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[39278]: 4.6 scrub ok 2024-02-05T08:20:17.530 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[39278]: 6.1f scrub starts 2024-02-05T08:20:17.530 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-mon[39278]: 6.1f scrub ok 2024-02-05T08:20:17.541 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:17.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:17.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:17.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:17.656 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:20:17.678 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:20:17.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:17.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:17.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:17.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:17.806 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:20:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:20:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:20:17.806 INFO:teuthology.orchestra.run.smithi134.stderr:Exporting 5.c info 5.c( empty local-lis/les=70/71 n=0 ec=70/70 lis/c=70/70 les/c/f=71/71/0 sis=70) 2024-02-05T08:20:17.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:17 smithi196 ceph-mon[33018]: pgmap v695: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s; 17108/274647 objects degraded (6.229%) 2024-02-05T08:20:17.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:17 smithi196 ceph-mon[33018]: 5.d scrub starts 2024-02-05T08:20:17.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:17 smithi196 ceph-mon[33018]: 5.d scrub ok 2024-02-05T08:20:17.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:17 smithi196 ceph-mon[33018]: 3.15 scrub starts 2024-02-05T08:20:17.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:17 smithi196 ceph-mon[33018]: 6.1e scrub starts 2024-02-05T08:20:17.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:17 smithi196 ceph-mon[33018]: 3.15 scrub ok 2024-02-05T08:20:17.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:17 smithi196 ceph-mon[33018]: 6.1e scrub ok 2024-02-05T08:20:17.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:17 smithi196 ceph-mon[33018]: 10.9 scrub ok 2024-02-05T08:20:17.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:17 smithi196 ceph-mon[33018]: 4.6 scrub starts 2024-02-05T08:20:17.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:17 smithi196 ceph-mon[33018]: 4.6 scrub ok 2024-02-05T08:20:17.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:17 smithi196 ceph-mon[33018]: 6.1f scrub starts 2024-02-05T08:20:17.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:17 smithi196 ceph-mon[33018]: 6.1f scrub ok 2024-02-05T08:20:17.818 INFO:teuthology.orchestra.run.smithi134.stderr:Export successful 2024-02-05T08:20:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:17.829 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:17.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:17.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:18.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:18.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:18.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:18.134 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 58 16 97000 96984 104.492 70.0625 0.00707516 0.00956451 2024-02-05T08:20:18.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:18.199 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:18.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:18.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:18.330 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:18.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:18.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:18.431 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:18.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:18.468 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:18.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:18.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:18.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:18.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:18.661 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:18 smithi134 ceph-mon[30086]: 3.11 scrub starts 2024-02-05T08:20:18.661 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:18 smithi134 ceph-mon[30086]: 3.11 scrub ok 2024-02-05T08:20:18.662 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:18 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:20:18.662 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:18 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:20:18.662 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:18 smithi134 ceph-mon[30086]: 6.1b scrub starts 2024-02-05T08:20:18.662 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:18 smithi134 ceph-mon[30086]: 6.1b scrub ok 2024-02-05T08:20:18.662 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:18 smithi134 ceph-mon[39278]: 3.11 scrub starts 2024-02-05T08:20:18.662 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:18 smithi134 ceph-mon[39278]: 3.11 scrub ok 2024-02-05T08:20:18.662 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:18 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:20:18.662 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:18 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:20:18.662 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:18 smithi134 ceph-mon[39278]: 6.1b scrub starts 2024-02-05T08:20:18.662 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:18 smithi134 ceph-mon[39278]: 6.1b scrub ok 2024-02-05T08:20:18.709 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:18.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:18.743 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:18.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:18 smithi196 ceph-mon[33018]: 3.11 scrub starts 2024-02-05T08:20:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:18 smithi196 ceph-mon[33018]: 3.11 scrub ok 2024-02-05T08:20:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:18 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:20:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:18 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:20:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:18 smithi196 ceph-mon[33018]: 6.1b scrub starts 2024-02-05T08:20:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:18 smithi196 ceph-mon[33018]: 6.1b scrub ok 2024-02-05T08:20:18.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:18.879 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:18.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:18.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:18.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:19.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:19.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:19.118 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:19.135 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 59 16 98250 98234 104.045 78.125 0.0102772 0.00960645 2024-02-05T08:20:19.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:19.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:19.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:19.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:19.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:19.300 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:19.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:19.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:19.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:19.448 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:19.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:19.479 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:20:19.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[39278]: pgmap v696: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s; 17823/280884 objects degraded (6.345%) 2024-02-05T08:20:19.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[39278]: 5.10 scrub starts 2024-02-05T08:20:19.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[39278]: 5.10 scrub ok 2024-02-05T08:20:19.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[39278]: 4.5 scrub starts 2024-02-05T08:20:19.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[39278]: 4.5 scrub ok 2024-02-05T08:20:19.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[39278]: 4.12 scrub starts 2024-02-05T08:20:19.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[39278]: 4.12 scrub ok 2024-02-05T08:20:19.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[30086]: pgmap v696: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s; 17823/280884 objects degraded (6.345%) 2024-02-05T08:20:19.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[30086]: 5.10 scrub starts 2024-02-05T08:20:19.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[30086]: 5.10 scrub ok 2024-02-05T08:20:19.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[30086]: 4.5 scrub starts 2024-02-05T08:20:19.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[30086]: 4.5 scrub ok 2024-02-05T08:20:19.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[30086]: 4.12 scrub starts 2024-02-05T08:20:19.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:19 smithi134 ceph-mon[30086]: 4.12 scrub ok 2024-02-05T08:20:19.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:19.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:19.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:19.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:19.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:19.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:19.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:19.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:19 smithi196 ceph-mon[33018]: pgmap v696: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.16k op/s; 17823/280884 objects degraded (6.345%) 2024-02-05T08:20:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:19 smithi196 ceph-mon[33018]: 5.10 scrub starts 2024-02-05T08:20:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:19 smithi196 ceph-mon[33018]: 5.10 scrub ok 2024-02-05T08:20:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:19 smithi196 ceph-mon[33018]: 4.5 scrub starts 2024-02-05T08:20:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:19 smithi196 ceph-mon[33018]: 4.5 scrub ok 2024-02-05T08:20:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:19 smithi196 ceph-mon[33018]: 4.12 scrub starts 2024-02-05T08:20:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:19 smithi196 ceph-mon[33018]: 4.12 scrub ok 2024-02-05T08:20:19.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:19.859 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:19.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:19.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:20.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:20.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:20.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:20.135 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:20:20.136086+0000 min lat: 0.00127789 max lat: 1.05979 avg lat: 0.00964361 2024-02-05T08:20:20.135 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:20:20.135 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 60 16 99532 99516 103.646 80.125 0.0141103 0.00964361 2024-02-05T08:20:20.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:20.181 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:20.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:20.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:20.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:20.314 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:20.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:20.416 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:20.476 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -n osd.1 --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 5.c 2024-02-05T08:20:20.477 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:20.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:20.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:20.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:20.615 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:20.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:20.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:20.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:20:20.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:20:20.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[30086]: 6.3 scrub starts 2024-02-05T08:20:20.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[30086]: 4.1a scrub starts 2024-02-05T08:20:20.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[30086]: 4.1a scrub ok 2024-02-05T08:20:20.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[30086]: 6.3 scrub ok 2024-02-05T08:20:20.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[30086]: 1.6 scrub starts 2024-02-05T08:20:20.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[30086]: 1.6 scrub ok 2024-02-05T08:20:20.679 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 17823/280884 objects degraded (6.345%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:20.679 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:20:20.680 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:20:20.680 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[39278]: 6.3 scrub starts 2024-02-05T08:20:20.680 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[39278]: 4.1a scrub starts 2024-02-05T08:20:20.680 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[39278]: 4.1a scrub ok 2024-02-05T08:20:20.680 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[39278]: 6.3 scrub ok 2024-02-05T08:20:20.680 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[39278]: 1.6 scrub starts 2024-02-05T08:20:20.680 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[39278]: 1.6 scrub ok 2024-02-05T08:20:20.680 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:20 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 17823/280884 objects degraded (6.345%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:20.717 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:20.781 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:20.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:20.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:20 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:20:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:20 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:20:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:20 smithi196 ceph-mon[33018]: 6.3 scrub starts 2024-02-05T08:20:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:20 smithi196 ceph-mon[33018]: 4.1a scrub starts 2024-02-05T08:20:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:20 smithi196 ceph-mon[33018]: 4.1a scrub ok 2024-02-05T08:20:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:20 smithi196 ceph-mon[33018]: 6.3 scrub ok 2024-02-05T08:20:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:20 smithi196 ceph-mon[33018]: 1.6 scrub starts 2024-02-05T08:20:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:20 smithi196 ceph-mon[33018]: 1.6 scrub ok 2024-02-05T08:20:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:20 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 17823/280884 objects degraded (6.345%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:20.883 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:20.926 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:20.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:21.028 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:21.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:21.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:21.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:21.135 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 61 16 100760 100744 103.205 76.75 0.00506403 0.00968479 2024-02-05T08:20:21.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:21.207 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:21.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:21.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:21.309 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:21.346 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:21.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:21.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:21.482 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:21.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:21.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:21.584 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:21.620 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:21.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:21.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:21.722 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:21.753 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[30086]: pgmap v697: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s; 18164/291384 objects degraded (6.234%) 2024-02-05T08:20:21.753 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[30086]: 5.4 scrub starts 2024-02-05T08:20:21.753 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[30086]: 5.4 scrub ok 2024-02-05T08:20:21.753 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[30086]: 3.14 scrub starts 2024-02-05T08:20:21.753 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[30086]: 3.14 scrub ok 2024-02-05T08:20:21.753 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[30086]: 3.b scrub starts 2024-02-05T08:20:21.753 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[30086]: 3.b scrub ok 2024-02-05T08:20:21.754 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[39278]: pgmap v697: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s; 18164/291384 objects degraded (6.234%) 2024-02-05T08:20:21.754 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[39278]: 5.4 scrub starts 2024-02-05T08:20:21.754 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[39278]: 5.4 scrub ok 2024-02-05T08:20:21.754 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[39278]: 3.14 scrub starts 2024-02-05T08:20:21.754 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[39278]: 3.14 scrub ok 2024-02-05T08:20:21.754 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[39278]: 3.b scrub starts 2024-02-05T08:20:21.754 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:21 smithi134 ceph-mon[39278]: 3.b scrub ok 2024-02-05T08:20:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:21.756 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:21.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:21 smithi196 ceph-mon[33018]: pgmap v697: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s; 18164/291384 objects degraded (6.234%) 2024-02-05T08:20:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:21 smithi196 ceph-mon[33018]: 5.4 scrub starts 2024-02-05T08:20:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:21 smithi196 ceph-mon[33018]: 5.4 scrub ok 2024-02-05T08:20:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:21 smithi196 ceph-mon[33018]: 3.14 scrub starts 2024-02-05T08:20:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:21 smithi196 ceph-mon[33018]: 3.14 scrub ok 2024-02-05T08:20:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:21 smithi196 ceph-mon[33018]: 3.b scrub starts 2024-02-05T08:20:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:21 smithi196 ceph-mon[33018]: 3.b scrub ok 2024-02-05T08:20:21.857 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:21.893 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:21.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:21.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:22.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:22.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:22.132 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/osd.1/config 2024-02-05T08:20:22.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:22.135 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 62 16 102056 102040 102.846 81 0.0143753 0.00971772 2024-02-05T08:20:22.183 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:22.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:22.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:22.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:22.323 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:22.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:22.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:22.425 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:22.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:22.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:22.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:22.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:22.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:22.706 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:22.739 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:22.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:22.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:22.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:22 smithi134 ceph-mon[30086]: 6.5 scrub starts 2024-02-05T08:20:22.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:22 smithi134 ceph-mon[30086]: 6.5 scrub ok 2024-02-05T08:20:22.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:22 smithi134 ceph-mon[39278]: 6.5 scrub starts 2024-02-05T08:20:22.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:22 smithi134 ceph-mon[39278]: 6.5 scrub ok 2024-02-05T08:20:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:22 smithi196 ceph-mon[33018]: 6.5 scrub starts 2024-02-05T08:20:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:22 smithi196 ceph-mon[33018]: 6.5 scrub ok 2024-02-05T08:20:22.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:22.881 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:22.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:22.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:23.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:23.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:23.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:23.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:23.135 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 63 15 103335 103320 102.484 80 0.0113586 0.0097535 2024-02-05T08:20:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:23.173 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:23.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:23.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:23.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:23.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:23.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:23.447 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:23.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:23.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:23.588 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:23.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:23.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:23.690 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:23.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:23.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:23.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:23.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[30086]: pgmap v698: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 988 B/s rd, 69 MiB/s wr, 1.10k op/s; 18164/291384 objects degraded (6.234%) 2024-02-05T08:20:23.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:20:23.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:20:23.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[30086]: osdmap e328: 8 total, 7 up, 8 in 2024-02-05T08:20:23.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[30086]: 4.15 scrub starts 2024-02-05T08:20:23.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[30086]: 4.15 scrub ok 2024-02-05T08:20:23.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:20:23.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[39278]: pgmap v698: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 988 B/s rd, 69 MiB/s wr, 1.10k op/s; 18164/291384 objects degraded (6.234%) 2024-02-05T08:20:23.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:20:23.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:20:23.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[39278]: osdmap e328: 8 total, 7 up, 8 in 2024-02-05T08:20:23.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[39278]: 4.15 scrub starts 2024-02-05T08:20:23.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[39278]: 4.15 scrub ok 2024-02-05T08:20:23.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:20:23.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:23 smithi196 ceph-mon[33018]: pgmap v698: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 988 B/s rd, 69 MiB/s wr, 1.10k op/s; 18164/291384 objects degraded (6.234%) 2024-02-05T08:20:23.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:23 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:20:23.789 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:23 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:20:23.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:23 smithi196 ceph-mon[33018]: osdmap e328: 8 total, 7 up, 8 in 2024-02-05T08:20:23.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:23 smithi196 ceph-mon[33018]: 4.15 scrub starts 2024-02-05T08:20:23.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:23 smithi196 ceph-mon[33018]: 4.15 scrub ok 2024-02-05T08:20:23.790 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:20:23.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:23.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:23.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:23.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:23.965 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:24.003 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:24.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:24.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:20:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:20:24.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:24.135 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 64 16 104538 104522 102.056 75.125 0.00688454 0.00979304 2024-02-05T08:20:24.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:24.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:24.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:24.245 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:24.282 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:24.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:24.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:24.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:24.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:24.527 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:24.573 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:24.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:24.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:24.675 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:24.713 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:24.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:24.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:24.718 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:24 smithi134 ceph-mon[30086]: 10.a scrub starts 2024-02-05T08:20:24.718 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:24 smithi134 ceph-mon[30086]: 6.11 scrub starts 2024-02-05T08:20:24.718 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:24 smithi134 ceph-mon[30086]: 6.11 scrub ok 2024-02-05T08:20:24.718 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:24 smithi134 ceph-mon[39278]: 10.a scrub starts 2024-02-05T08:20:24.718 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:24 smithi134 ceph-mon[39278]: 6.11 scrub starts 2024-02-05T08:20:24.718 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:24 smithi134 ceph-mon[39278]: 6.11 scrub ok 2024-02-05T08:20:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:24 smithi196 ceph-mon[33018]: 10.a scrub starts 2024-02-05T08:20:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:24 smithi196 ceph-mon[33018]: 6.11 scrub starts 2024-02-05T08:20:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:24 smithi196 ceph-mon[33018]: 6.11 scrub ok 2024-02-05T08:20:24.815 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:24.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:24.868 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:24.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:24.969 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:25.007 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:25.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:25.017 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:20:25.055 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:20:25.091 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:20:25.109 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:25.129 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:20:25.135 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 65 16 105661 105645 101.566 70.1875 0.0127573 0.00984078 2024-02-05T08:20:25.148 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:25.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:25.165 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:20:25.204 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:20:25.241 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:20:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:20:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:20:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:20:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:20:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:20:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:20:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:20:25.242 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:20:25.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:25.291 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:25.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:25.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:25.393 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:25.428 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:25.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:25.530 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:25.567 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:25.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:25.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:25.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:25.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:25.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:25.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:25.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[30086]: pgmap v700: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s; 19358/307611 objects degraded (6.293%) 2024-02-05T08:20:25.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:20:25.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:20:25.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:20:25.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:20:25.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 19358/307611 objects degraded (6.293%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:25.775 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/658615135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:20:25.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/127910533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:25.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[30086]: 1.1 scrub starts 2024-02-05T08:20:25.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[30086]: 1.1 scrub ok 2024-02-05T08:20:25.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:25.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[39278]: pgmap v700: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s; 19358/307611 objects degraded (6.293%) 2024-02-05T08:20:25.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:20:25.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:20:25.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:20:25.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:20:25.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 19358/307611 objects degraded (6.293%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:25.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/658615135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:20:25.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/127910533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:25.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[39278]: 1.1 scrub starts 2024-02-05T08:20:25.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:25 smithi134 ceph-mon[39278]: 1.1 scrub ok 2024-02-05T08:20:25.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:25 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:25 smithi196 ceph-mon[33018]: pgmap v700: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s; 19358/307611 objects degraded (6.293%) 2024-02-05T08:20:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:25 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:20:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:25 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:20:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:25 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:20:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:25 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:20:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:25 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 19358/307611 objects degraded (6.293%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:25 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/658615135' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:20:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:25 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/127910533' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:25 smithi196 ceph-mon[33018]: 1.1 scrub starts 2024-02-05T08:20:25.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:25 smithi196 ceph-mon[33018]: 1.1 scrub ok 2024-02-05T08:20:25.848 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:25.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:25.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:25.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:25.986 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:25.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:26.087 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:26.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:26.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:26.135 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 66 16 106882 106866 101.183 76.3125 0.00966626 0.00987861 2024-02-05T08:20:26.223 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:26.262 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:26.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:26.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:26.404 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:26.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:26.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:26.427 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:20:26.505 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:26.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:26.548 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:26.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:26.648 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:26.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:26.700 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:26.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:26.765 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:26 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:20:26.765 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:26 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:20:26.765 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:26 smithi134 ceph-mon[30086]: 10.a scrub ok 2024-02-05T08:20:26.765 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:26 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:20:26.765 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:26 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/658615135' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:20:26.765 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:26 smithi134 ceph-mon[30086]: osdmap e329: 8 total, 7 up, 8 in 2024-02-05T08:20:26.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:26 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:20:26.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:26 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:20:26.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:26 smithi134 ceph-mon[39278]: 10.a scrub ok 2024-02-05T08:20:26.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:26 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:20:26.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:26 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/658615135' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:20:26.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:26 smithi134 ceph-mon[39278]: osdmap e329: 8 total, 7 up, 8 in 2024-02-05T08:20:26.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:26 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:20:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:26 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:20:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:26 smithi196 ceph-mon[33018]: 10.a scrub ok 2024-02-05T08:20:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:26 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:20:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:26 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/658615135' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:20:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:26 smithi196 ceph-mon[33018]: osdmap e329: 8 total, 7 up, 8 in 2024-02-05T08:20:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:26.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:26.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:26.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:26.961 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:26.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:26.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:27.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:27.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:27.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:27.135 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 67 16 108120 108104 100.828 77.375 0.0107386 0.00991351 2024-02-05T08:20:27.201 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:27.235 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:27.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:27.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:27.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:27.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:27.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:27.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:27.508 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:27.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:27.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:27.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:27.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:27.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:27.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:27.746 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:27.784 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:27.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:27.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:27 smithi196 ceph-mon[33018]: pgmap v701: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s; 19358/309933 objects degraded (6.246%) 2024-02-05T08:20:27.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:27 smithi196 ceph-mon[33018]: 6.13 deep-scrub starts 2024-02-05T08:20:27.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:27 smithi196 ceph-mon[33018]: 6.13 deep-scrub ok 2024-02-05T08:20:27.885 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:27.921 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:27.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:27.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:28.023 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:28.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:27 smithi134 ceph-mon[30086]: pgmap v701: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s; 19358/309933 objects degraded (6.246%) 2024-02-05T08:20:28.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:27 smithi134 ceph-mon[30086]: 6.13 deep-scrub starts 2024-02-05T08:20:28.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:27 smithi134 ceph-mon[30086]: 6.13 deep-scrub ok 2024-02-05T08:20:28.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:20:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:20:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:20:28.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:27 smithi134 ceph-mon[39278]: pgmap v701: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s; 19358/309933 objects degraded (6.246%) 2024-02-05T08:20:28.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:27 smithi134 ceph-mon[39278]: 6.13 deep-scrub starts 2024-02-05T08:20:28.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:27 smithi134 ceph-mon[39278]: 6.13 deep-scrub ok 2024-02-05T08:20:28.060 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:28.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:28.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:28.136 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 68 15 109317 109302 100.446 74.875 0.0103291 0.00995125 2024-02-05T08:20:28.162 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:28.202 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:28.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:28.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:28.303 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:28.345 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:28.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:28.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:28.446 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:28.479 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:28.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:20:28.545 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:20:28.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:28.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:28.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:28.716 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:28.756 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:28.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:28.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[39278]: 3.3 scrub starts 2024-02-05T08:20:28.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[39278]: 3.3 scrub ok 2024-02-05T08:20:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[39278]: 1.2 scrub starts 2024-02-05T08:20:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[39278]: 1.2 scrub ok 2024-02-05T08:20:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[39278]: 5.11 scrub starts 2024-02-05T08:20:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[39278]: 5.11 scrub ok 2024-02-05T08:20:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[39278]: 1.4 scrub starts 2024-02-05T08:20:28.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[39278]: 1.4 scrub ok 2024-02-05T08:20:28.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[30086]: 3.3 scrub starts 2024-02-05T08:20:28.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[30086]: 3.3 scrub ok 2024-02-05T08:20:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[30086]: 1.2 scrub starts 2024-02-05T08:20:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[30086]: 1.2 scrub ok 2024-02-05T08:20:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[30086]: 5.11 scrub starts 2024-02-05T08:20:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[30086]: 5.11 scrub ok 2024-02-05T08:20:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[30086]: 1.4 scrub starts 2024-02-05T08:20:28.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:28 smithi134 ceph-mon[30086]: 1.4 scrub ok 2024-02-05T08:20:28.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:28 smithi196 ceph-mon[33018]: 3.3 scrub starts 2024-02-05T08:20:28.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:28 smithi196 ceph-mon[33018]: 3.3 scrub ok 2024-02-05T08:20:28.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:28 smithi196 ceph-mon[33018]: 1.2 scrub starts 2024-02-05T08:20:28.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:28 smithi196 ceph-mon[33018]: 1.2 scrub ok 2024-02-05T08:20:28.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:28 smithi196 ceph-mon[33018]: 5.11 scrub starts 2024-02-05T08:20:28.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:28 smithi196 ceph-mon[33018]: 5.11 scrub ok 2024-02-05T08:20:28.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:28 smithi196 ceph-mon[33018]: 1.4 scrub starts 2024-02-05T08:20:28.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:28 smithi196 ceph-mon[33018]: 1.4 scrub ok 2024-02-05T08:20:28.859 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:28.892 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:28.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:28.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:28.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:29.031 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:29.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:29.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:29.108 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:20:29.108 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:20:29.132 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:29.136 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 69 15 110600 110585 100.152 80.1875 0.0212587 0.00997993 2024-02-05T08:20:29.185 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:29.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:29.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:29.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:29.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:29.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:29.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:29.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:29.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:29.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:29.563 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:29.602 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:29.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:29.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:29.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:29.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:29.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:29 smithi134 ceph-mon[30086]: pgmap v703: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.19k op/s; 20457/320001 objects degraded (6.393%) 2024-02-05T08:20:29.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:29 smithi134 ceph-mon[39278]: pgmap v703: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.19k op/s; 20457/320001 objects degraded (6.393%) 2024-02-05T08:20:29.813 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:29 smithi196 ceph-mon[33018]: pgmap v703: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.19k op/s; 20457/320001 objects degraded (6.393%) 2024-02-05T08:20:29.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:29.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:29.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:29.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:29.971 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:30.007 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:30.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:30.109 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:30.136 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 70 16 111846 111830 99.8328 77.8125 0.013633 0.0100115 2024-02-05T08:20:30.152 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:30.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:30.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:30.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:30.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:30.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:30.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:30.390 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:30.431 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:30.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:30.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:30.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:30.546 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:20:30.577 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:30.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:30.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:30.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:30.713 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:30.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:30 smithi196 ceph-mon[33018]: 6.16 scrub starts 2024-02-05T08:20:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:30 smithi196 ceph-mon[33018]: 4.19 scrub starts 2024-02-05T08:20:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:30 smithi196 ceph-mon[33018]: 4.19 scrub ok 2024-02-05T08:20:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:30 smithi196 ceph-mon[33018]: 6.16 scrub ok 2024-02-05T08:20:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:30 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:30 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:30 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 20457/320001 objects degraded (6.393%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:30 smithi196 ceph-mon[33018]: 5.f scrub starts 2024-02-05T08:20:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:30 smithi196 ceph-mon[33018]: 5.f scrub ok 2024-02-05T08:20:30.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:30.840 INFO:teuthology.orchestra.run.smithi134.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-05T08:20:30.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:30.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:30.864 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[39278]: 6.16 scrub starts 2024-02-05T08:20:30.864 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[39278]: 4.19 scrub starts 2024-02-05T08:20:30.864 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[39278]: 4.19 scrub ok 2024-02-05T08:20:30.864 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[39278]: 6.16 scrub ok 2024-02-05T08:20:30.864 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:30.864 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:30.864 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 20457/320001 objects degraded (6.393%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:30.864 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[39278]: 5.f scrub starts 2024-02-05T08:20:30.864 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[39278]: 5.f scrub ok 2024-02-05T08:20:30.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[30086]: 6.16 scrub starts 2024-02-05T08:20:30.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[30086]: 4.19 scrub starts 2024-02-05T08:20:30.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[30086]: 4.19 scrub ok 2024-02-05T08:20:30.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[30086]: 6.16 scrub ok 2024-02-05T08:20:30.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:30.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:30.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 20457/320001 objects degraded (6.393%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:30.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[30086]: 5.f scrub starts 2024-02-05T08:20:30.866 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:30 smithi134 ceph-mon[30086]: 5.f scrub ok 2024-02-05T08:20:30.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:30.995 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:30.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:31.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:31.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:31.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:31.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:31.136 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 71 16 113059 113043 99.4944 75.8125 0.00893924 0.0100464 2024-02-05T08:20:31.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:31.268 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:31.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:31.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T08:20:31.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:31.419 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:31.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:31.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:31.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:31.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:31.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:31.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:31 smithi196 ceph-mon[33018]: pgmap v704: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s; 20457/328236 objects degraded (6.232%) 2024-02-05T08:20:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:31 smithi196 ceph-mon[33018]: 2.1 deep-scrub starts 2024-02-05T08:20:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:31 smithi196 ceph-mon[33018]: 2.1 deep-scrub ok 2024-02-05T08:20:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:31 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:20:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:31 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:20:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:31 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:31.874 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:31 smithi134 ceph-mon[30086]: pgmap v704: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s; 20457/328236 objects degraded (6.232%) 2024-02-05T08:20:31.874 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:31 smithi134 ceph-mon[30086]: 2.1 deep-scrub starts 2024-02-05T08:20:31.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:31 smithi134 ceph-mon[30086]: 2.1 deep-scrub ok 2024-02-05T08:20:31.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:31 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:20:31.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:31 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:20:31.875 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:31 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:31.875 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:31 smithi134 ceph-mon[39278]: pgmap v704: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.54k op/s; 20457/328236 objects degraded (6.232%) 2024-02-05T08:20:31.875 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:31 smithi134 ceph-mon[39278]: 2.1 deep-scrub starts 2024-02-05T08:20:31.875 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:31 smithi134 ceph-mon[39278]: 2.1 deep-scrub ok 2024-02-05T08:20:31.875 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:31 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:20:31.875 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:31 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:20:31.875 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:31 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:31.884 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:20:31.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:31.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:32.051 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:32.103 INFO:teuthology.orchestra.run.smithi134.stdout: marking collection for removal 2024-02-05T08:20:32.104 INFO:teuthology.orchestra.run.smithi134.stdout:setting '_remove' omap key 2024-02-05T08:20:32.105 INFO:teuthology.orchestra.run.smithi134.stdout:finish_remove_pgs 5.c_head removing 5.c 2024-02-05T08:20:32.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:32.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:32.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:32.112 INFO:teuthology.orchestra.run.smithi134.stdout:Remove successful 2024-02-05T08:20:32.136 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 72 16 114271 114255 99.1646 75.75 0.00779405 0.0100793 2024-02-05T08:20:32.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:32.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:32.266 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:20:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:32.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:32.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:20:32.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:32.413 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:32.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:32.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:32.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:32.548 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:32.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:32.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:32.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:32.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:32.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:32.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:32.810 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:32.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:32 smithi196 ceph-mon[33018]: 5.1 deep-scrub starts 2024-02-05T08:20:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:32 smithi196 ceph-mon[33018]: 5.1 deep-scrub ok 2024-02-05T08:20:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:32 smithi196 ceph-mon[33018]: 3.1b scrub starts 2024-02-05T08:20:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:32 smithi196 ceph-mon[33018]: 3.1b scrub ok 2024-02-05T08:20:32.910 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:32.951 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:32.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:32.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:33.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:32 smithi134 ceph-mon[30086]: 5.1 deep-scrub starts 2024-02-05T08:20:33.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:32 smithi134 ceph-mon[30086]: 5.1 deep-scrub ok 2024-02-05T08:20:33.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:32 smithi134 ceph-mon[30086]: 3.1b scrub starts 2024-02-05T08:20:33.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:32 smithi134 ceph-mon[30086]: 3.1b scrub ok 2024-02-05T08:20:33.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:32 smithi134 ceph-mon[39278]: 5.1 deep-scrub starts 2024-02-05T08:20:33.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:32 smithi134 ceph-mon[39278]: 5.1 deep-scrub ok 2024-02-05T08:20:33.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:32 smithi134 ceph-mon[39278]: 3.1b scrub starts 2024-02-05T08:20:33.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:32 smithi134 ceph-mon[39278]: 3.1b scrub ok 2024-02-05T08:20:33.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:33.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:33.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:33.136 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 73 16 115325 115309 98.7084 65.875 0.0117436 0.0101265 2024-02-05T08:20:33.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:33.224 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:33.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:33.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:33.326 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:33.362 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:33.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:33.464 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:33.499 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:33.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:33.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:33.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:33.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:33.638 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:33.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:33.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:33.773 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:33.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:33.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:33.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:33 smithi134 ceph-mon[30086]: pgmap v705: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 20457/328236 objects degraded (6.232%) 2024-02-05T08:20:33.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:33 smithi134 ceph-mon[30086]: 10.c scrub starts 2024-02-05T08:20:33.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:33 smithi134 ceph-mon[39278]: pgmap v705: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 20457/328236 objects degraded (6.232%) 2024-02-05T08:20:33.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:33 smithi134 ceph-mon[39278]: 10.c scrub starts 2024-02-05T08:20:33.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:33 smithi196 ceph-mon[33018]: pgmap v705: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 20457/328236 objects degraded (6.232%) 2024-02-05T08:20:33.800 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:33 smithi196 ceph-mon[33018]: 10.c scrub starts 2024-02-05T08:20:33.875 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:33.921 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:33.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:33.922 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -n osd.1 --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.c.1 2024-02-05T08:20:33.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:34.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:34.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:20:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:20:34.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:34.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:34.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:34.137 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 74 16 116422 116406 98.3003 68.5625 0.0119389 0.0101681 2024-02-05T08:20:34.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:34.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:34.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:34.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:34.303 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:34.348 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:34.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:34.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:34.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:34.498 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:34.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:34.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:34 smithi196 ceph-mon[33018]: 1.0 deep-scrub starts 2024-02-05T08:20:34.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:34 smithi196 ceph-mon[33018]: 1.0 deep-scrub ok 2024-02-05T08:20:34.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:34 smithi196 ceph-mon[33018]: 10.4 scrub starts 2024-02-05T08:20:34.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:34 smithi196 ceph-mon[33018]: 1.a scrub starts 2024-02-05T08:20:34.597 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:34 smithi196 ceph-mon[33018]: 1.a scrub ok 2024-02-05T08:20:34.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:34.642 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:34.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:34.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:34.724 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/osd.1/config 2024-02-05T08:20:34.734 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:20:34.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:34.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:34.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:34.851 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:34 smithi134 ceph-mon[39278]: 1.0 deep-scrub starts 2024-02-05T08:20:34.851 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:34 smithi134 ceph-mon[39278]: 1.0 deep-scrub ok 2024-02-05T08:20:34.851 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:34 smithi134 ceph-mon[39278]: 10.4 scrub starts 2024-02-05T08:20:34.851 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:34 smithi134 ceph-mon[39278]: 1.a scrub starts 2024-02-05T08:20:34.851 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:34 smithi134 ceph-mon[39278]: 1.a scrub ok 2024-02-05T08:20:34.851 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:34 smithi134 ceph-mon[30086]: 1.0 deep-scrub starts 2024-02-05T08:20:34.851 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:34 smithi134 ceph-mon[30086]: 1.0 deep-scrub ok 2024-02-05T08:20:34.851 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:34 smithi134 ceph-mon[30086]: 10.4 scrub starts 2024-02-05T08:20:34.852 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:34 smithi134 ceph-mon[30086]: 1.a scrub starts 2024-02-05T08:20:34.852 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:34 smithi134 ceph-mon[30086]: 1.a scrub ok 2024-02-05T08:20:34.883 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:34.923 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:34.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:34.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:35.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:35.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:35.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:35.131 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:20:35.131 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:20:35.137 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 75 16 117444 117428 97.8413 63.875 0.0183974 0.0102156 2024-02-05T08:20:35.165 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:35.204 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:35.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:35.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:35.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:35.355 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:35.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:35.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:35.457 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:35.499 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:35.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:35.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:35.650 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:35.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:35.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:35.681 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:35.681 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[30086]: pgmap v706: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s; 21587/343935 objects degraded (6.276%) 2024-02-05T08:20:35.681 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[30086]: 10.3 scrub starts 2024-02-05T08:20:35.681 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[30086]: 3.f scrub starts 2024-02-05T08:20:35.681 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[30086]: 3.f scrub ok 2024-02-05T08:20:35.681 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[30086]: 10.c scrub ok 2024-02-05T08:20:35.682 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 21587/343935 objects degraded (6.276%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:35.682 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[30086]: 10.5 scrub starts 2024-02-05T08:20:35.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:35.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[39278]: pgmap v706: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s; 21587/343935 objects degraded (6.276%) 2024-02-05T08:20:35.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[39278]: 10.3 scrub starts 2024-02-05T08:20:35.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[39278]: 3.f scrub starts 2024-02-05T08:20:35.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[39278]: 3.f scrub ok 2024-02-05T08:20:35.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[39278]: 10.c scrub ok 2024-02-05T08:20:35.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 21587/343935 objects degraded (6.276%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:35.682 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:35 smithi134 ceph-mon[39278]: 10.5 scrub starts 2024-02-05T08:20:35.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:35.792 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:35.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:35 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:35 smithi196 ceph-mon[33018]: pgmap v706: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s; 21587/343935 objects degraded (6.276%) 2024-02-05T08:20:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:35 smithi196 ceph-mon[33018]: 10.3 scrub starts 2024-02-05T08:20:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:35 smithi196 ceph-mon[33018]: 3.f scrub starts 2024-02-05T08:20:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:35 smithi196 ceph-mon[33018]: 3.f scrub ok 2024-02-05T08:20:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:35 smithi196 ceph-mon[33018]: 10.c scrub ok 2024-02-05T08:20:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:35 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 21587/343935 objects degraded (6.276%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:35 smithi196 ceph-mon[33018]: 10.5 scrub starts 2024-02-05T08:20:35.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T08:20:35.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:35.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:35.955 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:35.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:36.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:36.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:36.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:36.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:36.140 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 76 16 118383 118367 97.3218 58.6875 0.0154588 0.0102697 2024-02-05T08:20:36.204 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:36.242 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:36.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:36.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:36.340 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:20:36.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:36.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:36.399 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:36.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:36.501 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:36.539 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:36.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:36.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:36.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:36.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:36.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:36.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:36.797 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:20:36.797 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:20:36.797 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:20:36.797 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:20:36.797 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:20:36.797 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:20:36.797 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:20:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:20:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:20:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:20:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:20:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:20:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:20:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:20:36.798 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:20:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:36 smithi196 ceph-mon[33018]: 10.1 scrub starts 2024-02-05T08:20:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3707706786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:36 smithi196 ceph-mon[33018]: 1.7 deep-scrub starts 2024-02-05T08:20:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:36 smithi196 ceph-mon[33018]: 1.7 deep-scrub ok 2024-02-05T08:20:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:36 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:20:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:36 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:20:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:36 smithi196 ceph-mon[33018]: 10.6 scrub starts 2024-02-05T08:20:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:36 smithi196 ceph-mon[33018]: 10.3 scrub ok 2024-02-05T08:20:36.827 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:36.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:36.929 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:36.964 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:36.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:37.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[30086]: 10.1 scrub starts 2024-02-05T08:20:37.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3707706786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:37.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[30086]: 1.7 deep-scrub starts 2024-02-05T08:20:37.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[30086]: 1.7 deep-scrub ok 2024-02-05T08:20:37.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:20:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:20:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[30086]: 10.6 scrub starts 2024-02-05T08:20:37.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[30086]: 10.3 scrub ok 2024-02-05T08:20:37.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[39278]: 10.1 scrub starts 2024-02-05T08:20:37.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3707706786' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:37.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[39278]: 1.7 deep-scrub starts 2024-02-05T08:20:37.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[39278]: 1.7 deep-scrub ok 2024-02-05T08:20:37.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:20:37.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:20:37.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[39278]: 10.6 scrub starts 2024-02-05T08:20:37.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:36 smithi134 ceph-mon[39278]: 10.3 scrub ok 2024-02-05T08:20:37.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:37.114 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:37.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:37.140 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 77 16 119346 119330 96.8395 60.1875 0.0108999 0.0103218 2024-02-05T08:20:37.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:37.249 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:37.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:37.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:37.384 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:37.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:37.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:37.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:37.489 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:20:37.490 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:20:37.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:37.524 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:37.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:37.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:37.660 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:37.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:37.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:37.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[30086]: pgmap v707: 158 pgs: 2 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 98 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s; 21587/346113 objects degraded (6.237%) 2024-02-05T08:20:37.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[30086]: 10.4 scrub ok 2024-02-05T08:20:37.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[30086]: 6.2 scrub starts 2024-02-05T08:20:37.732 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[30086]: 6.2 scrub ok 2024-02-05T08:20:37.732 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[30086]: osdmap e330: 8 total, 7 up, 8 in 2024-02-05T08:20:37.732 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:20:37.732 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:20:37.732 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:20:37.732 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:20:37.732 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[30086]: 10.5 scrub ok 2024-02-05T08:20:37.732 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[30086]: 4.0 scrub starts 2024-02-05T08:20:37.732 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[30086]: 4.0 scrub ok 2024-02-05T08:20:37.732 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[39278]: pgmap v707: 158 pgs: 2 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 98 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s; 21587/346113 objects degraded (6.237%) 2024-02-05T08:20:37.732 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[39278]: 10.4 scrub ok 2024-02-05T08:20:37.733 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[39278]: 6.2 scrub starts 2024-02-05T08:20:37.733 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[39278]: 6.2 scrub ok 2024-02-05T08:20:37.733 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[39278]: osdmap e330: 8 total, 7 up, 8 in 2024-02-05T08:20:37.733 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:20:37.733 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:20:37.733 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:20:37.733 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:20:37.733 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[39278]: 10.5 scrub ok 2024-02-05T08:20:37.733 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[39278]: 4.0 scrub starts 2024-02-05T08:20:37.733 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-mon[39278]: 4.0 scrub ok 2024-02-05T08:20:37.762 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:37.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:37.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:37.892 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:37.932 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:37.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:37.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:38.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:20:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:20:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:20:38.034 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:38.063 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:38.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:38.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:37 smithi196 ceph-mon[33018]: pgmap v707: 158 pgs: 2 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 98 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.21k op/s; 21587/346113 objects degraded (6.237%) 2024-02-05T08:20:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:37 smithi196 ceph-mon[33018]: 10.4 scrub ok 2024-02-05T08:20:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:37 smithi196 ceph-mon[33018]: 6.2 scrub starts 2024-02-05T08:20:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:37 smithi196 ceph-mon[33018]: 6.2 scrub ok 2024-02-05T08:20:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:37 smithi196 ceph-mon[33018]: osdmap e330: 8 total, 7 up, 8 in 2024-02-05T08:20:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:37 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:20:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:37 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:20:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:37 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:20:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:37 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:20:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:37 smithi196 ceph-mon[33018]: 10.5 scrub ok 2024-02-05T08:20:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:37 smithi196 ceph-mon[33018]: 4.0 scrub starts 2024-02-05T08:20:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:37 smithi196 ceph-mon[33018]: 4.0 scrub ok 2024-02-05T08:20:38.140 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 78 16 120225 120209 96.3023 54.9375 0.0114264 0.0103791 2024-02-05T08:20:38.165 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:38.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:38.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:38.308 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:38.346 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:38.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:38.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:38.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:38.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:38.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:38.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:38.629 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:38.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:38.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:38.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:38.773 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:38.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:38 smithi196 ceph-mon[33018]: 10.e deep-scrub starts 2024-02-05T08:20:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:38 smithi196 ceph-mon[33018]: 6.1c scrub starts 2024-02-05T08:20:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:38 smithi196 ceph-mon[33018]: 6.1c scrub ok 2024-02-05T08:20:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:38 smithi196 ceph-mon[33018]: 10.1 scrub ok 2024-02-05T08:20:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:38 smithi196 ceph-mon[33018]: 10.8 scrub starts 2024-02-05T08:20:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:38 smithi196 ceph-mon[33018]: osdmap e331: 8 total, 7 up, 8 in 2024-02-05T08:20:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:38 smithi196 ceph-mon[33018]: 5.b scrub starts 2024-02-05T08:20:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:38 smithi196 ceph-mon[33018]: 5.b scrub ok 2024-02-05T08:20:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:38.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:20:38.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:38.908 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:38.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:38.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:39.010 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:39.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[39278]: 10.e deep-scrub starts 2024-02-05T08:20:39.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[39278]: 6.1c scrub starts 2024-02-05T08:20:39.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[39278]: 6.1c scrub ok 2024-02-05T08:20:39.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[39278]: 10.1 scrub ok 2024-02-05T08:20:39.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[39278]: 10.8 scrub starts 2024-02-05T08:20:39.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[39278]: osdmap e331: 8 total, 7 up, 8 in 2024-02-05T08:20:39.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[39278]: 5.b scrub starts 2024-02-05T08:20:39.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[39278]: 5.b scrub ok 2024-02-05T08:20:39.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:39.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:20:39.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[30086]: 10.e deep-scrub starts 2024-02-05T08:20:39.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[30086]: 6.1c scrub starts 2024-02-05T08:20:39.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[30086]: 6.1c scrub ok 2024-02-05T08:20:39.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[30086]: 10.1 scrub ok 2024-02-05T08:20:39.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[30086]: 10.8 scrub starts 2024-02-05T08:20:39.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[30086]: osdmap e331: 8 total, 7 up, 8 in 2024-02-05T08:20:39.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[30086]: 5.b scrub starts 2024-02-05T08:20:39.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[30086]: 5.b scrub ok 2024-02-05T08:20:39.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:39.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:20:39.044 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:39.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:39.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:39.140 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 79 16 121134 121118 95.8025 56.8125 0.00875546 0.010433 2024-02-05T08:20:39.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:39.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:39.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:39.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:39.315 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:39.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:39.415 INFO:teuthology.orchestra.run.smithi134.stderr:Creating an OSD daemon form without an OSD FSID value 2024-02-05T08:20:39.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:39.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:39.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:39.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:39.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:20:39.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:39.584 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:39.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:39.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:39.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:39.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:39.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:39.749 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:39 smithi196 ceph-mon[33018]: pgmap v710: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 97 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s; 22543/354696 objects degraded (6.356%) 2024-02-05T08:20:39.749 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:39 smithi196 ceph-mon[33018]: 5.e scrub starts 2024-02-05T08:20:39.749 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:39 smithi196 ceph-mon[33018]: 5.e scrub ok 2024-02-05T08:20:39.749 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:39 smithi196 ceph-mon[33018]: 10.f scrub starts 2024-02-05T08:20:39.749 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:39 smithi196 ceph-mon[33018]: 6.c scrub starts 2024-02-05T08:20:39.749 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:39 smithi196 ceph-mon[33018]: 6.c scrub ok 2024-02-05T08:20:39.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:39.858 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:39.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:39.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:39.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:39.963 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:20:39.963 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:20:39.963 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T08:20:39.963 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T08:20:39.963 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:20:39.996 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:39.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:39.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:40.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:39 smithi134 ceph-mon[30086]: pgmap v710: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 97 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s; 22543/354696 objects degraded (6.356%) 2024-02-05T08:20:40.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:39 smithi134 ceph-mon[30086]: 5.e scrub starts 2024-02-05T08:20:40.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:39 smithi134 ceph-mon[30086]: 5.e scrub ok 2024-02-05T08:20:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:39 smithi134 ceph-mon[30086]: 10.f scrub starts 2024-02-05T08:20:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:39 smithi134 ceph-mon[30086]: 6.c scrub starts 2024-02-05T08:20:40.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:39 smithi134 ceph-mon[30086]: 6.c scrub ok 2024-02-05T08:20:40.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:39 smithi134 ceph-mon[39278]: pgmap v710: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 97 active+clean; 7.2 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.10k op/s; 22543/354696 objects degraded (6.356%) 2024-02-05T08:20:40.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:39 smithi134 ceph-mon[39278]: 5.e scrub starts 2024-02-05T08:20:40.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:39 smithi134 ceph-mon[39278]: 5.e scrub ok 2024-02-05T08:20:40.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:39 smithi134 ceph-mon[39278]: 10.f scrub starts 2024-02-05T08:20:40.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:39 smithi134 ceph-mon[39278]: 6.c scrub starts 2024-02-05T08:20:40.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:39 smithi134 ceph-mon[39278]: 6.c scrub ok 2024-02-05T08:20:40.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:40.132 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:40.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:40.141 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:20:40.142009+0000 min lat: 0.00127789 max lat: 1.05979 avg lat: 0.0104903 2024-02-05T08:20:40.141 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:20:40.141 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 80 16 121986 121970 95.2706 53.25 0.0216045 0.0104903 2024-02-05T08:20:40.234 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:40.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:40.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:40.382 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:40.410 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:40.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:40.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:40.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:40.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:40.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:40.678 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:40.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:40.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:40.825 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:40.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:40.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T08:20:40.926 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:40.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:40 smithi196 ceph-mon[33018]: 10.8 scrub ok 2024-02-05T08:20:40.927 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:40 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 22543/354696 objects degraded (6.356%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:40.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:40.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:40.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:41.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:40 smithi134 ceph-mon[30086]: 10.8 scrub ok 2024-02-05T08:20:41.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:40 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 22543/354696 objects degraded (6.356%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:41.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:40 smithi134 ceph-mon[39278]: 10.8 scrub ok 2024-02-05T08:20:41.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:40 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 22543/354696 objects degraded (6.356%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:41.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:41.134 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:41.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:41.141 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 81 16 122697 122681 94.643 44.4375 0.0143473 0.0105576 2024-02-05T08:20:41.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:41.289 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:41.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:41.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:41.362 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:20:41.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:41.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:41.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:41.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:41.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:41.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:41.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:41.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:41.646 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:20:41.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:41.716 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:41.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:41.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:41.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:41 smithi196 ceph-mon[33018]: pgmap v711: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 97 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s; 22543/361098 objects degraded (6.243%) 2024-02-05T08:20:41.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:41 smithi196 ceph-mon[33018]: 10.d scrub starts 2024-02-05T08:20:41.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:41 smithi196 ceph-mon[33018]: 10.6 scrub ok 2024-02-05T08:20:41.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:41 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2815373405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:20:41.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:41 smithi196 ceph-mon[33018]: 10.f scrub ok 2024-02-05T08:20:41.817 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:41.855 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:41.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:41.957 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:41.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:41.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:41.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:42.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:41 smithi134 ceph-mon[30086]: pgmap v711: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 97 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s; 22543/361098 objects degraded (6.243%) 2024-02-05T08:20:42.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:41 smithi134 ceph-mon[30086]: 10.d scrub starts 2024-02-05T08:20:42.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:41 smithi134 ceph-mon[30086]: 10.6 scrub ok 2024-02-05T08:20:42.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:41 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2815373405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:20:42.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:41 smithi134 ceph-mon[30086]: 10.f scrub ok 2024-02-05T08:20:42.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:41 smithi134 ceph-mon[39278]: pgmap v711: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 97 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s; 22543/361098 objects degraded (6.243%) 2024-02-05T08:20:42.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:41 smithi134 ceph-mon[39278]: 10.d scrub starts 2024-02-05T08:20:42.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:41 smithi134 ceph-mon[39278]: 10.6 scrub ok 2024-02-05T08:20:42.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:41 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2815373405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:20:42.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:41 smithi134 ceph-mon[39278]: 10.f scrub ok 2024-02-05T08:20:42.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:42.141 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 82 16 123673 123657 94.2327 61 0.0286659 0.0106064 2024-02-05T08:20:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:42.143 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:42.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:42.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:42.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:42.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:42.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:42.429 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:42.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:42.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:42.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:42.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:42.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:42.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:42.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:42.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:42.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:42.831 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:42.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:42.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:42.933 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:42.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:42.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:42.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:42.988 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:42 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:20:42.988 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:42 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2815373405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:20:42.989 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:42 smithi134 ceph-mon[30086]: osdmap e332: 8 total, 7 up, 8 in 2024-02-05T08:20:42.989 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:42 smithi134 ceph-mon[30086]: pgmap v713: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 97 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 833 op/s; 22543/361098 objects degraded (6.243%) 2024-02-05T08:20:42.989 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:42 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:20:42.989 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:42 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:20:42.989 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:42 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:20:42.989 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:42 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2815373405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:20:42.989 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:42 smithi134 ceph-mon[39278]: osdmap e332: 8 total, 7 up, 8 in 2024-02-05T08:20:42.989 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:42 smithi134 ceph-mon[39278]: pgmap v713: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 97 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 833 op/s; 22543/361098 objects degraded (6.243%) 2024-02-05T08:20:42.989 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:42 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:20:42.989 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:42 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:20:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:42 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:20:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:42 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2815373405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:20:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:42 smithi196 ceph-mon[33018]: osdmap e332: 8 total, 7 up, 8 in 2024-02-05T08:20:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:42 smithi196 ceph-mon[33018]: pgmap v713: 158 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 97 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 833 op/s; 22543/361098 objects degraded (6.243%) 2024-02-05T08:20:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:42 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:20:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:42 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:20:43.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:43.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:43.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:43.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:43.141 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 83 16 124624 124608 93.8135 59.4375 0.00804129 0.0106549 2024-02-05T08:20:43.220 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:43.252 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:43.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:43.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:43.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:43.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:43.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:43.486 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:43.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:43.523 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:43.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:43.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:43.660 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:43.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:43.762 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:43.794 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:43.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:43.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:43 smithi196 ceph-mon[33018]: osdmap e333: 8 total, 7 up, 8 in 2024-02-05T08:20:43.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:43 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:20:43.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:43 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:20:43.896 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:43.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:43.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:43.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:43.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:43 smithi134 ceph-mon[30086]: osdmap e333: 8 total, 7 up, 8 in 2024-02-05T08:20:43.983 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:43 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:20:43.983 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:43 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:20:43.984 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:43 smithi134 ceph-mon[39278]: osdmap e333: 8 total, 7 up, 8 in 2024-02-05T08:20:43.984 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:43 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:20:43.984 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:43 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:20:44.010 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:20:44.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:44.090 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:44.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:44.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:44.141 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 84 15 125538 125523 93.3774 57.1875 0.00596362 0.0107037 2024-02-05T08:20:44.191 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:44.221 INFO:teuthology.orchestra.run.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e323 pg_nums {1={20=1,301=11},2={65=3},3={66=32},4={68=32},5={70=32},6={72=32},9={239=16},10={309=16}} deleted_pools 305,9) 2024-02-05T08:20:44.221 INFO:teuthology.orchestra.run.smithi134.stdout:Importing pgid 5.c 2024-02-05T08:20:44.221 INFO:teuthology.orchestra.run.smithi134.stdout:write_pg epoch 298 info 5.c( empty local-lis/les=70/71 n=0 ec=70/70 lis/c=70/70 les/c/f=71/71/0 sis=70) 2024-02-05T08:20:44.221 INFO:teuthology.orchestra.run.smithi134.stdout:Import successful 2024-02-05T08:20:44.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:44.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:44.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:44.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:20:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:20:44.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:44.355 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:44.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:44.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:44.507 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:44.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:44.608 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:44.637 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:44.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:44.738 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:44.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:44.779 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:44.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:44.815 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:44 smithi134 ceph-mon[30086]: osdmap e334: 8 total, 7 up, 8 in 2024-02-05T08:20:44.816 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:44 smithi134 ceph-mon[30086]: pgmap v716: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 98 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.7 KiB/s rd, 62 MiB/s wr, 999 op/s; 23367/372666 objects degraded (6.270%) 2024-02-05T08:20:44.816 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:44 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:44.817 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:44 smithi134 ceph-mon[39278]: osdmap e334: 8 total, 7 up, 8 in 2024-02-05T08:20:44.817 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:44 smithi134 ceph-mon[39278]: pgmap v716: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 98 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.7 KiB/s rd, 62 MiB/s wr, 999 op/s; 23367/372666 objects degraded (6.270%) 2024-02-05T08:20:44.817 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:44 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:44.880 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:44.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:44.915 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:44.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:45.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:44 smithi196 ceph-mon[33018]: osdmap e334: 8 total, 7 up, 8 in 2024-02-05T08:20:45.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:44 smithi196 ceph-mon[33018]: pgmap v716: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 98 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.7 KiB/s rd, 62 MiB/s wr, 999 op/s; 23367/372666 objects degraded (6.270%) 2024-02-05T08:20:45.002 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:44 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:45.016 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:45.048 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:45.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:45.141 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 85 16 126512 126496 92.9943 60.8125 0.01118 0.0107474 2024-02-05T08:20:45.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:45.180 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:45.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:45.227 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:20:45.227 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:20:45.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:20:45.227 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:20:45.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.227 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:131942 8.2 8:40ced295:::benchmark_data_smithi134_115836_object15282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:20:45.227 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:58.589962+0000", 2024-02-05T08:20:45.227 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 466.63249379600001, 2024-02-05T08:20:45.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.154613257, 2024-02-05T08:20:45.228 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.228 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:20:45.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:20:45.228 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 131942 2024-02-05T08:20:45.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.228 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.228 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.228 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.589962+0000", 2024-02-05T08:20:45.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.589962+0000", 2024-02-05T08:20:45.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.589963+0000", 2024-02-05T08:20:45.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0550000000000001e-06 2024-02-05T08:20:45.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.589968+0000", 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5929999999999999e-06 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.589981+0000", 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2729e-05 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.589988+0000", 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6629999999999993e-06 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.590030+0000", 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2045000000000001e-05 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.590111+0000", 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0148000000000006e-05 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.590185+0000", 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4167999999999999e-05 2024-02-05T08:20:45.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.592612+0000", 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024273319999999999 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.592751+0000", 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013892099999999999 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.744474+0000", 2024-02-05T08:20:45.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15172269599999999 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.744535+0000", 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1130000000000006e-05 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:58.744576+0000", 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0776999999999999e-05 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:134925 8.2 8:40737ef3:::benchmark_data_smithi134_115836_object18265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:02.535531+0000", 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 462.68692549000002, 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.170932836, 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 134925 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535531+0000", 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535531+0000", 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535565+0000", 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3880000000000001e-05 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535571+0000", 2024-02-05T08:20:45.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6390000000000003e-06 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535587+0000", 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5664000000000001e-05 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535595+0000", 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3690000000000007e-06 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535692+0000", 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7102999999999995e-05 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.535829+0000", 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013684299999999999 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.536032+0000", 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00020306 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538954+0000", 2024-02-05T08:20:45.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002921313 2024-02-05T08:20:45.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.539112+0000", 2024-02-05T08:20:45.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015816499999999999 2024-02-05T08:20:45.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.706382+0000", 2024-02-05T08:20:45.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16727075999999999 2024-02-05T08:20:45.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.706429+0000", 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6851000000000003e-05 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.706463+0000", 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4189000000000001e-05 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:134931 8.b 8:d4cd9ded:::benchmark_data_smithi134_115836_object18271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:20:45.239 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:02.538007+0000", 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 462.68444917599999, 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16854570699999999, 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 134931 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538007+0000", 2024-02-05T08:20:45.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538007+0000", 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538008+0000", 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3880000000000001e-06 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538018+0000", 2024-02-05T08:20:45.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5710000000000004e-06 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538033+0000", 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4928e-05 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538043+0000", 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0382e-05 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538111+0000", 2024-02-05T08:20:45.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7556000000000002e-05 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538257+0000", 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000146649 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2,3", 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.538415+0000", 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015738199999999999 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.540847+0000", 2024-02-05T08:20:45.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024317710000000001 2024-02-05T08:20:45.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.541060+0000", 2024-02-05T08:20:45.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021372900000000001 2024-02-05T08:20:45.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.541562+0000", 2024-02-05T08:20:45.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000502058 2024-02-05T08:20:45.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.706359+0000", 2024-02-05T08:20:45.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.164796741 2024-02-05T08:20:45.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.706428+0000", 2024-02-05T08:20:45.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8547999999999995e-05 2024-02-05T08:20:45.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:02.706553+0000", 2024-02-05T08:20:45.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012500400000000001 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:162947 8.b 8:d02f707b:::benchmark_data_smithi134_115836_object46287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:37.217853+0000", 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 428.00460283199999, 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14594627299999999, 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:20:45.246 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 162947 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.217853+0000", 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.217853+0000", 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.217855+0000", 2024-02-05T08:20:45.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.455e-06 2024-02-05T08:20:45.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.217860+0000", 2024-02-05T08:20:45.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4500000000000003e-06 2024-02-05T08:20:45.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.217877+0000", 2024-02-05T08:20:45.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6498e-05 2024-02-05T08:20:45.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.217884+0000", 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5560000000000002e-06 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.221862+0000", 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039781060000000004 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.221976+0000", 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113201 2024-02-05T08:20:45.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2,3", 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.222107+0000", 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000131743 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.224873+0000", 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027655790000000002 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.226643+0000", 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001769858 2024-02-05T08:20:45.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.231832+0000", 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051890779999999997 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.363612+0000", 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13178054 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.363681+0000", 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8775999999999993e-05 2024-02-05T08:20:45.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:37.363800+0000", 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000118433 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170415 8.b 8:d9f796c8:::benchmark_data_smithi134_115836_object53755:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.829318+0000", 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 419.39313854400001, 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.91218017299999998, 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.252 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170415 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829318+0000", 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829318+0000", 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829319+0000", 2024-02-05T08:20:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6640000000000001e-06 2024-02-05T08:20:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829335+0000", 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5282999999999999e-05 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829350+0000", 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5820000000000001e-05 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829374+0000", 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4136999999999999e-05 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829424+0000", 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9091000000000001e-05 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.829432+0000", 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6959999999999994e-06 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.734517+0000", 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.90508505800000005 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.734603+0000", 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6011e-05 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.734676+0000", 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2261999999999994e-05 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741372+0000", 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066968749999999997 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741411+0000", 2024-02-05T08:20:45.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8059999999999998e-05 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741439+0000", 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8787000000000001e-05 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741474+0000", 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4329999999999998e-05 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741498+0000", 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4099000000000001e-05 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170468 8.b 8:da1853ce:::benchmark_data_smithi134_115836_object53808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.873146+0000", 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 419.34930967499997, 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.86851456500000002, 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:20:45.259 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170468 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873146+0000", 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873146+0000", 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873148+0000", 2024-02-05T08:20:45.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1459999999999999e-06 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873152+0000", 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5040000000000004e-06 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873167+0000", 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4642e-05 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873176+0000", 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3370000000000006e-06 2024-02-05T08:20:45.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873241+0000", 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4597999999999994e-05 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.873249+0000", 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3100000000000001e-06 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.737138+0000", 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.86388931499999999 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.737244+0000", 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105386 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.737361+0000", 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117689 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741555+0000", 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041938380000000001 2024-02-05T08:20:45.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741589+0000", 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3438000000000001e-05 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741615+0000", 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6696e-05 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741643+0000", 2024-02-05T08:20:45.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.747e-05 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741661+0000", 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8196e-05 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170493 8.b 8:d1db0339:::benchmark_data_smithi134_115836_object53833:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.894206+0000", 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 419.32825030999999, 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.85198245399999994, 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.265 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.266 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.266 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:20:45.266 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:20:45.266 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170493 2024-02-05T08:20:45.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.266 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894206+0000", 2024-02-05T08:20:45.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894206+0000", 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894207+0000", 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.201e-06 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894211+0000", 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0779999999999997e-06 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894236+0000", 2024-02-05T08:20:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5241000000000001e-05 2024-02-05T08:20:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894246+0000", 2024-02-05T08:20:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2159999999999995e-06 2024-02-05T08:20:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894315+0000", 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9863000000000005e-05 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.894333+0000", 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7793999999999999e-05 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.739546+0000", 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.84521263000000002 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.739615+0000", 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9233000000000006e-05 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.739685+0000", 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0118999999999997e-05 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741667+0000", 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019816230000000001 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746011+0000", 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043444979999999996 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746094+0000", 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2254000000000002e-05 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746153+0000", 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9231999999999998e-05 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746188+0000", 2024-02-05T08:20:45.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5472000000000002e-05 2024-02-05T08:20:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170502 8.b 8:d6d92551:::benchmark_data_smithi134_115836_object53842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:20:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.904511+0000", 2024-02-05T08:20:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 419.31794537399998, 2024-02-05T08:20:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.84494767500000001, 2024-02-05T08:20:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.272 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170502 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904511+0000", 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904511+0000", 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904512+0000", 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5499999999999996e-07 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904516+0000", 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9140000000000001e-06 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904527+0000", 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1263999999999999e-05 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904533+0000", 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0279999999999997e-06 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904616+0000", 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2929999999999994e-05 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.904645+0000", 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9439000000000001e-05 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741703+0000", 2024-02-05T08:20:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.83705783 2024-02-05T08:20:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741809+0000", 2024-02-05T08:20:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105708 2024-02-05T08:20:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:20:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.741919+0000", 2024-02-05T08:20:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010994599999999999 2024-02-05T08:20:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746024+0000", 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041055070000000004 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746054+0000", 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0000000000000001e-05 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.749365+0000", 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003311072 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.749425+0000", 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9728999999999997e-05 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.749458+0000", 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3352999999999997e-05 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170522 8.b 8:d777389e:::benchmark_data_smithi134_115836_object53862:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.943354+0000", 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 419.27910232900001, 2024-02-05T08:20:45.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80622890899999999, 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170522 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943354+0000", 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943354+0000", 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943355+0000", 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3519999999999999e-06 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943360+0000", 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.237e-06 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943373+0000", 2024-02-05T08:20:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2850999999999999e-05 2024-02-05T08:20:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943381+0000", 2024-02-05T08:20:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4170000000000003e-06 2024-02-05T08:20:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943435+0000", 2024-02-05T08:20:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4799999999999997e-05 2024-02-05T08:20:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:20:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.943444+0000", 2024-02-05T08:20:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3650000000000004e-06 2024-02-05T08:20:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.743871+0000", 2024-02-05T08:20:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80042767599999998 2024-02-05T08:20:45.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.743978+0000", 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010660800000000001 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.744084+0000", 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010608 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.746203+0000", 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002119147 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.748861+0000", 2024-02-05T08:20:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026574300000000001 2024-02-05T08:20:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.749495+0000", 2024-02-05T08:20:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00063419099999999999 2024-02-05T08:20:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.749537+0000", 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2132000000000003e-05 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.749583+0000", 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5623e-05 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35870 9.c 9:306adcbc:::benchmark_data_smithi134_144349_object35869:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.102920+0000", 2024-02-05T08:20:45.285 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 315.11953601599998, 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19672384400000001, 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35870 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.102920+0000", 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.102920+0000", 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.102922+0000", 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6619999999999999e-06 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.103049+0000", 2024-02-05T08:20:45.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012733699999999999 2024-02-05T08:20:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.291776+0000", 2024-02-05T08:20:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18872659 2024-02-05T08:20:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.291787+0000", 2024-02-05T08:20:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1372999999999999e-05 2024-02-05T08:20:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.294056+0000", 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002268506 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.294198+0000", 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000142111 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.294379+0000", 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018153599999999999 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299279+0000", 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048995039999999998 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299293+0000", 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.467e-05 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299410+0000", 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011679 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299471+0000", 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0837e-05 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299644+0000", 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017292800000000001 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:35869 9.4 e250/239 9:2c743692:::benchmark_data_smithi134_144349_object35868:head v 250'2294, mlcod=249'2292)", 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.103107+0000", 2024-02-05T08:20:45.291 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 315.11934920900001, 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19259843099999999, 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.103107+0000", 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.103107+0000", 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.103107+0000", 2024-02-05T08:20:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3099999999999999e-07 2024-02-05T08:20:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.293 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.103178+0000", 2024-02-05T08:20:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0746000000000003e-05 2024-02-05T08:20:45.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.291775+0000", 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.188597234 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.291787+0000", 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1593999999999999e-05 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.292052+0000", 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026488199999999998 2024-02-05T08:20:45.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.292074+0000", 2024-02-05T08:20:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1967000000000002e-05 2024-02-05T08:20:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.295634+0000", 2024-02-05T08:20:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00356011 2024-02-05T08:20:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.295705+0000", 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1667000000000003e-05 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:35886 9.1 e250/239 9:8a1f8069:::benchmark_data_smithi134_144349_object35885:head v 250'2292, mlcod=249'2290)", 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.114067+0000", 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 315.10838886099998, 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.180242398, 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.114067+0000", 2024-02-05T08:20:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.114067+0000", 2024-02-05T08:20:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.114068+0000", 2024-02-05T08:20:45.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3799999999999999e-07 2024-02-05T08:20:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.114089+0000", 2024-02-05T08:20:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1841e-05 2024-02-05T08:20:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.291766+0000", 2024-02-05T08:20:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17767633699999999 2024-02-05T08:20:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.291783+0000", 2024-02-05T08:20:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7042e-05 2024-02-05T08:20:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.291973+0000", 2024-02-05T08:20:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019060099999999999 2024-02-05T08:20:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.291999+0000", 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5687999999999999e-05 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.294259+0000", 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022597310000000001 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.294310+0000", 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.092e-05 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.301 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:175455 9.c 9:36803ded:::benchmark_data_smithi134_144349_object43715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:17:23.988089+0000", 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 201.23436735499999, 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15127130699999999, 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 175455 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.302 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:23.988089+0000", 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:23.988089+0000", 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:23.988090+0000", 2024-02-05T08:20:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2130000000000001e-06 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:23.988095+0000", 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8860000000000003e-06 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:23.988105+0000", 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8910000000000007e-06 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:23.988112+0000", 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9569999999999996e-06 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:24.005686+0000", 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017574207000000001 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:24.005821+0000", 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000134936 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:24.005969+0000", 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014778 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:24.010247+0000", 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042782619999999997 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:24.010287+0000", 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9889999999999999e-05 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:24.139240+0000", 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12895335799999999 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:24.139298+0000", 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7358000000000002e-05 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:24.139360+0000", 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2569000000000004e-05 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.307 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:177256 9.c 9:35d5b1eb:::benchmark_data_smithi134_144349_object45516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T08:20:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:17:26.230073+0000", 2024-02-05T08:20:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 198.99238324300001, 2024-02-05T08:20:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17816562699999999, 2024-02-05T08:20:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:20:45.308 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 177256 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.230073+0000", 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.230073+0000", 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.230074+0000", 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4920000000000001e-06 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.230080+0000", 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2939999999999996e-06 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.230094+0000", 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4324999999999999e-05 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.230102+0000", 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7489999999999998e-06 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.256573+0000", 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.026471306999999999 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.256659+0000", 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6046999999999998e-05 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:20:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.256738+0000", 2024-02-05T08:20:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8936000000000005e-05 2024-02-05T08:20:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.258569+0000", 2024-02-05T08:20:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001831346 2024-02-05T08:20:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.258613+0000", 2024-02-05T08:20:45.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3149999999999999e-05 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.408115+0000", 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.149502831 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.408176+0000", 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1128000000000002e-05 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.408239+0000", 2024-02-05T08:20:45.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2021999999999997e-05 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:177337 9.c 9:37848b94:::benchmark_data_smithi134_144349_object45597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:17:26.352756+0000", 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 198.86970020999999, 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16719146400000001, 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:20:45.314 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 177337 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.352756+0000", 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.352756+0000", 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.352757+0000", 2024-02-05T08:20:45.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4890000000000001e-06 2024-02-05T08:20:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.352763+0000", 2024-02-05T08:20:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8529999999999997e-06 2024-02-05T08:20:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.352779+0000", 2024-02-05T08:20:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6237999999999999e-05 2024-02-05T08:20:45.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.352789+0000", 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2480000000000002e-06 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.416777+0000", 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.063988083000000001 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.416905+0000", 2024-02-05T08:20:45.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012856900000000001 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.417034+0000", 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012837999999999999 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.419766+0000", 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002732511 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.441286+0000", 2024-02-05T08:20:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021520108999999999 2024-02-05T08:20:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.519830+0000", 2024-02-05T08:20:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.078543158000000002 2024-02-05T08:20:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.519893+0000", 2024-02-05T08:20:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3120000000000006e-05 2024-02-05T08:20:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:17:26.519947+0000", 2024-02-05T08:20:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4706000000000001e-05 2024-02-05T08:20:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15714.0:120095 10.e 10:728d4736:::benchmark_data_smithi134_174861_object120094:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e331)", 2024-02-05T08:20:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:20:37.947352+0000", 2024-02-05T08:20:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 7.2751039909999999, 2024-02-05T08:20:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14615906300000001, 2024-02-05T08:20:45.320 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15714", 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 120095 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:37.947352+0000", 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:37.947352+0000", 2024-02-05T08:20:45.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:37.947354+0000", 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.595e-06 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:37.947385+0000", 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1236999999999999e-05 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:37.947400+0000", 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5158e-05 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:37.947408+0000", 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3329999999999997e-06 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:38.044173+0000", 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.096764731000000007 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:38.044316+0000", 2024-02-05T08:20:45.323 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014317 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,6", 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:38.044681+0000", 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036496500000000003 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:38.052289+0000", 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0076074259999999996 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:38.054621+0000", 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023323939999999998 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:38.093391+0000", 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.038770027999999998 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:38.093453+0000", 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1624000000000005e-05 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:38.093511+0000", 2024-02-05T08:20:45.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8402000000000001e-05 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15714.0:122289 10.e 10:784abf42:::benchmark_data_smithi134_174861_object122288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e331)", 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:20:40.528974+0000", 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.6934824380000002, 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.205701631, 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15714", 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 122289 2024-02-05T08:20:45.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.528974+0000", 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.528974+0000", 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.528975+0000", 2024-02-05T08:20:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.795e-06 2024-02-05T08:20:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.529006+0000", 2024-02-05T08:20:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0315e-05 2024-02-05T08:20:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.529023+0000", 2024-02-05T08:20:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7421000000000001e-05 2024-02-05T08:20:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.529031+0000", 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7810000000000005e-06 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.561872+0000", 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.032841045999999999 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.562010+0000", 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013806800000000001 2024-02-05T08:20:45.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,6", 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.562165+0000", 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000154804 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.566143+0000", 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039783199999999996 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.567429+0000", 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001286143 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.734572+0000", 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16714246099999999 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.734631+0000", 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9225999999999998e-05 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.734675+0000", 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4251e-05 2024-02-05T08:20:45.331 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15714.0:122299 10.e 10:7f2efcb0:::benchmark_data_smithi134_174861_object122298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e331)", 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:20:40.545046+0000", 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.6774103, 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.189759598, 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15714", 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 122299 2024-02-05T08:20:45.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.545046+0000", 2024-02-05T08:20:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.545046+0000", 2024-02-05T08:20:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.545048+0000", 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7370000000000001e-06 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.545087+0000", 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9756999999999999e-05 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.545102+0000", 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4970999999999999e-05 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.545112+0000", 2024-02-05T08:20:45.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3479999999999993e-06 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.564279+0000", 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019167232999999999 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.564436+0000", 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000156954 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,6", 2024-02-05T08:20:45.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.564613+0000", 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017715100000000001 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.567467+0000", 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0028540340000000001 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.567480+0000", 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2670000000000001e-05 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.337 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.734712+0000", 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.167232719 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.734760+0000", 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7318999999999999e-05 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.734805+0000", 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5704999999999997e-05 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.338 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15714.0:122301 10.e 10:716dcd7c:::benchmark_data_smithi134_174861_object122300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e331)", 2024-02-05T08:20:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:20:40.545127+0000", 2024-02-05T08:20:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 4.6773292099999999, 2024-02-05T08:20:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.189807536, 2024-02-05T08:20:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15714", 2024-02-05T08:20:45.339 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 122301 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.545127+0000", 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.545127+0000", 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.340 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.545127+0000", 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6599999999999996e-07 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.545162+0000", 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4576999999999998e-05 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.545170+0000", 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5279999999999998e-06 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.341 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.545176+0000", 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1970000000000001e-06 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.566193+0000", 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021016759999999999 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.566276+0000", 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3812000000000003e-05 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.342 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,6", 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.566383+0000", 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106261 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.567593+0000", 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0012101200000000001 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.568080+0000", 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00048744199999999999 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.343 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.734844+0000", 2024-02-05T08:20:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.166764154 2024-02-05T08:20:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.734890+0000", 2024-02-05T08:20:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5362999999999999e-05 2024-02-05T08:20:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:40.734934+0000", 2024-02-05T08:20:45.344 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4756000000000003e-05 2024-02-05T08:20:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15714.0:124252 10.e 10:715e46b0:::benchmark_data_smithi134_174861_object124251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-02-05T08:20:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:20:42.716269+0000", 2024-02-05T08:20:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 2.5061868459999999, 2024-02-05T08:20:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15025106399999999, 2024-02-05T08:20:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:20:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:20:45.345 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15714", 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 124252 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.716269+0000", 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.716269+0000", 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.346 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:20:45.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.716270+0000", 2024-02-05T08:20:45.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9800000000000003e-07 2024-02-05T08:20:45.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:20:45.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.716321+0000", 2024-02-05T08:20:45.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0578000000000003e-05 2024-02-05T08:20:45.347 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.347 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.347 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:20:45.347 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.716346+0000", 2024-02-05T08:20:45.347 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5114e-05 2024-02-05T08:20:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:20:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.716352+0000", 2024-02-05T08:20:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6980000000000001e-06 2024-02-05T08:20:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:20:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.743924+0000", 2024-02-05T08:20:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.027571875999999999 2024-02-05T08:20:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.348 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.744170+0000", 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00024554399999999999 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,6", 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.744318+0000", 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014788799999999999 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.751061+0000", 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067431069999999999 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.349 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:20:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.752048+0000", 2024-02-05T08:20:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00098732399999999993 2024-02-05T08:20:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:20:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.866447+0000", 2024-02-05T08:20:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11439935800000001 2024-02-05T08:20:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:20:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.866482+0000", 2024-02-05T08:20:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4623999999999999e-05 2024-02-05T08:20:45.350 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:20:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:20:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:20:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:20:42.866520+0000", 2024-02-05T08:20:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8254999999999999e-05 2024-02-05T08:20:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:20:45.351 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:20:45.351 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:20:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:45.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:45.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:45.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:45.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/exp.5.c.1 2024-02-05T08:20:45.495 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:45.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:45.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:45.632 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:45.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:45.734 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:45.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:45.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:45.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:45.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:45.916 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:45.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:46.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:46.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:45 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 23367/372666 objects degraded (6.270%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:46.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:45 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 23367/372666 objects degraded (6.270%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:46.051 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:46.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:46.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:46.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:45 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 23367/372666 objects degraded (6.270%), 25 pgs degraded (PG_DEGRADED) 2024-02-05T08:20:46.141 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 86 16 127495 127479 92.6273 61.4375 0.0151569 0.0107907 2024-02-05T08:20:46.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:46.190 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:46.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:46.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:46.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:46.329 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:46.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:46.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:46.430 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:46.462 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:46.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:46.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T08:20:46.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:46.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:46.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:46.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:46.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:20:46.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:46.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:46.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:46.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:46.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:46 smithi196 ceph-mon[33018]: pgmap v717: 158 pgs: 1 active+clean+scrubbing+deep, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 732 op/s; 23367/374271 objects degraded (6.243%) 2024-02-05T08:20:46.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:46.901 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:46.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:46.956 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:20:46.987 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:46 smithi134 ceph-mon[30086]: pgmap v717: 158 pgs: 1 active+clean+scrubbing+deep, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 732 op/s; 23367/374271 objects degraded (6.243%) 2024-02-05T08:20:46.988 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:46 smithi134 ceph-mon[39278]: pgmap v717: 158 pgs: 1 active+clean+scrubbing+deep, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.6 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 732 op/s; 23367/374271 objects degraded (6.243%) 2024-02-05T08:20:47.002 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:47.036 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:47.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:47.138 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:47.141 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 87 16 128419 128403 92.2261 57.75 0.00719483 0.010838 2024-02-05T08:20:47.182 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:47.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:47.284 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:47.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:47.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:47.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:47.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:47.476 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:47.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:47.483 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:20:47.492 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:20:47.521 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:20:47.553 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:20:47.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:47.601 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:20:47.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:47.618 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:47.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:47.640 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:20:47.677 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:20:47.709 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:20:47.709 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:20:47.709 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:20:47.709 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:20:47.710 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:20:47.710 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:20:47.710 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:20:47.710 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:20:47.710 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:20:47.718 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:47.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:47.757 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:47.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:47.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:47 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:20:47.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:47 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:20:47.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:47 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:20:47.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:47 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:20:47.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:47 smithi134 ceph-mon[30086]: osdmap e335: 8 total, 7 up, 8 in 2024-02-05T08:20:47.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:47 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1094421445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:47.778 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:20:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:20:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:20:47.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:47 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:20:47.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:47 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:20:47.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:47 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:20:47.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:47 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:20:47.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:47 smithi134 ceph-mon[39278]: osdmap e335: 8 total, 7 up, 8 in 2024-02-05T08:20:47.780 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:47 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1094421445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:47 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:20:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:47 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:20:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:47 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:20:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:47 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:20:47.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:47 smithi196 ceph-mon[33018]: osdmap e335: 8 total, 7 up, 8 in 2024-02-05T08:20:47.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:47 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1094421445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:47.858 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:47.891 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:47.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:47.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:48.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:48.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:48.130 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:48.142 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 88 16 129468 129452 91.9231 65.5625 0.00801315 0.0108739 2024-02-05T08:20:48.167 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:48.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:48.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:48.269 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:48.299 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:48.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:48.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:48.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:48.434 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:48.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:48.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:48.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:48.557 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:20:48.558 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:20:48.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:48.574 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:48.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:48.675 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:48.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:48.725 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:48.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:48.760 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:48 smithi134 ceph-mon[30086]: pgmap v719: 158 pgs: 1 active+clean+scrubbing+deep, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 878 op/s; 23654/376902 objects degraded (6.276%) 2024-02-05T08:20:48.760 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:48 smithi134 ceph-mon[39278]: pgmap v719: 158 pgs: 1 active+clean+scrubbing+deep, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 878 op/s; 23654/376902 objects degraded (6.276%) 2024-02-05T08:20:48.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:48.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:48.867 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:48.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:48.914 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:48 smithi196 ceph-mon[33018]: pgmap v719: 158 pgs: 1 active+clean+scrubbing+deep, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 55 MiB/s wr, 878 op/s; 23654/376902 objects degraded (6.276%) 2024-02-05T08:20:48.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:48.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:48.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:48.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:49.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:49.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:49.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:49.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:49.142 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 89 16 130529 130513 91.6352 66.3125 0.00322076 0.0109065 2024-02-05T08:20:49.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:49.266 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:49.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:49.367 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:49.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:49.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:49.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:49.493 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:49.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:49.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:49.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:49.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:49.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:49.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:49.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:49.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:49.808 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:49.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:49.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:49.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:49.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:49.945 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:49.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:50.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:50.078 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:50.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:50.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:50.142 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 90 11 131459 131448 91.2663 58.4375 0.0707578 0.0109495 2024-02-05T08:20:50.149 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:49 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 23654/376902 objects degraded (6.276%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:20:50.163 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:49 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 23654/376902 objects degraded (6.276%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:20:50.163 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:49 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 23654/376902 objects degraded (6.276%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:20:50.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:50.205 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:50.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:50.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:50.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:50.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:50.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:50.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:50.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:50.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:50.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:50.505 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-02-05T08:20:50.506 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:20:50.506 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_3 2024-02-05T08:20:50.506 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_3 by 10 min 8 2024-02-05T08:20:50.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:20:50.559 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:20:50.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:50.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:50.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:50.685 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:50.734 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:50.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:50.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:50.790 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:20:50.831 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:20:50.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:50.860 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:50.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:50.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:50.904 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:20:50.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:50.985 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:50.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:50.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:51.087 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:51.111 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:51.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:51.142 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Total time run: 90.0363 2024-02-05T08:20:51.142 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Total writes made: 131459 2024-02-05T08:20:51.142 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Write size: 65536 2024-02-05T08:20:51.142 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Object size: 65536 2024-02-05T08:20:51.142 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Bandwidth (MB/sec): 91.2541 2024-02-05T08:20:51.142 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev Bandwidth: 36.8586 2024-02-05T08:20:51.143 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max bandwidth (MB/sec): 227.625 2024-02-05T08:20:51.143 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min bandwidth (MB/sec): 20.875 2024-02-05T08:20:51.143 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Average IOPS: 1460 2024-02-05T08:20:51.143 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev IOPS: 589.737 2024-02-05T08:20:51.143 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max IOPS: 3642 2024-02-05T08:20:51.143 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min IOPS: 334 2024-02-05T08:20:51.143 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Average Latency(s): 0.0109533 2024-02-05T08:20:51.143 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev Latency(s): 0.0142034 2024-02-05T08:20:51.143 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max latency(s): 1.05979 2024-02-05T08:20:51.143 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min latency(s): 0.00127789 2024-02-05T08:20:51.146 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T08:20:51.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:51.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:51.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:51.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:51.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:50 smithi134 ceph-mon[30086]: pgmap v720: 158 pgs: 1 active+clean+scrubbing+deep, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 73 MiB/s wr, 1.17k op/s; 24236/388794 objects degraded (6.234%) 2024-02-05T08:20:51.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:50 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3933423329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:20:51.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:50 smithi134 ceph-mon[39278]: pgmap v720: 158 pgs: 1 active+clean+scrubbing+deep, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 73 MiB/s wr, 1.17k op/s; 24236/388794 objects degraded (6.234%) 2024-02-05T08:20:51.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:50 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3933423329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:20:51.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:50 smithi196 ceph-mon[33018]: pgmap v720: 158 pgs: 1 active+clean+scrubbing+deep, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 73 MiB/s wr, 1.17k op/s; 24236/388794 objects degraded (6.234%) 2024-02-05T08:20:51.289 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:50 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3933423329' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:20:51.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:51.383 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:51.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:51.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:51.523 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:51.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:51.624 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:51.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:51.662 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:51.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:51.763 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:51.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:51.802 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:51.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:51.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:51.936 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:51.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:52.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:52.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:52.080 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:52.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:52.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:52.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:52.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:52.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:52.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:51 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:20:52.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:51 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3933423329' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:20:52.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:51 smithi134 ceph-mon[30086]: osdmap e336: 8 total, 7 up, 8 in 2024-02-05T08:20:52.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:51 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:20:52.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:51 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3933423329' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:20:52.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:51 smithi134 ceph-mon[39278]: osdmap e336: 8 total, 7 up, 8 in 2024-02-05T08:20:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:51 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:20:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:51 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3933423329' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:20:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:51 smithi196 ceph-mon[33018]: osdmap e336: 8 total, 7 up, 8 in 2024-02-05T08:20:52.318 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:52.328 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:20:52.328 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:20:52.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:52.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:52.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:52.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:52.492 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:52.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:52.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:52.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:52.625 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:52.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:52.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:52.764 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:52.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:52.866 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:52.907 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:52.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:53.008 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:53.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:52 smithi134 ceph-mon[30086]: pgmap v722: 158 pgs: 1 active+clean+scrubbing+deep, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 672 op/s; 24236/388794 objects degraded (6.234%) 2024-02-05T08:20:53.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:52 smithi134 ceph-mon[39278]: pgmap v722: 158 pgs: 1 active+clean+scrubbing+deep, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 672 op/s; 24236/388794 objects degraded (6.234%) 2024-02-05T08:20:53.035 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:53.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:53.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:53.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:53.173 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:53.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:53.275 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:53.301 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:53.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:52 smithi196 ceph-mon[33018]: pgmap v722: 158 pgs: 1 active+clean+scrubbing+deep, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 672 op/s; 24236/388794 objects degraded (6.234%) 2024-02-05T08:20:53.403 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:53.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:53.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:53.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:53.541 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:53.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:53.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:53.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:53.715 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:53.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:53.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:53.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:53.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:53.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:53.984 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:53.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:53.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:54.087 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:54.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:54.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:54.222 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:54.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:54.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:54.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:54.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:20:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:20:54.357 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:54.395 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:54.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:54.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:54.497 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:54.532 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:54.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:54.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:20:54.633 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:54.667 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:54.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:54.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:54.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:54.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:20:54.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:54.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:20:54.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:54.809 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:54.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:54.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:20:54.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:20:54.910 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:54.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:54.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:54.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:55.043 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:55.077 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:55.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:55.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:55.179 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:55.215 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:55.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:55.317 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:55.353 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:55.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:55.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:55.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:55.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:55.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:55.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:55.590 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:55.624 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:55.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:55.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:55.670 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:55 smithi134 ceph-mon[30086]: pgmap v723: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 53 MiB/s wr, 1.07k op/s; 24333/389028 objects degraded (6.255%) 2024-02-05T08:20:55.670 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:55 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:55.670 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:55 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 24333/389028 objects degraded (6.255%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:20:55.671 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:55 smithi134 ceph-mon[39278]: pgmap v723: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 53 MiB/s wr, 1.07k op/s; 24333/389028 objects degraded (6.255%) 2024-02-05T08:20:55.671 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:55 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:55.671 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:55 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 24333/389028 objects degraded (6.255%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:20:55.726 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:55.766 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:55.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:55.797 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:20:55.803 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:20:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:55 smithi196 ceph-mon[33018]: pgmap v723: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.4 KiB/s rd, 53 MiB/s wr, 1.07k op/s; 24333/389028 objects degraded (6.255%) 2024-02-05T08:20:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:55 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:20:55.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:55 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 24333/389028 objects degraded (6.255%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:20:55.867 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:55.904 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:55.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:56.005 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:56.043 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:56.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:56.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:56.145 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:56.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:56.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:56.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:56.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:56.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:56.433 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:56.470 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:56.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:56.572 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:56.608 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:56 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1335776027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:56.608 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:56 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1335776027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:56.609 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:56.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:56.648 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":723,"stamp":"2024-02-05T08:20:53.812615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8486054682,"num_objects":129676,"num_object_clones":0,"num_object_copies":389028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24333,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129676,"num_whiteouts":0,"num_read":2117,"num_read_kb":1877,"num_write":133889,"num_write_kb":8404298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":221172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86726,"ondisk_log_size":86726,"up":416,"acting":416,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":28407604,"kb_used_data":23999840,"kb_used_omap":12,"kb_used_meta":4407283,"kb_avail":721553612,"statfs":{"total":767960285184,"available":738870898688,"internally_reserved":0,"allocated":24575836160,"data_stored":24573271771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":4513058355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":322306072,"num_objects":4919,"num_object_clones":0,"num_object_copies":14757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":966,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4919,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":8591,"num_write_kb":432257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003925"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":601,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:50.903840+0000","last_change":"2024-02-05T08:19:44.912725+0000","last_active":"2024-02-05T08:20:50.903840+0000","last_peered":"2024-02-05T08:20:50.903840+0000","last_clean":"2024-02-05T08:19:41.829565+0000","last_became_active":"2024-02-05T08:19:44.912725+0000","last_became_peered":"2024-02-05T08:19:44.912725+0000","last_unstale":"2024-02-05T08:20:50.903840+0000","last_undegraded":"2024-02-05T08:20:50.903840+0000","last_fullsized":"2024-02-05T08:19:44.874568+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:36.296981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:36.296981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:50.963586+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":32,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.025446+0000","last_change":"2024-02-05T08:19:44.903097+0000","last_active":"2024-02-05T08:20:51.025446+0000","last_peered":"2024-02-05T08:20:51.025446+0000","last_clean":"2024-02-05T08:04:44.536163+0000","last_became_active":"2024-02-05T08:19:44.903097+0000","last_became_peered":"2024-02-05T08:19:44.903097+0000","last_unstale":"2024-02-05T08:20:51.025446+0000","last_undegraded":"2024-02-05T08:20:51.025446+0000","last_fullsized":"2024-02-05T08:19:44.873443+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.301502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.301502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":600,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.720359+0000","last_change":"2024-02-05T08:19:25.569614+0000","last_active":"2024-02-05T08:20:46.720359+0000","last_peered":"2024-02-05T08:20:46.720359+0000","last_clean":"2024-02-05T08:20:46.720359+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:20:46.720359+0000","last_undegraded":"2024-02-05T08:20:46.720359+0000","last_fullsized":"2024-02-05T08:20:46.720359+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:25.569504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:25.569504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:50.479367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"236'19","reported_seq":662,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:50.903438+0000","last_change":"2024-02-05T08:19:44.920275+0000","last_active":"2024-02-05T08:20:50.903438+0000","last_peered":"2024-02-05T08:20:50.903438+0000","last_clean":"2024-02-05T08:19:41.830039+0000","last_became_active":"2024-02-05T08:19:44.920275+0000","last_became_peered":"2024-02-05T08:19:44.920275+0000","last_unstale":"2024-02-05T08:20:50.903438+0000","last_undegraded":"2024-02-05T08:19:44.874580+0000","last_fullsized":"2024-02-05T08:19:44.874470+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:01.602887+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:18:01.602887+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":101,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084849+0000","last_change":"2024-02-05T08:20:17.809613+0000","last_active":"2024-02-05T08:20:51.084849+0000","last_peered":"2024-02-05T08:20:51.084849+0000","last_clean":"2024-02-05T08:20:51.084849+0000","last_became_active":"2024-02-05T08:18:34.633083+0000","last_became_peered":"2024-02-05T08:18:34.633083+0000","last_unstale":"2024-02-05T08:20:51.084849+0000","last_undegraded":"2024-02-05T08:20:51.084849+0000","last_fullsized":"2024-02-05T08:20:51.084849+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:17.809200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:20:17.809200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:34.584011+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":609,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:50.903626+0000","last_change":"2024-02-05T08:19:44.919417+0000","last_active":"2024-02-05T08:20:50.903626+0000","last_peered":"2024-02-05T08:20:50.903626+0000","last_clean":"2024-02-05T08:19:41.831283+0000","last_became_active":"2024-02-05T08:19:44.919417+0000","last_became_peered":"2024-02-05T08:19:44.919417+0000","last_unstale":"2024-02-05T08:20:50.903626+0000","last_undegraded":"2024-02-05T08:20:50.903626+0000","last_fullsized":"2024-02-05T08:19:44.874665+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:39.536022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:18:39.536022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":105,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.085047+0000","last_change":"2024-02-05T08:19:44.927236+0000","last_active":"2024-02-05T08:20:51.085047+0000","last_peered":"2024-02-05T08:20:51.085047+0000","last_clean":"2024-02-05T08:19:41.830712+0000","last_became_active":"2024-02-05T08:19:44.927236+0000","last_became_peered":"2024-02-05T08:19:44.927236+0000","last_unstale":"2024-02-05T08:20:51.085047+0000","last_undegraded":"2024-02-05T08:20:51.085047+0000","last_fullsized":"2024-02-05T08:19:44.887381+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:47.183986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:18:47.183986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"236'34","reported_seq":125,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084526+0000","last_change":"2024-02-05T08:20:29.672533+0000","last_active":"2024-02-05T08:20:51.084526+0000","last_peered":"2024-02-05T08:20:51.084526+0000","last_clean":"2024-02-05T08:20:51.084526+0000","last_became_active":"2024-02-05T08:18:34.649433+0000","last_became_peered":"2024-02-05T08:18:34.649433+0000","last_unstale":"2024-02-05T08:20:51.084526+0000","last_undegraded":"2024-02-05T08:20:51.084526+0000","last_fullsized":"2024-02-05T08:20:51.084526+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:20:29.672381+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:20:29.672381+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:42.459911+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":590,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-02-05T08:20:46.719813+0000","last_change":"2024-02-05T08:19:44.902206+0000","last_active":"2024-02-05T08:20:46.719813+0000","last_peered":"2024-02-05T08:20:46.719813+0000","last_clean":"2024-02-05T08:19:41.834390+0000","last_became_active":"2024-02-05T08:19:44.902206+0000","last_became_peered":"2024-02-05T08:19:44.902206+0000","last_unstale":"2024-02-05T08:20:46.719813+0000","last_undegraded":"2024-02-05T08:20:46.719813+0000","last_fullsized":"2024-02-05T08:19:44.885918+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:21.661930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:21.661930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:50.479352+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":602,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.025752+0000","last_change":"2024-02-05T08:19:21.500610+0000","last_active":"2024-02-05T08:20:51.025752+0000","last_peered":"2024-02-05T08:20:51.025752+0000","last_clean":"2024-02-05T08:20:51.025752+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:20:51.025752+0000","last_undegraded":"2024-02-05T08:20:51.025752+0000","last_fullsized":"2024-02-05T08:20:51.025752+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:21.500492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:19:21.500492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":630,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.026148+0000","last_change":"2024-02-05T08:19:32.421512+0000","last_active":"2024-02-05T08:20:51.026148+0000","last_peered":"2024-02-05T08:20:51.026148+0000","last_clean":"2024-02-05T08:20:51.026148+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:20:51.026148+0000","last_undegraded":"2024-02-05T08:20:51.026148+0000","last_fullsized":"2024-02-05T08:20:51.026148+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_clean_scrub_stamp":"2024-02-05T08:19:32.421388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"263'19","reported_seq":673,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.904161+0000","last_change":"2024-02-05T08:20:19.142437+0000","last_active":"2024-02-05T08:20:50.904161+0000","last_peered":"2024-02-05T08:20:50.904161+0000","last_clean":"2024-02-05T08:20:50.904161+0000","last_became_active":"2024-02-05T08:18:34.636165+0000","last_became_peered":"2024-02-05T08:18:34.636165+0000","last_unstale":"2024-02-05T08:20:50.904161+0000","last_undegraded":"2024-02-05T08:20:50.904161+0000","last_fullsized":"2024-02-05T08:20:50.904161+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:20:19.142175+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.142175+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:27.368545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":605,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.025524+0000","last_change":"2024-02-05T08:19:44.901170+0000","last_active":"2024-02-05T08:20:51.025524+0000","last_peered":"2024-02-05T08:20:51.025524+0000","last_clean":"2024-02-05T08:19:41.833929+0000","last_became_active":"2024-02-05T08:19:44.901170+0000","last_became_peered":"2024-02-05T08:19:44.901170+0000","last_unstale":"2024-02-05T08:20:51.025524+0000","last_undegraded":"2024-02-05T08:20:51.025524+0000","last_fullsized":"2024-02-05T08:19:44.873575+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:32.611856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:32.611856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":606,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.084315+0000","last_change":"2024-02-05T08:19:44.921416+0000","last_active":"2024-02-05T08:20:51.084315+0000","last_peered":"2024-02-05T08:20:51.084315+0000","last_clean":"2024-02-05T08:19:41.834434+0000","last_became_active":"2024-02-05T08:19:44.921416+0000","last_became_peered":"2024-02-05T08:19:44.921416+0000","last_unstale":"2024-02-05T08:20:51.084315+0000","last_undegraded":"2024-02-05T08:20:51.084315+0000","last_fullsized":"2024-02-05T08:19:44.875209+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:40.260248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:40.260248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:58.951191+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":613,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.025897+0000","last_change":"2024-02-05T08:19:44.916712+0000","last_active":"2024-02-05T08:20:51.025897+0000","last_peered":"2024-02-05T08:20:51.025897+0000","last_clean":"2024-02-05T08:19:41.831063+0000","last_became_active":"2024-02-05T08:19:44.916712+0000","last_became_peered":"2024-02-05T08:19:44.916712+0000","last_unstale":"2024-02-05T08:20:51.025897+0000","last_undegraded":"2024-02-05T08:20:51.025897+0000","last_fullsized":"2024-02-05T08:19:44.898786+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:14.458198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.458198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":638,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:50.904264+0000","last_change":"2024-02-05T08:19:44.913761+0000","last_active":"2024-02-05T08:20:50.904264+0000","last_peered":"2024-02-05T08:20:50.904264+0000","last_clean":"2024-02-05T08:19:42.900179+0000","last_became_active":"2024-02-05T08:19:44.913761+0000","last_became_peered":"2024-02-05T08:19:44.913761+0000","last_unstale":"2024-02-05T08:20:50.904264+0000","last_undegraded":"2024-02-05T08:19:44.909323+0000","last_fullsized":"2024-02-05T08:19:44.909205+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-02-05T08:18:05.582303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:05.582303+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":611,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.912126+0000","last_change":"2024-02-05T08:20:16.267258+0000","last_active":"2024-02-05T08:20:50.912126+0000","last_peered":"2024-02-05T08:20:50.912126+0000","last_clean":"2024-02-05T08:20:50.912126+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:20:50.912126+0000","last_undegraded":"2024-02-05T08:20:50.912126+0000","last_fullsized":"2024-02-05T08:20:50.912126+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:16.266908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:20:16.266908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:24.672751+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":32,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:50.904400+0000","last_change":"2024-02-05T08:19:44.920889+0000","last_active":"2024-02-05T08:20:50.904400+0000","last_peered":"2024-02-05T08:20:50.904400+0000","last_clean":"2024-02-05T08:04:44.536163+0000","last_became_active":"2024-02-05T08:19:44.920889+0000","last_became_peered":"2024-02-05T08:19:44.920889+0000","last_unstale":"2024-02-05T08:20:50.904400+0000","last_undegraded":"2024-02-05T08:20:50.904400+0000","last_fullsized":"2024-02-05T08:19:44.913274+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_clean_scrub_stamp":"2024-02-05T08:19:36.737342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":599,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.719819+0000","last_change":"2024-02-05T08:20:31.566305+0000","last_active":"2024-02-05T08:20:46.719819+0000","last_peered":"2024-02-05T08:20:46.719819+0000","last_clean":"2024-02-05T08:20:46.719819+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:20:46.719819+0000","last_undegraded":"2024-02-05T08:20:46.719819+0000","last_fullsized":"2024-02-05T08:20:46.719819+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:31.566207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:31.566207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:41.126178+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"236'34","reported_seq":719,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:50.907431+0000","last_change":"2024-02-05T08:19:44.892882+0000","last_active":"2024-02-05T08:20:50.907431+0000","last_peered":"2024-02-05T08:20:50.907431+0000","last_clean":"2024-02-05T08:19:41.827204+0000","last_became_active":"2024-02-05T08:19:44.892882+0000","last_became_peered":"2024-02-05T08:19:44.892882+0000","last_unstale":"2024-02-05T08:20:50.907431+0000","last_undegraded":"2024-02-05T08:19:44.885361+0000","last_fullsized":"2024-02-05T08:19:44.885220+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:18:47.576225+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:18:47.576225+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":102,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.085637+0000","last_change":"2024-02-05T08:20:16.870703+0000","last_active":"2024-02-05T08:20:51.085637+0000","last_peered":"2024-02-05T08:20:51.085637+0000","last_clean":"2024-02-05T08:20:51.085637+0000","last_became_active":"2024-02-05T08:18:34.642494+0000","last_became_peered":"2024-02-05T08:18:34.642494+0000","last_unstale":"2024-02-05T08:20:51.085637+0000","last_undegraded":"2024-02-05T08:20:51.085637+0000","last_fullsized":"2024-02-05T08:20:51.085637+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:16.870561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_clean_scrub_stamp":"2024-02-05T08:20:16.870561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:34.793930+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":614,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.904868+0000","last_change":"2024-02-05T08:19:12.320431+0000","last_active":"2024-02-05T08:20:50.904868+0000","last_peered":"2024-02-05T08:20:50.904868+0000","last_clean":"2024-02-05T08:20:50.904868+0000","last_became_active":"2024-02-05T08:18:34.638652+0000","last_became_peered":"2024-02-05T08:18:34.638652+0000","last_unstale":"2024-02-05T08:20:50.904868+0000","last_undegraded":"2024-02-05T08:20:50.904868+0000","last_fullsized":"2024-02-05T08:20:50.904868+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:12.320317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:19:12.320317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":605,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:50.904801+0000","last_change":"2024-02-05T08:19:44.912456+0000","last_active":"2024-02-05T08:20:50.904801+0000","last_peered":"2024-02-05T08:20:50.904801+0000","last_clean":"2024-02-05T08:19:42.900163+0000","last_became_active":"2024-02-05T08:19:44.912456+0000","last_became_peered":"2024-02-05T08:19:44.912456+0000","last_unstale":"2024-02-05T08:20:50.904801+0000","last_undegraded":"2024-02-05T08:20:50.904801+0000","last_fullsized":"2024-02-05T08:19:44.874216+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:19.338699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:19:19.338699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"236'23","reported_seq":108,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:51.084941+0000","last_change":"2024-02-05T08:19:44.923092+0000","last_active":"2024-02-05T08:20:51.084941+0000","last_peered":"2024-02-05T08:20:51.084941+0000","last_clean":"2024-02-05T08:19:41.835633+0000","last_became_active":"2024-02-05T08:19:44.923092+0000","last_became_peered":"2024-02-05T08:19:44.923092+0000","last_unstale":"2024-02-05T08:20:51.084941+0000","last_undegraded":"2024-02-05T08:19:44.886225+0000","last_fullsized":"2024-02-05T08:19:44.886101+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'23","last_scrub_stamp":"2024-02-05T08:17:58.012827+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:17:58.012827+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":617,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.088746+0000","last_change":"2024-02-05T08:20:37.119269+0000","last_active":"2024-02-05T08:20:51.088746+0000","last_peered":"2024-02-05T08:20:51.088746+0000","last_clean":"2024-02-05T08:20:51.088746+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:20:51.088746+0000","last_undegraded":"2024-02-05T08:20:51.088746+0000","last_fullsized":"2024-02-05T08:20:51.088746+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:20:37.119142+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.119142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:41.418621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":454,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.779820+0000","last_change":"2024-02-05T08:20:17.417043+0000","last_active":"2024-02-05T08:20:51.779820+0000","last_peered":"2024-02-05T08:20:51.779820+0000","last_clean":"2024-02-05T08:20:51.779820+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:20:51.779820+0000","last_undegraded":"2024-02-05T08:20:51.779820+0000","last_fullsized":"2024-02-05T08:20:51.779820+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:20:17.416907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:17.416907+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:34.985509+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":559,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:51.084200+0000","last_change":"2024-02-05T08:19:44.915101+0000","last_active":"2024-02-05T08:20:51.084200+0000","last_peered":"2024-02-05T08:20:51.084200+0000","last_clean":"2024-02-05T08:18:29.609487+0000","last_became_active":"2024-02-05T08:19:44.915101+0000","last_became_peered":"2024-02-05T08:19:44.915101+0000","last_unstale":"2024-02-05T08:20:51.084200+0000","last_undegraded":"2024-02-05T08:19:44.874435+0000","last_fullsized":"2024-02-05T08:19:44.874324+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:19:28.743054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:28.743054+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:53.073454+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"236'20","reported_seq":699,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.720976+0000","last_change":"2024-02-05T08:19:35.669211+0000","last_active":"2024-02-05T08:20:46.720976+0000","last_peered":"2024-02-05T08:20:46.720976+0000","last_clean":"2024-02-05T08:20:46.720976+0000","last_became_active":"2024-02-05T08:18:34.645441+0000","last_became_peered":"2024-02-05T08:18:34.645441+0000","last_unstale":"2024-02-05T08:20:46.720976+0000","last_undegraded":"2024-02-05T08:20:46.720976+0000","last_fullsized":"2024-02-05T08:20:46.720976+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:19:35.668857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:35.668857+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:56.062065+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":31,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.025946+0000","last_change":"2024-02-05T08:19:44.923094+0000","last_active":"2024-02-05T08:20:51.025946+0000","last_peered":"2024-02-05T08:20:51.025946+0000","last_clean":"2024-02-05T08:04:46.546354+0000","last_became_active":"2024-02-05T08:19:44.923094+0000","last_became_peered":"2024-02-05T08:19:44.923094+0000","last_unstale":"2024-02-05T08:20:51.025946+0000","last_undegraded":"2024-02-05T08:20:51.025946+0000","last_fullsized":"2024-02-05T08:19:44.897835+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:46.550985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:46.550985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":592,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.721247+0000","last_change":"2024-02-05T08:19:32.665907+0000","last_active":"2024-02-05T08:20:46.721247+0000","last_peered":"2024-02-05T08:20:46.721247+0000","last_clean":"2024-02-05T08:20:46.721247+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:20:46.721247+0000","last_undegraded":"2024-02-05T08:20:46.721247+0000","last_fullsized":"2024-02-05T08:20:46.721247+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:32.665558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:32.231861+0000","last_clean_scrub_stamp":"2024-02-05T08:19:32.665558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:50.479391+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":104,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.084211+0000","last_change":"2024-02-05T08:19:44.892769+0000","last_active":"2024-02-05T08:20:51.084211+0000","last_peered":"2024-02-05T08:20:51.084211+0000","last_clean":"2024-02-05T08:19:41.827910+0000","last_became_active":"2024-02-05T08:19:44.892769+0000","last_became_peered":"2024-02-05T08:19:44.892769+0000","last_unstale":"2024-02-05T08:20:51.084211+0000","last_undegraded":"2024-02-05T08:20:51.084211+0000","last_fullsized":"2024-02-05T08:19:44.874377+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:11.049460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:19:11.049460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"236'19","reported_seq":365,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.906774+0000","last_change":"2024-02-05T08:19:34.457468+0000","last_active":"2024-02-05T08:20:50.906774+0000","last_peered":"2024-02-05T08:20:50.906774+0000","last_clean":"2024-02-05T08:20:50.906774+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:20:50.906774+0000","last_undegraded":"2024-02-05T08:20:50.906774+0000","last_fullsized":"2024-02-05T08:20:50.906774+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:19:34.457012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:34.457012+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":591,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084548+0000","last_change":"2024-02-05T08:19:26.157804+0000","last_active":"2024-02-05T08:20:51.084548+0000","last_peered":"2024-02-05T08:20:51.084548+0000","last_clean":"2024-02-05T08:20:51.084548+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:20:51.084548+0000","last_undegraded":"2024-02-05T08:20:51.084548+0000","last_fullsized":"2024-02-05T08:20:51.084548+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:26.157698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:26.157698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:56.024473+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":343,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.906882+0000","last_change":"2024-02-05T08:20:27.591858+0000","last_active":"2024-02-05T08:20:50.906882+0000","last_peered":"2024-02-05T08:20:50.906882+0000","last_clean":"2024-02-05T08:20:50.906882+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:20:50.906882+0000","last_undegraded":"2024-02-05T08:20:50.906882+0000","last_fullsized":"2024-02-05T08:20:50.906882+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:27.591651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:20:27.591651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:42.110964+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":611,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.720420+0000","last_change":"2024-02-05T08:19:34.697406+0000","last_active":"2024-02-05T08:20:46.720420+0000","last_peered":"2024-02-05T08:20:46.720420+0000","last_clean":"2024-02-05T08:20:46.720420+0000","last_became_active":"2024-02-05T08:18:34.644537+0000","last_became_peered":"2024-02-05T08:18:34.644537+0000","last_unstale":"2024-02-05T08:20:46.720420+0000","last_undegraded":"2024-02-05T08:20:46.720420+0000","last_fullsized":"2024-02-05T08:20:46.720420+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:34.697248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:34.697248+0000","last_clean_scrub_stamp":"2024-02-05T08:19:34.697248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:49.643061+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":110,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:51.084420+0000","last_change":"2024-02-05T08:19:44.911502+0000","last_active":"2024-02-05T08:20:51.084420+0000","last_peered":"2024-02-05T08:20:51.084420+0000","last_clean":"2024-02-05T08:19:41.828151+0000","last_became_active":"2024-02-05T08:19:44.911502+0000","last_became_peered":"2024-02-05T08:19:44.911502+0000","last_unstale":"2024-02-05T08:20:51.084420+0000","last_undegraded":"2024-02-05T08:19:44.874110+0000","last_fullsized":"2024-02-05T08:19:44.873994+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:19:23.023541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:23.023541+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":109,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084761+0000","last_change":"2024-02-05T08:20:26.891410+0000","last_active":"2024-02-05T08:20:51.084761+0000","last_peered":"2024-02-05T08:20:51.084761+0000","last_clean":"2024-02-05T08:20:51.084761+0000","last_became_active":"2024-02-05T08:18:34.651979+0000","last_became_peered":"2024-02-05T08:18:34.651979+0000","last_unstale":"2024-02-05T08:20:51.084761+0000","last_undegraded":"2024-02-05T08:20:51.084761+0000","last_fullsized":"2024-02-05T08:20:51.084761+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:26.891040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:26.891040+0000","last_clean_scrub_stamp":"2024-02-05T08:20:26.891040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:47.586543+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":623,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084513+0000","last_change":"2024-02-05T08:20:18.147831+0000","last_active":"2024-02-05T08:20:51.084513+0000","last_peered":"2024-02-05T08:20:51.084513+0000","last_clean":"2024-02-05T08:20:51.084513+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:20:51.084513+0000","last_undegraded":"2024-02-05T08:20:51.084513+0000","last_fullsized":"2024-02-05T08:20:51.084513+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:18.147762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:20:18.147762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:27.942302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":613,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.026005+0000","last_change":"2024-02-05T08:19:44.904157+0000","last_active":"2024-02-05T08:20:51.026005+0000","last_peered":"2024-02-05T08:20:51.026005+0000","last_clean":"2024-02-05T08:19:41.832352+0000","last_became_active":"2024-02-05T08:19:44.904157+0000","last_became_peered":"2024-02-05T08:19:44.904157+0000","last_unstale":"2024-02-05T08:20:51.026005+0000","last_undegraded":"2024-02-05T08:20:51.026005+0000","last_fullsized":"2024-02-05T08:19:44.901922+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:41.576882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:18:41.576882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"236'24","reported_seq":113,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084673+0000","last_change":"2024-02-05T08:20:11.675925+0000","last_active":"2024-02-05T08:20:51.084673+0000","last_peered":"2024-02-05T08:20:51.084673+0000","last_clean":"2024-02-05T08:20:51.084673+0000","last_became_active":"2024-02-05T08:18:34.631912+0000","last_became_peered":"2024-02-05T08:18:34.631912+0000","last_unstale":"2024-02-05T08:20:51.084673+0000","last_undegraded":"2024-02-05T08:20:51.084673+0000","last_fullsized":"2024-02-05T08:20:51.084673+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:20:11.675780+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:20:11.675780+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:35.282973+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":590,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-02-05T08:20:46.719364+0000","last_change":"2024-02-05T08:19:44.902556+0000","last_active":"2024-02-05T08:20:46.719364+0000","last_peered":"2024-02-05T08:20:46.719364+0000","last_clean":"2024-02-05T08:19:41.834123+0000","last_became_active":"2024-02-05T08:19:44.902556+0000","last_became_peered":"2024-02-05T08:19:44.902556+0000","last_unstale":"2024-02-05T08:20:46.719364+0000","last_undegraded":"2024-02-05T08:20:46.719364+0000","last_fullsized":"2024-02-05T08:19:44.874049+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:26.593364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:19:26.593364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:46.904274+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":104,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.084491+0000","last_change":"2024-02-05T08:19:44.892864+0000","last_active":"2024-02-05T08:20:51.084491+0000","last_peered":"2024-02-05T08:20:51.084491+0000","last_clean":"2024-02-05T08:19:41.831557+0000","last_became_active":"2024-02-05T08:19:44.892864+0000","last_became_peered":"2024-02-05T08:19:44.892864+0000","last_unstale":"2024-02-05T08:20:51.084491+0000","last_undegraded":"2024-02-05T08:20:51.084491+0000","last_fullsized":"2024-02-05T08:19:44.886732+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:44.133926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:44.133926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":623,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.088364+0000","last_change":"2024-02-05T08:20:16.189809+0000","last_active":"2024-02-05T08:20:51.088364+0000","last_peered":"2024-02-05T08:20:51.088364+0000","last_clean":"2024-02-05T08:20:51.088364+0000","last_became_active":"2024-02-05T08:18:34.642387+0000","last_became_peered":"2024-02-05T08:18:34.642387+0000","last_unstale":"2024-02-05T08:20:51.088364+0000","last_undegraded":"2024-02-05T08:20:51.088364+0000","last_fullsized":"2024-02-05T08:20:51.088364+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:16.189515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:16.189515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:24.853924+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"263'19","reported_seq":278,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.025361+0000","last_change":"2024-02-05T08:20:19.330332+0000","last_active":"2024-02-05T08:20:51.025361+0000","last_peered":"2024-02-05T08:20:51.025361+0000","last_clean":"2024-02-05T08:20:51.025361+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:20:51.025361+0000","last_undegraded":"2024-02-05T08:20:51.025361+0000","last_fullsized":"2024-02-05T08:20:51.025361+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:20:19.330198+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.330198+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:41.061827+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":109,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084557+0000","last_change":"2024-02-05T08:20:23.812466+0000","last_active":"2024-02-05T08:20:51.084557+0000","last_peered":"2024-02-05T08:20:51.084557+0000","last_clean":"2024-02-05T08:20:51.084557+0000","last_became_active":"2024-02-05T08:18:34.650359+0000","last_became_peered":"2024-02-05T08:18:34.650359+0000","last_unstale":"2024-02-05T08:20:51.084557+0000","last_undegraded":"2024-02-05T08:20:51.084557+0000","last_fullsized":"2024-02-05T08:20:51.084557+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:23.812353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:20:23.812353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:29.490140+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":546,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.025679+0000","last_change":"2024-02-05T08:19:44.903986+0000","last_active":"2024-02-05T08:20:51.025679+0000","last_peered":"2024-02-05T08:20:51.025679+0000","last_clean":"2024-02-05T08:18:29.613784+0000","last_became_active":"2024-02-05T08:19:44.903986+0000","last_became_peered":"2024-02-05T08:19:44.903986+0000","last_unstale":"2024-02-05T08:20:51.025679+0000","last_undegraded":"2024-02-05T08:20:51.025679+0000","last_fullsized":"2024-02-05T08:19:44.899948+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:17.633650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.633650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":630,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.904672+0000","last_change":"2024-02-05T08:20:20.151509+0000","last_active":"2024-02-05T08:20:50.904672+0000","last_peered":"2024-02-05T08:20:50.904672+0000","last_clean":"2024-02-05T08:20:50.904672+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:20:50.904672+0000","last_undegraded":"2024-02-05T08:20:50.904672+0000","last_fullsized":"2024-02-05T08:20:50.904672+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:20.151160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:20:20.151160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:40.580006+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"249'24","reported_seq":681,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:50.904417+0000","last_change":"2024-02-05T08:19:44.930725+0000","last_active":"2024-02-05T08:20:50.904417+0000","last_peered":"2024-02-05T08:20:50.904417+0000","last_clean":"2024-02-05T08:19:41.830358+0000","last_became_active":"2024-02-05T08:19:44.930725+0000","last_became_peered":"2024-02-05T08:19:44.930725+0000","last_unstale":"2024-02-05T08:20:50.904417+0000","last_undegraded":"2024-02-05T08:19:44.912297+0000","last_fullsized":"2024-02-05T08:19:44.910005+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:19:27.349541+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:19:27.349541+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:51.259959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":599,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.721352+0000","last_change":"2024-02-05T08:20:29.732543+0000","last_active":"2024-02-05T08:20:46.721352+0000","last_peered":"2024-02-05T08:20:46.721352+0000","last_clean":"2024-02-05T08:20:46.721352+0000","last_became_active":"2024-02-05T08:18:34.645027+0000","last_became_peered":"2024-02-05T08:18:34.645027+0000","last_unstale":"2024-02-05T08:20:46.721352+0000","last_undegraded":"2024-02-05T08:20:46.721352+0000","last_fullsized":"2024-02-05T08:20:46.721352+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:29.732422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:29.732422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:58.490572+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"325'9","reported_seq":809,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:51.026267+0000","last_change":"2024-02-05T08:19:44.900658+0000","last_active":"2024-02-05T08:20:51.026267+0000","last_peered":"2024-02-05T08:20:51.026267+0000","last_clean":"2024-02-05T08:19:41.831639+0000","last_became_active":"2024-02-05T08:19:44.900658+0000","last_became_peered":"2024-02-05T08:19:44.900658+0000","last_unstale":"2024-02-05T08:20:51.026267+0000","last_undegraded":"2024-02-05T08:19:44.898660+0000","last_fullsized":"2024-02-05T08:19:44.898552+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:19:12.452237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:12.452237+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":611,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084394+0000","last_change":"2024-02-05T08:19:25.118455+0000","last_active":"2024-02-05T08:20:51.084394+0000","last_peered":"2024-02-05T08:20:51.084394+0000","last_clean":"2024-02-05T08:20:51.084394+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:20:51.084394+0000","last_undegraded":"2024-02-05T08:20:51.084394+0000","last_fullsized":"2024-02-05T08:20:51.084394+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:25.118336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:25.118336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:56.024441+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":116,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:51.085009+0000","last_change":"2024-02-05T08:19:44.899031+0000","last_active":"2024-02-05T08:20:51.085009+0000","last_peered":"2024-02-05T08:20:51.085009+0000","last_clean":"2024-02-05T08:19:41.830913+0000","last_became_active":"2024-02-05T08:19:44.899031+0000","last_became_peered":"2024-02-05T08:19:44.899031+0000","last_unstale":"2024-02-05T08:20:51.085009+0000","last_undegraded":"2024-02-05T08:19:44.887736+0000","last_fullsized":"2024-02-05T08:19:44.887622+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-02-05T08:19:35.981313+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:19:35.981313+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":593,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.904653+0000","last_change":"2024-02-05T08:19:28.519254+0000","last_active":"2024-02-05T08:20:50.904653+0000","last_peered":"2024-02-05T08:20:50.904653+0000","last_clean":"2024-02-05T08:20:50.904653+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:20:50.904653+0000","last_undegraded":"2024-02-05T08:20:50.904653+0000","last_fullsized":"2024-02-05T08:20:50.904653+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:28.519149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:28.519149+0000","last_clean_scrub_stamp":"2024-02-05T08:19:28.519149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"295'10","reported_seq":134,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.085421+0000","last_change":"2024-02-05T08:20:14.713693+0000","last_active":"2024-02-05T08:20:51.085421+0000","last_peered":"2024-02-05T08:20:51.085421+0000","last_clean":"2024-02-05T08:20:51.085421+0000","last_became_active":"2024-02-05T08:18:34.644517+0000","last_became_peered":"2024-02-05T08:18:34.644517+0000","last_unstale":"2024-02-05T08:20:51.085421+0000","last_undegraded":"2024-02-05T08:20:51.085421+0000","last_fullsized":"2024-02-05T08:20:51.085421+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'10","last_scrub_stamp":"2024-02-05T08:20:14.713589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:14.713589+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:37.047228+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":611,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.721406+0000","last_change":"2024-02-05T08:19:38.660445+0000","last_active":"2024-02-05T08:20:46.721406+0000","last_peered":"2024-02-05T08:20:46.721406+0000","last_clean":"2024-02-05T08:20:46.721406+0000","last_became_active":"2024-02-05T08:18:34.635793+0000","last_became_peered":"2024-02-05T08:18:34.635793+0000","last_unstale":"2024-02-05T08:20:46.721406+0000","last_undegraded":"2024-02-05T08:20:46.721406+0000","last_fullsized":"2024-02-05T08:20:46.721406+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:38.659315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:19:38.659315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:53.108625+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":671,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.026402+0000","last_change":"2024-02-05T08:20:23.330119+0000","last_active":"2024-02-05T08:20:51.026402+0000","last_peered":"2024-02-05T08:20:51.026402+0000","last_clean":"2024-02-05T08:20:51.026402+0000","last_became_active":"2024-02-05T08:18:34.640741+0000","last_became_peered":"2024-02-05T08:18:34.640741+0000","last_unstale":"2024-02-05T08:20:51.026402+0000","last_undegraded":"2024-02-05T08:20:51.026402+0000","last_fullsized":"2024-02-05T08:20:51.026402+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:20:23.329973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:20:23.329973+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:50.365153+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":614,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.907697+0000","last_change":"2024-02-05T08:20:24.919443+0000","last_active":"2024-02-05T08:20:50.907697+0000","last_peered":"2024-02-05T08:20:50.907697+0000","last_clean":"2024-02-05T08:20:50.907697+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:20:50.907697+0000","last_undegraded":"2024-02-05T08:20:50.907697+0000","last_fullsized":"2024-02-05T08:20:50.907697+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:24.919305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:24.919305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:33.051917+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":104,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.085586+0000","last_change":"2024-02-05T08:19:44.899825+0000","last_active":"2024-02-05T08:20:51.085586+0000","last_peered":"2024-02-05T08:20:51.085586+0000","last_clean":"2024-02-05T08:19:41.832250+0000","last_became_active":"2024-02-05T08:19:44.899825+0000","last_became_peered":"2024-02-05T08:19:44.899825+0000","last_unstale":"2024-02-05T08:20:51.085586+0000","last_undegraded":"2024-02-05T08:20:51.085586+0000","last_fullsized":"2024-02-05T08:19:44.888998+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:37.028991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:37.028991+0000","last_clean_scrub_stamp":"2024-02-05T08:19:37.028991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":623,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084416+0000","last_change":"2024-02-05T08:20:17.150031+0000","last_active":"2024-02-05T08:20:51.084416+0000","last_peered":"2024-02-05T08:20:51.084416+0000","last_clean":"2024-02-05T08:20:51.084416+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:20:51.084416+0000","last_undegraded":"2024-02-05T08:20:51.084416+0000","last_fullsized":"2024-02-05T08:20:51.084416+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:17.149915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:20:17.149915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:32.105130+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"278'19","reported_seq":656,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.721052+0000","last_change":"2024-02-05T08:19:31.636110+0000","last_active":"2024-02-05T08:20:46.721052+0000","last_peered":"2024-02-05T08:20:46.721052+0000","last_clean":"2024-02-05T08:20:46.721052+0000","last_became_active":"2024-02-05T08:18:34.645881+0000","last_became_peered":"2024-02-05T08:18:34.645881+0000","last_unstale":"2024-02-05T08:20:46.721052+0000","last_undegraded":"2024-02-05T08:20:46.721052+0000","last_fullsized":"2024-02-05T08:20:46.721052+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-05T08:19:31.635979+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:19:31.635979+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:01.303372+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":611,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084566+0000","last_change":"2024-02-05T08:19:15.271802+0000","last_active":"2024-02-05T08:20:51.084566+0000","last_peered":"2024-02-05T08:20:51.084566+0000","last_clean":"2024-02-05T08:20:51.084566+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:20:51.084566+0000","last_undegraded":"2024-02-05T08:20:51.084566+0000","last_fullsized":"2024-02-05T08:20:51.084566+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:15.271671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.271671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:56.024556+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":617,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.026270+0000","last_change":"2024-02-05T08:19:44.903925+0000","last_active":"2024-02-05T08:20:51.026270+0000","last_peered":"2024-02-05T08:20:51.026270+0000","last_clean":"2024-02-05T08:19:41.832619+0000","last_became_active":"2024-02-05T08:19:44.903925+0000","last_became_peered":"2024-02-05T08:19:44.903925+0000","last_unstale":"2024-02-05T08:20:51.026270+0000","last_undegraded":"2024-02-05T08:20:51.026270+0000","last_fullsized":"2024-02-05T08:19:44.901051+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.486901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.486901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":340,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.906215+0000","last_change":"2024-02-05T08:19:16.518726+0000","last_active":"2024-02-05T08:20:50.906215+0000","last_peered":"2024-02-05T08:20:50.906215+0000","last_clean":"2024-02-05T08:20:50.906215+0000","last_became_active":"2024-02-05T08:12:29.684444+0000","last_became_peered":"2024-02-05T08:12:29.684444+0000","last_unstale":"2024-02-05T08:20:50.906215+0000","last_undegraded":"2024-02-05T08:20:50.906215+0000","last_fullsized":"2024-02-05T08:20:50.906215+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.518655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.518655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:54.561034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'15","reported_seq":104,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:51.085484+0000","last_change":"2024-02-05T08:19:44.899823+0000","last_active":"2024-02-05T08:20:51.085484+0000","last_peered":"2024-02-05T08:20:51.085484+0000","last_clean":"2024-02-05T08:19:41.832108+0000","last_became_active":"2024-02-05T08:19:44.899823+0000","last_became_peered":"2024-02-05T08:19:44.899823+0000","last_unstale":"2024-02-05T08:20:51.085484+0000","last_undegraded":"2024-02-05T08:19:44.888775+0000","last_fullsized":"2024-02-05T08:19:44.888665+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:17:54.101749+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.101749+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"336'8275","reported_seq":8423,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:53.225020+0000","last_change":"2024-02-05T08:20:40.673313+0000","last_active":"2024-02-05T08:20:53.225020+0000","last_peered":"2024-02-05T08:20:53.225020+0000","last_clean":"2024-02-05T08:20:53.225020+0000","last_became_active":"2024-02-05T08:19:07.332783+0000","last_became_peered":"2024-02-05T08:19:07.332783+0000","last_unstale":"2024-02-05T08:20:53.225020+0000","last_undegraded":"2024-02-05T08:20:53.225020+0000","last_fullsized":"2024-02-05T08:20:53.225020+0000","mapping_epoch":309,"log_start":"322'3100","ondisk_log_start":"322'3100","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'7546","last_scrub_stamp":"2024-02-05T08:20:40.673010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:40.673010+0000","objects_scrubbed":7412,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.387460+0000","scrub_duration":4450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521863168,"num_objects":7963,"num_object_clones":0,"num_object_copies":23889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8275,"num_write_kb":519616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":619,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.025886+0000","last_change":"2024-02-05T08:20:13.502530+0000","last_active":"2024-02-05T08:20:51.025886+0000","last_peered":"2024-02-05T08:20:51.025886+0000","last_clean":"2024-02-05T08:20:51.025886+0000","last_became_active":"2024-02-05T08:12:29.679849+0000","last_became_peered":"2024-02-05T08:12:29.679849+0000","last_unstale":"2024-02-05T08:20:51.025886+0000","last_undegraded":"2024-02-05T08:20:51.025886+0000","last_fullsized":"2024-02-05T08:20:51.025886+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:13.502417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:13.502417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:15.250048+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":632,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.088989+0000","last_change":"2024-02-05T08:20:34.173135+0000","last_active":"2024-02-05T08:20:51.088989+0000","last_peered":"2024-02-05T08:20:51.088989+0000","last_clean":"2024-02-05T08:20:51.088989+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:20:51.088989+0000","last_undegraded":"2024-02-05T08:20:51.088989+0000","last_fullsized":"2024-02-05T08:20:51.088989+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:20:34.173063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:34.173063+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:48.768654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"262'34","reported_seq":701,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.025407+0000","last_change":"2024-02-05T08:19:34.364882+0000","last_active":"2024-02-05T08:20:51.025407+0000","last_peered":"2024-02-05T08:20:51.025407+0000","last_clean":"2024-02-05T08:20:51.025407+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:20:51.025407+0000","last_undegraded":"2024-02-05T08:20:51.025407+0000","last_fullsized":"2024-02-05T08:20:51.025407+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'34","last_scrub_stamp":"2024-02-05T08:19:34.364738+0000","last_deep_scrub":"262'34","last_deep_scrub_stamp":"2024-02-05T08:19:34.364738+0000","last_clean_scrub_stamp":"2024-02-05T08:19:34.364738+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:59.589929+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":828,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.088585+0000","last_change":"2024-02-05T08:19:31.117664+0000","last_active":"2024-02-05T08:20:51.088585+0000","last_peered":"2024-02-05T08:20:51.088585+0000","last_clean":"2024-02-05T08:20:51.088585+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:20:51.088585+0000","last_undegraded":"2024-02-05T08:20:51.088585+0000","last_fullsized":"2024-02-05T08:20:51.088585+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_clean_scrub_stamp":"2024-02-05T08:19:31.117523+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:58.253463+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"336'8277","reported_seq":8341,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:52.898178+0000","last_change":"2024-02-05T08:19:07.346353+0000","last_active":"2024-02-05T08:20:52.898178+0000","last_peered":"2024-02-05T08:20:52.898178+0000","last_clean":"2024-02-05T08:20:52.898178+0000","last_became_active":"2024-02-05T08:19:07.346046+0000","last_became_peered":"2024-02-05T08:19:07.346046+0000","last_unstale":"2024-02-05T08:20:52.898178+0000","last_undegraded":"2024-02-05T08:20:52.898178+0000","last_fullsized":"2024-02-05T08:20:52.898178+0000","mapping_epoch":309,"log_start":"320'2700","ondisk_log_start":"320'2700","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":5577,"log_dups_size":0,"ondisk_log_size":5577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:54.560961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":525926400,"num_objects":8025,"num_object_clones":0,"num_object_copies":24075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8277,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":97,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.085085+0000","last_change":"2024-02-05T08:19:44.899425+0000","last_active":"2024-02-05T08:20:51.085085+0000","last_peered":"2024-02-05T08:20:51.085085+0000","last_clean":"2024-02-05T08:19:41.833419+0000","last_became_active":"2024-02-05T08:19:44.899425+0000","last_became_peered":"2024-02-05T08:19:44.899425+0000","last_unstale":"2024-02-05T08:20:51.085085+0000","last_undegraded":"2024-02-05T08:20:51.085085+0000","last_fullsized":"2024-02-05T08:19:44.885939+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:31.245736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:18:31.245736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:56.732499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":615,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.088910+0000","last_change":"2024-02-05T08:19:44.926408+0000","last_active":"2024-02-05T08:20:51.088910+0000","last_peered":"2024-02-05T08:20:51.088910+0000","last_clean":"2024-02-05T08:19:41.835482+0000","last_became_active":"2024-02-05T08:19:44.926408+0000","last_became_peered":"2024-02-05T08:19:44.926408+0000","last_unstale":"2024-02-05T08:20:51.088910+0000","last_undegraded":"2024-02-05T08:20:51.088910+0000","last_fullsized":"2024-02-05T08:19:44.901657+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:37.226829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:19:37.226829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:53.087750+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":670,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:50.903952+0000","last_change":"2024-02-05T08:19:44.912589+0000","last_active":"2024-02-05T08:20:50.903952+0000","last_peered":"2024-02-05T08:20:50.903952+0000","last_clean":"2024-02-05T08:19:41.830832+0000","last_became_active":"2024-02-05T08:19:44.912589+0000","last_became_peered":"2024-02-05T08:19:44.912589+0000","last_unstale":"2024-02-05T08:20:50.903952+0000","last_undegraded":"2024-02-05T08:19:44.909517+0000","last_fullsized":"2024-02-05T08:19:44.909410+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-02-05T08:18:02.629366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:02.629366+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":614,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:50.906697+0000","last_change":"2024-02-05T08:19:44.888161+0000","last_active":"2024-02-05T08:20:50.906697+0000","last_peered":"2024-02-05T08:20:50.906697+0000","last_clean":"2024-02-05T08:19:41.831112+0000","last_became_active":"2024-02-05T08:19:44.888161+0000","last_became_peered":"2024-02-05T08:19:44.888161+0000","last_unstale":"2024-02-05T08:20:50.906697+0000","last_undegraded":"2024-02-05T08:20:50.906697+0000","last_fullsized":"2024-02-05T08:19:44.872217+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:33.725647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:18:33.725647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"335'8122","reported_seq":8266,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:48.942826+0000","last_change":"2024-02-05T08:20:35.938379+0000","last_active":"2024-02-05T08:20:48.942826+0000","last_peered":"2024-02-05T08:20:48.942826+0000","last_clean":"2024-02-05T08:20:48.942826+0000","last_became_active":"2024-02-05T08:19:07.345271+0000","last_became_peered":"2024-02-05T08:19:07.345271+0000","last_unstale":"2024-02-05T08:20:48.942826+0000","last_undegraded":"2024-02-05T08:20:48.942826+0000","last_fullsized":"2024-02-05T08:20:48.942826+0000","mapping_epoch":309,"log_start":"322'3000","ondisk_log_start":"322'3000","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'7352","last_scrub_stamp":"2024-02-05T08:20:35.938230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:35.938230+0000","objects_scrubbed":7270,"log_size":5122,"log_dups_size":0,"ondisk_log_size":5122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.976602+0000","scrub_duration":2394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532283392,"num_objects":8122,"num_object_clones":0,"num_object_copies":24366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8122,"num_write_kb":519808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":612,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084128+0000","last_change":"2024-02-05T08:19:41.263058+0000","last_active":"2024-02-05T08:20:51.084128+0000","last_peered":"2024-02-05T08:20:51.084128+0000","last_clean":"2024-02-05T08:20:51.084128+0000","last_became_active":"2024-02-05T08:18:34.642239+0000","last_became_peered":"2024-02-05T08:18:34.642239+0000","last_unstale":"2024-02-05T08:20:51.084128+0000","last_undegraded":"2024-02-05T08:20:51.084128+0000","last_fullsized":"2024-02-05T08:20:51.084128+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:41.262935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:41.262935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:56.676913+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":619,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.907403+0000","last_change":"2024-02-05T08:20:37.971742+0000","last_active":"2024-02-05T08:20:50.907403+0000","last_peered":"2024-02-05T08:20:50.907403+0000","last_clean":"2024-02-05T08:20:50.907403+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:20:50.907403+0000","last_undegraded":"2024-02-05T08:20:50.907403+0000","last_fullsized":"2024-02-05T08:20:50.907403+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:37.971622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.971622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:50.211071+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":624,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.088767+0000","last_change":"2024-02-05T08:19:30.082858+0000","last_active":"2024-02-05T08:20:51.088767+0000","last_peered":"2024-02-05T08:20:51.088767+0000","last_clean":"2024-02-05T08:20:51.088767+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:20:51.088767+0000","last_undegraded":"2024-02-05T08:20:51.088767+0000","last_fullsized":"2024-02-05T08:20:51.088767+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_clean_scrub_stamp":"2024-02-05T08:19:30.082733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:56.024577+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"236'35","reported_seq":392,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.906564+0000","last_change":"2024-02-05T08:20:24.549048+0000","last_active":"2024-02-05T08:20:50.906564+0000","last_peered":"2024-02-05T08:20:50.906564+0000","last_clean":"2024-02-05T08:20:50.906564+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:20:50.906564+0000","last_undegraded":"2024-02-05T08:20:50.906564+0000","last_fullsized":"2024-02-05T08:20:50.906564+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'35","last_scrub_stamp":"2024-02-05T08:20:24.548947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:20:24.548947+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:41.121488+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"336'8529","reported_seq":8678,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:53.214281+0000","last_change":"2024-02-05T08:20:37.202482+0000","last_active":"2024-02-05T08:20:53.214281+0000","last_peered":"2024-02-05T08:20:53.214281+0000","last_clean":"2024-02-05T08:20:53.214281+0000","last_became_active":"2024-02-05T08:19:07.345197+0000","last_became_peered":"2024-02-05T08:19:07.345197+0000","last_unstale":"2024-02-05T08:20:53.214281+0000","last_undegraded":"2024-02-05T08:20:53.214281+0000","last_fullsized":"2024-02-05T08:20:53.214281+0000","mapping_epoch":309,"log_start":"320'2500","ondisk_log_start":"320'2500","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'7639","last_scrub_stamp":"2024-02-05T08:20:37.202377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.202377+0000","objects_scrubbed":7578,"log_size":6029,"log_dups_size":0,"ondisk_log_size":6029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:01.462904+0000","scrub_duration":2232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":541261824,"num_objects":8259,"num_object_clones":0,"num_object_copies":24777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8259,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8529,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":588,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.719891+0000","last_change":"2024-02-05T08:19:24.708102+0000","last_active":"2024-02-05T08:20:46.719891+0000","last_peered":"2024-02-05T08:20:46.719891+0000","last_clean":"2024-02-05T08:20:46.719891+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:20:46.719891+0000","last_undegraded":"2024-02-05T08:20:46.719891+0000","last_fullsized":"2024-02-05T08:20:46.719891+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_clean_scrub_stamp":"2024-02-05T08:19:24.707985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:50.479415+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":631,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.907094+0000","last_change":"2024-02-05T08:20:23.859434+0000","last_active":"2024-02-05T08:20:50.907094+0000","last_peered":"2024-02-05T08:20:50.907094+0000","last_clean":"2024-02-05T08:20:50.907094+0000","last_became_active":"2024-02-05T08:18:34.645830+0000","last_became_peered":"2024-02-05T08:18:34.645830+0000","last_unstale":"2024-02-05T08:20:50.907094+0000","last_undegraded":"2024-02-05T08:20:50.907094+0000","last_fullsized":"2024-02-05T08:20:50.907094+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:23.859245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:23.859245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:32.318976+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":635,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.025950+0000","last_change":"2024-02-05T08:19:17.520776+0000","last_active":"2024-02-05T08:20:51.025950+0000","last_peered":"2024-02-05T08:20:51.025950+0000","last_clean":"2024-02-05T08:20:51.025950+0000","last_became_active":"2024-02-05T08:18:34.633214+0000","last_became_peered":"2024-02-05T08:18:34.633214+0000","last_unstale":"2024-02-05T08:20:51.025950+0000","last_undegraded":"2024-02-05T08:20:51.025950+0000","last_fullsized":"2024-02-05T08:20:51.025950+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:17.520613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.520613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"262'19","reported_seq":628,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:46.720039+0000","last_change":"2024-02-05T08:19:44.914651+0000","last_active":"2024-02-05T08:20:46.720039+0000","last_peered":"2024-02-05T08:20:46.720039+0000","last_clean":"2024-02-05T08:19:41.838445+0000","last_became_active":"2024-02-05T08:19:44.914651+0000","last_became_peered":"2024-02-05T08:19:44.914651+0000","last_unstale":"2024-02-05T08:20:46.720039+0000","last_undegraded":"2024-02-05T08:19:44.886996+0000","last_fullsized":"2024-02-05T08:19:44.886890+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'19","last_scrub_stamp":"2024-02-05T08:19:17.554161+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.554161+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:50.479399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"336'8352","reported_seq":8491,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:52.929336+0000","last_change":"2024-02-05T08:20:25.764525+0000","last_active":"2024-02-05T08:20:52.929336+0000","last_peered":"2024-02-05T08:20:52.929336+0000","last_clean":"2024-02-05T08:20:52.929336+0000","last_became_active":"2024-02-05T08:19:07.331718+0000","last_became_peered":"2024-02-05T08:19:07.331718+0000","last_unstale":"2024-02-05T08:20:52.929336+0000","last_undegraded":"2024-02-05T08:20:52.929336+0000","last_fullsized":"2024-02-05T08:20:52.929336+0000","mapping_epoch":309,"log_start":"320'2700","ondisk_log_start":"320'2700","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'6676","last_scrub_stamp":"2024-02-05T08:20:25.764375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:25.764375+0000","objects_scrubbed":6586,"log_size":5652,"log_dups_size":0,"ondisk_log_size":5652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:42.802496+0000","scrub_duration":2281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":530448384,"num_objects":8094,"num_object_clones":0,"num_object_copies":24282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8352,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'130","reported_seq":385,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.025301+0000","last_change":"2024-02-05T08:20:25.365305+0000","last_active":"2024-02-05T08:20:51.025301+0000","last_peered":"2024-02-05T08:20:51.025301+0000","last_clean":"2024-02-05T08:20:51.025301+0000","last_became_active":"2024-02-05T08:19:01.816550+0000","last_became_peered":"2024-02-05T08:19:01.816550+0000","last_unstale":"2024-02-05T08:20:51.025301+0000","last_undegraded":"2024-02-05T08:20:51.025301+0000","last_fullsized":"2024-02-05T08:20:51.025301+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:25.365167+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:25.365167+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:32.657363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":324,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:50.911981+0000","last_change":"2024-02-05T08:19:44.925889+0000","last_active":"2024-02-05T08:20:50.911981+0000","last_peered":"2024-02-05T08:20:50.911981+0000","last_clean":"2024-02-05T08:19:41.831505+0000","last_became_active":"2024-02-05T08:19:44.925889+0000","last_became_peered":"2024-02-05T08:19:44.925889+0000","last_unstale":"2024-02-05T08:20:50.911981+0000","last_undegraded":"2024-02-05T08:20:50.911981+0000","last_fullsized":"2024-02-05T08:19:44.914970+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:54.817641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.817641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":646,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:51.025554+0000","last_change":"2024-02-05T08:19:44.903354+0000","last_active":"2024-02-05T08:20:51.025554+0000","last_peered":"2024-02-05T08:20:51.025554+0000","last_clean":"2024-02-05T08:19:41.832449+0000","last_became_active":"2024-02-05T08:19:44.903354+0000","last_became_peered":"2024-02-05T08:19:44.903354+0000","last_unstale":"2024-02-05T08:20:51.025554+0000","last_undegraded":"2024-02-05T08:19:44.873323+0000","last_fullsized":"2024-02-05T08:19:44.873213+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:18:48.543343+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:18:48.543343+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":623,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.903958+0000","last_change":"2024-02-05T08:20:27.119157+0000","last_active":"2024-02-05T08:20:50.903958+0000","last_peered":"2024-02-05T08:20:50.903958+0000","last_clean":"2024-02-05T08:20:50.903958+0000","last_became_active":"2024-02-05T08:12:29.670859+0000","last_became_peered":"2024-02-05T08:12:29.670859+0000","last_unstale":"2024-02-05T08:20:50.903958+0000","last_undegraded":"2024-02-05T08:20:50.903958+0000","last_fullsized":"2024-02-05T08:20:50.903958+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:27.119042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:20:27.119042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:41.963906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"325'53","reported_seq":651,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:50.907794+0000","last_change":"2024-02-05T08:19:44.892891+0000","last_active":"2024-02-05T08:20:50.907794+0000","last_peered":"2024-02-05T08:20:50.907794+0000","last_clean":"2024-02-05T08:14:49.167838+0000","last_became_active":"2024-02-05T08:19:44.892891+0000","last_became_peered":"2024-02-05T08:19:44.892891+0000","last_unstale":"2024-02-05T08:20:50.907794+0000","last_undegraded":"2024-02-05T08:19:44.886039+0000","last_fullsized":"2024-02-05T08:19:44.885927+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'52","last_scrub_stamp":"2024-02-05T08:19:15.601598+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.601598+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":587,"reported_epoch":335,"state":"active+undersized","last_fresh":"2024-02-05T08:20:46.720383+0000","last_change":"2024-02-05T08:19:44.924245+0000","last_active":"2024-02-05T08:20:46.720383+0000","last_peered":"2024-02-05T08:20:46.720383+0000","last_clean":"2024-02-05T08:19:41.838535+0000","last_became_active":"2024-02-05T08:19:44.924245+0000","last_became_peered":"2024-02-05T08:19:44.924245+0000","last_unstale":"2024-02-05T08:20:46.720383+0000","last_undegraded":"2024-02-05T08:20:46.720383+0000","last_fullsized":"2024-02-05T08:19:44.874710+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:14.615047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.615047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:50.479375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"336'8498","reported_seq":8631,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:53.260606+0000","last_change":"2024-02-05T08:20:16.549202+0000","last_active":"2024-02-05T08:20:53.260606+0000","last_peered":"2024-02-05T08:20:53.260606+0000","last_clean":"2024-02-05T08:20:53.260606+0000","last_became_active":"2024-02-05T08:19:07.333532+0000","last_became_peered":"2024-02-05T08:19:07.333532+0000","last_unstale":"2024-02-05T08:20:53.260606+0000","last_undegraded":"2024-02-05T08:20:53.260606+0000","last_fullsized":"2024-02-05T08:20:53.260606+0000","mapping_epoch":309,"log_start":"323'3300","ondisk_log_start":"323'3300","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'6042","last_scrub_stamp":"2024-02-05T08:20:16.548919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:16.548919+0000","objects_scrubbed":5971,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:44.635697+0000","scrub_duration":2107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535822336,"num_objects":8176,"num_object_clones":0,"num_object_copies":24528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8498,"num_write_kb":533568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'130","reported_seq":917,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084413+0000","last_change":"2024-02-05T08:20:27.164887+0000","last_active":"2024-02-05T08:20:51.084413+0000","last_peered":"2024-02-05T08:20:51.084413+0000","last_clean":"2024-02-05T08:20:51.084413+0000","last_became_active":"2024-02-05T08:19:01.813292+0000","last_became_peered":"2024-02-05T08:19:01.813292+0000","last_unstale":"2024-02-05T08:20:51.084413+0000","last_undegraded":"2024-02-05T08:20:51.084413+0000","last_fullsized":"2024-02-05T08:20:51.084413+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:27.164766+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:27.164766+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:40.161712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":624,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.088844+0000","last_change":"2024-02-05T08:20:21.189633+0000","last_active":"2024-02-05T08:20:51.088844+0000","last_peered":"2024-02-05T08:20:51.088844+0000","last_clean":"2024-02-05T08:20:51.088844+0000","last_became_active":"2024-02-05T08:18:34.643109+0000","last_became_peered":"2024-02-05T08:18:34.643109+0000","last_unstale":"2024-02-05T08:20:51.088844+0000","last_undegraded":"2024-02-05T08:20:51.088844+0000","last_fullsized":"2024-02-05T08:20:51.088844+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:21.189500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:21.189500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:37.050505+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":105,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:50.906631+0000","last_change":"2024-02-05T08:19:44.902069+0000","last_active":"2024-02-05T08:20:50.906631+0000","last_peered":"2024-02-05T08:20:50.906631+0000","last_clean":"2024-02-05T08:06:34.916685+0000","last_became_active":"2024-02-05T08:19:44.902069+0000","last_became_peered":"2024-02-05T08:19:44.902069+0000","last_unstale":"2024-02-05T08:20:50.906631+0000","last_undegraded":"2024-02-05T08:19:44.872249+0000","last_fullsized":"2024-02-05T08:19:44.872099+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:18:43.341831+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:18:43.341831+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":648,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.907004+0000","last_change":"2024-02-05T08:20:29.922938+0000","last_active":"2024-02-05T08:20:50.907004+0000","last_peered":"2024-02-05T08:20:50.907004+0000","last_clean":"2024-02-05T08:20:50.907004+0000","last_became_active":"2024-02-05T08:18:34.644683+0000","last_became_peered":"2024-02-05T08:18:34.644683+0000","last_unstale":"2024-02-05T08:20:50.907004+0000","last_undegraded":"2024-02-05T08:20:50.907004+0000","last_fullsized":"2024-02-05T08:20:50.907004+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_clean_scrub_stamp":"2024-02-05T08:20:29.922794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.839539+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'17","reported_seq":625,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.907742+0000","last_change":"2024-02-05T08:19:11.615846+0000","last_active":"2024-02-05T08:20:50.907742+0000","last_peered":"2024-02-05T08:20:50.907742+0000","last_clean":"2024-02-05T08:20:50.907742+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:20:50.907742+0000","last_undegraded":"2024-02-05T08:20:50.907742+0000","last_fullsized":"2024-02-05T08:20:50.907742+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'17","last_scrub_stamp":"2024-02-05T08:19:11.615763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:11.615763+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":627,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.910878+0000","last_change":"2024-02-05T08:20:37.014562+0000","last_active":"2024-02-05T08:20:50.910878+0000","last_peered":"2024-02-05T08:20:50.910878+0000","last_clean":"2024-02-05T08:20:50.910878+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:20:50.910878+0000","last_undegraded":"2024-02-05T08:20:50.910878+0000","last_fullsized":"2024-02-05T08:20:50.910878+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:37.014448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.014448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:02.259236+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"336'8257","reported_seq":8322,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:52.918632+0000","last_change":"2024-02-05T08:19:44.910611+0000","last_active":"2024-02-05T08:20:52.918632+0000","last_peered":"2024-02-05T08:20:52.918632+0000","last_clean":"2024-02-05T08:19:43.860522+0000","last_became_active":"2024-02-05T08:19:44.910611+0000","last_became_peered":"2024-02-05T08:19:44.910611+0000","last_unstale":"2024-02-05T08:20:52.918632+0000","last_undegraded":"2024-02-05T08:19:44.872215+0000","last_fullsized":"2024-02-05T08:19:44.872102+0000","mapping_epoch":324,"log_start":"320'2700","ondisk_log_start":"320'2700","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:54.560883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":527892480,"num_objects":8055,"num_object_clones":0,"num_object_copies":24165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8257,"num_write_kb":521984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":8055}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":611,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.026316+0000","last_change":"2024-02-05T08:19:19.557662+0000","last_active":"2024-02-05T08:20:51.026316+0000","last_peered":"2024-02-05T08:20:51.026316+0000","last_clean":"2024-02-05T08:20:51.026316+0000","last_became_active":"2024-02-05T08:18:34.633450+0000","last_became_peered":"2024-02-05T08:18:34.633450+0000","last_unstale":"2024-02-05T08:20:51.026316+0000","last_undegraded":"2024-02-05T08:20:51.026316+0000","last_fullsized":"2024-02-05T08:20:51.026316+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:19.557556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:19.557556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":910,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084232+0000","last_change":"2024-02-05T08:20:33.165619+0000","last_active":"2024-02-05T08:20:51.084232+0000","last_peered":"2024-02-05T08:20:51.084232+0000","last_clean":"2024-02-05T08:20:51.084232+0000","last_became_active":"2024-02-05T08:18:57.285362+0000","last_became_peered":"2024-02-05T08:18:57.285362+0000","last_unstale":"2024-02-05T08:20:51.084232+0000","last_undegraded":"2024-02-05T08:20:51.084232+0000","last_fullsized":"2024-02-05T08:20:51.084232+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_clean_scrub_stamp":"2024-02-05T08:20:33.165371+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:50.031969+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":102,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084512+0000","last_change":"2024-02-05T08:19:20.109021+0000","last_active":"2024-02-05T08:20:51.084512+0000","last_peered":"2024-02-05T08:20:51.084512+0000","last_clean":"2024-02-05T08:20:51.084512+0000","last_became_active":"2024-02-05T08:18:34.631974+0000","last_became_peered":"2024-02-05T08:18:34.631974+0000","last_unstale":"2024-02-05T08:20:51.084512+0000","last_undegraded":"2024-02-05T08:20:51.084512+0000","last_fullsized":"2024-02-05T08:20:51.084512+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:20.108944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:20.108944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"236'30","reported_seq":478,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.779606+0000","last_change":"2024-02-05T08:20:18.442377+0000","last_active":"2024-02-05T08:20:51.779606+0000","last_peered":"2024-02-05T08:20:51.779606+0000","last_clean":"2024-02-05T08:20:51.779606+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:20:51.779606+0000","last_undegraded":"2024-02-05T08:20:51.779606+0000","last_fullsized":"2024-02-05T08:20:51.779606+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:20:18.442244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:20:18.442244+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:40.320195+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":604,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.088788+0000","last_change":"2024-02-05T08:20:20.098080+0000","last_active":"2024-02-05T08:20:51.088788+0000","last_peered":"2024-02-05T08:20:51.088788+0000","last_clean":"2024-02-05T08:20:51.088788+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:20:51.088788+0000","last_undegraded":"2024-02-05T08:20:51.088788+0000","last_fullsized":"2024-02-05T08:20:51.088788+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:20.097959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_clean_scrub_stamp":"2024-02-05T08:20:20.097959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:30.955905+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"335'7994","reported_seq":8138,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:48.917014+0000","last_change":"2024-02-05T08:20:39.198575+0000","last_active":"2024-02-05T08:20:48.917014+0000","last_peered":"2024-02-05T08:20:48.917014+0000","last_clean":"2024-02-05T08:20:48.917014+0000","last_became_active":"2024-02-05T08:19:07.327007+0000","last_became_peered":"2024-02-05T08:19:07.327007+0000","last_unstale":"2024-02-05T08:20:48.917014+0000","last_undegraded":"2024-02-05T08:20:48.917014+0000","last_fullsized":"2024-02-05T08:20:48.917014+0000","mapping_epoch":309,"log_start":"322'2900","ondisk_log_start":"322'2900","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'7440","last_scrub_stamp":"2024-02-05T08:20:39.198303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:39.198303+0000","objects_scrubbed":7387,"log_size":5094,"log_dups_size":0,"ondisk_log_size":5094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:00.465206+0000","scrub_duration":1696,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":523894784,"num_objects":7994,"num_object_clones":0,"num_object_copies":23982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7994,"num_write_kb":511616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'130","reported_seq":582,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:50.906174+0000","last_change":"2024-02-05T08:19:44.916776+0000","last_active":"2024-02-05T08:20:50.906174+0000","last_peered":"2024-02-05T08:20:50.906174+0000","last_clean":"2024-02-05T08:12:23.979378+0000","last_became_active":"2024-02-05T08:19:44.916776+0000","last_became_peered":"2024-02-05T08:19:44.916776+0000","last_unstale":"2024-02-05T08:20:50.906174+0000","last_undegraded":"2024-02-05T08:20:50.906174+0000","last_fullsized":"2024-02-05T08:19:44.897597+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:07.594418+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:07.594418+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":543,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.026127+0000","last_change":"2024-02-05T08:19:44.903734+0000","last_active":"2024-02-05T08:20:51.026127+0000","last_peered":"2024-02-05T08:20:51.026127+0000","last_clean":"2024-02-05T08:18:29.610515+0000","last_became_active":"2024-02-05T08:19:44.903734+0000","last_became_peered":"2024-02-05T08:19:44.903734+0000","last_unstale":"2024-02-05T08:20:51.026127+0000","last_undegraded":"2024-02-05T08:20:51.026127+0000","last_fullsized":"2024-02-05T08:19:44.898755+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.646431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.646431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":612,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.719614+0000","last_change":"2024-02-05T08:20:12.695787+0000","last_active":"2024-02-05T08:20:46.719614+0000","last_peered":"2024-02-05T08:20:46.719614+0000","last_clean":"2024-02-05T08:20:46.719614+0000","last_became_active":"2024-02-05T08:18:34.635954+0000","last_became_peered":"2024-02-05T08:18:34.635954+0000","last_unstale":"2024-02-05T08:20:46.719614+0000","last_undegraded":"2024-02-05T08:20:46.719614+0000","last_fullsized":"2024-02-05T08:20:46.719614+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:12.695667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:12.695667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:37.193362+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":1593,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:52.023688+0000","last_change":"2024-02-05T08:19:44.901362+0000","last_active":"2024-02-05T08:20:52.023688+0000","last_peered":"2024-02-05T08:20:52.023688+0000","last_clean":"2024-02-05T08:19:43.389689+0000","last_became_active":"2024-02-05T08:19:44.901362+0000","last_became_peered":"2024-02-05T08:19:44.901362+0000","last_unstale":"2024-02-05T08:20:52.023688+0000","last_undegraded":"2024-02-05T08:19:44.875266+0000","last_fullsized":"2024-02-05T08:19:44.875151+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:19:14.179578+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.179578+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:56.024525+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":962,"num_read_kb":957,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":652,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.720615+0000","last_change":"2024-02-05T08:20:16.599470+0000","last_active":"2024-02-05T08:20:46.720615+0000","last_peered":"2024-02-05T08:20:46.720615+0000","last_clean":"2024-02-05T08:20:46.720615+0000","last_became_active":"2024-02-05T08:14:24.992820+0000","last_became_peered":"2024-02-05T08:14:24.992820+0000","last_unstale":"2024-02-05T08:20:46.720615+0000","last_undegraded":"2024-02-05T08:20:46.720615+0000","last_fullsized":"2024-02-05T08:20:46.720615+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:20:16.599202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:20:16.599202+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:35.823098+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":606,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.026543+0000","last_change":"2024-02-05T08:19:44.900624+0000","last_active":"2024-02-05T08:20:51.026543+0000","last_peered":"2024-02-05T08:20:51.026543+0000","last_clean":"2024-02-05T08:19:41.831836+0000","last_became_active":"2024-02-05T08:19:44.900624+0000","last_became_peered":"2024-02-05T08:19:44.900624+0000","last_unstale":"2024-02-05T08:20:51.026543+0000","last_undegraded":"2024-02-05T08:20:51.026543+0000","last_fullsized":"2024-02-05T08:19:44.899450+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:35.391824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:19:35.391824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.202567+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"336'8298","reported_seq":8408,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:52.929585+0000","last_change":"2024-02-05T08:20:41.668428+0000","last_active":"2024-02-05T08:20:52.929585+0000","last_peered":"2024-02-05T08:20:52.929585+0000","last_clean":"2024-02-05T08:20:52.929585+0000","last_became_active":"2024-02-05T08:19:07.332849+0000","last_became_peered":"2024-02-05T08:19:07.332849+0000","last_unstale":"2024-02-05T08:20:52.929585+0000","last_undegraded":"2024-02-05T08:20:52.929585+0000","last_fullsized":"2024-02-05T08:20:52.929585+0000","mapping_epoch":309,"log_start":"320'2700","ondisk_log_start":"320'2700","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":4199,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:54.560925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":526843928,"num_objects":8040,"num_object_clones":0,"num_object_copies":24120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8040,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8298,"num_write_kb":522753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'130","reported_seq":646,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.907394+0000","last_change":"2024-02-05T08:20:19.485708+0000","last_active":"2024-02-05T08:20:50.907394+0000","last_peered":"2024-02-05T08:20:50.907394+0000","last_clean":"2024-02-05T08:20:50.907394+0000","last_became_active":"2024-02-05T08:19:01.817743+0000","last_became_peered":"2024-02-05T08:19:01.817743+0000","last_unstale":"2024-02-05T08:20:50.907394+0000","last_undegraded":"2024-02-05T08:20:50.907394+0000","last_fullsized":"2024-02-05T08:20:50.907394+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:19.485585+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.485585+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:46.149289+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":310,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:50.906570+0000","last_change":"2024-02-05T08:19:44.901104+0000","last_active":"2024-02-05T08:20:50.906570+0000","last_peered":"2024-02-05T08:20:50.906570+0000","last_clean":"2024-02-05T08:12:25.219295+0000","last_became_active":"2024-02-05T08:19:44.901104+0000","last_became_peered":"2024-02-05T08:19:44.901104+0000","last_unstale":"2024-02-05T08:20:50.906570+0000","last_undegraded":"2024-02-05T08:20:50.906570+0000","last_fullsized":"2024-02-05T08:19:44.872029+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:27.862543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:27.862543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:50.911789+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":31,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:50.907432+0000","last_change":"2024-02-05T08:19:44.901166+0000","last_active":"2024-02-05T08:20:50.907432+0000","last_peered":"2024-02-05T08:20:50.907432+0000","last_clean":"2024-02-05T08:04:40.511631+0000","last_became_active":"2024-02-05T08:19:44.901166+0000","last_became_peered":"2024-02-05T08:19:44.901166+0000","last_unstale":"2024-02-05T08:20:50.907432+0000","last_undegraded":"2024-02-05T08:20:50.907432+0000","last_fullsized":"2024-02-05T08:19:44.885594+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:37.350622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.350622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":647,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.719896+0000","last_change":"2024-02-05T08:19:15.574020+0000","last_active":"2024-02-05T08:20:46.719896+0000","last_peered":"2024-02-05T08:20:46.719896+0000","last_clean":"2024-02-05T08:20:46.719896+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:20:46.719896+0000","last_undegraded":"2024-02-05T08:20:46.719896+0000","last_fullsized":"2024-02-05T08:20:46.719896+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-02-05T08:19:15.573897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.573897+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:50.479325+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":341,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.907099+0000","last_change":"2024-02-05T08:19:12.455979+0000","last_active":"2024-02-05T08:20:50.907099+0000","last_peered":"2024-02-05T08:20:50.907099+0000","last_clean":"2024-02-05T08:20:50.907099+0000","last_became_active":"2024-02-05T08:12:29.688534+0000","last_became_peered":"2024-02-05T08:12:29.688534+0000","last_unstale":"2024-02-05T08:20:50.907099+0000","last_undegraded":"2024-02-05T08:20:50.907099+0000","last_fullsized":"2024-02-05T08:20:50.907099+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:12.455899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:19:12.455899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:54.561004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"336'8321","reported_seq":8469,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:53.213347+0000","last_change":"2024-02-05T08:20:34.585093+0000","last_active":"2024-02-05T08:20:53.213347+0000","last_peered":"2024-02-05T08:20:53.213347+0000","last_clean":"2024-02-05T08:20:53.213347+0000","last_became_active":"2024-02-05T08:19:07.332176+0000","last_became_peered":"2024-02-05T08:19:07.332176+0000","last_unstale":"2024-02-05T08:20:53.213347+0000","last_undegraded":"2024-02-05T08:20:53.213347+0000","last_fullsized":"2024-02-05T08:20:53.213347+0000","mapping_epoch":309,"log_start":"323'3300","ondisk_log_start":"323'3300","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'7365","last_scrub_stamp":"2024-02-05T08:20:34.584959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:34.584959+0000","objects_scrubbed":7281,"log_size":5021,"log_dups_size":0,"ondisk_log_size":5021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:59.712909+0000","scrub_duration":2480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":529072128,"num_objects":8073,"num_object_clones":0,"num_object_copies":24219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8321,"num_write_kb":524608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"82'130","reported_seq":646,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.906636+0000","last_change":"2024-02-05T08:20:35.974697+0000","last_active":"2024-02-05T08:20:50.906636+0000","last_peered":"2024-02-05T08:20:50.906636+0000","last_clean":"2024-02-05T08:20:50.906636+0000","last_became_active":"2024-02-05T08:19:01.815446+0000","last_became_peered":"2024-02-05T08:19:01.815446+0000","last_unstale":"2024-02-05T08:20:50.906636+0000","last_undegraded":"2024-02-05T08:20:50.906636+0000","last_fullsized":"2024-02-05T08:20:50.906636+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:35.974462+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:35.974462+0000","last_clean_scrub_stamp":"2024-02-05T08:20:35.974462+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:59.595447+0000","scrub_duration":332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":600,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.719560+0000","last_change":"2024-02-05T08:19:10.488490+0000","last_active":"2024-02-05T08:20:46.719560+0000","last_peered":"2024-02-05T08:20:46.719560+0000","last_clean":"2024-02-05T08:20:46.719560+0000","last_became_active":"2024-02-05T08:18:34.636693+0000","last_became_peered":"2024-02-05T08:18:34.636693+0000","last_unstale":"2024-02-05T08:20:46.719560+0000","last_undegraded":"2024-02-05T08:20:46.719560+0000","last_fullsized":"2024-02-05T08:20:46.719560+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:10.488395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:10.488395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:50.479344+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":623,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.026262+0000","last_change":"2024-02-05T08:19:18.499832+0000","last_active":"2024-02-05T08:20:51.026262+0000","last_peered":"2024-02-05T08:20:51.026262+0000","last_clean":"2024-02-05T08:20:51.026262+0000","last_became_active":"2024-02-05T08:18:34.633263+0000","last_became_peered":"2024-02-05T08:18:34.633263+0000","last_unstale":"2024-02-05T08:20:51.026262+0000","last_undegraded":"2024-02-05T08:20:51.026262+0000","last_fullsized":"2024-02-05T08:20:51.026262+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:18.499743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:18.499743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'20","reported_seq":440,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:51.025670+0000","last_change":"2024-02-05T08:19:44.916082+0000","last_active":"2024-02-05T08:20:51.025670+0000","last_peered":"2024-02-05T08:20:51.025670+0000","last_clean":"2024-02-05T08:14:49.165927+0000","last_became_active":"2024-02-05T08:19:44.916082+0000","last_became_peered":"2024-02-05T08:19:44.916082+0000","last_unstale":"2024-02-05T08:20:51.025670+0000","last_undegraded":"2024-02-05T08:19:44.899919+0000","last_fullsized":"2024-02-05T08:19:44.899797+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:19:09.511509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:09.511509+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"277'11","reported_seq":815,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-02-05T08:20:46.720267+0000","last_change":"2024-02-05T08:19:33.688840+0000","last_active":"2024-02-05T08:20:46.720267+0000","last_peered":"2024-02-05T08:20:46.720267+0000","last_clean":"2024-02-05T08:20:46.720267+0000","last_became_active":"2024-02-05T08:12:29.683091+0000","last_became_peered":"2024-02-05T08:12:29.683091+0000","last_unstale":"2024-02-05T08:20:46.720267+0000","last_undegraded":"2024-02-05T08:20:46.720267+0000","last_fullsized":"2024-02-05T08:20:46.720267+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:19:33.688719+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_clean_scrub_stamp":"2024-02-05T08:19:33.688719+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:53.734388+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"336'8383","reported_seq":8531,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:52.927379+0000","last_change":"2024-02-05T08:20:41.146544+0000","last_active":"2024-02-05T08:20:52.927379+0000","last_peered":"2024-02-05T08:20:52.927379+0000","last_clean":"2024-02-05T08:20:52.927379+0000","last_became_active":"2024-02-05T08:19:07.346024+0000","last_became_peered":"2024-02-05T08:19:07.346024+0000","last_unstale":"2024-02-05T08:20:52.927379+0000","last_undegraded":"2024-02-05T08:20:52.927379+0000","last_fullsized":"2024-02-05T08:20:52.927379+0000","mapping_epoch":309,"log_start":"320'2800","ondisk_log_start":"320'2800","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'7704","last_scrub_stamp":"2024-02-05T08:20:41.146350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:41.146350+0000","objects_scrubbed":7648,"log_size":5583,"log_dups_size":0,"ondisk_log_size":5583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:00.018859+0000","scrub_duration":2644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532742144,"num_objects":8129,"num_object_clones":0,"num_object_copies":24387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8383,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'130","reported_seq":385,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084866+0000","last_change":"2024-02-05T08:20:27.728841+0000","last_active":"2024-02-05T08:20:51.084866+0000","last_peered":"2024-02-05T08:20:51.084866+0000","last_clean":"2024-02-05T08:20:51.084866+0000","last_became_active":"2024-02-05T08:19:01.819856+0000","last_became_peered":"2024-02-05T08:19:01.819856+0000","last_unstale":"2024-02-05T08:20:51.084866+0000","last_undegraded":"2024-02-05T08:20:51.084866+0000","last_fullsized":"2024-02-05T08:20:51.084866+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:27.728717+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:27.728717+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:56.586217+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":612,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.088375+0000","last_change":"2024-02-05T08:20:19.251720+0000","last_active":"2024-02-05T08:20:51.088375+0000","last_peered":"2024-02-05T08:20:51.088375+0000","last_clean":"2024-02-05T08:20:51.088375+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:20:51.088375+0000","last_undegraded":"2024-02-05T08:20:51.088375+0000","last_fullsized":"2024-02-05T08:20:51.088375+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:19.251464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.251464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:40.244202+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":614,"reported_epoch":335,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:46.720702+0000","last_change":"2024-02-05T08:19:44.892743+0000","last_active":"2024-02-05T08:20:46.720702+0000","last_peered":"2024-02-05T08:20:46.720702+0000","last_clean":"2024-02-05T08:19:41.834750+0000","last_became_active":"2024-02-05T08:19:44.892743+0000","last_became_peered":"2024-02-05T08:19:44.892743+0000","last_unstale":"2024-02-05T08:20:46.720702+0000","last_undegraded":"2024-02-05T08:19:44.887459+0000","last_fullsized":"2024-02-05T08:19:44.887352+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:19:11.483052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:11.483052+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:50.479301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"236'28","reported_seq":685,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.903794+0000","last_change":"2024-02-05T08:19:30.389683+0000","last_active":"2024-02-05T08:20:50.903794+0000","last_peered":"2024-02-05T08:20:50.903794+0000","last_clean":"2024-02-05T08:20:50.903794+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:20:50.903794+0000","last_undegraded":"2024-02-05T08:20:50.903794+0000","last_fullsized":"2024-02-05T08:20:50.903794+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:19:30.389554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:30.389554+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:52.595376+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"325'12","reported_seq":139,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:51.084811+0000","last_change":"2024-02-05T08:19:44.915079+0000","last_active":"2024-02-05T08:20:51.084811+0000","last_peered":"2024-02-05T08:20:51.084811+0000","last_clean":"2024-02-05T08:19:41.828253+0000","last_became_active":"2024-02-05T08:19:44.915079+0000","last_became_peered":"2024-02-05T08:19:44.915079+0000","last_unstale":"2024-02-05T08:20:51.084811+0000","last_undegraded":"2024-02-05T08:19:44.888020+0000","last_fullsized":"2024-02-05T08:19:44.887894+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'11","last_scrub_stamp":"2024-02-05T08:19:34.968809+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:19:34.968809+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:56.732610+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"336'10384","reported_seq":10566,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:52.585784+0000","last_change":"2024-02-05T08:20:51.167853+0000","last_active":"2024-02-05T08:20:52.585784+0000","last_peered":"2024-02-05T08:20:52.585784+0000","last_clean":"2024-02-05T08:20:52.585784+0000","last_became_active":"2024-02-05T08:19:07.331338+0000","last_became_peered":"2024-02-05T08:19:07.331338+0000","last_unstale":"2024-02-05T08:20:52.585784+0000","last_undegraded":"2024-02-05T08:20:52.585784+0000","last_fullsized":"2024-02-05T08:20:52.585784+0000","mapping_epoch":309,"log_start":"327'6002","ondisk_log_start":"327'6002","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":2817,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:55.914443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":530513920,"num_objects":8095,"num_object_clones":0,"num_object_copies":24285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8301,"num_write_kb":524672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'130","reported_seq":377,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.904229+0000","last_change":"2024-02-05T08:19:08.403635+0000","last_active":"2024-02-05T08:20:50.904229+0000","last_peered":"2024-02-05T08:20:50.904229+0000","last_clean":"2024-02-05T08:20:50.904229+0000","last_became_active":"2024-02-05T08:19:01.816923+0000","last_became_peered":"2024-02-05T08:19:01.816923+0000","last_unstale":"2024-02-05T08:20:50.904229+0000","last_undegraded":"2024-02-05T08:20:50.904229+0000","last_fullsized":"2024-02-05T08:20:50.904229+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:08.403509+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:08.403509+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":619,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.904022+0000","last_change":"2024-02-05T08:20:36.100335+0000","last_active":"2024-02-05T08:20:50.904022+0000","last_peered":"2024-02-05T08:20:50.904022+0000","last_clean":"2024-02-05T08:20:50.904022+0000","last_became_active":"2024-02-05T08:18:34.635908+0000","last_became_peered":"2024-02-05T08:18:34.635908+0000","last_unstale":"2024-02-05T08:20:50.904022+0000","last_undegraded":"2024-02-05T08:20:50.904022+0000","last_fullsized":"2024-02-05T08:20:50.904022+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:36.100208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:36.100208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:04.352882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":106,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.085483+0000","last_change":"2024-02-05T08:19:38.969220+0000","last_active":"2024-02-05T08:20:51.085483+0000","last_peered":"2024-02-05T08:20:51.085483+0000","last_clean":"2024-02-05T08:20:51.085483+0000","last_became_active":"2024-02-05T08:18:34.649808+0000","last_became_peered":"2024-02-05T08:18:34.649808+0000","last_unstale":"2024-02-05T08:20:51.085483+0000","last_undegraded":"2024-02-05T08:20:51.085483+0000","last_fullsized":"2024-02-05T08:20:51.085483+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:19:38.968895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:38.968895+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"236'38","reported_seq":128,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084260+0000","last_change":"2024-02-05T08:20:37.207904+0000","last_active":"2024-02-05T08:20:51.084260+0000","last_peered":"2024-02-05T08:20:51.084260+0000","last_clean":"2024-02-05T08:20:51.084260+0000","last_became_active":"2024-02-05T08:18:34.653505+0000","last_became_peered":"2024-02-05T08:18:34.653505+0000","last_unstale":"2024-02-05T08:20:51.084260+0000","last_undegraded":"2024-02-05T08:20:51.084260+0000","last_fullsized":"2024-02-05T08:20:51.084260+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'38","last_scrub_stamp":"2024-02-05T08:20:37.207762+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.207762+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:54.648572+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":615,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.903676+0000","last_change":"2024-02-05T08:20:31.133433+0000","last_active":"2024-02-05T08:20:50.903676+0000","last_peered":"2024-02-05T08:20:50.903676+0000","last_clean":"2024-02-05T08:20:50.903676+0000","last_became_active":"2024-02-05T08:18:34.636667+0000","last_became_peered":"2024-02-05T08:18:34.636667+0000","last_unstale":"2024-02-05T08:20:50.903676+0000","last_undegraded":"2024-02-05T08:20:50.903676+0000","last_fullsized":"2024-02-05T08:20:50.903676+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:31.133206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:31.133206+0000","last_clean_scrub_stamp":"2024-02-05T08:20:31.133206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:00.885317+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"336'8375","reported_seq":8521,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:52.657061+0000","last_change":"2024-02-05T08:20:37.390439+0000","last_active":"2024-02-05T08:20:52.657061+0000","last_peered":"2024-02-05T08:20:52.657061+0000","last_clean":"2024-02-05T08:20:52.657061+0000","last_became_active":"2024-02-05T08:19:07.330566+0000","last_became_peered":"2024-02-05T08:19:07.330566+0000","last_unstale":"2024-02-05T08:20:52.657061+0000","last_undegraded":"2024-02-05T08:20:52.657061+0000","last_fullsized":"2024-02-05T08:20:52.657061+0000","mapping_epoch":309,"log_start":"325'4000","ondisk_log_start":"325'4000","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'7463","last_scrub_stamp":"2024-02-05T08:20:37.390208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.390208+0000","objects_scrubbed":7384,"log_size":4375,"log_dups_size":0,"ondisk_log_size":4375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:50.900137+0000","scrub_duration":2278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534052864,"num_objects":8149,"num_object_clones":0,"num_object_copies":24447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8375,"num_write_kb":528768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'130","reported_seq":385,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.025392+0000","last_change":"2024-02-05T08:20:34.301845+0000","last_active":"2024-02-05T08:20:51.025392+0000","last_peered":"2024-02-05T08:20:51.025392+0000","last_clean":"2024-02-05T08:20:51.025392+0000","last_became_active":"2024-02-05T08:19:01.817169+0000","last_became_peered":"2024-02-05T08:19:01.817169+0000","last_unstale":"2024-02-05T08:20:51.025392+0000","last_undegraded":"2024-02-05T08:20:51.025392+0000","last_fullsized":"2024-02-05T08:20:51.025392+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:34.301740+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:34.301740+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:36.803503+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":602,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.025922+0000","last_change":"2024-02-05T08:19:44.900438+0000","last_active":"2024-02-05T08:20:51.025922+0000","last_peered":"2024-02-05T08:20:51.025922+0000","last_clean":"2024-02-05T08:19:41.831465+0000","last_became_active":"2024-02-05T08:19:44.900438+0000","last_became_peered":"2024-02-05T08:19:44.900438+0000","last_unstale":"2024-02-05T08:20:51.025922+0000","last_undegraded":"2024-02-05T08:20:51.025922+0000","last_fullsized":"2024-02-05T08:19:44.898161+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.613476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.613476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":806,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.912245+0000","last_change":"2024-02-05T08:20:38.198413+0000","last_active":"2024-02-05T08:20:50.912245+0000","last_peered":"2024-02-05T08:20:50.912245+0000","last_clean":"2024-02-05T08:20:50.912245+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:20:50.912245+0000","last_undegraded":"2024-02-05T08:20:50.912245+0000","last_fullsized":"2024-02-05T08:20:50.912245+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:20:38.198278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:38.198278+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:58.293117+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":105,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:51.084373+0000","last_change":"2024-02-05T08:19:44.902070+0000","last_active":"2024-02-05T08:20:51.084373+0000","last_peered":"2024-02-05T08:20:51.084373+0000","last_clean":"2024-02-05T08:19:41.828067+0000","last_became_active":"2024-02-05T08:19:44.902070+0000","last_became_peered":"2024-02-05T08:19:44.902070+0000","last_unstale":"2024-02-05T08:20:51.084373+0000","last_undegraded":"2024-02-05T08:20:51.084373+0000","last_fullsized":"2024-02-05T08:19:44.887805+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:07.027521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:07.027521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"236'27","reported_seq":610,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:51.085116+0000","last_change":"2024-02-05T08:19:44.912476+0000","last_active":"2024-02-05T08:20:51.085116+0000","last_peered":"2024-02-05T08:20:51.085116+0000","last_clean":"2024-02-05T08:18:29.610409+0000","last_became_active":"2024-02-05T08:19:44.912476+0000","last_became_peered":"2024-02-05T08:19:44.912476+0000","last_unstale":"2024-02-05T08:20:51.085116+0000","last_undegraded":"2024-02-05T08:19:44.888218+0000","last_fullsized":"2024-02-05T08:19:44.888087+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:18:40.384115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:40.384115+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"336'8306","reported_seq":8373,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:52.910840+0000","last_change":"2024-02-05T08:19:44.913164+0000","last_active":"2024-02-05T08:20:52.910840+0000","last_peered":"2024-02-05T08:20:52.910840+0000","last_clean":"2024-02-05T08:19:43.862508+0000","last_became_active":"2024-02-05T08:19:44.913164+0000","last_became_peered":"2024-02-05T08:19:44.913164+0000","last_unstale":"2024-02-05T08:20:52.910840+0000","last_undegraded":"2024-02-05T08:19:44.871956+0000","last_fullsized":"2024-02-05T08:19:44.871842+0000","mapping_epoch":324,"log_start":"320'2700","ondisk_log_start":"320'2700","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":5606,"log_dups_size":0,"ondisk_log_size":5606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:54.560682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":527171584,"num_objects":8044,"num_object_clones":0,"num_object_copies":24132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8306,"num_write_kb":523200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":8044}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":110,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.085338+0000","last_change":"2024-02-05T08:20:38.715971+0000","last_active":"2024-02-05T08:20:51.085338+0000","last_peered":"2024-02-05T08:20:51.085338+0000","last_clean":"2024-02-05T08:20:51.085338+0000","last_became_active":"2024-02-05T08:18:34.627261+0000","last_became_peered":"2024-02-05T08:18:34.627261+0000","last_unstale":"2024-02-05T08:20:51.085338+0000","last_undegraded":"2024-02-05T08:20:51.085338+0000","last_fullsized":"2024-02-05T08:20:51.085338+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:38.715837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:20:38.715837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:00.975975+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":623,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.026518+0000","last_change":"2024-02-05T08:20:30.366634+0000","last_active":"2024-02-05T08:20:51.026518+0000","last_peered":"2024-02-05T08:20:51.026518+0000","last_clean":"2024-02-05T08:20:51.026518+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:20:51.026518+0000","last_undegraded":"2024-02-05T08:20:51.026518+0000","last_fullsized":"2024-02-05T08:20:51.026518+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:30.366511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:30.366511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:51.599279+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":611,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.903697+0000","last_change":"2024-02-05T08:19:13.275219+0000","last_active":"2024-02-05T08:20:50.903697+0000","last_peered":"2024-02-05T08:20:50.903697+0000","last_clean":"2024-02-05T08:20:50.903697+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:20:50.903697+0000","last_undegraded":"2024-02-05T08:20:50.903697+0000","last_fullsized":"2024-02-05T08:20:50.903697+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:13.275109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:19:13.275109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"236'19","reported_seq":669,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:50.912102+0000","last_change":"2024-02-05T08:19:44.926373+0000","last_active":"2024-02-05T08:20:50.912102+0000","last_peered":"2024-02-05T08:20:50.912102+0000","last_clean":"2024-02-05T08:19:41.831642+0000","last_became_active":"2024-02-05T08:19:44.926373+0000","last_became_peered":"2024-02-05T08:19:44.926373+0000","last_unstale":"2024-02-05T08:20:50.912102+0000","last_undegraded":"2024-02-05T08:19:44.921915+0000","last_fullsized":"2024-02-05T08:19:44.916232+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:36.561837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:36.561837+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"336'8464","reported_seq":8613,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:53.207818+0000","last_change":"2024-02-05T08:20:36.341229+0000","last_active":"2024-02-05T08:20:53.207818+0000","last_peered":"2024-02-05T08:20:53.207818+0000","last_clean":"2024-02-05T08:20:53.207818+0000","last_became_active":"2024-02-05T08:19:07.346034+0000","last_became_peered":"2024-02-05T08:19:07.346034+0000","last_unstale":"2024-02-05T08:20:53.207818+0000","last_undegraded":"2024-02-05T08:20:53.207818+0000","last_fullsized":"2024-02-05T08:20:53.207818+0000","mapping_epoch":309,"log_start":"320'2400","ondisk_log_start":"320'2400","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'7452","last_scrub_stamp":"2024-02-05T08:20:36.340902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:36.340902+0000","objects_scrubbed":7382,"log_size":6064,"log_dups_size":0,"ondisk_log_size":6064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:46.511007+0000","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":531496960,"num_objects":8110,"num_object_clones":0,"num_object_copies":24330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8464,"num_write_kb":530368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'130","reported_seq":377,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.025274+0000","last_change":"2024-02-05T08:19:10.494455+0000","last_active":"2024-02-05T08:20:51.025274+0000","last_peered":"2024-02-05T08:20:51.025274+0000","last_clean":"2024-02-05T08:20:51.025274+0000","last_became_active":"2024-02-05T08:19:01.817126+0000","last_became_peered":"2024-02-05T08:19:01.817126+0000","last_unstale":"2024-02-05T08:20:51.025274+0000","last_undegraded":"2024-02-05T08:20:51.025274+0000","last_fullsized":"2024-02-05T08:20:51.025274+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:10.494376+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:10.494376+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":636,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.907815+0000","last_change":"2024-02-05T08:20:19.087445+0000","last_active":"2024-02-05T08:20:50.907815+0000","last_peered":"2024-02-05T08:20:50.907815+0000","last_clean":"2024-02-05T08:20:50.907815+0000","last_became_active":"2024-02-05T08:18:34.645067+0000","last_became_peered":"2024-02-05T08:18:34.645067+0000","last_unstale":"2024-02-05T08:20:50.907815+0000","last_undegraded":"2024-02-05T08:20:50.907815+0000","last_fullsized":"2024-02-05T08:20:50.907815+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:20:19.087309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.087309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:44.997318+0000","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":32,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:50.904734+0000","last_change":"2024-02-05T08:19:44.914853+0000","last_active":"2024-02-05T08:20:50.904734+0000","last_peered":"2024-02-05T08:20:50.904734+0000","last_clean":"2024-02-05T08:04:44.536163+0000","last_became_active":"2024-02-05T08:19:44.914853+0000","last_became_peered":"2024-02-05T08:19:44.914853+0000","last_unstale":"2024-02-05T08:20:50.904734+0000","last_undegraded":"2024-02-05T08:20:50.904734+0000","last_fullsized":"2024-02-05T08:19:44.909093+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:45.429303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:45.429303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":343,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:50.906656+0000","last_change":"2024-02-05T08:19:44.915255+0000","last_active":"2024-02-05T08:20:50.906656+0000","last_peered":"2024-02-05T08:20:50.906656+0000","last_clean":"2024-02-05T08:19:41.829592+0000","last_became_active":"2024-02-05T08:19:44.915255+0000","last_became_peered":"2024-02-05T08:19:44.915255+0000","last_unstale":"2024-02-05T08:20:50.906656+0000","last_undegraded":"2024-02-05T08:20:50.906656+0000","last_fullsized":"2024-02-05T08:19:44.872335+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:10.372287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:19:10.372287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:54.560975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"236'31","reported_seq":651,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:50.904539+0000","last_change":"2024-02-05T08:19:44.922652+0000","last_active":"2024-02-05T08:20:50.904539+0000","last_peered":"2024-02-05T08:20:50.904539+0000","last_clean":"2024-02-05T08:19:41.830877+0000","last_became_active":"2024-02-05T08:19:44.922652+0000","last_became_peered":"2024-02-05T08:19:44.922652+0000","last_unstale":"2024-02-05T08:20:50.904539+0000","last_undegraded":"2024-02-05T08:19:44.913741+0000","last_fullsized":"2024-02-05T08:19:44.913632+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'31","last_scrub_stamp":"2024-02-05T08:19:24.342770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:24.342770+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:51.983361+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"336'8398","reported_seq":8465,"reported_epoch":336,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:20:52.631132+0000","last_change":"2024-02-05T08:19:44.915552+0000","last_active":"2024-02-05T08:20:52.631132+0000","last_peered":"2024-02-05T08:20:52.631132+0000","last_clean":"2024-02-05T08:19:43.841958+0000","last_became_active":"2024-02-05T08:19:44.915552+0000","last_became_peered":"2024-02-05T08:19:44.915552+0000","last_unstale":"2024-02-05T08:20:52.631132+0000","last_undegraded":"2024-02-05T08:19:44.912000+0000","last_fullsized":"2024-02-05T08:19:44.910014+0000","mapping_epoch":324,"log_start":"325'4000","ondisk_log_start":"325'4000","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":4398,"log_dups_size":0,"ondisk_log_size":4398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:56.051746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":534118400,"num_objects":8150,"num_object_clones":0,"num_object_copies":24450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8150,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8398,"num_write_kb":529536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":8150}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"82'130","reported_seq":909,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084176+0000","last_change":"2024-02-05T08:19:09.153997+0000","last_active":"2024-02-05T08:20:51.084176+0000","last_peered":"2024-02-05T08:20:51.084176+0000","last_clean":"2024-02-05T08:20:51.084176+0000","last_became_active":"2024-02-05T08:19:01.816744+0000","last_became_peered":"2024-02-05T08:19:01.816744+0000","last_unstale":"2024-02-05T08:20:51.084176+0000","last_undegraded":"2024-02-05T08:20:51.084176+0000","last_fullsized":"2024-02-05T08:20:51.084176+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:09.153860+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:09.153860+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:20:56.024510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":594,"reported_epoch":336,"state":"active+undersized","last_fresh":"2024-02-05T08:20:50.912236+0000","last_change":"2024-02-05T08:19:44.927513+0000","last_active":"2024-02-05T08:20:50.912236+0000","last_peered":"2024-02-05T08:20:50.912236+0000","last_clean":"2024-02-05T08:19:41.831929+0000","last_became_active":"2024-02-05T08:19:44.927513+0000","last_became_peered":"2024-02-05T08:19:44.927513+0000","last_unstale":"2024-02-05T08:20:50.912236+0000","last_undegraded":"2024-02-05T08:20:50.912236+0000","last_fullsized":"2024-02-05T08:19:44.916139+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.282641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.282641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":823,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.907579+0000","last_change":"2024-02-05T08:20:15.874649+0000","last_active":"2024-02-05T08:20:50.907579+0000","last_peered":"2024-02-05T08:20:50.907579+0000","last_clean":"2024-02-05T08:20:50.907579+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:20:50.907579+0000","last_undegraded":"2024-02-05T08:20:50.907579+0000","last_fullsized":"2024-02-05T08:20:50.907579+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:20:15.874421+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:20:15.874421+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:42.693872+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":110,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:51.084991+0000","last_change":"2024-02-05T08:20:20.712863+0000","last_active":"2024-02-05T08:20:51.084991+0000","last_peered":"2024-02-05T08:20:51.084991+0000","last_clean":"2024-02-05T08:20:51.084991+0000","last_became_active":"2024-02-05T08:18:34.663738+0000","last_became_peered":"2024-02-05T08:18:34.663738+0000","last_unstale":"2024-02-05T08:20:51.084991+0000","last_undegraded":"2024-02-05T08:20:51.084991+0000","last_fullsized":"2024-02-05T08:20:51.084991+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:20.711675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:20.711675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:45.175334+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"236'20","reported_seq":709,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-02-05T08:20:50.904778+0000","last_change":"2024-02-05T08:20:12.117923+0000","last_active":"2024-02-05T08:20:50.904778+0000","last_peered":"2024-02-05T08:20:50.904778+0000","last_clean":"2024-02-05T08:20:50.904778+0000","last_became_active":"2024-02-05T08:18:34.637413+0000","last_became_peered":"2024-02-05T08:18:34.637413+0000","last_unstale":"2024-02-05T08:20:50.904778+0000","last_undegraded":"2024-02-05T08:20:50.904778+0000","last_fullsized":"2024-02-05T08:20:50.904778+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:20:12.117355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:20:12.117355+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:16.064157+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":8485404696,"num_objects":129478,"num_object_clones":0,"num_object_copies":388434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129478,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133150,"num_write_kb":8404033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24517615616,"data_stored":24517607472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84431,"ondisk_log_size":84431,"up":45,"acting":45,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":962,"num_read_kb":957,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":7,"acting":7,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":84,"acting":84,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":5462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":766,"ondisk_log_size":766,"up":81,"acting":81,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":82,"acting":82,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":85,"acting":85,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037959,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2628648,"kb_used_data":2069916,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":91116504,"statfs":{"total":95995035648,"available":93303300096,"internally_reserved":0,"allocated":2119593984,"data_stored":2119293414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":572127690},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":1.206,"15min":1.518},"min":{"1min":0.26600000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":1.54,"5min":27.376000000000001,"15min":60.127000000000002},"last":0.875},{"interface":"front","average":{"1min":0.81000000000000005,"5min":1.4510000000000001,"15min":1.6080000000000001},"min":{"1min":0.45100000000000001,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":2.609,"5min":27.456,"15min":58.283000000000001},"last":0.81100000000000005}]},{"osd":1,"last update":"Mon Feb 5 08:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.544,"15min":1.512},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.22900000000000001},"max":{"1min":2.7810000000000001,"5min":27.518999999999998,"15min":38.399999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.083,"5min":1.4330000000000001,"15min":1.3680000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.22500000000000001},"max":{"1min":4.2409999999999997,"5min":27.503,"15min":38.195999999999998},"last":0.32200000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.995,"15min":0.98899999999999999},"min":{"1min":0.32200000000000001,"5min":0.26100000000000001,"15min":0.24099999999999999},"max":{"1min":1.109,"5min":7.5709999999999997,"15min":7.6779999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.125,"15min":1.115},"min":{"1min":0.35699999999999998,"5min":0.26200000000000001,"15min":0.24299999999999999},"max":{"1min":3.4020000000000001,"5min":5.9400000000000004,"15min":8.9100000000000001},"last":0.48699999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":1.115,"15min":1.321},"min":{"1min":0.39000000000000001,"5min":0.29699999999999999,"15min":0.249},"max":{"1min":1.7749999999999999,"5min":5.984,"15min":58.515000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":1.085,"15min":1.3049999999999999},"min":{"1min":0.432,"5min":0.314,"15min":0.27000000000000002},"max":{"1min":1.234,"5min":5.1040000000000001,"15min":58.225999999999999},"last":0.78000000000000003}]},{"osd":4,"last update":"Mon Feb 5 08:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.1020000000000001,"15min":0.93100000000000005},"min":{"1min":0.30399999999999999,"5min":0.24099999999999999,"15min":0.22},"max":{"1min":1.5309999999999999,"5min":7.6340000000000003,"15min":49.601999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.1619999999999999,"15min":0.97299999999999998},"min":{"1min":0.33400000000000002,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":7.835,"5min":7.8399999999999999,"15min":49.795999999999999},"last":0.39100000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":1.343,"15min":1.016},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.24299999999999999},"max":{"1min":1.1850000000000001,"5min":8.0709999999999997,"15min":49.914000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":1.4950000000000001,"15min":1.0389999999999999},"min":{"1min":0.31900000000000001,"5min":0.308,"15min":0.27000000000000002},"max":{"1min":1.3819999999999999,"5min":9.4459999999999997,"15min":52.256},"last":0.44900000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:19:57 2024","interfaces":[{"interface":"back","average":{"1min":5.6529999999999996,"5min":2.1720000000000002,"15min":1.1339999999999999},"min":{"1min":0.36199999999999999,"5min":0.28799999999999998,"15min":0.28599999999999998},"max":{"1min":84.254999999999995,"5min":84.254999999999995,"15min":84.254999999999995},"last":2.1970000000000001},{"interface":"front","average":{"1min":5.8689999999999998,"5min":2.2080000000000002,"15min":1.1240000000000001},"min":{"1min":0.373,"5min":0.32500000000000001,"15min":0.27300000000000002},"max":{"1min":84.667000000000002,"5min":84.667000000000002,"15min":84.667000000000002},"last":0.36499999999999999}]}]},{"osd":6,"up_from":51,"seq":219043332301,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4058264,"kb_used_data":3625548,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":89686888,"statfs":{"total":95995035648,"available":91839373312,"internally_reserved":0,"allocated":3712561152,"data_stored":3712275463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":442759622},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.7549999999999999,"15min":1.7769999999999999},"min":{"1min":0.33200000000000002,"5min":0.215,"15min":0.215},"max":{"1min":8.2560000000000002,"5min":76.123000000000005,"15min":102.819},"last":1.8220000000000001},{"interface":"front","average":{"1min":1.206,"5min":1.6779999999999999,"15min":1.829},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":2.9809999999999999,"5min":76.037000000000006,"15min":102.75},"last":1.1499999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.90900000000000003,"15min":1.169},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.22700000000000001},"max":{"1min":1.1970000000000001,"5min":6.0229999999999997,"15min":23.773},"last":0.503},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.96299999999999997,"15min":1.1319999999999999},"min":{"1min":0.30299999999999999,"5min":0.26300000000000001,"15min":0.23400000000000001},"max":{"1min":3.0249999999999999,"5min":5.0209999999999999,"15min":23.719000000000001},"last":0.73699999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":2.0150000000000001,"5min":1.143,"15min":1.159},"min":{"1min":0.32300000000000001,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":7.3499999999999996,"5min":7.3499999999999996,"15min":22.503},"last":0.32000000000000001},{"interface":"front","average":{"1min":1.321,"5min":1.054,"15min":1.0589999999999999},"min":{"1min":0.28199999999999997,"5min":0.27300000000000002,"15min":0.23400000000000001},"max":{"1min":6.9720000000000004,"5min":6.9720000000000004,"15min":22.466000000000001},"last":0.38200000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.83499999999999996,"15min":1.3400000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":1.546,"5min":3.4689999999999999,"15min":102.438},"last":1.869},{"interface":"front","average":{"1min":1.615,"5min":1.1100000000000001,"15min":1.419},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29099999999999998},"max":{"1min":7.4000000000000004,"5min":7.4000000000000004,"15min":102.288},"last":1.4259999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1120000000000001,"15min":0.73999999999999999},"min":{"1min":0.30299999999999999,"5min":0.251,"15min":0.223},"max":{"1min":3.8780000000000001,"5min":4.9740000000000002,"15min":9.8970000000000002},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.679,"5min":1.143,"15min":0.79400000000000004},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":5.3789999999999996,"5min":5.3789999999999996,"15min":5.3789999999999996},"last":1.9330000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.5509999999999999,"15min":1.0900000000000001},"min":{"1min":0.33200000000000002,"5min":0.29299999999999998,"15min":0.24199999999999999},"max":{"1min":3.8079999999999998,"5min":24.675999999999998,"15min":43.643000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.204,"5min":1.452,"15min":1.075},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.7490000000000001,"5min":19.132000000000001,"15min":43.893000000000001},"last":1.9610000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:20:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":1.5900000000000001,"15min":0.93000000000000005},"min":{"1min":0.40500000000000003,"5min":0.376,"15min":0.29999999999999999},"max":{"1min":2.6720000000000002,"5min":24.728999999999999,"15min":24.728999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.774,"15min":0.996},"min":{"1min":0.375,"5min":0.32200000000000001,"15min":0.29299999999999998},"max":{"1min":5.4980000000000002,"5min":21.91,"15min":21.91},"last":1.8979999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1172192,"kb_used_data":617624,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92572960,"statfs":{"total":95995035648,"available":94794711040,"internally_reserved":0,"allocated":632446976,"data_stored":632002797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1659999999999999,"15min":1.625},"min":{"1min":0.221,"5min":0.183,"15min":0.183},"max":{"1min":1.0660000000000001,"5min":36.462000000000003,"15min":86.105000000000004},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.331,"15min":1.907},"min":{"1min":0.26300000000000001,"5min":0.21099999999999999,"15min":0.20699999999999999},"max":{"1min":8.0340000000000007,"5min":35.878,"15min":85.665000000000006},"last":0.40600000000000003}]},{"osd":2,"last update":"Mon Feb 5 08:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.84299999999999997,"15min":1.373},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.191},"max":{"1min":1.4930000000000001,"5min":4.2859999999999996,"15min":49.030999999999999},"last":2.855},{"interface":"front","average":{"1min":1.2210000000000001,"5min":0.92100000000000004,"15min":1.599},"min":{"1min":0.255,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":7.298,"5min":7.298,"15min":48.984000000000002},"last":2.5550000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.754,"15min":1.5189999999999999},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.20599999999999999},"max":{"1min":1.026,"5min":5.3289999999999997,"15min":49.067999999999998},"last":1.252},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.81699999999999995,"15min":1.46},"min":{"1min":0.32100000000000001,"5min":0.254,"15min":0.24199999999999999},"max":{"1min":2.3119999999999998,"5min":4.3529999999999998,"15min":48.734999999999999},"last":1.212}]},{"osd":4,"last update":"Mon Feb 5 08:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.4119999999999999,"15min":1.272},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.26000000000000001},"max":{"1min":1.2030000000000001,"5min":24.196999999999999,"15min":25.952000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":1.3100000000000001,"15min":1.3600000000000001},"min":{"1min":0.33700000000000002,"5min":0.31,"15min":0.27100000000000002},"max":{"1min":0.92100000000000004,"5min":24.215,"15min":25.922000000000001},"last":1.228}]},{"osd":5,"last update":"Mon Feb 5 08:18:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.492,"15min":1.3999999999999999},"min":{"1min":0.40100000000000002,"5min":0.39400000000000002,"15min":0.28699999999999998},"max":{"1min":3.1699999999999999,"5min":9.6820000000000004,"15min":54.552999999999997},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.296,"15min":1.4379999999999999},"min":{"1min":0.32000000000000001,"5min":0.311,"15min":0.28999999999999998},"max":{"1min":4.1239999999999997,"5min":6.0259999999999998,"15min":55.811},"last":1.1259999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.97399999999999998,"15min":1.2390000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.5069999999999999,"5min":3.6819999999999999,"15min":54.384},"last":1.196},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.0920000000000001,"15min":1.2470000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.34899999999999998},"max":{"1min":1.456,"5min":4.4370000000000003,"15min":55.210999999999999},"last":0.56399999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:18:52 2024","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.45,"15min":1.3089999999999999},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":5.7640000000000002,"5min":6.0579999999999998,"15min":13.749000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.3280000000000001,"15min":1.242},"min":{"1min":0.42399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":2.4020000000000001,"5min":4.6180000000000003,"15min":7.9640000000000004},"last":1.1579999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738602,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5818332,"kb_used_data":5187412,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":87926820,"statfs":{"total":95995035648,"available":90037063680,"internally_reserved":0,"allocated":5311909888,"data_stored":5311609027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.72199999999999998,"15min":1.79},"min":{"1min":0.185,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":1.4279999999999999,"5min":7.9580000000000002,"15min":66.772999999999996},"last":0.755},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.745,"15min":1.5409999999999999},"min":{"1min":0.27100000000000002,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":2.169,"5min":6.569,"15min":46.198999999999998},"last":1.016}]},{"osd":2,"last update":"Mon Feb 5 08:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.77500000000000002,"15min":1.542},"min":{"1min":0.26500000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.0149999999999997,"5min":6.7050000000000001,"15min":46.170999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.72099999999999997,"15min":1.671},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":2.7879999999999998,"5min":3.1360000000000001,"15min":46.543999999999997},"last":0.28399999999999997}]},{"osd":3,"last update":"Mon Feb 5 08:20:27 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":0.87,"15min":1.7549999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":5.9029999999999996,"5min":6.7439999999999998,"15min":67.888000000000005},"last":0.248},{"interface":"front","average":{"1min":1.2070000000000001,"5min":0.83299999999999996,"15min":2.04},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.22600000000000001},"max":{"1min":3.7749999999999999,"5min":3.7749999999999999,"15min":66.891999999999996},"last":0.376}]},{"osd":4,"last update":"Mon Feb 5 08:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":1.1040000000000001,"15min":1.2529999999999999},"min":{"1min":0.318,"5min":0.29299999999999998,"15min":0.26300000000000001},"max":{"1min":3.0680000000000001,"5min":22.920000000000002,"15min":22.920000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.2549999999999999,"15min":1.248},"min":{"1min":0.34499999999999997,"5min":0.33300000000000002,"15min":0.26900000000000002},"max":{"1min":4.5979999999999999,"5min":22.983000000000001,"15min":22.983000000000001},"last":0.35499999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.716,"15min":1.49},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.28299999999999997},"max":{"1min":1.992,"5min":38.979999999999997,"15min":38.979999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.7889999999999999,"15min":1.4450000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":6.0780000000000003,"5min":38.960999999999999,"15min":38.960999999999999},"last":0.46999999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:20:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.1759999999999999,"15min":1.24},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.30199999999999999},"max":{"1min":6.1109999999999998,"5min":6.9749999999999996,"15min":12.967000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.133,"15min":1.304},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.2509999999999999,"5min":4.1210000000000004,"15min":15.721},"last":0.623}]},{"osd":7,"last update":"Mon Feb 5 08:19:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":2.359,"15min":1.681},"min":{"1min":0.38700000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.27,"5min":68.241,"15min":68.241},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":2.4420000000000002,"15min":1.7310000000000001},"min":{"1min":0.45100000000000001,"5min":0.40400000000000003,"15min":0.33500000000000002},"max":{"1min":3.956,"5min":68.980000000000004,"15min":68.980000000000004},"last":0.501}]}]},{"osd":2,"up_from":18,"seq":77309411553,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3684396,"kb_used_data":3121444,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":90060756,"statfs":{"total":95995035648,"available":92222214144,"internally_reserved":0,"allocated":3196358656,"data_stored":3196049634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":576387526},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":2.2480000000000002,"15min":2.165},"min":{"1min":0.20000000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":1.913,"5min":107.155,"15min":107.155},"last":1.2609999999999999},{"interface":"front","average":{"1min":0.90800000000000003,"5min":2.3929999999999998,"15min":2.3370000000000002},"min":{"1min":0.25600000000000001,"5min":0.22600000000000001,"15min":0.20799999999999999},"max":{"1min":3.0990000000000002,"5min":107.221,"15min":107.221},"last":0.45000000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:18:54 2024","interfaces":[{"interface":"back","average":{"1min":2.4540000000000002,"5min":1.167,"15min":1.5860000000000001},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.17999999999999999},"max":{"1min":35.695999999999998,"5min":35.695999999999998,"15min":55.942},"last":0.28299999999999997},{"interface":"front","average":{"1min":2.9209999999999998,"5min":1.581,"15min":1.752},"min":{"1min":0.216,"5min":0.216,"15min":0.193},"max":{"1min":30.765000000000001,"5min":30.765000000000001,"15min":55.960999999999999},"last":0.79700000000000004}]},{"osd":3,"last update":"Mon Feb 5 08:20:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":0.90200000000000002,"15min":1.835},"min":{"1min":0.25800000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":6.5039999999999996,"5min":8.3629999999999995,"15min":68.108000000000004},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.242,"5min":0.75600000000000001,"15min":1.6379999999999999},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":6.101,"5min":6.6920000000000002,"15min":67.945999999999998},"last":2.7559999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:20:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.1850000000000001,"15min":1.448},"min":{"1min":0.36599999999999999,"5min":0.28100000000000003,"15min":0.27400000000000002},"max":{"1min":7.6189999999999998,"5min":8.2620000000000005,"15min":31.596},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.8129999999999999,"5min":1.171,"15min":1.3080000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.252},"max":{"1min":7.6509999999999998,"5min":9.5060000000000002,"15min":31.57},"last":2.6739999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:20:08 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.6140000000000001,"15min":1.575},"min":{"1min":0.27300000000000002,"5min":0.246,"15min":0.246},"max":{"1min":1.9390000000000001,"5min":26.373000000000001,"15min":58.799999999999997},"last":2.637},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.476,"15min":1.518},"min":{"1min":0.34100000000000003,"5min":0.32300000000000001,"15min":0.26900000000000002},"max":{"1min":3.1709999999999998,"5min":27.968,"15min":58.615000000000002},"last":3.5859999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:20:47 2024","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.1779999999999999,"15min":1.3009999999999999},"min":{"1min":0.44900000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":6.1609999999999996,"5min":8.3089999999999993,"15min":15.859},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.1799999999999999,"15min":1.167},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":6.2709999999999999,"5min":9.1739999999999995,"15min":9.1739999999999995},"last":2.722}]},{"osd":7,"last update":"Mon Feb 5 08:19:55 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.123,"15min":1.258},"min":{"1min":0.34699999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.71,"5min":7.7530000000000001,"15min":15.986000000000001},"last":3.1280000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":1.5189999999999999,"15min":1.454},"min":{"1min":0.49399999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.599,"5min":11.945,"15min":15.951000000000001},"last":0.77600000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117212,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3125376,"kb_used_data":2606900,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":90619776,"statfs":{"total":95995035648,"available":92794650624,"internally_reserved":0,"allocated":2669465600,"data_stored":2669140851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":530905545},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:20:23 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.4039999999999999,"15min":1.8420000000000001},"min":{"1min":0.23100000000000001,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":11.747999999999999,"5min":52.831000000000003,"15min":63.43},"last":0.251},{"interface":"front","average":{"1min":1.988,"5min":1.6399999999999999,"15min":1.9850000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.184},"max":{"1min":11.618,"5min":53.859999999999999,"15min":63.048999999999999},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.93500000000000005,"15min":1.5109999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.217},"max":{"1min":3.3119999999999998,"5min":23.154,"15min":51.405999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.89200000000000002,"15min":1.8260000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.17399999999999999},"max":{"1min":3.54,"5min":23.207000000000001,"15min":51.189999999999998},"last":2.1379999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:20:23 2024","interfaces":[{"interface":"back","average":{"1min":3.2570000000000001,"5min":1.147,"15min":1.712},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":35.277000000000001,"5min":35.277000000000001,"15min":62.902999999999999},"last":0.308},{"interface":"front","average":{"1min":2.9929999999999999,"5min":1.034,"15min":1.51},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":35.442999999999998,"5min":35.442999999999998,"15min":63.468000000000004},"last":0.38200000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:20:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":0.93999999999999995,"15min":1.107},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.298,"5min":6.298,"15min":7.7229999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":1.034,"5min":0.90800000000000003,"15min":1.149},"min":{"1min":0.33800000000000002,"5min":0.32900000000000001,"15min":0.29599999999999999},"max":{"1min":2.8220000000000001,"5min":5.8159999999999998,"15min":10.539999999999999},"last":0.39800000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":2.7709999999999999,"15min":1.794},"min":{"1min":0.34100000000000003,"5min":0.32500000000000001,"15min":0.255},"max":{"1min":2.5449999999999999,"5min":117.432,"15min":117.432},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.86499999999999999,"5min":2.7469999999999999,"15min":1.841},"min":{"1min":0.309,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":2.6779999999999999,"5min":114.828,"15min":114.828},"last":0.53400000000000003}]},{"osd":6,"last update":"Mon Feb 5 08:20:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":0.90800000000000003,"15min":1.206},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.4750000000000001,"5min":3.6360000000000001,"15min":10.622},"last":0.50800000000000001},{"interface":"front","average":{"1min":1.821,"5min":1.0269999999999999,"15min":1.2270000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.308},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":7.8639999999999999},"last":0.36899999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.212,"15min":1.764},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":6.3730000000000002,"5min":6.3730000000000002,"15min":114.84999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.0289999999999999,"15min":1.7809999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33600000000000002},"max":{"1min":3.0569999999999999,"5min":6.0529999999999999,"15min":114.871},"last":0.55100000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953687,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4724784,"kb_used_data":4166052,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":89020368,"statfs":{"total":95995035648,"available":91156856832,"internally_reserved":0,"allocated":4266037248,"data_stored":4265727157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.2430000000000001,"15min":1.7210000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.6470000000000002,"5min":25.600999999999999,"15min":73.078999999999994},"last":2.3210000000000002},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.3260000000000001,"15min":1.704},"min":{"1min":0.35799999999999998,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":2.4380000000000002,"5min":23.030999999999999,"15min":73.412000000000006},"last":3.7639999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.97299999999999998,"15min":1.486},"min":{"1min":0.30399999999999999,"5min":0.23300000000000001,"15min":0.21099999999999999},"max":{"1min":1.4590000000000001,"5min":8.3320000000000007,"15min":125.054},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.89300000000000002,"15min":1.387},"min":{"1min":0.309,"5min":0.25700000000000001,"15min":0.23499999999999999},"max":{"1min":1.1910000000000001,"5min":6.2370000000000001,"15min":125.08799999999999},"last":3.1699999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.056,"15min":1.3080000000000001},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.24199999999999999},"max":{"1min":3.9729999999999999,"5min":4.8159999999999998,"15min":96.849999999999994},"last":2.282},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.0580000000000001,"15min":1.359},"min":{"1min":0.37,"5min":0.27100000000000002,"15min":0.26300000000000001},"max":{"1min":3.3500000000000001,"5min":5.6879999999999997,"15min":96.143000000000001},"last":3.7229999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.181,"15min":1.9079999999999999},"min":{"1min":0.38600000000000001,"5min":0.33100000000000002,"15min":0.24399999999999999},"max":{"1min":8.9269999999999996,"5min":8.9269999999999996,"15min":73.602999999999994},"last":3.7959999999999998},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.1180000000000001,"15min":1.9119999999999999},"min":{"1min":0.35299999999999998,"5min":0.309,"15min":0.27200000000000002},"max":{"1min":4.2599999999999998,"5min":4.2599999999999998,"15min":74.817999999999998},"last":3.202}]},{"osd":5,"last update":"Mon Feb 5 08:20:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.282,"15min":0.92300000000000004},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.224},"max":{"1min":4.54,"5min":9.157,"15min":34.167000000000002},"last":3.2429999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.1599999999999999,"15min":0.95199999999999996},"min":{"1min":0.308,"5min":0.254,"15min":0.254},"max":{"1min":1.7589999999999999,"5min":5.6550000000000002,"15min":36.051000000000002},"last":3.8980000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.2070000000000001,"15min":0.79800000000000004},"min":{"1min":0.34499999999999997,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.5990000000000002,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":3.8359999999999999},{"interface":"front","average":{"1min":1.9319999999999999,"5min":1.2749999999999999,"15min":0.79600000000000004},"min":{"1min":0.29099999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.2969999999999997,"5min":5.5049999999999999,"15min":5.5049999999999999},"last":3.7810000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:20:48 2024","interfaces":[{"interface":"back","average":{"1min":2.1360000000000001,"5min":1.4279999999999999,"15min":0.91200000000000003},"min":{"1min":0.40300000000000002,"5min":0.307,"15min":0.27000000000000002},"max":{"1min":12.683999999999999,"5min":12.683999999999999,"15min":12.683999999999999},"last":2.4969999999999999},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.3120000000000001,"15min":0.878},"min":{"1min":0.36099999999999999,"5min":0.33900000000000002,"15min":0.27900000000000003},"max":{"1min":4.5110000000000001,"5min":4.6529999999999996,"15min":5.7450000000000001},"last":3.8180000000000001}]}]},{"osd":5,"up_from":42,"seq":180388626641,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3195612,"kb_used_data":2604944,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":90549540,"statfs":{"total":95995035648,"available":92722728960,"internally_reserved":0,"allocated":2667462656,"data_stored":2667173428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":1.8380000000000001,"15min":1.9590000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":4.5090000000000003,"5min":30.065999999999999,"15min":143.06299999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.655,"15min":1.889},"min":{"1min":0.314,"5min":0.248,"15min":0.20499999999999999},"max":{"1min":4.7910000000000004,"5min":30.173999999999999,"15min":143.09700000000001},"last":0.505}]},{"osd":1,"last update":"Mon Feb 5 08:18:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.321,"15min":1.163},"min":{"1min":0.42799999999999999,"5min":0.22500000000000001,"15min":0.215},"max":{"1min":2.8090000000000002,"5min":5.8940000000000001,"15min":46.200000000000003},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.1859999999999999,"15min":1.23},"min":{"1min":0.29699999999999999,"5min":0.246,"15min":0.22900000000000001},"max":{"1min":4.7969999999999997,"5min":5.2210000000000001,"15min":46.343000000000004},"last":0.35099999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.107,"15min":1.091},"min":{"1min":0.33300000000000002,"5min":0.28999999999999998,"15min":0.246},"max":{"1min":0.93200000000000005,"5min":7.1159999999999997,"15min":10.733000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.77500000000000002,"5min":1.3200000000000001,"15min":1.1899999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.2320000000000002,"5min":5.8250000000000002,"15min":8.0220000000000002},"last":0.52700000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":1.1930000000000001,"15min":1.734},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.28799999999999998},"max":{"1min":0.97099999999999997,"5min":7.165,"15min":143.03399999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.754,"5min":1.2569999999999999,"15min":1.837},"min":{"1min":0.34499999999999997,"5min":0.33800000000000002,"15min":0.28100000000000003},"max":{"1min":1.6890000000000001,"5min":5.8609999999999998,"15min":143.46799999999999},"last":0.49099999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":1.143,"15min":0.79300000000000004},"min":{"1min":0.27300000000000002,"5min":0.25600000000000001,"15min":0.23699999999999999},"max":{"1min":1.4470000000000001,"5min":5.3769999999999998,"15min":5.3769999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":1.2829999999999999,"15min":0.79700000000000004},"min":{"1min":0.38600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.98299999999999998,"5min":7.2380000000000004,"15min":7.2380000000000004},"last":0.35199999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:20:35 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.425,"15min":0.872},"min":{"1min":0.34599999999999997,"5min":0.33200000000000002,"15min":0.26600000000000001},"max":{"1min":4.0170000000000003,"5min":9.4100000000000001,"15min":9.4100000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.357,"15min":0.872},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":4.0510000000000002,"5min":7.194,"15min":7.194},"last":0.46500000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:19:56 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":1.708,"15min":0.98299999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.25900000000000001},"max":{"1min":3.722,"5min":9.6940000000000008,"15min":9.6940000000000008},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.628,"5min":1.4990000000000001,"15min":0.93400000000000005},"min":{"1min":0.38900000000000001,"5min":0.318,"15min":0.28199999999999997},"max":{"1min":1.4299999999999999,"5min":9.2360000000000007,"15min":9.2360000000000007},"last":0.42399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5330894848,"data_stored":5330894848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":645464064,"data_stored":645464064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3186688000,"data_stored":3186688000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2657615872,"data_stored":2657615872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4240445440,"data_stored":4240441368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2642345984,"data_stored":2642345984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3700363264,"data_stored":3700359192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2113798144,"data_stored":2113798144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:20:56.656 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-02-05T08:20:56.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:56.724 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:56 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1335776027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:20:56.739 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:56.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:56.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T08:20:56.806 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:20:56.806 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:20:56.806 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:20:56.806 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:20:56.806 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:20:56.807 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:20:56.807 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:20:56.807 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:20:56.807 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:20:56.807 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:20:56.807 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:20:56.807 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:20:56.807 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:20:56.807 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:20:56.807 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:20:56.842 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:56.869 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:56.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:56.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:56.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:20:57.005 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:57.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:20:57.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:57.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:57.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:57.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:57.182 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:20:57.244 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:57.276 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:57.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:57.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:57.378 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:57.415 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:57.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:57.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:57.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:57.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:57.655 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:57.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:57 smithi196 ceph-mon[33018]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:20:57.658 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:57 smithi196 ceph-mon[33018]: pgmap v724: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 1.00k op/s; 24333/388212 objects degraded (6.268%) 2024-02-05T08:20:57.693 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:57.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:57.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:57.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:57 smithi134 ceph-mon[30086]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:20:57.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:57 smithi134 ceph-mon[30086]: pgmap v724: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 1.00k op/s; 24333/388212 objects degraded (6.268%) 2024-02-05T08:20:57.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:57 smithi134 ceph-mon[39278]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:20:57.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:57 smithi134 ceph-mon[39278]: pgmap v724: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 1.00k op/s; 24333/388212 objects degraded (6.268%) 2024-02-05T08:20:57.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:57.828 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:20:57.829 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:20:57.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:57.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:57.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:57.870 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:20:57.936 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:57.977 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:57.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:57.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:58.005 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:20:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:20:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:20:58.079 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:20:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:58.110 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:58.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:20:58.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:58.249 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:58.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:58.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:58.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:58.386 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:58.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:58.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:58.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:58.529 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:58.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:58.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:58.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:58.679 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:58.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:58.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:58.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:58.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:58.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:58.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:58.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:59.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:59.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:59.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:59.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:20:59.237 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:20:59.238 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:20:59.238 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T08:20:59.238 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T08:20:59.238 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:20:59.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:59.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:59.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:20:59.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:59.374 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:59.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:59.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:59.475 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:20:59.500 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:59.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:20:59.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:20:59 smithi134 ceph-mon[30086]: pgmap v725: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 38 MiB/s wr, 942 op/s; 23946/384798 objects degraded (6.223%) 2024-02-05T08:20:59.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:20:59 smithi134 ceph-mon[39278]: pgmap v725: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 38 MiB/s wr, 942 op/s; 23946/384798 objects degraded (6.223%) 2024-02-05T08:20:59.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:20:59.637 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:59.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:59.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:20:59.738 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:20:59.770 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:20:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:59.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:20:59.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:20:59 smithi196 ceph-mon[33018]: pgmap v725: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.8 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 38 MiB/s wr, 942 op/s; 23946/384798 objects degraded (6.223%) 2024-02-05T08:20:59.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:20:59.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:20:59.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:20:59.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:20:59.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:20:59.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T08:21:00.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:00.070 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:00.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:00.171 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:00.215 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:00.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:00.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:00.287 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:00.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:00.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:00.366 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:00.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:00.387 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:21:00.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:00.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:00.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:00.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:00.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:00.658 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:00.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:00.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:00.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:00 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 23946/384798 objects degraded (6.223%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:00.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:00 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 23946/384798 objects degraded (6.223%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:00.796 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:00.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:00.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:00 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 23946/384798 objects degraded (6.223%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:00.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:00.933 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:00.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:00.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:01.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:01.065 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:01.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:01.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:01.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:01.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:01.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:01.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:01.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:01.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:01.491 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:21:01.494 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:01.529 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:01.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:01.631 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:01.657 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-02-05T08:21:01.657 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:21:01.657 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_3 2024-02-05T08:21:01.657 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:21:01.665 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:01.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:01.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:01.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:01 smithi134 ceph-mon[30086]: pgmap v726: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 MiB/s wr, 1.04k op/s; 23142/369969 objects degraded (6.255%) 2024-02-05T08:21:01.706 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:01 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1702742857' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:21:01.706 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:01 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:21:01.706 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:01 smithi134 ceph-mon[39278]: pgmap v726: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 MiB/s wr, 1.04k op/s; 23142/369969 objects degraded (6.255%) 2024-02-05T08:21:01.706 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:01 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1702742857' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:21:01.706 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:01 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:21:01.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:01.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:01.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:01.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:01.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:01 smithi196 ceph-mon[33018]: pgmap v726: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 MiB/s wr, 1.04k op/s; 23142/369969 objects degraded (6.255%) 2024-02-05T08:21:01.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:01 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1702742857' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:21:01.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:01 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:21:01.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:01.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:01.947 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:01.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:02.048 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:02.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:02.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:02.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:02.205 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:02.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:02.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:02.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:02.346 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:02.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:02.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:02.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:02.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:02.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:02.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:02.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:02.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:02.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:02.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:02.765 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:02.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:02.766 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:02 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:21:02.766 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:02 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:21:02.766 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:02 smithi134 ceph-mon[30086]: osdmap e337: 8 total, 7 up, 8 in 2024-02-05T08:21:02.766 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:02 smithi134 ceph-mon[30086]: 10.d scrub starts 2024-02-05T08:21:02.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:02 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:21:02.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:02 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:21:02.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:02 smithi134 ceph-mon[39278]: osdmap e337: 8 total, 7 up, 8 in 2024-02-05T08:21:02.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:02 smithi134 ceph-mon[39278]: 10.d scrub starts 2024-02-05T08:21:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:02 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:21:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:02 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:21:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:02 smithi196 ceph-mon[33018]: osdmap e337: 8 total, 7 up, 8 in 2024-02-05T08:21:02.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:02 smithi196 ceph-mon[33018]: 10.d scrub starts 2024-02-05T08:21:02.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:02.910 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:02.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:02.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:03.012 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:03.040 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:03.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:03.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:03.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:03.185 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:03.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:03.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:03.301 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:03.330 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:03.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:03.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:03.352 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:21:03.353 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:21:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:21:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:21:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:21:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136491 8.0 8:00352e96:::benchmark_data_smithi134_115836_object19831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:21:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.657285+0000", 2024-02-05T08:21:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.691081252, 2024-02-05T08:21:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.037191507, 2024-02-05T08:21:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:21:03.353 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136491 2024-02-05T08:21:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657285+0000", 2024-02-05T08:21:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.354 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657285+0000", 2024-02-05T08:21:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657286+0000", 2024-02-05T08:21:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4200000000000005e-07 2024-02-05T08:21:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657290+0000", 2024-02-05T08:21:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1289999999999998e-06 2024-02-05T08:21:03.355 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657303+0000", 2024-02-05T08:21:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2362e-05 2024-02-05T08:21:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657309+0000", 2024-02-05T08:21:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9159999999999998e-06 2024-02-05T08:21:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.356 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657364+0000", 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5350999999999999e-05 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657459+0000", 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5309999999999994e-05 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.657540+0000", 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0322999999999997e-05 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.357 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.659672+0000", 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021321059999999999 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.659795+0000", 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000123063 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.664541+0000", 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047457879999999999 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.358 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.678869+0000", 2024-02-05T08:21:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014328278999999999 2024-02-05T08:21:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.678879+0000", 2024-02-05T08:21:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0196e-05 2024-02-05T08:21:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.694389+0000", 2024-02-05T08:21:03.359 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.015510425 2024-02-05T08:21:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.694477+0000", 2024-02-05T08:21:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7416999999999993e-05 2024-02-05T08:21:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136499 8.0 8:0b745435:::benchmark_data_smithi134_115836_object19839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:21:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.669853+0000", 2024-02-05T08:21:03.360 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.67851399400001, 2024-02-05T08:21:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.025200141, 2024-02-05T08:21:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:21:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136499 2024-02-05T08:21:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669853+0000", 2024-02-05T08:21:03.361 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669853+0000", 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669854+0000", 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8000000000000004e-07 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669858+0000", 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1649999999999999e-06 2024-02-05T08:21:03.362 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669872+0000", 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4574e-05 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669878+0000", 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5770000000000001e-06 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.669926+0000", 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8191e-05 2024-02-05T08:21:03.363 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.670025+0000", 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8585999999999994e-05 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.670114+0000", 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9820000000000004e-05 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.672248+0000", 2024-02-05T08:21:03.364 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021340370000000001 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.672356+0000", 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107456 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.678926+0000", 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065703549999999999 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.678934+0000", 2024-02-05T08:21:03.365 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3599999999999998e-06 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.694985+0000", 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0160511160000001 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.695053+0000", 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8023999999999998e-05 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.366 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.367 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136506 8.0 8:02587487:::benchmark_data_smithi134_115836_object19846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.367 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.680322+0000", 2024-02-05T08:21:03.367 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.66804478699999, 2024-02-05T08:21:03.367 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.024271468, 2024-02-05T08:21:03.367 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.367 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.367 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.367 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.367 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.367 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136506 2024-02-05T08:21:03.367 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.367 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.367 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680322+0000", 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680322+0000", 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680323+0000", 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5859999999999999e-06 2024-02-05T08:21:03.368 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680336+0000", 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2333e-05 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680350+0000", 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4080000000000001e-05 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680359+0000", 2024-02-05T08:21:03.369 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0569999999999999e-06 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680413+0000", 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4431000000000001e-05 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.680421+0000", 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2570000000000001e-06 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.370 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.695096+0000", 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.014675564 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.695221+0000", 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000125186 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.695317+0000", 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5402000000000006e-05 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.371 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704478+0000", 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0091610249999999997 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704494+0000", 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5965999999999999e-05 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704537+0000", 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2824999999999997e-05 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.372 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704571+0000", 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4183999999999997e-05 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704593+0000", 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2572e-05 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.373 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136519 8.0 8:0144bb8f:::benchmark_data_smithi134_115836_object19859:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.697455+0000", 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.65091143000001, 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0072622899999999, 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136519 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.374 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.697455+0000", 2024-02-05T08:21:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.697455+0000", 2024-02-05T08:21:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.697456+0000", 2024-02-05T08:21:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6199999999999997e-07 2024-02-05T08:21:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.697460+0000", 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3649999999999997e-06 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.697471+0000", 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0987e-05 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.697480+0000", 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8759999999999994e-06 2024-02-05T08:21:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.700194+0000", 2024-02-05T08:21:03.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002713615 2024-02-05T08:21:03.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.700232+0000", 2024-02-05T08:21:03.377 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8225000000000001e-05 2024-02-05T08:21:03.377 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.377 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.377 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.377 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.698708+0000", 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99847636799999995 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.698833+0000", 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000124256 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.698962+0000", 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012896000000000001 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.378 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704500+0000", 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055384709999999997 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704635+0000", 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013537299999999999 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704670+0000", 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4434000000000003e-05 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.379 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704699+0000", 2024-02-05T08:21:03.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8796e-05 2024-02-05T08:21:03.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.380 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.380 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704717+0000", 2024-02-05T08:21:03.380 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8802000000000001e-05 2024-02-05T08:21:03.380 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.380 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.380 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.380 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.380 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.381 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136524 8.0 8:003515e7:::benchmark_data_smithi134_115836_object19864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.381 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.706080+0000", 2024-02-05T08:21:03.381 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.64228635400002, 2024-02-05T08:21:03.381 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0065054609999999, 2024-02-05T08:21:03.381 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.381 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.382 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.382 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.382 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.382 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136524 2024-02-05T08:21:03.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.382 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.382 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706080+0000", 2024-02-05T08:21:03.382 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.382 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.382 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.382 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706080+0000", 2024-02-05T08:21:03.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706081+0000", 2024-02-05T08:21:03.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5499999999999996e-07 2024-02-05T08:21:03.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.383 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.383 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.383 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706086+0000", 2024-02-05T08:21:03.383 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1089999999999997e-06 2024-02-05T08:21:03.383 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706102+0000", 2024-02-05T08:21:03.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5302e-05 2024-02-05T08:21:03.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706110+0000", 2024-02-05T08:21:03.384 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7029999999999995e-06 2024-02-05T08:21:03.384 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.384 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.384 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.384 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706277+0000", 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016661900000000001 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706286+0000", 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9299999999999992e-06 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.701831+0000", 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99554492900000002 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.385 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.701954+0000", 2024-02-05T08:21:03.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012360800000000001 2024-02-05T08:21:03.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.702061+0000", 2024-02-05T08:21:03.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106554 2024-02-05T08:21:03.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.386 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.386 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.386 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704762+0000", 2024-02-05T08:21:03.386 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002701331 2024-02-05T08:21:03.386 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704776+0000", 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3923e-05 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712452+0000", 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.007675734 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712534+0000", 2024-02-05T08:21:03.387 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2359000000000002e-05 2024-02-05T08:21:03.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.388 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.388 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712586+0000", 2024-02-05T08:21:03.388 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1405e-05 2024-02-05T08:21:03.388 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.388 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.388 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.388 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.388 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.388 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136525 8.0 8:0ae77d71:::benchmark_data_smithi134_115836_object19865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.388 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.706251+0000", 2024-02-05T08:21:03.388 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.64211516799998, 2024-02-05T08:21:03.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.012386306, 2024-02-05T08:21:03.389 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.389 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.389 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.389 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.389 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.389 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136525 2024-02-05T08:21:03.389 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.389 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.389 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.389 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.389 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706251+0000", 2024-02-05T08:21:03.389 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706251+0000", 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706252+0000", 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3500000000000005e-07 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.390 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706257+0000", 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6079999999999998e-06 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706271+0000", 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3642000000000001e-05 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706279+0000", 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1359999999999997e-06 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.391 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706314+0000", 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5213000000000003e-05 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.706319+0000", 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4979999999999997e-06 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704790+0000", 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99847085499999999 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.392 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704864+0000", 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3928000000000001e-05 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.704940+0000", 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5338000000000003e-05 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712682+0000", 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077425690000000004 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.393 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712699+0000", 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6458999999999999e-05 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718557+0000", 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058581809999999996 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718600+0000", 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3421999999999999e-05 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.394 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.395 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718638+0000", 2024-02-05T08:21:03.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7622e-05 2024-02-05T08:21:03.395 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.395 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.395 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.395 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.395 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.395 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136530 8.0 8:0397fdba:::benchmark_data_smithi134_115836_object19870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.395 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.712771+0000", 2024-02-05T08:21:03.395 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.63559544499998, 2024-02-05T08:21:03.395 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0059719009999999, 2024-02-05T08:21:03.395 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.395 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.396 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.396 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.396 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.396 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136530 2024-02-05T08:21:03.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.396 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.396 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712771+0000", 2024-02-05T08:21:03.396 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.396 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.396 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.396 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712771+0000", 2024-02-05T08:21:03.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712772+0000", 2024-02-05T08:21:03.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9599999999999999e-07 2024-02-05T08:21:03.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.397 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.397 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.397 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712779+0000", 2024-02-05T08:21:03.397 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9399999999999996e-06 2024-02-05T08:21:03.397 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712790+0000", 2024-02-05T08:21:03.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1144e-05 2024-02-05T08:21:03.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712797+0000", 2024-02-05T08:21:03.398 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.34e-06 2024-02-05T08:21:03.398 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.398 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.398 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.398 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712880+0000", 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2985000000000006e-05 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.712890+0000", 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7729999999999994e-06 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.707451+0000", 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99456112200000002 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.399 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.707568+0000", 2024-02-05T08:21:03.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117034 2024-02-05T08:21:03.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.707678+0000", 2024-02-05T08:21:03.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109516 2024-02-05T08:21:03.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.400 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.400 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.400 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712707+0000", 2024-02-05T08:21:03.400 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0050295879999999998 2024-02-05T08:21:03.400 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712761+0000", 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3804000000000002e-05 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718673+0000", 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059123350000000003 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718715+0000", 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1248999999999998e-05 2024-02-05T08:21:03.401 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718743+0000", 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8374999999999999e-05 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136533 8.0 8:0b2b077c:::benchmark_data_smithi134_115836_object19873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.716799+0000", 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.63156718499999, 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0020862020000001, 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.402 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136533 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716799+0000", 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.403 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716799+0000", 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716800+0000", 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.85e-07 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716804+0000", 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8829999999999999e-06 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.404 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716817+0000", 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3006e-05 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716824+0000", 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.328e-06 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716868+0000", 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3074999999999997e-05 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.405 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.716874+0000", 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0179999999999998e-06 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.710115+0000", 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99324165499999995 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.710199+0000", 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3902999999999999e-05 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.406 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.710270+0000", 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1204999999999993e-05 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712836+0000", 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0025652869999999999 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712850+0000", 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4688999999999999e-05 2024-02-05T08:21:03.407 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718799+0000", 2024-02-05T08:21:03.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059487899999999998 2024-02-05T08:21:03.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718848+0000", 2024-02-05T08:21:03.408 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9363999999999997e-05 2024-02-05T08:21:03.408 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.408 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.408 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.408 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718886+0000", 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7214e-05 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136552 8.0 8:0d51f24e:::benchmark_data_smithi134_115836_object19892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.737390+0000", 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.61097621699997, 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98161826200000002, 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.409 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136552 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737390+0000", 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737390+0000", 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.410 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737391+0000", 2024-02-05T08:21:03.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8999999999999995e-07 2024-02-05T08:21:03.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.411 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737396+0000", 2024-02-05T08:21:03.411 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8799999999999999e-06 2024-02-05T08:21:03.411 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.411 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.411 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737410+0000", 2024-02-05T08:21:03.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3597e-05 2024-02-05T08:21:03.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737418+0000", 2024-02-05T08:21:03.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.242e-06 2024-02-05T08:21:03.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.412 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.412 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.412 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737472+0000", 2024-02-05T08:21:03.412 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3678999999999999e-05 2024-02-05T08:21:03.412 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.737479+0000", 2024-02-05T08:21:03.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5780000000000001e-06 2024-02-05T08:21:03.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.413 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.712881+0000", 2024-02-05T08:21:03.413 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97540128100000001 2024-02-05T08:21:03.413 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.413 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.413 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.713006+0000", 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012547 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.713129+0000", 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000122784 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718514+0000", 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053848250000000002 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.414 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718528+0000", 2024-02-05T08:21:03.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4289e-05 2024-02-05T08:21:03.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718931+0000", 2024-02-05T08:21:03.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040299800000000001 2024-02-05T08:21:03.415 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.415 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.415 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.415 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.718976+0000", 2024-02-05T08:21:03.415 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4570000000000002e-05 2024-02-05T08:21:03.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.416 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.416 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.719009+0000", 2024-02-05T08:21:03.416 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3179000000000002e-05 2024-02-05T08:21:03.416 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.416 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.416 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.416 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.416 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.416 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136555 8.0 8:068b584f:::benchmark_data_smithi134_115836_object19895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.416 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.739036+0000", 2024-02-05T08:21:03.417 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.60933106800002, 2024-02-05T08:21:03.417 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98997460400000004, 2024-02-05T08:21:03.417 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.417 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.417 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.417 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.417 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.417 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136555 2024-02-05T08:21:03.417 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.417 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.417 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.417 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.417 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739036+0000", 2024-02-05T08:21:03.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739036+0000", 2024-02-05T08:21:03.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.418 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.418 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739036+0000", 2024-02-05T08:21:03.418 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8100000000000001e-07 2024-02-05T08:21:03.418 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.418 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739041+0000", 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4689999999999999e-06 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739056+0000", 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4739000000000001e-05 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739064+0000", 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5299999999999996e-06 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.419 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739122+0000", 2024-02-05T08:21:03.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7986999999999999e-05 2024-02-05T08:21:03.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.739131+0000", 2024-02-05T08:21:03.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5699999999999993e-06 2024-02-05T08:21:03.420 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.420 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.420 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.420 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.715762+0000", 2024-02-05T08:21:03.420 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97663171199999999 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.715882+0000", 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119974 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.716003+0000", 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000120401 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.421 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.719060+0000", 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030575480000000002 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.728903+0000", 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0098422530000000005 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.728924+0000", 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0871000000000001e-05 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.728972+0000", 2024-02-05T08:21:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8936e-05 2024-02-05T08:21:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.424 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:21:03.424 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:21:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729010+0000", 2024-02-05T08:21:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7732999999999999e-05 2024-02-05T08:21:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136557 8.0 8:08e98751:::benchmark_data_smithi134_115836_object19897:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.742226+0000", 2024-02-05T08:21:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.60614024699998, 2024-02-05T08:21:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99639465800000004, 2024-02-05T08:21:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136557 2024-02-05T08:21:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742226+0000", 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742226+0000", 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742228+0000", 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.113e-06 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742232+0000", 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3309999999999997e-06 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742248+0000", 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6107e-05 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742256+0000", 2024-02-05T08:21:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4330000000000004e-06 2024-02-05T08:21:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742341+0000", 2024-02-05T08:21:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4172999999999995e-05 2024-02-05T08:21:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.742349+0000", 2024-02-05T08:21:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3799999999999994e-06 2024-02-05T08:21:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.719097+0000", 2024-02-05T08:21:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97674827900000005 2024-02-05T08:21:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.719221+0000", 2024-02-05T08:21:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000124084 2024-02-05T08:21:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.719387+0000", 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016600999999999999 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729020+0000", 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0096322219999999993 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729069+0000", 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9456999999999998e-05 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738523+0000", 2024-02-05T08:21:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.009454278 2024-02-05T08:21:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738579+0000", 2024-02-05T08:21:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5782999999999998e-05 2024-02-05T08:21:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738621+0000", 2024-02-05T08:21:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2008000000000002e-05 2024-02-05T08:21:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136562 8.0 8:0611ba46:::benchmark_data_smithi134_115836_object19902:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.745425+0000", 2024-02-05T08:21:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.60294166800003, 2024-02-05T08:21:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99331518600000002, 2024-02-05T08:21:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136562 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745425+0000", 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745425+0000", 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745426+0000", 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8899999999999998e-07 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745431+0000", 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6870000000000002e-06 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745458+0000", 2024-02-05T08:21:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7885000000000002e-05 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745469+0000", 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0129000000000001e-05 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745567+0000", 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8684999999999995e-05 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.435 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.745578+0000", 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0891e-05 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.722102+0000", 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97652399700000003 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.722202+0000", 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100271 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.436 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.722420+0000", 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021719600000000001 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729025+0000", 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066054420000000004 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729107+0000", 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1979999999999998e-05 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.437 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738665+0000", 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0095575179999999992 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738708+0000", 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.354e-05 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738740+0000", 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2076000000000003e-05 2024-02-05T08:21:03.438 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136567 8.0 8:09cb9fd1:::benchmark_data_smithi134_115836_object19907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.757390+0000", 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.590976883, 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99000443400000004, 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136567 2024-02-05T08:21:03.439 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.440 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757390+0000", 2024-02-05T08:21:03.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.440 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.440 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757390+0000", 2024-02-05T08:21:03.440 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.440 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.440 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757391+0000", 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.61e-07 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757394+0000", 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5899999999999999e-06 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757406+0000", 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1481000000000001e-05 2024-02-05T08:21:03.441 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757413+0000", 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4059999999999999e-06 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757473+0000", 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9667999999999999e-05 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.757481+0000", 2024-02-05T08:21:03.442 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.4530000000000002e-06 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.725852+0000", 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96837125899999998 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.726036+0000", 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00018403000000000001 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.443 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.726148+0000", 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011178799999999999 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729141+0000", 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0029930619999999999 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738471+0000", 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0093298789999999993 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.444 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.747294+0000", 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0088230219999999998 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.747363+0000", 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8349999999999994e-05 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.747394+0000", 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1684999999999999e-05 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.445 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136569 8.0 8:020c7c43:::benchmark_data_smithi134_115836_object19909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.761088+0000", 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.58727817599998, 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.98660557500000001, 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136569 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.446 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761088+0000", 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761088+0000", 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761089+0000", 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7900000000000007e-07 2024-02-05T08:21:03.447 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761093+0000", 2024-02-05T08:21:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8079999999999998e-06 2024-02-05T08:21:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761103+0000", 2024-02-05T08:21:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0249e-05 2024-02-05T08:21:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.448 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761110+0000", 2024-02-05T08:21:03.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0599999999999996e-06 2024-02-05T08:21:03.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761153+0000", 2024-02-05T08:21:03.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3733999999999998e-05 2024-02-05T08:21:03.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.449 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.449 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.761161+0000", 2024-02-05T08:21:03.449 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6639999999999998e-06 2024-02-05T08:21:03.449 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.449 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729171+0000", 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96800960300000005 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729280+0000", 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010945100000000001 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.729396+0000", 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116348 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.450 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738459+0000", 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0090630169999999996 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738476+0000", 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6878e-05 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.747607+0000", 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0091306809999999999 2024-02-05T08:21:03.451 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.747659+0000", 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2420000000000003e-05 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.747694+0000", 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4683e-05 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.452 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136574 8.0 8:0b467986:::benchmark_data_smithi134_115836_object19914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.774612+0000", 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.5737542, 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97985992499999996, 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136574 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774612+0000", 2024-02-05T08:21:03.453 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774612+0000", 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774614+0000", 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.629e-06 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.454 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774620+0000", 2024-02-05T08:21:03.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1920000000000002e-06 2024-02-05T08:21:03.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774637+0000", 2024-02-05T08:21:03.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6438000000000001e-05 2024-02-05T08:21:03.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.455 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.455 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774647+0000", 2024-02-05T08:21:03.455 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0020000000000001e-05 2024-02-05T08:21:03.455 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.455 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774713+0000", 2024-02-05T08:21:03.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5846000000000006e-05 2024-02-05T08:21:03.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.774725+0000", 2024-02-05T08:21:03.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2101e-05 2024-02-05T08:21:03.456 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.456 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.456 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.456 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.731889+0000", 2024-02-05T08:21:03.456 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.957164601 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.731990+0000", 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000100874 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.732097+0000", 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000107283 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.457 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738480+0000", 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063826600000000001 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.738866+0000", 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00038624999999999998 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.754413+0000", 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.015546739 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.458 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.754449+0000", 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5766999999999998e-05 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.754472+0000", 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3524999999999999e-05 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:136593 8.0 8:0c9c3211:::benchmark_data_smithi134_115836_object19933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:04.850131+0000", 2024-02-05T08:21:03.459 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 478.498235744, 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.90448195099999995, 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 136593 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850131+0000", 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.460 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850131+0000", 2024-02-05T08:21:03.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850132+0000", 2024-02-05T08:21:03.461 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.432e-06 2024-02-05T08:21:03.461 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.461 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.461 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.461 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850137+0000", 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.301e-06 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850148+0000", 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1627000000000001e-05 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850155+0000", 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2879999999999998e-06 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.462 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850215+0000", 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0081000000000003e-05 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:04.850226+0000", 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1097999999999999e-05 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.735002+0000", 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88477587000000002 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.735508+0000", 2024-02-05T08:21:03.463 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050598699999999995 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.735660+0000", 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000152409 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.742042+0000", 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0063815520000000004 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.742099+0000", 2024-02-05T08:21:03.464 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7327000000000002e-05 2024-02-05T08:21:03.465 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.465 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.465 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.465 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.754523+0000", 2024-02-05T08:21:03.465 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012424117 2024-02-05T08:21:03.465 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:03.466 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:21:03.466 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.466 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.466 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.754575+0000", 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2299999999999997e-05 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:05.754613+0000", 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7561999999999997e-05 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170401 8.8 8:15576685:::benchmark_data_smithi134_115836_object53741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.792841+0000", 2024-02-05T08:21:03.467 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 437.555525461, 2024-02-05T08:21:03.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94320472300000002, 2024-02-05T08:21:03.468 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.468 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.468 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.468 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.468 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.468 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170401 2024-02-05T08:21:03.468 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.468 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.468 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.468 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.468 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792841+0000", 2024-02-05T08:21:03.468 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792841+0000", 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792842+0000", 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2109999999999999e-06 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.469 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792846+0000", 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5260000000000002e-06 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792857+0000", 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.135e-05 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792865+0000", 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8930000000000005e-06 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.470 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792926+0000", 2024-02-05T08:21:03.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0949000000000002e-05 2024-02-05T08:21:03.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792935+0000", 2024-02-05T08:21:03.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7050000000000005e-06 2024-02-05T08:21:03.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.471 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.471 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.471 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.733064+0000", 2024-02-05T08:21:03.471 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94012932299999996 2024-02-05T08:21:03.471 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.733194+0000", 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012970500000000001 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.733276+0000", 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2328000000000002e-05 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.735903+0000", 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026266029999999999 2024-02-05T08:21:03.472 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.735915+0000", 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1817999999999999e-05 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.735956+0000", 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1635999999999999e-05 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.736003+0000", 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7089000000000003e-05 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.473 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.736046+0000", 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2586999999999997e-05 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170446 8.8 8:1cf91133:::benchmark_data_smithi134_115836_object53786:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.854446+0000", 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 437.49392082200001, 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88840418799999998, 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.474 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170446 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854446+0000", 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.475 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854446+0000", 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854447+0000", 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9099999999999991e-07 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854450+0000", 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6179999999999999e-06 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.476 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854462+0000", 2024-02-05T08:21:03.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1548e-05 2024-02-05T08:21:03.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854468+0000", 2024-02-05T08:21:03.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7300000000000002e-06 2024-02-05T08:21:03.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.477 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.477 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.477 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854532+0000", 2024-02-05T08:21:03.477 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3870999999999999e-05 2024-02-05T08:21:03.477 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.854541+0000", 2024-02-05T08:21:03.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2080000000000006e-06 2024-02-05T08:21:03.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.736085+0000", 2024-02-05T08:21:03.478 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88154407899999998 2024-02-05T08:21:03.478 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.478 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.478 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.478 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.736150+0000", 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5520999999999998e-05 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.736211+0000", 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0817999999999999e-05 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.740393+0000", 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0041813190000000002 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.479 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.742713+0000", 2024-02-05T08:21:03.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0023207359999999999 2024-02-05T08:21:03.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.742773+0000", 2024-02-05T08:21:03.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9789e-05 2024-02-05T08:21:03.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.480 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.480 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.480 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.742813+0000", 2024-02-05T08:21:03.480 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9972999999999998e-05 2024-02-05T08:21:03.480 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.481 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.481 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.742850+0000", 2024-02-05T08:21:03.481 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6986999999999997e-05 2024-02-05T08:21:03.481 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.481 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.481 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.481 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.481 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.481 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170458 8.8 8:11a83214:::benchmark_data_smithi134_115836_object53798:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:21:03.481 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.863750+0000", 2024-02-05T08:21:03.481 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 437.48461684799997, 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88835640999999999, 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170458 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863750+0000", 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.482 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863750+0000", 2024-02-05T08:21:03.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.483 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863751+0000", 2024-02-05T08:21:03.483 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2999999999999999e-07 2024-02-05T08:21:03.483 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.483 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.483 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863754+0000", 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5310000000000002e-06 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863764+0000", 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8770000000000005e-06 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863771+0000", 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9979999999999997e-06 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.484 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863806+0000", 2024-02-05T08:21:03.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4671000000000001e-05 2024-02-05T08:21:03.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.863812+0000", 2024-02-05T08:21:03.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2790000000000004e-06 2024-02-05T08:21:03.485 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.485 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.485 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.485 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.740517+0000", 2024-02-05T08:21:03.485 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.876704539 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.740625+0000", 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108942 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.740705+0000", 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9560000000000004e-05 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.745209+0000", 2024-02-05T08:21:03.486 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045041869999999998 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.745218+0000", 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.405e-06 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.751999+0000", 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067812380000000002 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.752057+0000", 2024-02-05T08:21:03.487 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7773999999999999e-05 2024-02-05T08:21:03.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.488 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.488 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.752106+0000", 2024-02-05T08:21:03.488 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9579000000000001e-05 2024-02-05T08:21:03.488 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.488 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.488 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.488 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.488 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.488 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170464 8.8 8:17746572:::benchmark_data_smithi134_115836_object53804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:21:03.488 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.870050+0000", 2024-02-05T08:21:03.488 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 437.47831671, 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88894854099999998, 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170464 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870050+0000", 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.489 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:03.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870050+0000", 2024-02-05T08:21:03.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:03.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:03.490 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870051+0000", 2024-02-05T08:21:03.490 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7400000000000002e-07 2024-02-05T08:21:03.490 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.490 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.490 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:03.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870055+0000", 2024-02-05T08:21:03.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.561e-06 2024-02-05T08:21:03.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:03.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870068+0000", 2024-02-05T08:21:03.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2458e-05 2024-02-05T08:21:03.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.491 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.491 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:03.491 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870075+0000", 2024-02-05T08:21:03.491 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1890000000000001e-06 2024-02-05T08:21:03.491 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870130+0000", 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5297999999999998e-05 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.870140+0000", 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2990000000000002e-06 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.745256+0000", 2024-02-05T08:21:03.492 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.875116271 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.745380+0000", 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012443 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.745491+0000", 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110257 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.752163+0000", 2024-02-05T08:21:03.493 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0066722159999999999 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.752182+0000", 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9623000000000002e-05 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.758912+0000", 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067298669999999996 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:03.494 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.758962+0000", 2024-02-05T08:21:03.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0207999999999998e-05 2024-02-05T08:21:03.495 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:03.495 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:03.495 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:03.495 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.758998+0000", 2024-02-05T08:21:03.495 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5989999999999999e-05 2024-02-05T08:21:03.495 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.495 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.495 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.495 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:03.495 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:03.495 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:21:03.496 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:21:03.532 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:03.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:03.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:03.535 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:21:03.573 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:21:03.574 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:21:03.574 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:21:03.574 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:21:03.574 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:21:03.574 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:21:03.574 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:21:03.574 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:21:03.574 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:21:03.634 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:03.670 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:03.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:03.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:03.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:03.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:03 smithi134 ceph-mon[30086]: pgmap v728: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 MiB/s wr, 1.04k op/s; 23142/369969 objects degraded (6.255%) 2024-02-05T08:21:03.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:03 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1684181470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:03.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:03 smithi134 ceph-mon[39278]: pgmap v728: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 MiB/s wr, 1.04k op/s; 23142/369969 objects degraded (6.255%) 2024-02-05T08:21:03.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:03 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1684181470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:03.809 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:03.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:03.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:03 smithi196 ceph-mon[33018]: pgmap v728: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.1 KiB/s rd, 13 MiB/s wr, 1.04k op/s; 23142/369969 objects degraded (6.255%) 2024-02-05T08:21:03.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:03 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1684181470' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:03.910 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:03.950 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:03.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:03.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:04.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:04.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:04.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:04.103 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:21:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:21:04.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:04.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:04.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:04.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:04.361 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:04.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:04.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:04.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:04.505 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:04.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:04.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:04.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T08:21:04.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:04.685 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:04.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:04.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:04.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:04 smithi134 ceph-mon[30086]: 3.12 scrub starts 2024-02-05T08:21:04.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:04 smithi134 ceph-mon[30086]: 3.12 scrub ok 2024-02-05T08:21:04.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:04 smithi134 ceph-mon[30086]: 10.d scrub ok 2024-02-05T08:21:04.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:04 smithi134 ceph-mon[39278]: 3.12 scrub starts 2024-02-05T08:21:04.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:04 smithi134 ceph-mon[39278]: 3.12 scrub ok 2024-02-05T08:21:04.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:04 smithi134 ceph-mon[39278]: 10.d scrub ok 2024-02-05T08:21:04.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:04.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:21:04.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:04 smithi196 ceph-mon[33018]: 3.12 scrub starts 2024-02-05T08:21:04.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:04 smithi196 ceph-mon[33018]: 3.12 scrub ok 2024-02-05T08:21:04.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:04 smithi196 ceph-mon[33018]: 10.d scrub ok 2024-02-05T08:21:04.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:04.829 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:04.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:21:04.906 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:21:04.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:04.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:04.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:05.071 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:05.099 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:05.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:05.108 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:21:05.201 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:05.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:05.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:05.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:05.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:05.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:05.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:05.509 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:05.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:05.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:05.634 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:05.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:05.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:05.736 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:05.771 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:05.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:05.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:05.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:05 smithi134 ceph-mon[39278]: pgmap v729: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 934 KiB/s wr, 1.20k op/s; 21953/353775 objects degraded (6.205%) 2024-02-05T08:21:05.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:05 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:05.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:05 smithi134 ceph-mon[39278]: 10.e scrub starts 2024-02-05T08:21:05.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:05 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 21953/353775 objects degraded (6.205%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:05.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:05 smithi134 ceph-mon[30086]: pgmap v729: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 934 KiB/s wr, 1.20k op/s; 21953/353775 objects degraded (6.205%) 2024-02-05T08:21:05.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:05 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:05.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:05 smithi134 ceph-mon[30086]: 10.e scrub starts 2024-02-05T08:21:05.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:05 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 21953/353775 objects degraded (6.205%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:05 smithi196 ceph-mon[33018]: pgmap v729: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 934 KiB/s wr, 1.20k op/s; 21953/353775 objects degraded (6.205%) 2024-02-05T08:21:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:05 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:05 smithi196 ceph-mon[33018]: 10.e scrub starts 2024-02-05T08:21:05.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:05 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 21953/353775 objects degraded (6.205%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:05.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:05.908 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:05.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:05.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:06.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:06.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:06.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:06.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:06.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:06.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:06.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:06.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:06.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:06.328 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:06.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:06.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:06.470 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:06.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:06.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:06.559 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:06 smithi134 ceph-mon[30086]: 1.5 scrub starts 2024-02-05T08:21:06.559 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:06 smithi134 ceph-mon[30086]: 1.5 scrub ok 2024-02-05T08:21:06.559 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:06 smithi134 ceph-mon[39278]: 1.5 scrub starts 2024-02-05T08:21:06.559 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:06 smithi134 ceph-mon[39278]: 1.5 scrub ok 2024-02-05T08:21:06.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:06.607 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:06.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:06.682 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:06.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:06.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:06.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:06.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:06.754 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:06 smithi196 ceph-mon[33018]: 1.5 scrub starts 2024-02-05T08:21:06.754 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:06 smithi196 ceph-mon[33018]: 1.5 scrub ok 2024-02-05T08:21:06.844 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:06.880 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:06.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:06.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:06.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:21:06.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:07.037 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:07.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:07.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:07.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:07.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:07.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:07.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:07.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:07.314 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:07.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:07.416 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:07.438 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:21:07.443 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:21:07.456 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:07.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:07.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:07.558 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:07.592 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:07.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:07.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:07.694 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:07.730 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:07.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:07.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:07.752 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[30086]: pgmap v730: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 21953/351552 objects degraded (6.245%) 2024-02-05T08:21:07.752 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[30086]: 3.2 scrub starts 2024-02-05T08:21:07.752 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[30086]: 3.2 scrub ok 2024-02-05T08:21:07.752 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[30086]: 10.e scrub ok 2024-02-05T08:21:07.752 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[30086]: 10.7 scrub starts 2024-02-05T08:21:07.752 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[30086]: 1.8 scrub starts 2024-02-05T08:21:07.752 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[30086]: 1.8 scrub ok 2024-02-05T08:21:07.752 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[39278]: pgmap v730: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 21953/351552 objects degraded (6.245%) 2024-02-05T08:21:07.753 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[39278]: 3.2 scrub starts 2024-02-05T08:21:07.753 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[39278]: 3.2 scrub ok 2024-02-05T08:21:07.753 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[39278]: 10.e scrub ok 2024-02-05T08:21:07.753 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[39278]: 10.7 scrub starts 2024-02-05T08:21:07.753 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[39278]: 1.8 scrub starts 2024-02-05T08:21:07.753 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-mon[39278]: 1.8 scrub ok 2024-02-05T08:21:07.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:07 smithi196 ceph-mon[33018]: pgmap v730: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 21953/351552 objects degraded (6.245%) 2024-02-05T08:21:07.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:07 smithi196 ceph-mon[33018]: 3.2 scrub starts 2024-02-05T08:21:07.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:07 smithi196 ceph-mon[33018]: 3.2 scrub ok 2024-02-05T08:21:07.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:07 smithi196 ceph-mon[33018]: 10.e scrub ok 2024-02-05T08:21:07.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:07 smithi196 ceph-mon[33018]: 10.7 scrub starts 2024-02-05T08:21:07.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:07 smithi196 ceph-mon[33018]: 1.8 scrub starts 2024-02-05T08:21:07.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:07 smithi196 ceph-mon[33018]: 1.8 scrub ok 2024-02-05T08:21:07.832 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:07.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:07.868 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:07.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:07.969 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:08.006 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:08.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:08.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:21:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:21:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:21:08.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:08.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:08.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:08.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:08.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:08.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:08.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:08.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:08.405 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:08.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:08.506 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:08.544 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:08.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:08.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:08.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:08.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:08.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:08.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:08.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:08 smithi134 ceph-mon[30086]: 5.1c scrub starts 2024-02-05T08:21:08.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:08 smithi134 ceph-mon[30086]: 5.1c scrub ok 2024-02-05T08:21:08.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:08 smithi134 ceph-mon[30086]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:21:08.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:08 smithi134 ceph-mon[30086]: 5.2 scrub starts 2024-02-05T08:21:08.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:08 smithi134 ceph-mon[30086]: 5.2 scrub ok 2024-02-05T08:21:08.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:21:08.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:08 smithi134 ceph-mon[39278]: 5.1c scrub starts 2024-02-05T08:21:08.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:08 smithi134 ceph-mon[39278]: 5.1c scrub ok 2024-02-05T08:21:08.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:08 smithi134 ceph-mon[39278]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:21:08.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:08 smithi134 ceph-mon[39278]: 5.2 scrub starts 2024-02-05T08:21:08.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:08 smithi134 ceph-mon[39278]: 5.2 scrub ok 2024-02-05T08:21:08.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:21:08.783 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":730,"stamp":"2024-02-05T08:21:05.819731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7667378970,"num_objects":117184,"num_object_clones":0,"num_object_copies":351552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117184,"num_whiteouts":0,"num_read":2127,"num_read_kb":1887,"num_write":146673,"num_write_kb":8413642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":221172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87110,"ondisk_log_size":87110,"up":416,"acting":416,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":26185620,"kb_used_data":21767044,"kb_used_omap":12,"kb_used_meta":4418419,"kb_avail":723775596,"statfs":{"total":767960285184,"available":741146210304,"internally_reserved":0,"allocated":22289453056,"data_stored":22286852099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":4524461619},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,2,10,6,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-800849920,"num_objects":-12220,"num_object_clones":0,"num_object_copies":-36660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2380,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12220,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12220,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004867"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":603,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.488262+0000","last_change":"2024-02-05T08:19:44.912725+0000","last_active":"2024-02-05T08:21:01.488262+0000","last_peered":"2024-02-05T08:21:01.488262+0000","last_clean":"2024-02-05T08:19:41.829565+0000","last_became_active":"2024-02-05T08:19:44.912725+0000","last_became_peered":"2024-02-05T08:19:44.912725+0000","last_unstale":"2024-02-05T08:21:01.488262+0000","last_undegraded":"2024-02-05T08:21:01.488262+0000","last_fullsized":"2024-02-05T08:19:44.874568+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:36.296981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:36.296981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135681+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":34,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.502042+0000","last_change":"2024-02-05T08:19:44.903097+0000","last_active":"2024-02-05T08:21:01.502042+0000","last_peered":"2024-02-05T08:21:01.502042+0000","last_clean":"2024-02-05T08:04:44.536163+0000","last_became_active":"2024-02-05T08:19:44.903097+0000","last_became_peered":"2024-02-05T08:19:44.903097+0000","last_unstale":"2024-02-05T08:21:01.502042+0000","last_undegraded":"2024-02-05T08:21:01.502042+0000","last_fullsized":"2024-02-05T08:19:44.873443+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.301502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.301502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":604,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.493875+0000","last_change":"2024-02-05T08:19:25.569614+0000","last_active":"2024-02-05T08:21:01.493875+0000","last_peered":"2024-02-05T08:21:01.493875+0000","last_clean":"2024-02-05T08:21:01.493875+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:21:01.493875+0000","last_undegraded":"2024-02-05T08:21:01.493875+0000","last_fullsized":"2024-02-05T08:21:01.493875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:25.569504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:25.569504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.355083+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"236'19","reported_seq":664,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.491330+0000","last_change":"2024-02-05T08:19:44.920275+0000","last_active":"2024-02-05T08:21:01.491330+0000","last_peered":"2024-02-05T08:21:01.491330+0000","last_clean":"2024-02-05T08:19:41.830039+0000","last_became_active":"2024-02-05T08:19:44.920275+0000","last_became_peered":"2024-02-05T08:19:44.920275+0000","last_unstale":"2024-02-05T08:21:01.491330+0000","last_undegraded":"2024-02-05T08:19:44.874580+0000","last_fullsized":"2024-02-05T08:19:44.874470+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:01.602887+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:18:01.602887+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135378+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":103,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.492747+0000","last_change":"2024-02-05T08:20:17.809613+0000","last_active":"2024-02-05T08:21:01.492747+0000","last_peered":"2024-02-05T08:21:01.492747+0000","last_clean":"2024-02-05T08:21:01.492747+0000","last_became_active":"2024-02-05T08:18:34.633083+0000","last_became_peered":"2024-02-05T08:18:34.633083+0000","last_unstale":"2024-02-05T08:21:01.492747+0000","last_undegraded":"2024-02-05T08:21:01.492747+0000","last_fullsized":"2024-02-05T08:21:01.492747+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:17.809200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:20:17.809200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:34.584011+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":611,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.487936+0000","last_change":"2024-02-05T08:19:44.919417+0000","last_active":"2024-02-05T08:21:01.487936+0000","last_peered":"2024-02-05T08:21:01.487936+0000","last_clean":"2024-02-05T08:19:41.831283+0000","last_became_active":"2024-02-05T08:19:44.919417+0000","last_became_peered":"2024-02-05T08:19:44.919417+0000","last_unstale":"2024-02-05T08:21:01.487936+0000","last_undegraded":"2024-02-05T08:21:01.487936+0000","last_fullsized":"2024-02-05T08:19:44.874665+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:39.536022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:18:39.536022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135452+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":107,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.492147+0000","last_change":"2024-02-05T08:19:44.927236+0000","last_active":"2024-02-05T08:21:01.492147+0000","last_peered":"2024-02-05T08:21:01.492147+0000","last_clean":"2024-02-05T08:19:41.830712+0000","last_became_active":"2024-02-05T08:19:44.927236+0000","last_became_peered":"2024-02-05T08:19:44.927236+0000","last_unstale":"2024-02-05T08:21:01.492147+0000","last_undegraded":"2024-02-05T08:21:01.492147+0000","last_fullsized":"2024-02-05T08:19:44.887381+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:47.183986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:18:47.183986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"236'34","reported_seq":128,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.490348+0000","last_change":"2024-02-05T08:20:29.672533+0000","last_active":"2024-02-05T08:21:01.490348+0000","last_peered":"2024-02-05T08:21:01.490348+0000","last_clean":"2024-02-05T08:21:01.490348+0000","last_became_active":"2024-02-05T08:18:34.649433+0000","last_became_peered":"2024-02-05T08:18:34.649433+0000","last_unstale":"2024-02-05T08:21:01.490348+0000","last_undegraded":"2024-02-05T08:21:01.490348+0000","last_fullsized":"2024-02-05T08:21:01.490348+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:20:29.672381+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:20:29.672381+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:42.459911+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":594,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.493810+0000","last_change":"2024-02-05T08:19:44.902206+0000","last_active":"2024-02-05T08:21:01.493810+0000","last_peered":"2024-02-05T08:21:01.493810+0000","last_clean":"2024-02-05T08:19:41.834390+0000","last_became_active":"2024-02-05T08:19:44.902206+0000","last_became_peered":"2024-02-05T08:19:44.902206+0000","last_unstale":"2024-02-05T08:21:01.493810+0000","last_undegraded":"2024-02-05T08:21:01.493810+0000","last_fullsized":"2024-02-05T08:19:44.885918+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:21.661930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:21.661930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.355058+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":604,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.501475+0000","last_change":"2024-02-05T08:19:21.500610+0000","last_active":"2024-02-05T08:21:01.501475+0000","last_peered":"2024-02-05T08:21:01.501475+0000","last_clean":"2024-02-05T08:21:01.501475+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:21:01.501475+0000","last_undegraded":"2024-02-05T08:21:01.501475+0000","last_fullsized":"2024-02-05T08:21:01.501475+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:21.500492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:19:21.500492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319370+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":632,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.502118+0000","last_change":"2024-02-05T08:19:32.421512+0000","last_active":"2024-02-05T08:21:01.502118+0000","last_peered":"2024-02-05T08:21:01.502118+0000","last_clean":"2024-02-05T08:21:01.502118+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:21:01.502118+0000","last_undegraded":"2024-02-05T08:21:01.502118+0000","last_fullsized":"2024-02-05T08:21:01.502118+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_clean_scrub_stamp":"2024-02-05T08:19:32.421388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319395+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"263'19","reported_seq":675,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.488572+0000","last_change":"2024-02-05T08:20:19.142437+0000","last_active":"2024-02-05T08:21:01.488572+0000","last_peered":"2024-02-05T08:21:01.488572+0000","last_clean":"2024-02-05T08:21:01.488572+0000","last_became_active":"2024-02-05T08:18:34.636165+0000","last_became_peered":"2024-02-05T08:18:34.636165+0000","last_unstale":"2024-02-05T08:21:01.488572+0000","last_undegraded":"2024-02-05T08:21:01.488572+0000","last_fullsized":"2024-02-05T08:21:01.488572+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:20:19.142175+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.142175+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:27.368545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":607,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.502231+0000","last_change":"2024-02-05T08:19:44.901170+0000","last_active":"2024-02-05T08:21:01.502231+0000","last_peered":"2024-02-05T08:21:01.502231+0000","last_clean":"2024-02-05T08:19:41.833929+0000","last_became_active":"2024-02-05T08:19:44.901170+0000","last_became_peered":"2024-02-05T08:19:44.901170+0000","last_unstale":"2024-02-05T08:21:01.502231+0000","last_undegraded":"2024-02-05T08:21:01.502231+0000","last_fullsized":"2024-02-05T08:19:44.873575+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:32.611856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:32.611856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319152+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":608,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.496223+0000","last_change":"2024-02-05T08:19:44.921416+0000","last_active":"2024-02-05T08:21:01.496223+0000","last_peered":"2024-02-05T08:21:01.496223+0000","last_clean":"2024-02-05T08:19:41.834434+0000","last_became_active":"2024-02-05T08:19:44.921416+0000","last_became_peered":"2024-02-05T08:19:44.921416+0000","last_unstale":"2024-02-05T08:21:01.496223+0000","last_undegraded":"2024-02-05T08:21:01.496223+0000","last_fullsized":"2024-02-05T08:19:44.875209+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:40.260248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:40.260248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:05.017002+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":615,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.501490+0000","last_change":"2024-02-05T08:19:44.916712+0000","last_active":"2024-02-05T08:21:01.501490+0000","last_peered":"2024-02-05T08:21:01.501490+0000","last_clean":"2024-02-05T08:19:41.831063+0000","last_became_active":"2024-02-05T08:19:44.916712+0000","last_became_peered":"2024-02-05T08:19:44.916712+0000","last_unstale":"2024-02-05T08:21:01.501490+0000","last_undegraded":"2024-02-05T08:21:01.501490+0000","last_fullsized":"2024-02-05T08:19:44.898786+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:14.458198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.458198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":640,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.488359+0000","last_change":"2024-02-05T08:19:44.913761+0000","last_active":"2024-02-05T08:21:01.488359+0000","last_peered":"2024-02-05T08:21:01.488359+0000","last_clean":"2024-02-05T08:19:42.900179+0000","last_became_active":"2024-02-05T08:19:44.913761+0000","last_became_peered":"2024-02-05T08:19:44.913761+0000","last_unstale":"2024-02-05T08:21:01.488359+0000","last_undegraded":"2024-02-05T08:19:44.909323+0000","last_fullsized":"2024-02-05T08:19:44.909205+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-02-05T08:18:05.582303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:05.582303+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135392+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":614,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495281+0000","last_change":"2024-02-05T08:20:16.267258+0000","last_active":"2024-02-05T08:21:01.495281+0000","last_peered":"2024-02-05T08:21:01.495281+0000","last_clean":"2024-02-05T08:21:01.495281+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:21:01.495281+0000","last_undegraded":"2024-02-05T08:21:01.495281+0000","last_fullsized":"2024-02-05T08:21:01.495281+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:16.266908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:20:16.266908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:24.672751+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":34,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.489267+0000","last_change":"2024-02-05T08:19:44.920889+0000","last_active":"2024-02-05T08:21:01.489267+0000","last_peered":"2024-02-05T08:21:01.489267+0000","last_clean":"2024-02-05T08:04:44.536163+0000","last_became_active":"2024-02-05T08:19:44.920889+0000","last_became_peered":"2024-02-05T08:19:44.920889+0000","last_unstale":"2024-02-05T08:21:01.489267+0000","last_undegraded":"2024-02-05T08:21:01.489267+0000","last_fullsized":"2024-02-05T08:19:44.913274+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_clean_scrub_stamp":"2024-02-05T08:19:36.737342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":603,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495101+0000","last_change":"2024-02-05T08:20:31.566305+0000","last_active":"2024-02-05T08:21:01.495101+0000","last_peered":"2024-02-05T08:21:01.495101+0000","last_clean":"2024-02-05T08:21:01.495101+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:21:01.495101+0000","last_undegraded":"2024-02-05T08:21:01.495101+0000","last_fullsized":"2024-02-05T08:21:01.495101+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:31.566207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:31.566207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:41.126178+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"236'34","reported_seq":721,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.495971+0000","last_change":"2024-02-05T08:19:44.892882+0000","last_active":"2024-02-05T08:21:01.495971+0000","last_peered":"2024-02-05T08:21:01.495971+0000","last_clean":"2024-02-05T08:19:41.827204+0000","last_became_active":"2024-02-05T08:19:44.892882+0000","last_became_peered":"2024-02-05T08:19:44.892882+0000","last_unstale":"2024-02-05T08:21:01.495971+0000","last_undegraded":"2024-02-05T08:19:44.885361+0000","last_fullsized":"2024-02-05T08:19:44.885220+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:18:47.576225+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:18:47.576225+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":104,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495591+0000","last_change":"2024-02-05T08:20:16.870703+0000","last_active":"2024-02-05T08:21:01.495591+0000","last_peered":"2024-02-05T08:21:01.495591+0000","last_clean":"2024-02-05T08:21:01.495591+0000","last_became_active":"2024-02-05T08:18:34.642494+0000","last_became_peered":"2024-02-05T08:18:34.642494+0000","last_unstale":"2024-02-05T08:21:01.495591+0000","last_undegraded":"2024-02-05T08:21:01.495591+0000","last_fullsized":"2024-02-05T08:21:01.495591+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:16.870561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_clean_scrub_stamp":"2024-02-05T08:20:16.870561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:34.793930+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":616,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.490383+0000","last_change":"2024-02-05T08:19:12.320431+0000","last_active":"2024-02-05T08:21:01.490383+0000","last_peered":"2024-02-05T08:21:01.490383+0000","last_clean":"2024-02-05T08:21:01.490383+0000","last_became_active":"2024-02-05T08:18:34.638652+0000","last_became_peered":"2024-02-05T08:18:34.638652+0000","last_unstale":"2024-02-05T08:21:01.490383+0000","last_undegraded":"2024-02-05T08:21:01.490383+0000","last_fullsized":"2024-02-05T08:21:01.490383+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:12.320317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:19:12.320317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":607,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.488850+0000","last_change":"2024-02-05T08:19:44.912456+0000","last_active":"2024-02-05T08:21:01.488850+0000","last_peered":"2024-02-05T08:21:01.488850+0000","last_clean":"2024-02-05T08:19:42.900163+0000","last_became_active":"2024-02-05T08:19:44.912456+0000","last_became_peered":"2024-02-05T08:19:44.912456+0000","last_unstale":"2024-02-05T08:21:01.488850+0000","last_undegraded":"2024-02-05T08:21:01.488850+0000","last_fullsized":"2024-02-05T08:19:44.874216+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:19.338699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:19:19.338699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135569+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"236'23","reported_seq":110,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.492410+0000","last_change":"2024-02-05T08:19:44.923092+0000","last_active":"2024-02-05T08:21:01.492410+0000","last_peered":"2024-02-05T08:21:01.492410+0000","last_clean":"2024-02-05T08:19:41.835633+0000","last_became_active":"2024-02-05T08:19:44.923092+0000","last_became_peered":"2024-02-05T08:19:44.923092+0000","last_unstale":"2024-02-05T08:21:01.492410+0000","last_undegraded":"2024-02-05T08:19:44.886225+0000","last_fullsized":"2024-02-05T08:19:44.886101+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'23","last_scrub_stamp":"2024-02-05T08:17:58.012827+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:17:58.012827+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780667+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":619,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.497141+0000","last_change":"2024-02-05T08:20:37.119269+0000","last_active":"2024-02-05T08:21:01.497141+0000","last_peered":"2024-02-05T08:21:01.497141+0000","last_clean":"2024-02-05T08:21:01.497141+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:21:01.497141+0000","last_undegraded":"2024-02-05T08:21:01.497141+0000","last_fullsized":"2024-02-05T08:21:01.497141+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:20:37.119142+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.119142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:41.418621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":458,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.493892+0000","last_change":"2024-02-05T08:20:17.417043+0000","last_active":"2024-02-05T08:21:01.493892+0000","last_peered":"2024-02-05T08:21:01.493892+0000","last_clean":"2024-02-05T08:21:01.493892+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:21:01.493892+0000","last_undegraded":"2024-02-05T08:21:01.493892+0000","last_fullsized":"2024-02-05T08:21:01.493892+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:20:17.416907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:17.416907+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:34.985509+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":561,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.489854+0000","last_change":"2024-02-05T08:19:44.915101+0000","last_active":"2024-02-05T08:21:01.489854+0000","last_peered":"2024-02-05T08:21:01.489854+0000","last_clean":"2024-02-05T08:18:29.609487+0000","last_became_active":"2024-02-05T08:19:44.915101+0000","last_became_peered":"2024-02-05T08:19:44.915101+0000","last_unstale":"2024-02-05T08:21:01.489854+0000","last_undegraded":"2024-02-05T08:19:44.874435+0000","last_fullsized":"2024-02-05T08:19:44.874324+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:19:28.743054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:28.743054+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780945+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"236'20","reported_seq":703,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.496668+0000","last_change":"2024-02-05T08:19:35.669211+0000","last_active":"2024-02-05T08:21:01.496668+0000","last_peered":"2024-02-05T08:21:01.496668+0000","last_clean":"2024-02-05T08:21:01.496668+0000","last_became_active":"2024-02-05T08:18:34.645441+0000","last_became_peered":"2024-02-05T08:18:34.645441+0000","last_unstale":"2024-02-05T08:21:01.496668+0000","last_undegraded":"2024-02-05T08:21:01.496668+0000","last_fullsized":"2024-02-05T08:21:01.496668+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:19:35.668857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:35.668857+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.355232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":33,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.501752+0000","last_change":"2024-02-05T08:19:44.923094+0000","last_active":"2024-02-05T08:21:01.501752+0000","last_peered":"2024-02-05T08:21:01.501752+0000","last_clean":"2024-02-05T08:04:46.546354+0000","last_became_active":"2024-02-05T08:19:44.923094+0000","last_became_peered":"2024-02-05T08:19:44.923094+0000","last_unstale":"2024-02-05T08:21:01.501752+0000","last_undegraded":"2024-02-05T08:21:01.501752+0000","last_fullsized":"2024-02-05T08:19:44.897835+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:46.550985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:46.550985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":597,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.496906+0000","last_change":"2024-02-05T08:19:32.665907+0000","last_active":"2024-02-05T08:21:01.496906+0000","last_peered":"2024-02-05T08:21:01.496906+0000","last_clean":"2024-02-05T08:21:01.496906+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:21:01.496906+0000","last_undegraded":"2024-02-05T08:21:01.496906+0000","last_fullsized":"2024-02-05T08:21:01.496906+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:32.665558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:32.231861+0000","last_clean_scrub_stamp":"2024-02-05T08:19:32.665558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.355121+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":107,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.489951+0000","last_change":"2024-02-05T08:19:44.892769+0000","last_active":"2024-02-05T08:21:01.489951+0000","last_peered":"2024-02-05T08:21:01.489951+0000","last_clean":"2024-02-05T08:19:41.827910+0000","last_became_active":"2024-02-05T08:19:44.892769+0000","last_became_peered":"2024-02-05T08:19:44.892769+0000","last_unstale":"2024-02-05T08:21:01.489951+0000","last_undegraded":"2024-02-05T08:21:01.489951+0000","last_fullsized":"2024-02-05T08:19:44.874377+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:11.049460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:19:11.049460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"236'19","reported_seq":367,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.493735+0000","last_change":"2024-02-05T08:19:34.457468+0000","last_active":"2024-02-05T08:21:01.493735+0000","last_peered":"2024-02-05T08:21:01.493735+0000","last_clean":"2024-02-05T08:21:01.493735+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:21:01.493735+0000","last_undegraded":"2024-02-05T08:21:01.493735+0000","last_fullsized":"2024-02-05T08:21:01.493735+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:19:34.457012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:34.457012+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":593,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.496475+0000","last_change":"2024-02-05T08:19:26.157804+0000","last_active":"2024-02-05T08:21:01.496475+0000","last_peered":"2024-02-05T08:21:01.496475+0000","last_clean":"2024-02-05T08:21:01.496475+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:21:01.496475+0000","last_undegraded":"2024-02-05T08:21:01.496475+0000","last_fullsized":"2024-02-05T08:21:01.496475+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:26.157698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:26.157698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:05.016827+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":345,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495128+0000","last_change":"2024-02-05T08:20:27.591858+0000","last_active":"2024-02-05T08:21:01.495128+0000","last_peered":"2024-02-05T08:21:01.495128+0000","last_clean":"2024-02-05T08:21:01.495128+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:21:01.495128+0000","last_undegraded":"2024-02-05T08:21:01.495128+0000","last_fullsized":"2024-02-05T08:21:01.495128+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:27.591651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:20:27.591651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:42.110964+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":615,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.494074+0000","last_change":"2024-02-05T08:19:34.697406+0000","last_active":"2024-02-05T08:21:01.494074+0000","last_peered":"2024-02-05T08:21:01.494074+0000","last_clean":"2024-02-05T08:21:01.494074+0000","last_became_active":"2024-02-05T08:18:34.644537+0000","last_became_peered":"2024-02-05T08:18:34.644537+0000","last_unstale":"2024-02-05T08:21:01.494074+0000","last_undegraded":"2024-02-05T08:21:01.494074+0000","last_fullsized":"2024-02-05T08:21:01.494074+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:34.697248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:34.697248+0000","last_clean_scrub_stamp":"2024-02-05T08:19:34.697248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.355203+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":113,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.490291+0000","last_change":"2024-02-05T08:19:44.911502+0000","last_active":"2024-02-05T08:21:01.490291+0000","last_peered":"2024-02-05T08:21:01.490291+0000","last_clean":"2024-02-05T08:19:41.828151+0000","last_became_active":"2024-02-05T08:19:44.911502+0000","last_became_peered":"2024-02-05T08:19:44.911502+0000","last_unstale":"2024-02-05T08:21:01.490291+0000","last_undegraded":"2024-02-05T08:19:44.874110+0000","last_fullsized":"2024-02-05T08:19:44.873994+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:19:23.023541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:23.023541+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780757+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":112,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.492660+0000","last_change":"2024-02-05T08:20:26.891410+0000","last_active":"2024-02-05T08:21:01.492660+0000","last_peered":"2024-02-05T08:21:01.492660+0000","last_clean":"2024-02-05T08:21:01.492660+0000","last_became_active":"2024-02-05T08:18:34.651979+0000","last_became_peered":"2024-02-05T08:18:34.651979+0000","last_unstale":"2024-02-05T08:21:01.492660+0000","last_undegraded":"2024-02-05T08:21:01.492660+0000","last_fullsized":"2024-02-05T08:21:01.492660+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:26.891040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:26.891040+0000","last_clean_scrub_stamp":"2024-02-05T08:20:26.891040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:47.586543+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":625,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.496100+0000","last_change":"2024-02-05T08:20:18.147831+0000","last_active":"2024-02-05T08:21:01.496100+0000","last_peered":"2024-02-05T08:21:01.496100+0000","last_clean":"2024-02-05T08:21:01.496100+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:21:01.496100+0000","last_undegraded":"2024-02-05T08:21:01.496100+0000","last_fullsized":"2024-02-05T08:21:01.496100+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:18.147762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:20:18.147762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:27.942302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":615,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.502838+0000","last_change":"2024-02-05T08:19:44.904157+0000","last_active":"2024-02-05T08:21:01.502838+0000","last_peered":"2024-02-05T08:21:01.502838+0000","last_clean":"2024-02-05T08:19:41.832352+0000","last_became_active":"2024-02-05T08:19:44.904157+0000","last_became_peered":"2024-02-05T08:19:44.904157+0000","last_unstale":"2024-02-05T08:21:01.502838+0000","last_undegraded":"2024-02-05T08:21:01.502838+0000","last_fullsized":"2024-02-05T08:19:44.901922+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:41.576882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:18:41.576882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"236'24","reported_seq":116,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.491129+0000","last_change":"2024-02-05T08:20:11.675925+0000","last_active":"2024-02-05T08:21:01.491129+0000","last_peered":"2024-02-05T08:21:01.491129+0000","last_clean":"2024-02-05T08:21:01.491129+0000","last_became_active":"2024-02-05T08:18:34.631912+0000","last_became_peered":"2024-02-05T08:18:34.631912+0000","last_unstale":"2024-02-05T08:21:01.491129+0000","last_undegraded":"2024-02-05T08:21:01.491129+0000","last_fullsized":"2024-02-05T08:21:01.491129+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:20:11.675780+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:20:11.675780+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:35.282973+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":594,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.493557+0000","last_change":"2024-02-05T08:19:44.902556+0000","last_active":"2024-02-05T08:21:01.493557+0000","last_peered":"2024-02-05T08:21:01.493557+0000","last_clean":"2024-02-05T08:19:41.834123+0000","last_became_active":"2024-02-05T08:19:44.902556+0000","last_became_peered":"2024-02-05T08:19:44.902556+0000","last_unstale":"2024-02-05T08:21:01.493557+0000","last_undegraded":"2024-02-05T08:21:01.493557+0000","last_fullsized":"2024-02-05T08:19:44.874049+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:26.593364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:19:26.593364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.355172+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":107,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.491830+0000","last_change":"2024-02-05T08:19:44.892864+0000","last_active":"2024-02-05T08:21:01.491830+0000","last_peered":"2024-02-05T08:21:01.491830+0000","last_clean":"2024-02-05T08:19:41.831557+0000","last_became_active":"2024-02-05T08:19:44.892864+0000","last_became_peered":"2024-02-05T08:19:44.892864+0000","last_unstale":"2024-02-05T08:21:01.491830+0000","last_undegraded":"2024-02-05T08:21:01.491830+0000","last_fullsized":"2024-02-05T08:19:44.886732+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:44.133926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:44.133926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":625,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.497193+0000","last_change":"2024-02-05T08:20:16.189809+0000","last_active":"2024-02-05T08:21:01.497193+0000","last_peered":"2024-02-05T08:21:01.497193+0000","last_clean":"2024-02-05T08:21:01.497193+0000","last_became_active":"2024-02-05T08:18:34.642387+0000","last_became_peered":"2024-02-05T08:18:34.642387+0000","last_unstale":"2024-02-05T08:21:01.497193+0000","last_undegraded":"2024-02-05T08:21:01.497193+0000","last_fullsized":"2024-02-05T08:21:01.497193+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:16.189515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:16.189515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:24.853924+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"263'19","reported_seq":280,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.501630+0000","last_change":"2024-02-05T08:20:19.330332+0000","last_active":"2024-02-05T08:21:01.501630+0000","last_peered":"2024-02-05T08:21:01.501630+0000","last_clean":"2024-02-05T08:21:01.501630+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:21:01.501630+0000","last_undegraded":"2024-02-05T08:21:01.501630+0000","last_fullsized":"2024-02-05T08:21:01.501630+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:20:19.330198+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.330198+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:41.061827+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":112,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.490500+0000","last_change":"2024-02-05T08:20:23.812466+0000","last_active":"2024-02-05T08:21:01.490500+0000","last_peered":"2024-02-05T08:21:01.490500+0000","last_clean":"2024-02-05T08:21:01.490500+0000","last_became_active":"2024-02-05T08:18:34.650359+0000","last_became_peered":"2024-02-05T08:18:34.650359+0000","last_unstale":"2024-02-05T08:21:01.490500+0000","last_undegraded":"2024-02-05T08:21:01.490500+0000","last_fullsized":"2024-02-05T08:21:01.490500+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:23.812353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:20:23.812353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:29.490140+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":548,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.502433+0000","last_change":"2024-02-05T08:19:44.903986+0000","last_active":"2024-02-05T08:21:01.502433+0000","last_peered":"2024-02-05T08:21:01.502433+0000","last_clean":"2024-02-05T08:18:29.613784+0000","last_became_active":"2024-02-05T08:19:44.903986+0000","last_became_peered":"2024-02-05T08:19:44.903986+0000","last_unstale":"2024-02-05T08:21:01.502433+0000","last_undegraded":"2024-02-05T08:21:01.502433+0000","last_fullsized":"2024-02-05T08:19:44.899948+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:17.633650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.633650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":632,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.490018+0000","last_change":"2024-02-05T08:20:20.151509+0000","last_active":"2024-02-05T08:21:01.490018+0000","last_peered":"2024-02-05T08:21:01.490018+0000","last_clean":"2024-02-05T08:21:01.490018+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:21:01.490018+0000","last_undegraded":"2024-02-05T08:21:01.490018+0000","last_fullsized":"2024-02-05T08:21:01.490018+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:20.151160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:20:20.151160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:40.580006+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"249'24","reported_seq":683,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.488626+0000","last_change":"2024-02-05T08:19:44.930725+0000","last_active":"2024-02-05T08:21:01.488626+0000","last_peered":"2024-02-05T08:21:01.488626+0000","last_clean":"2024-02-05T08:19:41.830358+0000","last_became_active":"2024-02-05T08:19:44.930725+0000","last_became_peered":"2024-02-05T08:19:44.930725+0000","last_unstale":"2024-02-05T08:21:01.488626+0000","last_undegraded":"2024-02-05T08:19:44.912297+0000","last_fullsized":"2024-02-05T08:19:44.910005+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:19:27.349541+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:19:27.349541+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":603,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.494453+0000","last_change":"2024-02-05T08:20:29.732543+0000","last_active":"2024-02-05T08:21:01.494453+0000","last_peered":"2024-02-05T08:21:01.494453+0000","last_clean":"2024-02-05T08:21:01.494453+0000","last_became_active":"2024-02-05T08:18:34.645027+0000","last_became_peered":"2024-02-05T08:18:34.645027+0000","last_unstale":"2024-02-05T08:21:01.494453+0000","last_undegraded":"2024-02-05T08:21:01.494453+0000","last_fullsized":"2024-02-05T08:21:01.494453+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:29.732422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:29.732422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:58.490572+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"325'9","reported_seq":813,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.502300+0000","last_change":"2024-02-05T08:19:44.900658+0000","last_active":"2024-02-05T08:21:01.502300+0000","last_peered":"2024-02-05T08:21:01.502300+0000","last_clean":"2024-02-05T08:19:41.831639+0000","last_became_active":"2024-02-05T08:19:44.900658+0000","last_became_peered":"2024-02-05T08:19:44.900658+0000","last_unstale":"2024-02-05T08:21:01.502300+0000","last_undegraded":"2024-02-05T08:19:44.898660+0000","last_fullsized":"2024-02-05T08:19:44.898552+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:19:12.452237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:12.452237+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":614,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.496672+0000","last_change":"2024-02-05T08:19:25.118455+0000","last_active":"2024-02-05T08:21:01.496672+0000","last_peered":"2024-02-05T08:21:01.496672+0000","last_clean":"2024-02-05T08:21:01.496672+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:21:01.496672+0000","last_undegraded":"2024-02-05T08:21:01.496672+0000","last_fullsized":"2024-02-05T08:21:01.496672+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:25.118336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:25.118336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:05.016799+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":119,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.494741+0000","last_change":"2024-02-05T08:19:44.899031+0000","last_active":"2024-02-05T08:21:01.494741+0000","last_peered":"2024-02-05T08:21:01.494741+0000","last_clean":"2024-02-05T08:19:41.830913+0000","last_became_active":"2024-02-05T08:19:44.899031+0000","last_became_peered":"2024-02-05T08:19:44.899031+0000","last_unstale":"2024-02-05T08:21:01.494741+0000","last_undegraded":"2024-02-05T08:19:44.887736+0000","last_fullsized":"2024-02-05T08:19:44.887622+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-02-05T08:19:35.981313+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:19:35.981313+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780822+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":595,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.488912+0000","last_change":"2024-02-05T08:19:28.519254+0000","last_active":"2024-02-05T08:21:01.488912+0000","last_peered":"2024-02-05T08:21:01.488912+0000","last_clean":"2024-02-05T08:21:01.488912+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:21:01.488912+0000","last_undegraded":"2024-02-05T08:21:01.488912+0000","last_fullsized":"2024-02-05T08:21:01.488912+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:28.519149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:28.519149+0000","last_clean_scrub_stamp":"2024-02-05T08:19:28.519149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135654+0000","scrub_duration":154,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"295'10","reported_seq":139,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495215+0000","last_change":"2024-02-05T08:20:14.713693+0000","last_active":"2024-02-05T08:21:01.495215+0000","last_peered":"2024-02-05T08:21:01.495215+0000","last_clean":"2024-02-05T08:21:01.495215+0000","last_became_active":"2024-02-05T08:18:34.644517+0000","last_became_peered":"2024-02-05T08:18:34.644517+0000","last_unstale":"2024-02-05T08:21:01.495215+0000","last_undegraded":"2024-02-05T08:21:01.495215+0000","last_fullsized":"2024-02-05T08:21:01.495215+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'10","last_scrub_stamp":"2024-02-05T08:20:14.713589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:14.713589+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:37.047228+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":624,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:03.366506+0000","last_change":"2024-02-05T08:21:03.366506+0000","last_active":"2024-02-05T08:21:03.366506+0000","last_peered":"2024-02-05T08:21:03.366506+0000","last_clean":"2024-02-05T08:21:03.366506+0000","last_became_active":"2024-02-05T08:18:34.635793+0000","last_became_peered":"2024-02-05T08:18:34.635793+0000","last_unstale":"2024-02-05T08:21:03.366506+0000","last_undegraded":"2024-02-05T08:21:03.366506+0000","last_fullsized":"2024-02-05T08:21:03.366506+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:03.366172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:21:03.366172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:10.583764+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":673,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.502533+0000","last_change":"2024-02-05T08:20:23.330119+0000","last_active":"2024-02-05T08:21:01.502533+0000","last_peered":"2024-02-05T08:21:01.502533+0000","last_clean":"2024-02-05T08:21:01.502533+0000","last_became_active":"2024-02-05T08:18:34.640741+0000","last_became_peered":"2024-02-05T08:18:34.640741+0000","last_unstale":"2024-02-05T08:21:01.502533+0000","last_undegraded":"2024-02-05T08:21:01.502533+0000","last_fullsized":"2024-02-05T08:21:01.502533+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:20:23.329973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:20:23.329973+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:50.365153+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":616,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.497795+0000","last_change":"2024-02-05T08:20:24.919443+0000","last_active":"2024-02-05T08:21:01.497795+0000","last_peered":"2024-02-05T08:21:01.497795+0000","last_clean":"2024-02-05T08:21:01.497795+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:21:01.497795+0000","last_undegraded":"2024-02-05T08:21:01.497795+0000","last_fullsized":"2024-02-05T08:21:01.497795+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:24.919305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:24.919305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:33.051917+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":107,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.495889+0000","last_change":"2024-02-05T08:19:44.899825+0000","last_active":"2024-02-05T08:21:01.495889+0000","last_peered":"2024-02-05T08:21:01.495889+0000","last_clean":"2024-02-05T08:19:41.832250+0000","last_became_active":"2024-02-05T08:19:44.899825+0000","last_became_peered":"2024-02-05T08:19:44.899825+0000","last_unstale":"2024-02-05T08:21:01.495889+0000","last_undegraded":"2024-02-05T08:21:01.495889+0000","last_fullsized":"2024-02-05T08:19:44.888998+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:37.028991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:37.028991+0000","last_clean_scrub_stamp":"2024-02-05T08:19:37.028991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780836+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":626,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.497994+0000","last_change":"2024-02-05T08:20:17.150031+0000","last_active":"2024-02-05T08:21:01.497994+0000","last_peered":"2024-02-05T08:21:01.497994+0000","last_clean":"2024-02-05T08:21:01.497994+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:21:01.497994+0000","last_undegraded":"2024-02-05T08:21:01.497994+0000","last_fullsized":"2024-02-05T08:21:01.497994+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:17.149915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:20:17.149915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:32.105130+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"278'19","reported_seq":660,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.496762+0000","last_change":"2024-02-05T08:19:31.636110+0000","last_active":"2024-02-05T08:21:01.496762+0000","last_peered":"2024-02-05T08:21:01.496762+0000","last_clean":"2024-02-05T08:21:01.496762+0000","last_became_active":"2024-02-05T08:18:34.645881+0000","last_became_peered":"2024-02-05T08:18:34.645881+0000","last_unstale":"2024-02-05T08:21:01.496762+0000","last_undegraded":"2024-02-05T08:21:01.496762+0000","last_fullsized":"2024-02-05T08:21:01.496762+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-05T08:19:31.635979+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:19:31.635979+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":613,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.497115+0000","last_change":"2024-02-05T08:19:15.271802+0000","last_active":"2024-02-05T08:21:01.497115+0000","last_peered":"2024-02-05T08:21:01.497115+0000","last_clean":"2024-02-05T08:21:01.497115+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:21:01.497115+0000","last_undegraded":"2024-02-05T08:21:01.497115+0000","last_fullsized":"2024-02-05T08:21:01.497115+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:15.271671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.271671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:05.016898+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":619,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.503169+0000","last_change":"2024-02-05T08:19:44.903925+0000","last_active":"2024-02-05T08:21:01.503169+0000","last_peered":"2024-02-05T08:21:01.503169+0000","last_clean":"2024-02-05T08:19:41.832619+0000","last_became_active":"2024-02-05T08:19:44.903925+0000","last_became_peered":"2024-02-05T08:19:44.903925+0000","last_unstale":"2024-02-05T08:21:01.503169+0000","last_undegraded":"2024-02-05T08:21:01.503169+0000","last_fullsized":"2024-02-05T08:19:44.901051+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.486901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.486901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":342,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.492953+0000","last_change":"2024-02-05T08:19:16.518726+0000","last_active":"2024-02-05T08:21:01.492953+0000","last_peered":"2024-02-05T08:21:01.492953+0000","last_clean":"2024-02-05T08:21:01.492953+0000","last_became_active":"2024-02-05T08:12:29.684444+0000","last_became_peered":"2024-02-05T08:12:29.684444+0000","last_unstale":"2024-02-05T08:21:01.492953+0000","last_undegraded":"2024-02-05T08:21:01.492953+0000","last_fullsized":"2024-02-05T08:21:01.492953+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.518655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.518655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'15","reported_seq":107,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.495486+0000","last_change":"2024-02-05T08:19:44.899823+0000","last_active":"2024-02-05T08:21:01.495486+0000","last_peered":"2024-02-05T08:21:01.495486+0000","last_clean":"2024-02-05T08:19:41.832108+0000","last_became_active":"2024-02-05T08:19:44.899823+0000","last_became_peered":"2024-02-05T08:19:44.899823+0000","last_unstale":"2024-02-05T08:21:01.495486+0000","last_undegraded":"2024-02-05T08:19:44.888775+0000","last_fullsized":"2024-02-05T08:19:44.888665+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:17:54.101749+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.101749+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780651+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"337'9059","reported_seq":9209,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:03.271907+0000","last_change":"2024-02-05T08:20:40.673313+0000","last_active":"2024-02-05T08:21:03.271907+0000","last_peered":"2024-02-05T08:21:03.271907+0000","last_clean":"2024-02-05T08:21:03.271907+0000","last_became_active":"2024-02-05T08:19:07.332783+0000","last_became_peered":"2024-02-05T08:19:07.332783+0000","last_unstale":"2024-02-05T08:21:03.271907+0000","last_undegraded":"2024-02-05T08:21:03.271907+0000","last_fullsized":"2024-02-05T08:21:03.271907+0000","mapping_epoch":309,"log_start":"325'3800","ondisk_log_start":"325'3800","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'7546","last_scrub_stamp":"2024-02-05T08:20:40.673010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:40.673010+0000","objects_scrubbed":7412,"log_size":5259,"log_dups_size":0,"ondisk_log_size":5259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.387460+0000","scrub_duration":4450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470482944,"num_objects":7179,"num_object_clones":0,"num_object_copies":21537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9059,"num_write_kb":519616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":621,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.502592+0000","last_change":"2024-02-05T08:20:13.502530+0000","last_active":"2024-02-05T08:21:01.502592+0000","last_peered":"2024-02-05T08:21:01.502592+0000","last_clean":"2024-02-05T08:21:01.502592+0000","last_became_active":"2024-02-05T08:12:29.679849+0000","last_became_peered":"2024-02-05T08:12:29.679849+0000","last_unstale":"2024-02-05T08:21:01.502592+0000","last_undegraded":"2024-02-05T08:21:01.502592+0000","last_fullsized":"2024-02-05T08:21:01.502592+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:13.502417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:13.502417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:15.250048+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":634,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.497335+0000","last_change":"2024-02-05T08:20:34.173135+0000","last_active":"2024-02-05T08:21:01.497335+0000","last_peered":"2024-02-05T08:21:01.497335+0000","last_clean":"2024-02-05T08:21:01.497335+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:21:01.497335+0000","last_undegraded":"2024-02-05T08:21:01.497335+0000","last_fullsized":"2024-02-05T08:21:01.497335+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:20:34.173063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:34.173063+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:48.768654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"262'34","reported_seq":703,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.501210+0000","last_change":"2024-02-05T08:19:34.364882+0000","last_active":"2024-02-05T08:21:01.501210+0000","last_peered":"2024-02-05T08:21:01.501210+0000","last_clean":"2024-02-05T08:21:01.501210+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:21:01.501210+0000","last_undegraded":"2024-02-05T08:21:01.501210+0000","last_fullsized":"2024-02-05T08:21:01.501210+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'34","last_scrub_stamp":"2024-02-05T08:19:34.364738+0000","last_deep_scrub":"262'34","last_deep_scrub_stamp":"2024-02-05T08:19:34.364738+0000","last_clean_scrub_stamp":"2024-02-05T08:19:34.364738+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319485+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":832,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.498114+0000","last_change":"2024-02-05T08:19:31.117664+0000","last_active":"2024-02-05T08:21:01.498114+0000","last_peered":"2024-02-05T08:21:01.498114+0000","last_clean":"2024-02-05T08:21:01.498114+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:21:01.498114+0000","last_undegraded":"2024-02-05T08:21:01.498114+0000","last_fullsized":"2024-02-05T08:21:01.498114+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_clean_scrub_stamp":"2024-02-05T08:19:31.117523+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:05.016990+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"337'8993","reported_seq":9059,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:02.953437+0000","last_change":"2024-02-05T08:19:07.346353+0000","last_active":"2024-02-05T08:21:02.953437+0000","last_peered":"2024-02-05T08:21:02.953437+0000","last_clean":"2024-02-05T08:21:02.953437+0000","last_became_active":"2024-02-05T08:19:07.346046+0000","last_became_peered":"2024-02-05T08:19:07.346046+0000","last_unstale":"2024-02-05T08:21:02.953437+0000","last_undegraded":"2024-02-05T08:21:02.953437+0000","last_fullsized":"2024-02-05T08:21:02.953437+0000","mapping_epoch":309,"log_start":"325'3400","ondisk_log_start":"325'3400","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":5593,"log_dups_size":0,"ondisk_log_size":5593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":479002624,"num_objects":7309,"num_object_clones":0,"num_object_copies":21927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8993,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":99,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.492822+0000","last_change":"2024-02-05T08:19:44.899425+0000","last_active":"2024-02-05T08:21:01.492822+0000","last_peered":"2024-02-05T08:21:01.492822+0000","last_clean":"2024-02-05T08:19:41.833419+0000","last_became_active":"2024-02-05T08:19:44.899425+0000","last_became_peered":"2024-02-05T08:19:44.899425+0000","last_unstale":"2024-02-05T08:21:01.492822+0000","last_undegraded":"2024-02-05T08:21:01.492822+0000","last_fullsized":"2024-02-05T08:19:44.885939+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:31.245736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:18:31.245736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":617,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.497708+0000","last_change":"2024-02-05T08:19:44.926408+0000","last_active":"2024-02-05T08:21:01.497708+0000","last_peered":"2024-02-05T08:21:01.497708+0000","last_clean":"2024-02-05T08:19:41.835482+0000","last_became_active":"2024-02-05T08:19:44.926408+0000","last_became_peered":"2024-02-05T08:19:44.926408+0000","last_unstale":"2024-02-05T08:21:01.497708+0000","last_undegraded":"2024-02-05T08:21:01.497708+0000","last_fullsized":"2024-02-05T08:19:44.901657+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:37.226829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:19:37.226829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:05.016935+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":672,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.491732+0000","last_change":"2024-02-05T08:19:44.912589+0000","last_active":"2024-02-05T08:21:01.491732+0000","last_peered":"2024-02-05T08:21:01.491732+0000","last_clean":"2024-02-05T08:19:41.830832+0000","last_became_active":"2024-02-05T08:19:44.912589+0000","last_became_peered":"2024-02-05T08:19:44.912589+0000","last_unstale":"2024-02-05T08:21:01.491732+0000","last_undegraded":"2024-02-05T08:19:44.909517+0000","last_fullsized":"2024-02-05T08:19:44.909410+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-02-05T08:18:02.629366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:02.629366+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":616,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.495003+0000","last_change":"2024-02-05T08:19:44.888161+0000","last_active":"2024-02-05T08:21:01.495003+0000","last_peered":"2024-02-05T08:21:01.495003+0000","last_clean":"2024-02-05T08:19:41.831112+0000","last_became_active":"2024-02-05T08:19:44.888161+0000","last_became_peered":"2024-02-05T08:19:44.888161+0000","last_unstale":"2024-02-05T08:21:01.495003+0000","last_undegraded":"2024-02-05T08:21:01.495003+0000","last_fullsized":"2024-02-05T08:19:44.872217+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:33.725647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:18:33.725647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"337'9169","reported_seq":9317,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:03.959588+0000","last_change":"2024-02-05T08:20:35.938379+0000","last_active":"2024-02-05T08:21:03.959588+0000","last_peered":"2024-02-05T08:21:03.959588+0000","last_clean":"2024-02-05T08:21:03.959588+0000","last_became_active":"2024-02-05T08:19:07.345271+0000","last_became_peered":"2024-02-05T08:19:07.345271+0000","last_unstale":"2024-02-05T08:21:03.959588+0000","last_undegraded":"2024-02-05T08:21:03.959588+0000","last_fullsized":"2024-02-05T08:21:03.959588+0000","mapping_epoch":309,"log_start":"325'4000","ondisk_log_start":"325'4000","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'7352","last_scrub_stamp":"2024-02-05T08:20:35.938230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:35.938230+0000","objects_scrubbed":7270,"log_size":5169,"log_dups_size":0,"ondisk_log_size":5169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.976602+0000","scrub_duration":2394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472973312,"num_objects":7217,"num_object_clones":0,"num_object_copies":21651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9169,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":614,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495757+0000","last_change":"2024-02-05T08:19:41.263058+0000","last_active":"2024-02-05T08:21:01.495757+0000","last_peered":"2024-02-05T08:21:01.495757+0000","last_clean":"2024-02-05T08:21:01.495757+0000","last_became_active":"2024-02-05T08:18:34.642239+0000","last_became_peered":"2024-02-05T08:18:34.642239+0000","last_unstale":"2024-02-05T08:21:01.495757+0000","last_undegraded":"2024-02-05T08:21:01.495757+0000","last_fullsized":"2024-02-05T08:21:01.495757+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:41.262935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:41.262935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:05.016961+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":621,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.497123+0000","last_change":"2024-02-05T08:20:37.971742+0000","last_active":"2024-02-05T08:21:01.497123+0000","last_peered":"2024-02-05T08:21:01.497123+0000","last_clean":"2024-02-05T08:21:01.497123+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:21:01.497123+0000","last_undegraded":"2024-02-05T08:21:01.497123+0000","last_fullsized":"2024-02-05T08:21:01.497123+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:37.971622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.971622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:50.211071+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":627,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.497383+0000","last_change":"2024-02-05T08:19:30.082858+0000","last_active":"2024-02-05T08:21:01.497383+0000","last_peered":"2024-02-05T08:21:01.497383+0000","last_clean":"2024-02-05T08:21:01.497383+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:21:01.497383+0000","last_undegraded":"2024-02-05T08:21:01.497383+0000","last_fullsized":"2024-02-05T08:21:01.497383+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_clean_scrub_stamp":"2024-02-05T08:19:30.082733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:05.016923+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"236'35","reported_seq":394,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.492755+0000","last_change":"2024-02-05T08:20:24.549048+0000","last_active":"2024-02-05T08:21:01.492755+0000","last_peered":"2024-02-05T08:21:01.492755+0000","last_clean":"2024-02-05T08:21:01.492755+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:21:01.492755+0000","last_undegraded":"2024-02-05T08:21:01.492755+0000","last_fullsized":"2024-02-05T08:21:01.492755+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'35","last_scrub_stamp":"2024-02-05T08:20:24.548947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:20:24.548947+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:41.121488+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"337'9275","reported_seq":9426,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:03.226464+0000","last_change":"2024-02-05T08:20:37.202482+0000","last_active":"2024-02-05T08:21:03.226464+0000","last_peered":"2024-02-05T08:21:03.226464+0000","last_clean":"2024-02-05T08:21:03.226464+0000","last_became_active":"2024-02-05T08:19:07.345197+0000","last_became_peered":"2024-02-05T08:19:07.345197+0000","last_unstale":"2024-02-05T08:21:03.226464+0000","last_undegraded":"2024-02-05T08:21:03.226464+0000","last_fullsized":"2024-02-05T08:21:03.226464+0000","mapping_epoch":309,"log_start":"323'3200","ondisk_log_start":"323'3200","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'7639","last_scrub_stamp":"2024-02-05T08:20:37.202377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.202377+0000","objects_scrubbed":7578,"log_size":6075,"log_dups_size":0,"ondisk_log_size":6075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:01.462904+0000","scrub_duration":2232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492371968,"num_objects":7513,"num_object_clones":0,"num_object_copies":22539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9275,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":592,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495578+0000","last_change":"2024-02-05T08:19:24.708102+0000","last_active":"2024-02-05T08:21:01.495578+0000","last_peered":"2024-02-05T08:21:01.495578+0000","last_clean":"2024-02-05T08:21:01.495578+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:21:01.495578+0000","last_undegraded":"2024-02-05T08:21:01.495578+0000","last_fullsized":"2024-02-05T08:21:01.495578+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_clean_scrub_stamp":"2024-02-05T08:19:24.707985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.355160+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":633,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495590+0000","last_change":"2024-02-05T08:20:23.859434+0000","last_active":"2024-02-05T08:21:01.495590+0000","last_peered":"2024-02-05T08:21:01.495590+0000","last_clean":"2024-02-05T08:21:01.495590+0000","last_became_active":"2024-02-05T08:18:34.645830+0000","last_became_peered":"2024-02-05T08:18:34.645830+0000","last_unstale":"2024-02-05T08:21:01.495590+0000","last_undegraded":"2024-02-05T08:21:01.495590+0000","last_fullsized":"2024-02-05T08:21:01.495590+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:23.859245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:23.859245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:32.318976+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":637,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.501530+0000","last_change":"2024-02-05T08:19:17.520776+0000","last_active":"2024-02-05T08:21:01.501530+0000","last_peered":"2024-02-05T08:21:01.501530+0000","last_clean":"2024-02-05T08:21:01.501530+0000","last_became_active":"2024-02-05T08:18:34.633214+0000","last_became_peered":"2024-02-05T08:18:34.633214+0000","last_unstale":"2024-02-05T08:21:01.501530+0000","last_undegraded":"2024-02-05T08:21:01.501530+0000","last_fullsized":"2024-02-05T08:21:01.501530+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:17.520613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.520613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319345+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"262'19","reported_seq":632,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.495554+0000","last_change":"2024-02-05T08:19:44.914651+0000","last_active":"2024-02-05T08:21:01.495554+0000","last_peered":"2024-02-05T08:21:01.495554+0000","last_clean":"2024-02-05T08:19:41.838445+0000","last_became_active":"2024-02-05T08:19:44.914651+0000","last_became_peered":"2024-02-05T08:19:44.914651+0000","last_unstale":"2024-02-05T08:21:01.495554+0000","last_undegraded":"2024-02-05T08:19:44.886996+0000","last_fullsized":"2024-02-05T08:19:44.886890+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'19","last_scrub_stamp":"2024-02-05T08:19:17.554161+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.554161+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.355134+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"337'9114","reported_seq":9255,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:02.930233+0000","last_change":"2024-02-05T08:20:25.764525+0000","last_active":"2024-02-05T08:21:02.930233+0000","last_peered":"2024-02-05T08:21:02.930233+0000","last_clean":"2024-02-05T08:21:02.930233+0000","last_became_active":"2024-02-05T08:19:07.331718+0000","last_became_peered":"2024-02-05T08:19:07.331718+0000","last_unstale":"2024-02-05T08:21:02.930233+0000","last_undegraded":"2024-02-05T08:21:02.930233+0000","last_fullsized":"2024-02-05T08:21:02.930233+0000","mapping_epoch":309,"log_start":"325'3500","ondisk_log_start":"325'3500","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"328'6676","last_scrub_stamp":"2024-02-05T08:20:25.764375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:25.764375+0000","objects_scrubbed":6586,"log_size":5614,"log_dups_size":0,"ondisk_log_size":5614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:42.802496+0000","scrub_duration":2281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480509952,"num_objects":7332,"num_object_clones":0,"num_object_copies":21996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9114,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'130","reported_seq":387,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.501297+0000","last_change":"2024-02-05T08:20:25.365305+0000","last_active":"2024-02-05T08:21:01.501297+0000","last_peered":"2024-02-05T08:21:01.501297+0000","last_clean":"2024-02-05T08:21:01.501297+0000","last_became_active":"2024-02-05T08:19:01.816550+0000","last_became_peered":"2024-02-05T08:19:01.816550+0000","last_unstale":"2024-02-05T08:21:01.501297+0000","last_undegraded":"2024-02-05T08:21:01.501297+0000","last_fullsized":"2024-02-05T08:21:01.501297+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:25.365167+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:25.365167+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:32.657363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":326,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.494901+0000","last_change":"2024-02-05T08:19:44.925889+0000","last_active":"2024-02-05T08:21:01.494901+0000","last_peered":"2024-02-05T08:21:01.494901+0000","last_clean":"2024-02-05T08:19:41.831505+0000","last_became_active":"2024-02-05T08:19:44.925889+0000","last_became_peered":"2024-02-05T08:19:44.925889+0000","last_unstale":"2024-02-05T08:21:01.494901+0000","last_undegraded":"2024-02-05T08:21:01.494901+0000","last_fullsized":"2024-02-05T08:19:44.914970+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:54.817641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.817641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135359+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":648,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.501976+0000","last_change":"2024-02-05T08:19:44.903354+0000","last_active":"2024-02-05T08:21:01.501976+0000","last_peered":"2024-02-05T08:21:01.501976+0000","last_clean":"2024-02-05T08:19:41.832449+0000","last_became_active":"2024-02-05T08:19:44.903354+0000","last_became_peered":"2024-02-05T08:19:44.903354+0000","last_unstale":"2024-02-05T08:21:01.501976+0000","last_undegraded":"2024-02-05T08:19:44.873323+0000","last_fullsized":"2024-02-05T08:19:44.873213+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:18:48.543343+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:18:48.543343+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":625,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.488169+0000","last_change":"2024-02-05T08:20:27.119157+0000","last_active":"2024-02-05T08:21:01.488169+0000","last_peered":"2024-02-05T08:21:01.488169+0000","last_clean":"2024-02-05T08:21:01.488169+0000","last_became_active":"2024-02-05T08:12:29.670859+0000","last_became_peered":"2024-02-05T08:12:29.670859+0000","last_unstale":"2024-02-05T08:21:01.488169+0000","last_undegraded":"2024-02-05T08:21:01.488169+0000","last_fullsized":"2024-02-05T08:21:01.488169+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:27.119042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:20:27.119042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:41.963906+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"325'53","reported_seq":655,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.497354+0000","last_change":"2024-02-05T08:19:44.892891+0000","last_active":"2024-02-05T08:21:01.497354+0000","last_peered":"2024-02-05T08:21:01.497354+0000","last_clean":"2024-02-05T08:14:49.167838+0000","last_became_active":"2024-02-05T08:19:44.892891+0000","last_became_peered":"2024-02-05T08:19:44.892891+0000","last_unstale":"2024-02-05T08:21:01.497354+0000","last_undegraded":"2024-02-05T08:19:44.886039+0000","last_fullsized":"2024-02-05T08:19:44.885927+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'52","last_scrub_stamp":"2024-02-05T08:19:15.601598+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.601598+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":591,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.496289+0000","last_change":"2024-02-05T08:19:44.924245+0000","last_active":"2024-02-05T08:21:01.496289+0000","last_peered":"2024-02-05T08:21:01.496289+0000","last_clean":"2024-02-05T08:19:41.838535+0000","last_became_active":"2024-02-05T08:19:44.924245+0000","last_became_peered":"2024-02-05T08:19:44.924245+0000","last_unstale":"2024-02-05T08:21:01.496289+0000","last_undegraded":"2024-02-05T08:21:01.496289+0000","last_fullsized":"2024-02-05T08:19:44.874710+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:14.615047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.615047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.355096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"337'9312","reported_seq":9447,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:03.294129+0000","last_change":"2024-02-05T08:20:16.549202+0000","last_active":"2024-02-05T08:21:03.294129+0000","last_peered":"2024-02-05T08:21:03.294129+0000","last_clean":"2024-02-05T08:21:03.294129+0000","last_became_active":"2024-02-05T08:19:07.333532+0000","last_became_peered":"2024-02-05T08:19:07.333532+0000","last_unstale":"2024-02-05T08:21:03.294129+0000","last_undegraded":"2024-02-05T08:21:03.294129+0000","last_fullsized":"2024-02-05T08:21:03.294129+0000","mapping_epoch":309,"log_start":"325'4100","ondisk_log_start":"325'4100","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'6042","last_scrub_stamp":"2024-02-05T08:20:16.548919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:16.548919+0000","objects_scrubbed":5971,"log_size":5212,"log_dups_size":0,"ondisk_log_size":5212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:44.635697+0000","scrub_duration":2107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482476032,"num_objects":7362,"num_object_clones":0,"num_object_copies":22086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9312,"num_write_kb":533568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'130","reported_seq":920,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.496432+0000","last_change":"2024-02-05T08:20:27.164887+0000","last_active":"2024-02-05T08:21:01.496432+0000","last_peered":"2024-02-05T08:21:01.496432+0000","last_clean":"2024-02-05T08:21:01.496432+0000","last_became_active":"2024-02-05T08:19:01.813292+0000","last_became_peered":"2024-02-05T08:19:01.813292+0000","last_unstale":"2024-02-05T08:21:01.496432+0000","last_undegraded":"2024-02-05T08:21:01.496432+0000","last_fullsized":"2024-02-05T08:21:01.496432+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:27.164766+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:27.164766+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:40.161712+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":626,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.497507+0000","last_change":"2024-02-05T08:20:21.189633+0000","last_active":"2024-02-05T08:21:01.497507+0000","last_peered":"2024-02-05T08:21:01.497507+0000","last_clean":"2024-02-05T08:21:01.497507+0000","last_became_active":"2024-02-05T08:18:34.643109+0000","last_became_peered":"2024-02-05T08:18:34.643109+0000","last_unstale":"2024-02-05T08:21:01.497507+0000","last_undegraded":"2024-02-05T08:21:01.497507+0000","last_fullsized":"2024-02-05T08:21:01.497507+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:21.189500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:21.189500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:37.050505+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":107,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.494763+0000","last_change":"2024-02-05T08:19:44.902069+0000","last_active":"2024-02-05T08:21:01.494763+0000","last_peered":"2024-02-05T08:21:01.494763+0000","last_clean":"2024-02-05T08:06:34.916685+0000","last_became_active":"2024-02-05T08:19:44.902069+0000","last_became_peered":"2024-02-05T08:19:44.902069+0000","last_unstale":"2024-02-05T08:21:01.494763+0000","last_undegraded":"2024-02-05T08:19:44.872249+0000","last_fullsized":"2024-02-05T08:19:44.872099+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:18:43.341831+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:18:43.341831+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":650,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495778+0000","last_change":"2024-02-05T08:20:29.922938+0000","last_active":"2024-02-05T08:21:01.495778+0000","last_peered":"2024-02-05T08:21:01.495778+0000","last_clean":"2024-02-05T08:21:01.495778+0000","last_became_active":"2024-02-05T08:18:34.644683+0000","last_became_peered":"2024-02-05T08:18:34.644683+0000","last_unstale":"2024-02-05T08:21:01.495778+0000","last_undegraded":"2024-02-05T08:21:01.495778+0000","last_fullsized":"2024-02-05T08:21:01.495778+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_clean_scrub_stamp":"2024-02-05T08:20:29.922794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.839539+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'17","reported_seq":627,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.498611+0000","last_change":"2024-02-05T08:19:11.615846+0000","last_active":"2024-02-05T08:21:01.498611+0000","last_peered":"2024-02-05T08:21:01.498611+0000","last_clean":"2024-02-05T08:21:01.498611+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:21:01.498611+0000","last_undegraded":"2024-02-05T08:21:01.498611+0000","last_fullsized":"2024-02-05T08:21:01.498611+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'17","last_scrub_stamp":"2024-02-05T08:19:11.615763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:11.615763+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":629,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.500893+0000","last_change":"2024-02-05T08:20:37.014562+0000","last_active":"2024-02-05T08:21:01.500893+0000","last_peered":"2024-02-05T08:21:01.500893+0000","last_clean":"2024-02-05T08:21:01.500893+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:21:01.500893+0000","last_undegraded":"2024-02-05T08:21:01.500893+0000","last_fullsized":"2024-02-05T08:21:01.500893+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:37.014448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.014448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:02.259236+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"337'9092","reported_seq":9157,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:02.947447+0000","last_change":"2024-02-05T08:19:44.910611+0000","last_active":"2024-02-05T08:21:02.947447+0000","last_peered":"2024-02-05T08:21:02.947447+0000","last_clean":"2024-02-05T08:19:43.860522+0000","last_became_active":"2024-02-05T08:19:44.910611+0000","last_became_peered":"2024-02-05T08:19:44.910611+0000","last_unstale":"2024-02-05T08:21:02.947447+0000","last_undegraded":"2024-02-05T08:19:44.872215+0000","last_fullsized":"2024-02-05T08:19:44.872102+0000","mapping_epoch":324,"log_start":"325'3500","ondisk_log_start":"325'3500","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":5592,"log_dups_size":0,"ondisk_log_size":5592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.678873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473169920,"num_objects":7220,"num_object_clones":0,"num_object_copies":21660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9092,"num_write_kb":521984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":7220}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":613,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.502962+0000","last_change":"2024-02-05T08:19:19.557662+0000","last_active":"2024-02-05T08:21:01.502962+0000","last_peered":"2024-02-05T08:21:01.502962+0000","last_clean":"2024-02-05T08:21:01.502962+0000","last_became_active":"2024-02-05T08:18:34.633450+0000","last_became_peered":"2024-02-05T08:18:34.633450+0000","last_unstale":"2024-02-05T08:21:01.502962+0000","last_undegraded":"2024-02-05T08:21:01.502962+0000","last_fullsized":"2024-02-05T08:21:01.502962+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:19.557556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:19.557556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319420+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":912,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495878+0000","last_change":"2024-02-05T08:20:33.165619+0000","last_active":"2024-02-05T08:21:01.495878+0000","last_peered":"2024-02-05T08:21:01.495878+0000","last_clean":"2024-02-05T08:21:01.495878+0000","last_became_active":"2024-02-05T08:18:57.285362+0000","last_became_peered":"2024-02-05T08:18:57.285362+0000","last_unstale":"2024-02-05T08:21:01.495878+0000","last_undegraded":"2024-02-05T08:21:01.495878+0000","last_fullsized":"2024-02-05T08:21:01.495878+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_clean_scrub_stamp":"2024-02-05T08:20:33.165371+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:50.031969+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":104,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.490949+0000","last_change":"2024-02-05T08:19:20.109021+0000","last_active":"2024-02-05T08:21:01.490949+0000","last_peered":"2024-02-05T08:21:01.490949+0000","last_clean":"2024-02-05T08:21:01.490949+0000","last_became_active":"2024-02-05T08:18:34.631974+0000","last_became_peered":"2024-02-05T08:18:34.631974+0000","last_unstale":"2024-02-05T08:21:01.490949+0000","last_undegraded":"2024-02-05T08:21:01.490949+0000","last_fullsized":"2024-02-05T08:21:01.490949+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:20.108944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:20.108944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"236'30","reported_seq":482,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.492663+0000","last_change":"2024-02-05T08:20:18.442377+0000","last_active":"2024-02-05T08:21:01.492663+0000","last_peered":"2024-02-05T08:21:01.492663+0000","last_clean":"2024-02-05T08:21:01.492663+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:21:01.492663+0000","last_undegraded":"2024-02-05T08:21:01.492663+0000","last_fullsized":"2024-02-05T08:21:01.492663+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:20:18.442244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:20:18.442244+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:40.320195+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":606,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.497285+0000","last_change":"2024-02-05T08:20:20.098080+0000","last_active":"2024-02-05T08:21:01.497285+0000","last_peered":"2024-02-05T08:21:01.497285+0000","last_clean":"2024-02-05T08:21:01.497285+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:21:01.497285+0000","last_undegraded":"2024-02-05T08:21:01.497285+0000","last_fullsized":"2024-02-05T08:21:01.497285+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:20.097959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_clean_scrub_stamp":"2024-02-05T08:20:20.097959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:30.955905+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"337'8989","reported_seq":9137,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:03.928977+0000","last_change":"2024-02-05T08:20:39.198575+0000","last_active":"2024-02-05T08:21:03.928977+0000","last_peered":"2024-02-05T08:21:03.928977+0000","last_clean":"2024-02-05T08:21:03.928977+0000","last_became_active":"2024-02-05T08:19:07.327007+0000","last_became_peered":"2024-02-05T08:19:07.327007+0000","last_unstale":"2024-02-05T08:21:03.928977+0000","last_undegraded":"2024-02-05T08:21:03.928977+0000","last_fullsized":"2024-02-05T08:21:03.928977+0000","mapping_epoch":309,"log_start":"325'3900","ondisk_log_start":"325'3900","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'7440","last_scrub_stamp":"2024-02-05T08:20:39.198303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:39.198303+0000","objects_scrubbed":7387,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:00.465206+0000","scrub_duration":1696,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468516864,"num_objects":7149,"num_object_clones":0,"num_object_copies":21447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8989,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'130","reported_seq":584,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.492956+0000","last_change":"2024-02-05T08:19:44.916776+0000","last_active":"2024-02-05T08:21:01.492956+0000","last_peered":"2024-02-05T08:21:01.492956+0000","last_clean":"2024-02-05T08:12:23.979378+0000","last_became_active":"2024-02-05T08:19:44.916776+0000","last_became_peered":"2024-02-05T08:19:44.916776+0000","last_unstale":"2024-02-05T08:21:01.492956+0000","last_undegraded":"2024-02-05T08:21:01.492956+0000","last_fullsized":"2024-02-05T08:19:44.897597+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:07.594418+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:07.594418+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:01.629277+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":545,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.501843+0000","last_change":"2024-02-05T08:19:44.903734+0000","last_active":"2024-02-05T08:21:01.501843+0000","last_peered":"2024-02-05T08:21:01.501843+0000","last_clean":"2024-02-05T08:18:29.610515+0000","last_became_active":"2024-02-05T08:19:44.903734+0000","last_became_peered":"2024-02-05T08:19:44.903734+0000","last_unstale":"2024-02-05T08:21:01.501843+0000","last_undegraded":"2024-02-05T08:21:01.501843+0000","last_fullsized":"2024-02-05T08:19:44.898755+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.646431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.646431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319457+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":616,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495293+0000","last_change":"2024-02-05T08:20:12.695787+0000","last_active":"2024-02-05T08:21:01.495293+0000","last_peered":"2024-02-05T08:21:01.495293+0000","last_clean":"2024-02-05T08:21:01.495293+0000","last_became_active":"2024-02-05T08:18:34.635954+0000","last_became_peered":"2024-02-05T08:18:34.635954+0000","last_unstale":"2024-02-05T08:21:01.495293+0000","last_undegraded":"2024-02-05T08:21:01.495293+0000","last_fullsized":"2024-02-05T08:21:01.495293+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:12.695667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:12.695667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:37.193362+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":1605,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:02.051840+0000","last_change":"2024-02-05T08:19:44.901362+0000","last_active":"2024-02-05T08:21:02.051840+0000","last_peered":"2024-02-05T08:21:02.051840+0000","last_clean":"2024-02-05T08:19:43.389689+0000","last_became_active":"2024-02-05T08:19:44.901362+0000","last_became_peered":"2024-02-05T08:19:44.901362+0000","last_unstale":"2024-02-05T08:21:02.051840+0000","last_undegraded":"2024-02-05T08:19:44.875266+0000","last_fullsized":"2024-02-05T08:19:44.875151+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:19:14.179578+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.179578+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:05.016872+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":972,"num_read_kb":967,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":656,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.494531+0000","last_change":"2024-02-05T08:20:16.599470+0000","last_active":"2024-02-05T08:21:01.494531+0000","last_peered":"2024-02-05T08:21:01.494531+0000","last_clean":"2024-02-05T08:21:01.494531+0000","last_became_active":"2024-02-05T08:14:24.992820+0000","last_became_peered":"2024-02-05T08:14:24.992820+0000","last_unstale":"2024-02-05T08:21:01.494531+0000","last_undegraded":"2024-02-05T08:21:01.494531+0000","last_fullsized":"2024-02-05T08:21:01.494531+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:20:16.599202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:20:16.599202+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:35.823098+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":608,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.503187+0000","last_change":"2024-02-05T08:19:44.900624+0000","last_active":"2024-02-05T08:21:01.503187+0000","last_peered":"2024-02-05T08:21:01.503187+0000","last_clean":"2024-02-05T08:19:41.831836+0000","last_became_active":"2024-02-05T08:19:44.900624+0000","last_became_peered":"2024-02-05T08:19:44.900624+0000","last_unstale":"2024-02-05T08:21:01.503187+0000","last_undegraded":"2024-02-05T08:21:01.503187+0000","last_fullsized":"2024-02-05T08:19:44.899450+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:35.391824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:19:35.391824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.202567+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"337'9038","reported_seq":9189,"reported_epoch":337,"state":"active+clean+scrubbing","last_fresh":"2024-02-05T08:21:02.962136+0000","last_change":"2024-02-05T08:21:01.679040+0000","last_active":"2024-02-05T08:21:02.962136+0000","last_peered":"2024-02-05T08:21:02.962136+0000","last_clean":"2024-02-05T08:21:02.962136+0000","last_became_active":"2024-02-05T08:19:07.332849+0000","last_became_peered":"2024-02-05T08:19:07.332849+0000","last_unstale":"2024-02-05T08:21:02.962136+0000","last_undegraded":"2024-02-05T08:21:02.962136+0000","last_fullsized":"2024-02-05T08:21:02.962136+0000","mapping_epoch":309,"log_start":"325'3400","ondisk_log_start":"325'3400","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":3499,"log_size":5638,"log_dups_size":0,"ondisk_log_size":5638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478347288,"num_objects":7300,"num_object_clones":0,"num_object_copies":21900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7300,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9038,"num_write_kb":522753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'130","reported_seq":648,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.493128+0000","last_change":"2024-02-05T08:20:19.485708+0000","last_active":"2024-02-05T08:21:01.493128+0000","last_peered":"2024-02-05T08:21:01.493128+0000","last_clean":"2024-02-05T08:21:01.493128+0000","last_became_active":"2024-02-05T08:19:01.817743+0000","last_became_peered":"2024-02-05T08:19:01.817743+0000","last_unstale":"2024-02-05T08:21:01.493128+0000","last_undegraded":"2024-02-05T08:21:01.493128+0000","last_fullsized":"2024-02-05T08:21:01.493128+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:19.485585+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.485585+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:46.149289+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":312,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.494664+0000","last_change":"2024-02-05T08:19:44.901104+0000","last_active":"2024-02-05T08:21:01.494664+0000","last_peered":"2024-02-05T08:21:01.494664+0000","last_clean":"2024-02-05T08:12:25.219295+0000","last_became_active":"2024-02-05T08:19:44.901104+0000","last_became_peered":"2024-02-05T08:19:44.901104+0000","last_unstale":"2024-02-05T08:21:01.494664+0000","last_undegraded":"2024-02-05T08:21:01.494664+0000","last_fullsized":"2024-02-05T08:19:44.872029+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:27.862543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:27.862543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.496169+0000","last_change":"2024-02-05T08:19:44.901166+0000","last_active":"2024-02-05T08:21:01.496169+0000","last_peered":"2024-02-05T08:21:01.496169+0000","last_clean":"2024-02-05T08:04:40.511631+0000","last_became_active":"2024-02-05T08:19:44.901166+0000","last_became_peered":"2024-02-05T08:19:44.901166+0000","last_unstale":"2024-02-05T08:21:01.496169+0000","last_undegraded":"2024-02-05T08:21:01.496169+0000","last_fullsized":"2024-02-05T08:19:44.885594+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:37.350622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.350622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":651,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495242+0000","last_change":"2024-02-05T08:19:15.574020+0000","last_active":"2024-02-05T08:21:01.495242+0000","last_peered":"2024-02-05T08:21:01.495242+0000","last_clean":"2024-02-05T08:21:01.495242+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:21:01.495242+0000","last_undegraded":"2024-02-05T08:21:01.495242+0000","last_fullsized":"2024-02-05T08:21:01.495242+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-02-05T08:19:15.573897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.573897+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.355019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":343,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.492674+0000","last_change":"2024-02-05T08:19:12.455979+0000","last_active":"2024-02-05T08:21:01.492674+0000","last_peered":"2024-02-05T08:21:01.492674+0000","last_clean":"2024-02-05T08:21:01.492674+0000","last_became_active":"2024-02-05T08:12:29.688534+0000","last_became_peered":"2024-02-05T08:12:29.688534+0000","last_unstale":"2024-02-05T08:21:01.492674+0000","last_undegraded":"2024-02-05T08:21:01.492674+0000","last_fullsized":"2024-02-05T08:21:01.492674+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:12.455899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:19:12.455899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"337'9045","reported_seq":9193,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:03.241719+0000","last_change":"2024-02-05T08:20:34.585093+0000","last_active":"2024-02-05T08:21:03.241719+0000","last_peered":"2024-02-05T08:21:03.241719+0000","last_clean":"2024-02-05T08:21:03.241719+0000","last_became_active":"2024-02-05T08:19:07.332176+0000","last_became_peered":"2024-02-05T08:19:07.332176+0000","last_unstale":"2024-02-05T08:21:03.241719+0000","last_undegraded":"2024-02-05T08:21:03.241719+0000","last_fullsized":"2024-02-05T08:21:03.241719+0000","mapping_epoch":309,"log_start":"325'4000","ondisk_log_start":"325'4000","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'7365","last_scrub_stamp":"2024-02-05T08:20:34.584959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:34.584959+0000","objects_scrubbed":7281,"log_size":5045,"log_dups_size":0,"ondisk_log_size":5045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:59.712909+0000","scrub_duration":2480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481624064,"num_objects":7349,"num_object_clones":0,"num_object_copies":22047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9045,"num_write_kb":524608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"82'130","reported_seq":648,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.493557+0000","last_change":"2024-02-05T08:20:35.974697+0000","last_active":"2024-02-05T08:21:01.493557+0000","last_peered":"2024-02-05T08:21:01.493557+0000","last_clean":"2024-02-05T08:21:01.493557+0000","last_became_active":"2024-02-05T08:19:01.815446+0000","last_became_peered":"2024-02-05T08:19:01.815446+0000","last_unstale":"2024-02-05T08:21:01.493557+0000","last_undegraded":"2024-02-05T08:21:01.493557+0000","last_fullsized":"2024-02-05T08:21:01.493557+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:35.974462+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:35.974462+0000","last_clean_scrub_stamp":"2024-02-05T08:20:35.974462+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:59.595447+0000","scrub_duration":332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":605,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.493616+0000","last_change":"2024-02-05T08:19:10.488490+0000","last_active":"2024-02-05T08:21:01.493616+0000","last_peered":"2024-02-05T08:21:01.493616+0000","last_clean":"2024-02-05T08:21:01.493616+0000","last_became_active":"2024-02-05T08:18:34.636693+0000","last_became_peered":"2024-02-05T08:18:34.636693+0000","last_unstale":"2024-02-05T08:21:01.493616+0000","last_undegraded":"2024-02-05T08:21:01.493616+0000","last_fullsized":"2024-02-05T08:21:01.493616+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:10.488395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:10.488395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.355045+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":625,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.502401+0000","last_change":"2024-02-05T08:19:18.499832+0000","last_active":"2024-02-05T08:21:01.502401+0000","last_peered":"2024-02-05T08:21:01.502401+0000","last_clean":"2024-02-05T08:21:01.502401+0000","last_became_active":"2024-02-05T08:18:34.633263+0000","last_became_peered":"2024-02-05T08:18:34.633263+0000","last_unstale":"2024-02-05T08:21:01.502401+0000","last_undegraded":"2024-02-05T08:21:01.502401+0000","last_fullsized":"2024-02-05T08:21:01.502401+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:18.499743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:18.499743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319444+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'20","reported_seq":442,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.502226+0000","last_change":"2024-02-05T08:19:44.916082+0000","last_active":"2024-02-05T08:21:01.502226+0000","last_peered":"2024-02-05T08:21:01.502226+0000","last_clean":"2024-02-05T08:14:49.165927+0000","last_became_active":"2024-02-05T08:19:44.916082+0000","last_became_peered":"2024-02-05T08:19:44.916082+0000","last_unstale":"2024-02-05T08:21:01.502226+0000","last_undegraded":"2024-02-05T08:19:44.899919+0000","last_fullsized":"2024-02-05T08:19:44.899797+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:19:09.511509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:09.511509+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319299+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"277'11","reported_seq":822,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495808+0000","last_change":"2024-02-05T08:19:33.688840+0000","last_active":"2024-02-05T08:21:01.495808+0000","last_peered":"2024-02-05T08:21:01.495808+0000","last_clean":"2024-02-05T08:21:01.495808+0000","last_became_active":"2024-02-05T08:12:29.683091+0000","last_became_peered":"2024-02-05T08:12:29.683091+0000","last_unstale":"2024-02-05T08:21:01.495808+0000","last_undegraded":"2024-02-05T08:21:01.495808+0000","last_fullsized":"2024-02-05T08:21:01.495808+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:19:33.688719+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_clean_scrub_stamp":"2024-02-05T08:19:33.688719+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"337'9168","reported_seq":9319,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:02.942045+0000","last_change":"2024-02-05T08:20:41.146544+0000","last_active":"2024-02-05T08:21:02.942045+0000","last_peered":"2024-02-05T08:21:02.942045+0000","last_clean":"2024-02-05T08:21:02.942045+0000","last_became_active":"2024-02-05T08:19:07.346024+0000","last_became_peered":"2024-02-05T08:19:07.346024+0000","last_unstale":"2024-02-05T08:21:02.942045+0000","last_undegraded":"2024-02-05T08:21:02.942045+0000","last_fullsized":"2024-02-05T08:21:02.942045+0000","mapping_epoch":309,"log_start":"325'3600","ondisk_log_start":"325'3600","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'7704","last_scrub_stamp":"2024-02-05T08:20:41.146350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:41.146350+0000","objects_scrubbed":7648,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:00.018859+0000","scrub_duration":2644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481296384,"num_objects":7344,"num_object_clones":0,"num_object_copies":22032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9168,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'130","reported_seq":387,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.492098+0000","last_change":"2024-02-05T08:20:27.728841+0000","last_active":"2024-02-05T08:21:01.492098+0000","last_peered":"2024-02-05T08:21:01.492098+0000","last_clean":"2024-02-05T08:21:01.492098+0000","last_became_active":"2024-02-05T08:19:01.819856+0000","last_became_peered":"2024-02-05T08:19:01.819856+0000","last_unstale":"2024-02-05T08:21:01.492098+0000","last_undegraded":"2024-02-05T08:21:01.492098+0000","last_fullsized":"2024-02-05T08:21:01.492098+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:27.728717+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:27.728717+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:56.586217+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":614,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.496890+0000","last_change":"2024-02-05T08:20:19.251720+0000","last_active":"2024-02-05T08:21:01.496890+0000","last_peered":"2024-02-05T08:21:01.496890+0000","last_clean":"2024-02-05T08:21:01.496890+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:21:01.496890+0000","last_undegraded":"2024-02-05T08:21:01.496890+0000","last_fullsized":"2024-02-05T08:21:01.496890+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:19.251464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.251464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:40.244202+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":618,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.496466+0000","last_change":"2024-02-05T08:19:44.892743+0000","last_active":"2024-02-05T08:21:01.496466+0000","last_peered":"2024-02-05T08:21:01.496466+0000","last_clean":"2024-02-05T08:19:41.834750+0000","last_became_active":"2024-02-05T08:19:44.892743+0000","last_became_peered":"2024-02-05T08:19:44.892743+0000","last_unstale":"2024-02-05T08:21:01.496466+0000","last_undegraded":"2024-02-05T08:19:44.887459+0000","last_fullsized":"2024-02-05T08:19:44.887352+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:19:11.483052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:11.483052+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.354982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"236'28","reported_seq":687,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.491555+0000","last_change":"2024-02-05T08:19:30.389683+0000","last_active":"2024-02-05T08:21:01.491555+0000","last_peered":"2024-02-05T08:21:01.491555+0000","last_clean":"2024-02-05T08:21:01.491555+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:21:01.491555+0000","last_undegraded":"2024-02-05T08:21:01.491555+0000","last_fullsized":"2024-02-05T08:21:01.491555+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:19:30.389554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:30.389554+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:04.118531+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"325'12","reported_seq":143,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.491374+0000","last_change":"2024-02-05T08:19:44.915079+0000","last_active":"2024-02-05T08:21:01.491374+0000","last_peered":"2024-02-05T08:21:01.491374+0000","last_clean":"2024-02-05T08:19:41.828253+0000","last_became_active":"2024-02-05T08:19:44.915079+0000","last_became_peered":"2024-02-05T08:19:44.915079+0000","last_unstale":"2024-02-05T08:21:01.491374+0000","last_undegraded":"2024-02-05T08:19:44.888020+0000","last_fullsized":"2024-02-05T08:19:44.887894+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'11","last_scrub_stamp":"2024-02-05T08:19:34.968809+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:19:34.968809+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780773+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"337'11134","reported_seq":11318,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:02.658532+0000","last_change":"2024-02-05T08:20:51.167853+0000","last_active":"2024-02-05T08:21:02.658532+0000","last_peered":"2024-02-05T08:21:02.658532+0000","last_clean":"2024-02-05T08:21:02.658532+0000","last_became_active":"2024-02-05T08:19:07.331338+0000","last_became_peered":"2024-02-05T08:19:07.331338+0000","last_unstale":"2024-02-05T08:21:02.658532+0000","last_undegraded":"2024-02-05T08:21:02.658532+0000","last_fullsized":"2024-02-05T08:21:02.658532+0000","mapping_epoch":309,"log_start":"329'6702","ondisk_log_start":"329'6702","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":2817,"log_size":4432,"log_dups_size":0,"ondisk_log_size":4432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481361920,"num_objects":7345,"num_object_clones":0,"num_object_copies":22035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9051,"num_write_kb":524672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'130","reported_seq":379,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.489936+0000","last_change":"2024-02-05T08:19:08.403635+0000","last_active":"2024-02-05T08:21:01.489936+0000","last_peered":"2024-02-05T08:21:01.489936+0000","last_clean":"2024-02-05T08:21:01.489936+0000","last_became_active":"2024-02-05T08:19:01.816923+0000","last_became_peered":"2024-02-05T08:19:01.816923+0000","last_unstale":"2024-02-05T08:21:01.489936+0000","last_undegraded":"2024-02-05T08:21:01.489936+0000","last_fullsized":"2024-02-05T08:21:01.489936+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:08.403509+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:08.403509+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135556+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":621,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.488434+0000","last_change":"2024-02-05T08:20:36.100335+0000","last_active":"2024-02-05T08:21:01.488434+0000","last_peered":"2024-02-05T08:21:01.488434+0000","last_clean":"2024-02-05T08:21:01.488434+0000","last_became_active":"2024-02-05T08:18:34.635908+0000","last_became_peered":"2024-02-05T08:18:34.635908+0000","last_unstale":"2024-02-05T08:21:01.488434+0000","last_undegraded":"2024-02-05T08:21:01.488434+0000","last_fullsized":"2024-02-05T08:21:01.488434+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:36.100208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:36.100208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:04.352882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":108,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.492393+0000","last_change":"2024-02-05T08:19:38.969220+0000","last_active":"2024-02-05T08:21:01.492393+0000","last_peered":"2024-02-05T08:21:01.492393+0000","last_clean":"2024-02-05T08:21:01.492393+0000","last_became_active":"2024-02-05T08:18:34.649808+0000","last_became_peered":"2024-02-05T08:18:34.649808+0000","last_unstale":"2024-02-05T08:21:01.492393+0000","last_undegraded":"2024-02-05T08:21:01.492393+0000","last_fullsized":"2024-02-05T08:21:01.492393+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:19:38.968895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:38.968895+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780868+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"236'38","reported_seq":130,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.490035+0000","last_change":"2024-02-05T08:20:37.207904+0000","last_active":"2024-02-05T08:21:01.490035+0000","last_peered":"2024-02-05T08:21:01.490035+0000","last_clean":"2024-02-05T08:21:01.490035+0000","last_became_active":"2024-02-05T08:18:34.653505+0000","last_became_peered":"2024-02-05T08:18:34.653505+0000","last_unstale":"2024-02-05T08:21:01.490035+0000","last_undegraded":"2024-02-05T08:21:01.490035+0000","last_fullsized":"2024-02-05T08:21:01.490035+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'38","last_scrub_stamp":"2024-02-05T08:20:37.207762+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.207762+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:54.648572+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":617,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.491064+0000","last_change":"2024-02-05T08:20:31.133433+0000","last_active":"2024-02-05T08:21:01.491064+0000","last_peered":"2024-02-05T08:21:01.491064+0000","last_clean":"2024-02-05T08:21:01.491064+0000","last_became_active":"2024-02-05T08:18:34.636667+0000","last_became_peered":"2024-02-05T08:18:34.636667+0000","last_unstale":"2024-02-05T08:21:01.491064+0000","last_undegraded":"2024-02-05T08:21:01.491064+0000","last_fullsized":"2024-02-05T08:21:01.491064+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:31.133206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:31.133206+0000","last_clean_scrub_stamp":"2024-02-05T08:20:31.133206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:00.885317+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"337'9183","reported_seq":9331,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:02.647607+0000","last_change":"2024-02-05T08:20:37.390439+0000","last_active":"2024-02-05T08:21:02.647607+0000","last_peered":"2024-02-05T08:21:02.647607+0000","last_clean":"2024-02-05T08:21:02.647607+0000","last_became_active":"2024-02-05T08:19:07.330566+0000","last_became_peered":"2024-02-05T08:19:07.330566+0000","last_unstale":"2024-02-05T08:21:02.647607+0000","last_undegraded":"2024-02-05T08:21:02.647607+0000","last_fullsized":"2024-02-05T08:21:02.647607+0000","mapping_epoch":309,"log_start":"326'4800","ondisk_log_start":"326'4800","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'7463","last_scrub_stamp":"2024-02-05T08:20:37.390208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.390208+0000","objects_scrubbed":7384,"log_size":4383,"log_dups_size":0,"ondisk_log_size":4383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:50.900137+0000","scrub_duration":2278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":481099776,"num_objects":7341,"num_object_clones":0,"num_object_copies":22023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9183,"num_write_kb":528768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'130","reported_seq":387,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.501720+0000","last_change":"2024-02-05T08:20:34.301845+0000","last_active":"2024-02-05T08:21:01.501720+0000","last_peered":"2024-02-05T08:21:01.501720+0000","last_clean":"2024-02-05T08:21:01.501720+0000","last_became_active":"2024-02-05T08:19:01.817169+0000","last_became_peered":"2024-02-05T08:19:01.817169+0000","last_unstale":"2024-02-05T08:21:01.501720+0000","last_undegraded":"2024-02-05T08:21:01.501720+0000","last_fullsized":"2024-02-05T08:21:01.501720+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:34.301740+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:34.301740+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:36.803503+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":604,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.501932+0000","last_change":"2024-02-05T08:19:44.900438+0000","last_active":"2024-02-05T08:21:01.501932+0000","last_peered":"2024-02-05T08:21:01.501932+0000","last_clean":"2024-02-05T08:19:41.831465+0000","last_became_active":"2024-02-05T08:19:44.900438+0000","last_became_peered":"2024-02-05T08:19:44.900438+0000","last_unstale":"2024-02-05T08:21:01.501932+0000","last_undegraded":"2024-02-05T08:21:01.501932+0000","last_fullsized":"2024-02-05T08:19:44.898161+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.613476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.613476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319180+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":810,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.495409+0000","last_change":"2024-02-05T08:20:38.198413+0000","last_active":"2024-02-05T08:21:01.495409+0000","last_peered":"2024-02-05T08:21:01.495409+0000","last_clean":"2024-02-05T08:21:01.495409+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:21:01.495409+0000","last_undegraded":"2024-02-05T08:21:01.495409+0000","last_fullsized":"2024-02-05T08:21:01.495409+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:20:38.198278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:38.198278+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:58.293117+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":107,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.490120+0000","last_change":"2024-02-05T08:19:44.902070+0000","last_active":"2024-02-05T08:21:01.490120+0000","last_peered":"2024-02-05T08:21:01.490120+0000","last_clean":"2024-02-05T08:19:41.828067+0000","last_became_active":"2024-02-05T08:19:44.902070+0000","last_became_peered":"2024-02-05T08:19:44.902070+0000","last_unstale":"2024-02-05T08:21:01.490120+0000","last_undegraded":"2024-02-05T08:21:01.490120+0000","last_fullsized":"2024-02-05T08:19:44.887805+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:07.027521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:07.027521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"236'27","reported_seq":612,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.491954+0000","last_change":"2024-02-05T08:19:44.912476+0000","last_active":"2024-02-05T08:21:01.491954+0000","last_peered":"2024-02-05T08:21:01.491954+0000","last_clean":"2024-02-05T08:18:29.610409+0000","last_became_active":"2024-02-05T08:19:44.912476+0000","last_became_peered":"2024-02-05T08:19:44.912476+0000","last_unstale":"2024-02-05T08:21:01.491954+0000","last_undegraded":"2024-02-05T08:19:44.888218+0000","last_fullsized":"2024-02-05T08:19:44.888087+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:18:40.384115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:40.384115+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:03.780681+0000","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"337'9107","reported_seq":9176,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:02.927776+0000","last_change":"2024-02-05T08:19:44.913164+0000","last_active":"2024-02-05T08:21:02.927776+0000","last_peered":"2024-02-05T08:21:02.927776+0000","last_clean":"2024-02-05T08:19:43.862508+0000","last_became_active":"2024-02-05T08:19:44.913164+0000","last_became_peered":"2024-02-05T08:19:44.913164+0000","last_unstale":"2024-02-05T08:21:02.927776+0000","last_undegraded":"2024-02-05T08:19:44.871956+0000","last_fullsized":"2024-02-05T08:19:44.871842+0000","mapping_epoch":324,"log_start":"325'3500","ondisk_log_start":"325'3500","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.678857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474677248,"num_objects":7243,"num_object_clones":0,"num_object_copies":21729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9107,"num_write_kb":523200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":7243}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":112,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.494918+0000","last_change":"2024-02-05T08:20:38.715971+0000","last_active":"2024-02-05T08:21:01.494918+0000","last_peered":"2024-02-05T08:21:01.494918+0000","last_clean":"2024-02-05T08:21:01.494918+0000","last_became_active":"2024-02-05T08:18:34.627261+0000","last_became_peered":"2024-02-05T08:18:34.627261+0000","last_unstale":"2024-02-05T08:21:01.494918+0000","last_undegraded":"2024-02-05T08:21:01.494918+0000","last_fullsized":"2024-02-05T08:21:01.494918+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:38.715837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:20:38.715837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:00.975975+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":625,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.503182+0000","last_change":"2024-02-05T08:20:30.366634+0000","last_active":"2024-02-05T08:21:01.503182+0000","last_peered":"2024-02-05T08:21:01.503182+0000","last_clean":"2024-02-05T08:21:01.503182+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:21:01.503182+0000","last_undegraded":"2024-02-05T08:21:01.503182+0000","last_fullsized":"2024-02-05T08:21:01.503182+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:30.366511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:30.366511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:51.599279+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":613,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.491139+0000","last_change":"2024-02-05T08:19:13.275219+0000","last_active":"2024-02-05T08:21:01.491139+0000","last_peered":"2024-02-05T08:21:01.491139+0000","last_clean":"2024-02-05T08:21:01.491139+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:21:01.491139+0000","last_undegraded":"2024-02-05T08:21:01.491139+0000","last_fullsized":"2024-02-05T08:21:01.491139+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:13.275109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:19:13.275109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"236'19","reported_seq":671,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.495112+0000","last_change":"2024-02-05T08:19:44.926373+0000","last_active":"2024-02-05T08:21:01.495112+0000","last_peered":"2024-02-05T08:21:01.495112+0000","last_clean":"2024-02-05T08:19:41.831642+0000","last_became_active":"2024-02-05T08:19:44.926373+0000","last_became_peered":"2024-02-05T08:19:44.926373+0000","last_unstale":"2024-02-05T08:21:01.495112+0000","last_undegraded":"2024-02-05T08:19:44.921915+0000","last_fullsized":"2024-02-05T08:19:44.916232+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:36.561837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:36.561837+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135427+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"337'9197","reported_seq":9348,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:03.232485+0000","last_change":"2024-02-05T08:20:36.341229+0000","last_active":"2024-02-05T08:21:03.232485+0000","last_peered":"2024-02-05T08:21:03.232485+0000","last_clean":"2024-02-05T08:21:03.232485+0000","last_became_active":"2024-02-05T08:19:07.346034+0000","last_became_peered":"2024-02-05T08:19:07.346034+0000","last_unstale":"2024-02-05T08:21:03.232485+0000","last_undegraded":"2024-02-05T08:21:03.232485+0000","last_fullsized":"2024-02-05T08:21:03.232485+0000","mapping_epoch":309,"log_start":"322'3100","ondisk_log_start":"322'3100","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'7452","last_scrub_stamp":"2024-02-05T08:20:36.340902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:36.340902+0000","objects_scrubbed":7382,"log_size":6097,"log_dups_size":0,"ondisk_log_size":6097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:46.511007+0000","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":483459072,"num_objects":7377,"num_object_clones":0,"num_object_copies":22131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9197,"num_write_kb":530368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'130","reported_seq":379,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.501019+0000","last_change":"2024-02-05T08:19:10.494455+0000","last_active":"2024-02-05T08:21:01.501019+0000","last_peered":"2024-02-05T08:21:01.501019+0000","last_clean":"2024-02-05T08:21:01.501019+0000","last_became_active":"2024-02-05T08:19:01.817126+0000","last_became_peered":"2024-02-05T08:19:01.817126+0000","last_unstale":"2024-02-05T08:21:01.501019+0000","last_undegraded":"2024-02-05T08:21:01.501019+0000","last_fullsized":"2024-02-05T08:21:01.501019+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:10.494376+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:10.494376+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:06.319285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":638,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.498770+0000","last_change":"2024-02-05T08:20:19.087445+0000","last_active":"2024-02-05T08:21:01.498770+0000","last_peered":"2024-02-05T08:21:01.498770+0000","last_clean":"2024-02-05T08:21:01.498770+0000","last_became_active":"2024-02-05T08:18:34.645067+0000","last_became_peered":"2024-02-05T08:18:34.645067+0000","last_unstale":"2024-02-05T08:21:01.498770+0000","last_undegraded":"2024-02-05T08:21:01.498770+0000","last_fullsized":"2024-02-05T08:21:01.498770+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:20:19.087309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.087309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:44.997318+0000","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":34,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.490105+0000","last_change":"2024-02-05T08:19:44.914853+0000","last_active":"2024-02-05T08:21:01.490105+0000","last_peered":"2024-02-05T08:21:01.490105+0000","last_clean":"2024-02-05T08:04:44.536163+0000","last_became_active":"2024-02-05T08:19:44.914853+0000","last_became_peered":"2024-02-05T08:19:44.914853+0000","last_unstale":"2024-02-05T08:21:01.490105+0000","last_undegraded":"2024-02-05T08:21:01.490105+0000","last_fullsized":"2024-02-05T08:19:44.909093+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:45.429303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:45.429303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":345,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.492844+0000","last_change":"2024-02-05T08:19:44.915255+0000","last_active":"2024-02-05T08:21:01.492844+0000","last_peered":"2024-02-05T08:21:01.492844+0000","last_clean":"2024-02-05T08:19:41.829592+0000","last_became_active":"2024-02-05T08:19:44.915255+0000","last_became_peered":"2024-02-05T08:19:44.915255+0000","last_unstale":"2024-02-05T08:21:01.492844+0000","last_undegraded":"2024-02-05T08:21:01.492844+0000","last_fullsized":"2024-02-05T08:19:44.872335+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:10.372287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:19:10.372287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"236'31","reported_seq":653,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:01.489503+0000","last_change":"2024-02-05T08:19:44.922652+0000","last_active":"2024-02-05T08:21:01.489503+0000","last_peered":"2024-02-05T08:21:01.489503+0000","last_clean":"2024-02-05T08:19:41.830877+0000","last_became_active":"2024-02-05T08:19:44.922652+0000","last_became_peered":"2024-02-05T08:19:44.922652+0000","last_unstale":"2024-02-05T08:21:01.489503+0000","last_undegraded":"2024-02-05T08:19:44.913741+0000","last_fullsized":"2024-02-05T08:19:44.913632+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'31","last_scrub_stamp":"2024-02-05T08:19:24.342770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:24.342770+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135714+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"337'9142","reported_seq":9211,"reported_epoch":337,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:02.666335+0000","last_change":"2024-02-05T08:19:44.915552+0000","last_active":"2024-02-05T08:21:02.666335+0000","last_peered":"2024-02-05T08:21:02.666335+0000","last_clean":"2024-02-05T08:19:43.841958+0000","last_became_active":"2024-02-05T08:19:44.915552+0000","last_became_peered":"2024-02-05T08:19:44.915552+0000","last_unstale":"2024-02-05T08:21:02.666335+0000","last_undegraded":"2024-02-05T08:19:44.912000+0000","last_fullsized":"2024-02-05T08:19:44.910014+0000","mapping_epoch":324,"log_start":"326'4700","ondisk_log_start":"326'4700","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":485359616,"num_objects":7406,"num_object_clones":0,"num_object_copies":22218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7406,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9142,"num_write_kb":529536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":7406}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"82'130","reported_seq":911,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.497512+0000","last_change":"2024-02-05T08:19:09.153997+0000","last_active":"2024-02-05T08:21:01.497512+0000","last_peered":"2024-02-05T08:21:01.497512+0000","last_clean":"2024-02-05T08:21:01.497512+0000","last_became_active":"2024-02-05T08:19:01.816744+0000","last_became_peered":"2024-02-05T08:19:01.816744+0000","last_unstale":"2024-02-05T08:21:01.497512+0000","last_undegraded":"2024-02-05T08:21:01.497512+0000","last_fullsized":"2024-02-05T08:21:01.497512+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:09.153860+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:09.153860+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:05.016857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":596,"reported_epoch":337,"state":"active+undersized","last_fresh":"2024-02-05T08:21:01.495459+0000","last_change":"2024-02-05T08:19:44.927513+0000","last_active":"2024-02-05T08:21:01.495459+0000","last_peered":"2024-02-05T08:21:01.495459+0000","last_clean":"2024-02-05T08:19:41.831929+0000","last_became_active":"2024-02-05T08:19:44.927513+0000","last_became_peered":"2024-02-05T08:19:44.927513+0000","last_unstale":"2024-02-05T08:21:01.495459+0000","last_undegraded":"2024-02-05T08:21:01.495459+0000","last_fullsized":"2024-02-05T08:19:44.916139+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.282641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.282641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:02.135485+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":827,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.498236+0000","last_change":"2024-02-05T08:20:15.874649+0000","last_active":"2024-02-05T08:21:01.498236+0000","last_peered":"2024-02-05T08:21:01.498236+0000","last_clean":"2024-02-05T08:21:01.498236+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:21:01.498236+0000","last_undegraded":"2024-02-05T08:21:01.498236+0000","last_fullsized":"2024-02-05T08:21:01.498236+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:20:15.874421+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:20:15.874421+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:42.693872+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":112,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.496145+0000","last_change":"2024-02-05T08:20:20.712863+0000","last_active":"2024-02-05T08:21:01.496145+0000","last_peered":"2024-02-05T08:21:01.496145+0000","last_clean":"2024-02-05T08:21:01.496145+0000","last_became_active":"2024-02-05T08:18:34.663738+0000","last_became_peered":"2024-02-05T08:18:34.663738+0000","last_unstale":"2024-02-05T08:21:01.496145+0000","last_undegraded":"2024-02-05T08:21:01.496145+0000","last_fullsized":"2024-02-05T08:21:01.496145+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:20.711675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:20.711675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:45.175334+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"236'20","reported_seq":711,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-02-05T08:21:01.490283+0000","last_change":"2024-02-05T08:20:12.117923+0000","last_active":"2024-02-05T08:21:01.490283+0000","last_peered":"2024-02-05T08:21:01.490283+0000","last_clean":"2024-02-05T08:21:01.490283+0000","last_became_active":"2024-02-05T08:18:34.637413+0000","last_became_peered":"2024-02-05T08:18:34.637413+0000","last_unstale":"2024-02-05T08:21:01.490283+0000","last_undegraded":"2024-02-05T08:21:01.490283+0000","last_fullsized":"2024-02-05T08:21:01.490283+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:20:12.117355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:20:12.117355+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:16.064157+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":7666728984,"num_objects":116986,"num_object_clones":0,"num_object_copies":350958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21869,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":116986,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145934,"num_write_kb":8413377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22189322240,"data_stored":22189310024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84815,"ondisk_log_size":84815,"up":45,"acting":45,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":972,"num_read_kb":967,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":7,"acting":7,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":84,"acting":84,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":5462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":766,"ondisk_log_size":766,"up":81,"acting":81,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":82,"acting":82,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":85,"acting":85,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037961,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2426672,"kb_used_data":1876264,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":91318480,"statfs":{"total":95995035648,"available":93510123520,"internally_reserved":0,"allocated":1921294336,"data_stored":1920989704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":563608010},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:20:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.3120000000000001,"15min":1.5760000000000001},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.0069999999999997,"5min":27.376000000000001,"15min":60.127000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.4419999999999999,"15min":1.661},"min":{"1min":0.254,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":4.2169999999999996,"5min":27.456,"15min":58.283000000000001},"last":0.24099999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.544,"15min":1.512},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.22900000000000001},"max":{"1min":2.7810000000000001,"5min":27.518999999999998,"15min":38.399999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.083,"5min":1.4330000000000001,"15min":1.3680000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.22500000000000001},"max":{"1min":4.2409999999999997,"5min":27.503,"15min":38.195999999999998},"last":0.32200000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:20:57 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.071,"15min":1.034},"min":{"1min":0.27200000000000002,"5min":0.26100000000000001,"15min":0.24099999999999999},"max":{"1min":4.0949999999999998,"5min":7.5709999999999997,"15min":7.6779999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.2070000000000001,"15min":1.173},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.24299999999999999},"max":{"1min":4.7199999999999998,"5min":5.9400000000000004,"15min":8.9100000000000001},"last":0.51300000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:20:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.135,"15min":1.3640000000000001},"min":{"1min":0.31,"5min":0.29699999999999999,"15min":0.249},"max":{"1min":4.6840000000000002,"5min":5.984,"15min":58.515000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.244,"5min":1.0620000000000001,"15min":1.3520000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.27000000000000002},"max":{"1min":2.573,"5min":4.0960000000000001,"15min":58.225999999999999},"last":0.34100000000000003}]},{"osd":4,"last update":"Mon Feb 5 08:20:57 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.1220000000000001,"15min":0.98699999999999999},"min":{"1min":0.26600000000000001,"5min":0.24099999999999999,"15min":0.22},"max":{"1min":3.9809999999999999,"5min":7.6340000000000003,"15min":49.601999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.3100000000000001,"15min":1.0629999999999999},"min":{"1min":0.39100000000000001,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":4.6360000000000001,"5min":7.8399999999999999,"15min":49.795999999999999},"last":0.69599999999999995}]},{"osd":5,"last update":"Mon Feb 5 08:20:57 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.204,"15min":1.056},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.24299999999999999},"max":{"1min":3.044,"5min":5.1769999999999996,"15min":49.914000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.127,"5min":1.452,"15min":1.081},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.27000000000000002},"max":{"1min":4.0309999999999997,"5min":9.4459999999999997,"15min":52.256},"last":0.60199999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:20:57 2024","interfaces":[{"interface":"back","average":{"1min":2.0699999999999998,"5min":2.2919999999999998,"15min":1.238},"min":{"1min":0.48999999999999999,"5min":0.28799999999999998,"15min":0.28599999999999998},"max":{"1min":5.8550000000000004,"5min":84.254999999999995,"15min":84.254999999999995},"last":0.373},{"interface":"front","average":{"1min":1.8520000000000001,"5min":2.347,"15min":1.2150000000000001},"min":{"1min":0.36499999999999999,"5min":0.32500000000000001,"15min":0.27300000000000002},"max":{"1min":7.1859999999999999,"5min":84.667000000000002,"15min":84.667000000000002},"last":0.67000000000000004}]}]},{"osd":6,"up_from":51,"seq":219043332303,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3731172,"kb_used_data":3276568,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":90013980,"statfs":{"total":95995035648,"available":92174315520,"internally_reserved":0,"allocated":3355205632,"data_stored":3354915881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":465500614},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.7549999999999999,"15min":1.7769999999999999},"min":{"1min":0.33200000000000002,"5min":0.215,"15min":0.215},"max":{"1min":8.2560000000000002,"5min":76.123000000000005,"15min":102.819},"last":1.637},{"interface":"front","average":{"1min":1.206,"5min":1.6779999999999999,"15min":1.829},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":2.9809999999999999,"5min":76.037000000000006,"15min":102.75},"last":1.425}]},{"osd":1,"last update":"Mon Feb 5 08:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.90900000000000003,"15min":1.169},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.22700000000000001},"max":{"1min":1.1970000000000001,"5min":6.0229999999999997,"15min":23.773},"last":0.503},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.96299999999999997,"15min":1.1319999999999999},"min":{"1min":0.30299999999999999,"5min":0.26300000000000001,"15min":0.23400000000000001},"max":{"1min":3.0249999999999999,"5min":5.0209999999999999,"15min":23.719000000000001},"last":0.73699999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":2.0150000000000001,"5min":1.143,"15min":1.159},"min":{"1min":0.32300000000000001,"5min":0.27700000000000002,"15min":0.26300000000000001},"max":{"1min":7.3499999999999996,"5min":7.3499999999999996,"15min":22.503},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.321,"5min":1.054,"15min":1.0589999999999999},"min":{"1min":0.28199999999999997,"5min":0.27300000000000002,"15min":0.23400000000000001},"max":{"1min":6.9720000000000004,"5min":6.9720000000000004,"15min":22.466000000000001},"last":1.494}]},{"osd":3,"last update":"Mon Feb 5 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.83499999999999996,"15min":1.3400000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":1.546,"5min":3.4689999999999999,"15min":102.438},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.615,"5min":1.1100000000000001,"15min":1.419},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29099999999999998},"max":{"1min":7.4000000000000004,"5min":7.4000000000000004,"15min":102.288},"last":0.622}]},{"osd":4,"last update":"Mon Feb 5 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.1120000000000001,"15min":0.73999999999999999},"min":{"1min":0.30299999999999999,"5min":0.251,"15min":0.223},"max":{"1min":3.8780000000000001,"5min":4.9740000000000002,"15min":9.8970000000000002},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.679,"5min":1.143,"15min":0.79400000000000004},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":5.3789999999999996,"5min":5.3789999999999996,"15min":5.3789999999999996},"last":1.534}]},{"osd":5,"last update":"Mon Feb 5 08:20:33 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.5509999999999999,"15min":1.0900000000000001},"min":{"1min":0.33200000000000002,"5min":0.29299999999999998,"15min":0.24199999999999999},"max":{"1min":3.8079999999999998,"5min":24.675999999999998,"15min":43.643000000000001},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.204,"5min":1.452,"15min":1.075},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":2.7490000000000001,"5min":19.132000000000001,"15min":43.893000000000001},"last":1.6120000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:20:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":1.5900000000000001,"15min":0.93000000000000005},"min":{"1min":0.40500000000000003,"5min":0.376,"15min":0.29999999999999999},"max":{"1min":2.6720000000000002,"5min":24.728999999999999,"15min":24.728999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.774,"15min":0.996},"min":{"1min":0.375,"5min":0.32200000000000001,"15min":0.29299999999999998},"max":{"1min":5.4980000000000002,"5min":21.91,"15min":21.91},"last":1.5600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1172192,"kb_used_data":617624,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92572960,"statfs":{"total":95995035648,"available":94794711040,"internally_reserved":0,"allocated":632446976,"data_stored":632002797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1659999999999999,"15min":1.625},"min":{"1min":0.221,"5min":0.183,"15min":0.183},"max":{"1min":1.0660000000000001,"5min":36.462000000000003,"15min":86.105000000000004},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.331,"15min":1.907},"min":{"1min":0.26300000000000001,"5min":0.21099999999999999,"15min":0.20699999999999999},"max":{"1min":8.0340000000000007,"5min":35.878,"15min":85.665000000000006},"last":0.40600000000000003}]},{"osd":2,"last update":"Mon Feb 5 08:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.84299999999999997,"15min":1.373},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.191},"max":{"1min":1.4930000000000001,"5min":4.2859999999999996,"15min":49.030999999999999},"last":2.855},{"interface":"front","average":{"1min":1.2210000000000001,"5min":0.92100000000000004,"15min":1.599},"min":{"1min":0.255,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":7.298,"5min":7.298,"15min":48.984000000000002},"last":2.5550000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.754,"15min":1.5189999999999999},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.20599999999999999},"max":{"1min":1.026,"5min":5.3289999999999997,"15min":49.067999999999998},"last":1.252},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.81699999999999995,"15min":1.46},"min":{"1min":0.32100000000000001,"5min":0.254,"15min":0.24199999999999999},"max":{"1min":2.3119999999999998,"5min":4.3529999999999998,"15min":48.734999999999999},"last":1.212}]},{"osd":4,"last update":"Mon Feb 5 08:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.4119999999999999,"15min":1.272},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.26000000000000001},"max":{"1min":1.2030000000000001,"5min":24.196999999999999,"15min":25.952000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":1.3100000000000001,"15min":1.3600000000000001},"min":{"1min":0.33700000000000002,"5min":0.31,"15min":0.27100000000000002},"max":{"1min":0.92100000000000004,"5min":24.215,"15min":25.922000000000001},"last":1.228}]},{"osd":5,"last update":"Mon Feb 5 08:18:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.492,"15min":1.3999999999999999},"min":{"1min":0.40100000000000002,"5min":0.39400000000000002,"15min":0.28699999999999998},"max":{"1min":3.1699999999999999,"5min":9.6820000000000004,"15min":54.552999999999997},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.296,"15min":1.4379999999999999},"min":{"1min":0.32000000000000001,"5min":0.311,"15min":0.28999999999999998},"max":{"1min":4.1239999999999997,"5min":6.0259999999999998,"15min":55.811},"last":1.1259999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.97399999999999998,"15min":1.2390000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.5069999999999999,"5min":3.6819999999999999,"15min":54.384},"last":1.196},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.0920000000000001,"15min":1.2470000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.34899999999999998},"max":{"1min":1.456,"5min":4.4370000000000003,"15min":55.210999999999999},"last":0.56399999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:18:52 2024","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.45,"15min":1.3089999999999999},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":5.7640000000000002,"5min":6.0579999999999998,"15min":13.749000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.3280000000000001,"15min":1.242},"min":{"1min":0.42399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":2.4020000000000001,"5min":4.6180000000000003,"15min":7.9640000000000004},"last":1.1579999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738605,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5301688,"kb_used_data":4666544,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":88443464,"statfs":{"total":95995035648,"available":90566107136,"internally_reserved":0,"allocated":4778541056,"data_stored":4778227999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.72199999999999998,"15min":1.79},"min":{"1min":0.185,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":1.4279999999999999,"5min":7.9580000000000002,"15min":66.772999999999996},"last":0.755},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.745,"15min":1.5409999999999999},"min":{"1min":0.27100000000000002,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":2.169,"5min":6.569,"15min":46.198999999999998},"last":1.016}]},{"osd":2,"last update":"Mon Feb 5 08:20:56 2024","interfaces":[{"interface":"back","average":{"1min":6.3049999999999997,"5min":1.869,"15min":1.9359999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":105.96599999999999,"5min":105.96599999999999,"15min":105.96599999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":5.8769999999999998,"5min":1.7589999999999999,"15min":2.0350000000000001},"min":{"1min":0.19900000000000001,"5min":0.182,"15min":0.182},"max":{"1min":106.137,"5min":106.137,"15min":106.137},"last":1.0860000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:20:27 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":0.87,"15min":1.7549999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":5.9029999999999996,"5min":6.7439999999999998,"15min":67.888000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.2070000000000001,"5min":0.83299999999999996,"15min":2.04},"min":{"1min":0.26900000000000002,"5min":0.23999999999999999,"15min":0.22600000000000001},"max":{"1min":3.7749999999999999,"5min":3.7749999999999999,"15min":66.891999999999996},"last":1.2210000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:20:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.4179999999999999,"15min":1.3839999999999999},"min":{"1min":0.34100000000000003,"5min":0.29299999999999998,"15min":0.26300000000000001},"max":{"1min":30.914000000000001,"5min":30.914000000000001,"15min":30.914000000000001},"last":1.3400000000000001},{"interface":"front","average":{"1min":2.859,"5min":1.609,"15min":1.3959999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.26900000000000002},"max":{"1min":30.93,"5min":30.93,"15min":30.93},"last":0.92100000000000004}]},{"osd":5,"last update":"Mon Feb 5 08:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":1.716,"15min":1.49},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.28299999999999997},"max":{"1min":1.992,"5min":38.979999999999997,"15min":38.979999999999997},"last":1.476},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.7889999999999999,"15min":1.4450000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":6.0780000000000003,"5min":38.960999999999999,"15min":38.960999999999999},"last":0.94799999999999995}]},{"osd":6,"last update":"Mon Feb 5 08:20:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.1759999999999999,"15min":1.24},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.30199999999999999},"max":{"1min":6.1109999999999998,"5min":6.9749999999999996,"15min":12.967000000000001},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.133,"15min":1.304},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.2509999999999999,"5min":4.1210000000000004,"15min":15.721},"last":1.444}]},{"osd":7,"last update":"Mon Feb 5 08:20:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":2.367,"15min":1.74},"min":{"1min":0.317,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.6079999999999997,"5min":68.241,"15min":68.241},"last":1.298},{"interface":"front","average":{"1min":1.7010000000000001,"5min":2.48,"15min":1.804},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":6.1369999999999996,"5min":68.980000000000004,"15min":68.980000000000004},"last":2.121}]}]},{"osd":2,"up_from":18,"seq":77309411555,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3397756,"kb_used_data":2829296,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":90347396,"statfs":{"total":95995035648,"available":92515733504,"internally_reserved":0,"allocated":2897199104,"data_stored":2896886020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":582089158},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.244,"15min":2.1989999999999998},"min":{"1min":0.23499999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":6.181,"5min":30.731999999999999,"15min":107.155},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.4159999999999999,"15min":2.3809999999999998},"min":{"1min":0.26900000000000002,"5min":0.22600000000000001,"15min":0.20799999999999999},"max":{"1min":6.468,"5min":30.838999999999999,"15min":107.221},"last":1.2969999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:18:54 2024","interfaces":[{"interface":"back","average":{"1min":2.4540000000000002,"5min":1.167,"15min":1.5860000000000001},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.17999999999999999},"max":{"1min":35.695999999999998,"5min":35.695999999999998,"15min":55.942},"last":0.28299999999999997},{"interface":"front","average":{"1min":2.9209999999999998,"5min":1.581,"15min":1.752},"min":{"1min":0.216,"5min":0.216,"15min":0.193},"max":{"1min":30.765000000000001,"5min":30.765000000000001,"15min":55.960999999999999},"last":0.79700000000000004}]},{"osd":3,"last update":"Mon Feb 5 08:20:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":0.90200000000000002,"15min":1.835},"min":{"1min":0.25800000000000001,"5min":0.222,"15min":0.20499999999999999},"max":{"1min":6.5039999999999996,"5min":8.3629999999999995,"15min":68.108000000000004},"last":1.157},{"interface":"front","average":{"1min":1.242,"5min":0.75600000000000001,"15min":1.6379999999999999},"min":{"1min":0.23999999999999999,"5min":0.223,"15min":0.223},"max":{"1min":6.101,"5min":6.6920000000000002,"15min":67.945999999999998},"last":0.36699999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:20:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.1850000000000001,"15min":1.448},"min":{"1min":0.36599999999999999,"5min":0.28100000000000003,"15min":0.27400000000000002},"max":{"1min":7.6189999999999998,"5min":8.2620000000000005,"15min":31.596},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.8129999999999999,"5min":1.171,"15min":1.3080000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.252},"max":{"1min":7.6509999999999998,"5min":9.5060000000000002,"15min":31.57},"last":0.41299999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:20:08 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.6140000000000001,"15min":1.575},"min":{"1min":0.27300000000000002,"5min":0.246,"15min":0.246},"max":{"1min":1.9390000000000001,"5min":26.373000000000001,"15min":58.799999999999997},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.476,"15min":1.518},"min":{"1min":0.34100000000000003,"5min":0.32300000000000001,"15min":0.26900000000000002},"max":{"1min":3.1709999999999998,"5min":27.968,"15min":58.615000000000002},"last":0.61499999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:20:47 2024","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.1779999999999999,"15min":1.3009999999999999},"min":{"1min":0.44900000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":6.1609999999999996,"5min":8.3089999999999993,"15min":15.859},"last":1.226},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.1799999999999999,"15min":1.167},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":6.2709999999999999,"5min":9.1739999999999995,"15min":9.1739999999999995},"last":0.72299999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.2030000000000001,"15min":1.2929999999999999},"min":{"1min":0.35299999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.1280000000000001,"5min":7.7530000000000001,"15min":15.986000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.587,"15min":1.4950000000000001},"min":{"1min":0.45600000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":6.2080000000000002,"5min":11.945,"15min":15.951000000000001},"last":1.26}]}]},{"osd":3,"up_from":27,"seq":115964117214,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2883528,"kb_used_data":2369216,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":90861624,"statfs":{"total":95995035648,"available":93042302976,"internally_reserved":0,"allocated":2426077184,"data_stored":2425748373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":526645705},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:20:23 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.4039999999999999,"15min":1.8420000000000001},"min":{"1min":0.23100000000000001,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":11.747999999999999,"5min":52.831000000000003,"15min":63.43},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.988,"5min":1.6399999999999999,"15min":1.9850000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.184},"max":{"1min":11.618,"5min":53.859999999999999,"15min":63.048999999999999},"last":0.317}]},{"osd":1,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.93500000000000005,"15min":1.5109999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.217},"max":{"1min":3.3119999999999998,"5min":23.154,"15min":51.405999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.89200000000000002,"15min":1.8260000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.17399999999999999},"max":{"1min":3.54,"5min":23.207000000000001,"15min":51.189999999999998},"last":2.1379999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:20:23 2024","interfaces":[{"interface":"back","average":{"1min":3.2570000000000001,"5min":1.147,"15min":1.712},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":35.277000000000001,"5min":35.277000000000001,"15min":62.902999999999999},"last":1.3120000000000001},{"interface":"front","average":{"1min":2.9929999999999999,"5min":1.034,"15min":1.51},"min":{"1min":0.23300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":35.442999999999998,"5min":35.442999999999998,"15min":63.468000000000004},"last":2.633}]},{"osd":4,"last update":"Mon Feb 5 08:20:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":0.93999999999999995,"15min":1.107},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.298,"5min":6.298,"15min":7.7229999999999999},"last":1.994},{"interface":"front","average":{"1min":1.034,"5min":0.90800000000000003,"15min":1.149},"min":{"1min":0.33800000000000002,"5min":0.32900000000000001,"15min":0.29599999999999999},"max":{"1min":2.8220000000000001,"5min":5.8159999999999998,"15min":10.539999999999999},"last":0.69299999999999995}]},{"osd":5,"last update":"Mon Feb 5 08:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":2.7709999999999999,"15min":1.794},"min":{"1min":0.34100000000000003,"5min":0.32500000000000001,"15min":0.255},"max":{"1min":2.5449999999999999,"5min":117.432,"15min":117.432},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":2.7469999999999999,"15min":1.841},"min":{"1min":0.309,"5min":0.309,"15min":0.30299999999999999},"max":{"1min":2.6779999999999999,"5min":114.828,"15min":114.828},"last":0.27600000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:20:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":0.90800000000000003,"15min":1.206},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.4750000000000001,"5min":3.6360000000000001,"15min":10.622},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.821,"5min":1.0269999999999999,"15min":1.2270000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.308},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":7.8639999999999999},"last":1.345}]},{"osd":7,"last update":"Mon Feb 5 08:20:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.212,"15min":1.764},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":6.3730000000000002,"5min":6.3730000000000002,"15min":114.84999999999999},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.0289999999999999,"15min":1.7809999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33600000000000002},"max":{"1min":3.0569999999999999,"5min":6.0529999999999999,"15min":114.871},"last":0.38800000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953689,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4322876,"kb_used_data":3772464,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":89422276,"statfs":{"total":95995035648,"available":91568410624,"internally_reserved":0,"allocated":3863003136,"data_stored":3862688983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.2430000000000001,"15min":1.7210000000000001},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":8.6470000000000002,"5min":25.600999999999999,"15min":73.078999999999994},"last":1.6479999999999999},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.3260000000000001,"15min":1.704},"min":{"1min":0.35799999999999998,"5min":0.23499999999999999,"15min":0.21099999999999999},"max":{"1min":2.4380000000000002,"5min":23.030999999999999,"15min":73.412000000000006},"last":0.27600000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.97299999999999998,"15min":1.486},"min":{"1min":0.30399999999999999,"5min":0.23300000000000001,"15min":0.21099999999999999},"max":{"1min":1.4590000000000001,"5min":8.3320000000000007,"15min":125.054},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.89300000000000002,"15min":1.387},"min":{"1min":0.309,"5min":0.25700000000000001,"15min":0.23499999999999999},"max":{"1min":1.1910000000000001,"5min":6.2370000000000001,"15min":125.08799999999999},"last":3.1699999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.056,"15min":1.3080000000000001},"min":{"1min":0.27500000000000002,"5min":0.26500000000000001,"15min":0.24199999999999999},"max":{"1min":3.9729999999999999,"5min":4.8159999999999998,"15min":96.849999999999994},"last":1.6799999999999999},{"interface":"front","average":{"1min":0.94399999999999995,"5min":1.0580000000000001,"15min":1.359},"min":{"1min":0.37,"5min":0.27100000000000002,"15min":0.26300000000000001},"max":{"1min":3.3500000000000001,"5min":5.6879999999999997,"15min":96.143000000000001},"last":1.3200000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6200000000000001,"5min":1.181,"15min":1.9079999999999999},"min":{"1min":0.38600000000000001,"5min":0.33100000000000002,"15min":0.24399999999999999},"max":{"1min":8.9269999999999996,"5min":8.9269999999999996,"15min":73.602999999999994},"last":0.312},{"interface":"front","average":{"1min":1.7230000000000001,"5min":1.1180000000000001,"15min":1.9119999999999999},"min":{"1min":0.35299999999999998,"5min":0.309,"15min":0.27200000000000002},"max":{"1min":4.2599999999999998,"5min":4.2599999999999998,"15min":74.817999999999998},"last":0.372}]},{"osd":5,"last update":"Mon Feb 5 08:20:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.282,"15min":0.92300000000000004},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.224},"max":{"1min":4.54,"5min":9.157,"15min":34.167000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.1599999999999999,"15min":0.95199999999999996},"min":{"1min":0.308,"5min":0.254,"15min":0.254},"max":{"1min":1.7589999999999999,"5min":5.6550000000000002,"15min":36.051000000000002},"last":0.24199999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:20:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.2070000000000001,"15min":0.79800000000000004},"min":{"1min":0.34499999999999997,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.5990000000000002,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":2.1240000000000001},{"interface":"front","average":{"1min":1.9319999999999999,"5min":1.2749999999999999,"15min":0.79600000000000004},"min":{"1min":0.29099999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.2969999999999997,"5min":5.5049999999999999,"15min":5.5049999999999999},"last":2.0070000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:20:48 2024","interfaces":[{"interface":"back","average":{"1min":2.1360000000000001,"5min":1.4279999999999999,"15min":0.91200000000000003},"min":{"1min":0.40300000000000002,"5min":0.307,"15min":0.27000000000000002},"max":{"1min":12.683999999999999,"5min":12.683999999999999,"15min":12.683999999999999},"last":2.7970000000000002},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.3120000000000001,"15min":0.878},"min":{"1min":0.36099999999999999,"5min":0.33900000000000002,"15min":0.27900000000000003},"max":{"1min":4.5110000000000001,"5min":4.6529999999999996,"15min":5.7450000000000001},"last":1.355}]}]},{"osd":5,"up_from":42,"seq":180388626643,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2949736,"kb_used_data":2359068,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":90795416,"statfs":{"total":95995035648,"available":92974505984,"internally_reserved":0,"allocated":2415685632,"data_stored":2415392342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.7210000000000001,"15min":1.9830000000000001},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":3.0739999999999998,"5min":30.065999999999999,"15min":143.06299999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.6759999999999999,"15min":1.9279999999999999},"min":{"1min":0.27600000000000002,"5min":0.248,"15min":0.20499999999999999},"max":{"1min":4.0350000000000001,"5min":30.173999999999999,"15min":143.09700000000001},"last":3.0859999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:18:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.321,"15min":1.163},"min":{"1min":0.42799999999999999,"5min":0.22500000000000001,"15min":0.215},"max":{"1min":2.8090000000000002,"5min":5.8940000000000001,"15min":46.200000000000003},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.1859999999999999,"15min":1.23},"min":{"1min":0.29699999999999999,"5min":0.246,"15min":0.22900000000000001},"max":{"1min":4.7969999999999997,"5min":5.2210000000000001,"15min":46.343000000000004},"last":0.35099999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.282,"15min":1.177},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":17.09,"5min":17.09,"15min":17.09},"last":0.52200000000000002},{"interface":"front","average":{"1min":2.1600000000000001,"5min":1.458,"15min":1.298},"min":{"1min":0.33100000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":17.515999999999998,"5min":17.515999999999998,"15min":17.515999999999998},"last":0.80800000000000005}]},{"osd":3,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.1679999999999999,"15min":1.76},"min":{"1min":0.34000000000000002,"5min":0.29199999999999998,"15min":0.28799999999999998},"max":{"1min":4.0819999999999999,"5min":7.165,"15min":143.03399999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.139,"15min":1.863},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.1030000000000002,"5min":5.8609999999999998,"15min":143.46799999999999},"last":0.83599999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1020000000000001,"15min":0.84199999999999997},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.7389999999999999,"5min":7.7389999999999999,"15min":7.7389999999999999},"last":3.1230000000000002},{"interface":"front","average":{"1min":0.877,"5min":1.244,"15min":0.82599999999999996},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.23999999999999999},"max":{"1min":4.1349999999999998,"5min":7.2380000000000004,"15min":7.2380000000000004},"last":0.54700000000000004}]},{"osd":6,"last update":"Mon Feb 5 08:20:35 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.425,"15min":0.872},"min":{"1min":0.34599999999999997,"5min":0.33200000000000002,"15min":0.26600000000000001},"max":{"1min":4.0170000000000003,"5min":9.4100000000000001,"15min":9.4100000000000001},"last":3.871},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.357,"15min":0.872},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.316},"max":{"1min":4.0510000000000002,"5min":7.194,"15min":7.194},"last":0.56899999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.5089999999999999,"15min":1.034},"min":{"1min":0.35399999999999998,"5min":0.28799999999999998,"15min":0.25900000000000001},"max":{"1min":5.0250000000000004,"5min":7.0330000000000004,"15min":9.6940000000000008},"last":5.0250000000000004},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.464,"15min":0.97399999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":4.1440000000000001,"5min":9.2360000000000007,"15min":9.2360000000000007},"last":3.149}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4747563008,"data_stored":4747558936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":645464064,"data_stored":645464064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2889744384,"data_stored":2889744384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2411921408,"data_stored":2411921408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3841396736,"data_stored":3841392664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2392326144,"data_stored":2392326144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3346599936,"data_stored":3346595864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1914306560,"data_stored":1914306560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:21:08.790 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool set unique_pool_3 pgp_num 16 2024-02-05T08:21:08.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:08 smithi196 ceph-mon[33018]: 5.1c scrub starts 2024-02-05T08:21:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:08 smithi196 ceph-mon[33018]: 5.1c scrub ok 2024-02-05T08:21:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:08 smithi196 ceph-mon[33018]: from='client.15819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:21:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:08 smithi196 ceph-mon[33018]: 5.2 scrub starts 2024-02-05T08:21:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:08 smithi196 ceph-mon[33018]: 5.2 scrub ok 2024-02-05T08:21:08.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:21:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:08.821 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:08.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:08.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:08.949 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:08.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:09.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:09.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:09.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:09.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:09.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:09.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:09.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:09.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:09.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:09.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:09.368 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:09.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:09.439 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:09.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:09.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:09.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:09.604 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:09.642 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:09.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:09.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:09.664 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[30086]: pgmap v731: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s; 21603/348450 objects degraded (6.200%) 2024-02-05T08:21:09.664 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[30086]: 3.9 scrub starts 2024-02-05T08:21:09.664 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[30086]: 3.9 scrub ok 2024-02-05T08:21:09.664 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[30086]: 3.7 scrub starts 2024-02-05T08:21:09.665 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[30086]: 3.7 scrub ok 2024-02-05T08:21:09.665 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[30086]: 3.10 scrub starts 2024-02-05T08:21:09.665 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[30086]: 3.10 scrub ok 2024-02-05T08:21:09.665 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[39278]: pgmap v731: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s; 21603/348450 objects degraded (6.200%) 2024-02-05T08:21:09.665 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[39278]: 3.9 scrub starts 2024-02-05T08:21:09.665 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[39278]: 3.9 scrub ok 2024-02-05T08:21:09.665 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[39278]: 3.7 scrub starts 2024-02-05T08:21:09.665 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[39278]: 3.7 scrub ok 2024-02-05T08:21:09.665 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[39278]: 3.10 scrub starts 2024-02-05T08:21:09.665 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:09 smithi134 ceph-mon[39278]: 3.10 scrub ok 2024-02-05T08:21:09.744 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:09.780 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:09.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:09.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:09 smithi196 ceph-mon[33018]: pgmap v731: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 7.1 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s; 21603/348450 objects degraded (6.200%) 2024-02-05T08:21:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:09 smithi196 ceph-mon[33018]: 3.9 scrub starts 2024-02-05T08:21:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:09 smithi196 ceph-mon[33018]: 3.9 scrub ok 2024-02-05T08:21:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:09 smithi196 ceph-mon[33018]: 3.7 scrub starts 2024-02-05T08:21:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:09 smithi196 ceph-mon[33018]: 3.7 scrub ok 2024-02-05T08:21:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:09 smithi196 ceph-mon[33018]: 3.10 scrub starts 2024-02-05T08:21:09.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:09 smithi196 ceph-mon[33018]: 3.10 scrub ok 2024-02-05T08:21:09.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:09.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:09.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:09.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:10.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:10.048 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:10.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:10.149 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:10.179 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:10.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:10.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:10.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:10.324 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:10.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:10.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:10.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:10.465 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:10.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:10.567 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:10.605 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:10.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:10.706 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:10.727 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:21:10.727 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:21:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:10.749 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:10.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:10 smithi196 ceph-mon[33018]: 3.13 scrub starts 2024-02-05T08:21:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:10 smithi196 ceph-mon[33018]: 3.13 scrub ok 2024-02-05T08:21:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:10 smithi196 ceph-mon[33018]: 10.7 scrub ok 2024-02-05T08:21:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:10 smithi196 ceph-mon[33018]: 4.1f scrub starts 2024-02-05T08:21:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:10 smithi196 ceph-mon[33018]: 4.1f scrub ok 2024-02-05T08:21:10.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:10 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 21603/348450 objects degraded (6.200%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:10.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:10.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:10.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:10.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:11.023 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:11.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:11.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:10 smithi134 ceph-mon[30086]: 3.13 scrub starts 2024-02-05T08:21:11.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:10 smithi134 ceph-mon[30086]: 3.13 scrub ok 2024-02-05T08:21:11.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:10 smithi134 ceph-mon[30086]: 10.7 scrub ok 2024-02-05T08:21:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:10 smithi134 ceph-mon[30086]: 4.1f scrub starts 2024-02-05T08:21:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:10 smithi134 ceph-mon[30086]: 4.1f scrub ok 2024-02-05T08:21:11.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:10 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 21603/348450 objects degraded (6.200%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:10 smithi134 ceph-mon[39278]: 3.13 scrub starts 2024-02-05T08:21:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:10 smithi134 ceph-mon[39278]: 3.13 scrub ok 2024-02-05T08:21:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:10 smithi134 ceph-mon[39278]: 10.7 scrub ok 2024-02-05T08:21:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:10 smithi134 ceph-mon[39278]: 4.1f scrub starts 2024-02-05T08:21:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:10 smithi134 ceph-mon[39278]: 4.1f scrub ok 2024-02-05T08:21:11.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:10 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 21603/348450 objects degraded (6.200%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:11.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:11.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:11.163 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:11.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:11.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:11.298 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:11.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:11.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:11.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:11.438 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:11.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:11.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T08:21:11.540 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:11.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:11.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:11.687 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:21:11.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:11.736 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:11.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[30086]: pgmap v732: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 20943/334866 objects degraded (6.254%) 2024-02-05T08:21:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[30086]: 6.17 scrub starts 2024-02-05T08:21:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[30086]: 4.7 scrub starts 2024-02-05T08:21:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[30086]: 4.7 scrub ok 2024-02-05T08:21:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[30086]: 6.17 scrub ok 2024-02-05T08:21:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1421031270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:11.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:11.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[39278]: pgmap v732: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 20943/334866 objects degraded (6.254%) 2024-02-05T08:21:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[39278]: 6.17 scrub starts 2024-02-05T08:21:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[39278]: 4.7 scrub starts 2024-02-05T08:21:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[39278]: 4.7 scrub ok 2024-02-05T08:21:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[39278]: 6.17 scrub ok 2024-02-05T08:21:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1421031270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:11.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:11 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:11 smithi196 ceph-mon[33018]: pgmap v732: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 20943/334866 objects degraded (6.254%) 2024-02-05T08:21:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:11 smithi196 ceph-mon[33018]: 6.17 scrub starts 2024-02-05T08:21:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:11 smithi196 ceph-mon[33018]: 4.7 scrub starts 2024-02-05T08:21:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:11 smithi196 ceph-mon[33018]: 4.7 scrub ok 2024-02-05T08:21:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:11 smithi196 ceph-mon[33018]: 6.17 scrub ok 2024-02-05T08:21:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:11 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1421031270' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:11.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:11 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:11.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:11.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:11.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:11.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:11.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:12.030 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:12.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:12.059 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:21:12.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:12.179 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:12.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:12.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:12.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:12.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:12.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:12.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:12.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:12.462 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:12.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:12.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:12.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:12.612 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:12.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:12.707 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:12 smithi134 ceph-mon[30086]: 4.3 scrub starts 2024-02-05T08:21:12.707 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:12 smithi134 ceph-mon[30086]: 4.3 scrub ok 2024-02-05T08:21:12.707 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:12 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:21:12.707 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:12 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:21:12.708 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:12 smithi134 ceph-mon[30086]: osdmap e338: 8 total, 7 up, 8 in 2024-02-05T08:21:12.708 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:12 smithi134 ceph-mon[39278]: 4.3 scrub starts 2024-02-05T08:21:12.708 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:12 smithi134 ceph-mon[39278]: 4.3 scrub ok 2024-02-05T08:21:12.708 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:12 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:21:12.708 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:12 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:21:12.708 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:12 smithi134 ceph-mon[39278]: osdmap e338: 8 total, 7 up, 8 in 2024-02-05T08:21:12.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:12.732 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:12.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:12.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:12.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:12 smithi196 ceph-mon[33018]: 4.3 scrub starts 2024-02-05T08:21:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:12 smithi196 ceph-mon[33018]: 4.3 scrub ok 2024-02-05T08:21:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:12 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:21:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:12 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:21:12.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:12 smithi196 ceph-mon[33018]: osdmap e338: 8 total, 7 up, 8 in 2024-02-05T08:21:12.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:12.896 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:12.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:12.997 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:13.037 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:13.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:13.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:13.172 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:13.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:13.274 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:13.316 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:13.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:13.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:13.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:13.442 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:21:13.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:13.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:13.474 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:21:13.509 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:21:13.552 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:21:13.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:13.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:13.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:13.595 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:21:13.604 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:13 smithi134 ceph-mon[30086]: pgmap v734: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 20943/334866 objects degraded (6.254%) 2024-02-05T08:21:13.605 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:13 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1663806886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:13.605 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:13 smithi134 ceph-mon[39278]: pgmap v734: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 20943/334866 objects degraded (6.254%) 2024-02-05T08:21:13.605 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:13 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1663806886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:13.634 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:21:13.670 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:21:13.671 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:21:13.671 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:21:13.671 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:21:13.671 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:21:13.671 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:21:13.671 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:21:13.671 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:21:13.671 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:21:13.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:13.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:13.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:13.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:13.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:13 smithi196 ceph-mon[33018]: pgmap v734: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.17k op/s; 20943/334866 objects degraded (6.254%) 2024-02-05T08:21:13.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:13 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1663806886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:13.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:13.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:13.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:13.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:13.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:13.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:13.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:14.099 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:14.132 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:14.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:14.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:14.158 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:21:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:21:14.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:14.266 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:14.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:14.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:14.368 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:14.402 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:14.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:14.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:14.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:21:14.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:14.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:14.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:21:14.563 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:21:14.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:14.678 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:14.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:14.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:14.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:14.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:14.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:14.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:14.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:15.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:15.095 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:15.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:15.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:15.196 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:15.232 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:15.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:15.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:15.371 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:15.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:15.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:15.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:15.508 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:15.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:15.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:15.610 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:15.636 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[30086]: pgmap v735: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 19829/319101 objects degraded (6.214%) 2024-02-05T08:21:15.636 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:15.637 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[30086]: 4.1 scrub starts 2024-02-05T08:21:15.637 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[30086]: 4.1 scrub ok 2024-02-05T08:21:15.637 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[30086]: 6.12 deep-scrub starts 2024-02-05T08:21:15.637 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[30086]: 6.12 deep-scrub ok 2024-02-05T08:21:15.637 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[30086]: 6.0 scrub starts 2024-02-05T08:21:15.637 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[30086]: 6.0 scrub ok 2024-02-05T08:21:15.637 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 19829/319101 objects degraded (6.214%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:15.637 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[30086]: 3.c scrub starts 2024-02-05T08:21:15.637 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[30086]: 3.c scrub ok 2024-02-05T08:21:15.637 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[39278]: pgmap v735: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 19829/319101 objects degraded (6.214%) 2024-02-05T08:21:15.637 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:15.637 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[39278]: 4.1 scrub starts 2024-02-05T08:21:15.637 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[39278]: 4.1 scrub ok 2024-02-05T08:21:15.638 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[39278]: 6.12 deep-scrub starts 2024-02-05T08:21:15.638 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[39278]: 6.12 deep-scrub ok 2024-02-05T08:21:15.638 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[39278]: 6.0 scrub starts 2024-02-05T08:21:15.638 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[39278]: 6.0 scrub ok 2024-02-05T08:21:15.638 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 19829/319101 objects degraded (6.214%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:15.638 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[39278]: 3.c scrub starts 2024-02-05T08:21:15.638 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:15 smithi134 ceph-mon[39278]: 3.c scrub ok 2024-02-05T08:21:15.662 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:15.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:15.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:15.666 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:21:15.666 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:21:15.666 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T08:21:15.666 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T08:21:15.666 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:21:15.692 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:15.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:15.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:15.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:15.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:15 smithi196 ceph-mon[33018]: pgmap v735: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 19829/319101 objects degraded (6.214%) 2024-02-05T08:21:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:15 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:15 smithi196 ceph-mon[33018]: 4.1 scrub starts 2024-02-05T08:21:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:15 smithi196 ceph-mon[33018]: 4.1 scrub ok 2024-02-05T08:21:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:15 smithi196 ceph-mon[33018]: 6.12 deep-scrub starts 2024-02-05T08:21:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:15 smithi196 ceph-mon[33018]: 6.12 deep-scrub ok 2024-02-05T08:21:15.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:15 smithi196 ceph-mon[33018]: 6.0 scrub starts 2024-02-05T08:21:15.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:15 smithi196 ceph-mon[33018]: 6.0 scrub ok 2024-02-05T08:21:15.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:15 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 19829/319101 objects degraded (6.214%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:15.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:15 smithi196 ceph-mon[33018]: 3.c scrub starts 2024-02-05T08:21:15.816 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:15 smithi196 ceph-mon[33018]: 3.c scrub ok 2024-02-05T08:21:15.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:15.941 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:15.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:16.043 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:16.084 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:16.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:16.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:16.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:16.222 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:16.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:16.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:16.361 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:16.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:16.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:16.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:16.499 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:16.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:16.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:16.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:21:16.594 INFO:teuthology.orchestra.run.smithi134.stderr:set pool 10 pgp_num to 16 2024-02-05T08:21:16.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:16.643 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:16.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:16.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:16.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:16.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:16.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:16 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1866909837' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T08:21:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:16 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T08:21:16.886 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:16.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:16.925 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:16.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:16.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T08:21:17.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:17.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:16 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1866909837' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T08:21:17.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:16 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T08:21:17.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:16 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1866909837' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T08:21:17.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:16 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T08:21:17.107 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:17.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:17.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:17.258 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:17.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:17.359 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:17.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:17.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:17.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:17.455 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:21:17.513 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:17.564 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:17.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:17.665 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:17.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:17.700 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:17.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:17.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:17 smithi134 ceph-mon[30086]: pgmap v736: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 19829/316890 objects degraded (6.257%) 2024-02-05T08:21:17.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:17 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-02-05T08:21:17.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:17 smithi134 ceph-mon[30086]: osdmap e339: 8 total, 7 up, 8 in 2024-02-05T08:21:17.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:17 smithi134 ceph-mon[39278]: pgmap v736: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 19829/316890 objects degraded (6.257%) 2024-02-05T08:21:17.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:17 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-02-05T08:21:17.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:17 smithi134 ceph-mon[39278]: osdmap e339: 8 total, 7 up, 8 in 2024-02-05T08:21:17.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:17.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:17.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:17.866 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:17 smithi196 ceph-mon[33018]: pgmap v736: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 19829/316890 objects degraded (6.257%) 2024-02-05T08:21:17.866 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:17 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-02-05T08:21:17.866 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:17 smithi196 ceph-mon[33018]: osdmap e339: 8 total, 7 up, 8 in 2024-02-05T08:21:17.947 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:17.979 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:17.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:18.004 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:21:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:21:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:21:18.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:18.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:18.122 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:18.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:18.223 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:18.263 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:18.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:18.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:18.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:18.401 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:18.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:18.499 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:18.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:18.538 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:18.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:18.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:18.633 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:18 smithi134 ceph-mon[30086]: 4.c scrub starts 2024-02-05T08:21:18.633 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:18 smithi134 ceph-mon[30086]: 4.c scrub ok 2024-02-05T08:21:18.633 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:18 smithi134 ceph-mon[30086]: 5.1b scrub starts 2024-02-05T08:21:18.633 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:18 smithi134 ceph-mon[30086]: 5.1b scrub ok 2024-02-05T08:21:18.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:18 smithi134 ceph-mon[39278]: 4.c scrub starts 2024-02-05T08:21:18.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:18 smithi134 ceph-mon[39278]: 4.c scrub ok 2024-02-05T08:21:18.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:18 smithi134 ceph-mon[39278]: 5.1b scrub starts 2024-02-05T08:21:18.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:18 smithi134 ceph-mon[39278]: 5.1b scrub ok 2024-02-05T08:21:18.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:18.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:18.674 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:18.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:18.775 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:18.809 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:18.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:18.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:18.912 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:18 smithi196 ceph-mon[33018]: 4.c scrub starts 2024-02-05T08:21:18.912 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:18 smithi196 ceph-mon[33018]: 4.c scrub ok 2024-02-05T08:21:18.912 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:18 smithi196 ceph-mon[33018]: 5.1b scrub starts 2024-02-05T08:21:18.913 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:18 smithi196 ceph-mon[33018]: 5.1b scrub ok 2024-02-05T08:21:18.940 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:18.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:19.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:19.080 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:19.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:19.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:19.218 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:19.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:19.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:19.320 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:19.373 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:19.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:19.474 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:19.513 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:19.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:19.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:19.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:19.615 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:21:19.658 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:19.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:19.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:19 smithi196 ceph-mon[33018]: pgmap v738: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 894 op/s; 19433/313398 objects degraded (6.201%) 2024-02-05T08:21:19.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:19 smithi196 ceph-mon[33018]: 3.1f scrub starts 2024-02-05T08:21:19.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:19 smithi196 ceph-mon[33018]: 3.1f scrub ok 2024-02-05T08:21:19.746 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:19 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1295287682' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:21:19.759 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:19.770 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:21:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:21:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:21:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:21:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:108961 8.9 8.107267e9 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:21:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:29.041581+0000", 2024-02-05T08:21:19.770 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 530.664184471, 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62651763699999996, 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108961 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041581+0000", 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.771 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041581+0000", 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041582+0000", 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5570000000000001e-06 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041733+0000", 2024-02-05T08:21:19.772 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015074700000000001 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041745+0000", 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1508999999999999e-05 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041755+0000", 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0533000000000001e-05 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.773 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041803+0000", 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8022999999999997e-05 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.041811+0000", 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0460000000000006e-06 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668084+0000", 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.62627265600000004 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.774 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668098+0000", 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4566000000000001e-05 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:108978 8.9 8.d5065aa9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:29.058870+0000", 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 530.646895245, 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.60925642300000005, 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.775 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108978 2024-02-05T08:21:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058870+0000", 2024-02-05T08:21:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058870+0000", 2024-02-05T08:21:19.776 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058872+0000", 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.655e-06 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058938+0000", 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6568000000000004e-05 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.777 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058949+0000", 2024-02-05T08:21:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.058e-05 2024-02-05T08:21:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058957+0000", 2024-02-05T08:21:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8099999999999998e-06 2024-02-05T08:21:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.058997+0000", 2024-02-05T08:21:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0797999999999998e-05 2024-02-05T08:21:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.778 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059004+0000", 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7510000000000002e-06 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668120+0000", 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.60911546800000005 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668126+0000", 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.793e-06 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.779 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.780 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:108980 8.5 8.acde60b5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:29.059192+0000", 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 530.64657289599995, 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.61158693799999997, 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 108980 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.781 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059192+0000", 2024-02-05T08:21:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059192+0000", 2024-02-05T08:21:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059193+0000", 2024-02-05T08:21:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4900000000000005e-07 2024-02-05T08:21:19.782 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059259+0000", 2024-02-05T08:21:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5699999999999998e-05 2024-02-05T08:21:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059272+0000", 2024-02-05T08:21:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3658e-05 2024-02-05T08:21:19.783 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059281+0000", 2024-02-05T08:21:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9469999999999992e-06 2024-02-05T08:21:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.784 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059332+0000", 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0247999999999997e-05 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.059338+0000", 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.4659999999999996e-06 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.670768+0000", 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.61143003200000001 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.785 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.670779+0000", 2024-02-05T08:21:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1138000000000001e-05 2024-02-05T08:21:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:109000 8.9 8.6538e8d9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:21:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:12:29.089503+0000", 2024-02-05T08:21:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 530.61626215000001, 2024-02-05T08:21:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.57864895999999999, 2024-02-05T08:21:19.786 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:21:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 109000 2024-02-05T08:21:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089503+0000", 2024-02-05T08:21:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.787 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089503+0000", 2024-02-05T08:21:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089505+0000", 2024-02-05T08:21:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.559e-06 2024-02-05T08:21:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089552+0000", 2024-02-05T08:21:19.788 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7407000000000002e-05 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089568+0000", 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5722999999999998e-05 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089576+0000", 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1759999999999993e-06 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.789 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089638+0000", 2024-02-05T08:21:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2442999999999998e-05 2024-02-05T08:21:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.089646+0000", 2024-02-05T08:21:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2629999999999997e-06 2024-02-05T08:21:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668145+0000", 2024-02-05T08:21:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.57849936599999996 2024-02-05T08:21:19.790 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:12:29.668152+0000", 2024-02-05T08:21:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0230000000000004e-06 2024-02-05T08:21:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170181 8.9 8:97b09017:::benchmark_data_smithi134_115836_object53521:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.702283+0000", 2024-02-05T08:21:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 455.00348214799999, 2024-02-05T08:21:19.791 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.090149636, 2024-02-05T08:21:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:21:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170181 2024-02-05T08:21:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.702283+0000", 2024-02-05T08:21:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.792 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.702283+0000", 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.702285+0000", 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5990000000000001e-06 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.702290+0000", 2024-02-05T08:21:19.793 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7989999999999999e-06 2024-02-05T08:21:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.702308+0000", 2024-02-05T08:21:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7918e-05 2024-02-05T08:21:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.702318+0000", 2024-02-05T08:21:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0064999999999999e-05 2024-02-05T08:21:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.794 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.707786+0000", 2024-02-05T08:21:19.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054679469999999999 2024-02-05T08:21:19.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-02-05T08:21:19.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.707795+0000", 2024-02-05T08:21:19.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6729999999999998e-06 2024-02-05T08:21:19.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.795 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.795 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.795 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.716814+0000", 2024-02-05T08:21:19.795 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0090188639999999997 2024-02-05T08:21:19.795 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.716824+0000", 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9709999999999999e-06 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.757276+0000", 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.040452063 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for missing object", 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.757390+0000", 2024-02-05T08:21:19.796 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114508 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792380+0000", 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.034989287000000001 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792433+0000", 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2942000000000002e-05 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.797 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170200 8.9 8:9eea0c4e:::benchmark_data_smithi134_115836_object53540:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.725435+0000", 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.980329778, 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0773159670000001, 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170200 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.798 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725435+0000", 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725435+0000", 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725437+0000", 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.463e-06 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.799 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725443+0000", 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5990000000000001e-06 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725454+0000", 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1479e-05 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725462+0000", 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9170000000000006e-06 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.800 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725517+0000", 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.533e-05 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.725526+0000", 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.738e-06 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.757616+0000", 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.032089603 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.801 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.757697+0000", 2024-02-05T08:21:19.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1055000000000002e-05 2024-02-05T08:21:19.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.802 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.758135+0000", 2024-02-05T08:21:19.802 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00043868999999999999 2024-02-05T08:21:19.802 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.802 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.802 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.779652+0000", 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.021517172000000001 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.786094+0000", 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0064418239999999996 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.802651+0000", 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016556479999999998 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.803 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.802688+0000", 2024-02-05T08:21:19.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7465000000000001e-05 2024-02-05T08:21:19.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.804 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.804 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.804 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.802751+0000", 2024-02-05T08:21:19.804 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3151999999999994e-05 2024-02-05T08:21:19.804 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.804 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.804 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.804 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.805 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.805 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170221 8.9 8:92d7f30d:::benchmark_data_smithi134_115836_object53561:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.805 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.739479+0000", 2024-02-05T08:21:19.805 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.96628663000001, 2024-02-05T08:21:19.805 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0641018680000001, 2024-02-05T08:21:19.805 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.805 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.805 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.805 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.805 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.805 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170221 2024-02-05T08:21:19.805 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.805 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739479+0000", 2024-02-05T08:21:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739479+0000", 2024-02-05T08:21:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.806 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739480+0000", 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.584e-06 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739486+0000", 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6099999999999997e-06 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739497+0000", 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1277000000000001e-05 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.807 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739507+0000", 2024-02-05T08:21:19.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7459999999999995e-06 2024-02-05T08:21:19.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739593+0000", 2024-02-05T08:21:19.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6298999999999995e-05 2024-02-05T08:21:19.808 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.808 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.808 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.808 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739603+0000", 2024-02-05T08:21:19.808 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.011e-05 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.773653+0000", 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0340492779999999 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.773740+0000", 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7072999999999998e-05 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.809 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.773820+0000", 2024-02-05T08:21:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0129000000000005e-05 2024-02-05T08:21:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.779658+0000", 2024-02-05T08:21:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058380710000000002 2024-02-05T08:21:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.788894+0000", 2024-02-05T08:21:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0092360129999999995 2024-02-05T08:21:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803498+0000", 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014604114 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803528+0000", 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0465999999999999e-05 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803580+0000", 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2098000000000002e-05 2024-02-05T08:21:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170223 8.9 8:95eaee90:::benchmark_data_smithi134_115836_object53563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.739633+0000", 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.96613195399999, 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.064076367, 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170223 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739633+0000", 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739633+0000", 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739634+0000", 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0999999999999998e-07 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739640+0000", 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3749999999999999e-06 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739650+0000", 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6670000000000008e-06 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739657+0000", 2024-02-05T08:21:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0990000000000001e-06 2024-02-05T08:21:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739725+0000", 2024-02-05T08:21:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7936999999999997e-05 2024-02-05T08:21:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.739737+0000", 2024-02-05T08:21:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1558e-05 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.775454+0000", 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.035717086 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.775553+0000", 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9755999999999998e-05 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.775642+0000", 2024-02-05T08:21:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8261000000000001e-05 2024-02-05T08:21:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.782170+0000", 2024-02-05T08:21:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065284360000000003 2024-02-05T08:21:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.788944+0000", 2024-02-05T08:21:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0067739209999999996 2024-02-05T08:21:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803613+0000", 2024-02-05T08:21:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014668936 2024-02-05T08:21:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803651+0000", 2024-02-05T08:21:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7505000000000001e-05 2024-02-05T08:21:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803710+0000", 2024-02-05T08:21:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9120000000000003e-05 2024-02-05T08:21:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170234 8.9 8:9450d93c:::benchmark_data_smithi134_115836_object53574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.755021+0000", 2024-02-05T08:21:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.95074378499999, 2024-02-05T08:21:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.048796149, 2024-02-05T08:21:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170234 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755021+0000", 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755021+0000", 2024-02-05T08:21:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755024+0000", 2024-02-05T08:21:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1169999999999998e-06 2024-02-05T08:21:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755029+0000", 2024-02-05T08:21:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8119999999999996e-06 2024-02-05T08:21:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755042+0000", 2024-02-05T08:21:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2541000000000001e-05 2024-02-05T08:21:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755051+0000", 2024-02-05T08:21:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2250000000000006e-06 2024-02-05T08:21:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755134+0000", 2024-02-05T08:21:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3196999999999996e-05 2024-02-05T08:21:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.755145+0000", 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0212e-05 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.776979+0000", 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.021834707 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.777109+0000", 2024-02-05T08:21:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000129868 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.777181+0000", 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1717000000000005e-05 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.782174+0000", 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0049935209999999999 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.791943+0000", 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0097690020000000006 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803736+0000", 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.011792810000000001 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803766+0000", 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.006e-05 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.803818+0000", 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1360000000000003e-05 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170243 8.9 8:90429ea4:::benchmark_data_smithi134_115836_object53583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.761267+0000", 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.94449786199999, 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0487509880000001, 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170243 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761267+0000", 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761267+0000", 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761269+0000", 2024-02-05T08:21:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4559999999999999e-06 2024-02-05T08:21:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761274+0000", 2024-02-05T08:21:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7090000000000001e-06 2024-02-05T08:21:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761286+0000", 2024-02-05T08:21:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2884999999999999e-05 2024-02-05T08:21:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761294+0000", 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7209999999999994e-06 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761361+0000", 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6507000000000005e-05 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.761371+0000", 2024-02-05T08:21:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0225e-05 2024-02-05T08:21:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.778407+0000", 2024-02-05T08:21:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0170360919999999 2024-02-05T08:21:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.778486+0000", 2024-02-05T08:21:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8974000000000007e-05 2024-02-05T08:21:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.778554+0000", 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8254000000000001e-05 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.784715+0000", 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.006161265 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792019+0000", 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0073035729999999998 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.809736+0000", 2024-02-05T08:21:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017717096000000002 2024-02-05T08:21:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.809843+0000", 2024-02-05T08:21:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010713399999999999 2024-02-05T08:21:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.810018+0000", 2024-02-05T08:21:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017509700000000001 2024-02-05T08:21:19.833 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.833 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.833 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.833 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.833 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170265 8.9 8:93c691bd:::benchmark_data_smithi134_115836_object53605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.833 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.786248+0000", 2024-02-05T08:21:19.833 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.91951735800001, 2024-02-05T08:21:19.833 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0239562790000001, 2024-02-05T08:21:19.833 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.833 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170265 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786248+0000", 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.834 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786248+0000", 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786249+0000", 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.051e-06 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786254+0000", 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4500000000000003e-06 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.835 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786270+0000", 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5780000000000001e-05 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786278+0000", 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2539999999999992e-06 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786346+0000", 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7643000000000003e-05 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.836 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.786355+0000", 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0690000000000008e-06 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.779698+0000", 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99334322799999997 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.779812+0000", 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113348 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.837 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.779943+0000", 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013156600000000001 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.786038+0000", 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0060945799999999996 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792480+0000", 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0064420859999999996 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.838 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.810091+0000", 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.017610614 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.810128+0000", 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7329999999999997e-05 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.810204+0000", 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6279999999999995e-05 2024-02-05T08:21:19.839 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.840 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.840 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.840 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.840 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.840 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170309 8.9 8:96bff997:::benchmark_data_smithi134_115836_object53649:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.840 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.842207+0000", 2024-02-05T08:21:19.840 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.86355815399997, 2024-02-05T08:21:19.840 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99189979299999997, 2024-02-05T08:21:19.840 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.840 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.840 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.840 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.840 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.841 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170309 2024-02-05T08:21:19.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.841 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.841 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842207+0000", 2024-02-05T08:21:19.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.841 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.841 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.841 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842207+0000", 2024-02-05T08:21:19.841 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842209+0000", 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.764e-06 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842215+0000", 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0630000000000002e-06 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842231+0000", 2024-02-05T08:21:19.842 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5920999999999999e-05 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842240+0000", 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4919999999999999e-06 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842307+0000", 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6278000000000005e-05 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.843 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.842318+0000", 2024-02-05T08:21:19.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1601e-05 2024-02-05T08:21:19.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.844 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.781005+0000", 2024-02-05T08:21:19.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.93868725799999997 2024-02-05T08:21:19.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.844 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.844 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.781107+0000", 2024-02-05T08:21:19.844 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101585 2024-02-05T08:21:19.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.844 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.781179+0000", 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1582999999999995e-05 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.787700+0000", 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065215459999999996 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792531+0000", 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048308650000000002 2024-02-05T08:21:19.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834010+0000", 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.041478810999999997 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834066+0000", 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6192e-05 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834107+0000", 2024-02-05T08:21:19.846 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0834000000000002e-05 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170330 8.9 8:92c65df7:::benchmark_data_smithi134_115836_object53670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.865357+0000", 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.84040861599999, 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96889826400000001, 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.847 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.848 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170330 2024-02-05T08:21:19.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.848 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.848 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.848 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.848 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865357+0000", 2024-02-05T08:21:19.848 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.848 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.848 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.848 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865357+0000", 2024-02-05T08:21:19.848 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865358+0000", 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4839999999999999e-06 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865364+0000", 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1020000000000002e-06 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865383+0000", 2024-02-05T08:21:19.849 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8620000000000001e-05 2024-02-05T08:21:19.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865392+0000", 2024-02-05T08:21:19.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8689999999999993e-06 2024-02-05T08:21:19.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.850 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865477+0000", 2024-02-05T08:21:19.850 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5654000000000004e-05 2024-02-05T08:21:19.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.850 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.850 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.865490+0000", 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2308e-05 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.782216+0000", 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.91672598400000005 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.782347+0000", 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013109100000000001 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.851 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.852 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.782515+0000", 2024-02-05T08:21:19.852 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000168593 2024-02-05T08:21:19.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.852 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.852 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.787704+0000", 2024-02-05T08:21:19.852 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0051884519999999996 2024-02-05T08:21:19.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.852 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.852 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.852 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.794555+0000", 2024-02-05T08:21:19.852 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0068516999999999996 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834163+0000", 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039607246999999998 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834220+0000", 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6884e-05 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.853 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834255+0000", 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5275999999999999e-05 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170336 8.9 8:9638bc28:::benchmark_data_smithi134_115836_object53676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.869806+0000", 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.835959436, 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.96456107099999999, 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.854 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170336 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869806+0000", 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869806+0000", 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.855 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.856 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.856 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869807+0000", 2024-02-05T08:21:19.856 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3e-06 2024-02-05T08:21:19.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.856 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.856 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.856 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869811+0000", 2024-02-05T08:21:19.856 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.216e-06 2024-02-05T08:21:19.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.856 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.856 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.856 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869830+0000", 2024-02-05T08:21:19.856 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8218000000000001e-05 2024-02-05T08:21:19.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.857 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.857 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.857 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869843+0000", 2024-02-05T08:21:19.857 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2977e-05 2024-02-05T08:21:19.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.857 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.857 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.857 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869900+0000", 2024-02-05T08:21:19.857 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7697e-05 2024-02-05T08:21:19.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.857 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.857 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.869909+0000", 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7849999999999997e-06 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.783591+0000", 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.91368222399999999 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.783670+0000", 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8839999999999997e-05 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.858 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.783743+0000", 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2649000000000002e-05 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.787707+0000", 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039647909999999996 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.794616+0000", 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0069084580000000001 2024-02-05T08:21:19.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834295+0000", 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039678791999999997 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834332+0000", 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7585e-05 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834367+0000", 2024-02-05T08:21:19.860 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4539000000000003e-05 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170349 8.9 8:9486e005:::benchmark_data_smithi134_115836_object53689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.884848+0000", 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.82091766799999, 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94965263499999997, 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.861 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170349 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884848+0000", 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884848+0000", 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.862 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884849+0000", 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.079e-06 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884853+0000", 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0400000000000003e-06 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884863+0000", 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0521e-05 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.863 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884870+0000", 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0509999999999999e-06 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884913+0000", 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2555000000000002e-05 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.884919+0000", 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1510000000000001e-06 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.784743+0000", 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.89982445499999997 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.784894+0000", 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015011199999999999 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.785027+0000", 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013361800000000001 2024-02-05T08:21:19.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.788953+0000", 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0039257370000000003 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.796649+0000", 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.007695744 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834414+0000", 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.037765493999999997 2024-02-05T08:21:19.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834466+0000", 2024-02-05T08:21:19.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1848000000000002e-05 2024-02-05T08:21:19.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.834500+0000", 2024-02-05T08:21:19.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4230000000000003e-05 2024-02-05T08:21:19.867 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.867 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.867 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.868 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170351 8.9 8:968af758:::benchmark_data_smithi134_115836_object53691:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.868 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.889654+0000", 2024-02-05T08:21:19.868 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.81611148799999, 2024-02-05T08:21:19.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94761108199999999, 2024-02-05T08:21:19.868 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.868 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.868 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.868 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.868 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.868 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170351 2024-02-05T08:21:19.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.868 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889654+0000", 2024-02-05T08:21:19.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889654+0000", 2024-02-05T08:21:19.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889655+0000", 2024-02-05T08:21:19.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.35e-06 2024-02-05T08:21:19.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889661+0000", 2024-02-05T08:21:19.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5949999999999998e-06 2024-02-05T08:21:19.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889673+0000", 2024-02-05T08:21:19.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2588e-05 2024-02-05T08:21:19.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889683+0000", 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8260000000000005e-06 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889744+0000", 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0723000000000001e-05 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.889753+0000", 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3270000000000007e-06 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.786146+0000", 2024-02-05T08:21:19.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.89639291600000004 2024-02-05T08:21:19.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.786261+0000", 2024-02-05T08:21:19.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000114831 2024-02-05T08:21:19.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.786400+0000", 2024-02-05T08:21:19.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013884300000000001 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.791892+0000", 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054924190000000001 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.796690+0000", 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047979820000000001 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837182+0000", 2024-02-05T08:21:19.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040491744000000003 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837229+0000", 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7046999999999999e-05 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837265+0000", 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5890999999999999e-05 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.874 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170364 8.9 8:96172a8c:::benchmark_data_smithi134_115836_object53704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.875 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.908364+0000", 2024-02-05T08:21:19.875 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.797401218, 2024-02-05T08:21:19.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.92901532799999997, 2024-02-05T08:21:19.875 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.875 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.875 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.875 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.875 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.875 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170364 2024-02-05T08:21:19.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.875 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908364+0000", 2024-02-05T08:21:19.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908364+0000", 2024-02-05T08:21:19.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908365+0000", 2024-02-05T08:21:19.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.034e-06 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908369+0000", 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8430000000000003e-06 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908379+0000", 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0528e-05 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908385+0000", 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0480000000000004e-06 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908435+0000", 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9166999999999999e-05 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.908441+0000", 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5409999999999997e-06 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.787740+0000", 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87929909299999998 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.787810+0000", 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9851999999999995e-05 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.787877+0000", 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6863e-05 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.791898+0000", 2024-02-05T08:21:19.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0040212030000000001 2024-02-05T08:21:19.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.796750+0000", 2024-02-05T08:21:19.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048520569999999999 2024-02-05T08:21:19.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837303+0000", 2024-02-05T08:21:19.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040552610000000003 2024-02-05T08:21:19.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.881 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837350+0000", 2024-02-05T08:21:19.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7002000000000002e-05 2024-02-05T08:21:19.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.881 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837379+0000", 2024-02-05T08:21:19.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9487000000000001e-05 2024-02-05T08:21:19.881 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.881 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.882 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.882 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170377 8.9 8:9cb6fb33:::benchmark_data_smithi134_115836_object53717:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.882 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.927537+0000", 2024-02-05T08:21:19.882 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.7782282, 2024-02-05T08:21:19.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.90994256200000001, 2024-02-05T08:21:19.882 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.882 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.882 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.882 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.882 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.882 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170377 2024-02-05T08:21:19.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.883 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927537+0000", 2024-02-05T08:21:19.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.883 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.883 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927537+0000", 2024-02-05T08:21:19.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927538+0000", 2024-02-05T08:21:19.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0860000000000001e-06 2024-02-05T08:21:19.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927542+0000", 2024-02-05T08:21:19.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0319999999999997e-06 2024-02-05T08:21:19.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927554+0000", 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1647999999999999e-05 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927560+0000", 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.5810000000000002e-06 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927622+0000", 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1507000000000006e-05 2024-02-05T08:21:19.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.927632+0000", 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.045e-05 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.788974+0000", 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.86134118900000001 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.789086+0000", 2024-02-05T08:21:19.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011277699999999999 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.789219+0000", 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013251800000000001 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.791958+0000", 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027390470000000001 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.796792+0000", 2024-02-05T08:21:19.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048341469999999996 2024-02-05T08:21:19.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837412+0000", 2024-02-05T08:21:19.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040620235999999997 2024-02-05T08:21:19.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837451+0000", 2024-02-05T08:21:19.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8872000000000003e-05 2024-02-05T08:21:19.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837480+0000", 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8472000000000001e-05 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170391 8.9 8:9bfec3b7:::benchmark_data_smithi134_115836_object53731:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.958818+0000", 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.74694748899998, 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.87876214900000005, 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.890 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170391 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958818+0000", 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958818+0000", 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958819+0000", 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2240000000000001e-06 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958823+0000", 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1659999999999996e-06 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958835+0000", 2024-02-05T08:21:19.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1590000000000001e-05 2024-02-05T08:21:19.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958841+0000", 2024-02-05T08:21:19.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3879999999999997e-06 2024-02-05T08:21:19.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958894+0000", 2024-02-05T08:21:19.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3151e-05 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.958901+0000", 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8719999999999996e-06 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.790448+0000", 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.831546813 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.790573+0000", 2024-02-05T08:21:19.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000124673 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.790704+0000", 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013115100000000001 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792495+0000", 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001791614 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.796825+0000", 2024-02-05T08:21:19.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0043295790000000001 2024-02-05T08:21:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837511+0000", 2024-02-05T08:21:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040686310000000003 2024-02-05T08:21:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837550+0000", 2024-02-05T08:21:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8303999999999998e-05 2024-02-05T08:21:19.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837580+0000", 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0314000000000001e-05 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170400 8.9 8:9216392e:::benchmark_data_smithi134_115836_object53740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:44.988572+0000", 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 454.71719284699998, 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.84910828500000002, 2024-02-05T08:21:19.897 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170400 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988572+0000", 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988572+0000", 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988574+0000", 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.761e-06 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988580+0000", 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6659999999999996e-06 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988595+0000", 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5441e-05 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988604+0000", 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7250000000000003e-06 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988667+0000", 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3484000000000004e-05 2024-02-05T08:21:19.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:44.988679+0000", 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1958e-05 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792565+0000", 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.80388560200000003 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792682+0000", 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000117037 2024-02-05T08:21:19.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,5", 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792813+0000", 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013053500000000001 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.794221+0000", 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014081300000000001 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.797909+0000", 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036880709999999998 2024-02-05T08:21:19.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837613+0000", 2024-02-05T08:21:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.039704383000000003 2024-02-05T08:21:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:21:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837652+0000", 2024-02-05T08:21:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8931999999999999e-05 2024-02-05T08:21:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:21:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:21:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:21:19.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.837681+0000", 2024-02-05T08:21:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8560000000000001e-05 2024-02-05T08:21:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:21:19.904 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:21:19.904 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:21:19.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:19.904 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:19.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:20.005 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:20.020 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:19 smithi134 ceph-mon[30086]: pgmap v738: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 894 op/s; 19433/313398 objects degraded (6.201%) 2024-02-05T08:21:20.020 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:19 smithi134 ceph-mon[30086]: 3.1f scrub starts 2024-02-05T08:21:20.020 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:19 smithi134 ceph-mon[30086]: 3.1f scrub ok 2024-02-05T08:21:20.020 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:19 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1295287682' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:21:20.020 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:19 smithi134 ceph-mon[39278]: pgmap v738: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 894 op/s; 19433/313398 objects degraded (6.201%) 2024-02-05T08:21:20.020 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:19 smithi134 ceph-mon[39278]: 3.1f scrub starts 2024-02-05T08:21:20.021 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:19 smithi134 ceph-mon[39278]: 3.1f scrub ok 2024-02-05T08:21:20.021 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:19 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1295287682' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:21:20.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:20.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:20.144 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:20.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:20.178 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:20.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:20.279 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:20.315 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:20.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:20.417 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:20.455 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:20.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:20.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:20.590 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:20.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:20.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:20.602 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[30086]: 1.9 scrub starts 2024-02-05T08:21:20.603 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[30086]: 1.9 scrub ok 2024-02-05T08:21:20.603 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[30086]: 5.1e deep-scrub starts 2024-02-05T08:21:20.603 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[30086]: 5.1e deep-scrub ok 2024-02-05T08:21:20.603 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:21:20.603 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1295287682' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:21:20.603 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[30086]: osdmap e340: 8 total, 7 up, 8 in 2024-02-05T08:21:20.603 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 19433/313398 objects degraded (6.201%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:20.692 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:20.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:20.727 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:20.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:20.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:20 smithi196 ceph-mon[33018]: 1.9 scrub starts 2024-02-05T08:21:20.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:20 smithi196 ceph-mon[33018]: 1.9 scrub ok 2024-02-05T08:21:20.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:20 smithi196 ceph-mon[33018]: 5.1e deep-scrub starts 2024-02-05T08:21:20.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:20 smithi196 ceph-mon[33018]: 5.1e deep-scrub ok 2024-02-05T08:21:20.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:20 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:21:20.810 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:20 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1295287682' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:21:20.811 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:20 smithi196 ceph-mon[33018]: osdmap e340: 8 total, 7 up, 8 in 2024-02-05T08:21:20.811 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:20 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 19433/313398 objects degraded (6.201%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:20.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:20.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T08:21:20.869 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:20.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:20.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:20.909 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[39278]: 1.9 scrub starts 2024-02-05T08:21:20.909 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[39278]: 1.9 scrub ok 2024-02-05T08:21:20.909 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[39278]: 5.1e deep-scrub starts 2024-02-05T08:21:20.909 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[39278]: 5.1e deep-scrub ok 2024-02-05T08:21:20.910 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:21:20.910 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1295287682' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:21:20.910 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[39278]: osdmap e340: 8 total, 7 up, 8 in 2024-02-05T08:21:20.910 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:20 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 19433/313398 objects degraded (6.201%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:20.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:21.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:21.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:21.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:21.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:21.171 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:21.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:21.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:21.313 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:21:21.328 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:21.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:21.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:21.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:21.467 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:21.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:21.566 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:21:21.569 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:21.605 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:21.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:21.608 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:21:21.645 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:21:21.683 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:21:21.707 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:21.722 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:21:21.741 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:21.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:21.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:21.759 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:21:21.796 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:21:21.796 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:21:21.796 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:21:21.797 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:21:21.797 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:21:21.797 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:21:21.797 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:21:21.797 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:21:21.797 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:21:21.797 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:21 smithi196 ceph-mon[33018]: pgmap v740: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 18697/298461 objects degraded (6.264%) 2024-02-05T08:21:21.797 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:21 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2260958284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:21.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:21.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:21.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:21.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:21.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:22.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:22.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:22.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:21 smithi134 ceph-mon[30086]: pgmap v740: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 18697/298461 objects degraded (6.264%) 2024-02-05T08:21:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:21 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2260958284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:22.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:21 smithi134 ceph-mon[39278]: pgmap v740: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s; 18697/298461 objects degraded (6.264%) 2024-02-05T08:21:22.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:21 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2260958284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:22.119 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:22.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:22.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:22.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:22.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:22.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:22.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:22.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:21:22.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:22.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:22.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:22.529 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:22.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:22.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:22.593 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:21:22.594 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:21:22.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:22.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:22.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:22.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:22.775 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:22.804 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:22.838 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:22.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:22.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:22.909 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-02-05T08:21:22.909 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:21:22.909 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-05T08:21:22.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T08:21:22.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:22.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:22.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:22.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:23.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:23.139 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:23.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:23.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:23.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:23.299 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:23.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:23.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:23.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:23.428 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:23.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:23.529 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:23.572 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:23.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:23.673 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:23.705 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:23.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:23.752 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:21:23.753 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:21:23.807 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:23.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:23 smithi196 ceph-mon[33018]: pgmap v741: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 860 op/s; 18697/298461 objects degraded (6.264%) 2024-02-05T08:21:23.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:21:23.840 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:23.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:23.942 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:23.984 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:23.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:23.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:24.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:23 smithi134 ceph-mon[30086]: pgmap v741: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 860 op/s; 18697/298461 objects degraded (6.264%) 2024-02-05T08:21:24.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:21:24.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:23 smithi134 ceph-mon[39278]: pgmap v741: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 860 op/s; 18697/298461 objects degraded (6.264%) 2024-02-05T08:21:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:21:24.086 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:24.117 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:21:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:21:24.117 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:21:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=sqlstore.transactions t=2024-02-05T08:21:23.832493634Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-05T08:21:24.117 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:21:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=sqlstore.transactions t=2024-02-05T08:21:23.844116789Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-02-05T08:21:24.117 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:21:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=sqlstore.transactions t=2024-02-05T08:21:23.855432593Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-02-05T08:21:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:24.118 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:24.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:24.190 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:24.219 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:24.264 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:24.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:24.366 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:24.407 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:24.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:24.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:24.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T08:21:24.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:24.580 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:24.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:24.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:24.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:21:24.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:24.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:24.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:24.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:24.883 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:24.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:24.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:24.941 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:21:24.985 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:25.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:25.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:25.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:25.127 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:25.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:25.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:25.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:25.307 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:25.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:25.408 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:25.448 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:25.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:25.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:25.550 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:25.583 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:25.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:25.663 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:25 smithi134 ceph-mon[30086]: pgmap v742: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 17519/282294 objects degraded (6.206%) 2024-02-05T08:21:25.663 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:25 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:25.663 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:25 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 17519/282294 objects degraded (6.206%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:25.663 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:25 smithi134 ceph-mon[39278]: pgmap v742: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 17519/282294 objects degraded (6.206%) 2024-02-05T08:21:25.664 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:25 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:25.664 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:25 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 17519/282294 objects degraded (6.206%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:25.684 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:25.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:25.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:25.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:25 smithi196 ceph-mon[33018]: pgmap v742: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 17519/282294 objects degraded (6.206%) 2024-02-05T08:21:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:25 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:25 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 17519/282294 objects degraded (6.206%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:25.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:25.856 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:25.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:25.957 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:26.007 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:26.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:26.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:26.109 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:26.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:26.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:26.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:26.293 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:26.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:26.395 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:26.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:26.447 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:26.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:26.548 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:26.593 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:26.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:26.651 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:21:26.694 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:26.723 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:26.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:26.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:26 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2144764171' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:26.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:26.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:26.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:26.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:26.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:27.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:27.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:27.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:26 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2144764171' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:27.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:26 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2144764171' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:27.065 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:21:27.066 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:21:27.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:27.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:27.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:27.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:27.273 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:27.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:27.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:27.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:27.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:27.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:27.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:27.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:27.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:27.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:27.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:27.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:27.685 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:27.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:27.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:27.787 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:27.806 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:27 smithi196 ceph-mon[33018]: pgmap v743: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.33k op/s; 17519/280065 objects degraded (6.255%) 2024-02-05T08:21:27.806 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:27 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:21:27.806 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:27 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2144764171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:21:27.806 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:27 smithi196 ceph-mon[33018]: osdmap e341: 8 total, 7 up, 8 in 2024-02-05T08:21:27.857 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T08:21:27.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:27.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:27.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:27.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:27.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:27.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:28.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:27 smithi134 ceph-mon[30086]: pgmap v743: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.33k op/s; 17519/280065 objects degraded (6.255%) 2024-02-05T08:21:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:27 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:21:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:27 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2144764171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:21:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:27 smithi134 ceph-mon[30086]: osdmap e341: 8 total, 7 up, 8 in 2024-02-05T08:21:28.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:21:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:21:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:21:28.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:27 smithi134 ceph-mon[39278]: pgmap v743: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.33k op/s; 17519/280065 objects degraded (6.255%) 2024-02-05T08:21:28.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:27 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:21:28.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:27 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2144764171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:21:28.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:27 smithi134 ceph-mon[39278]: osdmap e341: 8 total, 7 up, 8 in 2024-02-05T08:21:28.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:28.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:28.143 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:28.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:28.244 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:28.290 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:28.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:28.302 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:21:28.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:28.430 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:28.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:28.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:28.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:28.572 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:28.572 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:28.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:28.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:28.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:28.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:28.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:28.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:28.798 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:28.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:28.843 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:28.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:28.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:28.978 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:28.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:28.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:29.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:29.115 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:29.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:29.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:29.217 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:29.255 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:29.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:29.357 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:29.393 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:29.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:29.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:29.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:29.529 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:29.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:29.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:29.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:29.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:29.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:29.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:29.761 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:21:29.761 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:21:29.761 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 341, 2024-02-05T08:21:29.761 INFO:teuthology.orchestra.run.smithi134.stdout: "fsid": "43e13aa4-c3fc-11ee-95b6-87774f69a715", 2024-02-05T08:21:29.761 INFO:teuthology.orchestra.run.smithi134.stdout: "created": "2024-02-05T07:59:09.321262+0000", 2024-02-05T08:21:29.762 INFO:teuthology.orchestra.run.smithi134.stdout: "modified": "2024-02-05T08:21:26.627737+0000", 2024-02-05T08:21:29.762 INFO:teuthology.orchestra.run.smithi134.stdout: "last_up_change": "2024-02-05T08:19:43.862128+0000", 2024-02-05T08:21:29.762 INFO:teuthology.orchestra.run.smithi134.stdout: "last_in_change": "2024-02-05T08:18:33.608335+0000", 2024-02-05T08:21:29.762 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T08:21:29.762 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_num": 5806080, 2024-02-05T08:21:29.762 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_set": [ 2024-02-05T08:21:29.762 INFO:teuthology.orchestra.run.smithi134.stdout: "nodeep-scrub", 2024-02-05T08:21:29.762 INFO:teuthology.orchestra.run.smithi134.stdout: "noscrub", 2024-02-05T08:21:29.762 INFO:teuthology.orchestra.run.smithi134.stdout: "pglog_hardlimit", 2024-02-05T08:21:29.762 INFO:teuthology.orchestra.run.smithi134.stdout: "purged_snapdirs", 2024-02-05T08:21:29.762 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_deletes", 2024-02-05T08:21:29.762 INFO:teuthology.orchestra.run.smithi134.stdout: "sortbitwise" 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_version": 18, 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_snapshot": "", 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_max": 10, 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: "max_osd": 8, 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: "require_min_compat_client": "luminous", 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: "min_compat_client": "luminous", 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: "require_osd_release": "squid", 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: "allow_crimson": false, 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: "pools": [ 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.763 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 1, 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": ".mgr", 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:02:21.034012+0000", 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:21:29.764 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:21:29.765 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 11, 2024-02-05T08:21:29.765 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 11, 2024-02-05T08:21:29.765 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 11, 2024-02-05T08:21:29.765 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 11, 2024-02-05T08:21:29.765 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 11, 2024-02-05T08:21:29.765 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:21:29.765 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:21:29.765 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:21:29.765 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:21:29.765 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:21:29.765 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:21:29.765 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:21:29.765 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "303", 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "301", 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:21:29.766 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.767 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:21:29.768 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:21:29.768 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:21:29.768 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:21:29.768 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:21:29.768 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:21:29.768 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:21:29.768 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:21:29.768 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:21:29.768 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:21:29.768 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:21:29.768 INFO:teuthology.orchestra.run.smithi134.stdout: "options": { 2024-02-05T08:21:29.768 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_max": 32, 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_min": 1 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr": {} 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.1700000762939453, 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.1700000762939453, 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.9099999666213989, 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.9099999666213989, 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:21:29.769 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 2, 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "datapool", 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:39.400746+0000", 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 8193, 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:21:29.770 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:21:29.771 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:21:29.771 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:21:29.771 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:21:29.771 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:21:29.771 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 3, 2024-02-05T08:21:29.771 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 3, 2024-02-05T08:21:29.771 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 3, 2024-02-05T08:21:29.771 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 3, 2024-02-05T08:21:29.771 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 3, 2024-02-05T08:21:29.771 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:21:29.771 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:21:29.771 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:21:29.771 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "70", 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 2, 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 70, 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:21:29.772 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:21:29.773 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:21:29.774 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd": {} 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.6700000762939453, 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.6700000762939453, 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.75, 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 2, 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 2, 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:21:29.775 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 3, 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": ".rgw.root", 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:40.005769+0000", 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:21:29.776 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:21:29.777 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:21:29.778 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.778 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "68", 2024-02-05T08:21:29.778 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:21:29.778 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:21:29.778 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:21:29.778 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:21:29.778 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:21:29.778 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:21:29.778 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:21:29.778 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:21:29.778 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:21:29.778 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:21:29.778 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:21:29.779 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:21:29.780 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.7400000095367432, 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.7400000095367432, 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.5299999713897705, 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.5299999713897705, 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.781 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 4, 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.log", 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:41.851891+0000", 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:21:29.782 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:21:29.783 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:21:29.783 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:21:29.783 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:21:29.783 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:21:29.783 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:21:29.783 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:21:29.783 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:21:29.783 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:21:29.783 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:21:29.783 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:21:29.783 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:21:29.783 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:21:29.783 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:21:29.784 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:21:29.784 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.784 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "70", 2024-02-05T08:21:29.784 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:21:29.784 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:21:29.784 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:21:29.784 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:21:29.784 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:21:29.784 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:21:29.784 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:21:29.784 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:21:29.784 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:21:29.784 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:21:29.785 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:21:29.786 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.2400000095367432, 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.2400000095367432, 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.9700000286102295, 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.9700000286102295, 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:21:29.787 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 5, 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.control", 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:43.793804+0000", 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:21:29.788 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:21:29.789 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:21:29.789 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:21:29.789 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:21:29.789 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:21:29.789 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:21:29.789 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:21:29.789 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:21:29.789 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:21:29.789 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:21:29.789 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "72", 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:21:29.790 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:21:29.791 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:21:29.792 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.7400000095367432, 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.7400000095367432, 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:21:29.793 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.5299999713897705, 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.5299999713897705, 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 6, 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.meta", 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:45.853999+0000", 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:21:29.794 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:21:29.795 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "74", 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:21:29.796 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:21:29.797 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:21:29.798 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: "options": { 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_bias": 4 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.4900000095367432, 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.4900000095367432, 2024-02-05T08:21:29.799 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:21:29.800 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.309999942779541, 2024-02-05T08:21:29.800 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.309999942779541, 2024-02-05T08:21:29.800 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:21:29.800 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:21:29.800 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:21:29.800 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.800 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.800 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.800 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 10, 2024-02-05T08:21:29.800 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "unique_pool_3", 2024-02-05T08:21:29.800 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:19:06.157645+0000", 2024-02-05T08:21:29.800 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:21:29.800 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 16, 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 16, 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 16, 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 16, 2024-02-05T08:21:29.801 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 16, 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "339", 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:21:29.802 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:21:29.803 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:21:29.804 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "rados": {} 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.9900000095367432, 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.9900000095367432, 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 2.630000114440918, 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 2.630000114440918, 2024-02-05T08:21:29.805 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: "osds": [ 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 0, 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "d90a5ea0-9a19-452e-9564-ca7b7a7c45e4", 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:21:29.806 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 8, 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 324, 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6802", 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.807 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6803", 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6804", 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.808 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6805", 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6808", 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.809 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6809", 2024-02-05T08:21:29.810 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:21:29.810 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.810 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.810 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.810 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:21:29.810 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.810 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.810 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:29.811 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6806", 2024-02-05T08:21:29.812 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:21:29.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.812 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.812 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.812 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6807", 2024-02-05T08:21:29.812 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:21:29.812 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.812 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.812 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.812 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6803/2540271081", 2024-02-05T08:21:29.812 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6805/2540271081", 2024-02-05T08:21:29.812 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6809/2540271081", 2024-02-05T08:21:29.812 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6807/2540271081", 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 1, 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4", 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 0, 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:21:29.813 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 13, 2024-02-05T08:21:29.814 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 309, 2024-02-05T08:21:29.814 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 324, 2024-02-05T08:21:29.814 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:21:29.814 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:21:29.814 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.814 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.814 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.814 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6810", 2024-02-05T08:21:29.814 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:21:29.814 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.814 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.815 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.815 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6811", 2024-02-05T08:21:29.815 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:21:29.815 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.815 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.815 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.815 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:21:29.816 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.816 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.816 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.816 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6812", 2024-02-05T08:21:29.816 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:21:29.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.816 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.816 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.816 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6813", 2024-02-05T08:21:29.816 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:21:29.816 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.816 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.816 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6816", 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6817", 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.817 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6814", 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6815", 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2032802960 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.818 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6811/2032802960", 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6813/2032802960", 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6817/2032802960", 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6815/2032802960", 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: "exists" 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 2, 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708", 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:21:29.819 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 18, 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 324, 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6818", 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.820 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.821 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6819", 2024-02-05T08:21:29.821 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:21:29.821 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.821 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.821 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.821 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:21:29.821 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.821 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6820", 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6821", 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.822 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.823 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.823 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6824", 2024-02-05T08:21:29.823 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:21:29.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.823 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.823 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.823 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6825", 2024-02-05T08:21:29.823 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:21:29.823 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.823 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.823 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.823 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:21:29.823 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6822", 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6823", 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6819/153827987", 2024-02-05T08:21:29.824 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6821/153827987", 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6825/153827987", 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6823/153827987", 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 3, 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "6774e330-3202-4671-894e-172cc2f6189d", 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:21:29.825 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 27, 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 324, 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6826", 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.826 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6827", 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6828", 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.827 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6829", 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6832", 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.828 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6833", 2024-02-05T08:21:29.829 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:21:29.829 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.829 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.829 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:21:29.829 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.829 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.829 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.829 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6830", 2024-02-05T08:21:29.829 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:21:29.829 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.829 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.829 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6831", 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6827/1527507056", 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6829/1527507056", 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6833/1527507056", 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6831/1527507056", 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.830 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.831 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.831 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 4, 2024-02-05T08:21:29.831 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693", 2024-02-05T08:21:29.831 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:21:29.831 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:21:29.831 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:21:29.831 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:21:29.831 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:21:29.831 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:21:29.831 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 32, 2024-02-05T08:21:29.831 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 324, 2024-02-05T08:21:29.831 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:21:29.832 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:21:29.832 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:21:29.832 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.832 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.832 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.832 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6800", 2024-02-05T08:21:29.832 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:21:29.832 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.832 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.832 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.832 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6801", 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6802", 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.833 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6803", 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6806", 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.834 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6807", 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6804", 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6805", 2024-02-05T08:21:29.835 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:21:29.836 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.836 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.836 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.836 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6801/3005156808", 2024-02-05T08:21:29.836 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6803/3005156808", 2024-02-05T08:21:29.836 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3005156808", 2024-02-05T08:21:29.836 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3005156808", 2024-02-05T08:21:29.836 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:21:29.836 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:21:29.836 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:21:29.836 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.836 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.836 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 5, 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955", 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 42, 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 324, 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:21:29.837 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6808", 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6809", 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:21:29.838 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6810", 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6811", 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.839 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.840 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.840 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6814", 2024-02-05T08:21:29.840 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:21:29.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.840 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.840 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.840 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6815", 2024-02-05T08:21:29.840 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:21:29.840 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.840 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.840 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.840 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:21:29.840 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6812", 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6813", 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6809/1603224792", 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6811/1603224792", 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6815/1603224792", 2024-02-05T08:21:29.841 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6813/1603224792", 2024-02-05T08:21:29.842 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:21:29.842 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:21:29.842 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:21:29.842 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.842 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.842 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.842 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 6, 2024-02-05T08:21:29.842 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a", 2024-02-05T08:21:29.842 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:21:29.842 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:21:29.842 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:21:29.842 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:21:29.842 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 51, 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 324, 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6816", 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.843 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6817", 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6818", 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.844 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6819", 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6822", 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.845 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6823", 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6820", 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.846 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6821", 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6817/3727413143", 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6819/3727413143", 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6823/3727413143", 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6821/3727413143", 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.847 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 7, 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057", 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 60, 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 324, 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:21:29.848 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.849 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.849 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.849 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6824", 2024-02-05T08:21:29.849 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:21:29.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.849 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.849 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.849 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6825", 2024-02-05T08:21:29.849 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:21:29.849 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.849 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.849 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.849 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6826", 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6827", 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.850 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6830", 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6831", 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:21:29.851 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:21:29.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.852 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:21:29.852 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6828", 2024-02-05T08:21:29.852 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:21:29.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.852 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.852 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:21:29.852 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6829", 2024-02-05T08:21:29.852 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:21:29.852 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.852 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.852 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.852 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6825/4206571010", 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6827/4206571010", 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6831/4206571010", 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6829/4206571010", 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_xinfo": [ 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 0, 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:21:29.853 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:21:29.854 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:21:29.854 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:21:29.854 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:21:29.854 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:20:46.624000+0000", 2024-02-05T08:21:29.854 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:21:29.854 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.854 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.854 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 1, 2024-02-05T08:21:29.854 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:21:29.854 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:21:29.854 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:21:29.854 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:21:29.854 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:21:29.855 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:19:38.848711+0000", 2024-02-05T08:21:29.855 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 323 2024-02-05T08:21:29.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.855 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.855 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 2, 2024-02-05T08:21:29.855 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:21:29.855 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:21:29.855 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:21:29.855 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:21:29.855 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:21:29.855 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:20:36.781950+0000", 2024-02-05T08:21:29.855 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:21:29.855 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.856 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.856 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 3, 2024-02-05T08:21:29.856 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:21:29.856 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:21:29.856 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:21:29.856 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:21:29.856 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:21:29.856 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:20:36.057611+0000", 2024-02-05T08:21:29.856 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:21:29.856 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.856 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.856 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 4, 2024-02-05T08:21:29.856 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:21:29.857 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:21:29.857 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:21:29.857 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:21:29.857 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:21:29.857 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:20:46.010664+0000", 2024-02-05T08:21:29.857 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:21:29.857 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.857 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.857 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 5, 2024-02-05T08:21:29.857 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:21:29.857 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:21:29.857 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:21:29.857 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:20:41.987727+0000", 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 6, 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:20:42.674824+0000", 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:21:29.858 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.859 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.859 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 7, 2024-02-05T08:21:29.859 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:21:29.859 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:21:29.859 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:21:29.859 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:21:29.859 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:21:29.859 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:20:21.973509+0000", 2024-02-05T08:21:29.859 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:21:29.859 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.859 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:21:29.859 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap": [ 2024-02-05T08:21:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "3.1e", 2024-02-05T08:21:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: "osds": [ 2024-02-05T08:21:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: 3, 2024-02-05T08:21:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: 4, 2024-02-05T08:21:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: 1 2024-02-05T08:21:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "6.6", 2024-02-05T08:21:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: "osds": [ 2024-02-05T08:21:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: 4, 2024-02-05T08:21:29.860 INFO:teuthology.orchestra.run.smithi134.stdout: 1, 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: 3 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap_items": [ 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "3.6", 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 4, 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.861 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.6", 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 5 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.7", 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:21:29.862 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.a", 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.12", 2024-02-05T08:21:29.863 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 5 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.1f", 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.864 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap_primaries": [], 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_temp": [], 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_temp": [], 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: "blocklist": { 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/788812699": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/1046500326": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/464208089": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/301941854": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/3607168099": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/673876301": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/4225724311": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:21:29.865 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/1525062017": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/3607168099": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/673876301": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3200884805": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/2952444615": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3325835814": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3161126612": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/69097091": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/1525062017": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/1645475987": "2024-02-06T08:05:21.465844+0000" 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: "range_blocklist": {}, 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profiles": { 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: "default": { 2024-02-05T08:21:29.866 INFO:teuthology.orchestra.run.smithi134.stdout: "crush-failure-domain": "osd", 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: "k": "2", 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: "m": "1", 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: "plugin": "jerasure", 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: "technique": "reed_sol_van" 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps_queue": [], 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: "new_removed_snaps": [], 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: "new_purged_snaps": [], 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_node_flags": {}, 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: "device_class_flags": {}, 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode": { 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode_enabled": false, 2024-02-05T08:21:29.867 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_bucket_count": 0, 2024-02-05T08:21:29.868 INFO:teuthology.orchestra.run.smithi134.stdout: "degraded_stretch_mode": 0, 2024-02-05T08:21:29.868 INFO:teuthology.orchestra.run.smithi134.stdout: "recovering_stretch_mode": 0, 2024-02-05T08:21:29.868 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode_bucket": 0 2024-02-05T08:21:29.868 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:29.868 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:21:29.900 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:29 smithi196 ceph-mon[33018]: pgmap v745: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 622 B/s rd, 0 B/s wr, 898 op/s; 17112/276318 objects degraded (6.193%) 2024-02-05T08:21:29.902 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:29.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:29.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:30.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:30.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:29 smithi134 ceph-mon[30086]: pgmap v745: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 622 B/s rd, 0 B/s wr, 898 op/s; 17112/276318 objects degraded (6.193%) 2024-02-05T08:21:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:29 smithi134 ceph-mon[39278]: pgmap v745: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 622 B/s rd, 0 B/s wr, 898 op/s; 17112/276318 objects degraded (6.193%) 2024-02-05T08:21:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:30.040 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:30.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:30.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:30.178 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:30.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:30.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:30.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:30.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:30.420 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:30.452 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:30.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:30.541 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:21:30.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:30.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:30.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:30.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:30.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:30.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:30.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:30.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:30.829 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:30 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1514687651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:21:30.829 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:30 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 17112/276318 objects degraded (6.193%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:30.829 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:30 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1514687651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:21:30.829 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:30 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 17112/276318 objects degraded (6.193%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:30.834 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:30.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:30.888 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:30.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:30.893 INFO:tasks.thrashosds.thrasher:j is {'epoch': 341, 'fsid': '43e13aa4-c3fc-11ee-95b6-87774f69a715', 'created': '2024-02-05T07:59:09.321262+0000', 'modified': '2024-02-05T08:21:26.627737+0000', 'last_up_change': '2024-02-05T08:19:43.862128+0000', 'last_in_change': '2024-02-05T08:18:33.608335+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T08:02:21.034012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '303', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '301', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1700000762939453, 'score_stable': 2.1700000762939453, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T08:04:39.400746+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T08:04:40.005769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T08:04:41.851891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T08:04:43.793804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T08:04:45.853999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-02-05T08:19:06.157645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '339', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'd90a5ea0-9a19-452e-9564-ca7b7a7c45e4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 324, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6802', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 2540271081}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6804', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 2540271081}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6808', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2540271081}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6806', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 2540271081}]}, 'public_addr': '172.21.15.134:6803/2540271081', 'cluster_addr': '172.21.15.134:6805/2540271081', 'heartbeat_back_addr': '172.21.15.134:6809/2540271081', 'heartbeat_front_addr': '172.21.15.134:6807/2540271081', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 309, 'down_at': 324, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6810', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 2032802960}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6812', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 2032802960}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6816', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6817', 'nonce': 2032802960}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6814', 'nonce': 2032802960}, {'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 2032802960}]}, 'public_addr': '172.21.15.134:6811/2032802960', 'cluster_addr': '172.21.15.134:6813/2032802960', 'heartbeat_back_addr': '172.21.15.134:6817/2032802960', 'heartbeat_front_addr': '172.21.15.134:6815/2032802960', 'state': ['exists']}, {'osd': 2, 'uuid': '078e578b-855b-49ff-83bd-0548a9c5a708', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 324, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6818', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6819', 'nonce': 153827987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6820', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6821', 'nonce': 153827987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 153827987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6822', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6823', 'nonce': 153827987}]}, 'public_addr': '172.21.15.134:6819/153827987', 'cluster_addr': '172.21.15.134:6821/153827987', 'heartbeat_back_addr': '172.21.15.134:6825/153827987', 'heartbeat_front_addr': '172.21.15.134:6823/153827987', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6774e330-3202-4671-894e-172cc2f6189d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 324, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 1527507056}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6828', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6829', 'nonce': 1527507056}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6832', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6833', 'nonce': 1527507056}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6830', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6831', 'nonce': 1527507056}]}, 'public_addr': '172.21.15.134:6827/1527507056', 'cluster_addr': '172.21.15.134:6829/1527507056', 'heartbeat_back_addr': '172.21.15.134:6833/1527507056', 'heartbeat_front_addr': '172.21.15.134:6831/1527507056', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '52f70cfa-ab0e-4751-a5f8-1045c61c1693', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 324, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3005156808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3005156808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3005156808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3005156808}]}, 'public_addr': '172.21.15.196:6801/3005156808', 'cluster_addr': '172.21.15.196:6803/3005156808', 'heartbeat_back_addr': '172.21.15.196:6807/3005156808', 'heartbeat_front_addr': '172.21.15.196:6805/3005156808', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b4b21769-d40f-43fd-b7e9-2477b74d3955', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 324, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 1603224792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 1603224792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 1603224792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 1603224792}]}, 'public_addr': '172.21.15.196:6809/1603224792', 'cluster_addr': '172.21.15.196:6811/1603224792', 'heartbeat_back_addr': '172.21.15.196:6815/1603224792', 'heartbeat_front_addr': '172.21.15.196:6813/1603224792', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'af1cced4-0015-47f5-b105-27641bfe4f4a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 324, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 3727413143}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 3727413143}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 3727413143}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 3727413143}]}, 'public_addr': '172.21.15.196:6817/3727413143', 'cluster_addr': '172.21.15.196:6819/3727413143', 'heartbeat_back_addr': '172.21.15.196:6823/3727413143', 'heartbeat_front_addr': '172.21.15.196:6821/3727413143', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c54beeae-9f7c-4723-a369-13bb531b4057', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 324, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 4206571010}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 4206571010}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 4206571010}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 4206571010}]}, 'public_addr': '172.21.15.196:6825/4206571010', 'cluster_addr': '172.21.15.196:6827/4206571010', 'heartbeat_back_addr': '172.21.15.196:6831/4206571010', 'heartbeat_front_addr': '172.21.15.196:6829/4206571010', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:20:46.624000+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:19:38.848711+0000', 'dead_epoch': 323}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:20:36.781950+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:20:36.057611+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:20:46.010664+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:20:41.987727+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:20:42.674824+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:20:21.973509+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.1e', 'osds': [3, 4, 1]}, {'pgid': '6.6', 'osds': [4, 1, 3]}], 'pg_upmap_items': [{'pgid': '3.6', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.134:0/788812699': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/1046500326': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/464208089': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/301941854': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:6800/3607168099': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6800/673876301': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/4225724311': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6800/1525062017': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:6801/3607168099': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6801/673876301': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/3200884805': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/2952444615': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:0/3325835814': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/3161126612': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:0/69097091': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:6801/1525062017': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/1645475987': '2024-02-06T08:05:21.465844+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T08:21:30.893 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.1e 2024-02-05T08:21:30.893 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd rm-pg-upmap 3.1e 2024-02-05T08:21:30.938 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:30 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1514687651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:21:30.939 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:30 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 17112/276318 objects degraded (6.193%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:30.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:31.024 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:31.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:31.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:31.125 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:31.165 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:31.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:31.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:31.313 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:31.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:31.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:31.415 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:31.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:31.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:31.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:31.529 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:31.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:31.622 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:31.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:31.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:31.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:31.773 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:31.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:31.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:31.815 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:21:31.816 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:21:31.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:31.922 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:31 smithi134 ceph-mon[39278]: pgmap v746: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 16346/260994 objects degraded (6.263%) 2024-02-05T08:21:31.922 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:31 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:21:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:31 smithi134 ceph-mon[30086]: pgmap v746: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 16346/260994 objects degraded (6.263%) 2024-02-05T08:21:31.922 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:31 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:21:31.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:31.924 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:31.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:32.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:32.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:31 smithi196 ceph-mon[33018]: pgmap v746: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 16346/260994 objects degraded (6.263%) 2024-02-05T08:21:32.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:31 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:21:32.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:32.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:32.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:32.172 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:32.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:32.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:32.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:32.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:32.314 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:21:32.315 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:21:32.315 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:21:32.315 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:21:32.320 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:21:32.325 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:32.341 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:32.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:32.360 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:21:32.398 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:21:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:21:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:21:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:21:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:21:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:21:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:21:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:21:32.399 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:21:32.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:32.484 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:32.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:32.585 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:32.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:32.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:32.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:32 smithi134 ceph-mon[39278]: pgmap v747: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 16346/260994 objects degraded (6.263%) 2024-02-05T08:21:32.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:32 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2222695040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:32.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:32 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:32.689 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:32 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:32.690 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:32 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:32.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:32 smithi134 ceph-mon[30086]: pgmap v747: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 16346/260994 objects degraded (6.263%) 2024-02-05T08:21:32.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:32 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2222695040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:32.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:32 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:32.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:32 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:32.690 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:32 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:32.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:32.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:32.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:32.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:32 smithi196 ceph-mon[33018]: pgmap v747: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 16346/260994 objects degraded (6.263%) 2024-02-05T08:21:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:32 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2222695040' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:32 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:32 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:32.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:32 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:32.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:32.915 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:32.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:32.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T08:21:33.017 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:33.048 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:33.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:33.150 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:33.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:33.188 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:33.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:33.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:33.328 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:33.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:33.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:33.430 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:33.473 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:33.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:33.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:33.612 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:33.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:33.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:33.714 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:33.748 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:33.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:33.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:33.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:33.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:33.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:33.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:33.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:34.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:33 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:34.029 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:34.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:34.030 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:33 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:34.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:33 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:34.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:21:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:21:34.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:34.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:34.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:34.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:34.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:34.304 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:34.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:34.405 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:34.443 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:34.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:34.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:34.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:34.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:34.687 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:34.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:34.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:34.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:34.766 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:34 smithi134 ceph-mon[30086]: pgmap v748: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 15253/245328 objects degraded (6.217%) 2024-02-05T08:21:34.767 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:34 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:34.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:34 smithi134 ceph-mon[39278]: pgmap v748: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 15253/245328 objects degraded (6.217%) 2024-02-05T08:21:34.767 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:34 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:34.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:34.864 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:34.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:34.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:35.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:35.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:34 smithi196 ceph-mon[33018]: pgmap v748: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 15253/245328 objects degraded (6.217%) 2024-02-05T08:21:35.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:34 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:35.104 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:35.142 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:35.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:35.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:35.281 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:35.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:35.383 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:35.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:35.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:35.481 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:21:35.482 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:21:35.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:35.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:35.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:35.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:35.713 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:35.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:35.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:35.815 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:35.842 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:35 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 15253/245328 objects degraded (6.217%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:35.843 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:35 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 15253/245328 objects degraded (6.217%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:35.865 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:35.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:35.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:35.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:35.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:35.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:35.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:35 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 15253/245328 objects degraded (6.217%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:36.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:36.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:36.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:36.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:36.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:36.288 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:36.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:36.390 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:36.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:36.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:36.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:36.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:36.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:36.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:36.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:36.665 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:36.702 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:36.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:36.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:36.804 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:36.837 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:36.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:36.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:36.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:36.968 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:36 smithi134 ceph-mon[39278]: pgmap v749: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 15253/242886 objects degraded (6.280%) 2024-02-05T08:21:36.969 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:36 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:21:36.969 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:36 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:21:36.969 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:36 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:36.969 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/60482000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:21:36.969 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:36 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:21:36.969 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:21:36.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:36 smithi134 ceph-mon[30086]: pgmap v749: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 15253/242886 objects degraded (6.280%) 2024-02-05T08:21:36.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:36 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:21:36.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:36 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:21:36.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:36 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:36.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/60482000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:21:36.972 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:36 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:21:36.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:36.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:36.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:36.980 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:36 smithi196 ceph-mon[33018]: pgmap v749: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 15253/242886 objects degraded (6.280%) 2024-02-05T08:21:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:36 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:21:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:36 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:21:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:36 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/60482000' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:21:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:36 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:21:37.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:37.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:37.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:37.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:37.259 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:37.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:37.361 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:37.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:37.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:37.483 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:21:37.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:37.580 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:37.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:37.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:37.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:37.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:37.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:37.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:37.844 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:37.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:37.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:37.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:37.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:37.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:38.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:21:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:21:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:21:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:37 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:21:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:37 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:21:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:37 smithi134 ceph-mon[30086]: osdmap e342: 8 total, 7 up, 8 in 2024-02-05T08:21:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:37 smithi134 ceph-mon[30086]: 3.1d scrub starts 2024-02-05T08:21:38.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:37 smithi134 ceph-mon[30086]: 3.1d scrub ok 2024-02-05T08:21:38.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:37 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:21:38.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:37 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:21:38.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:37 smithi134 ceph-mon[39278]: osdmap e342: 8 total, 7 up, 8 in 2024-02-05T08:21:38.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:37 smithi134 ceph-mon[39278]: 3.1d scrub starts 2024-02-05T08:21:38.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:37 smithi134 ceph-mon[39278]: 3.1d scrub ok 2024-02-05T08:21:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:37 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:21:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:37 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:21:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:37 smithi196 ceph-mon[33018]: osdmap e342: 8 total, 7 up, 8 in 2024-02-05T08:21:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:37 smithi196 ceph-mon[33018]: 3.1d scrub starts 2024-02-05T08:21:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:37 smithi196 ceph-mon[33018]: 3.1d scrub ok 2024-02-05T08:21:38.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:38.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:38.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:38.225 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:38.261 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:38.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:38.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:38.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:38.400 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:38.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:38.502 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:38.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:38.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:38.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:38.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:38.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:38.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:38.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:38.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:38.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:38.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:38.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:38.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:38.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:38.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:38.991 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[30086]: 6.9 scrub starts 2024-02-05T08:21:38.991 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[30086]: 6.15 scrub starts 2024-02-05T08:21:38.991 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[30086]: 6.9 scrub ok 2024-02-05T08:21:38.991 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[30086]: 6.15 scrub ok 2024-02-05T08:21:38.991 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[30086]: pgmap v751: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 14904/239604 objects degraded (6.220%) 2024-02-05T08:21:38.991 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[30086]: 6.7 scrub starts 2024-02-05T08:21:38.991 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:21:38.992 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[39278]: 6.9 scrub starts 2024-02-05T08:21:38.992 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[39278]: 6.15 scrub starts 2024-02-05T08:21:38.992 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[39278]: 6.9 scrub ok 2024-02-05T08:21:38.992 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[39278]: 6.15 scrub ok 2024-02-05T08:21:38.992 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[39278]: pgmap v751: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 14904/239604 objects degraded (6.220%) 2024-02-05T08:21:38.992 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[39278]: 6.7 scrub starts 2024-02-05T08:21:38.992 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:21:39.040 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:39.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:38 smithi196 ceph-mon[33018]: 6.9 scrub starts 2024-02-05T08:21:39.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:38 smithi196 ceph-mon[33018]: 6.15 scrub starts 2024-02-05T08:21:39.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:38 smithi196 ceph-mon[33018]: 6.9 scrub ok 2024-02-05T08:21:39.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:38 smithi196 ceph-mon[33018]: 6.15 scrub ok 2024-02-05T08:21:39.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:38 smithi196 ceph-mon[33018]: pgmap v751: 158 pgs: 25 active+undersized+degraded, 33 active+undersized, 100 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s; 14904/239604 objects degraded (6.220%) 2024-02-05T08:21:39.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:38 smithi196 ceph-mon[33018]: 6.7 scrub starts 2024-02-05T08:21:39.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:21:39.082 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:39.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:39.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:39.224 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:39.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:39.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:39.325 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:39.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:39.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:39.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:39.476 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:39.517 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:39.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:39.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:39.620 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:39.658 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:39.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:39.760 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:39.799 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:39.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:39.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:39.922 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:39.930 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:21:39.934 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:39.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:39.974 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:39 smithi134 ceph-mon[39278]: 3.17 scrub starts 2024-02-05T08:21:39.974 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:39 smithi134 ceph-mon[39278]: 3.17 scrub ok 2024-02-05T08:21:39.974 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:39 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:21:39.974 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:39 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:21:39.974 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:39 smithi134 ceph-mon[39278]: 6.7 scrub ok 2024-02-05T08:21:39.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:39 smithi134 ceph-mon[30086]: 3.17 scrub starts 2024-02-05T08:21:39.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:39 smithi134 ceph-mon[30086]: 3.17 scrub ok 2024-02-05T08:21:39.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:39 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:21:39.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:39 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:21:39.975 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:39 smithi134 ceph-mon[30086]: 6.7 scrub ok 2024-02-05T08:21:40.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:39 smithi196 ceph-mon[33018]: 3.17 scrub starts 2024-02-05T08:21:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:39 smithi196 ceph-mon[33018]: 3.17 scrub ok 2024-02-05T08:21:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:39 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:21:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:39 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:21:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:39 smithi196 ceph-mon[33018]: 6.7 scrub ok 2024-02-05T08:21:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:40.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:40.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:40.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:40.207 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:40.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:40.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:40.309 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:40.344 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:40.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:40.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:40.446 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:40.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:40.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:40.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:40.586 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:40.620 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:40.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:40.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:40.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:40.766 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:40.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:40.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:40.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:40 smithi134 ceph-mon[30086]: 5.3 scrub starts 2024-02-05T08:21:40.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:40 smithi134 ceph-mon[30086]: 5.3 scrub ok 2024-02-05T08:21:40.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:40 smithi134 ceph-mon[30086]: pgmap v752: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s; 14182/224673 objects degraded (6.312%) 2024-02-05T08:21:40.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:40 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 14904/239604 objects degraded (6.220%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:40.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:40 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1604331329' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "3.1e"}]: dispatch 2024-02-05T08:21:40.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:40 smithi134 ceph-mon[39278]: 5.3 scrub starts 2024-02-05T08:21:40.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:40 smithi134 ceph-mon[39278]: 5.3 scrub ok 2024-02-05T08:21:40.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:40 smithi134 ceph-mon[39278]: pgmap v752: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s; 14182/224673 objects degraded (6.312%) 2024-02-05T08:21:40.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:40 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 14904/239604 objects degraded (6.220%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:40.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:40 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1604331329' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "3.1e"}]: dispatch 2024-02-05T08:21:40.866 INFO:teuthology.orchestra.run.smithi134.stderr:clear 3.1e pg_upmap mapping 2024-02-05T08:21:40.868 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:40.912 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:40.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:40.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:41.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:41.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:41.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:40 smithi196 ceph-mon[33018]: 5.3 scrub starts 2024-02-05T08:21:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:40 smithi196 ceph-mon[33018]: 5.3 scrub ok 2024-02-05T08:21:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:40 smithi196 ceph-mon[33018]: pgmap v752: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s; 14182/224673 objects degraded (6.312%) 2024-02-05T08:21:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:40 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 14904/239604 objects degraded (6.220%), 25 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:40 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1604331329' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "3.1e"}]: dispatch 2024-02-05T08:21:41.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:41.189 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:41.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:41.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:41.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:41.328 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:41.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:41.431 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:41.469 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:41.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:41.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:41.571 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:41.605 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:41.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:41.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:41.743 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:41.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:41.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:41.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:41.882 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:41.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:41.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:42.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:21:42.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:21:42.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[30086]: 3.d scrub starts 2024-02-05T08:21:42.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[30086]: 3.d scrub ok 2024-02-05T08:21:42.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[30086]: 3.5 scrub starts 2024-02-05T08:21:42.008 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[30086]: 3.5 scrub ok 2024-02-05T08:21:42.009 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1604331329' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "3.1e"}]': finished 2024-02-05T08:21:42.009 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[30086]: osdmap e343: 8 total, 7 up, 8 in 2024-02-05T08:21:42.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:21:42.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:21:42.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[39278]: 3.d scrub starts 2024-02-05T08:21:42.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[39278]: 3.d scrub ok 2024-02-05T08:21:42.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[39278]: 3.5 scrub starts 2024-02-05T08:21:42.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[39278]: 3.5 scrub ok 2024-02-05T08:21:42.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1604331329' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "3.1e"}]': finished 2024-02-05T08:21:42.009 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:41 smithi134 ceph-mon[39278]: osdmap e343: 8 total, 7 up, 8 in 2024-02-05T08:21:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:42.019 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:42.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:41 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:21:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:41 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:21:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:41 smithi196 ceph-mon[33018]: 3.d scrub starts 2024-02-05T08:21:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:41 smithi196 ceph-mon[33018]: 3.d scrub ok 2024-02-05T08:21:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:41 smithi196 ceph-mon[33018]: 3.5 scrub starts 2024-02-05T08:21:42.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:41 smithi196 ceph-mon[33018]: 3.5 scrub ok 2024-02-05T08:21:42.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:41 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1604331329' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "3.1e"}]': finished 2024-02-05T08:21:42.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:41 smithi196 ceph-mon[33018]: osdmap e343: 8 total, 7 up, 8 in 2024-02-05T08:21:42.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:42.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:42.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:42.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:42.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:42.291 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:42.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:42.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:42.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:42.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:42.431 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:42.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:42.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:42.593 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:42.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:42.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:42.733 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:42.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:42.835 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:42.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:42.871 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:42.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:42.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:43.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:43.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:43.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:43.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:42 smithi134 ceph-mon[30086]: pgmap v754: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 860 op/s; 14182/224673 objects degraded (6.312%) 2024-02-05T08:21:43.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:42 smithi134 ceph-mon[30086]: osdmap e344: 8 total, 7 up, 8 in 2024-02-05T08:21:43.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:42 smithi134 ceph-mon[39278]: pgmap v754: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 860 op/s; 14182/224673 objects degraded (6.312%) 2024-02-05T08:21:43.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:42 smithi134 ceph-mon[39278]: osdmap e344: 8 total, 7 up, 8 in 2024-02-05T08:21:43.107 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:43.144 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:43.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:43.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:43.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:43.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:43.285 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:43.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:43.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:42 smithi196 ceph-mon[33018]: pgmap v754: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 33 active+undersized, 99 active+clean; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 860 op/s; 14182/224673 objects degraded (6.312%) 2024-02-05T08:21:43.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:42 smithi196 ceph-mon[33018]: osdmap e344: 8 total, 7 up, 8 in 2024-02-05T08:21:43.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:43.429 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:43.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:43.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:43.531 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:43.566 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:43.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:43.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:43.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:43.672 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:21:43.672 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:21:43.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.672 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15714.0:195304 10.5 e344/309 10:a757f515:::benchmark_data_smithi134_174861_object63842:head v 344'12416, mlcod=344'12411)", 2024-02-05T08:21:43.672 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:21:43.652798+0000", 2024-02-05T08:21:43.672 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.011957010000000001, 2024-02-05T08:21:43.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.011989843, 2024-02-05T08:21:43.672 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:43.672 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:43.672 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:43.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.672 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:43.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.652798+0000", 2024-02-05T08:21:43.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:43.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.652798+0000", 2024-02-05T08:21:43.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:43.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.652799+0000", 2024-02-05T08:21:43.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1699999999999997e-07 2024-02-05T08:21:43.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:43.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.652805+0000", 2024-02-05T08:21:43.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6429999999999998e-06 2024-02-05T08:21:43.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:43.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.652819+0000", 2024-02-05T08:21:43.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3366999999999999e-05 2024-02-05T08:21:43.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:43.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.652827+0000", 2024-02-05T08:21:43.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9300000000000003e-06 2024-02-05T08:21:43.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:43.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.653925+0000", 2024-02-05T08:21:43.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010984 2024-02-05T08:21:43.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:43.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.653944+0000", 2024-02-05T08:21:43.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9001999999999999e-05 2024-02-05T08:21:43.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:43.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.664789+0000", 2024-02-05T08:21:43.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010845100999999999 2024-02-05T08:21:43.677 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.677 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:43.677 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.678 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15714.0:195317 10.8 10:1c52db44:::benchmark_data_smithi134_174861_object63855:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-02-05T08:21:43.678 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:21:43.654940+0000", 2024-02-05T08:21:43.678 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0098147080000000001, 2024-02-05T08:21:43.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0099324489999999994, 2024-02-05T08:21:43.678 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:43.678 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:21:43.678 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:43.678 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15714", 2024-02-05T08:21:43.678 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:21:43.678 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 195317 2024-02-05T08:21:43.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.654940+0000", 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.654940+0000", 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:43.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.654942+0000", 2024-02-05T08:21:43.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3009999999999999e-06 2024-02-05T08:21:43.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:43.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.654947+0000", 2024-02-05T08:21:43.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0100000000000003e-06 2024-02-05T08:21:43.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:43.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.654963+0000", 2024-02-05T08:21:43.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6821999999999999e-05 2024-02-05T08:21:43.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.654972+0000", 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6519999999999995e-06 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.656153+0000", 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001180431 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.656274+0000", 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000121227 2024-02-05T08:21:43.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,7", 2024-02-05T08:21:43.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.656416+0000", 2024-02-05T08:21:43.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000142008 2024-02-05T08:21:43.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:43.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.660548+0000", 2024-02-05T08:21:43.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0041318520000000001 2024-02-05T08:21:43.682 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.682 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:43.682 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15714.0:195309 10.5 e344/309 10:a34e2a3f:::benchmark_data_smithi134_174861_object63847:head v 344'12417, mlcod=344'12411)", 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:21:43.660326+0000", 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0044287290000000002, 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0046369330000000002, 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.660326+0000", 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.683 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.660326+0000", 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.660327+0000", 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.8999999999999998e-07 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.660330+0000", 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7110000000000001e-06 2024-02-05T08:21:43.684 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.660338+0000", 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3440000000000003e-06 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.660342+0000", 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3749999999999996e-06 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.660401+0000", 2024-02-05T08:21:43.685 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9273e-05 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.660423+0000", 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2116999999999998e-05 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15714.0:195327 10.8 10:128c156d:::benchmark_data_smithi134_174861_object63865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:21:43.663144+0000", 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0016107000000000001, 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001887931, 2024-02-05T08:21:43.686 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15714", 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 195327 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663144+0000", 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.687 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663144+0000", 2024-02-05T08:21:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663146+0000", 2024-02-05T08:21:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.206e-06 2024-02-05T08:21:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663150+0000", 2024-02-05T08:21:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.729e-06 2024-02-05T08:21:43.688 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663162+0000", 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1647e-05 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663170+0000", 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4300000000000006e-06 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663237+0000", 2024-02-05T08:21:43.689 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6670999999999999e-05 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663376+0000", 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013859699999999999 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,7", 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663531+0000", 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000154976 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.690 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop_reply(client.15714.0:195317 10.8 e344/309)", 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:21:43.663342+0000", 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.001413299, 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001766763, 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663342+0000", 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:43.691 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663342+0000", 2024-02-05T08:21:43.692 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.692 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:43.692 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663343+0000", 2024-02-05T08:21:43.692 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6199999999999996e-07 2024-02-05T08:21:43.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.692 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:43.692 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663351+0000", 2024-02-05T08:21:43.692 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6540000000000005e-06 2024-02-05T08:21:43.692 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.692 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663363+0000", 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1833e-05 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663371+0000", 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9300000000000003e-06 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.693 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15714.0:195325 10.6 e344/309 10:6b94c3f4:::benchmark_data_smithi134_174861_object63863:head v 344'12077, mlcod=344'12075)", 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:21:43.663411+0000", 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0013440430000000001, 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017496269999999999, 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663411+0000", 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:43.694 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663411+0000", 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663411+0000", 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7399999999999998e-07 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663418+0000", 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5270000000000004e-06 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.695 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663426+0000", 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8429999999999993e-06 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663432+0000", 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0399999999999998e-06 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663495+0000", 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3446000000000002e-05 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.696 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663515+0000", 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9969000000000001e-05 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15714.0:195328 10.8 10.b175a48 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:21:43.663851+0000", 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.00090439700000000004, 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0013807769999999999, 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:43.697 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15714", 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 195328 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663851+0000", 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:43.698 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663851+0000", 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663852+0000", 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3600000000000002e-07 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663872+0000", 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0321000000000001e-05 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.699 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663901+0000", 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9597000000000001e-05 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:43.663914+0000", 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2883e-05 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 7 2024-02-05T08:21:43.700 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:21:43.713 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:43.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:43.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:43.815 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:43.840 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:21:43.840 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:21:43.854 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:43.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:43.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:43.862 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:21:43.900 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:21:43.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:43 smithi134 ceph-mon[39278]: 6.1e scrub starts 2024-02-05T08:21:43.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:43 smithi134 ceph-mon[39278]: 6.1e scrub ok 2024-02-05T08:21:43.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:43 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2971198079' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:43.938 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:43 smithi134 ceph-mon[30086]: 6.1e scrub starts 2024-02-05T08:21:43.938 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:43 smithi134 ceph-mon[30086]: 6.1e scrub ok 2024-02-05T08:21:43.938 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:43 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2971198079' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:43.939 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:21:43.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:43.982 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:21:43.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:43.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:44.018 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:21:44.018 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:21:44.018 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:21:44.018 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:21:44.019 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:21:44.019 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:21:44.019 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:21:44.019 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:21:44.019 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:21:44.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:44.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:44.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:44.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:44.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:43 smithi196 ceph-mon[33018]: 6.1e scrub starts 2024-02-05T08:21:44.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:43 smithi196 ceph-mon[33018]: 6.1e scrub ok 2024-02-05T08:21:44.215 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:43 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2971198079' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:44.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:21:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:21:44.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:44.277 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:44.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:44.381 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:44.415 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:44.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:44.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:44.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:44.554 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:44.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:44.642 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T08:21:44.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:44.716 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:44.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:44.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:44.818 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:44.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:44.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:44.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:45.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:45.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:44 smithi196 ceph-mon[33018]: pgmap v756: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.71k op/s; 12972/207573 objects degraded (6.249%) 2024-02-05T08:21:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:44 smithi196 ceph-mon[33018]: 4.1a scrub starts 2024-02-05T08:21:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:44 smithi196 ceph-mon[33018]: 4.1a scrub ok 2024-02-05T08:21:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:44 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:44 smithi196 ceph-mon[33018]: 6.8 scrub starts 2024-02-05T08:21:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:44 smithi196 ceph-mon[33018]: 6.8 scrub ok 2024-02-05T08:21:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:44 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 12972/207573 objects degraded (6.249%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:45.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:45.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:45.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:45.233 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[30086]: pgmap v756: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.71k op/s; 12972/207573 objects degraded (6.249%) 2024-02-05T08:21:45.233 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[30086]: 4.1a scrub starts 2024-02-05T08:21:45.233 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[30086]: 4.1a scrub ok 2024-02-05T08:21:45.233 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:45.233 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[30086]: 6.8 scrub starts 2024-02-05T08:21:45.233 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[30086]: 6.8 scrub ok 2024-02-05T08:21:45.234 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 12972/207573 objects degraded (6.249%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:45.234 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[39278]: pgmap v756: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.71k op/s; 12972/207573 objects degraded (6.249%) 2024-02-05T08:21:45.234 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[39278]: 4.1a scrub starts 2024-02-05T08:21:45.234 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[39278]: 4.1a scrub ok 2024-02-05T08:21:45.234 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:45.234 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[39278]: 6.8 scrub starts 2024-02-05T08:21:45.234 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[39278]: 6.8 scrub ok 2024-02-05T08:21:45.234 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:44 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 12972/207573 objects degraded (6.249%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:45.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:45.271 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:45.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:45.374 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:45.397 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:45.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:45.499 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:45.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:45.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:45.638 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:45.667 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:45.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:45.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:45.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:45.807 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:45.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:45.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:45.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:45.945 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:45.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:46.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:46.077 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:46.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:46.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:46.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:46.208 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:46.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:46.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:46.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:45 smithi134 ceph-mon[30086]: 4.8 scrub starts 2024-02-05T08:21:46.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:45 smithi134 ceph-mon[30086]: 4.8 scrub ok 2024-02-05T08:21:46.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:45 smithi134 ceph-mon[30086]: 3.14 scrub starts 2024-02-05T08:21:46.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:45 smithi134 ceph-mon[30086]: 3.14 scrub ok 2024-02-05T08:21:46.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:45 smithi134 ceph-mon[39278]: 4.8 scrub starts 2024-02-05T08:21:46.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:45 smithi134 ceph-mon[39278]: 4.8 scrub ok 2024-02-05T08:21:46.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:45 smithi134 ceph-mon[39278]: 3.14 scrub starts 2024-02-05T08:21:46.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:45 smithi134 ceph-mon[39278]: 3.14 scrub ok 2024-02-05T08:21:46.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:46.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:45 smithi196 ceph-mon[33018]: 4.8 scrub starts 2024-02-05T08:21:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:45 smithi196 ceph-mon[33018]: 4.8 scrub ok 2024-02-05T08:21:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:45 smithi196 ceph-mon[33018]: 3.14 scrub starts 2024-02-05T08:21:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:45 smithi196 ceph-mon[33018]: 3.14 scrub ok 2024-02-05T08:21:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:46.351 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:46.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:46.453 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:46.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:46.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:46.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:46.572 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:21:46.572 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:21:46.592 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:46.630 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:46.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:46.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:46.643 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:46.706 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:46.731 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:46.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:46.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:46.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:46.871 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:46.905 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:46.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:47.007 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:47.041 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:47.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:47.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:47.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:47.181 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:47.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:47.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[30086]: 6.a scrub starts 2024-02-05T08:21:47.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[30086]: 6.a scrub ok 2024-02-05T08:21:47.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[30086]: 10.a scrub starts 2024-02-05T08:21:47.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[30086]: 3.1e scrub starts 2024-02-05T08:21:47.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[30086]: 3.1e scrub ok 2024-02-05T08:21:47.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[30086]: pgmap v757: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 12972/205182 objects degraded (6.322%) 2024-02-05T08:21:47.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[30086]: 5.9 scrub starts 2024-02-05T08:21:47.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[30086]: 5.9 scrub ok 2024-02-05T08:21:47.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[39278]: 6.a scrub starts 2024-02-05T08:21:47.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[39278]: 6.a scrub ok 2024-02-05T08:21:47.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[39278]: 10.a scrub starts 2024-02-05T08:21:47.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[39278]: 3.1e scrub starts 2024-02-05T08:21:47.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[39278]: 3.1e scrub ok 2024-02-05T08:21:47.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[39278]: pgmap v757: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 12972/205182 objects degraded (6.322%) 2024-02-05T08:21:47.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[39278]: 5.9 scrub starts 2024-02-05T08:21:47.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:46 smithi134 ceph-mon[39278]: 5.9 scrub ok 2024-02-05T08:21:47.281 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:47.312 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:47.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:46 smithi196 ceph-mon[33018]: 6.a scrub starts 2024-02-05T08:21:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:46 smithi196 ceph-mon[33018]: 6.a scrub ok 2024-02-05T08:21:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:46 smithi196 ceph-mon[33018]: 10.a scrub starts 2024-02-05T08:21:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:46 smithi196 ceph-mon[33018]: 3.1e scrub starts 2024-02-05T08:21:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:46 smithi196 ceph-mon[33018]: 3.1e scrub ok 2024-02-05T08:21:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:46 smithi196 ceph-mon[33018]: pgmap v757: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 12972/205182 objects degraded (6.322%) 2024-02-05T08:21:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:46 smithi196 ceph-mon[33018]: 5.9 scrub starts 2024-02-05T08:21:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:46 smithi196 ceph-mon[33018]: 5.9 scrub ok 2024-02-05T08:21:47.414 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:47.452 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:47.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:47.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:47.554 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:47.588 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:47.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:47.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:47.690 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:47.731 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:47.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:47.832 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:47.866 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:47.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:47.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:47.935 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:21:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:21:47.935 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[30086]: 10.9 scrub starts 2024-02-05T08:21:47.935 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:21:47.935 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:21:47.936 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[30086]: 3.3 scrub starts 2024-02-05T08:21:47.936 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[30086]: 3.3 scrub ok 2024-02-05T08:21:47.936 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[30086]: 6.11 scrub starts 2024-02-05T08:21:47.936 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[30086]: 6.11 scrub ok 2024-02-05T08:21:47.936 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[30086]: 10.a scrub ok 2024-02-05T08:21:47.936 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:21:47.936 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:21:47.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[39278]: 10.9 scrub starts 2024-02-05T08:21:47.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:21:47.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:21:47.936 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[39278]: 3.3 scrub starts 2024-02-05T08:21:47.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[39278]: 3.3 scrub ok 2024-02-05T08:21:47.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[39278]: 6.11 scrub starts 2024-02-05T08:21:47.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[39278]: 6.11 scrub ok 2024-02-05T08:21:47.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[39278]: 10.a scrub ok 2024-02-05T08:21:47.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:21:47.937 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:47 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:21:47.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:48.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:48.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:48.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:48.107 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:48.119 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-02-05T08:21:48.119 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:21:48.120 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:21:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:48.147 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:48.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:48.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:48.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:48.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:47 smithi196 ceph-mon[33018]: 10.9 scrub starts 2024-02-05T08:21:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:47 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:21:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:47 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:21:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:47 smithi196 ceph-mon[33018]: 3.3 scrub starts 2024-02-05T08:21:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:47 smithi196 ceph-mon[33018]: 3.3 scrub ok 2024-02-05T08:21:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:47 smithi196 ceph-mon[33018]: 6.11 scrub starts 2024-02-05T08:21:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:47 smithi196 ceph-mon[33018]: 6.11 scrub ok 2024-02-05T08:21:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:47 smithi196 ceph-mon[33018]: 10.a scrub ok 2024-02-05T08:21:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:47 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:21:48.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:47 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:21:48.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:48.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:48.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:48.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:48.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:48.554 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:48.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:48.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:21:48.656 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:48.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:48.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:48.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:48.797 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:48.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:48.839 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:48.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:48.909 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:48 smithi134 ceph-mon[30086]: 1.1 scrub starts 2024-02-05T08:21:48.909 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:48 smithi134 ceph-mon[30086]: 1.1 scrub ok 2024-02-05T08:21:48.909 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:48 smithi134 ceph-mon[30086]: pgmap v758: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 4.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 947 op/s; 12605/201915 objects degraded (6.243%) 2024-02-05T08:21:48.909 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:48 smithi134 ceph-mon[30086]: osdmap e345: 8 total, 7 up, 8 in 2024-02-05T08:21:48.909 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:48 smithi134 ceph-mon[30086]: 10.9 scrub ok 2024-02-05T08:21:48.940 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:48.974 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:48.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:48.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:49.075 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:49.111 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:49.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:49.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:49.240 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:48 smithi134 ceph-mon[39278]: 1.1 scrub starts 2024-02-05T08:21:49.240 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:48 smithi134 ceph-mon[39278]: 1.1 scrub ok 2024-02-05T08:21:49.240 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:48 smithi134 ceph-mon[39278]: pgmap v758: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 4.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 947 op/s; 12605/201915 objects degraded (6.243%) 2024-02-05T08:21:49.240 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:48 smithi134 ceph-mon[39278]: osdmap e345: 8 total, 7 up, 8 in 2024-02-05T08:21:49.240 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:48 smithi134 ceph-mon[39278]: 10.9 scrub ok 2024-02-05T08:21:49.251 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:49.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:49.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:48 smithi196 ceph-mon[33018]: 1.1 scrub starts 2024-02-05T08:21:49.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:48 smithi196 ceph-mon[33018]: 1.1 scrub ok 2024-02-05T08:21:49.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:48 smithi196 ceph-mon[33018]: pgmap v758: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 4.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 947 op/s; 12605/201915 objects degraded (6.243%) 2024-02-05T08:21:49.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:48 smithi196 ceph-mon[33018]: osdmap e345: 8 total, 7 up, 8 in 2024-02-05T08:21:49.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:48 smithi196 ceph-mon[33018]: 10.9 scrub ok 2024-02-05T08:21:49.353 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:49.387 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:49.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:49.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:49.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:49.524 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:49.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:49.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:49.663 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:49.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:49.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:49.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:49.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:49.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:49.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:49.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:49.910 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[30086]: 1.a scrub starts 2024-02-05T08:21:49.911 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[30086]: 1.a scrub ok 2024-02-05T08:21:49.911 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[30086]: 4.5 scrub starts 2024-02-05T08:21:49.911 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[30086]: 4.5 scrub ok 2024-02-05T08:21:49.911 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[30086]: 4.1e scrub starts 2024-02-05T08:21:49.911 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[30086]: 4.1e scrub ok 2024-02-05T08:21:49.911 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 12605/201915 objects degraded (6.243%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:49.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:49.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:50.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:50.083 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:50.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:50.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:50.185 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:50.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:50.219 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:50.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:50.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[39278]: 1.a scrub starts 2024-02-05T08:21:50.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[39278]: 1.a scrub ok 2024-02-05T08:21:50.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[39278]: 4.5 scrub starts 2024-02-05T08:21:50.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[39278]: 4.5 scrub ok 2024-02-05T08:21:50.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[39278]: 4.1e scrub starts 2024-02-05T08:21:50.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[39278]: 4.1e scrub ok 2024-02-05T08:21:50.242 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:49 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 12605/201915 objects degraded (6.243%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:49 smithi196 ceph-mon[33018]: 1.a scrub starts 2024-02-05T08:21:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:49 smithi196 ceph-mon[33018]: 1.a scrub ok 2024-02-05T08:21:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:49 smithi196 ceph-mon[33018]: 4.5 scrub starts 2024-02-05T08:21:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:49 smithi196 ceph-mon[33018]: 4.5 scrub ok 2024-02-05T08:21:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:49 smithi196 ceph-mon[33018]: 4.1e scrub starts 2024-02-05T08:21:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:49 smithi196 ceph-mon[33018]: 4.1e scrub ok 2024-02-05T08:21:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:49 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 12605/201915 objects degraded (6.243%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:50.320 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:50.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:50.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:50.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:50.460 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:50.504 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:50.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:50.606 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:50.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:50.643 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:50.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:50.665 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:21:50.665 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:21:50.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:50.773 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:50.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:50.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:50.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:50.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:50.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:50.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:51.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:51.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:51.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:51.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:51.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:51.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:51.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:51.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:50 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:21:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:50 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:21:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:50 smithi134 ceph-mon[30086]: pgmap v760: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 11848/187584 objects degraded (6.316%) 2024-02-05T08:21:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:50 smithi134 ceph-mon[30086]: 4.16 scrub starts 2024-02-05T08:21:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:50 smithi134 ceph-mon[30086]: 4.16 scrub ok 2024-02-05T08:21:51.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:50 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:21:51.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:50 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:21:51.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:50 smithi134 ceph-mon[39278]: pgmap v760: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 11848/187584 objects degraded (6.316%) 2024-02-05T08:21:51.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:50 smithi134 ceph-mon[39278]: 4.16 scrub starts 2024-02-05T08:21:51.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:50 smithi134 ceph-mon[39278]: 4.16 scrub ok 2024-02-05T08:21:51.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:50 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:21:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:50 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:21:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:50 smithi196 ceph-mon[33018]: pgmap v760: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 11848/187584 objects degraded (6.316%) 2024-02-05T08:21:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:50 smithi196 ceph-mon[33018]: 4.16 scrub starts 2024-02-05T08:21:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:50 smithi196 ceph-mon[33018]: 4.16 scrub ok 2024-02-05T08:21:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:51.317 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:51.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:51.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:51.450 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:51.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:51.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:51.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:51.593 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:51.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:51.694 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:51.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:51.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:51.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:51.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:51.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:51.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:51.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T08:21:51.898 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:51.967 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:51.971 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:21:51.971 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:52.040 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:52.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:52.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:52.116 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:51 smithi134 ceph-mon[30086]: 5.11 scrub starts 2024-02-05T08:21:52.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:51 smithi134 ceph-mon[30086]: 5.11 scrub ok 2024-02-05T08:21:52.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:51 smithi134 ceph-mon[30086]: 6.1b scrub starts 2024-02-05T08:21:52.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:51 smithi134 ceph-mon[30086]: 6.1b scrub ok 2024-02-05T08:21:52.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:51 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/7486491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:52.119 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:51 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:52.119 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:51 smithi134 ceph-mon[39278]: 5.11 scrub starts 2024-02-05T08:21:52.119 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:51 smithi134 ceph-mon[39278]: 5.11 scrub ok 2024-02-05T08:21:52.120 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:51 smithi134 ceph-mon[39278]: 6.1b scrub starts 2024-02-05T08:21:52.120 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:51 smithi134 ceph-mon[39278]: 6.1b scrub ok 2024-02-05T08:21:52.120 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:51 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/7486491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:52.120 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:51 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:52.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:52.179 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:52.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:52.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:52.281 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:51 smithi196 ceph-mon[33018]: 5.11 scrub starts 2024-02-05T08:21:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:51 smithi196 ceph-mon[33018]: 5.11 scrub ok 2024-02-05T08:21:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:51 smithi196 ceph-mon[33018]: 6.1b scrub starts 2024-02-05T08:21:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:51 smithi196 ceph-mon[33018]: 6.1b scrub ok 2024-02-05T08:21:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:51 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/7486491' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:51 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:21:52.316 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:52.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:52.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:52.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:52.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:52.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:52.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:52.598 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:52.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:52.700 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:52.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:52.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:52.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:52.837 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:52.869 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:52.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:52.971 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:53.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:53.012 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:53.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:53.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:52 smithi134 ceph-mon[30086]: pgmap v761: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.24k op/s; 11848/187584 objects degraded (6.316%) 2024-02-05T08:21:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:52 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:21:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:52 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:21:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:52 smithi134 ceph-mon[30086]: osdmap e346: 8 total, 7 up, 8 in 2024-02-05T08:21:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:52 smithi134 ceph-mon[30086]: 4.6 scrub starts 2024-02-05T08:21:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:52 smithi134 ceph-mon[30086]: 4.6 scrub ok 2024-02-05T08:21:53.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:52 smithi134 ceph-mon[39278]: pgmap v761: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.24k op/s; 11848/187584 objects degraded (6.316%) 2024-02-05T08:21:53.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:52 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:21:53.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:52 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:21:53.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:52 smithi134 ceph-mon[39278]: osdmap e346: 8 total, 7 up, 8 in 2024-02-05T08:21:53.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:52 smithi134 ceph-mon[39278]: 4.6 scrub starts 2024-02-05T08:21:53.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:52 smithi134 ceph-mon[39278]: 4.6 scrub ok 2024-02-05T08:21:53.114 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:53.151 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:53.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:53.253 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:53.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:53.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:52 smithi196 ceph-mon[33018]: pgmap v761: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.24k op/s; 11848/187584 objects degraded (6.316%) 2024-02-05T08:21:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:52 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:21:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:52 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:21:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:52 smithi196 ceph-mon[33018]: osdmap e346: 8 total, 7 up, 8 in 2024-02-05T08:21:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:52 smithi196 ceph-mon[33018]: 4.6 scrub starts 2024-02-05T08:21:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:52 smithi196 ceph-mon[33018]: 4.6 scrub ok 2024-02-05T08:21:53.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:53.424 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:53.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:53.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:53.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:53.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:53.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:53.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:53.694 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:53.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:53.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:53.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:53.833 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:53.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:53.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:53.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:53.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:53.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:54.071 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:54.107 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:54.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:54.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:54.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:54.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:54.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:21:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:21:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[30086]: osdmap e347: 8 total, 7 up, 8 in 2024-02-05T08:21:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[30086]: 3.15 scrub starts 2024-02-05T08:21:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[30086]: 3.15 scrub ok 2024-02-05T08:21:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[30086]: 4.12 scrub starts 2024-02-05T08:21:54.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[30086]: 4.12 scrub ok 2024-02-05T08:21:54.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:54.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:21:54.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:21:54.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:21:54.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[39278]: osdmap e347: 8 total, 7 up, 8 in 2024-02-05T08:21:54.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[39278]: 3.15 scrub starts 2024-02-05T08:21:54.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[39278]: 3.15 scrub ok 2024-02-05T08:21:54.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[39278]: 4.12 scrub starts 2024-02-05T08:21:54.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[39278]: 4.12 scrub ok 2024-02-05T08:21:54.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:54.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:21:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:53 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:21:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:53 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:21:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:53 smithi196 ceph-mon[33018]: osdmap e347: 8 total, 7 up, 8 in 2024-02-05T08:21:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:53 smithi196 ceph-mon[33018]: 3.15 scrub starts 2024-02-05T08:21:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:53 smithi196 ceph-mon[33018]: 3.15 scrub ok 2024-02-05T08:21:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:53 smithi196 ceph-mon[33018]: 4.12 scrub starts 2024-02-05T08:21:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:53 smithi196 ceph-mon[33018]: 4.12 scrub ok 2024-02-05T08:21:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:21:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:21:54.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:21:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:21:54.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:54.381 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:54.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:54.483 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:54.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:54.532 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:54.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:54.633 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:54.666 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:54.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:54.768 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:54.808 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:54.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:54.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:54.910 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:54.945 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:54.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:55.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:55.079 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:55.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:55.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:55.198 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[30086]: 10.f scrub starts 2024-02-05T08:21:55.198 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[30086]: pgmap v764: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.74k op/s; 10569/170538 objects degraded (6.197%) 2024-02-05T08:21:55.198 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:55.198 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[30086]: 5.d scrub starts 2024-02-05T08:21:55.198 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[30086]: 5.d scrub ok 2024-02-05T08:21:55.198 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[30086]: 5.10 deep-scrub starts 2024-02-05T08:21:55.198 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[30086]: 5.10 deep-scrub ok 2024-02-05T08:21:55.198 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:21:55.198 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:21:55.198 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 10569/170538 objects degraded (6.197%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:55.198 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[39278]: 10.f scrub starts 2024-02-05T08:21:55.199 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[39278]: pgmap v764: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.74k op/s; 10569/170538 objects degraded (6.197%) 2024-02-05T08:21:55.199 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:55.199 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[39278]: 5.d scrub starts 2024-02-05T08:21:55.199 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[39278]: 5.d scrub ok 2024-02-05T08:21:55.199 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[39278]: 5.10 deep-scrub starts 2024-02-05T08:21:55.199 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[39278]: 5.10 deep-scrub ok 2024-02-05T08:21:55.199 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:21:55.199 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:21:55.199 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:54 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 10569/170538 objects degraded (6.197%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:55.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:55.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:54 smithi196 ceph-mon[33018]: 10.f scrub starts 2024-02-05T08:21:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:54 smithi196 ceph-mon[33018]: pgmap v764: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 3.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.74k op/s; 10569/170538 objects degraded (6.197%) 2024-02-05T08:21:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:54 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:21:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:54 smithi196 ceph-mon[33018]: 5.d scrub starts 2024-02-05T08:21:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:54 smithi196 ceph-mon[33018]: 5.d scrub ok 2024-02-05T08:21:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:54 smithi196 ceph-mon[33018]: 5.10 deep-scrub starts 2024-02-05T08:21:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:54 smithi196 ceph-mon[33018]: 5.10 deep-scrub ok 2024-02-05T08:21:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:54 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:21:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:54 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:21:55.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:54 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 10569/170538 objects degraded (6.197%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:21:55.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:55.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:55.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:55.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:55.455 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:55.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:55.488 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:55.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:55.589 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:55.631 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:55.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:55.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:55.767 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:55.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:55.863 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:21:55.866 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:21:55.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:21:55.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:55.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:55.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:21:56.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:56.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:56.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:56.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:56.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:56.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:56.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:56.245 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:55 smithi134 ceph-mon[30086]: 4.15 scrub starts 2024-02-05T08:21:56.245 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:55 smithi134 ceph-mon[30086]: 4.15 scrub ok 2024-02-05T08:21:56.245 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:55 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:21:56.245 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:55 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:21:56.245 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:55 smithi134 ceph-mon[30086]: 10.f scrub ok 2024-02-05T08:21:56.245 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:55 smithi134 ceph-mon[30086]: osdmap e348: 8 total, 7 up, 8 in 2024-02-05T08:21:56.246 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:55 smithi134 ceph-mon[39278]: 4.15 scrub starts 2024-02-05T08:21:56.246 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:55 smithi134 ceph-mon[39278]: 4.15 scrub ok 2024-02-05T08:21:56.246 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:55 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:21:56.246 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:55 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:21:56.246 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:55 smithi134 ceph-mon[39278]: 10.f scrub ok 2024-02-05T08:21:56.246 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:55 smithi134 ceph-mon[39278]: osdmap e348: 8 total, 7 up, 8 in 2024-02-05T08:21:56.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:55 smithi196 ceph-mon[33018]: 4.15 scrub starts 2024-02-05T08:21:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:55 smithi196 ceph-mon[33018]: 4.15 scrub ok 2024-02-05T08:21:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:55 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:21:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:55 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:21:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:55 smithi196 ceph-mon[33018]: 10.f scrub ok 2024-02-05T08:21:56.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:55 smithi196 ceph-mon[33018]: osdmap e348: 8 total, 7 up, 8 in 2024-02-05T08:21:56.330 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:56.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:56.431 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:56.466 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:56.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:56.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:56.603 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:56.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:56.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:56.728 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:21:56.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:56.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:56.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:56.750 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":765,"stamp":"2024-02-05T08:21:54.974512+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3660049178,"num_objects":56037,"num_object_clones":0,"num_object_copies":168111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10569,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56037,"num_whiteouts":0,"num_read":2177,"num_read_kb":1937,"num_write":207820,"num_write_kb":8413642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":221172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86388,"ondisk_log_size":86388,"up":417,"acting":417,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":475,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15416244,"kb_used_data":10773908,"kb_used_omap":12,"kb_used_meta":4641907,"kb_avail":734544972,"statfs":{"total":767960285184,"available":752174051328,"internally_reserved":0,"allocated":11032481792,"data_stored":11029896365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":4753313331},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-738459648,"num_objects":-11268,"num_object_clones":0,"num_object_copies":-33804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2036,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11268,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11268,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.066363"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":621,"reported_epoch":346,"state":"active+undersized","last_fresh":"2024-02-05T08:21:51.954700+0000","last_change":"2024-02-05T08:19:44.912725+0000","last_active":"2024-02-05T08:21:51.954700+0000","last_peered":"2024-02-05T08:21:51.954700+0000","last_clean":"2024-02-05T08:19:41.829565+0000","last_became_active":"2024-02-05T08:19:44.912725+0000","last_became_peered":"2024-02-05T08:19:44.912725+0000","last_unstale":"2024-02-05T08:21:51.954700+0000","last_undegraded":"2024-02-05T08:21:51.954700+0000","last_fullsized":"2024-02-05T08:19:44.874568+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:36.296981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:36.296981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.909999+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":54,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.970090+0000","last_change":"2024-02-05T08:19:44.903097+0000","last_active":"2024-02-05T08:21:52.970090+0000","last_peered":"2024-02-05T08:21:52.970090+0000","last_clean":"2024-02-05T08:04:44.536163+0000","last_became_active":"2024-02-05T08:19:44.903097+0000","last_became_peered":"2024-02-05T08:19:44.903097+0000","last_unstale":"2024-02-05T08:21:52.970090+0000","last_undegraded":"2024-02-05T08:21:52.970090+0000","last_fullsized":"2024-02-05T08:19:44.873443+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.301502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.301502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":633,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.959145+0000","last_change":"2024-02-05T08:21:18.141386+0000","last_active":"2024-02-05T08:21:52.959145+0000","last_peered":"2024-02-05T08:21:52.959145+0000","last_clean":"2024-02-05T08:21:52.959145+0000","last_became_active":"2024-02-05T08:04:41.536416+0000","last_became_peered":"2024-02-05T08:04:41.536416+0000","last_unstale":"2024-02-05T08:21:52.959145+0000","last_undegraded":"2024-02-05T08:21:52.959145+0000","last_fullsized":"2024-02-05T08:21:52.959145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:18.141284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:18.141284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:22.319759+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"236'19","reported_seq":683,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:51.954464+0000","last_change":"2024-02-05T08:19:44.920275+0000","last_active":"2024-02-05T08:21:51.954464+0000","last_peered":"2024-02-05T08:21:51.954464+0000","last_clean":"2024-02-05T08:19:41.830039+0000","last_became_active":"2024-02-05T08:19:44.920275+0000","last_became_peered":"2024-02-05T08:19:44.920275+0000","last_unstale":"2024-02-05T08:21:51.954464+0000","last_undegraded":"2024-02-05T08:19:44.874580+0000","last_fullsized":"2024-02-05T08:19:44.874470+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:01.602887+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:18:01.602887+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.909918+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":132,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.967720+0000","last_change":"2024-02-05T08:21:51.007314+0000","last_active":"2024-02-05T08:21:52.967720+0000","last_peered":"2024-02-05T08:21:52.967720+0000","last_clean":"2024-02-05T08:21:52.967720+0000","last_became_active":"2024-02-05T08:18:34.633083+0000","last_became_peered":"2024-02-05T08:18:34.633083+0000","last_unstale":"2024-02-05T08:21:52.967720+0000","last_undegraded":"2024-02-05T08:21:52.967720+0000","last_fullsized":"2024-02-05T08:21:52.967720+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:51.006901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:21:51.006901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:16.387843+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":630,"reported_epoch":346,"state":"active+undersized","last_fresh":"2024-02-05T08:21:51.955321+0000","last_change":"2024-02-05T08:19:44.919417+0000","last_active":"2024-02-05T08:21:51.955321+0000","last_peered":"2024-02-05T08:21:51.955321+0000","last_clean":"2024-02-05T08:19:41.831283+0000","last_became_active":"2024-02-05T08:19:44.919417+0000","last_became_peered":"2024-02-05T08:19:44.919417+0000","last_unstale":"2024-02-05T08:21:51.955321+0000","last_undegraded":"2024-02-05T08:21:51.955321+0000","last_fullsized":"2024-02-05T08:19:44.874665+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:39.536022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:18:39.536022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.909961+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":141,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.967300+0000","last_change":"2024-02-05T08:21:45.823410+0000","last_active":"2024-02-05T08:21:52.967300+0000","last_peered":"2024-02-05T08:21:52.967300+0000","last_clean":"2024-02-05T08:21:52.967300+0000","last_became_active":"2024-02-05T08:21:41.880038+0000","last_became_peered":"2024-02-05T08:21:41.880038+0000","last_unstale":"2024-02-05T08:21:52.967300+0000","last_undegraded":"2024-02-05T08:21:52.967300+0000","last_fullsized":"2024-02-05T08:21:52.967300+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:45.823300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:21:45.823300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:05.550303+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"236'34","reported_seq":148,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.966962+0000","last_change":"2024-02-05T08:20:29.672533+0000","last_active":"2024-02-05T08:21:52.966962+0000","last_peered":"2024-02-05T08:21:52.966962+0000","last_clean":"2024-02-05T08:21:52.966962+0000","last_became_active":"2024-02-05T08:18:34.649433+0000","last_became_peered":"2024-02-05T08:18:34.649433+0000","last_unstale":"2024-02-05T08:21:52.966962+0000","last_undegraded":"2024-02-05T08:21:52.966962+0000","last_fullsized":"2024-02-05T08:21:52.966962+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:20:29.672381+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:20:29.672381+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":615,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.959537+0000","last_change":"2024-02-05T08:19:44.902206+0000","last_active":"2024-02-05T08:21:52.959537+0000","last_peered":"2024-02-05T08:21:52.959537+0000","last_clean":"2024-02-05T08:19:41.834390+0000","last_became_active":"2024-02-05T08:19:44.902206+0000","last_became_peered":"2024-02-05T08:19:44.902206+0000","last_unstale":"2024-02-05T08:21:52.959537+0000","last_undegraded":"2024-02-05T08:21:52.959537+0000","last_fullsized":"2024-02-05T08:19:44.885918+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:21.661930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:21.661930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:54.230291+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":632,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.973728+0000","last_change":"2024-02-05T08:21:18.395522+0000","last_active":"2024-02-05T08:21:52.973728+0000","last_peered":"2024-02-05T08:21:52.973728+0000","last_clean":"2024-02-05T08:21:52.973728+0000","last_became_active":"2024-02-05T08:04:45.560008+0000","last_became_peered":"2024-02-05T08:04:45.560008+0000","last_unstale":"2024-02-05T08:21:52.973728+0000","last_undegraded":"2024-02-05T08:21:52.973728+0000","last_fullsized":"2024-02-05T08:21:52.973728+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:18.395414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:21:18.395414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:45.950771+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":660,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.973150+0000","last_change":"2024-02-05T08:21:37.323987+0000","last_active":"2024-02-05T08:21:52.973150+0000","last_peered":"2024-02-05T08:21:52.973150+0000","last_clean":"2024-02-05T08:21:52.973150+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:21:52.973150+0000","last_undegraded":"2024-02-05T08:21:52.973150+0000","last_fullsized":"2024-02-05T08:21:52.973150+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:37.323869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_clean_scrub_stamp":"2024-02-05T08:21:37.323869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:46.746148+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"263'19","reported_seq":701,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:51.954860+0000","last_change":"2024-02-05T08:21:43.861190+0000","last_active":"2024-02-05T08:21:51.954860+0000","last_peered":"2024-02-05T08:21:51.954860+0000","last_clean":"2024-02-05T08:21:51.954860+0000","last_became_active":"2024-02-05T08:18:34.636165+0000","last_became_peered":"2024-02-05T08:18:34.636165+0000","last_unstale":"2024-02-05T08:21:51.954860+0000","last_undegraded":"2024-02-05T08:21:51.954860+0000","last_fullsized":"2024-02-05T08:21:51.954860+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:21:43.861060+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:21:43.861060+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:11.505620+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":628,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.972202+0000","last_change":"2024-02-05T08:19:44.901170+0000","last_active":"2024-02-05T08:21:52.972202+0000","last_peered":"2024-02-05T08:21:52.972202+0000","last_clean":"2024-02-05T08:19:41.833929+0000","last_became_active":"2024-02-05T08:19:44.901170+0000","last_became_peered":"2024-02-05T08:19:44.901170+0000","last_unstale":"2024-02-05T08:21:52.972202+0000","last_undegraded":"2024-02-05T08:21:52.972202+0000","last_fullsized":"2024-02-05T08:19:44.873575+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:32.611856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:32.611856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":628,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.961402+0000","last_change":"2024-02-05T08:19:44.921416+0000","last_active":"2024-02-05T08:21:52.961402+0000","last_peered":"2024-02-05T08:21:52.961402+0000","last_clean":"2024-02-05T08:19:41.834434+0000","last_became_active":"2024-02-05T08:19:44.921416+0000","last_became_peered":"2024-02-05T08:19:44.921416+0000","last_unstale":"2024-02-05T08:21:52.961402+0000","last_undegraded":"2024-02-05T08:21:52.961402+0000","last_fullsized":"2024-02-05T08:19:44.875209+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:40.260248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:40.260248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":635,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.973855+0000","last_change":"2024-02-05T08:19:44.916712+0000","last_active":"2024-02-05T08:21:52.973855+0000","last_peered":"2024-02-05T08:21:52.973855+0000","last_clean":"2024-02-05T08:19:41.831063+0000","last_became_active":"2024-02-05T08:19:44.916712+0000","last_became_peered":"2024-02-05T08:19:44.916712+0000","last_unstale":"2024-02-05T08:21:52.973855+0000","last_undegraded":"2024-02-05T08:21:52.973855+0000","last_fullsized":"2024-02-05T08:19:44.898786+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:14.458198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.458198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":658,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:51.954996+0000","last_change":"2024-02-05T08:19:44.913761+0000","last_active":"2024-02-05T08:21:51.954996+0000","last_peered":"2024-02-05T08:21:51.954996+0000","last_clean":"2024-02-05T08:19:42.900179+0000","last_became_active":"2024-02-05T08:19:44.913761+0000","last_became_peered":"2024-02-05T08:19:44.913761+0000","last_unstale":"2024-02-05T08:21:51.954996+0000","last_undegraded":"2024-02-05T08:19:44.909323+0000","last_fullsized":"2024-02-05T08:19:44.909205+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-02-05T08:18:05.582303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:05.582303+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.909928+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":640,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:51.956531+0000","last_change":"2024-02-05T08:21:42.867663+0000","last_active":"2024-02-05T08:21:51.956531+0000","last_peered":"2024-02-05T08:21:51.956531+0000","last_clean":"2024-02-05T08:21:51.956531+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:21:51.956531+0000","last_undegraded":"2024-02-05T08:21:51.956531+0000","last_fullsized":"2024-02-05T08:21:51.956531+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:42.867555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:21:42.867555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:53.230473+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":52,"reported_epoch":346,"state":"active+undersized","last_fresh":"2024-02-05T08:21:51.955824+0000","last_change":"2024-02-05T08:19:44.920889+0000","last_active":"2024-02-05T08:21:51.955824+0000","last_peered":"2024-02-05T08:21:51.955824+0000","last_clean":"2024-02-05T08:04:44.536163+0000","last_became_active":"2024-02-05T08:19:44.920889+0000","last_became_peered":"2024-02-05T08:19:44.920889+0000","last_unstale":"2024-02-05T08:21:51.955824+0000","last_undegraded":"2024-02-05T08:21:51.955824+0000","last_fullsized":"2024-02-05T08:19:44.913274+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_clean_scrub_stamp":"2024-02-05T08:19:36.737342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.909992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":623,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.960600+0000","last_change":"2024-02-05T08:20:31.566305+0000","last_active":"2024-02-05T08:21:52.960600+0000","last_peered":"2024-02-05T08:21:52.960600+0000","last_clean":"2024-02-05T08:21:52.960600+0000","last_became_active":"2024-02-05T08:04:41.541551+0000","last_became_peered":"2024-02-05T08:04:41.541551+0000","last_unstale":"2024-02-05T08:21:52.960600+0000","last_undegraded":"2024-02-05T08:21:52.960600+0000","last_fullsized":"2024-02-05T08:21:52.960600+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:31.566207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:31.566207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"236'34","reported_seq":741,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.961914+0000","last_change":"2024-02-05T08:19:44.892882+0000","last_active":"2024-02-05T08:21:52.961914+0000","last_peered":"2024-02-05T08:21:52.961914+0000","last_clean":"2024-02-05T08:19:41.827204+0000","last_became_active":"2024-02-05T08:19:44.892882+0000","last_became_peered":"2024-02-05T08:19:44.892882+0000","last_unstale":"2024-02-05T08:21:52.961914+0000","last_undegraded":"2024-02-05T08:19:44.885361+0000","last_fullsized":"2024-02-05T08:19:44.885220+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:18:47.576225+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:18:47.576225+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":124,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.970136+0000","last_change":"2024-02-05T08:20:16.870703+0000","last_active":"2024-02-05T08:21:52.970136+0000","last_peered":"2024-02-05T08:21:52.970136+0000","last_clean":"2024-02-05T08:21:52.970136+0000","last_became_active":"2024-02-05T08:18:34.642494+0000","last_became_peered":"2024-02-05T08:18:34.642494+0000","last_unstale":"2024-02-05T08:21:52.970136+0000","last_undegraded":"2024-02-05T08:21:52.970136+0000","last_fullsized":"2024-02-05T08:21:52.970136+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:16.870561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_clean_scrub_stamp":"2024-02-05T08:20:16.870561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":642,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:51.955983+0000","last_change":"2024-02-05T08:21:07.003807+0000","last_active":"2024-02-05T08:21:51.955983+0000","last_peered":"2024-02-05T08:21:51.955983+0000","last_clean":"2024-02-05T08:21:51.955983+0000","last_became_active":"2024-02-05T08:18:34.638652+0000","last_became_peered":"2024-02-05T08:18:34.638652+0000","last_unstale":"2024-02-05T08:21:51.955983+0000","last_undegraded":"2024-02-05T08:21:51.955983+0000","last_fullsized":"2024-02-05T08:21:51.955983+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:07.003702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:21:07.003702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:14.547409+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":625,"reported_epoch":346,"state":"active+undersized","last_fresh":"2024-02-05T08:21:51.954950+0000","last_change":"2024-02-05T08:19:44.912456+0000","last_active":"2024-02-05T08:21:51.954950+0000","last_peered":"2024-02-05T08:21:51.954950+0000","last_clean":"2024-02-05T08:19:42.900163+0000","last_became_active":"2024-02-05T08:19:44.912456+0000","last_became_peered":"2024-02-05T08:19:44.912456+0000","last_unstale":"2024-02-05T08:21:51.954950+0000","last_undegraded":"2024-02-05T08:21:51.954950+0000","last_fullsized":"2024-02-05T08:19:44.874216+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:19.338699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:19:19.338699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.909985+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"236'23","reported_seq":131,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.967093+0000","last_change":"2024-02-05T08:19:44.923092+0000","last_active":"2024-02-05T08:21:52.967093+0000","last_peered":"2024-02-05T08:21:52.967093+0000","last_clean":"2024-02-05T08:19:41.835633+0000","last_became_active":"2024-02-05T08:19:44.923092+0000","last_became_peered":"2024-02-05T08:19:44.923092+0000","last_unstale":"2024-02-05T08:21:52.967093+0000","last_undegraded":"2024-02-05T08:19:44.886225+0000","last_fullsized":"2024-02-05T08:19:44.886101+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'23","last_scrub_stamp":"2024-02-05T08:17:58.012827+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:17:58.012827+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.878326+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":639,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.964841+0000","last_change":"2024-02-05T08:20:37.119269+0000","last_active":"2024-02-05T08:21:52.964841+0000","last_peered":"2024-02-05T08:21:52.964841+0000","last_clean":"2024-02-05T08:21:52.964841+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:21:52.964841+0000","last_undegraded":"2024-02-05T08:21:52.964841+0000","last_fullsized":"2024-02-05T08:21:52.964841+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:20:37.119142+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.119142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":496,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.960503+0000","last_change":"2024-02-05T08:21:46.613709+0000","last_active":"2024-02-05T08:21:52.960503+0000","last_peered":"2024-02-05T08:21:52.960503+0000","last_clean":"2024-02-05T08:21:52.960503+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:21:52.960503+0000","last_undegraded":"2024-02-05T08:21:52.960503+0000","last_fullsized":"2024-02-05T08:21:52.960503+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:21:46.613469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:21:46.613469+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:47.607944+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":581,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.966490+0000","last_change":"2024-02-05T08:19:44.915101+0000","last_active":"2024-02-05T08:21:52.966490+0000","last_peered":"2024-02-05T08:21:52.966490+0000","last_clean":"2024-02-05T08:18:29.609487+0000","last_became_active":"2024-02-05T08:19:44.915101+0000","last_became_peered":"2024-02-05T08:19:44.915101+0000","last_unstale":"2024-02-05T08:21:52.966490+0000","last_undegraded":"2024-02-05T08:19:44.874435+0000","last_fullsized":"2024-02-05T08:19:44.874324+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:19:28.743054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:28.743054+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"236'20","reported_seq":731,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.960700+0000","last_change":"2024-02-05T08:21:49.240508+0000","last_active":"2024-02-05T08:21:52.960700+0000","last_peered":"2024-02-05T08:21:52.960700+0000","last_clean":"2024-02-05T08:21:52.960700+0000","last_became_active":"2024-02-05T08:18:34.645441+0000","last_became_peered":"2024-02-05T08:18:34.645441+0000","last_unstale":"2024-02-05T08:21:52.960700+0000","last_undegraded":"2024-02-05T08:21:52.960700+0000","last_fullsized":"2024-02-05T08:21:52.960700+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:21:49.240369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:49.240369+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:06.167971+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":53,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.972806+0000","last_change":"2024-02-05T08:19:44.923094+0000","last_active":"2024-02-05T08:21:52.972806+0000","last_peered":"2024-02-05T08:21:52.972806+0000","last_clean":"2024-02-05T08:04:46.546354+0000","last_became_active":"2024-02-05T08:19:44.923094+0000","last_became_peered":"2024-02-05T08:19:44.923094+0000","last_unstale":"2024-02-05T08:21:52.972806+0000","last_undegraded":"2024-02-05T08:21:52.972806+0000","last_fullsized":"2024-02-05T08:19:44.897835+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:46.550985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:46.550985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":625,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.961037+0000","last_change":"2024-02-05T08:21:19.107687+0000","last_active":"2024-02-05T08:21:52.961037+0000","last_peered":"2024-02-05T08:21:52.961037+0000","last_clean":"2024-02-05T08:21:52.961037+0000","last_became_active":"2024-02-05T08:04:45.564463+0000","last_became_peered":"2024-02-05T08:04:45.564463+0000","last_unstale":"2024-02-05T08:21:52.961037+0000","last_undegraded":"2024-02-05T08:21:52.961037+0000","last_fullsized":"2024-02-05T08:21:52.961037+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:19.107555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:19.107555+0000","last_clean_scrub_stamp":"2024-02-05T08:21:19.107555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:46.449485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":127,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.966536+0000","last_change":"2024-02-05T08:19:44.892769+0000","last_active":"2024-02-05T08:21:52.966536+0000","last_peered":"2024-02-05T08:21:52.966536+0000","last_clean":"2024-02-05T08:19:41.827910+0000","last_became_active":"2024-02-05T08:19:44.892769+0000","last_became_peered":"2024-02-05T08:19:44.892769+0000","last_unstale":"2024-02-05T08:21:52.966536+0000","last_undegraded":"2024-02-05T08:21:52.966536+0000","last_fullsized":"2024-02-05T08:19:44.874377+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:11.049460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:19:11.049460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"236'19","reported_seq":395,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.960349+0000","last_change":"2024-02-05T08:21:09.755205+0000","last_active":"2024-02-05T08:21:52.960349+0000","last_peered":"2024-02-05T08:21:52.960349+0000","last_clean":"2024-02-05T08:21:52.960349+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:21:52.960349+0000","last_undegraded":"2024-02-05T08:21:52.960349+0000","last_fullsized":"2024-02-05T08:21:52.960349+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:21:09.755115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:09.755115+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:38.649362+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":622,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.961650+0000","last_change":"2024-02-05T08:21:14.159752+0000","last_active":"2024-02-05T08:21:52.961650+0000","last_peered":"2024-02-05T08:21:52.961650+0000","last_clean":"2024-02-05T08:21:52.961650+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:21:52.961650+0000","last_undegraded":"2024-02-05T08:21:52.961650+0000","last_fullsized":"2024-02-05T08:21:52.961650+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:14.159628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:14.159628+0000","last_clean_scrub_stamp":"2024-02-05T08:21:14.159628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:30.209411+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":374,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.957805+0000","last_change":"2024-02-05T08:21:50.638160+0000","last_active":"2024-02-05T08:21:52.957805+0000","last_peered":"2024-02-05T08:21:52.957805+0000","last_clean":"2024-02-05T08:21:52.957805+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:21:52.957805+0000","last_undegraded":"2024-02-05T08:21:52.957805+0000","last_fullsized":"2024-02-05T08:21:52.957805+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:50.638065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:21:50.638065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:10.341134+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":644,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.959441+0000","last_change":"2024-02-05T08:21:38.069833+0000","last_active":"2024-02-05T08:21:52.959441+0000","last_peered":"2024-02-05T08:21:52.959441+0000","last_clean":"2024-02-05T08:21:52.959441+0000","last_became_active":"2024-02-05T08:18:34.644537+0000","last_became_peered":"2024-02-05T08:18:34.644537+0000","last_unstale":"2024-02-05T08:21:52.959441+0000","last_undegraded":"2024-02-05T08:21:52.959441+0000","last_fullsized":"2024-02-05T08:21:52.959441+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:38.069723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:34.697248+0000","last_clean_scrub_stamp":"2024-02-05T08:21:38.069723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:45.195388+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":133,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.966970+0000","last_change":"2024-02-05T08:19:44.911502+0000","last_active":"2024-02-05T08:21:52.966970+0000","last_peered":"2024-02-05T08:21:52.966970+0000","last_clean":"2024-02-05T08:19:41.828151+0000","last_became_active":"2024-02-05T08:19:44.911502+0000","last_became_peered":"2024-02-05T08:19:44.911502+0000","last_unstale":"2024-02-05T08:21:52.966970+0000","last_undegraded":"2024-02-05T08:19:44.874110+0000","last_fullsized":"2024-02-05T08:19:44.873994+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:19:23.023541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:23.023541+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":132,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.967549+0000","last_change":"2024-02-05T08:20:26.891410+0000","last_active":"2024-02-05T08:21:52.967549+0000","last_peered":"2024-02-05T08:21:52.967549+0000","last_clean":"2024-02-05T08:21:52.967549+0000","last_became_active":"2024-02-05T08:18:34.651979+0000","last_became_peered":"2024-02-05T08:18:34.651979+0000","last_unstale":"2024-02-05T08:21:52.967549+0000","last_undegraded":"2024-02-05T08:21:52.967549+0000","last_fullsized":"2024-02-05T08:21:52.967549+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:26.891040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:26.891040+0000","last_clean_scrub_stamp":"2024-02-05T08:20:26.891040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":646,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.964072+0000","last_change":"2024-02-05T08:20:18.147831+0000","last_active":"2024-02-05T08:21:52.964072+0000","last_peered":"2024-02-05T08:21:52.964072+0000","last_clean":"2024-02-05T08:21:52.964072+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:21:52.964072+0000","last_undegraded":"2024-02-05T08:21:52.964072+0000","last_fullsized":"2024-02-05T08:21:52.964072+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:18.147762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:36.353351+0000","last_clean_scrub_stamp":"2024-02-05T08:20:18.147762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":636,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.973621+0000","last_change":"2024-02-05T08:19:44.904157+0000","last_active":"2024-02-05T08:21:52.973621+0000","last_peered":"2024-02-05T08:21:52.973621+0000","last_clean":"2024-02-05T08:19:41.832352+0000","last_became_active":"2024-02-05T08:19:44.904157+0000","last_became_peered":"2024-02-05T08:19:44.904157+0000","last_unstale":"2024-02-05T08:21:52.973621+0000","last_undegraded":"2024-02-05T08:21:52.973621+0000","last_fullsized":"2024-02-05T08:19:44.901922+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:41.576882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:18:41.576882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"236'24","reported_seq":136,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.967419+0000","last_change":"2024-02-05T08:20:11.675925+0000","last_active":"2024-02-05T08:21:52.967419+0000","last_peered":"2024-02-05T08:21:52.967419+0000","last_clean":"2024-02-05T08:21:52.967419+0000","last_became_active":"2024-02-05T08:18:34.631912+0000","last_became_peered":"2024-02-05T08:18:34.631912+0000","last_unstale":"2024-02-05T08:21:52.967419+0000","last_undegraded":"2024-02-05T08:21:52.967419+0000","last_fullsized":"2024-02-05T08:21:52.967419+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:20:11.675780+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:20:11.675780+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":615,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.959032+0000","last_change":"2024-02-05T08:19:44.902556+0000","last_active":"2024-02-05T08:21:52.959032+0000","last_peered":"2024-02-05T08:21:52.959032+0000","last_clean":"2024-02-05T08:19:41.834123+0000","last_became_active":"2024-02-05T08:19:44.902556+0000","last_became_peered":"2024-02-05T08:19:44.902556+0000","last_unstale":"2024-02-05T08:21:52.959032+0000","last_undegraded":"2024-02-05T08:21:52.959032+0000","last_fullsized":"2024-02-05T08:19:44.874049+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:26.593364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:19:26.593364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:54.230329+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":127,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.967521+0000","last_change":"2024-02-05T08:19:44.892864+0000","last_active":"2024-02-05T08:21:52.967521+0000","last_peered":"2024-02-05T08:21:52.967521+0000","last_clean":"2024-02-05T08:19:41.831557+0000","last_became_active":"2024-02-05T08:19:44.892864+0000","last_became_peered":"2024-02-05T08:19:44.892864+0000","last_unstale":"2024-02-05T08:21:52.967521+0000","last_undegraded":"2024-02-05T08:21:52.967521+0000","last_fullsized":"2024-02-05T08:19:44.886732+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:44.133926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:44.133926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":654,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:53.174387+0000","last_change":"2024-02-05T08:21:53.174425+0000","last_active":"2024-02-05T08:21:53.174387+0000","last_peered":"2024-02-05T08:21:53.174387+0000","last_clean":"2024-02-05T08:21:53.174387+0000","last_became_active":"2024-02-05T08:18:34.642387+0000","last_became_peered":"2024-02-05T08:18:34.642387+0000","last_unstale":"2024-02-05T08:21:53.174387+0000","last_undegraded":"2024-02-05T08:21:53.174387+0000","last_fullsized":"2024-02-05T08:21:53.174387+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:53.173888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:53.173888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:58.513833+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"263'19","reported_seq":309,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:53.426955+0000","last_change":"2024-02-05T08:21:53.427022+0000","last_active":"2024-02-05T08:21:53.426955+0000","last_peered":"2024-02-05T08:21:53.426955+0000","last_clean":"2024-02-05T08:21:53.426955+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:21:53.426955+0000","last_undegraded":"2024-02-05T08:21:53.426955+0000","last_fullsized":"2024-02-05T08:21:53.426955+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:21:53.422885+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_clean_scrub_stamp":"2024-02-05T08:21:53.422885+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:14.476216+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":140,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.967121+0000","last_change":"2024-02-05T08:21:46.884029+0000","last_active":"2024-02-05T08:21:52.967121+0000","last_peered":"2024-02-05T08:21:52.967121+0000","last_clean":"2024-02-05T08:21:52.967121+0000","last_became_active":"2024-02-05T08:18:34.650359+0000","last_became_peered":"2024-02-05T08:18:34.650359+0000","last_unstale":"2024-02-05T08:21:52.967121+0000","last_undegraded":"2024-02-05T08:21:52.967121+0000","last_fullsized":"2024-02-05T08:21:52.967121+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:46.883886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:21:46.883886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:58.515954+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":568,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.973336+0000","last_change":"2024-02-05T08:19:44.903986+0000","last_active":"2024-02-05T08:21:52.973336+0000","last_peered":"2024-02-05T08:21:52.973336+0000","last_clean":"2024-02-05T08:18:29.613784+0000","last_became_active":"2024-02-05T08:19:44.903986+0000","last_became_peered":"2024-02-05T08:19:44.903986+0000","last_unstale":"2024-02-05T08:21:52.973336+0000","last_undegraded":"2024-02-05T08:21:52.973336+0000","last_fullsized":"2024-02-05T08:19:44.899948+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:17.633650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.633650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":659,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:51.955548+0000","last_change":"2024-02-05T08:21:44.871777+0000","last_active":"2024-02-05T08:21:51.955548+0000","last_peered":"2024-02-05T08:21:51.955548+0000","last_clean":"2024-02-05T08:21:51.955548+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:21:51.955548+0000","last_undegraded":"2024-02-05T08:21:51.955548+0000","last_fullsized":"2024-02-05T08:21:51.955548+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:44.871668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:21:44.871668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:04.896469+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"249'24","reported_seq":702,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:51.955230+0000","last_change":"2024-02-05T08:19:44.930725+0000","last_active":"2024-02-05T08:21:51.955230+0000","last_peered":"2024-02-05T08:21:51.955230+0000","last_clean":"2024-02-05T08:19:41.830358+0000","last_became_active":"2024-02-05T08:19:44.930725+0000","last_became_peered":"2024-02-05T08:19:44.930725+0000","last_unstale":"2024-02-05T08:21:51.955230+0000","last_undegraded":"2024-02-05T08:19:44.912297+0000","last_fullsized":"2024-02-05T08:19:44.910005+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:19:27.349541+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:19:27.349541+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.910008+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":624,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.960255+0000","last_change":"2024-02-05T08:20:29.732543+0000","last_active":"2024-02-05T08:21:52.960255+0000","last_peered":"2024-02-05T08:21:52.960255+0000","last_clean":"2024-02-05T08:21:52.960255+0000","last_became_active":"2024-02-05T08:18:34.645027+0000","last_became_peered":"2024-02-05T08:18:34.645027+0000","last_unstale":"2024-02-05T08:21:52.960255+0000","last_undegraded":"2024-02-05T08:21:52.960255+0000","last_fullsized":"2024-02-05T08:21:52.960255+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:29.732422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:29.732422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:58.490572+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"325'9","reported_seq":844,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.973231+0000","last_change":"2024-02-05T08:19:44.900658+0000","last_active":"2024-02-05T08:21:52.973231+0000","last_peered":"2024-02-05T08:21:52.973231+0000","last_clean":"2024-02-05T08:19:41.831639+0000","last_became_active":"2024-02-05T08:19:44.900658+0000","last_became_peered":"2024-02-05T08:19:44.900658+0000","last_unstale":"2024-02-05T08:21:52.973231+0000","last_undegraded":"2024-02-05T08:19:44.898660+0000","last_fullsized":"2024-02-05T08:19:44.898552+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:19:12.452237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:12.452237+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[{"shards":"0,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":642,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.961813+0000","last_change":"2024-02-05T08:21:09.098128+0000","last_active":"2024-02-05T08:21:52.961813+0000","last_peered":"2024-02-05T08:21:52.961813+0000","last_clean":"2024-02-05T08:21:52.961813+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:21:52.961813+0000","last_undegraded":"2024-02-05T08:21:52.961813+0000","last_fullsized":"2024-02-05T08:21:52.961813+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:09.098005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:09.098005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:33.909120+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":139,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.968083+0000","last_change":"2024-02-05T08:19:44.899031+0000","last_active":"2024-02-05T08:21:52.968083+0000","last_peered":"2024-02-05T08:21:52.968083+0000","last_clean":"2024-02-05T08:19:41.830913+0000","last_became_active":"2024-02-05T08:19:44.899031+0000","last_became_peered":"2024-02-05T08:19:44.899031+0000","last_unstale":"2024-02-05T08:21:52.968083+0000","last_undegraded":"2024-02-05T08:19:44.887736+0000","last_fullsized":"2024-02-05T08:19:44.887622+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-02-05T08:19:35.981313+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:19:35.981313+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":622,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:51.955963+0000","last_change":"2024-02-05T08:21:10.125938+0000","last_active":"2024-02-05T08:21:51.955963+0000","last_peered":"2024-02-05T08:21:51.955963+0000","last_clean":"2024-02-05T08:21:51.955963+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:21:51.955963+0000","last_undegraded":"2024-02-05T08:21:51.955963+0000","last_fullsized":"2024-02-05T08:21:51.955963+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:10.125818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:28.519149+0000","last_clean_scrub_stamp":"2024-02-05T08:21:10.125818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:30.234304+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"295'10","reported_seq":169,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.968289+0000","last_change":"2024-02-05T08:20:14.713693+0000","last_active":"2024-02-05T08:21:52.968289+0000","last_peered":"2024-02-05T08:21:52.968289+0000","last_clean":"2024-02-05T08:21:52.968289+0000","last_became_active":"2024-02-05T08:18:34.644517+0000","last_became_peered":"2024-02-05T08:18:34.644517+0000","last_unstale":"2024-02-05T08:21:52.968289+0000","last_undegraded":"2024-02-05T08:21:52.968289+0000","last_fullsized":"2024-02-05T08:21:52.968289+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'10","last_scrub_stamp":"2024-02-05T08:20:14.713589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:14.713589+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":644,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.961175+0000","last_change":"2024-02-05T08:21:03.366506+0000","last_active":"2024-02-05T08:21:52.961175+0000","last_peered":"2024-02-05T08:21:52.961175+0000","last_clean":"2024-02-05T08:21:52.961175+0000","last_became_active":"2024-02-05T08:18:34.635793+0000","last_became_peered":"2024-02-05T08:18:34.635793+0000","last_unstale":"2024-02-05T08:21:52.961175+0000","last_undegraded":"2024-02-05T08:21:52.961175+0000","last_fullsized":"2024-02-05T08:21:52.961175+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:03.366172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:21:03.366172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:10.583764+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":694,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.973544+0000","last_change":"2024-02-05T08:20:23.330119+0000","last_active":"2024-02-05T08:21:52.973544+0000","last_peered":"2024-02-05T08:21:52.973544+0000","last_clean":"2024-02-05T08:21:52.973544+0000","last_became_active":"2024-02-05T08:18:34.640741+0000","last_became_peered":"2024-02-05T08:18:34.640741+0000","last_unstale":"2024-02-05T08:21:52.973544+0000","last_undegraded":"2024-02-05T08:21:52.973544+0000","last_fullsized":"2024-02-05T08:21:52.973544+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:20:23.329973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:20:23.329973+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":645,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.962329+0000","last_change":"2024-02-05T08:21:40.074672+0000","last_active":"2024-02-05T08:21:52.962329+0000","last_peered":"2024-02-05T08:21:52.962329+0000","last_clean":"2024-02-05T08:21:52.962329+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:21:52.962329+0000","last_undegraded":"2024-02-05T08:21:52.962329+0000","last_fullsized":"2024-02-05T08:21:52.962329+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:40.074544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:40.074544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:40.581670+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":127,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.969986+0000","last_change":"2024-02-05T08:19:44.899825+0000","last_active":"2024-02-05T08:21:52.969986+0000","last_peered":"2024-02-05T08:21:52.969986+0000","last_clean":"2024-02-05T08:19:41.832250+0000","last_became_active":"2024-02-05T08:19:44.899825+0000","last_became_peered":"2024-02-05T08:19:44.899825+0000","last_unstale":"2024-02-05T08:21:52.969986+0000","last_undegraded":"2024-02-05T08:21:52.969986+0000","last_fullsized":"2024-02-05T08:19:44.888998+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:37.028991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:37.028991+0000","last_clean_scrub_stamp":"2024-02-05T08:19:37.028991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":646,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.961458+0000","last_change":"2024-02-05T08:20:17.150031+0000","last_active":"2024-02-05T08:21:52.961458+0000","last_peered":"2024-02-05T08:21:52.961458+0000","last_clean":"2024-02-05T08:21:52.961458+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:21:52.961458+0000","last_undegraded":"2024-02-05T08:21:52.961458+0000","last_fullsized":"2024-02-05T08:21:52.961458+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:17.149915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:20:17.149915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"278'19","reported_seq":689,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.960812+0000","last_change":"2024-02-05T08:21:50.244248+0000","last_active":"2024-02-05T08:21:52.960812+0000","last_peered":"2024-02-05T08:21:52.960812+0000","last_clean":"2024-02-05T08:21:52.960812+0000","last_became_active":"2024-02-05T08:18:34.645881+0000","last_became_peered":"2024-02-05T08:18:34.645881+0000","last_unstale":"2024-02-05T08:21:52.960812+0000","last_undegraded":"2024-02-05T08:21:52.960812+0000","last_fullsized":"2024-02-05T08:21:52.960812+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-05T08:21:50.244148+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:21:50.244148+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:57.308560+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":642,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.962291+0000","last_change":"2024-02-05T08:21:37.234586+0000","last_active":"2024-02-05T08:21:52.962291+0000","last_peered":"2024-02-05T08:21:52.962291+0000","last_clean":"2024-02-05T08:21:52.962291+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:21:52.962291+0000","last_undegraded":"2024-02-05T08:21:52.962291+0000","last_fullsized":"2024-02-05T08:21:52.962291+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:37.234451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:37.234451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:04.799794+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":640,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.973873+0000","last_change":"2024-02-05T08:19:44.903925+0000","last_active":"2024-02-05T08:21:52.973873+0000","last_peered":"2024-02-05T08:21:52.973873+0000","last_clean":"2024-02-05T08:19:41.832619+0000","last_became_active":"2024-02-05T08:19:44.903925+0000","last_became_peered":"2024-02-05T08:19:44.903925+0000","last_unstale":"2024-02-05T08:21:52.973873+0000","last_undegraded":"2024-02-05T08:21:52.973873+0000","last_fullsized":"2024-02-05T08:19:44.901051+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.486901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.486901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":371,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.957853+0000","last_change":"2024-02-05T08:21:08.817280+0000","last_active":"2024-02-05T08:21:52.957853+0000","last_peered":"2024-02-05T08:21:52.957853+0000","last_clean":"2024-02-05T08:21:52.957853+0000","last_became_active":"2024-02-05T08:12:29.684444+0000","last_became_peered":"2024-02-05T08:12:29.684444+0000","last_unstale":"2024-02-05T08:21:52.957853+0000","last_undegraded":"2024-02-05T08:21:52.957853+0000","last_fullsized":"2024-02-05T08:21:52.957853+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:08.817073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:08.817073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:22.242885+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'15","reported_seq":127,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.969740+0000","last_change":"2024-02-05T08:19:44.899823+0000","last_active":"2024-02-05T08:21:52.969740+0000","last_peered":"2024-02-05T08:21:52.969740+0000","last_clean":"2024-02-05T08:19:41.832108+0000","last_became_active":"2024-02-05T08:19:44.899823+0000","last_became_peered":"2024-02-05T08:19:44.899823+0000","last_unstale":"2024-02-05T08:21:52.969740+0000","last_undegraded":"2024-02-05T08:19:44.888775+0000","last_fullsized":"2024-02-05T08:19:44.888665+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:17:54.101749+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.101749+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.878309+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"347'12847","reported_seq":13018,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:53.517183+0000","last_change":"2024-02-05T08:20:40.673313+0000","last_active":"2024-02-05T08:21:53.517183+0000","last_peered":"2024-02-05T08:21:53.517183+0000","last_clean":"2024-02-05T08:21:53.517183+0000","last_became_active":"2024-02-05T08:19:07.332783+0000","last_became_peered":"2024-02-05T08:19:07.332783+0000","last_unstale":"2024-02-05T08:21:53.517183+0000","last_undegraded":"2024-02-05T08:21:53.517183+0000","last_fullsized":"2024-02-05T08:21:53.517183+0000","mapping_epoch":309,"log_start":"332'7600","ondisk_log_start":"332'7600","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'7546","last_scrub_stamp":"2024-02-05T08:20:40.673010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:40.673010+0000","objects_scrubbed":7412,"log_size":5247,"log_dups_size":0,"ondisk_log_size":5247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:59.449412+0000","scrub_duration":4450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222232576,"num_objects":3391,"num_object_clones":0,"num_object_copies":10173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12847,"num_write_kb":519616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":649,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.973517+0000","last_change":"2024-02-05T08:21:45.532512+0000","last_active":"2024-02-05T08:21:52.973517+0000","last_peered":"2024-02-05T08:21:52.973517+0000","last_clean":"2024-02-05T08:21:52.973517+0000","last_became_active":"2024-02-05T08:12:29.679849+0000","last_became_peered":"2024-02-05T08:12:29.679849+0000","last_unstale":"2024-02-05T08:21:52.973517+0000","last_undegraded":"2024-02-05T08:21:52.973517+0000","last_fullsized":"2024-02-05T08:21:52.973517+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:45.532212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:45.532212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:55.262707+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":654,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.963487+0000","last_change":"2024-02-05T08:20:34.173135+0000","last_active":"2024-02-05T08:21:52.963487+0000","last_peered":"2024-02-05T08:21:52.963487+0000","last_clean":"2024-02-05T08:21:52.963487+0000","last_became_active":"2024-02-05T08:04:41.545361+0000","last_became_peered":"2024-02-05T08:04:41.545361+0000","last_unstale":"2024-02-05T08:21:52.963487+0000","last_undegraded":"2024-02-05T08:21:52.963487+0000","last_fullsized":"2024-02-05T08:21:52.963487+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:20:34.173063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:34.173063+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"262'34","reported_seq":731,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.967936+0000","last_change":"2024-02-05T08:21:44.548331+0000","last_active":"2024-02-05T08:21:52.967936+0000","last_peered":"2024-02-05T08:21:52.967936+0000","last_clean":"2024-02-05T08:21:52.967936+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:21:52.967936+0000","last_undegraded":"2024-02-05T08:21:52.967936+0000","last_fullsized":"2024-02-05T08:21:52.967936+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'34","last_scrub_stamp":"2024-02-05T08:21:44.548195+0000","last_deep_scrub":"262'34","last_deep_scrub_stamp":"2024-02-05T08:19:34.364738+0000","last_clean_scrub_stamp":"2024-02-05T08:21:44.548195+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:04.070224+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":870,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.961699+0000","last_change":"2024-02-05T08:21:46.165287+0000","last_active":"2024-02-05T08:21:52.961699+0000","last_peered":"2024-02-05T08:21:52.961699+0000","last_clean":"2024-02-05T08:21:52.961699+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:21:52.961699+0000","last_undegraded":"2024-02-05T08:21:52.961699+0000","last_fullsized":"2024-02-05T08:21:52.961699+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:21:46.164949+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_clean_scrub_stamp":"2024-02-05T08:21:46.164949+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:52.940791+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"347'12855","reported_seq":13021,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:53.034204+0000","last_change":"2024-02-05T08:21:09.538977+0000","last_active":"2024-02-05T08:21:53.034204+0000","last_peered":"2024-02-05T08:21:53.034204+0000","last_clean":"2024-02-05T08:21:53.034204+0000","last_became_active":"2024-02-05T08:19:07.346046+0000","last_became_peered":"2024-02-05T08:19:07.346046+0000","last_unstale":"2024-02-05T08:21:53.034204+0000","last_undegraded":"2024-02-05T08:21:53.034204+0000","last_fullsized":"2024-02-05T08:21:53.034204+0000","mapping_epoch":309,"log_start":"329'7335","ondisk_log_start":"329'7335","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'9467","last_scrub_stamp":"2024-02-05T08:21:09.538779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:21:09.538779+0000","objects_scrubbed":6962,"log_size":5520,"log_dups_size":0,"ondisk_log_size":5520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:39.064414+0000","scrub_duration":2843,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225902592,"num_objects":3447,"num_object_clones":0,"num_object_copies":10341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12855,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":119,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.967764+0000","last_change":"2024-02-05T08:19:44.899425+0000","last_active":"2024-02-05T08:21:52.967764+0000","last_peered":"2024-02-05T08:21:52.967764+0000","last_clean":"2024-02-05T08:19:41.833419+0000","last_became_active":"2024-02-05T08:19:44.899425+0000","last_became_peered":"2024-02-05T08:19:44.899425+0000","last_unstale":"2024-02-05T08:21:52.967764+0000","last_undegraded":"2024-02-05T08:21:52.967764+0000","last_fullsized":"2024-02-05T08:19:44.885939+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:31.245736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:18:31.245736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":637,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.962990+0000","last_change":"2024-02-05T08:19:44.926408+0000","last_active":"2024-02-05T08:21:52.962990+0000","last_peered":"2024-02-05T08:21:52.962990+0000","last_clean":"2024-02-05T08:19:41.835482+0000","last_became_active":"2024-02-05T08:19:44.926408+0000","last_became_peered":"2024-02-05T08:19:44.926408+0000","last_unstale":"2024-02-05T08:21:52.962990+0000","last_undegraded":"2024-02-05T08:21:52.962990+0000","last_fullsized":"2024-02-05T08:19:44.901657+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:37.226829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:19:37.226829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":690,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:51.955219+0000","last_change":"2024-02-05T08:19:44.912589+0000","last_active":"2024-02-05T08:21:51.955219+0000","last_peered":"2024-02-05T08:21:51.955219+0000","last_clean":"2024-02-05T08:19:41.830832+0000","last_became_active":"2024-02-05T08:19:44.912589+0000","last_became_peered":"2024-02-05T08:19:44.912589+0000","last_unstale":"2024-02-05T08:21:51.955219+0000","last_undegraded":"2024-02-05T08:19:44.909517+0000","last_fullsized":"2024-02-05T08:19:44.909410+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-02-05T08:18:02.629366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:02.629366+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.909936+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":636,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.963307+0000","last_change":"2024-02-05T08:19:44.888161+0000","last_active":"2024-02-05T08:21:52.963307+0000","last_peered":"2024-02-05T08:21:52.963307+0000","last_clean":"2024-02-05T08:19:41.831112+0000","last_became_active":"2024-02-05T08:19:44.888161+0000","last_became_peered":"2024-02-05T08:19:44.888161+0000","last_unstale":"2024-02-05T08:21:52.963307+0000","last_undegraded":"2024-02-05T08:21:52.963307+0000","last_fullsized":"2024-02-05T08:19:44.872217+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:33.725647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:18:33.725647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"347'12913","reported_seq":13082,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:54.037816+0000","last_change":"2024-02-05T08:20:35.938379+0000","last_active":"2024-02-05T08:21:54.037816+0000","last_peered":"2024-02-05T08:21:54.037816+0000","last_clean":"2024-02-05T08:21:54.037816+0000","last_became_active":"2024-02-05T08:19:07.345271+0000","last_became_peered":"2024-02-05T08:19:07.345271+0000","last_unstale":"2024-02-05T08:21:54.037816+0000","last_undegraded":"2024-02-05T08:21:54.037816+0000","last_fullsized":"2024-02-05T08:21:54.037816+0000","mapping_epoch":309,"log_start":"334'7800","ondisk_log_start":"334'7800","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'7352","last_scrub_stamp":"2024-02-05T08:20:35.938230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:35.938230+0000","objects_scrubbed":7270,"log_size":5113,"log_dups_size":0,"ondisk_log_size":5113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:58.519110+0000","scrub_duration":2394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227606528,"num_objects":3473,"num_object_clones":0,"num_object_copies":10419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12913,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":642,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.963444+0000","last_change":"2024-02-05T08:21:44.100658+0000","last_active":"2024-02-05T08:21:52.963444+0000","last_peered":"2024-02-05T08:21:52.963444+0000","last_clean":"2024-02-05T08:21:52.963444+0000","last_became_active":"2024-02-05T08:18:34.642239+0000","last_became_peered":"2024-02-05T08:18:34.642239+0000","last_unstale":"2024-02-05T08:21:52.963444+0000","last_undegraded":"2024-02-05T08:21:52.963444+0000","last_fullsized":"2024-02-05T08:21:52.963444+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:44.100428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:44.100428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:55.743623+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":641,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.962002+0000","last_change":"2024-02-05T08:20:37.971742+0000","last_active":"2024-02-05T08:21:52.962002+0000","last_peered":"2024-02-05T08:21:52.962002+0000","last_clean":"2024-02-05T08:21:52.962002+0000","last_became_active":"2024-02-05T08:04:45.567801+0000","last_became_peered":"2024-02-05T08:04:45.567801+0000","last_unstale":"2024-02-05T08:21:52.962002+0000","last_undegraded":"2024-02-05T08:21:52.962002+0000","last_fullsized":"2024-02-05T08:21:52.962002+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:37.971622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.971622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":655,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.962560+0000","last_change":"2024-02-05T08:21:40.182453+0000","last_active":"2024-02-05T08:21:52.962560+0000","last_peered":"2024-02-05T08:21:52.962560+0000","last_clean":"2024-02-05T08:21:52.962560+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:21:52.962560+0000","last_undegraded":"2024-02-05T08:21:52.962560+0000","last_fullsized":"2024-02-05T08:21:52.962560+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:40.182334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_clean_scrub_stamp":"2024-02-05T08:21:40.182334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:09.265384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"236'35","reported_seq":423,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.960028+0000","last_change":"2024-02-05T08:21:49.635918+0000","last_active":"2024-02-05T08:21:52.960028+0000","last_peered":"2024-02-05T08:21:52.960028+0000","last_clean":"2024-02-05T08:21:52.960028+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:21:52.960028+0000","last_undegraded":"2024-02-05T08:21:52.960028+0000","last_fullsized":"2024-02-05T08:21:52.960028+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'35","last_scrub_stamp":"2024-02-05T08:21:49.635790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:49.635790+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:08.866578+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"347'13180","reported_seq":13353,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:53.296270+0000","last_change":"2024-02-05T08:20:37.202482+0000","last_active":"2024-02-05T08:21:53.296270+0000","last_peered":"2024-02-05T08:21:53.296270+0000","last_clean":"2024-02-05T08:21:53.296270+0000","last_became_active":"2024-02-05T08:19:07.345197+0000","last_became_peered":"2024-02-05T08:19:07.345197+0000","last_unstale":"2024-02-05T08:21:53.296270+0000","last_undegraded":"2024-02-05T08:21:53.296270+0000","last_fullsized":"2024-02-05T08:21:53.296270+0000","mapping_epoch":309,"log_start":"329'7284","ondisk_log_start":"329'7284","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'7639","last_scrub_stamp":"2024-02-05T08:20:37.202377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.202377+0000","objects_scrubbed":7578,"log_size":5896,"log_dups_size":0,"ondisk_log_size":5896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236453888,"num_objects":3608,"num_object_clones":0,"num_object_copies":10824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13180,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":620,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.959501+0000","last_change":"2024-02-05T08:21:37.204657+0000","last_active":"2024-02-05T08:21:52.959501+0000","last_peered":"2024-02-05T08:21:52.959501+0000","last_clean":"2024-02-05T08:21:52.959501+0000","last_became_active":"2024-02-05T08:04:47.581778+0000","last_became_peered":"2024-02-05T08:04:47.581778+0000","last_unstale":"2024-02-05T08:21:52.959501+0000","last_undegraded":"2024-02-05T08:21:52.959501+0000","last_fullsized":"2024-02-05T08:21:52.959501+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:37.204550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_clean_scrub_stamp":"2024-02-05T08:21:37.204550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:49.557015+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":661,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.962489+0000","last_change":"2024-02-05T08:21:38.087273+0000","last_active":"2024-02-05T08:21:52.962489+0000","last_peered":"2024-02-05T08:21:52.962489+0000","last_clean":"2024-02-05T08:21:52.962489+0000","last_became_active":"2024-02-05T08:18:34.645830+0000","last_became_peered":"2024-02-05T08:18:34.645830+0000","last_unstale":"2024-02-05T08:21:52.962489+0000","last_undegraded":"2024-02-05T08:21:52.962489+0000","last_fullsized":"2024-02-05T08:21:52.962489+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:38.087157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:21:38.087157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:43.004194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":665,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.973985+0000","last_change":"2024-02-05T08:21:15.391166+0000","last_active":"2024-02-05T08:21:52.973985+0000","last_peered":"2024-02-05T08:21:52.973985+0000","last_clean":"2024-02-05T08:21:52.973985+0000","last_became_active":"2024-02-05T08:18:34.633214+0000","last_became_peered":"2024-02-05T08:18:34.633214+0000","last_unstale":"2024-02-05T08:21:52.973985+0000","last_undegraded":"2024-02-05T08:21:52.973985+0000","last_fullsized":"2024-02-05T08:21:52.973985+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:15.391056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:15.391056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:31.413224+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"262'19","reported_seq":652,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.960720+0000","last_change":"2024-02-05T08:19:44.914651+0000","last_active":"2024-02-05T08:21:52.960720+0000","last_peered":"2024-02-05T08:21:52.960720+0000","last_clean":"2024-02-05T08:19:41.838445+0000","last_became_active":"2024-02-05T08:19:44.914651+0000","last_became_peered":"2024-02-05T08:19:44.914651+0000","last_unstale":"2024-02-05T08:21:52.960720+0000","last_undegraded":"2024-02-05T08:19:44.886996+0000","last_fullsized":"2024-02-05T08:19:44.886890+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'19","last_scrub_stamp":"2024-02-05T08:19:17.554161+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.554161+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:54.230317+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"347'12957","reported_seq":13170,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:53.071868+0000","last_change":"2024-02-05T08:21:47.018537+0000","last_active":"2024-02-05T08:21:53.071868+0000","last_peered":"2024-02-05T08:21:53.071868+0000","last_clean":"2024-02-05T08:21:53.071868+0000","last_became_active":"2024-02-05T08:19:07.331718+0000","last_became_peered":"2024-02-05T08:19:07.331718+0000","last_unstale":"2024-02-05T08:21:53.071868+0000","last_undegraded":"2024-02-05T08:21:53.071868+0000","last_fullsized":"2024-02-05T08:21:53.071868+0000","mapping_epoch":309,"log_start":"329'7438","ondisk_log_start":"329'7438","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'12470","last_scrub_stamp":"2024-02-05T08:21:47.018241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:21:47.018241+0000","objects_scrubbed":4021,"log_size":5519,"log_dups_size":0,"ondisk_log_size":5519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:05.978475+0000","scrub_duration":1486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228655104,"num_objects":3489,"num_object_clones":0,"num_object_copies":10467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12957,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'130","reported_seq":416,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.967792+0000","last_change":"2024-02-05T08:21:47.509836+0000","last_active":"2024-02-05T08:21:52.967792+0000","last_peered":"2024-02-05T08:21:52.967792+0000","last_clean":"2024-02-05T08:21:52.967792+0000","last_became_active":"2024-02-05T08:19:01.816550+0000","last_became_peered":"2024-02-05T08:19:01.816550+0000","last_unstale":"2024-02-05T08:21:52.967792+0000","last_undegraded":"2024-02-05T08:21:52.967792+0000","last_fullsized":"2024-02-05T08:21:52.967792+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:47.509712+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:47.509712+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:47.870316+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":345,"reported_epoch":346,"state":"active+undersized","last_fresh":"2024-02-05T08:21:51.956265+0000","last_change":"2024-02-05T08:19:44.925889+0000","last_active":"2024-02-05T08:21:51.956265+0000","last_peered":"2024-02-05T08:21:51.956265+0000","last_clean":"2024-02-05T08:19:41.831505+0000","last_became_active":"2024-02-05T08:19:44.925889+0000","last_became_peered":"2024-02-05T08:19:44.925889+0000","last_unstale":"2024-02-05T08:21:51.956265+0000","last_undegraded":"2024-02-05T08:21:51.956265+0000","last_fullsized":"2024-02-05T08:19:44.914970+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:54.817641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.817641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.909906+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":669,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.972909+0000","last_change":"2024-02-05T08:19:44.903354+0000","last_active":"2024-02-05T08:21:52.972909+0000","last_peered":"2024-02-05T08:21:52.972909+0000","last_clean":"2024-02-05T08:19:41.832449+0000","last_became_active":"2024-02-05T08:19:44.903354+0000","last_became_peered":"2024-02-05T08:19:44.903354+0000","last_unstale":"2024-02-05T08:21:52.972909+0000","last_undegraded":"2024-02-05T08:19:44.873323+0000","last_fullsized":"2024-02-05T08:19:44.873213+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:18:48.543343+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:18:48.543343+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":652,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:51.954815+0000","last_change":"2024-02-05T08:21:46.849854+0000","last_active":"2024-02-05T08:21:51.954815+0000","last_peered":"2024-02-05T08:21:51.954815+0000","last_clean":"2024-02-05T08:21:51.954815+0000","last_became_active":"2024-02-05T08:12:29.670859+0000","last_became_peered":"2024-02-05T08:12:29.670859+0000","last_unstale":"2024-02-05T08:21:51.954815+0000","last_undegraded":"2024-02-05T08:21:51.954815+0000","last_fullsized":"2024-02-05T08:21:51.954815+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:46.849759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:21:46.849759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:00.294002+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"325'53","reported_seq":686,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.962360+0000","last_change":"2024-02-05T08:19:44.892891+0000","last_active":"2024-02-05T08:21:52.962360+0000","last_peered":"2024-02-05T08:21:52.962360+0000","last_clean":"2024-02-05T08:14:49.167838+0000","last_became_active":"2024-02-05T08:19:44.892891+0000","last_became_peered":"2024-02-05T08:19:44.892891+0000","last_unstale":"2024-02-05T08:21:52.962360+0000","last_undegraded":"2024-02-05T08:19:44.886039+0000","last_fullsized":"2024-02-05T08:19:44.885927+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'52","last_scrub_stamp":"2024-02-05T08:19:15.601598+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.601598+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":612,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.959303+0000","last_change":"2024-02-05T08:19:44.924245+0000","last_active":"2024-02-05T08:21:52.959303+0000","last_peered":"2024-02-05T08:21:52.959303+0000","last_clean":"2024-02-05T08:19:41.838535+0000","last_became_active":"2024-02-05T08:19:44.924245+0000","last_became_peered":"2024-02-05T08:19:44.924245+0000","last_unstale":"2024-02-05T08:21:52.959303+0000","last_undegraded":"2024-02-05T08:21:52.959303+0000","last_fullsized":"2024-02-05T08:19:44.874710+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:14.615047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.615047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:54.230303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"347'13142","reported_seq":13348,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:53.525825+0000","last_change":"2024-02-05T08:21:48.036801+0000","last_active":"2024-02-05T08:21:53.525825+0000","last_peered":"2024-02-05T08:21:53.525825+0000","last_clean":"2024-02-05T08:21:53.525825+0000","last_became_active":"2024-02-05T08:19:07.333532+0000","last_became_peered":"2024-02-05T08:19:07.333532+0000","last_unstale":"2024-02-05T08:21:53.525825+0000","last_undegraded":"2024-02-05T08:21:53.525825+0000","last_fullsized":"2024-02-05T08:21:53.525825+0000","mapping_epoch":309,"log_start":"333'7900","ondisk_log_start":"333'7900","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'12709","last_scrub_stamp":"2024-02-05T08:21:48.036494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:21:48.036494+0000","objects_scrubbed":4030,"log_size":5242,"log_dups_size":0,"ondisk_log_size":5242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:13.805552+0000","scrub_duration":1518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231473152,"num_objects":3532,"num_object_clones":0,"num_object_copies":10596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13142,"num_write_kb":533568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'130","reported_seq":940,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.961566+0000","last_change":"2024-02-05T08:20:27.164887+0000","last_active":"2024-02-05T08:21:52.961566+0000","last_peered":"2024-02-05T08:21:52.961566+0000","last_clean":"2024-02-05T08:21:52.961566+0000","last_became_active":"2024-02-05T08:19:01.813292+0000","last_became_peered":"2024-02-05T08:19:01.813292+0000","last_unstale":"2024-02-05T08:21:52.961566+0000","last_undegraded":"2024-02-05T08:21:52.961566+0000","last_fullsized":"2024-02-05T08:21:52.961566+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:27.164766+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:27.164766+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":647,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.962697+0000","last_change":"2024-02-05T08:20:21.189633+0000","last_active":"2024-02-05T08:21:52.962697+0000","last_peered":"2024-02-05T08:21:52.962697+0000","last_clean":"2024-02-05T08:21:52.962697+0000","last_became_active":"2024-02-05T08:18:34.643109+0000","last_became_peered":"2024-02-05T08:18:34.643109+0000","last_unstale":"2024-02-05T08:21:52.962697+0000","last_undegraded":"2024-02-05T08:21:52.962697+0000","last_fullsized":"2024-02-05T08:21:52.962697+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:21.189500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:21.189500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":128,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.963264+0000","last_change":"2024-02-05T08:19:44.902069+0000","last_active":"2024-02-05T08:21:52.963264+0000","last_peered":"2024-02-05T08:21:52.963264+0000","last_clean":"2024-02-05T08:06:34.916685+0000","last_became_active":"2024-02-05T08:19:44.902069+0000","last_became_peered":"2024-02-05T08:19:44.902069+0000","last_unstale":"2024-02-05T08:21:52.963264+0000","last_undegraded":"2024-02-05T08:19:44.872249+0000","last_fullsized":"2024-02-05T08:19:44.872099+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:18:43.341831+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:18:43.341831+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":671,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.961538+0000","last_change":"2024-02-05T08:20:29.922938+0000","last_active":"2024-02-05T08:21:52.961538+0000","last_peered":"2024-02-05T08:21:52.961538+0000","last_clean":"2024-02-05T08:21:52.961538+0000","last_became_active":"2024-02-05T08:18:34.644683+0000","last_became_peered":"2024-02-05T08:18:34.644683+0000","last_unstale":"2024-02-05T08:21:52.961538+0000","last_undegraded":"2024-02-05T08:21:52.961538+0000","last_fullsized":"2024-02-05T08:21:52.961538+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_clean_scrub_stamp":"2024-02-05T08:20:29.922794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.839539+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'17","reported_seq":655,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.962338+0000","last_change":"2024-02-05T08:21:10.019548+0000","last_active":"2024-02-05T08:21:52.962338+0000","last_peered":"2024-02-05T08:21:52.962338+0000","last_clean":"2024-02-05T08:21:52.962338+0000","last_became_active":"2024-02-05T08:05:03.788596+0000","last_became_peered":"2024-02-05T08:05:03.788596+0000","last_unstale":"2024-02-05T08:21:52.962338+0000","last_undegraded":"2024-02-05T08:21:52.962338+0000","last_fullsized":"2024-02-05T08:21:52.962338+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'17","last_scrub_stamp":"2024-02-05T08:21:10.019219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:10.019219+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:29.604160+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":649,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.962158+0000","last_change":"2024-02-05T08:20:37.014562+0000","last_active":"2024-02-05T08:21:52.962158+0000","last_peered":"2024-02-05T08:21:52.962158+0000","last_clean":"2024-02-05T08:21:52.962158+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:21:52.962158+0000","last_undegraded":"2024-02-05T08:21:52.962158+0000","last_fullsized":"2024-02-05T08:21:52.962158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:37.014448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.014448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:02.259236+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"347'12844","reported_seq":12933,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:53.024651+0000","last_change":"2024-02-05T08:19:44.910611+0000","last_active":"2024-02-05T08:21:53.024651+0000","last_peered":"2024-02-05T08:21:53.024651+0000","last_clean":"2024-02-05T08:19:43.860522+0000","last_became_active":"2024-02-05T08:19:44.910611+0000","last_became_peered":"2024-02-05T08:19:44.910611+0000","last_unstale":"2024-02-05T08:21:53.024651+0000","last_undegraded":"2024-02-05T08:19:44.872215+0000","last_fullsized":"2024-02-05T08:19:44.872102+0000","mapping_epoch":324,"log_start":"329'7326","ondisk_log_start":"329'7326","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":5518,"log_dups_size":0,"ondisk_log_size":5518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.626751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227278848,"num_objects":3468,"num_object_clones":0,"num_object_copies":10404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3468,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12844,"num_write_kb":521984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":3468}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":642,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.974592+0000","last_change":"2024-02-05T08:21:38.543400+0000","last_active":"2024-02-05T08:21:52.974592+0000","last_peered":"2024-02-05T08:21:52.974592+0000","last_clean":"2024-02-05T08:21:52.974592+0000","last_became_active":"2024-02-05T08:18:34.633450+0000","last_became_peered":"2024-02-05T08:18:34.633450+0000","last_unstale":"2024-02-05T08:21:52.974592+0000","last_undegraded":"2024-02-05T08:21:52.974592+0000","last_fullsized":"2024-02-05T08:21:52.974592+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:38.543288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:38.543288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:59.833175+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":932,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.963804+0000","last_change":"2024-02-05T08:20:33.165619+0000","last_active":"2024-02-05T08:21:52.963804+0000","last_peered":"2024-02-05T08:21:52.963804+0000","last_clean":"2024-02-05T08:21:52.963804+0000","last_became_active":"2024-02-05T08:18:57.285362+0000","last_became_peered":"2024-02-05T08:18:57.285362+0000","last_unstale":"2024-02-05T08:21:52.963804+0000","last_undegraded":"2024-02-05T08:21:52.963804+0000","last_fullsized":"2024-02-05T08:21:52.963804+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_clean_scrub_stamp":"2024-02-05T08:20:33.165371+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":132,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.966953+0000","last_change":"2024-02-05T08:21:06.644882+0000","last_active":"2024-02-05T08:21:52.966953+0000","last_peered":"2024-02-05T08:21:52.966953+0000","last_clean":"2024-02-05T08:21:52.966953+0000","last_became_active":"2024-02-05T08:18:34.631974+0000","last_became_peered":"2024-02-05T08:18:34.631974+0000","last_unstale":"2024-02-05T08:21:52.966953+0000","last_undegraded":"2024-02-05T08:21:52.966953+0000","last_fullsized":"2024-02-05T08:21:52.966953+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:06.644775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:06.644775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:10.975758+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"236'30","reported_seq":520,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.968386+0000","last_change":"2024-02-05T08:21:48.670539+0000","last_active":"2024-02-05T08:21:52.968386+0000","last_peered":"2024-02-05T08:21:52.968386+0000","last_clean":"2024-02-05T08:21:52.968386+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:21:52.968386+0000","last_undegraded":"2024-02-05T08:21:52.968386+0000","last_fullsized":"2024-02-05T08:21:52.968386+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:21:48.670245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:48.670245+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:04.208440+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":627,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.964964+0000","last_change":"2024-02-05T08:20:20.098080+0000","last_active":"2024-02-05T08:21:52.964964+0000","last_peered":"2024-02-05T08:21:52.964964+0000","last_clean":"2024-02-05T08:21:52.964964+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:21:52.964964+0000","last_undegraded":"2024-02-05T08:21:52.964964+0000","last_fullsized":"2024-02-05T08:21:52.964964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:20.097959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_clean_scrub_stamp":"2024-02-05T08:20:20.097959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"347'12726","reported_seq":12895,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:54.057344+0000","last_change":"2024-02-05T08:20:39.198575+0000","last_active":"2024-02-05T08:21:54.057344+0000","last_peered":"2024-02-05T08:21:54.057344+0000","last_clean":"2024-02-05T08:21:54.057344+0000","last_became_active":"2024-02-05T08:19:07.327007+0000","last_became_peered":"2024-02-05T08:19:07.327007+0000","last_unstale":"2024-02-05T08:21:54.057344+0000","last_undegraded":"2024-02-05T08:21:54.057344+0000","last_fullsized":"2024-02-05T08:21:54.057344+0000","mapping_epoch":309,"log_start":"332'7600","ondisk_log_start":"332'7600","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'7440","last_scrub_stamp":"2024-02-05T08:20:39.198303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:39.198303+0000","objects_scrubbed":7387,"log_size":5126,"log_dups_size":0,"ondisk_log_size":5126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:09.169271+0000","scrub_duration":1696,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223608832,"num_objects":3412,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12726,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"82'130","reported_seq":605,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.958294+0000","last_change":"2024-02-05T08:19:44.916776+0000","last_active":"2024-02-05T08:21:52.958294+0000","last_peered":"2024-02-05T08:21:52.958294+0000","last_clean":"2024-02-05T08:12:23.979378+0000","last_became_active":"2024-02-05T08:19:44.916776+0000","last_became_peered":"2024-02-05T08:19:44.916776+0000","last_unstale":"2024-02-05T08:21:52.958294+0000","last_undegraded":"2024-02-05T08:21:52.958294+0000","last_fullsized":"2024-02-05T08:19:44.897597+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:07.594418+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:07.594418+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.626733+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":566,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.974303+0000","last_change":"2024-02-05T08:19:44.903734+0000","last_active":"2024-02-05T08:21:52.974303+0000","last_peered":"2024-02-05T08:21:52.974303+0000","last_clean":"2024-02-05T08:18:29.610515+0000","last_became_active":"2024-02-05T08:19:44.903734+0000","last_became_peered":"2024-02-05T08:19:44.903734+0000","last_unstale":"2024-02-05T08:21:52.974303+0000","last_undegraded":"2024-02-05T08:21:52.974303+0000","last_fullsized":"2024-02-05T08:19:44.898755+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.646431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.646431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:58.411476+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":637,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.959255+0000","last_change":"2024-02-05T08:20:12.695787+0000","last_active":"2024-02-05T08:21:52.959255+0000","last_peered":"2024-02-05T08:21:52.959255+0000","last_clean":"2024-02-05T08:21:52.959255+0000","last_became_active":"2024-02-05T08:18:34.635954+0000","last_became_peered":"2024-02-05T08:18:34.635954+0000","last_unstale":"2024-02-05T08:21:52.959255+0000","last_undegraded":"2024-02-05T08:21:52.959255+0000","last_fullsized":"2024-02-05T08:21:52.959255+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:12.695667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:12.695667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":1675,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.963705+0000","last_change":"2024-02-05T08:19:44.901362+0000","last_active":"2024-02-05T08:21:52.963705+0000","last_peered":"2024-02-05T08:21:52.963705+0000","last_clean":"2024-02-05T08:19:43.389689+0000","last_became_active":"2024-02-05T08:19:44.901362+0000","last_became_peered":"2024-02-05T08:19:44.901362+0000","last_unstale":"2024-02-05T08:21:52.963705+0000","last_undegraded":"2024-02-05T08:19:44.875266+0000","last_fullsized":"2024-02-05T08:19:44.875151+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:19:14.179578+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.179578+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1022,"num_read_kb":1017,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":684,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.960400+0000","last_change":"2024-02-05T08:21:52.253152+0000","last_active":"2024-02-05T08:21:52.960400+0000","last_peered":"2024-02-05T08:21:52.960400+0000","last_clean":"2024-02-05T08:21:52.960400+0000","last_became_active":"2024-02-05T08:14:24.992820+0000","last_became_peered":"2024-02-05T08:14:24.992820+0000","last_unstale":"2024-02-05T08:21:52.960400+0000","last_undegraded":"2024-02-05T08:21:52.960400+0000","last_fullsized":"2024-02-05T08:21:52.960400+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:21:52.253002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:52.253002+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:19.397758+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":628,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.974806+0000","last_change":"2024-02-05T08:19:44.900624+0000","last_active":"2024-02-05T08:21:52.974806+0000","last_peered":"2024-02-05T08:21:52.974806+0000","last_clean":"2024-02-05T08:19:41.831836+0000","last_became_active":"2024-02-05T08:19:44.900624+0000","last_became_peered":"2024-02-05T08:19:44.900624+0000","last_unstale":"2024-02-05T08:21:52.974806+0000","last_undegraded":"2024-02-05T08:21:52.974806+0000","last_fullsized":"2024-02-05T08:19:44.899450+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:35.391824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:19:35.391824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"347'12844","reported_seq":13057,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:53.089150+0000","last_change":"2024-02-05T08:21:04.237956+0000","last_active":"2024-02-05T08:21:53.089150+0000","last_peered":"2024-02-05T08:21:53.089150+0000","last_clean":"2024-02-05T08:21:53.089150+0000","last_became_active":"2024-02-05T08:19:07.332849+0000","last_became_peered":"2024-02-05T08:19:07.332849+0000","last_unstale":"2024-02-05T08:21:53.089150+0000","last_undegraded":"2024-02-05T08:21:53.089150+0000","last_fullsized":"2024-02-05T08:21:53.089150+0000","mapping_epoch":309,"log_start":"330'7385","ondisk_log_start":"330'7385","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'9124","last_scrub_stamp":"2024-02-05T08:21:04.237762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:21:04.237762+0000","objects_scrubbed":7297,"log_size":5459,"log_dups_size":0,"ondisk_log_size":5459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:20.139534+0000","scrub_duration":2557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228917272,"num_objects":3494,"num_object_clones":0,"num_object_copies":10482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12844,"num_write_kb":522753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'130","reported_seq":669,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.959907+0000","last_change":"2024-02-05T08:20:19.485708+0000","last_active":"2024-02-05T08:21:52.959907+0000","last_peered":"2024-02-05T08:21:52.959907+0000","last_clean":"2024-02-05T08:21:52.959907+0000","last_became_active":"2024-02-05T08:19:01.817743+0000","last_became_peered":"2024-02-05T08:19:01.817743+0000","last_unstale":"2024-02-05T08:21:52.959907+0000","last_undegraded":"2024-02-05T08:21:52.959907+0000","last_fullsized":"2024-02-05T08:21:52.959907+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:19.485585+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.485585+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":333,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.957595+0000","last_change":"2024-02-05T08:19:44.901104+0000","last_active":"2024-02-05T08:21:52.957595+0000","last_peered":"2024-02-05T08:21:52.957595+0000","last_clean":"2024-02-05T08:12:25.219295+0000","last_became_active":"2024-02-05T08:19:44.901104+0000","last_became_peered":"2024-02-05T08:19:44.901104+0000","last_unstale":"2024-02-05T08:21:52.957595+0000","last_undegraded":"2024-02-05T08:21:52.957595+0000","last_fullsized":"2024-02-05T08:19:44.872029+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:27.862543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:27.862543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.630632+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":54,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.962199+0000","last_change":"2024-02-05T08:19:44.901166+0000","last_active":"2024-02-05T08:21:52.962199+0000","last_peered":"2024-02-05T08:21:52.962199+0000","last_clean":"2024-02-05T08:04:40.511631+0000","last_became_active":"2024-02-05T08:19:44.901166+0000","last_became_peered":"2024-02-05T08:19:44.901166+0000","last_unstale":"2024-02-05T08:21:52.962199+0000","last_undegraded":"2024-02-05T08:21:52.962199+0000","last_fullsized":"2024-02-05T08:19:44.885594+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:37.350622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.350622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":680,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.960602+0000","last_change":"2024-02-05T08:21:11.323387+0000","last_active":"2024-02-05T08:21:52.960602+0000","last_peered":"2024-02-05T08:21:52.960602+0000","last_clean":"2024-02-05T08:21:52.960602+0000","last_became_active":"2024-02-05T08:04:43.606027+0000","last_became_peered":"2024-02-05T08:04:43.606027+0000","last_unstale":"2024-02-05T08:21:52.960602+0000","last_undegraded":"2024-02-05T08:21:52.960602+0000","last_fullsized":"2024-02-05T08:21:52.960602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-02-05T08:21:11.323094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:11.323094+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:17.012111+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":371,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.959937+0000","last_change":"2024-02-05T08:21:07.746354+0000","last_active":"2024-02-05T08:21:52.959937+0000","last_peered":"2024-02-05T08:21:52.959937+0000","last_clean":"2024-02-05T08:21:52.959937+0000","last_became_active":"2024-02-05T08:12:29.688534+0000","last_became_peered":"2024-02-05T08:12:29.688534+0000","last_unstale":"2024-02-05T08:21:52.959937+0000","last_undegraded":"2024-02-05T08:21:52.959937+0000","last_fullsized":"2024-02-05T08:21:52.959937+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:07.746072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:21:07.746072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:26.863248+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"347'12969","reported_seq":13140,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:53.216700+0000","last_change":"2024-02-05T08:20:34.585093+0000","last_active":"2024-02-05T08:21:53.216700+0000","last_peered":"2024-02-05T08:21:53.216700+0000","last_clean":"2024-02-05T08:21:53.216700+0000","last_became_active":"2024-02-05T08:19:07.332176+0000","last_became_peered":"2024-02-05T08:19:07.332176+0000","last_unstale":"2024-02-05T08:21:53.216700+0000","last_undegraded":"2024-02-05T08:21:53.216700+0000","last_fullsized":"2024-02-05T08:21:53.216700+0000","mapping_epoch":309,"log_start":"334'7900","ondisk_log_start":"334'7900","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'7365","last_scrub_stamp":"2024-02-05T08:20:34.584959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:34.584959+0000","objects_scrubbed":7281,"log_size":5069,"log_dups_size":0,"ondisk_log_size":5069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224460800,"num_objects":3425,"num_object_clones":0,"num_object_copies":10275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12969,"num_write_kb":524608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"82'130","reported_seq":669,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.960214+0000","last_change":"2024-02-05T08:20:35.974697+0000","last_active":"2024-02-05T08:21:52.960214+0000","last_peered":"2024-02-05T08:21:52.960214+0000","last_clean":"2024-02-05T08:21:52.960214+0000","last_became_active":"2024-02-05T08:19:01.815446+0000","last_became_peered":"2024-02-05T08:19:01.815446+0000","last_unstale":"2024-02-05T08:21:52.960214+0000","last_undegraded":"2024-02-05T08:21:52.960214+0000","last_fullsized":"2024-02-05T08:21:52.960214+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:35.974462+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:35.974462+0000","last_clean_scrub_stamp":"2024-02-05T08:20:35.974462+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:59.595447+0000","scrub_duration":332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":633,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.959304+0000","last_change":"2024-02-05T08:21:14.350292+0000","last_active":"2024-02-05T08:21:52.959304+0000","last_peered":"2024-02-05T08:21:52.959304+0000","last_clean":"2024-02-05T08:21:52.959304+0000","last_became_active":"2024-02-05T08:18:34.636693+0000","last_became_peered":"2024-02-05T08:18:34.636693+0000","last_unstale":"2024-02-05T08:21:52.959304+0000","last_undegraded":"2024-02-05T08:21:52.959304+0000","last_fullsized":"2024-02-05T08:21:52.959304+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:14.350181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:14.350181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:42.191258+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":654,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.973415+0000","last_change":"2024-02-05T08:21:40.544316+0000","last_active":"2024-02-05T08:21:52.973415+0000","last_peered":"2024-02-05T08:21:52.973415+0000","last_clean":"2024-02-05T08:21:52.973415+0000","last_became_active":"2024-02-05T08:18:34.633263+0000","last_became_peered":"2024-02-05T08:18:34.633263+0000","last_unstale":"2024-02-05T08:21:52.973415+0000","last_undegraded":"2024-02-05T08:21:52.973415+0000","last_fullsized":"2024-02-05T08:21:52.973415+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:40.544202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:40.544202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:46.579918+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'20","reported_seq":463,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.973228+0000","last_change":"2024-02-05T08:19:44.916082+0000","last_active":"2024-02-05T08:21:52.973228+0000","last_peered":"2024-02-05T08:21:52.973228+0000","last_clean":"2024-02-05T08:14:49.165927+0000","last_became_active":"2024-02-05T08:19:44.916082+0000","last_became_peered":"2024-02-05T08:19:44.916082+0000","last_unstale":"2024-02-05T08:21:52.973228+0000","last_undegraded":"2024-02-05T08:19:44.899919+0000","last_fullsized":"2024-02-05T08:19:44.899797+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:19:09.511509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:09.511509+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:58.411445+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"277'11","reported_seq":861,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.960921+0000","last_change":"2024-02-05T08:21:39.122006+0000","last_active":"2024-02-05T08:21:52.960921+0000","last_peered":"2024-02-05T08:21:52.960921+0000","last_clean":"2024-02-05T08:21:52.960921+0000","last_became_active":"2024-02-05T08:12:29.683091+0000","last_became_peered":"2024-02-05T08:12:29.683091+0000","last_unstale":"2024-02-05T08:21:52.960921+0000","last_undegraded":"2024-02-05T08:21:52.960921+0000","last_fullsized":"2024-02-05T08:21:52.960921+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:21:39.121890+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_clean_scrub_stamp":"2024-02-05T08:21:39.121890+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:41.515934+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"347'13050","reported_seq":13223,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:53.048000+0000","last_change":"2024-02-05T08:20:41.146544+0000","last_active":"2024-02-05T08:21:53.048000+0000","last_peered":"2024-02-05T08:21:53.048000+0000","last_clean":"2024-02-05T08:21:53.048000+0000","last_became_active":"2024-02-05T08:19:07.346024+0000","last_became_peered":"2024-02-05T08:19:07.346024+0000","last_unstale":"2024-02-05T08:21:53.048000+0000","last_undegraded":"2024-02-05T08:21:53.048000+0000","last_fullsized":"2024-02-05T08:21:53.048000+0000","mapping_epoch":309,"log_start":"331'7538","ondisk_log_start":"331'7538","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'7704","last_scrub_stamp":"2024-02-05T08:20:41.146350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:41.146350+0000","objects_scrubbed":7648,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226885632,"num_objects":3462,"num_object_clones":0,"num_object_copies":10386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13050,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'130","reported_seq":408,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.968049+0000","last_change":"2024-02-05T08:20:27.728841+0000","last_active":"2024-02-05T08:21:52.968049+0000","last_peered":"2024-02-05T08:21:52.968049+0000","last_clean":"2024-02-05T08:21:52.968049+0000","last_became_active":"2024-02-05T08:19:01.819856+0000","last_became_peered":"2024-02-05T08:19:01.819856+0000","last_unstale":"2024-02-05T08:21:52.968049+0000","last_undegraded":"2024-02-05T08:21:52.968049+0000","last_fullsized":"2024-02-05T08:21:52.968049+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:27.728717+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:27.728717+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:56.586217+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":635,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.962064+0000","last_change":"2024-02-05T08:20:19.251720+0000","last_active":"2024-02-05T08:21:52.962064+0000","last_peered":"2024-02-05T08:21:52.962064+0000","last_clean":"2024-02-05T08:21:52.962064+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:21:52.962064+0000","last_undegraded":"2024-02-05T08:21:52.962064+0000","last_fullsized":"2024-02-05T08:21:52.962064+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:19.251464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.251464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":638,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.960492+0000","last_change":"2024-02-05T08:19:44.892743+0000","last_active":"2024-02-05T08:21:52.960492+0000","last_peered":"2024-02-05T08:21:52.960492+0000","last_clean":"2024-02-05T08:19:41.834750+0000","last_became_active":"2024-02-05T08:19:44.892743+0000","last_became_peered":"2024-02-05T08:19:44.892743+0000","last_unstale":"2024-02-05T08:21:52.960492+0000","last_undegraded":"2024-02-05T08:19:44.887459+0000","last_fullsized":"2024-02-05T08:19:44.887352+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:19:11.483052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:11.483052+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:54.230276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"236'28","reported_seq":714,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:51.954882+0000","last_change":"2024-02-05T08:21:13.958221+0000","last_active":"2024-02-05T08:21:51.954882+0000","last_peered":"2024-02-05T08:21:51.954882+0000","last_clean":"2024-02-05T08:21:51.954882+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:21:51.954882+0000","last_undegraded":"2024-02-05T08:21:51.954882+0000","last_fullsized":"2024-02-05T08:21:51.954882+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:21:13.957934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:13.957934+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:34.456435+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"325'12","reported_seq":173,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.967165+0000","last_change":"2024-02-05T08:19:44.915079+0000","last_active":"2024-02-05T08:21:52.967165+0000","last_peered":"2024-02-05T08:21:52.967165+0000","last_clean":"2024-02-05T08:19:41.828253+0000","last_became_active":"2024-02-05T08:19:44.915079+0000","last_became_peered":"2024-02-05T08:19:44.915079+0000","last_unstale":"2024-02-05T08:21:52.967165+0000","last_undegraded":"2024-02-05T08:19:44.888020+0000","last_fullsized":"2024-02-05T08:19:44.887894+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'11","last_scrub_stamp":"2024-02-05T08:19:34.968809+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:19:34.968809+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"346'14965","reported_seq":15248,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.726984+0000","last_change":"2024-02-05T08:21:06.695781+0000","last_active":"2024-02-05T08:21:52.726984+0000","last_peered":"2024-02-05T08:21:52.726984+0000","last_clean":"2024-02-05T08:21:52.726984+0000","last_became_active":"2024-02-05T08:19:07.331338+0000","last_became_peered":"2024-02-05T08:19:07.331338+0000","last_unstale":"2024-02-05T08:21:52.726984+0000","last_undegraded":"2024-02-05T08:21:52.726984+0000","last_fullsized":"2024-02-05T08:21:52.726984+0000","mapping_epoch":309,"log_start":"336'10502","ondisk_log_start":"336'10502","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'11401","last_scrub_stamp":"2024-02-05T08:21:06.695186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:21:06.695186+0000","objects_scrubbed":7153,"log_size":4463,"log_dups_size":0,"ondisk_log_size":4463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:27.523490+0000","scrub_duration":2629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230293504,"num_objects":3514,"num_object_clones":0,"num_object_copies":10542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3514,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12882,"num_write_kb":524672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'130","reported_seq":406,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:51.955563+0000","last_change":"2024-02-05T08:21:05.054369+0000","last_active":"2024-02-05T08:21:51.955563+0000","last_peered":"2024-02-05T08:21:51.955563+0000","last_clean":"2024-02-05T08:21:51.955563+0000","last_became_active":"2024-02-05T08:19:01.816923+0000","last_became_peered":"2024-02-05T08:19:01.816923+0000","last_unstale":"2024-02-05T08:21:51.955563+0000","last_undegraded":"2024-02-05T08:21:51.955563+0000","last_fullsized":"2024-02-05T08:21:51.955563+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:05.054231+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:05.054231+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:22.716414+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":640,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:51.954711+0000","last_change":"2024-02-05T08:20:36.100335+0000","last_active":"2024-02-05T08:21:51.954711+0000","last_peered":"2024-02-05T08:21:51.954711+0000","last_clean":"2024-02-05T08:21:51.954711+0000","last_became_active":"2024-02-05T08:18:34.635908+0000","last_became_peered":"2024-02-05T08:18:34.635908+0000","last_unstale":"2024-02-05T08:21:51.954711+0000","last_undegraded":"2024-02-05T08:21:51.954711+0000","last_fullsized":"2024-02-05T08:21:51.954711+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:36.100208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:36.100208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:04.352882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":136,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.969653+0000","last_change":"2024-02-05T08:21:08.559594+0000","last_active":"2024-02-05T08:21:52.969653+0000","last_peered":"2024-02-05T08:21:52.969653+0000","last_clean":"2024-02-05T08:21:52.969653+0000","last_became_active":"2024-02-05T08:18:34.649808+0000","last_became_peered":"2024-02-05T08:18:34.649808+0000","last_unstale":"2024-02-05T08:21:52.969653+0000","last_undegraded":"2024-02-05T08:21:52.969653+0000","last_fullsized":"2024-02-05T08:21:52.969653+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:21:08.559493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:08.559493+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:17.634233+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"236'38","reported_seq":150,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.966873+0000","last_change":"2024-02-05T08:20:37.207904+0000","last_active":"2024-02-05T08:21:52.966873+0000","last_peered":"2024-02-05T08:21:52.966873+0000","last_clean":"2024-02-05T08:21:52.966873+0000","last_became_active":"2024-02-05T08:18:34.653505+0000","last_became_peered":"2024-02-05T08:18:34.653505+0000","last_unstale":"2024-02-05T08:21:52.966873+0000","last_undegraded":"2024-02-05T08:21:52.966873+0000","last_fullsized":"2024-02-05T08:21:52.966873+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'38","last_scrub_stamp":"2024-02-05T08:20:37.207762+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.207762+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:54.648572+0000","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":636,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:51.954651+0000","last_change":"2024-02-05T08:20:31.133433+0000","last_active":"2024-02-05T08:21:51.954651+0000","last_peered":"2024-02-05T08:21:51.954651+0000","last_clean":"2024-02-05T08:21:51.954651+0000","last_became_active":"2024-02-05T08:18:34.636667+0000","last_became_peered":"2024-02-05T08:18:34.636667+0000","last_unstale":"2024-02-05T08:21:51.954651+0000","last_undegraded":"2024-02-05T08:21:51.954651+0000","last_fullsized":"2024-02-05T08:21:51.954651+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:31.133206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:31.133206+0000","last_clean_scrub_stamp":"2024-02-05T08:20:31.133206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:00.885317+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"346'12940","reported_seq":13108,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.708860+0000","last_change":"2024-02-05T08:20:37.390439+0000","last_active":"2024-02-05T08:21:52.708860+0000","last_peered":"2024-02-05T08:21:52.708860+0000","last_clean":"2024-02-05T08:21:52.708860+0000","last_became_active":"2024-02-05T08:19:07.330566+0000","last_became_peered":"2024-02-05T08:19:07.330566+0000","last_unstale":"2024-02-05T08:21:52.708860+0000","last_undegraded":"2024-02-05T08:21:52.708860+0000","last_fullsized":"2024-02-05T08:21:52.708860+0000","mapping_epoch":309,"log_start":"336'8500","ondisk_log_start":"336'8500","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"330'7463","last_scrub_stamp":"2024-02-05T08:20:37.390208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.390208+0000","objects_scrubbed":7384,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.604218+0000","scrub_duration":2278,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234881024,"num_objects":3584,"num_object_clones":0,"num_object_copies":10752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12940,"num_write_kb":528768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'130","reported_seq":416,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.972607+0000","last_change":"2024-02-05T08:21:48.475705+0000","last_active":"2024-02-05T08:21:52.972607+0000","last_peered":"2024-02-05T08:21:52.972607+0000","last_clean":"2024-02-05T08:21:52.972607+0000","last_became_active":"2024-02-05T08:19:01.817169+0000","last_became_peered":"2024-02-05T08:19:01.817169+0000","last_unstale":"2024-02-05T08:21:52.972607+0000","last_undegraded":"2024-02-05T08:21:52.972607+0000","last_fullsized":"2024-02-05T08:21:52.972607+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:48.475580+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:48.475580+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:03.051786+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":624,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.972991+0000","last_change":"2024-02-05T08:19:44.900438+0000","last_active":"2024-02-05T08:21:52.972991+0000","last_peered":"2024-02-05T08:21:52.972991+0000","last_clean":"2024-02-05T08:19:41.831465+0000","last_became_active":"2024-02-05T08:19:44.900438+0000","last_became_peered":"2024-02-05T08:19:44.900438+0000","last_unstale":"2024-02-05T08:21:52.972991+0000","last_undegraded":"2024-02-05T08:21:52.972991+0000","last_fullsized":"2024-02-05T08:19:44.898161+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.613476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.613476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":838,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:51.956618+0000","last_change":"2024-02-05T08:20:38.198413+0000","last_active":"2024-02-05T08:21:51.956618+0000","last_peered":"2024-02-05T08:21:51.956618+0000","last_clean":"2024-02-05T08:21:51.956618+0000","last_became_active":"2024-02-05T08:04:45.566357+0000","last_became_peered":"2024-02-05T08:04:45.566357+0000","last_unstale":"2024-02-05T08:21:51.956618+0000","last_undegraded":"2024-02-05T08:21:51.956618+0000","last_fullsized":"2024-02-05T08:21:51.956618+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:20:38.198278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:38.198278+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:58.293117+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":127,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.966730+0000","last_change":"2024-02-05T08:19:44.902070+0000","last_active":"2024-02-05T08:21:52.966730+0000","last_peered":"2024-02-05T08:21:52.966730+0000","last_clean":"2024-02-05T08:19:41.828067+0000","last_became_active":"2024-02-05T08:19:44.902070+0000","last_became_peered":"2024-02-05T08:19:44.902070+0000","last_unstale":"2024-02-05T08:21:52.966730+0000","last_undegraded":"2024-02-05T08:21:52.966730+0000","last_fullsized":"2024-02-05T08:19:44.887805+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:07.027521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:07.027521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"236'27","reported_seq":632,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.969237+0000","last_change":"2024-02-05T08:19:44.912476+0000","last_active":"2024-02-05T08:21:52.969237+0000","last_peered":"2024-02-05T08:21:52.969237+0000","last_clean":"2024-02-05T08:18:29.610409+0000","last_became_active":"2024-02-05T08:19:44.912476+0000","last_became_peered":"2024-02-05T08:19:44.912476+0000","last_unstale":"2024-02-05T08:21:52.969237+0000","last_undegraded":"2024-02-05T08:19:44.888218+0000","last_fullsized":"2024-02-05T08:19:44.888087+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:18:40.384115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:40.384115+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"347'12888","reported_seq":12979,"reported_epoch":347,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:53.068408+0000","last_change":"2024-02-05T08:19:44.913164+0000","last_active":"2024-02-05T08:21:53.068408+0000","last_peered":"2024-02-05T08:21:53.068408+0000","last_clean":"2024-02-05T08:19:43.862508+0000","last_became_active":"2024-02-05T08:19:44.913164+0000","last_became_peered":"2024-02-05T08:19:44.913164+0000","last_unstale":"2024-02-05T08:21:53.068408+0000","last_undegraded":"2024-02-05T08:19:44.871956+0000","last_fullsized":"2024-02-05T08:19:44.871842+0000","mapping_epoch":324,"log_start":"329'7379","ondisk_log_start":"329'7379","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":5509,"log_dups_size":0,"ondisk_log_size":5509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.630594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226885632,"num_objects":3462,"num_object_clones":0,"num_object_copies":10386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12888,"num_write_kb":523200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":3462}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":132,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.968141+0000","last_change":"2024-02-05T08:20:38.715971+0000","last_active":"2024-02-05T08:21:52.968141+0000","last_peered":"2024-02-05T08:21:52.968141+0000","last_clean":"2024-02-05T08:21:52.968141+0000","last_became_active":"2024-02-05T08:18:34.627261+0000","last_became_peered":"2024-02-05T08:18:34.627261+0000","last_unstale":"2024-02-05T08:21:52.968141+0000","last_undegraded":"2024-02-05T08:21:52.968141+0000","last_fullsized":"2024-02-05T08:21:52.968141+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:38.715837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:20:38.715837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:00.975975+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":645,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.974877+0000","last_change":"2024-02-05T08:20:30.366634+0000","last_active":"2024-02-05T08:21:52.974877+0000","last_peered":"2024-02-05T08:21:52.974877+0000","last_clean":"2024-02-05T08:21:52.974877+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:21:52.974877+0000","last_undegraded":"2024-02-05T08:21:52.974877+0000","last_fullsized":"2024-02-05T08:21:52.974877+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:30.366511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:30.366511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":639,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:51.954735+0000","last_change":"2024-02-05T08:21:07.952159+0000","last_active":"2024-02-05T08:21:51.954735+0000","last_peered":"2024-02-05T08:21:51.954735+0000","last_clean":"2024-02-05T08:21:51.954735+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:21:51.954735+0000","last_undegraded":"2024-02-05T08:21:51.954735+0000","last_fullsized":"2024-02-05T08:21:51.954735+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:07.952031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:21:07.952031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:31.866127+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"236'19","reported_seq":689,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:51.956461+0000","last_change":"2024-02-05T08:19:44.926373+0000","last_active":"2024-02-05T08:21:51.956461+0000","last_peered":"2024-02-05T08:21:51.956461+0000","last_clean":"2024-02-05T08:19:41.831642+0000","last_became_active":"2024-02-05T08:19:44.926373+0000","last_became_peered":"2024-02-05T08:19:44.926373+0000","last_unstale":"2024-02-05T08:21:51.956461+0000","last_undegraded":"2024-02-05T08:19:44.921915+0000","last_fullsized":"2024-02-05T08:19:44.916232+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:36.561837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:36.561837+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.909944+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"347'13051","reported_seq":13224,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:53.289330+0000","last_change":"2024-02-05T08:20:36.341229+0000","last_active":"2024-02-05T08:21:53.289330+0000","last_peered":"2024-02-05T08:21:53.289330+0000","last_clean":"2024-02-05T08:21:53.289330+0000","last_became_active":"2024-02-05T08:19:07.346034+0000","last_became_peered":"2024-02-05T08:19:07.346034+0000","last_unstale":"2024-02-05T08:21:53.289330+0000","last_undegraded":"2024-02-05T08:21:53.289330+0000","last_fullsized":"2024-02-05T08:21:53.289330+0000","mapping_epoch":309,"log_start":"329'7084","ondisk_log_start":"329'7084","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'7452","last_scrub_stamp":"2024-02-05T08:20:36.340902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:36.340902+0000","objects_scrubbed":7382,"log_size":5967,"log_dups_size":0,"ondisk_log_size":5967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2388,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230883328,"num_objects":3523,"num_object_clones":0,"num_object_copies":10569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13051,"num_write_kb":530368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'130","reported_seq":408,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.967727+0000","last_change":"2024-02-05T08:21:07.292072+0000","last_active":"2024-02-05T08:21:52.967727+0000","last_peered":"2024-02-05T08:21:52.967727+0000","last_clean":"2024-02-05T08:21:52.967727+0000","last_became_active":"2024-02-05T08:19:01.817126+0000","last_became_peered":"2024-02-05T08:19:01.817126+0000","last_unstale":"2024-02-05T08:21:52.967727+0000","last_undegraded":"2024-02-05T08:21:52.967727+0000","last_fullsized":"2024-02-05T08:21:52.967727+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:07.291933+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:07.291933+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:30.946377+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":658,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.962508+0000","last_change":"2024-02-05T08:20:19.087445+0000","last_active":"2024-02-05T08:21:52.962508+0000","last_peered":"2024-02-05T08:21:52.962508+0000","last_clean":"2024-02-05T08:21:52.962508+0000","last_became_active":"2024-02-05T08:18:34.645067+0000","last_became_peered":"2024-02-05T08:18:34.645067+0000","last_unstale":"2024-02-05T08:21:52.962508+0000","last_undegraded":"2024-02-05T08:21:52.962508+0000","last_fullsized":"2024-02-05T08:21:52.962508+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:20:19.087309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.087309+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":52,"reported_epoch":346,"state":"active+undersized","last_fresh":"2024-02-05T08:21:51.955641+0000","last_change":"2024-02-05T08:19:44.914853+0000","last_active":"2024-02-05T08:21:51.955641+0000","last_peered":"2024-02-05T08:21:51.955641+0000","last_clean":"2024-02-05T08:04:44.536163+0000","last_became_active":"2024-02-05T08:19:44.914853+0000","last_became_peered":"2024-02-05T08:19:44.914853+0000","last_unstale":"2024-02-05T08:21:51.955641+0000","last_undegraded":"2024-02-05T08:21:51.955641+0000","last_fullsized":"2024-02-05T08:19:44.909093+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:45.429303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:45.429303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.909953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":366,"reported_epoch":347,"state":"active+undersized","last_fresh":"2024-02-05T08:21:52.962517+0000","last_change":"2024-02-05T08:19:44.915255+0000","last_active":"2024-02-05T08:21:52.962517+0000","last_peered":"2024-02-05T08:21:52.962517+0000","last_clean":"2024-02-05T08:19:41.829592+0000","last_became_active":"2024-02-05T08:19:44.915255+0000","last_became_peered":"2024-02-05T08:19:44.915255+0000","last_unstale":"2024-02-05T08:21:52.962517+0000","last_undegraded":"2024-02-05T08:21:52.962517+0000","last_fullsized":"2024-02-05T08:19:44.872335+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:10.372287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:19:10.372287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:55.630619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"236'31","reported_seq":671,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:51.956226+0000","last_change":"2024-02-05T08:19:44.922652+0000","last_active":"2024-02-05T08:21:51.956226+0000","last_peered":"2024-02-05T08:21:51.956226+0000","last_clean":"2024-02-05T08:19:41.830877+0000","last_became_active":"2024-02-05T08:19:44.922652+0000","last_became_peered":"2024-02-05T08:19:44.922652+0000","last_unstale":"2024-02-05T08:21:51.956226+0000","last_undegraded":"2024-02-05T08:19:44.913741+0000","last_fullsized":"2024-02-05T08:19:44.913632+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'31","last_scrub_stamp":"2024-02-05T08:19:24.342770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:24.342770+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.910016+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"346'12993","reported_seq":13082,"reported_epoch":346,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:21:52.718008+0000","last_change":"2024-02-05T08:19:44.915552+0000","last_active":"2024-02-05T08:21:52.718008+0000","last_peered":"2024-02-05T08:21:52.718008+0000","last_clean":"2024-02-05T08:19:43.841958+0000","last_became_active":"2024-02-05T08:19:44.915552+0000","last_became_peered":"2024-02-05T08:19:44.915552+0000","last_unstale":"2024-02-05T08:21:52.718008+0000","last_undegraded":"2024-02-05T08:19:44.912000+0000","last_fullsized":"2024-02-05T08:19:44.910014+0000","mapping_epoch":324,"log_start":"336'8500","ondisk_log_start":"336'8500","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":4493,"log_dups_size":0,"ondisk_log_size":4493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.909977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232980480,"num_objects":3555,"num_object_clones":0,"num_object_copies":10665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3555,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12993,"num_write_kb":529536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":3555}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"82'130","reported_seq":940,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.961037+0000","last_change":"2024-02-05T08:21:19.070269+0000","last_active":"2024-02-05T08:21:52.961037+0000","last_peered":"2024-02-05T08:21:52.961037+0000","last_clean":"2024-02-05T08:21:52.961037+0000","last_became_active":"2024-02-05T08:19:01.816744+0000","last_became_peered":"2024-02-05T08:19:01.816744+0000","last_unstale":"2024-02-05T08:21:52.961037+0000","last_undegraded":"2024-02-05T08:21:52.961037+0000","last_fullsized":"2024-02-05T08:21:52.961037+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:19.069725+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:19.069725+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:33.977459+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":614,"reported_epoch":346,"state":"active+undersized","last_fresh":"2024-02-05T08:21:51.956654+0000","last_change":"2024-02-05T08:19:44.927513+0000","last_active":"2024-02-05T08:21:51.956654+0000","last_peered":"2024-02-05T08:21:51.956654+0000","last_clean":"2024-02-05T08:19:41.831929+0000","last_became_active":"2024-02-05T08:19:44.927513+0000","last_became_peered":"2024-02-05T08:19:44.927513+0000","last_unstale":"2024-02-05T08:21:51.956654+0000","last_undegraded":"2024-02-05T08:21:51.956654+0000","last_fullsized":"2024-02-05T08:19:44.916139+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.282641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.282641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:21:53.909969+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":857,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.961880+0000","last_change":"2024-02-05T08:20:15.874649+0000","last_active":"2024-02-05T08:21:52.961880+0000","last_peered":"2024-02-05T08:21:52.961880+0000","last_clean":"2024-02-05T08:21:52.961880+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:21:52.961880+0000","last_undegraded":"2024-02-05T08:21:52.961880+0000","last_fullsized":"2024-02-05T08:21:52.961880+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:20:15.874421+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:20:15.874421+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":132,"reported_epoch":347,"state":"active+clean","last_fresh":"2024-02-05T08:21:52.967932+0000","last_change":"2024-02-05T08:20:20.712863+0000","last_active":"2024-02-05T08:21:52.967932+0000","last_peered":"2024-02-05T08:21:52.967932+0000","last_clean":"2024-02-05T08:21:52.967932+0000","last_became_active":"2024-02-05T08:18:34.663738+0000","last_became_peered":"2024-02-05T08:18:34.663738+0000","last_unstale":"2024-02-05T08:21:52.967932+0000","last_undegraded":"2024-02-05T08:21:52.967932+0000","last_fullsized":"2024-02-05T08:21:52.967932+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:20.711675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:20.711675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"236'20","reported_seq":737,"reported_epoch":346,"state":"active+clean","last_fresh":"2024-02-05T08:21:51.955888+0000","last_change":"2024-02-05T08:21:16.960225+0000","last_active":"2024-02-05T08:21:51.955888+0000","last_peered":"2024-02-05T08:21:51.955888+0000","last_clean":"2024-02-05T08:21:51.955888+0000","last_became_active":"2024-02-05T08:18:34.637413+0000","last_became_peered":"2024-02-05T08:18:34.637413+0000","last_unstale":"2024-02-05T08:21:51.955888+0000","last_undegraded":"2024-02-05T08:21:51.955888+0000","last_fullsized":"2024-02-05T08:21:51.955888+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:21:16.959305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:16.959305+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:29.189495+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3659399192,"num_objects":55839,"num_object_clones":0,"num_object_copies":167517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10485,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55839,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207081,"num_write_kb":8413377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10911363072,"data_stored":10911350856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84093,"ondisk_log_size":84093,"up":45,"acting":45,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1022,"num_read_kb":1017,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":7,"acting":7,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":32,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":85,"acting":85,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":77,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":5462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":766,"ondisk_log_size":766,"up":81,"acting":81,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80,"ondisk_log_size":80,"up":82,"acting":82,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":85,"acting":85,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037971,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1479928,"kb_used_data":898920,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92265224,"statfs":{"total":95995035648,"available":94479589376,"internally_reserved":0,"allocated":920494080,"data_stored":920189348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:20:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.3120000000000001,"15min":1.5760000000000001},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.0069999999999997,"5min":27.376000000000001,"15min":60.127000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.4419999999999999,"15min":1.661},"min":{"1min":0.254,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":4.2169999999999996,"5min":27.456,"15min":58.283000000000001},"last":0.40500000000000003}]},{"osd":1,"last update":"Mon Feb 5 08:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.544,"15min":1.512},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.22900000000000001},"max":{"1min":2.7810000000000001,"5min":27.518999999999998,"15min":38.399999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.083,"5min":1.4330000000000001,"15min":1.3680000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.22500000000000001},"max":{"1min":4.2409999999999997,"5min":27.503,"15min":38.195999999999998},"last":0.32200000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:20:57 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.071,"15min":1.034},"min":{"1min":0.27200000000000002,"5min":0.26100000000000001,"15min":0.24099999999999999},"max":{"1min":4.0949999999999998,"5min":7.5709999999999997,"15min":7.6779999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.2070000000000001,"15min":1.173},"min":{"1min":0.27200000000000002,"5min":0.26200000000000001,"15min":0.24299999999999999},"max":{"1min":4.7199999999999998,"5min":5.9400000000000004,"15min":8.9100000000000001},"last":0.63}]},{"osd":3,"last update":"Mon Feb 5 08:20:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.135,"15min":1.3640000000000001},"min":{"1min":0.31,"5min":0.29699999999999999,"15min":0.249},"max":{"1min":4.6840000000000002,"5min":5.984,"15min":58.515000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.244,"5min":1.0620000000000001,"15min":1.3520000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.27000000000000002},"max":{"1min":2.573,"5min":4.0960000000000001,"15min":58.225999999999999},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:20:57 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.1220000000000001,"15min":0.98699999999999999},"min":{"1min":0.26600000000000001,"5min":0.24099999999999999,"15min":0.22},"max":{"1min":3.9809999999999999,"5min":7.6340000000000003,"15min":49.601999999999997},"last":1.498},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.3100000000000001,"15min":1.0629999999999999},"min":{"1min":0.39100000000000001,"5min":0.26400000000000001,"15min":0.249},"max":{"1min":4.6360000000000001,"5min":7.8399999999999999,"15min":49.795999999999999},"last":1.681}]},{"osd":5,"last update":"Mon Feb 5 08:20:57 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.204,"15min":1.056},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.24299999999999999},"max":{"1min":3.044,"5min":5.1769999999999996,"15min":49.914000000000001},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.452,"15min":1.081},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.27000000000000002},"max":{"1min":4.0309999999999997,"5min":9.4459999999999997,"15min":52.256},"last":0.54300000000000004}]},{"osd":6,"last update":"Mon Feb 5 08:20:57 2024","interfaces":[{"interface":"back","average":{"1min":2.0699999999999998,"5min":2.2919999999999998,"15min":1.238},"min":{"1min":0.48999999999999999,"5min":0.28799999999999998,"15min":0.28599999999999998},"max":{"1min":5.8550000000000004,"5min":84.254999999999995,"15min":84.254999999999995},"last":0.439},{"interface":"front","average":{"1min":1.8520000000000001,"5min":2.347,"15min":1.2150000000000001},"min":{"1min":0.36499999999999999,"5min":0.32500000000000001,"15min":0.27300000000000002},"max":{"1min":7.1859999999999999,"5min":84.667000000000002,"15min":84.667000000000002},"last":2.3220000000000001}]}]},{"osd":6,"up_from":51,"seq":219043332313,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2059480,"kb_used_data":1572940,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91685672,"statfs":{"total":95995035648,"available":93886128128,"internally_reserved":0,"allocated":1610690560,"data_stored":1610404754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":498203078},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:21:35 2024","interfaces":[{"interface":"back","average":{"1min":2.23,"5min":1.996,"15min":1.889},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":14.836,"5min":76.123000000000005,"15min":102.819},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.80300000000000005,"5min":1.6779999999999999,"15min":1.845},"min":{"1min":0.26800000000000002,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":2.359,"5min":76.037000000000006,"15min":102.75},"last":0.69699999999999995}]},{"osd":1,"last update":"Mon Feb 5 08:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.90900000000000003,"15min":1.169},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.22700000000000001},"max":{"1min":1.1970000000000001,"5min":6.0229999999999997,"15min":23.773},"last":0.503},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.96299999999999997,"15min":1.1319999999999999},"min":{"1min":0.30299999999999999,"5min":0.26300000000000001,"15min":0.23400000000000001},"max":{"1min":3.0249999999999999,"5min":5.0209999999999999,"15min":23.719000000000001},"last":0.73699999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":1.1479999999999999,"15min":1.1819999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.1539999999999999,"5min":7.3499999999999996,"15min":22.503},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.044,"15min":1.0669999999999999},"min":{"1min":0.36399999999999999,"5min":0.27300000000000002,"15min":0.23400000000000001},"max":{"1min":1.494,"5min":6.9720000000000004,"15min":22.466000000000001},"last":0.56899999999999995}]},{"osd":3,"last update":"Mon Feb 5 08:21:35 2024","interfaces":[{"interface":"back","average":{"1min":2.1320000000000001,"5min":1.073,"15min":1.4430000000000001},"min":{"1min":0.29799999999999999,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":14.957000000000001,"5min":14.957000000000001,"15min":102.438},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.083,"5min":1.1399999999999999,"15min":1.452},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29099999999999998},"max":{"1min":3.1899999999999999,"5min":7.4000000000000004,"15min":102.288},"last":0.30599999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:21:35 2024","interfaces":[{"interface":"back","average":{"1min":5.0199999999999996,"5min":1.8660000000000001,"15min":1.042},"min":{"1min":0.34999999999999998,"5min":0.251,"15min":0.223},"max":{"1min":46.125,"5min":46.125,"15min":46.125},"last":0.44500000000000001},{"interface":"front","average":{"1min":5.6689999999999996,"5min":2.0619999999999998,"15min":1.139},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":48.817999999999998,"5min":48.817999999999998,"15min":48.817999999999998},"last":4.7149999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":1.1870000000000001,"15min":1.1100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.24199999999999999},"max":{"1min":2.8479999999999999,"5min":5.4340000000000002,"15min":43.643000000000001},"last":0.995},{"interface":"front","average":{"1min":1.8720000000000001,"5min":1.3580000000000001,"15min":1.1659999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":43.893000000000001},"last":3.5590000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:21:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.6339999999999999,"15min":0.96799999999999997},"min":{"1min":0.314,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":3.9510000000000001,"5min":24.728999999999999,"15min":24.728999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.8160000000000001,"15min":1.044},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.29299999999999998},"max":{"1min":4.492,"5min":21.91,"15min":21.91},"last":0.51500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1172192,"kb_used_data":617624,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92572960,"statfs":{"total":95995035648,"available":94794711040,"internally_reserved":0,"allocated":632446976,"data_stored":632002797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1659999999999999,"15min":1.625},"min":{"1min":0.221,"5min":0.183,"15min":0.183},"max":{"1min":1.0660000000000001,"5min":36.462000000000003,"15min":86.105000000000004},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.331,"15min":1.907},"min":{"1min":0.26300000000000001,"5min":0.21099999999999999,"15min":0.20699999999999999},"max":{"1min":8.0340000000000007,"5min":35.878,"15min":85.665000000000006},"last":0.40600000000000003}]},{"osd":2,"last update":"Mon Feb 5 08:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.84299999999999997,"15min":1.373},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.191},"max":{"1min":1.4930000000000001,"5min":4.2859999999999996,"15min":49.030999999999999},"last":2.855},{"interface":"front","average":{"1min":1.2210000000000001,"5min":0.92100000000000004,"15min":1.599},"min":{"1min":0.255,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":7.298,"5min":7.298,"15min":48.984000000000002},"last":2.5550000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.754,"15min":1.5189999999999999},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.20599999999999999},"max":{"1min":1.026,"5min":5.3289999999999997,"15min":49.067999999999998},"last":1.252},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.81699999999999995,"15min":1.46},"min":{"1min":0.32100000000000001,"5min":0.254,"15min":0.24199999999999999},"max":{"1min":2.3119999999999998,"5min":4.3529999999999998,"15min":48.734999999999999},"last":1.212}]},{"osd":4,"last update":"Mon Feb 5 08:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.4119999999999999,"15min":1.272},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.26000000000000001},"max":{"1min":1.2030000000000001,"5min":24.196999999999999,"15min":25.952000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":1.3100000000000001,"15min":1.3600000000000001},"min":{"1min":0.33700000000000002,"5min":0.31,"15min":0.27100000000000002},"max":{"1min":0.92100000000000004,"5min":24.215,"15min":25.922000000000001},"last":1.228}]},{"osd":5,"last update":"Mon Feb 5 08:18:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.492,"15min":1.3999999999999999},"min":{"1min":0.40100000000000002,"5min":0.39400000000000002,"15min":0.28699999999999998},"max":{"1min":3.1699999999999999,"5min":9.6820000000000004,"15min":54.552999999999997},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.296,"15min":1.4379999999999999},"min":{"1min":0.32000000000000001,"5min":0.311,"15min":0.28999999999999998},"max":{"1min":4.1239999999999997,"5min":6.0259999999999998,"15min":55.811},"last":1.1259999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.97399999999999998,"15min":1.2390000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.5069999999999999,"5min":3.6819999999999999,"15min":54.384},"last":1.196},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.0920000000000001,"15min":1.2470000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.34899999999999998},"max":{"1min":1.456,"5min":4.4370000000000003,"15min":55.210999999999999},"last":0.56399999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:18:52 2024","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.45,"15min":1.3089999999999999},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":5.7640000000000002,"5min":6.0579999999999998,"15min":13.749000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.3280000000000001,"15min":1.242},"min":{"1min":0.42399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":2.4020000000000001,"5min":4.6180000000000003,"15min":7.9640000000000004},"last":1.1579999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738615,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2955456,"kb_used_data":2238448,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":90789696,"statfs":{"total":95995035648,"available":92968648704,"internally_reserved":0,"allocated":2292170752,"data_stored":2291857595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":734198213},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:19:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.72199999999999998,"15min":1.79},"min":{"1min":0.185,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":1.4279999999999999,"5min":7.9580000000000002,"15min":66.772999999999996},"last":0.755},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.745,"15min":1.5409999999999999},"min":{"1min":0.27100000000000002,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":2.169,"5min":6.569,"15min":46.198999999999998},"last":1.016}]},{"osd":2,"last update":"Mon Feb 5 08:20:56 2024","interfaces":[{"interface":"back","average":{"1min":6.3049999999999997,"5min":1.869,"15min":1.9359999999999999},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":105.96599999999999,"5min":105.96599999999999,"15min":105.96599999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":5.8769999999999998,"5min":1.7589999999999999,"15min":2.0350000000000001},"min":{"1min":0.19900000000000001,"5min":0.182,"15min":0.182},"max":{"1min":106.137,"5min":106.137,"15min":106.137},"last":0.66600000000000004}]},{"osd":3,"last update":"Mon Feb 5 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":0.95599999999999996,"15min":1.8149999999999999},"min":{"1min":0.223,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":5.226,"5min":5.9029999999999996,"15min":67.888000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":1.113,"5min":0.88100000000000001,"15min":2.0840000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":5.6950000000000003,"5min":5.6950000000000003,"15min":66.891999999999996},"last":0.32800000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:20:51 2024","interfaces":[{"interface":"back","average":{"1min":2.536,"5min":1.4179999999999999,"15min":1.3839999999999999},"min":{"1min":0.34100000000000003,"5min":0.29299999999999998,"15min":0.26300000000000001},"max":{"1min":30.914000000000001,"5min":30.914000000000001,"15min":30.914000000000001},"last":3.5859999999999999},{"interface":"front","average":{"1min":2.859,"5min":1.609,"15min":1.3959999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.26900000000000002},"max":{"1min":30.93,"5min":30.93,"15min":30.93},"last":5.5510000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:21:12 2024","interfaces":[{"interface":"back","average":{"1min":2.9740000000000002,"5min":1.915,"15min":1.6479999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.28299999999999997},"max":{"1min":33.112000000000002,"5min":38.979999999999997,"15min":38.979999999999997},"last":0.75},{"interface":"front","average":{"1min":3.254,"5min":2.048,"15min":1.6240000000000001},"min":{"1min":0.435,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":33.137999999999998,"5min":38.960999999999999,"15min":38.960999999999999},"last":0.78000000000000003}]},{"osd":6,"last update":"Mon Feb 5 08:21:39 2024","interfaces":[{"interface":"back","average":{"1min":1.9890000000000001,"5min":1.278,"15min":1.333},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.30199999999999999},"max":{"1min":8.2370000000000001,"5min":8.2370000000000001,"15min":12.967000000000001},"last":5.5830000000000002},{"interface":"front","average":{"1min":2.0830000000000002,"5min":1.284,"15min":1.401},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":7.3730000000000002,"5min":7.3730000000000002,"15min":15.721},"last":2.919}]},{"osd":7,"last update":"Mon Feb 5 08:20:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":2.367,"15min":1.74},"min":{"1min":0.317,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.6079999999999997,"5min":68.241,"15min":68.241},"last":3.6739999999999999},{"interface":"front","average":{"1min":1.7010000000000001,"5min":2.48,"15min":1.804},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":6.1369999999999996,"5min":68.980000000000004,"15min":68.980000000000004},"last":5.5069999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411565,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1931072,"kb_used_data":1358448,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":91814080,"statfs":{"total":95995035648,"available":94017617920,"internally_reserved":0,"allocated":1391050752,"data_stored":1390737568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":586348998},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":1.244,"15min":2.1989999999999998},"min":{"1min":0.23499999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":6.181,"5min":30.731999999999999,"15min":107.155},"last":1.262},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.4159999999999999,"15min":2.3809999999999998},"min":{"1min":0.26900000000000002,"5min":0.22600000000000001,"15min":0.20799999999999999},"max":{"1min":6.468,"5min":30.838999999999999,"15min":107.221},"last":0.83799999999999997}]},{"osd":1,"last update":"Mon Feb 5 08:18:54 2024","interfaces":[{"interface":"back","average":{"1min":2.4540000000000002,"5min":1.167,"15min":1.5860000000000001},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.17999999999999999},"max":{"1min":35.695999999999998,"5min":35.695999999999998,"15min":55.942},"last":0.28299999999999997},{"interface":"front","average":{"1min":2.9209999999999998,"5min":1.581,"15min":1.752},"min":{"1min":0.216,"5min":0.216,"15min":0.193},"max":{"1min":30.765000000000001,"5min":30.765000000000001,"15min":55.960999999999999},"last":0.79700000000000004}]},{"osd":3,"last update":"Mon Feb 5 08:21:21 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":0.95799999999999996,"15min":1.879},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20499999999999999},"max":{"1min":5.1970000000000001,"5min":8.3629999999999995,"15min":68.108000000000004},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.876,"5min":0.81899999999999995,"15min":1.669},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":4.7519999999999998,"5min":6.6920000000000002,"15min":67.945999999999998},"last":0.69399999999999995}]},{"osd":4,"last update":"Mon Feb 5 08:21:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.226,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.27400000000000002},"max":{"1min":7.2549999999999999,"5min":7.6189999999999998,"15min":31.596},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.089,"5min":1.2490000000000001,"15min":1.343},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.252},"max":{"1min":2.6739999999999999,"5min":9.5060000000000002,"15min":31.57},"last":0.46200000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:21:11 2024","interfaces":[{"interface":"back","average":{"1min":1.829,"5min":1.718,"15min":1.6579999999999999},"min":{"1min":0.32600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":9.516,"5min":26.373000000000001,"15min":58.799999999999997},"last":0.496},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5529999999999999,"15min":1.5800000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.26900000000000002},"max":{"1min":9.3209999999999997,"5min":27.968,"15min":58.615000000000002},"last":2.3519999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:21:48 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":1.266,"15min":1.377},"min":{"1min":0.39300000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":7.1059999999999999,"5min":7.1059999999999999,"15min":15.859},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.252,"15min":1.2070000000000001},"min":{"1min":0.35999999999999999,"5min":0.317,"15min":0.317},"max":{"1min":4.6390000000000002,"5min":9.1739999999999995,"15min":9.1739999999999995},"last":0.72199999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.2030000000000001,"15min":1.2929999999999999},"min":{"1min":0.35299999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.1280000000000001,"5min":7.7530000000000001,"15min":15.986000000000001},"last":2.294},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.587,"15min":1.4950000000000001},"min":{"1min":0.45600000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":6.2080000000000002,"5min":11.945,"15min":15.951000000000001},"last":0.57799999999999996}]}]},{"osd":3,"up_from":27,"seq":115964117224,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1659904,"kb_used_data":1137268,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92085248,"statfs":{"total":95995035648,"available":94295293952,"internally_reserved":0,"allocated":1164562432,"data_stored":1164237566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":535165385},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.399,"15min":1.855},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":1.5960000000000001,"5min":52.831000000000003,"15min":63.43},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":1.546,"15min":1.998},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.4009999999999998,"5min":53.859999999999999,"15min":63.048999999999999},"last":1.3260000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.93500000000000005,"15min":1.5109999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.217},"max":{"1min":3.3119999999999998,"5min":23.154,"15min":51.405999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.89200000000000002,"15min":1.8260000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.17399999999999999},"max":{"1min":3.54,"5min":23.207000000000001,"15min":51.189999999999998},"last":2.1379999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.1519999999999999,"15min":1.724},"min":{"1min":0.23000000000000001,"5min":0.22,"15min":0.22},"max":{"1min":2.23,"5min":35.277000000000001,"15min":62.902999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.048,"15min":1.5249999999999999},"min":{"1min":0.23400000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.633,"5min":35.442999999999998,"15min":63.468000000000004},"last":4.1950000000000003}]},{"osd":4,"last update":"Mon Feb 5 08:21:46 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.90700000000000003,"15min":1.1359999999999999},"min":{"1min":0.28999999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.7329999999999997,"5min":6.298,"15min":7.7229999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.755,"5min":0.872,"15min":1.1599999999999999},"min":{"1min":0.34000000000000002,"5min":0.32900000000000001,"15min":0.29599999999999999},"max":{"1min":2.3610000000000002,"5min":5.8159999999999998,"15min":10.539999999999999},"last":1.274}]},{"osd":5,"last update":"Mon Feb 5 08:21:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0740000000000001,"15min":1.8240000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.255},"max":{"1min":6.343,"5min":11.519,"15min":117.432},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.86299999999999999,"5min":1.163,"15min":1.853},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.472,"5min":11.071,"15min":114.828},"last":1.0069999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:21:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":0.98799999999999999,"15min":1.2470000000000001},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.2149999999999999,"5min":3.6360000000000001,"15min":10.622},"last":1.214},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.056,"15min":1.2470000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.431,"5min":7.0439999999999996,"15min":7.8639999999999999},"last":0.77500000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:21:50 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.1100000000000001,"15min":1.798},"min":{"1min":0.41999999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":4.7629999999999999,"5min":6.3730000000000002,"15min":114.84999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.129,"5min":0.96099999999999997,"15min":1.8160000000000001},"min":{"1min":0.38800000000000001,"5min":0.38700000000000001,"15min":0.33600000000000002},"max":{"1min":4.4290000000000003,"5min":4.4290000000000003,"15min":114.871},"last":1.2989999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953699,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2434208,"kb_used_data":1839192,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91310944,"statfs":{"total":95995035648,"available":93502406656,"internally_reserved":0,"allocated":1883332608,"data_stored":1883026495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.3240000000000001,"15min":1.7909999999999999},"min":{"1min":0.26700000000000002,"5min":0.192,"15min":0.192},"max":{"1min":4.1420000000000003,"5min":25.600999999999999,"15min":73.078999999999994},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.585,"5min":1.385,"15min":1.7749999999999999},"min":{"1min":0.27600000000000002,"5min":0.23599999999999999,"15min":0.21099999999999999},"max":{"1min":4.3789999999999996,"5min":23.030999999999999,"15min":73.412000000000006},"last":3.0209999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.97299999999999998,"15min":1.486},"min":{"1min":0.30399999999999999,"5min":0.23300000000000001,"15min":0.21099999999999999},"max":{"1min":1.4590000000000001,"5min":8.3320000000000007,"15min":125.054},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.89300000000000002,"15min":1.387},"min":{"1min":0.309,"5min":0.25700000000000001,"15min":0.23499999999999999},"max":{"1min":1.1910000000000001,"5min":6.2370000000000001,"15min":125.08799999999999},"last":3.1699999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.0660000000000001,"15min":1.3759999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1849999999999996,"5min":4.8159999999999998,"15min":96.849999999999994},"last":0.32800000000000001},{"interface":"front","average":{"1min":1.599,"5min":1.123,"15min":1.4299999999999999},"min":{"1min":0.371,"5min":0.27100000000000002,"15min":0.26300000000000001},"max":{"1min":4.4290000000000003,"5min":5.6879999999999997,"15min":96.143000000000001},"last":3.0790000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":1.3220000000000001,"15min":1.986},"min":{"1min":0.312,"5min":0.312,"15min":0.24399999999999999},"max":{"1min":4.4550000000000001,"5min":8.9269999999999996,"15min":73.602999999999994},"last":3.0550000000000002},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1419999999999999,"15min":1.954},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.27200000000000002},"max":{"1min":3.2879999999999998,"5min":4.2599999999999998,"15min":74.817999999999998},"last":2.9550000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:21:17 2024","interfaces":[{"interface":"back","average":{"1min":1.774,"5min":1.365,"15min":1.0089999999999999},"min":{"1min":0.33400000000000002,"5min":0.29399999999999998,"15min":0.224},"max":{"1min":6.1909999999999998,"5min":9.157,"15min":34.167000000000002},"last":4.9279999999999999},{"interface":"front","average":{"1min":2.2029999999999998,"5min":1.3959999999999999,"15min":1.0660000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":7.851,"5min":7.851,"15min":36.051000000000002},"last":5.0209999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":2.194,"5min":1.4039999999999999,"15min":0.91000000000000003},"min":{"1min":0.42899999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.8929999999999998,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":3.1080000000000001},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.3,"15min":0.86099999999999999},"min":{"1min":0.40799999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.7810000000000001,"5min":4.2969999999999997,"15min":5.5049999999999999},"last":3.4780000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4339999999999999,"15min":0.96899999999999997},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.27000000000000002},"max":{"1min":4.282,"5min":12.683999999999999,"15min":12.683999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.407,"15min":0.94699999999999995},"min":{"1min":0.434,"5min":0.33900000000000002,"15min":0.27900000000000003},"max":{"1min":6.4180000000000001,"5min":6.4180000000000001,"15min":6.4180000000000001},"last":3.286}]}]},{"osd":5,"up_from":42,"seq":180388626653,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1724004,"kb_used_data":1111068,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92021148,"statfs":{"total":95995035648,"available":94229655552,"internally_reserved":0,"allocated":1137733632,"data_stored":1137440242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.7210000000000001,"15min":1.9830000000000001},"min":{"1min":0.246,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":3.0739999999999998,"5min":30.065999999999999,"15min":143.06299999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.6759999999999999,"15min":1.9279999999999999},"min":{"1min":0.27600000000000002,"5min":0.248,"15min":0.20499999999999999},"max":{"1min":4.0350000000000001,"5min":30.173999999999999,"15min":143.09700000000001},"last":0.39100000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:18:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.321,"15min":1.163},"min":{"1min":0.42799999999999999,"5min":0.22500000000000001,"15min":0.215},"max":{"1min":2.8090000000000002,"5min":5.8940000000000001,"15min":46.200000000000003},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.1859999999999999,"15min":1.23},"min":{"1min":0.29699999999999999,"5min":0.246,"15min":0.22900000000000001},"max":{"1min":4.7969999999999997,"5min":5.2210000000000001,"15min":46.343000000000004},"last":0.35099999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.282,"15min":1.177},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":17.09,"5min":17.09,"15min":17.09},"last":0.36599999999999999},{"interface":"front","average":{"1min":2.1600000000000001,"5min":1.458,"15min":1.298},"min":{"1min":0.33100000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":17.515999999999998,"5min":17.515999999999998,"15min":17.515999999999998},"last":0.746}]},{"osd":3,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":1.1679999999999999,"15min":1.76},"min":{"1min":0.34000000000000002,"5min":0.29199999999999998,"15min":0.28799999999999998},"max":{"1min":4.0819999999999999,"5min":7.165,"15min":143.03399999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.139,"15min":1.863},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.1030000000000002,"5min":5.8609999999999998,"15min":143.46799999999999},"last":3.617}]},{"osd":4,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1020000000000001,"15min":0.84199999999999997},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.7389999999999999,"5min":7.7389999999999999,"15min":7.7389999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.877,"5min":1.244,"15min":0.82599999999999996},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.23999999999999999},"max":{"1min":4.1349999999999998,"5min":7.2380000000000004,"15min":7.2380000000000004},"last":0.56399999999999995}]},{"osd":6,"last update":"Mon Feb 5 08:21:40 2024","interfaces":[{"interface":"back","average":{"1min":3.1179999999999999,"5min":1.6899999999999999,"15min":1.046},"min":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.26600000000000001},"max":{"1min":30.581,"5min":30.581,"15min":30.581},"last":1.145},{"interface":"front","average":{"1min":3.077,"5min":1.627,"15min":1.04},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.316},"max":{"1min":30.620999999999999,"5min":30.620999999999999,"15min":30.620999999999999},"last":0.76100000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:20:58 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.5089999999999999,"15min":1.034},"min":{"1min":0.35399999999999998,"5min":0.28799999999999998,"15min":0.25900000000000001},"max":{"1min":5.0250000000000004,"5min":7.0330000000000004,"15min":9.6940000000000008},"last":0.40300000000000002},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.464,"15min":0.97399999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":4.1440000000000001,"5min":9.2360000000000007,"15min":9.2360000000000007},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2244022272,"data_stored":2244018200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":645464064,"data_stored":645464064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1380319232,"data_stored":1380319232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1152385024,"data_stored":1152385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1848643584,"data_stored":1848639512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1132855296,"data_stored":1132855296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1592004608,"data_stored":1592000536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":915668992,"data_stored":915668992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:21:56.841 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:56.875 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:56.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:56.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:56.979 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:56.984 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:21:57.015 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:57.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:57.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:57.117 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:57.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:57.162 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:57.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:57.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:56 smithi134 ceph-mon[30086]: from='client.15885 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:21:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:56 smithi134 ceph-mon[30086]: pgmap v766: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 3.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.08k op/s; 10569/168111 objects degraded (6.287%) 2024-02-05T08:21:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:56 smithi134 ceph-mon[30086]: 10.1 scrub starts 2024-02-05T08:21:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:56 smithi134 ceph-mon[39278]: from='client.15885 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:21:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:56 smithi134 ceph-mon[39278]: pgmap v766: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 3.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.08k op/s; 10569/168111 objects degraded (6.287%) 2024-02-05T08:21:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:56 smithi134 ceph-mon[39278]: 10.1 scrub starts 2024-02-05T08:21:57.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:57.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:56 smithi196 ceph-mon[33018]: from='client.15885 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:21:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:56 smithi196 ceph-mon[33018]: pgmap v766: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 3.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.08k op/s; 10569/168111 objects degraded (6.287%) 2024-02-05T08:21:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:56 smithi196 ceph-mon[33018]: 10.1 scrub starts 2024-02-05T08:21:57.407 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:57.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:57.444 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:57.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:57.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:57.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:57.578 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:57.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:57.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:57.718 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:57.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:57.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:57.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:57.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:57.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:57.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:57.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:57.996 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:21:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:21:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:21:57.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:57.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:57.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:58.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:58.139 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:58.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:58.241 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:58.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:57 smithi134 ceph-mon[30086]: 1.6 scrub starts 2024-02-05T08:21:58.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:57 smithi134 ceph-mon[30086]: 1.6 scrub ok 2024-02-05T08:21:58.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:57 smithi134 ceph-mon[30086]: 3.1 scrub starts 2024-02-05T08:21:58.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:57 smithi134 ceph-mon[30086]: 3.1 scrub ok 2024-02-05T08:21:58.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:57 smithi134 ceph-mon[39278]: 1.6 scrub starts 2024-02-05T08:21:58.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:57 smithi134 ceph-mon[39278]: 1.6 scrub ok 2024-02-05T08:21:58.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:57 smithi134 ceph-mon[39278]: 3.1 scrub starts 2024-02-05T08:21:58.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:58 smithi134 ceph-mon[39278]: 3.1 scrub ok 2024-02-05T08:21:58.278 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:58.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:58.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:57 smithi196 ceph-mon[33018]: 1.6 scrub starts 2024-02-05T08:21:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:57 smithi196 ceph-mon[33018]: 1.6 scrub ok 2024-02-05T08:21:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:57 smithi196 ceph-mon[33018]: 3.1 scrub starts 2024-02-05T08:21:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:57 smithi196 ceph-mon[33018]: 3.1 scrub ok 2024-02-05T08:21:58.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:58.415 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:58.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:58.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:58.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:58.550 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:58.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:58.651 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:58.708 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:58.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:58.740 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:21:58.778 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:21:58.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:21:58.830 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:21:58.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:58.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:58.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:21:58.869 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:21:58.910 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:21:58.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:21:58.949 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:21:58.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:58.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:21:58.984 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:21:58.984 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:21:58.984 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:21:58.984 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:21:58.984 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:21:58.984 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:21:58.985 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:21:58.985 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:21:58.985 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:21:59.083 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:21:59.130 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:59.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:59.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:21:59.231 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:21:59.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:59.272 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:59.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:21:59.272 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:59 smithi134 ceph-mon[30086]: pgmap v767: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 32 active+undersized, 100 active+clean; 3.3 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s; 10157/164403 objects degraded (6.178%) 2024-02-05T08:21:59.273 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:59 smithi134 ceph-mon[30086]: 10.1 scrub ok 2024-02-05T08:21:59.273 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:59 smithi134 ceph-mon[30086]: 10.3 scrub starts 2024-02-05T08:21:59.273 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:59 smithi134 ceph-mon[30086]: 5.4 scrub starts 2024-02-05T08:21:59.273 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:59 smithi134 ceph-mon[30086]: 5.4 scrub ok 2024-02-05T08:21:59.273 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:21:59 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2724042004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:59.273 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:59 smithi134 ceph-mon[39278]: pgmap v767: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 32 active+undersized, 100 active+clean; 3.3 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s; 10157/164403 objects degraded (6.178%) 2024-02-05T08:21:59.273 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:59 smithi134 ceph-mon[39278]: 10.1 scrub ok 2024-02-05T08:21:59.273 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:59 smithi134 ceph-mon[39278]: 10.3 scrub starts 2024-02-05T08:21:59.273 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:59 smithi134 ceph-mon[39278]: 5.4 scrub starts 2024-02-05T08:21:59.273 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:59 smithi134 ceph-mon[39278]: 5.4 scrub ok 2024-02-05T08:21:59.273 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:21:59 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2724042004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:59 smithi196 ceph-mon[33018]: pgmap v767: 158 pgs: 1 active+clean+scrubbing, 25 active+undersized+degraded, 32 active+undersized, 100 active+clean; 3.3 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s; 10157/164403 objects degraded (6.178%) 2024-02-05T08:21:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:59 smithi196 ceph-mon[33018]: 10.1 scrub ok 2024-02-05T08:21:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:59 smithi196 ceph-mon[33018]: 10.3 scrub starts 2024-02-05T08:21:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:59 smithi196 ceph-mon[33018]: 5.4 scrub starts 2024-02-05T08:21:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:59 smithi196 ceph-mon[33018]: 5.4 scrub ok 2024-02-05T08:21:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:21:59 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2724042004' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:21:59.373 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:59.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:59.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:59.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:59.468 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:21:59.468 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:21:59.468 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:21:59.468 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108936 8.6 8:6ce9955c:::benchmark_data_smithi134_115836_object108935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:28.999547+0000", 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 570.26827128800005, 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.767948115, 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108936 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.469 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999547+0000", 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999547+0000", 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999548+0000", 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.195e-06 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.470 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999572+0000", 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3781000000000002e-05 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999583+0000", 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1143e-05 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:28.999591+0000", 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1140000000000006e-06 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.471 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.472 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.005098+0000", 2024-02-05T08:21:59.472 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0055070800000000001 2024-02-05T08:21:59.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.472 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.472 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.005332+0000", 2024-02-05T08:21:59.472 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023392899999999999 2024-02-05T08:21:59.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.472 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,7", 2024-02-05T08:21:59.472 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.005485+0000", 2024-02-05T08:21:59.472 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015303800000000001 2024-02-05T08:21:59.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.015634+0000", 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010149004 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.015665+0000", 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0966000000000001e-05 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.041481+0000", 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.025815647000000001 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.041495+0000", 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4079e-05 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666718+0000", 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.62522305499999997 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666730+0000", 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2157000000000001e-05 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.474 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.767398+0000", 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1006681920000001 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.767495+0000", 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6735000000000001e-05 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108967 8.6 8:6b4bec94:::benchmark_data_smithi134_115836_object108966:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.046490+0000", 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 570.22132811899996, 2024-02-05T08:21:59.475 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.731470512, 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108967 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046490+0000", 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.476 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046490+0000", 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046492+0000", 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7060000000000001e-06 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046518+0000", 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5791000000000001e-05 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046527+0000", 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3980000000000005e-06 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.046537+0000", 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0380999999999999e-05 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.053715+0000", 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0071773959999999999 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.053726+0000", 2024-02-05T08:21:59.478 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1117999999999999e-05 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666815+0000", 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.61308943100000002 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666824+0000", 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0059999999999998e-06 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.767788+0000", 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1009632439999999 2024-02-05T08:21:59.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.767916+0000", 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012850399999999999 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.768062+0000", 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014636200000000001 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777461+0000", 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0093981900000000007 2024-02-05T08:21:59.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777525+0000", 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4302999999999997e-05 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777717+0000", 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019194500000000001 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777765+0000", 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8143999999999998e-05 2024-02-05T08:21:59.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777864+0000", 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9068999999999996e-05 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777961+0000", 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6523999999999999e-05 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108968 8.6 8:680b4c21:::benchmark_data_smithi134_115836_object108967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.048233+0000", 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 570.21958561600002, 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7300381819999999, 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108968 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048233+0000", 2024-02-05T08:21:59.483 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048233+0000", 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048235+0000", 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.018e-06 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048260+0000", 2024-02-05T08:21:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5247000000000001e-05 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048272+0000", 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2483000000000001e-05 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.048345+0000", 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2766000000000001e-05 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.053751+0000", 2024-02-05T08:21:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0054057139999999998 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.053759+0000", 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0479999999999999e-06 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666852+0000", 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.61309340899999998 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666861+0000", 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5750000000000001e-06 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.770884+0000", 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1040229189999999 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.771021+0000", 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000137353 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.771230+0000", 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020888999999999999 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.777723+0000", 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00649338 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778025+0000", 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00030212499999999998 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778073+0000", 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7438000000000002e-05 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778116+0000", 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3133000000000002e-05 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778154+0000", 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7744999999999999e-05 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778271+0000", 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000116939 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108988 8.6 8:6273284f:::benchmark_data_smithi134_115836_object108987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.068824+0000", 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 570.19899378699995, 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7214295480000001, 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108988 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068824+0000", 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068824+0000", 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068826+0000", 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.35e-06 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068848+0000", 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2093e-05 2024-02-05T08:21:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068861+0000", 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2918000000000001e-05 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068871+0000", 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.009e-05 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068950+0000", 2024-02-05T08:21:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8747999999999999e-05 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.068962+0000", 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2023e-05 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666889+0000", 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.59792759500000003 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666898+0000", 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4479999999999994e-06 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778301+0000", 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.111403232 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778649+0000", 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00034766799999999998 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.778800+0000", 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000151101 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788785+0000", 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0099855810000000003 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788832+0000", 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6394000000000003e-05 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788885+0000", 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3412999999999998e-05 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788973+0000", 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7843000000000006e-05 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.789172+0000", 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019889199999999999 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790254+0000", 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0010821590000000001 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108994 8.6 8:69b35254:::benchmark_data_smithi134_115836_object108993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.072494+0000", 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 570.19532447400002, 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7180740800000001, 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108994 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072494+0000", 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072494+0000", 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072496+0000", 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.815e-06 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072522+0000", 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6954e-05 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072539+0000", 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6602000000000001e-05 2024-02-05T08:21:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072548+0000", 2024-02-05T08:21:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3070000000000008e-06 2024-02-05T08:21:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072616+0000", 2024-02-05T08:21:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7382e-05 2024-02-05T08:21:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072628+0000", 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1772e-05 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666926+0000", 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.59429891000000001 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666935+0000", 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0989999999999999e-06 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.782050+0000", 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1151151159999999 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.782294+0000", 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000243849 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.782456+0000", 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016257700000000001 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788847+0000", 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.006391112 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.788929+0000", 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1520000000000006e-05 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790307+0000", 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0013783059999999999 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790406+0000", 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9007999999999998e-05 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790467+0000", 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1057999999999997e-05 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790568+0000", 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010069300000000001 2024-02-05T08:21:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108995 8.6 8:6351071d:::benchmark_data_smithi134_115836_object108994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.072933+0000", 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 570.194885537, 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7266677289999999, 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108995 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072933+0000", 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072933+0000", 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072934+0000", 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0410000000000001e-06 2024-02-05T08:21:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072956+0000", 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2197000000000001e-05 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072971+0000", 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5152e-05 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.072979+0000", 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9689999999999994e-06 2024-02-05T08:21:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.073032+0000", 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3375000000000003e-05 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.073041+0000", 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0480000000000004e-06 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666963+0000", 2024-02-05T08:21:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.59392174399999997 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666972+0000", 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4500000000000004e-06 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.785402+0000", 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1184303840000001 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.785534+0000", 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013186 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,6,7", 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.785689+0000", 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000155487 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790323+0000", 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0046338509999999996 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790357+0000", 2024-02-05T08:21:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3657e-05 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.790612+0000", 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00025531600000000001 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.799422+0000", 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0088102289999999993 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.799489+0000", 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6593000000000004e-05 2024-02-05T08:21:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.799600+0000", 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111376 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:108980 8.5 8:ad067b35:::benchmark_data_smithi134_115836_object108979:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e188)", 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:29.663590+0000", 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 569.60422845999994, 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1061829890000001, 2024-02-05T08:21:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 108980 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.663590+0000", 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.663590+0000", 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.663591+0000", 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7680000000000001e-06 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.663619+0000", 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7832999999999999e-05 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.665311+0000", 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0016921149999999999 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.665334+0000", 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2626e-05 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666524+0000", 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011904350000000001 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:29.666534+0000", 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3009999999999995e-06 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.761805+0000", 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.095271168 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.763439+0000", 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001634251 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 4,6,7", 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.763616+0000", 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000176711 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769220+0000", 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0056040200000000004 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769258+0000", 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8101e-05 2024-02-05T08:21:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769293+0000", 2024-02-05T08:21:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4687999999999997e-05 2024-02-05T08:21:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769610+0000", 2024-02-05T08:21:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031678300000000001 2024-02-05T08:21:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769668+0000", 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8451e-05 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:30.769773+0000", 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104738 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:134904 8.e 8:7d534b82:::benchmark_data_smithi134_115836_object18244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.506668+0000", 2024-02-05T08:21:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 536.76115000699997, 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.19134764500000001, 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 134904 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506668+0000", 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506668+0000", 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506669+0000", 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4900000000000004e-07 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506674+0000", 2024-02-05T08:21:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0529999999999997e-06 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506688+0000", 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4287999999999999e-05 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.506696+0000", 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1709999999999996e-06 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.516764+0000", 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010068344999999999 2024-02-05T08:21:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.516879+0000", 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114751 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.516995+0000", 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115844 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.521656+0000", 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0046613660000000001 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.528669+0000", 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0070134309999999997 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.697929+0000", 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16926000699999999 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.697981+0000", 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1521999999999999e-05 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.698016+0000", 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4918e-05 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:134932 8.e 8:747554f1:::benchmark_data_smithi134_115836_object18272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.538807+0000", 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 536.72901106799998, 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.18031301799999999, 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 134932 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538807+0000", 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538807+0000", 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538808+0000", 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.33e-06 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538813+0000", 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9429999999999999e-06 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.694877+0000", 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15606357700000001 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.694895+0000", 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8104000000000001e-05 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.698658+0000", 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0037627429999999998 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.698821+0000", 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016332799999999999 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.698943+0000", 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012196 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.708664+0000", 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0097209140000000006 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.708709+0000", 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5241999999999998e-05 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.718868+0000", 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010159062999999999 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.719076+0000", 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020797 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.719120+0000", 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3844000000000002e-05 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15087.0:162972 8.8 e214/194 8:14305b79:::benchmark_data_smithi134_115836_object46312:head v 214'10137, mlcod=214'10135)", 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:37.286547+0000", 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 501.98127139600001, 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.187515187, 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286547+0000", 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286547+0000", 2024-02-05T08:21:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286547+0000", 2024-02-05T08:21:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8800000000000002e-07 2024-02-05T08:21:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286553+0000", 2024-02-05T08:21:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9819999999999996e-06 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286560+0000", 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1380000000000001e-06 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286567+0000", 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3600000000000001e-06 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286636+0000", 2024-02-05T08:21:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8868999999999994e-05 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.286659+0000", 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.304e-05 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.474023+0000", 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.187364261 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.474062+0000", 2024-02-05T08:21:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8949000000000003e-05 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "MOSDPGPush(8.8 214/194 [PushOp(8:1b8b759a:::benchmark_data_smithi134_115836_object58675:head, version: 173'3646, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:1b8b759a:::benchmark_data_smithi134_115836_object58675:head@173'3646, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:37.291477+0000", 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 501.97634083000003, 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.18357304299999999, 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.291477+0000", 2024-02-05T08:21:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.291477+0000", 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.291478+0000", 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0800000000000004e-07 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.291523+0000", 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5318999999999998e-05 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.464807+0000", 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.17328333600000001 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.464836+0000", 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9229000000000001e-05 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.474831+0000", 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0099948740000000008 2024-02-05T08:21:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.474844+0000", 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3338e-05 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:37.475050+0000", 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020613900000000001 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170535 8.c 8:30bf855a:::benchmark_data_smithi134_115836_object53875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:21:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.749814+0000", 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 492.51800395399999, 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.99481441999999998, 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170535 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749814+0000", 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749814+0000", 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749816+0000", 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.547e-06 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749821+0000", 2024-02-05T08:21:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8679999999999998e-06 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749834+0000", 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3169999999999999e-05 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749842+0000", 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3229999999999998e-06 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749901+0000", 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8464000000000001e-05 2024-02-05T08:21:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.749909+0000", 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8550000000000008e-06 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.733527+0000", 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.98361745899999997 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.733637+0000", 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109769 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.733763+0000", 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000126191 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744320+0000", 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010556640000000001 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744459+0000", 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013946999999999999 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744506+0000", 2024-02-05T08:21:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7512000000000002e-05 2024-02-05T08:21:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744579+0000", 2024-02-05T08:21:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2857999999999999e-05 2024-02-05T08:21:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744629+0000", 2024-02-05T08:21:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9293999999999999e-05 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170561 8.c 8:320abcbc:::benchmark_data_smithi134_115836_object53901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.770164+0000", 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 492.49765386199999, 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.97474936999999995, 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170561 2024-02-05T08:21:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770164+0000", 2024-02-05T08:21:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770164+0000", 2024-02-05T08:21:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770166+0000", 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6750000000000001e-06 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770187+0000", 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1461999999999999e-05 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770200+0000", 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2884000000000001e-05 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770208+0000", 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6839999999999996e-06 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770266+0000", 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8081000000000002e-05 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.770275+0000", 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4579999999999993e-06 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.736290+0000", 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.96601568100000001 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.736375+0000", 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5116000000000003e-05 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.736487+0000", 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111376 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744402+0000", 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0079155340000000001 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744684+0000", 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00028183599999999999 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744761+0000", 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7120999999999996e-05 2024-02-05T08:21:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744873+0000", 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011210299999999999 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744914+0000", 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0358999999999998e-05 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170591 8.c 8:3e90cc49:::benchmark_data_smithi134_115836_object53931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:21:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.795150+0000", 2024-02-05T08:21:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 492.47266831000002, 2024-02-05T08:21:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.949924875, 2024-02-05T08:21:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170591 2024-02-05T08:21:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795150+0000", 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795150+0000", 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795151+0000", 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2049999999999999e-06 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795156+0000", 2024-02-05T08:21:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.634e-06 2024-02-05T08:21:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795168+0000", 2024-02-05T08:21:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2119999999999999e-05 2024-02-05T08:21:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795181+0000", 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3655999999999999e-05 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795293+0000", 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011196600000000001 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795312+0000", 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8717000000000001e-05 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.738715+0000", 2024-02-05T08:21:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94340282600000003 2024-02-05T08:21:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.738798+0000", 2024-02-05T08:21:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3123999999999999e-05 2024-02-05T08:21:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:21:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.738868+0000", 2024-02-05T08:21:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9904e-05 2024-02-05T08:21:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744471+0000", 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0056029870000000002 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744956+0000", 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00048495400000000001 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744999+0000", 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3164999999999998e-05 2024-02-05T08:21:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745044+0000", 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4780999999999997e-05 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745075+0000", 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0836000000000001e-05 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170592 8.c 8:3f22b68d:::benchmark_data_smithi134_115836_object53932:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.795212+0000", 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 492.472605789, 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95563532299999998, 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170592 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795212+0000", 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795212+0000", 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795213+0000", 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3200000000000005e-07 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795216+0000", 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7470000000000001e-06 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795223+0000", 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9240000000000001e-06 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795228+0000", 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7729999999999996e-06 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795334+0000", 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000105321 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.795339+0000", 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7440000000000004e-06 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.741396+0000", 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94605645199999999 2024-02-05T08:21:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.741504+0000", 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107666 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.741620+0000", 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011629900000000001 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.744703+0000", 2024-02-05T08:21:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030836259999999999 2024-02-05T08:21:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750711+0000", 2024-02-05T08:21:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0060076019999999999 2024-02-05T08:21:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750749+0000", 2024-02-05T08:21:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8243e-05 2024-02-05T08:21:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750814+0000", 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4344999999999995e-05 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750848+0000", 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4048999999999999e-05 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170598 8.c 8:396b7832:::benchmark_data_smithi134_115836_object53938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:21:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.801793+0000", 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 492.466024866, 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95466330200000005, 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170598 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801793+0000", 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801793+0000", 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801794+0000", 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0699999999999999e-06 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801798+0000", 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9999999999999998e-06 2024-02-05T08:21:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801812+0000", 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3257e-05 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801820+0000", 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.5380000000000002e-06 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801881+0000", 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0494000000000001e-05 2024-02-05T08:21:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801890+0000", 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5859999999999994e-06 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745103+0000", 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94321233000000004 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745242+0000", 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000139357 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.745347+0000", 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000105308 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750885+0000", 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0055373130000000003 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756344+0000", 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.005459518 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756359+0000", 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5268999999999999e-05 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756414+0000", 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4469999999999999e-05 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756457+0000", 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2792e-05 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170600 8.c 8:3e055f0a:::benchmark_data_smithi134_115836_object53940:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.801861+0000", 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 492.46595766899998, 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95753421999999999, 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170600 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801861+0000", 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801861+0000", 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801861+0000", 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7099999999999997e-07 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801863+0000", 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4109999999999999e-06 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801869+0000", 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.789e-06 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801874+0000", 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3310000000000003e-06 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801917+0000", 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2317000000000002e-05 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.801924+0000", 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4080000000000001e-06 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.748074+0000", 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94614989999999999 2024-02-05T08:21:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.748198+0000", 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012395200000000001 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.748296+0000", 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8288000000000005e-05 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756466+0000", 2024-02-05T08:21:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0081697049999999993 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756527+0000", 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1192000000000006e-05 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759307+0000", 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0027796460000000002 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759358+0000", 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0918000000000003e-05 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759395+0000", 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6992000000000001e-05 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170603 8.c 8:335f5460:::benchmark_data_smithi134_115836_object53943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.805707+0000", 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 492.46211100699998, 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95838726600000002, 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170603 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805707+0000", 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805707+0000", 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805709+0000", 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.415e-06 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805713+0000", 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6020000000000002e-06 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805725+0000", 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1729000000000001e-05 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805732+0000", 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4839999999999998e-06 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805784+0000", 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1134000000000002e-05 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.805792+0000", 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6440000000000006e-06 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.750912+0000", 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94512030199999997 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.751005+0000", 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2283999999999999e-05 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.751107+0000", 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000101828 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756471+0000", 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0053649029999999999 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759438+0000", 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002966266 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764000+0000", 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0045619750000000002 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764055+0000", 2024-02-05T08:21:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5158000000000003e-05 2024-02-05T08:21:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764094+0000", 2024-02-05T08:21:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9542000000000002e-05 2024-02-05T08:21:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170608 8.c 8:33618a8a:::benchmark_data_smithi134_115836_object53948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.807038+0000", 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 492.46077977700003, 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95716985099999996, 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170608 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807038+0000", 2024-02-05T08:21:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807038+0000", 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807039+0000", 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4e-07 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807044+0000", 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9660000000000004e-06 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807056+0000", 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1437000000000001e-05 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807064+0000", 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2779999999999993e-06 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807115+0000", 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0725e-05 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.807122+0000", 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4449999999999999e-06 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.753804+0000", 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.94668188499999995 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.753906+0000", 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000102166 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.753985+0000", 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8432999999999999e-05 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756543+0000", 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0025579399999999999 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759479+0000", 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0029361249999999999 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764128+0000", 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0046493369999999999 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764168+0000", 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9558999999999999e-05 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764208+0000", 2024-02-05T08:21:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0714999999999999e-05 2024-02-05T08:21:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:170625 8.c 8:322ad108:::benchmark_data_smithi134_115836_object53965:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-02-05T08:21:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:46.834138+0000", 2024-02-05T08:21:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 492.43368055799999, 2024-02-05T08:21:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.93039029699999998, 2024-02-05T08:21:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 170625 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834138+0000", 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834138+0000", 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834138+0000", 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8100000000000001e-07 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834143+0000", 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0929999999999996e-06 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834158+0000", 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5057e-05 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834166+0000", 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2390000000000002e-06 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834213+0000", 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7029e-05 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:46.834219+0000", 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3319999999999997e-06 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756569+0000", 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.92234994100000001 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756684+0000", 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115053 2024-02-05T08:21:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,7", 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.756836+0000", 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015174599999999999 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759502+0000", 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0026656649999999998 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.759930+0000", 2024-02-05T08:21:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00042810999999999999 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764311+0000", 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0043816990000000002 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764492+0000", 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018032500000000001 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:47.764528+0000", 2024-02-05T08:21:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6127000000000002e-05 2024-02-05T08:21:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:21:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:21:59.598 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:21:59.598 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:21:59.632 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:21:59.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:59.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:21:59.734 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:59.768 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:59.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:59.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:21:59.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:21:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:21:59.913 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:21:59.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:00.014 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:00.048 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:00.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:00.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:00.150 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:00.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:00.185 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:00.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[30086]: 6.1f scrub starts 2024-02-05T08:22:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[30086]: 10.3 scrub ok 2024-02-05T08:22:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:22:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:22:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[30086]: osdmap e349: 8 total, 7 up, 8 in 2024-02-05T08:22:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[30086]: 6.1f scrub ok 2024-02-05T08:22:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 10157/164403 objects degraded (6.178%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:00.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[39278]: 6.1f scrub starts 2024-02-05T08:22:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[39278]: 10.3 scrub ok 2024-02-05T08:22:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:22:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:22:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[39278]: osdmap e349: 8 total, 7 up, 8 in 2024-02-05T08:22:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[39278]: 6.1f scrub ok 2024-02-05T08:22:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:00 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 10157/164403 objects degraded (6.178%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:00.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:00 smithi196 ceph-mon[33018]: 6.1f scrub starts 2024-02-05T08:22:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:00 smithi196 ceph-mon[33018]: 10.3 scrub ok 2024-02-05T08:22:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:00 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:22:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:00 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:22:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:00 smithi196 ceph-mon[33018]: osdmap e349: 8 total, 7 up, 8 in 2024-02-05T08:22:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:00 smithi196 ceph-mon[33018]: 6.1f scrub ok 2024-02-05T08:22:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:00 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 10157/164403 objects degraded (6.178%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:00.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:00.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:00.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:00.355 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:00.427 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:00.466 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:00.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:00.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:00.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:00.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:00.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:00.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:00.736 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:00.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:00.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:00.873 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:00.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:00.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:00.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:01.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:01.012 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:01.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:01.045 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T08:22:01.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:01.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:01.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:01.248 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:01.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:01 smithi134 ceph-mon[30086]: pgmap v769: 158 pgs: 2 active+clean+scrubbing, 25 active+undersized+degraded, 32 active+undersized, 99 active+clean; 3.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.70k op/s; 9349/148845 objects degraded (6.281%) 2024-02-05T08:22:01.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:01 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:22:01.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:01 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:22:01.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:01 smithi134 ceph-mon[39278]: pgmap v769: 158 pgs: 2 active+clean+scrubbing, 25 active+undersized+degraded, 32 active+undersized, 99 active+clean; 3.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.70k op/s; 9349/148845 objects degraded (6.281%) 2024-02-05T08:22:01.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:01 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:22:01.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:01 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:22:01.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:01.283 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:01.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:01 smithi196 ceph-mon[33018]: pgmap v769: 158 pgs: 2 active+clean+scrubbing, 25 active+undersized+degraded, 32 active+undersized, 99 active+clean; 3.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.70k op/s; 9349/148845 objects degraded (6.281%) 2024-02-05T08:22:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:01 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:22:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:01 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:22:01.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:01.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:01.427 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:01.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:01.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:01.563 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:01.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:01.665 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:01.698 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:01.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:01.748 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:22:01.749 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:22:01.759 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 6, 3] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-02-05T08:22:01.759 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:22:01.759 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 4, 5, 7, 6, 3] 2024-02-05T08:22:01.760 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd out 0 2024-02-05T08:22:01.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:01.835 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:01.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:01.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:01.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:01.982 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:01.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:02.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:02.124 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:02.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:02.225 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:02.262 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:02.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:02.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:02 smithi134 ceph-mon[30086]: osdmap e350: 8 total, 7 up, 8 in 2024-02-05T08:22:02.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:02 smithi134 ceph-mon[39278]: osdmap e350: 8 total, 7 up, 8 in 2024-02-05T08:22:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:02 smithi196 ceph-mon[33018]: osdmap e350: 8 total, 7 up, 8 in 2024-02-05T08:22:02.364 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:02.402 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:02.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:02.504 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:02.540 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:02.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:02.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:02.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:02.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:02.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:02.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:02.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:02.812 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:02.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:02.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:02.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:02.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:02.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:03.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:03.064 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:03.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:03.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:03.144 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:03.188 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:03.223 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:03.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:03 smithi196 ceph-mon[33018]: 4.11 scrub starts 2024-02-05T08:22:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:03 smithi196 ceph-mon[33018]: 4.11 scrub ok 2024-02-05T08:22:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:03 smithi196 ceph-mon[33018]: pgmap v771: 158 pgs: 2 active+clean+scrubbing, 25 active+undersized+degraded, 32 active+undersized, 99 active+clean; 3.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 933 op/s; 9349/148845 objects degraded (6.281%) 2024-02-05T08:22:03.325 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:03.364 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:03.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:03.379 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:03 smithi134 ceph-mon[30086]: 4.11 scrub starts 2024-02-05T08:22:03.379 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:03 smithi134 ceph-mon[30086]: 4.11 scrub ok 2024-02-05T08:22:03.379 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:03 smithi134 ceph-mon[30086]: pgmap v771: 158 pgs: 2 active+clean+scrubbing, 25 active+undersized+degraded, 32 active+undersized, 99 active+clean; 3.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 933 op/s; 9349/148845 objects degraded (6.281%) 2024-02-05T08:22:03.379 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:03 smithi134 ceph-mon[39278]: 4.11 scrub starts 2024-02-05T08:22:03.379 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:03 smithi134 ceph-mon[39278]: 4.11 scrub ok 2024-02-05T08:22:03.379 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:03 smithi134 ceph-mon[39278]: pgmap v771: 158 pgs: 2 active+clean+scrubbing, 25 active+undersized+degraded, 32 active+undersized, 99 active+clean; 3.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 933 op/s; 9349/148845 objects degraded (6.281%) 2024-02-05T08:22:03.466 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:03.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:03.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:03.606 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:03.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:03.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:03.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:03.749 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:22:03.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:03.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:03.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:03.880 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:03.908 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:03.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:04.010 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:04.053 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:04.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:04.155 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:04.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:04.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:04.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:04 smithi134 ceph-mon[30086]: 5.f scrub starts 2024-02-05T08:22:04.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:04 smithi134 ceph-mon[30086]: 5.f scrub ok 2024-02-05T08:22:04.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:04 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:22:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:04 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:22:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:04 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:22:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:04 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:22:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:04 smithi134 ceph-mon[39278]: 5.f scrub starts 2024-02-05T08:22:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:04 smithi134 ceph-mon[39278]: 5.f scrub ok 2024-02-05T08:22:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:04 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:22:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:04 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:22:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:04 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:22:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:04 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:22:04.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:04.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:22:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:22:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:04 smithi196 ceph-mon[33018]: 5.f scrub starts 2024-02-05T08:22:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:04 smithi196 ceph-mon[33018]: 5.f scrub ok 2024-02-05T08:22:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:04 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:22:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:04 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:22:04.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:04 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:22:04.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:04 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:22:04.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:04.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:04.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:04.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:04.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:04.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:04.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:04.593 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:04.633 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:04.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:04.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:04.735 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:04.769 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:04.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:04.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:04.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:04.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:05.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:05.043 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:05.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:05.144 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:05.189 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:05.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:05.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:05.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:05.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:05 smithi196 ceph-mon[33018]: 5.14 scrub starts 2024-02-05T08:22:05.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:05 smithi196 ceph-mon[33018]: 5.14 scrub ok 2024-02-05T08:22:05.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:05 smithi196 ceph-mon[33018]: 1.7 scrub starts 2024-02-05T08:22:05.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:05 smithi196 ceph-mon[33018]: 1.7 scrub ok 2024-02-05T08:22:05.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:05 smithi196 ceph-mon[33018]: pgmap v772: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 8193/132405 objects degraded (6.188%) 2024-02-05T08:22:05.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:05 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:05.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:05 smithi196 ceph-mon[33018]: osdmap e351: 8 total, 7 up, 8 in 2024-02-05T08:22:05.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:05 smithi196 ceph-mon[33018]: 3.1b deep-scrub starts 2024-02-05T08:22:05.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:05 smithi196 ceph-mon[33018]: 3.1b deep-scrub ok 2024-02-05T08:22:05.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:05 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 8193/132405 objects degraded (6.188%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:05.326 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:05.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:05.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:05.464 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:05.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:05.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:05.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[30086]: 5.14 scrub starts 2024-02-05T08:22:05.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[30086]: 5.14 scrub ok 2024-02-05T08:22:05.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[30086]: 1.7 scrub starts 2024-02-05T08:22:05.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[30086]: 1.7 scrub ok 2024-02-05T08:22:05.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[30086]: pgmap v772: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 8193/132405 objects degraded (6.188%) 2024-02-05T08:22:05.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:05.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[30086]: osdmap e351: 8 total, 7 up, 8 in 2024-02-05T08:22:05.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[30086]: 3.1b deep-scrub starts 2024-02-05T08:22:05.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[30086]: 3.1b deep-scrub ok 2024-02-05T08:22:05.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 8193/132405 objects degraded (6.188%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:05.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[39278]: 5.14 scrub starts 2024-02-05T08:22:05.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[39278]: 5.14 scrub ok 2024-02-05T08:22:05.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[39278]: 1.7 scrub starts 2024-02-05T08:22:05.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[39278]: 1.7 scrub ok 2024-02-05T08:22:05.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[39278]: pgmap v772: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 8193/132405 objects degraded (6.188%) 2024-02-05T08:22:05.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:05.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[39278]: osdmap e351: 8 total, 7 up, 8 in 2024-02-05T08:22:05.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[39278]: 3.1b deep-scrub starts 2024-02-05T08:22:05.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[39278]: 3.1b deep-scrub ok 2024-02-05T08:22:05.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:05 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 8193/132405 objects degraded (6.188%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:05.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:05.600 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:05.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:05.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:05.702 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:05.742 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:05.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:05.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:05.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:05.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:05.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:05.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:05.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:06.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:06.021 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:06.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:06.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:06.157 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:06.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:06.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:06.211 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:06 smithi134 ceph-mon[30086]: 4.19 scrub starts 2024-02-05T08:22:06.211 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:06 smithi134 ceph-mon[30086]: 4.19 scrub ok 2024-02-05T08:22:06.211 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:06 smithi134 ceph-mon[30086]: 3.11 scrub starts 2024-02-05T08:22:06.211 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:06 smithi134 ceph-mon[30086]: 3.11 scrub ok 2024-02-05T08:22:06.212 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:06 smithi134 ceph-mon[39278]: 4.19 scrub starts 2024-02-05T08:22:06.212 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:06 smithi134 ceph-mon[39278]: 4.19 scrub ok 2024-02-05T08:22:06.212 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:06 smithi134 ceph-mon[39278]: 3.11 scrub starts 2024-02-05T08:22:06.212 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:06 smithi134 ceph-mon[39278]: 3.11 scrub ok 2024-02-05T08:22:06.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:06.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:06.307 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:06.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:06.408 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:06.434 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:22:06.434 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:22:06.434 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.434 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15714.0:223917 10.8 e351/309 10:12a61490:::benchmark_data_smithi134_174861_object92455:head v 351'13686, mlcod=351'13682)", 2024-02-05T08:22:06.434 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:22:06.430963+0000", 2024-02-05T08:22:06.434 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.001788502, 2024-02-05T08:22:06.434 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023280620000000001, 2024-02-05T08:22:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:22:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.430963+0000", 2024-02-05T08:22:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.430963+0000", 2024-02-05T08:22:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:06.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.436 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:06.436 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.430964+0000", 2024-02-05T08:22:06.436 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6300000000000004e-07 2024-02-05T08:22:06.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.436 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:06.436 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.430971+0000", 2024-02-05T08:22:06.436 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7090000000000002e-06 2024-02-05T08:22:06.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.436 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:06.436 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.430982+0000", 2024-02-05T08:22:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1045e-05 2024-02-05T08:22:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.430990+0000", 2024-02-05T08:22:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8550000000000002e-06 2024-02-05T08:22:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.431066+0000", 2024-02-05T08:22:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5359999999999997e-05 2024-02-05T08:22:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.438 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:06.438 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.431085+0000", 2024-02-05T08:22:06.438 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9287999999999999e-05 2024-02-05T08:22:06.438 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:06.438 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:06.438 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:06.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.438 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15714.0:223906 10.f e351/309 10:f3f678cd:::benchmark_data_smithi134_174861_object92444:head v 351'14105, mlcod=351'14102)", 2024-02-05T08:22:06.438 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:22:06.411091+0000", 2024-02-05T08:22:06.438 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.021659918, 2024-02-05T08:22:06.438 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.022275335, 2024-02-05T08:22:06.438 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.411091+0000", 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.411091+0000", 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:06.440 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.411092+0000", 2024-02-05T08:22:06.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9599999999999998e-07 2024-02-05T08:22:06.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.440 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:06.440 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.411100+0000", 2024-02-05T08:22:06.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9750000000000007e-06 2024-02-05T08:22:06.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.440 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:06.440 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.411113+0000", 2024-02-05T08:22:06.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2798e-05 2024-02-05T08:22:06.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.411120+0000", 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3089999999999998e-06 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.411265+0000", 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014438 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.411288+0000", 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3478000000000001e-05 2024-02-05T08:22:06.441 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15714.0:223916 10.f e351/309 10:fef3dea4:::benchmark_data_smithi134_174861_object92454:head v 351'14106, mlcod=351'14103)", 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:22:06.421113+0000", 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.011638114999999999, 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012343854, 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.421113+0000", 2024-02-05T08:22:06.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:06.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.443 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:06.443 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.421113+0000", 2024-02-05T08:22:06.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:06.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.443 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:06.443 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.421114+0000", 2024-02-05T08:22:06.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6800000000000009e-07 2024-02-05T08:22:06.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.443 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:06.444 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.421123+0000", 2024-02-05T08:22:06.444 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.507e-06 2024-02-05T08:22:06.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.444 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:06.444 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.421136+0000", 2024-02-05T08:22:06.444 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3642000000000001e-05 2024-02-05T08:22:06.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.444 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:06.444 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.421144+0000", 2024-02-05T08:22:06.444 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0320000000000003e-06 2024-02-05T08:22:06.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.427943+0000", 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0067987880000000001 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.428040+0000", 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7277999999999999e-05 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15714.0:223924 10.f e351/309)", 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:22:06.428781+0000", 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 0.0039701739999999999, 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0047349719999999996, 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "queued for pg", 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.428781+0000", 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.446 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.428781+0000", 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.428782+0000", 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.54e-07 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.428792+0000", 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7759999999999992e-06 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.447 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.428802+0000", 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0197000000000001e-05 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:06.428809+0000", 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4880000000000001e-06 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 4 2024-02-05T08:22:06.448 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:22:06.453 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:06.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:06.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:06 smithi196 ceph-mon[33018]: 4.19 scrub starts 2024-02-05T08:22:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:06 smithi196 ceph-mon[33018]: 4.19 scrub ok 2024-02-05T08:22:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:06 smithi196 ceph-mon[33018]: 3.11 scrub starts 2024-02-05T08:22:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:06 smithi196 ceph-mon[33018]: 3.11 scrub ok 2024-02-05T08:22:06.589 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:06.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:06.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:06.691 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:06.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:06.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:06.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:06.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:06.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:06.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:06.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:07.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:07.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:07.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:07.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:07.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:07.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:07.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:07.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:07.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:07 smithi134 ceph-mon[30086]: pgmap v774: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 746 B/s rd, 0 B/s wr, 1.03k op/s; 8193/129861 objects degraded (6.309%) 2024-02-05T08:22:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:07 smithi134 ceph-mon[30086]: 10.5 scrub starts 2024-02-05T08:22:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:07 smithi134 ceph-mon[30086]: 5.e scrub starts 2024-02-05T08:22:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:07 smithi134 ceph-mon[30086]: 5.e scrub ok 2024-02-05T08:22:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:07 smithi134 ceph-mon[30086]: 10.6 scrub starts 2024-02-05T08:22:07.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a[30063]: 2024-02-05T08:22:07.124+0000 7f4523bf3700 -1 mon.a@0(leader).osd e351 definitely_dead 0 2024-02-05T08:22:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:07 smithi134 ceph-mon[39278]: pgmap v774: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 746 B/s rd, 0 B/s wr, 1.03k op/s; 8193/129861 objects degraded (6.309%) 2024-02-05T08:22:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:07 smithi134 ceph-mon[39278]: 10.5 scrub starts 2024-02-05T08:22:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:07 smithi134 ceph-mon[39278]: 5.e scrub starts 2024-02-05T08:22:07.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:07 smithi134 ceph-mon[39278]: 5.e scrub ok 2024-02-05T08:22:07.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:07 smithi134 ceph-mon[39278]: 10.6 scrub starts 2024-02-05T08:22:07.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:07.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:07.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:07.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:07.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:07.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:07.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:07.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:07 smithi196 ceph-mon[33018]: pgmap v774: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 2.6 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 746 B/s rd, 0 B/s wr, 1.03k op/s; 8193/129861 objects degraded (6.309%) 2024-02-05T08:22:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:07 smithi196 ceph-mon[33018]: 10.5 scrub starts 2024-02-05T08:22:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:07 smithi196 ceph-mon[33018]: 5.e scrub starts 2024-02-05T08:22:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:07 smithi196 ceph-mon[33018]: 5.e scrub ok 2024-02-05T08:22:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:07 smithi196 ceph-mon[33018]: 10.6 scrub starts 2024-02-05T08:22:07.663 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:07.709 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:07.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:07.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:07.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:07.854 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:07.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:07.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:07.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:07.994 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:07.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:07.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:08.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:22:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:22:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:22:08.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:08.098 INFO:teuthology.orchestra.run.smithi134.stderr:marked out osd.0. 2024-02-05T08:22:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:08.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:08.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:08.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:08.255 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:08.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:08.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:08.357 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:08.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:08.382 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:08.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:08.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:08.509 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:08.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:08.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[30086]: 10.5 scrub ok 2024-02-05T08:22:08.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[30086]: 10.6 scrub ok 2024-02-05T08:22:08.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[30086]: 5.1 scrub starts 2024-02-05T08:22:08.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[30086]: 5.1 scrub ok 2024-02-05T08:22:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1207737158' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-05T08:22:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-05T08:22:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[30086]: Client client.admin marked osd.0 out, while it was still marked up 2024-02-05T08:22:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[39278]: 10.5 scrub ok 2024-02-05T08:22:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[39278]: 10.6 scrub ok 2024-02-05T08:22:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[39278]: 5.1 scrub starts 2024-02-05T08:22:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[39278]: 5.1 scrub ok 2024-02-05T08:22:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1207737158' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-05T08:22:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-05T08:22:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:08 smithi134 ceph-mon[39278]: Client client.admin marked osd.0 out, while it was still marked up 2024-02-05T08:22:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:08 smithi196 ceph-mon[33018]: 10.5 scrub ok 2024-02-05T08:22:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:08 smithi196 ceph-mon[33018]: 10.6 scrub ok 2024-02-05T08:22:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:08 smithi196 ceph-mon[33018]: 5.1 scrub starts 2024-02-05T08:22:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:08 smithi196 ceph-mon[33018]: 5.1 scrub ok 2024-02-05T08:22:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:08 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1207737158' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-05T08:22:08.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:08 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-02-05T08:22:08.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:08 smithi196 ceph-mon[33018]: Client client.admin marked osd.0 out, while it was still marked up 2024-02-05T08:22:08.611 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:08.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:08.637 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:08.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:08.738 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:08.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:08.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:08.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:08.865 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:08.890 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:08.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:08.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:08.992 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:09.015 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:09.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:09.089 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:22:09.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:09.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:09.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:09.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:09.243 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:09.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:09.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:09.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:09.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:09.392 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:09.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:09.494 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:09.518 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:09.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:09.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:09.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[39278]: 3.b scrub starts 2024-02-05T08:22:09.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[39278]: 3.b scrub ok 2024-02-05T08:22:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[39278]: pgmap v775: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 932 op/s; 7861/126471 objects degraded (6.216%) 2024-02-05T08:22:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-05T08:22:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[39278]: osdmap e352: 8 total, 7 up, 7 in 2024-02-05T08:22:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[39278]: 6.2 deep-scrub starts 2024-02-05T08:22:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[39278]: 6.2 deep-scrub ok 2024-02-05T08:22:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/481737917' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:22:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:22:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:22:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:22:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:22:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[39278]: osdmap e353: 8 total, 7 up, 7 in 2024-02-05T08:22:09.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[30086]: 3.b scrub starts 2024-02-05T08:22:09.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[30086]: 3.b scrub ok 2024-02-05T08:22:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[30086]: pgmap v775: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 932 op/s; 7861/126471 objects degraded (6.216%) 2024-02-05T08:22:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-05T08:22:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[30086]: osdmap e352: 8 total, 7 up, 7 in 2024-02-05T08:22:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[30086]: 6.2 deep-scrub starts 2024-02-05T08:22:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[30086]: 6.2 deep-scrub ok 2024-02-05T08:22:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/481737917' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:22:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:22:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:22:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:22:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:22:09.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:09 smithi134 ceph-mon[30086]: osdmap e353: 8 total, 7 up, 7 in 2024-02-05T08:22:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:09 smithi196 ceph-mon[33018]: 3.b scrub starts 2024-02-05T08:22:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:09 smithi196 ceph-mon[33018]: 3.b scrub ok 2024-02-05T08:22:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:09 smithi196 ceph-mon[33018]: pgmap v775: 158 pgs: 25 active+undersized+degraded, 32 active+undersized, 101 active+clean; 2.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 932 op/s; 7861/126471 objects degraded (6.216%) 2024-02-05T08:22:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:09 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-02-05T08:22:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:09 smithi196 ceph-mon[33018]: osdmap e352: 8 total, 7 up, 7 in 2024-02-05T08:22:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:09 smithi196 ceph-mon[33018]: 6.2 deep-scrub starts 2024-02-05T08:22:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:09 smithi196 ceph-mon[33018]: 6.2 deep-scrub ok 2024-02-05T08:22:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:09 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/481737917' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:22:09.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:09 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:22:09.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:09 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:22:09.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:09 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:22:09.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:09 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:22:09.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:09 smithi196 ceph-mon[33018]: osdmap e353: 8 total, 7 up, 7 in 2024-02-05T08:22:09.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:09.643 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:09.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:09.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:09.744 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:09.768 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:09.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:09.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:09.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:09.895 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:09.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:09.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:10.021 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:10.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:10.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:10.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:10.160 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:10.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:10.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:10.262 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:10.300 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:10.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:10.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:10.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:10 smithi134 ceph-mon[30086]: 6.5 scrub starts 2024-02-05T08:22:10.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:10 smithi134 ceph-mon[30086]: 6.5 scrub ok 2024-02-05T08:22:10.441 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:10 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 7861/126471 objects degraded (6.216%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:10.441 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:10 smithi134 ceph-mon[30086]: osdmap e354: 8 total, 7 up, 7 in 2024-02-05T08:22:10.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:10 smithi134 ceph-mon[39278]: 6.5 scrub starts 2024-02-05T08:22:10.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:10 smithi134 ceph-mon[39278]: 6.5 scrub ok 2024-02-05T08:22:10.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:10 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 7861/126471 objects degraded (6.216%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:10.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:10 smithi134 ceph-mon[39278]: osdmap e354: 8 total, 7 up, 7 in 2024-02-05T08:22:10.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:10.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:10.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:10.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:10 smithi196 ceph-mon[33018]: 6.5 scrub starts 2024-02-05T08:22:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:10 smithi196 ceph-mon[33018]: 6.5 scrub ok 2024-02-05T08:22:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:10 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 7861/126471 objects degraded (6.216%), 25 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:10 smithi196 ceph-mon[33018]: osdmap e354: 8 total, 7 up, 7 in 2024-02-05T08:22:10.581 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:10.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:10.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:10.683 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:10.721 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:10.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:10.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:10.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:10.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:10.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:10.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:10.996 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:10.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:11.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:11.144 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:11.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:11.187 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[30086]: pgmap v778: 158 pgs: 9 unknown, 38 peering, 21 active+undersized+degraded, 25 active+undersized, 65 active+clean; 2.3 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 4759/112893 objects degraded (4.215%) 2024-02-05T08:22:11.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[30086]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 351'14168 2024-02-05T08:22:11.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[30086]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 351'13846 2024-02-05T08:22:11.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[30086]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 351'14006 2024-02-05T08:22:11.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[30086]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 351'14420 2024-02-05T08:22:11.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[30086]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 351'13985 2024-02-05T08:22:11.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[30086]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 351'14164 2024-02-05T08:22:11.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[30086]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 351'14085 2024-02-05T08:22:11.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[30086]: Health check failed: Reduced data availability: 11 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-02-05T08:22:11.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[39278]: pgmap v778: 158 pgs: 9 unknown, 38 peering, 21 active+undersized+degraded, 25 active+undersized, 65 active+clean; 2.3 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 4759/112893 objects degraded (4.215%) 2024-02-05T08:22:11.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[39278]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 351'14168 2024-02-05T08:22:11.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[39278]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 351'13846 2024-02-05T08:22:11.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[39278]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 351'14006 2024-02-05T08:22:11.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[39278]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 351'14420 2024-02-05T08:22:11.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[39278]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 351'13985 2024-02-05T08:22:11.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[39278]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 351'14164 2024-02-05T08:22:11.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[39278]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 351'14085 2024-02-05T08:22:11.189 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:11 smithi134 ceph-mon[39278]: Health check failed: Reduced data availability: 11 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-02-05T08:22:11.190 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T08:22:11.247 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:11.301 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:11.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:11.403 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:11.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:11.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:11.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:11 smithi196 ceph-mon[33018]: pgmap v778: 158 pgs: 9 unknown, 38 peering, 21 active+undersized+degraded, 25 active+undersized, 65 active+clean; 2.3 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 4759/112893 objects degraded (4.215%) 2024-02-05T08:22:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:11 smithi196 ceph-mon[33018]: 10.c starting backfill to osd.3 from (0'0,0'0] MAX to 351'14168 2024-02-05T08:22:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:11 smithi196 ceph-mon[33018]: 10.8 starting backfill to osd.5 from (0'0,0'0] MAX to 351'13846 2024-02-05T08:22:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:11 smithi196 ceph-mon[33018]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 351'14006 2024-02-05T08:22:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:11 smithi196 ceph-mon[33018]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 351'14420 2024-02-05T08:22:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:11 smithi196 ceph-mon[33018]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 351'13985 2024-02-05T08:22:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:11 smithi196 ceph-mon[33018]: 10.2 starting backfill to osd.3 from (0'0,0'0] MAX to 351'14164 2024-02-05T08:22:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:11 smithi196 ceph-mon[33018]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 351'14085 2024-02-05T08:22:11.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:11 smithi196 ceph-mon[33018]: Health check failed: Reduced data availability: 11 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-02-05T08:22:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:11.586 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:11.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:11.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:11.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:11.730 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:11.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:11.756 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:11.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:11.898 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:11.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:11.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:12.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:12.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:12.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:12.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:12.184 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:12.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:12.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:12.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:12.327 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:12.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:12.429 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:12.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:12.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:12.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:12 smithi134 ceph-mon[30086]: osdmap e355: 8 total, 7 up, 7 in 2024-02-05T08:22:12.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:12 smithi134 ceph-mon[39278]: osdmap e355: 8 total, 7 up, 7 in 2024-02-05T08:22:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:12 smithi196 ceph-mon[33018]: osdmap e355: 8 total, 7 up, 7 in 2024-02-05T08:22:12.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:12.602 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:12.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:12.704 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:12.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:12.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:12.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:12.845 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:12.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:12.882 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:12.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:12.983 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:13.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:13.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:13.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:13.119 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:13.156 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:13.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:13.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:13.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:13.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:13.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:13.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:13.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:13.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:13.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:13 smithi134 ceph-mon[30086]: pgmap v781: 158 pgs: 9 unknown, 38 peering, 21 active+undersized+degraded, 25 active+undersized, 65 active+clean; 2.3 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.13k op/s; 4759/112893 objects degraded (4.215%) 2024-02-05T08:22:13.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:13 smithi134 ceph-mon[30086]: osdmap e356: 8 total, 7 up, 7 in 2024-02-05T08:22:13.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:13 smithi134 ceph-mon[39278]: pgmap v781: 158 pgs: 9 unknown, 38 peering, 21 active+undersized+degraded, 25 active+undersized, 65 active+clean; 2.3 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.13k op/s; 4759/112893 objects degraded (4.215%) 2024-02-05T08:22:13.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:13 smithi134 ceph-mon[39278]: osdmap e356: 8 total, 7 up, 7 in 2024-02-05T08:22:13.536 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:13.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:13 smithi196 ceph-mon[33018]: pgmap v781: 158 pgs: 9 unknown, 38 peering, 21 active+undersized+degraded, 25 active+undersized, 65 active+clean; 2.3 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.13k op/s; 4759/112893 objects degraded (4.215%) 2024-02-05T08:22:13.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:13 smithi196 ceph-mon[33018]: osdmap e356: 8 total, 7 up, 7 in 2024-02-05T08:22:13.573 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:13.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:13.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:13.675 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:13.713 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:13.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:13.804 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:22:13.815 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:13.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:13.891 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:13.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:13.993 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:14.072 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:14.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:14.175 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:14.192 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:22:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:22:14.212 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:14.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:14.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:14.314 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:14.348 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:14.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:14.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:14.486 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:14.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:14.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:14.488 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:14 smithi134 ceph-mon[30086]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:22:14.488 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:14 smithi134 ceph-mon[30086]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 8 pgs peering) 2024-02-05T08:22:14.488 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:14 smithi134 ceph-mon[39278]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:22:14.488 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:14 smithi134 ceph-mon[39278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 8 pgs peering) 2024-02-05T08:22:14.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:14 smithi196 ceph-mon[33018]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T08:22:14.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:14 smithi196 ceph-mon[33018]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 8 pgs peering) 2024-02-05T08:22:14.588 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:14.630 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:14.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:14.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:14.732 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:14.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:14.771 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:14.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:14.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:14.918 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:14.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:14.977 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:15.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:15.059 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:15.110 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:15.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:15.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:15.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:15 smithi134 ceph-mon[30086]: pgmap v783: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfill_toofull, 28 active+clean+remapped; 2.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 855 B/s rd, 0 B/s wr, 768 op/s; 6049/101865 objects degraded (5.938%); 20224/101865 objects misplaced (19.854%); 20 MiB/s, 325 objects/s recovering 2024-02-05T08:22:15.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:15 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:15.243 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:15 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 6049/101865 objects degraded (5.938%), 21 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:15.243 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:15 smithi134 ceph-mon[39278]: pgmap v783: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfill_toofull, 28 active+clean+remapped; 2.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 855 B/s rd, 0 B/s wr, 768 op/s; 6049/101865 objects degraded (5.938%); 20224/101865 objects misplaced (19.854%); 20 MiB/s, 325 objects/s recovering 2024-02-05T08:22:15.243 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:15 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:15.243 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:15 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 6049/101865 objects degraded (5.938%), 21 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:15.250 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:15.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:15.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:15.352 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:15.391 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:15.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:15.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:15.493 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:15.527 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:15.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:15.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:15 smithi196 ceph-mon[33018]: pgmap v783: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfill_toofull, 28 active+clean+remapped; 2.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 855 B/s rd, 0 B/s wr, 768 op/s; 6049/101865 objects degraded (5.938%); 20224/101865 objects misplaced (19.854%); 20 MiB/s, 325 objects/s recovering 2024-02-05T08:22:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:15 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:15 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 6049/101865 objects degraded (5.938%), 21 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:15.629 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:15.667 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:15.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:15.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:15.769 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:15.812 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:15.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:15.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:15.948 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:15.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:15.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:16.050 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:16.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:16.092 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:16.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:16.193 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:16.230 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:16.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:16.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:16.332 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:16.370 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:16.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:16.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:16.511 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:16.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:16.613 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:16.652 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:16.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:16.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:16.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:16.795 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:16.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:16.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:16.935 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:16.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:17.037 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:17.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:17.079 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:17.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:17.180 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:17.217 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:17.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:17.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:17.319 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:17.360 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:17.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:17.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:17.506 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:17.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:17.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:17 smithi134 ceph-mon[30086]: pgmap v784: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfill_toofull, 28 active+clean+remapped; 2.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 613 op/s; 6049/101865 objects degraded (5.938%); 20224/101865 objects misplaced (19.854%); 16 MiB/s, 259 objects/s recovering 2024-02-05T08:22:17.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:17 smithi134 ceph-mon[39278]: pgmap v784: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfill_toofull, 28 active+clean+remapped; 2.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 613 op/s; 6049/101865 objects degraded (5.938%); 20224/101865 objects misplaced (19.854%); 16 MiB/s, 259 objects/s recovering 2024-02-05T08:22:17.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:17 smithi196 ceph-mon[33018]: pgmap v784: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 1 active+undersized+degraded+remapped+backfill_toofull, 28 active+clean+remapped; 2.1 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 613 op/s; 6049/101865 objects degraded (5.938%); 20224/101865 objects misplaced (19.854%); 16 MiB/s, 259 objects/s recovering 2024-02-05T08:22:17.609 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:17.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:17.646 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:17.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:17.747 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:17.789 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:17.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:17.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:17.905 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:22:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:22:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:22:17.905 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:22:17.929 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:17.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:17.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:17.953 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:22:17.994 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:22:18.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:18.038 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:22:18.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:18.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:18.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:18.077 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:22:18.116 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:22:18.154 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:22:18.154 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:22:18.154 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:22:18.154 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:22:18.154 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:22:18.155 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:22:18.155 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:22:18.155 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:22:18.155 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:22:18.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:18.209 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:18.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:18.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:18.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:18.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:18.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:18.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:18.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:18.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:18.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:18.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:18 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1568267022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:22:18.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:18 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1568267022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:22:18.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:18 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1568267022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:22:18.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:18.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:18.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:18.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:18.725 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:18.760 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:18.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:18.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:18.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:18.897 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:18.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:18.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:18.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:19.027 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:22:19.028 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:22:19.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:19.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:19.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:19.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:19.190 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:19.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:19.263 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:19 smithi134 ceph-mon[30086]: pgmap v785: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 2.0 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 526 B/s rd, 0 B/s wr, 663 op/s; 5637/97401 objects degraded (5.787%); 19107/97401 objects misplaced (19.617%); 12 MiB/s, 199 objects/s recovering 2024-02-05T08:22:19.263 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:19 smithi134 ceph-mon[39278]: pgmap v785: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 2.0 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 526 B/s rd, 0 B/s wr, 663 op/s; 5637/97401 objects degraded (5.787%); 19107/97401 objects misplaced (19.617%); 12 MiB/s, 199 objects/s recovering 2024-02-05T08:22:19.291 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:19.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:19.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:19.432 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:19.474 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:19.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:19.539 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 7, 6, 3] out_osds: [0] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-02-05T08:22:19.539 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:22:19.539 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_3 2024-02-05T08:22:19.540 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:22:19.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:19 smithi196 ceph-mon[33018]: pgmap v785: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 2.0 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 526 B/s rd, 0 B/s wr, 663 op/s; 5637/97401 objects degraded (5.787%); 19107/97401 objects misplaced (19.617%); 12 MiB/s, 199 objects/s recovering 2024-02-05T08:22:19.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:19.608 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:19.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:19.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:19.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:19.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:19.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:19.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:19.844 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:19.859 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:22:19.859 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:22:19.912 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:19.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:20.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:20.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:20.054 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:20.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:20.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:20.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:20.195 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:20.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:20.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:20.334 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:20.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:20.436 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:20.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:20.476 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:20.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:20.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:20 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 5637/97401 objects degraded (5.787%), 21 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:20.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:20 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1101568385' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:22:20.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:20 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:22:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:20 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 5637/97401 objects degraded (5.787%), 21 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:20 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1101568385' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:22:20.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:20 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:22:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:20 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 5637/97401 objects degraded (5.787%), 21 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:20 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1101568385' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:22:20.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:20 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:22:20.577 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:20.615 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:20.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:20.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:20.733 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T08:22:20.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:20.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:20.814 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:20.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:20.882 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:22:20.898 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:20.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:21.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:21.029 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:22:21.043 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:21.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:21.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:21.181 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:21.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:21.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:21 smithi134 ceph-mon[30086]: pgmap v786: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s; 4379/84585 objects degraded (5.177%); 14926/84585 objects misplaced (17.646%); 30 MiB/s, 478 objects/s recovering 2024-02-05T08:22:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:21 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:22:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:21 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:22:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:21 smithi134 ceph-mon[30086]: osdmap e357: 8 total, 7 up, 7 in 2024-02-05T08:22:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:21 smithi134 ceph-mon[39278]: pgmap v786: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s; 4379/84585 objects degraded (5.177%); 14926/84585 objects misplaced (17.646%); 30 MiB/s, 478 objects/s recovering 2024-02-05T08:22:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:21 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:22:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:21 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:22:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:21 smithi134 ceph-mon[39278]: osdmap e357: 8 total, 7 up, 7 in 2024-02-05T08:22:21.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:21.336 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:21.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:21.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:21.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:21.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:21.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:21 smithi196 ceph-mon[33018]: pgmap v786: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s; 4379/84585 objects degraded (5.177%); 14926/84585 objects misplaced (17.646%); 30 MiB/s, 478 objects/s recovering 2024-02-05T08:22:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:21 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:22:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:21 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:22:21.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:21 smithi196 ceph-mon[33018]: osdmap e357: 8 total, 7 up, 7 in 2024-02-05T08:22:21.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:21.620 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:21.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:21.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:21.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:21.760 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:21.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:21.861 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:21.897 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:21.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:21.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:22.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:22.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:22.138 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:22.174 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:22.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:22.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:22.313 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:22.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:22.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:22.415 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:22.432 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:22.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:22.477 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:22.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:22.578 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:22.615 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:22.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:22.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:22.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:22.755 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:22.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:22.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:22.857 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:22.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:22.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:23.016 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:23.043 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:23.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:23.075 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:23.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:23.179 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:23.222 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:23.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:23.270 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:23 smithi134 ceph-mon[30086]: pgmap v788: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 945 B/s rd, 0 B/s wr, 969 op/s; 4379/84585 objects degraded (5.177%); 14926/84585 objects misplaced (17.646%); 27 MiB/s, 430 objects/s recovering 2024-02-05T08:22:23.270 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:23 smithi134 ceph-mon[39278]: pgmap v788: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 945 B/s rd, 0 B/s wr, 969 op/s; 4379/84585 objects degraded (5.177%); 14926/84585 objects misplaced (17.646%); 27 MiB/s, 430 objects/s recovering 2024-02-05T08:22:23.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:23.358 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:23.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:23.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:23.460 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:23.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:23.496 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:23.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:23 smithi196 ceph-mon[33018]: pgmap v788: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.7 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 945 B/s rd, 0 B/s wr, 969 op/s; 4379/84585 objects degraded (5.177%); 14926/84585 objects misplaced (17.646%); 27 MiB/s, 430 objects/s recovering 2024-02-05T08:22:23.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:23.637 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:23.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:23.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:23.740 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:23.778 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:23.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:23.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:23.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:23.916 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:23.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:23.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:23.950 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:22:24.018 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:24.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:24.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:24.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:24.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:24.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:24.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:24.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:24.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:22:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:22:24.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:24.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:24.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:24.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:24.451 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:24.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:24.553 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:24.578 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:24.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:24.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:24.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:24.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:24.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:24 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:22:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:24 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:22:24.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:24 smithi196 ceph-mon[33018]: osdmap e358: 8 total, 7 up, 7 in 2024-02-05T08:22:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:24.831 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:24.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:24.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:24.958 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:24 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:22:24.958 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:24 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:22:24.958 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:24 smithi134 ceph-mon[30086]: osdmap e358: 8 total, 7 up, 7 in 2024-02-05T08:22:24.958 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:24 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:22:24.959 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:24 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:22:24.959 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:24 smithi134 ceph-mon[39278]: osdmap e358: 8 total, 7 up, 7 in 2024-02-05T08:22:24.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:24.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:24.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:25.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:25.088 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:25.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:25.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:25.218 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:25.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:25.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:25.320 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:25.367 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:25.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:25.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:25.506 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:25.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:25.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:25.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:25.643 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:25.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:25.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:25.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:25 smithi134 ceph-mon[30086]: pgmap v789: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 2912/67620 objects degraded (4.306%); 10276/67620 objects misplaced (15.197%); 33 MiB/s, 520 objects/s recovering 2024-02-05T08:22:25.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:25 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:25.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:25 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 2912/67620 objects degraded (4.306%), 21 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:25.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:25 smithi134 ceph-mon[30086]: osdmap e359: 8 total, 7 up, 7 in 2024-02-05T08:22:25.706 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:25 smithi134 ceph-mon[39278]: pgmap v789: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 2912/67620 objects degraded (4.306%); 10276/67620 objects misplaced (15.197%); 33 MiB/s, 520 objects/s recovering 2024-02-05T08:22:25.706 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:25 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:25.706 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:25 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 2912/67620 objects degraded (4.306%), 21 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:25.706 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:25 smithi134 ceph-mon[39278]: osdmap e359: 8 total, 7 up, 7 in 2024-02-05T08:22:25.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:25.759 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:22:25.782 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:25.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:25.802 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:22:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:25 smithi196 ceph-mon[33018]: pgmap v789: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s; 2912/67620 objects degraded (4.306%); 10276/67620 objects misplaced (15.197%); 33 MiB/s, 520 objects/s recovering 2024-02-05T08:22:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:25 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:25 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 2912/67620 objects degraded (4.306%), 21 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:25 smithi196 ceph-mon[33018]: osdmap e359: 8 total, 7 up, 7 in 2024-02-05T08:22:25.840 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:22:25.874 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:22:25.884 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:25.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:25.931 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:25.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:25.931 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:22:25.975 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:22:26.018 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:22:26.019 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:22:26.019 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:22:26.019 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:22:26.019 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:22:26.019 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:22:26.019 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:22:26.019 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:22:26.019 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:22:26.032 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:26.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:26.072 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:26.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:26.174 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:26.205 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:26.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:26.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:26.307 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:26.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:26.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:26.445 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:26.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:26.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:26.583 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:26.615 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:26.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:26.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:26 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3486786471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:22:26.886 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:26.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:26.963 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:26 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3486786471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:22:26.963 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:26 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3486786471' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:22:26.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:27.023 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:27.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:27.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:27.125 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:27.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:27.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:27.246 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:22:27.250 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:22:27.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:27.300 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:27.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:27.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:27.441 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:27.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:27.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:27.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:27.561 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:22:27.561 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:22:27.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:22:27.561 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:22:27.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.562 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:131934 8.f 8:fd4b76ad:::benchmark_data_smithi134_115836_object15274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:22:27.562 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:58.583316+0000", 2024-02-05T08:22:27.562 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 568.97367199600001, 2024-02-05T08:22:27.562 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15811914799999999, 2024-02-05T08:22:27.562 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.562 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.562 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.562 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:27.562 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:27.562 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 131934 2024-02-05T08:22:27.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.562 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.562 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583316+0000", 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583316+0000", 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583318+0000", 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.593e-06 2024-02-05T08:22:27.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583324+0000", 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3620000000000002e-06 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583342+0000", 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7396000000000001e-05 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.583351+0000", 2024-02-05T08:22:27.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7429999999999997e-06 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.589777+0000", 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0064258070000000004 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.589907+0000", 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000129913 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:22:27.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.590038+0000", 2024-02-05T08:22:27.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013120100000000001 2024-02-05T08:22:27.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.595790+0000", 2024-02-05T08:22:27.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0057520330000000001 2024-02-05T08:22:27.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596232+0000", 2024-02-05T08:22:27.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000442075 2024-02-05T08:22:27.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600111+0000", 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003878898 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.603530+0000", 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0034189889999999999 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.741275+0000", 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13774504400000001 2024-02-05T08:22:27.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.741435+0000", 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000160094 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:131938 8.f 8:fc014979:::benchmark_data_smithi134_115836_object15278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:58.588880+0000", 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 568.96810814599996, 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15328541100000001, 2024-02-05T08:22:27.568 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 131938 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588880+0000", 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588880+0000", 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588882+0000", 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6500000000000001e-06 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588888+0000", 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0839999999999997e-06 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588902+0000", 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4208e-05 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588912+0000", 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7839999999999998e-06 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.592812+0000", 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003899881 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.592939+0000", 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012735700000000001 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.593065+0000", 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000125659 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.599122+0000", 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0060574239999999996 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600149+0000", 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001027095 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600698+0000", 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00054859500000000003 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.741935+0000", 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.141237063 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.742052+0000", 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000117544 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.742165+0000", 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011306699999999999 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:131939 8.f 8:fbb4e894:::benchmark_data_smithi134_115836_object15279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:22:27.574 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:58.588930+0000", 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 568.96805793500005, 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15448906300000001, 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 131939 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588930+0000", 2024-02-05T08:22:27.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588930+0000", 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588931+0000", 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0800000000000005e-07 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588933+0000", 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5749999999999999e-06 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588941+0000", 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9489999999999996e-06 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.588947+0000", 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8749999999999997e-06 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596264+0000", 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0073165060000000004 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596396+0000", 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013257000000000001 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596529+0000", 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013312400000000001 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600168+0000", 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036390340000000002 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600367+0000", 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019905500000000001 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600806+0000", 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00043826 2024-02-05T08:22:27.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.743237+0000", 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14243134299999999 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.743303+0000", 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6328999999999995e-05 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.743419+0000", 2024-02-05T08:22:27.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011593499999999999 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:134926 8.f 8:f72340dc:::benchmark_data_smithi134_115836_object18266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.535476+0000", 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 565.02151258200001, 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16460388500000001, 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:27.581 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 134926 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535476+0000", 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535476+0000", 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535477+0000", 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8160000000000001e-06 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535482+0000", 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9400000000000001e-06 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535494+0000", 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1657999999999999e-05 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535503+0000", 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7129999999999994e-06 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535674+0000", 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017146300000000001 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535806+0000", 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000131707 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,3,4", 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.535947+0000", 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014141300000000001 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.539323+0000", 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033756630000000001 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.539337+0000", 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4443e-05 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.540469+0000", 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011320169999999999 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.699796+0000", 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15932692800000001 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.699972+0000", 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000175513 2024-02-05T08:22:27.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.700079+0000", 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107611 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15087.0:134935 8.d e199/157 8:bc5808d6:::benchmark_data_smithi134_115836_object18275:head v 199'8633, mlcod=199'8631)", 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.544759+0000", 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 565.01222880600005, 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15565003399999999, 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.587 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.588 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.544759+0000", 2024-02-05T08:22:27.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.544759+0000", 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.544760+0000", 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.06e-07 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.544764+0000", 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1629999999999998e-06 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.694991+0000", 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15022724000000001 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.694999+0000", 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2339999999999994e-06 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.695129+0000", 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012998799999999999 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.695151+0000", 2024-02-05T08:22:27.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1145000000000002e-05 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.700293+0000", 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0051418979999999998 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.700409+0000", 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011686 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:162662 8.f 8:f41fb899:::benchmark_data_smithi134_115836_object46002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:36.809228+0000", 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 530.74775994399999, 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16428367499999999, 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 162662 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809228+0000", 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809228+0000", 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809230+0000", 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.488e-06 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809239+0000", 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2180000000000005e-06 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809248+0000", 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4040000000000001e-06 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809255+0000", 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5119999999999997e-06 2024-02-05T08:22:27.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809371+0000", 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115905 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809483+0000", 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000112136 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.809632+0000", 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000149285 2024-02-05T08:22:27.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.815527+0000", 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0058951819999999997 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.819817+0000", 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042893899999999997 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973380+0000", 2024-02-05T08:22:27.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15356336900000001 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973474+0000", 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3459000000000001e-05 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973512+0000", 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8327000000000001e-05 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:162673 8.f 8:f0c11af9:::benchmark_data_smithi134_115836_object46013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:36.816303+0000", 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 530.740685525, 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.157358891, 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 162673 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.599 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816303+0000", 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816303+0000", 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816304+0000", 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2449999999999999e-06 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816308+0000", 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2679999999999996e-06 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816318+0000", 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4779999999999997e-06 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816324+0000", 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.725e-06 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816371+0000", 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6983000000000001e-05 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816476+0000", 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000104721 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.816581+0000", 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010494500000000001 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.823858+0000", 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0072768090000000004 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.823889+0000", 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1165999999999999e-05 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973555+0000", 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14966643499999999 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973629+0000", 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3936000000000005e-05 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973662+0000", 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.218e-05 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:162674 8.f 8:f8e2bae8:::benchmark_data_smithi134_115836_object46014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-02-05T08:22:27.604 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:36.818247+0000", 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 530.73874147000004, 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.155548574, 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 162674 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818247+0000", 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818247+0000", 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818248+0000", 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.113e-06 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818251+0000", 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.647e-06 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818262+0000", 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0519e-05 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.818269+0000", 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3679999999999996e-06 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.819987+0000", 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017176610000000001 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.820094+0000", 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000106597 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.820209+0000", 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114962 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.823966+0000", 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0037573390000000002 2024-02-05T08:22:27.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.825554+0000", 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001587647 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973698+0000", 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14814437499999999 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973765+0000", 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7377000000000002e-05 2024-02-05T08:22:27.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.973795+0000", 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9969000000000001e-05 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:197385 8.f 8:f14f197d:::benchmark_data_smithi134_115836_object80725:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:14:07.714003+0000", 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 499.84298502500002, 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14987962099999999, 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.610 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 197385 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714003+0000", 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714003+0000", 2024-02-05T08:22:27.611 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714004+0000", 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.173e-06 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714008+0000", 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8130000000000002e-06 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714019+0000", 2024-02-05T08:22:27.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1114999999999999e-05 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.714026+0000", 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2690000000000002e-06 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.717045+0000", 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030182009999999999 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.717183+0000", 2024-02-05T08:22:27.613 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013826499999999999 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.717300+0000", 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000116941 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.724371+0000", 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0070706450000000004 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.726733+0000", 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023622299999999999 2024-02-05T08:22:27.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.863757+0000", 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13702392899999999 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.863841+0000", 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4518999999999995e-05 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.863883+0000", 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1520999999999998e-05 2024-02-05T08:22:27.615 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:197386 8.f 8:ffb8808e:::benchmark_data_smithi134_115836_object80726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:14:07.715023+0000", 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 499.84196546200002, 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15136122299999999, 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:27.616 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 197386 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715023+0000", 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715023+0000", 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715023+0000", 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.6700000000000003e-07 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715027+0000", 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7100000000000001e-06 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715038+0000", 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1262e-05 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.618 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.715046+0000", 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7919999999999993e-06 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.720250+0000", 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0052037560000000004 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.721564+0000", 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001314052 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:22:27.619 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.721676+0000", 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000112118 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.726778+0000", 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0051023309999999999 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.728765+0000", 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0019865550000000001 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.620 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866283+0000", 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.137518106 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866340+0000", 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7144e-05 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:14:07.866384+0000", 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3630000000000001e-05 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.621 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15714.0:122285 10.5 10:a179a804:::benchmark_data_smithi134_174861_object122284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e331)", 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:20:40.528061+0000", 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 107.028927344, 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15752057999999999, 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15714", 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 122285 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.622 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.528061+0000", 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.528061+0000", 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.528062+0000", 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.133e-06 2024-02-05T08:22:27.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.528084+0000", 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2480000000000002e-05 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.528097+0000", 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2492e-05 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.528104+0000", 2024-02-05T08:22:27.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5159999999999997e-06 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.531602+0000", 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0034971310000000001 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.531732+0000", 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013086200000000001 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.531872+0000", 2024-02-05T08:22:27.625 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000139111 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.535303+0000", 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.003431945 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.539419+0000", 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0041160019999999997 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.685480+0000", 2024-02-05T08:22:27.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14606095899999999 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.685540+0000", 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9494999999999999e-05 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.685581+0000", 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1454e-05 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.628 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(osd.4.0:83066 10.e e331/309 10:715af47c:::benchmark_data_smithi134_174861_object20715:head v 331'8086, mlcod=331'8081)", 2024-02-05T08:22:27.628 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:20:40.531490+0000", 2024-02-05T08:22:27.628 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 107.025497852, 2024-02-05T08:22:27.628 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16206124399999999, 2024-02-05T08:22:27.628 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.628 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.628 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:27.629 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:27.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.531490+0000", 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.531490+0000", 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.531491+0000", 2024-02-05T08:22:27.630 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1200000000000002e-07 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.531498+0000", 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3839999999999999e-06 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.531509+0000", 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0369e-05 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.531517+0000", 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9370000000000004e-06 2024-02-05T08:22:27.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.533841+0000", 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0023243809999999999 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.533883+0000", 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2314999999999997e-05 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.693497+0000", 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15961314800000001 2024-02-05T08:22:27.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.693552+0000", 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4898000000000002e-05 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(osd.4.0:83067 10.e e331/309 10:715b7719:::benchmark_data_smithi134_174861_object2817:head v 331'8087, mlcod=331'8081)", 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:20:40.533749+0000", 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 107.023239198, 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.164406884, 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.633 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.533749+0000", 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.533749+0000", 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.533750+0000", 2024-02-05T08:22:27.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6800000000000009e-07 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.533757+0000", 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2130000000000002e-06 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.533766+0000", 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.3680000000000008e-06 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.533775+0000", 2024-02-05T08:22:27.635 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1780000000000003e-06 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.536268+0000", 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0024932769999999999 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.536287+0000", 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8612999999999999e-05 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.636 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.698113+0000", 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16182670699999999 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.698156+0000", 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2559999999999999e-05 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15714.0:122291 10.5 10:af51fe4c:::benchmark_data_smithi134_174861_object122290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e331)", 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:20:40.535626+0000", 2024-02-05T08:22:27.637 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 107.021362326, 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.150073071, 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15714", 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 122291 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.535626+0000", 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.535626+0000", 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.535627+0000", 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.054e-06 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.535650+0000", 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3164999999999999e-05 2024-02-05T08:22:27.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.640 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.640 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.535663+0000", 2024-02-05T08:22:27.640 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2879e-05 2024-02-05T08:22:27.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.640 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.640 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.535672+0000", 2024-02-05T08:22:27.640 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.969e-06 2024-02-05T08:22:27.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.640 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.640 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.535732+0000", 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0424999999999998e-05 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.535865+0000", 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000133091 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.536071+0000", 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000205873 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.641 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.539492+0000", 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00342078 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.540876+0000", 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00138346 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.685621+0000", 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.144745177 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.685667+0000", 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6347999999999997e-05 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.685699+0000", 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1850000000000002e-05 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(osd.4.0:83068 10.e e331/309 10:715ca208:::benchmark_data_smithi134_174861_object16581:head v 331'8088, mlcod=331'8081)", 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:20:40.536696+0000", 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 107.020292553, 2024-02-05T08:22:27.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.161502696, 2024-02-05T08:22:27.644 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.644 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.644 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.536696+0000", 2024-02-05T08:22:27.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.536696+0000", 2024-02-05T08:22:27.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.536696+0000", 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5199999999999996e-07 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.536702+0000", 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2870000000000004e-06 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.536710+0000", 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2349999999999999e-06 2024-02-05T08:22:27.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.536716+0000", 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9529999999999996e-06 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.538585+0000", 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001869105 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.538603+0000", 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8573e-05 2024-02-05T08:22:27.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.698159+0000", 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15955525600000001 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.698198+0000", 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9635000000000003e-05 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(osd.4.0:83069 10.e e331/309 10:715e238b:::benchmark_data_smithi134_174861_object25954:head v 331'8089, mlcod=331'8081)", 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:20:40.539373+0000", 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 107.017614927, 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.158857573, 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.539373+0000", 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.648 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.539373+0000", 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.539374+0000", 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0210000000000001e-06 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.539383+0000", 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7159999999999992e-06 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.539395+0000", 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1582e-05 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.539404+0000", 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1530000000000003e-06 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.540711+0000", 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001306981 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.540727+0000", 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6521999999999999e-05 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.698201+0000", 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.157473524 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.698231+0000", 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0074e-05 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.651 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(osd.4.0:83070 10.e e331/309 10:71602309:::benchmark_data_smithi134_174861_object56777:head v 331'8090, mlcod=331'8081)", 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:20:40.542254+0000", 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 107.014734557, 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15602084699999999, 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.542254+0000", 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.542254+0000", 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.542254+0000", 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7199999999999998e-07 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.542261+0000", 2024-02-05T08:22:27.653 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8580000000000002e-06 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.684975+0000", 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.142713434 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.684992+0000", 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7056000000000001e-05 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.685243+0000", 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00025148500000000002 2024-02-05T08:22:27.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.685264+0000", 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0602999999999999e-05 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.698233+0000", 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012969105 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:20:40.698274+0000", 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1533999999999999e-05 2024-02-05T08:22:27.655 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15714.0:225875 10.5 10:ae0287c1:::benchmark_data_smithi134_174861_object94413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:22:08.098469+0000", 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 19.458518903000002, 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2056242849999999, 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15714", 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:22:27.656 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 225875 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.098469+0000", 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.098469+0000", 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.098471+0000", 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5740000000000001e-06 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.098477+0000", 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7570000000000001e-06 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.098489+0000", 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2299e-05 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.098498+0000", 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7309999999999999e-06 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.098632+0000", 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013439300000000001 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.098644+0000", 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2462e-05 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:09.089258+0000", 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.99061398899999997 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:09.089274+0000", 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5298000000000002e-05 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.278287+0000", 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.189013651 2024-02-05T08:22:27.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.278409+0000", 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000121873 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3,7", 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.279583+0000", 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0011733690000000001 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.291927+0000", 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012344152000000001 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.299369+0000", 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0074425869999999996 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.299621+0000", 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00025158599999999999 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.303912+0000", 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042907869999999999 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.303973+0000", 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1457000000000005e-05 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.304094+0000", 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012032 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.663 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15714.0:225882 10.8 10:1fdd998c:::benchmark_data_smithi134_174861_object94420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:22:08.118491+0000", 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 19.438497604999998, 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1445450789999998, 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15714", 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 225882 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.118491+0000", 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.118491+0000", 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.118492+0000", 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.409e-06 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.118498+0000", 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7400000000000001e-06 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.118510+0000", 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2618e-05 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.118518+0000", 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9310000000000008e-06 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.119880+0000", 2024-02-05T08:22:27.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001361637 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.119891+0000", 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1358999999999999e-05 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:09.093395+0000", 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.97350327299999995 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:27.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:09.093407+0000", 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.204e-05 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.245151+0000", 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1517443489999999 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.247413+0000", 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0022622330000000002 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5,7", 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.247677+0000", 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000263885 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.255943+0000", 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0082659610000000005 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.255999+0000", 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6505000000000002e-05 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.256269+0000", 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000269332 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.262851+0000", 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0065822659999999998 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.262915+0000", 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3554999999999997e-05 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.263036+0000", 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000120986 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15714.0:225885 10.5 10:a1f88be3:::benchmark_data_smithi134_174861_object94423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:22:08.123443+0000", 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 19.433544958999999, 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1812996349999998, 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:27.671 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15714", 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 225885 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.123443+0000", 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.123443+0000", 2024-02-05T08:22:27.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.123445+0000", 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1900000000000002e-06 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.123454+0000", 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9479999999999997e-06 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.123474+0000", 2024-02-05T08:22:27.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9347999999999998e-05 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.123508+0000", 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4031e-05 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.123615+0000", 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000107557 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:08.123633+0000", 2024-02-05T08:22:27.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7737999999999999e-05 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:09.089345+0000", 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.96571228399999998 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:09.089357+0000", 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1982e-05 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.291981+0000", 2024-02-05T08:22:27.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.202623596 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.292278+0000", 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00029736600000000003 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,3,7", 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.292376+0000", 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7336000000000005e-05 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.299419+0000", 2024-02-05T08:22:27.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0070438230000000003 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.299511+0000", 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1520000000000005e-05 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.299627+0000", 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000115747 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:27.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.304375+0000", 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0047480669999999999 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.304431+0000", 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5930000000000002e-05 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:10.304743+0000", 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00031217199999999999 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:27.678 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:27.679 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:22:27.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:27.762 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:27.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:27.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:27.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:27 smithi134 ceph-mon[30086]: pgmap v792: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 942 op/s; 2912/67620 objects degraded (4.306%); 10276/67620 objects misplaced (15.197%); 27 MiB/s, 428 objects/s recovering 2024-02-05T08:22:27.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:27 smithi134 ceph-mon[30086]: from='client.25733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:22:27.777 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:22:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:22:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:22:27.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:27 smithi134 ceph-mon[39278]: pgmap v792: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 942 op/s; 2912/67620 objects degraded (4.306%); 10276/67620 objects misplaced (15.197%); 27 MiB/s, 428 objects/s recovering 2024-02-05T08:22:27.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:27 smithi134 ceph-mon[39278]: from='client.25733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:22:27.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:27 smithi196 ceph-mon[33018]: pgmap v792: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.4 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 942 op/s; 2912/67620 objects degraded (4.306%); 10276/67620 objects misplaced (15.197%); 27 MiB/s, 428 objects/s recovering 2024-02-05T08:22:27.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:27 smithi196 ceph-mon[33018]: from='client.25733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:22:27.863 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:27.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:27.903 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:27.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:28.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:28.040 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:28.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:28.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:28.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:28.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:28.283 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:28.315 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:28.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:28.417 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:28.447 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:28.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:28.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:28.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:28.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:28.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:28.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:28.721 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:28.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:28.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:28.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:28.844 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T08:22:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:28.868 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:28.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:28.970 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:29.008 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:29.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:29.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:29.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:29.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:29.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:29.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:29.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:29.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:22:29.360 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:22:29.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:29.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:29.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:29.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:29.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:29.558 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:29 smithi134 ceph-mon[30086]: pgmap v793: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 731 B/s rd, 0 B/s wr, 996 op/s; 2595/63684 objects degraded (4.075%); 9274/63684 objects misplaced (14.563%); 23 MiB/s, 367 objects/s recovering 2024-02-05T08:22:29.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:29.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:29.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:29.664 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:29.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:29.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:29.757 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:29.798 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":792,"stamp":"2024-02-05T08:22:25.869594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1464789786,"num_objects":22540,"num_object_clones":0,"num_object_copies":67620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2912,"num_objects_misplaced":10276,"num_objects_unfound":0,"num_objects_dirty":22540,"num_whiteouts":0,"num_read":2198,"num_read_kb":1958,"num_write":241317,"num_write_kb":8413642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7064,"num_bytes_recovered":441083767,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77308,"ondisk_log_size":77308,"up":403,"acting":431,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":461,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":8197780,"kb_used_data":4141724,"kb_used_omap":10,"kb_used_meta":4055413,"kb_avail":648018284,"statfs":{"total":671965249536,"available":663570722816,"internally_reserved":0,"allocated":4241125376,"data_stored":4238815671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11154,"internal_metadata":4152743022},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-370606080,"num_objects":-5655,"num_object_clones":0,"num_object_copies":-16965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1467,"num_objects_misplaced":-4650,"num_objects_unfound":0,"num_objects_dirty":-5655,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5655,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2574,"num_bytes_recovered":168689664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003134"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":644,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.891267+0000","last_change":"2024-02-05T08:19:44.912725+0000","last_active":"2024-02-05T08:22:20.891267+0000","last_peered":"2024-02-05T08:22:20.891267+0000","last_clean":"2024-02-05T08:19:41.829565+0000","last_became_active":"2024-02-05T08:19:44.912725+0000","last_became_peered":"2024-02-05T08:19:44.912725+0000","last_unstale":"2024-02-05T08:22:20.891267+0000","last_undegraded":"2024-02-05T08:22:20.891267+0000","last_fullsized":"2024-02-05T08:19:44.874568+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:36.296981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:36.296981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":75,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.897511+0000","last_change":"2024-02-05T08:19:44.903097+0000","last_active":"2024-02-05T08:22:20.897511+0000","last_peered":"2024-02-05T08:22:20.897511+0000","last_clean":"2024-02-05T08:04:44.536163+0000","last_became_active":"2024-02-05T08:19:44.903097+0000","last_became_peered":"2024-02-05T08:19:44.903097+0000","last_unstale":"2024-02-05T08:22:20.897511+0000","last_undegraded":"2024-02-05T08:22:20.897511+0000","last_fullsized":"2024-02-05T08:19:44.873443+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.301502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.301502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.894069+0000","last_change":"2024-02-05T08:22:09.102282+0000","last_active":"2024-02-05T08:22:20.894069+0000","last_peered":"2024-02-05T08:22:20.894069+0000","last_clean":"2024-02-05T08:22:20.894069+0000","last_became_active":"2024-02-05T08:22:09.101941+0000","last_became_peered":"2024-02-05T08:22:09.101941+0000","last_unstale":"2024-02-05T08:22:20.894069+0000","last_undegraded":"2024-02-05T08:22:20.894069+0000","last_fullsized":"2024-02-05T08:22:20.894069+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:18.141284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:18.141284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:30.873596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"236'19","reported_seq":705,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.896627+0000","last_change":"2024-02-05T08:19:44.920275+0000","last_active":"2024-02-05T08:22:20.896627+0000","last_peered":"2024-02-05T08:22:20.896627+0000","last_clean":"2024-02-05T08:19:41.830039+0000","last_became_active":"2024-02-05T08:19:44.920275+0000","last_became_peered":"2024-02-05T08:19:44.920275+0000","last_unstale":"2024-02-05T08:22:20.896627+0000","last_undegraded":"2024-02-05T08:19:44.874580+0000","last_fullsized":"2024-02-05T08:19:44.874470+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:01.602887+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:18:01.602887+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":152,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.898417+0000","last_change":"2024-02-05T08:21:51.007314+0000","last_active":"2024-02-05T08:22:20.898417+0000","last_peered":"2024-02-05T08:22:20.898417+0000","last_clean":"2024-02-05T08:22:20.898417+0000","last_became_active":"2024-02-05T08:18:34.633083+0000","last_became_peered":"2024-02-05T08:18:34.633083+0000","last_unstale":"2024-02-05T08:22:20.898417+0000","last_undegraded":"2024-02-05T08:22:20.898417+0000","last_fullsized":"2024-02-05T08:22:20.898417+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:51.006901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:21:51.006901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:16.387843+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":652,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.896937+0000","last_change":"2024-02-05T08:19:44.919417+0000","last_active":"2024-02-05T08:22:20.896937+0000","last_peered":"2024-02-05T08:22:20.896937+0000","last_clean":"2024-02-05T08:19:41.831283+0000","last_became_active":"2024-02-05T08:19:44.919417+0000","last_became_peered":"2024-02-05T08:19:44.919417+0000","last_unstale":"2024-02-05T08:22:20.896937+0000","last_undegraded":"2024-02-05T08:22:20.896937+0000","last_fullsized":"2024-02-05T08:19:44.874665+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:39.536022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:19.262611+0000","last_clean_scrub_stamp":"2024-02-05T08:18:39.536022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":161,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.897701+0000","last_change":"2024-02-05T08:21:45.823410+0000","last_active":"2024-02-05T08:22:20.897701+0000","last_peered":"2024-02-05T08:22:20.897701+0000","last_clean":"2024-02-05T08:22:20.897701+0000","last_became_active":"2024-02-05T08:21:41.880038+0000","last_became_peered":"2024-02-05T08:21:41.880038+0000","last_unstale":"2024-02-05T08:22:20.897701+0000","last_undegraded":"2024-02-05T08:22:20.897701+0000","last_fullsized":"2024-02-05T08:22:20.897701+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:45.823300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:21:45.823300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:05.550303+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"236'34","reported_seq":198,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.897321+0000","last_change":"2024-02-05T08:22:09.119504+0000","last_active":"2024-02-05T08:22:20.897321+0000","last_peered":"2024-02-05T08:22:20.897321+0000","last_clean":"2024-02-05T08:22:20.897321+0000","last_became_active":"2024-02-05T08:22:09.095896+0000","last_became_peered":"2024-02-05T08:22:09.095896+0000","last_unstale":"2024-02-05T08:22:20.897321+0000","last_undegraded":"2024-02-05T08:22:20.897321+0000","last_fullsized":"2024-02-05T08:22:20.897321+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:22:04.719477+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:22:04.719477+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:12.239099+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":583,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.884578+0000","last_change":"2024-02-05T08:22:10.119765+0000","last_active":"2024-02-05T08:22:20.884578+0000","last_peered":"2024-02-05T08:22:20.884578+0000","last_clean":"2024-02-05T08:22:20.884578+0000","last_became_active":"2024-02-05T08:22:10.119359+0000","last_became_peered":"2024-02-05T08:22:10.119359+0000","last_unstale":"2024-02-05T08:22:20.884578+0000","last_undegraded":"2024-02-05T08:22:20.884578+0000","last_fullsized":"2024-02-05T08:22:20.884578+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:21.661930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:21.661930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,0],"avail_no_missing":["7","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":659,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.893159+0000","last_change":"2024-02-05T08:22:09.109069+0000","last_active":"2024-02-05T08:22:20.893159+0000","last_peered":"2024-02-05T08:22:20.893159+0000","last_clean":"2024-02-05T08:22:20.893159+0000","last_became_active":"2024-02-05T08:22:09.108745+0000","last_became_peered":"2024-02-05T08:22:09.108745+0000","last_unstale":"2024-02-05T08:22:20.893159+0000","last_undegraded":"2024-02-05T08:22:20.893159+0000","last_fullsized":"2024-02-05T08:22:20.893159+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:18.395414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:35.612720+0000","last_clean_scrub_stamp":"2024-02-05T08:21:18.395414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:32.048707+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":681,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.893423+0000","last_change":"2024-02-05T08:21:37.323987+0000","last_active":"2024-02-05T08:22:20.893423+0000","last_peered":"2024-02-05T08:22:20.893423+0000","last_clean":"2024-02-05T08:22:20.893423+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:22:20.893423+0000","last_undegraded":"2024-02-05T08:22:20.893423+0000","last_fullsized":"2024-02-05T08:22:20.893423+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:37.323869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_clean_scrub_stamp":"2024-02-05T08:21:37.323869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:46.746148+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"263'19","reported_seq":740,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.891661+0000","last_change":"2024-02-05T08:22:09.118822+0000","last_active":"2024-02-05T08:22:20.891661+0000","last_peered":"2024-02-05T08:22:20.891661+0000","last_clean":"2024-02-05T08:22:20.891661+0000","last_became_active":"2024-02-05T08:22:09.102371+0000","last_became_peered":"2024-02-05T08:22:09.102371+0000","last_unstale":"2024-02-05T08:22:20.891661+0000","last_undegraded":"2024-02-05T08:22:20.891661+0000","last_fullsized":"2024-02-05T08:22:20.891661+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:21:43.861060+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:21:43.861060+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:12.490657+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":648,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.897633+0000","last_change":"2024-02-05T08:19:44.901170+0000","last_active":"2024-02-05T08:22:20.897633+0000","last_peered":"2024-02-05T08:22:20.897633+0000","last_clean":"2024-02-05T08:19:41.833929+0000","last_became_active":"2024-02-05T08:19:44.901170+0000","last_became_peered":"2024-02-05T08:19:44.901170+0000","last_unstale":"2024-02-05T08:22:20.897633+0000","last_undegraded":"2024-02-05T08:22:20.897633+0000","last_fullsized":"2024-02-05T08:19:44.873575+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:32.611856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:32.611856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":649,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.887553+0000","last_change":"2024-02-05T08:19:44.921416+0000","last_active":"2024-02-05T08:22:20.887553+0000","last_peered":"2024-02-05T08:22:20.887553+0000","last_clean":"2024-02-05T08:19:41.834434+0000","last_became_active":"2024-02-05T08:19:44.921416+0000","last_became_peered":"2024-02-05T08:19:44.921416+0000","last_unstale":"2024-02-05T08:22:20.887553+0000","last_undegraded":"2024-02-05T08:22:20.887553+0000","last_fullsized":"2024-02-05T08:19:44.875209+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:40.260248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:40.260248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":656,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.893140+0000","last_change":"2024-02-05T08:19:44.916712+0000","last_active":"2024-02-05T08:22:20.893140+0000","last_peered":"2024-02-05T08:22:20.893140+0000","last_clean":"2024-02-05T08:19:41.831063+0000","last_became_active":"2024-02-05T08:19:44.916712+0000","last_became_peered":"2024-02-05T08:19:44.916712+0000","last_unstale":"2024-02-05T08:22:20.893140+0000","last_undegraded":"2024-02-05T08:22:20.893140+0000","last_fullsized":"2024-02-05T08:19:44.898786+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:14.458198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.458198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":681,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.895598+0000","last_change":"2024-02-05T08:19:44.913761+0000","last_active":"2024-02-05T08:22:20.895598+0000","last_peered":"2024-02-05T08:22:20.895598+0000","last_clean":"2024-02-05T08:19:42.900179+0000","last_became_active":"2024-02-05T08:19:44.913761+0000","last_became_peered":"2024-02-05T08:19:44.913761+0000","last_unstale":"2024-02-05T08:22:20.895598+0000","last_undegraded":"2024-02-05T08:19:44.909323+0000","last_fullsized":"2024-02-05T08:19:44.909205+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-02-05T08:18:05.582303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:05.582303+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":662,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.900161+0000","last_change":"2024-02-05T08:21:42.867663+0000","last_active":"2024-02-05T08:22:20.900161+0000","last_peered":"2024-02-05T08:22:20.900161+0000","last_clean":"2024-02-05T08:22:20.900161+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:22:20.900161+0000","last_undegraded":"2024-02-05T08:22:20.900161+0000","last_fullsized":"2024-02-05T08:22:20.900161+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:42.867555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:21:42.867555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:53.230473+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":80,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.895292+0000","last_change":"2024-02-05T08:22:10.106207+0000","last_active":"2024-02-05T08:22:20.895292+0000","last_peered":"2024-02-05T08:22:20.895292+0000","last_clean":"2024-02-05T08:22:20.895292+0000","last_became_active":"2024-02-05T08:22:10.105741+0000","last_became_peered":"2024-02-05T08:22:10.105741+0000","last_unstale":"2024-02-05T08:22:20.895292+0000","last_undegraded":"2024-02-05T08:22:20.895292+0000","last_fullsized":"2024-02-05T08:22:20.895292+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_clean_scrub_stamp":"2024-02-05T08:19:36.737342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,0],"avail_no_missing":["4","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.895410+0000","last_change":"2024-02-05T08:22:10.109383+0000","last_active":"2024-02-05T08:22:20.895410+0000","last_peered":"2024-02-05T08:22:20.895410+0000","last_clean":"2024-02-05T08:22:20.895410+0000","last_became_active":"2024-02-05T08:22:10.108335+0000","last_became_peered":"2024-02-05T08:22:10.108335+0000","last_unstale":"2024-02-05T08:22:20.895410+0000","last_undegraded":"2024-02-05T08:22:20.895410+0000","last_fullsized":"2024-02-05T08:22:20.895410+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:04.290125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:04.290125+0000","last_clean_scrub_stamp":"2024-02-05T08:22:04.290125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:24.614667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"236'34","reported_seq":762,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.882594+0000","last_change":"2024-02-05T08:19:44.892882+0000","last_active":"2024-02-05T08:22:20.882594+0000","last_peered":"2024-02-05T08:22:20.882594+0000","last_clean":"2024-02-05T08:19:41.827204+0000","last_became_active":"2024-02-05T08:19:44.892882+0000","last_became_peered":"2024-02-05T08:19:44.892882+0000","last_unstale":"2024-02-05T08:22:20.882594+0000","last_undegraded":"2024-02-05T08:19:44.885361+0000","last_fullsized":"2024-02-05T08:19:44.885220+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:18:47.576225+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:18:47.576225+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":7}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":152,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.900426+0000","last_change":"2024-02-05T08:21:59.021213+0000","last_active":"2024-02-05T08:22:20.900426+0000","last_peered":"2024-02-05T08:22:20.900426+0000","last_clean":"2024-02-05T08:22:20.900426+0000","last_became_active":"2024-02-05T08:18:34.642494+0000","last_became_peered":"2024-02-05T08:18:34.642494+0000","last_unstale":"2024-02-05T08:22:20.900426+0000","last_undegraded":"2024-02-05T08:22:20.900426+0000","last_fullsized":"2024-02-05T08:22:20.900426+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:59.021074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_clean_scrub_stamp":"2024-02-05T08:21:59.021074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:09.416637+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":665,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.899061+0000","last_change":"2024-02-05T08:21:07.003807+0000","last_active":"2024-02-05T08:22:20.899061+0000","last_peered":"2024-02-05T08:22:20.899061+0000","last_clean":"2024-02-05T08:22:20.899061+0000","last_became_active":"2024-02-05T08:18:34.638652+0000","last_became_peered":"2024-02-05T08:18:34.638652+0000","last_unstale":"2024-02-05T08:22:20.899061+0000","last_undegraded":"2024-02-05T08:22:20.899061+0000","last_fullsized":"2024-02-05T08:22:20.899061+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:07.003702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:21:07.003702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":648,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.891982+0000","last_change":"2024-02-05T08:19:44.912456+0000","last_active":"2024-02-05T08:22:20.891982+0000","last_peered":"2024-02-05T08:22:20.891982+0000","last_clean":"2024-02-05T08:19:42.900163+0000","last_became_active":"2024-02-05T08:19:44.912456+0000","last_became_peered":"2024-02-05T08:19:44.912456+0000","last_unstale":"2024-02-05T08:22:20.891982+0000","last_undegraded":"2024-02-05T08:22:20.891982+0000","last_fullsized":"2024-02-05T08:19:44.874216+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:19.338699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:19:19.338699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"236'23","reported_seq":151,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.898402+0000","last_change":"2024-02-05T08:19:44.923092+0000","last_active":"2024-02-05T08:22:20.898402+0000","last_peered":"2024-02-05T08:22:20.898402+0000","last_clean":"2024-02-05T08:19:41.835633+0000","last_became_active":"2024-02-05T08:19:44.923092+0000","last_became_peered":"2024-02-05T08:19:44.923092+0000","last_unstale":"2024-02-05T08:22:20.898402+0000","last_undegraded":"2024-02-05T08:19:44.886225+0000","last_fullsized":"2024-02-05T08:19:44.886101+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'23","last_scrub_stamp":"2024-02-05T08:17:58.012827+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:17:58.012827+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":660,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.888378+0000","last_change":"2024-02-05T08:20:37.119269+0000","last_active":"2024-02-05T08:22:20.888378+0000","last_peered":"2024-02-05T08:22:20.888378+0000","last_clean":"2024-02-05T08:22:20.888378+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:22:20.888378+0000","last_undegraded":"2024-02-05T08:22:20.888378+0000","last_fullsized":"2024-02-05T08:22:20.888378+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:20:37.119142+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.119142+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":522,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.892972+0000","last_change":"2024-02-05T08:21:46.613709+0000","last_active":"2024-02-05T08:22:20.892972+0000","last_peered":"2024-02-05T08:22:20.892972+0000","last_clean":"2024-02-05T08:22:20.892972+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:22:20.892972+0000","last_undegraded":"2024-02-05T08:22:20.892972+0000","last_fullsized":"2024-02-05T08:22:20.892972+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:21:46.613469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:21:46.613469+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:47.607944+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":602,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.897016+0000","last_change":"2024-02-05T08:19:44.915101+0000","last_active":"2024-02-05T08:22:20.897016+0000","last_peered":"2024-02-05T08:22:20.897016+0000","last_clean":"2024-02-05T08:18:29.609487+0000","last_became_active":"2024-02-05T08:19:44.915101+0000","last_became_peered":"2024-02-05T08:19:44.915101+0000","last_unstale":"2024-02-05T08:22:20.897016+0000","last_undegraded":"2024-02-05T08:19:44.874435+0000","last_fullsized":"2024-02-05T08:19:44.874324+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:19:28.743054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:28.743054+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"236'20","reported_seq":473,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.888562+0000","last_change":"2024-02-05T08:22:10.099322+0000","last_active":"2024-02-05T08:22:20.888562+0000","last_peered":"2024-02-05T08:22:20.888562+0000","last_clean":"2024-02-05T08:22:20.888562+0000","last_became_active":"2024-02-05T08:22:10.096130+0000","last_became_peered":"2024-02-05T08:22:10.096130+0000","last_unstale":"2024-02-05T08:22:20.888562+0000","last_undegraded":"2024-02-05T08:22:20.888562+0000","last_fullsized":"2024-02-05T08:22:20.888562+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:21:49.240369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:49.240369+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:02.612940+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,0],"avail_no_missing":["7","0","3"],"object_location_counts":[{"shards":"0,3,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":74,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.893165+0000","last_change":"2024-02-05T08:19:44.923094+0000","last_active":"2024-02-05T08:22:20.893165+0000","last_peered":"2024-02-05T08:22:20.893165+0000","last_clean":"2024-02-05T08:04:46.546354+0000","last_became_active":"2024-02-05T08:19:44.923094+0000","last_became_peered":"2024-02-05T08:19:44.923094+0000","last_unstale":"2024-02-05T08:22:20.893165+0000","last_undegraded":"2024-02-05T08:22:20.893165+0000","last_fullsized":"2024-02-05T08:19:44.897835+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:46.550985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:46.550985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":19,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.889351+0000","last_change":"2024-02-05T08:22:10.096359+0000","last_active":"2024-02-05T08:22:20.889351+0000","last_peered":"2024-02-05T08:22:20.889351+0000","last_clean":"2024-02-05T08:22:20.889351+0000","last_became_active":"2024-02-05T08:22:10.094420+0000","last_became_peered":"2024-02-05T08:22:10.094420+0000","last_unstale":"2024-02-05T08:22:20.889351+0000","last_undegraded":"2024-02-05T08:22:20.889351+0000","last_fullsized":"2024-02-05T08:22:20.889351+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:19.107555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:19.107555+0000","last_clean_scrub_stamp":"2024-02-05T08:21:19.107555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:47.004351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2,0],"avail_no_missing":["7","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":152,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.896873+0000","last_change":"2024-02-05T08:22:10.109079+0000","last_active":"2024-02-05T08:22:20.896873+0000","last_peered":"2024-02-05T08:22:20.896873+0000","last_clean":"2024-02-05T08:22:20.896873+0000","last_became_active":"2024-02-05T08:22:10.108643+0000","last_became_peered":"2024-02-05T08:22:10.108643+0000","last_unstale":"2024-02-05T08:22:20.896873+0000","last_undegraded":"2024-02-05T08:22:20.896873+0000","last_fullsized":"2024-02-05T08:22:20.896873+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:11.049460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:19:11.049460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2,0],"avail_no_missing":["3","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"236'19","reported_seq":415,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.892784+0000","last_change":"2024-02-05T08:21:09.755205+0000","last_active":"2024-02-05T08:22:20.892784+0000","last_peered":"2024-02-05T08:22:20.892784+0000","last_clean":"2024-02-05T08:22:20.892784+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:22:20.892784+0000","last_undegraded":"2024-02-05T08:22:20.892784+0000","last_fullsized":"2024-02-05T08:22:20.892784+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:21:09.755115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:09.755115+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:38.649362+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":642,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.887824+0000","last_change":"2024-02-05T08:21:14.159752+0000","last_active":"2024-02-05T08:22:20.887824+0000","last_peered":"2024-02-05T08:22:20.887824+0000","last_clean":"2024-02-05T08:22:20.887824+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:22:20.887824+0000","last_undegraded":"2024-02-05T08:22:20.887824+0000","last_fullsized":"2024-02-05T08:22:20.887824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:14.159628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:14.159628+0000","last_clean_scrub_stamp":"2024-02-05T08:21:14.159628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:30.209411+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":394,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.889399+0000","last_change":"2024-02-05T08:21:50.638160+0000","last_active":"2024-02-05T08:22:20.889399+0000","last_peered":"2024-02-05T08:22:20.889399+0000","last_clean":"2024-02-05T08:22:20.889399+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:22:20.889399+0000","last_undegraded":"2024-02-05T08:22:20.889399+0000","last_fullsized":"2024-02-05T08:22:20.889399+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:50.638065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:21:50.638065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:10.341134+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":533,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.894147+0000","last_change":"2024-02-05T08:22:09.104313+0000","last_active":"2024-02-05T08:22:20.894147+0000","last_peered":"2024-02-05T08:22:20.894147+0000","last_clean":"2024-02-05T08:22:20.894147+0000","last_became_active":"2024-02-05T08:22:09.104064+0000","last_became_peered":"2024-02-05T08:22:09.104064+0000","last_unstale":"2024-02-05T08:22:20.894147+0000","last_undegraded":"2024-02-05T08:22:20.894147+0000","last_fullsized":"2024-02-05T08:22:20.894147+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:38.069723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:34.697248+0000","last_clean_scrub_stamp":"2024-02-05T08:21:38.069723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:42.076515+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":153,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.897405+0000","last_change":"2024-02-05T08:19:44.911502+0000","last_active":"2024-02-05T08:22:20.897405+0000","last_peered":"2024-02-05T08:22:20.897405+0000","last_clean":"2024-02-05T08:19:41.828151+0000","last_became_active":"2024-02-05T08:19:44.911502+0000","last_became_peered":"2024-02-05T08:19:44.911502+0000","last_unstale":"2024-02-05T08:22:20.897405+0000","last_undegraded":"2024-02-05T08:19:44.874110+0000","last_fullsized":"2024-02-05T08:19:44.873994+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:19:23.023541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:23.023541+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":158,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.898328+0000","last_change":"2024-02-05T08:22:09.096116+0000","last_active":"2024-02-05T08:22:20.898328+0000","last_peered":"2024-02-05T08:22:20.898328+0000","last_clean":"2024-02-05T08:22:20.898328+0000","last_became_active":"2024-02-05T08:22:09.095805+0000","last_became_peered":"2024-02-05T08:22:09.095805+0000","last_unstale":"2024-02-05T08:22:20.898328+0000","last_undegraded":"2024-02-05T08:22:20.898328+0000","last_fullsized":"2024-02-05T08:22:20.898328+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:26.891040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:26.891040+0000","last_clean_scrub_stamp":"2024-02-05T08:20:26.891040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":674,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.886618+0000","last_change":"2024-02-05T08:21:54.156544+0000","last_active":"2024-02-05T08:22:20.886618+0000","last_peered":"2024-02-05T08:22:20.886618+0000","last_clean":"2024-02-05T08:22:20.886618+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:22:20.886618+0000","last_undegraded":"2024-02-05T08:22:20.886618+0000","last_fullsized":"2024-02-05T08:22:20.886618+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:54.156080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:54.156080+0000","last_clean_scrub_stamp":"2024-02-05T08:21:54.156080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:08.226750+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":656,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.897824+0000","last_change":"2024-02-05T08:19:44.904157+0000","last_active":"2024-02-05T08:22:20.897824+0000","last_peered":"2024-02-05T08:22:20.897824+0000","last_clean":"2024-02-05T08:19:41.832352+0000","last_became_active":"2024-02-05T08:19:44.904157+0000","last_became_peered":"2024-02-05T08:19:44.904157+0000","last_unstale":"2024-02-05T08:22:20.897824+0000","last_undegraded":"2024-02-05T08:22:20.897824+0000","last_fullsized":"2024-02-05T08:19:44.901922+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:41.576882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:18:41.576882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"236'24","reported_seq":164,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.897746+0000","last_change":"2024-02-05T08:22:01.724542+0000","last_active":"2024-02-05T08:22:20.897746+0000","last_peered":"2024-02-05T08:22:20.897746+0000","last_clean":"2024-02-05T08:22:20.897746+0000","last_became_active":"2024-02-05T08:18:34.631912+0000","last_became_peered":"2024-02-05T08:18:34.631912+0000","last_unstale":"2024-02-05T08:22:20.897746+0000","last_undegraded":"2024-02-05T08:22:20.897746+0000","last_fullsized":"2024-02-05T08:22:20.897746+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:22:01.724412+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:22:01.724412+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:12.000220+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":583,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.892981+0000","last_change":"2024-02-05T08:22:10.101418+0000","last_active":"2024-02-05T08:22:20.892981+0000","last_peered":"2024-02-05T08:22:20.892981+0000","last_clean":"2024-02-05T08:22:20.892981+0000","last_became_active":"2024-02-05T08:22:10.100923+0000","last_became_peered":"2024-02-05T08:22:10.100923+0000","last_unstale":"2024-02-05T08:22:20.892981+0000","last_undegraded":"2024-02-05T08:22:20.892981+0000","last_fullsized":"2024-02-05T08:22:20.892981+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:26.593364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:19:26.593364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6,0],"avail_no_missing":["5","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":152,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.898145+0000","last_change":"2024-02-05T08:22:10.091668+0000","last_active":"2024-02-05T08:22:20.898145+0000","last_peered":"2024-02-05T08:22:20.898145+0000","last_clean":"2024-02-05T08:22:20.898145+0000","last_became_active":"2024-02-05T08:22:10.091216+0000","last_became_peered":"2024-02-05T08:22:10.091216+0000","last_unstale":"2024-02-05T08:22:20.898145+0000","last_undegraded":"2024-02-05T08:22:20.898145+0000","last_fullsized":"2024-02-05T08:22:20.898145+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:44.133926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:44.133926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,0],"avail_no_missing":["3","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":674,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.887698+0000","last_change":"2024-02-05T08:21:53.174425+0000","last_active":"2024-02-05T08:22:20.887698+0000","last_peered":"2024-02-05T08:22:20.887698+0000","last_clean":"2024-02-05T08:22:20.887698+0000","last_became_active":"2024-02-05T08:18:34.642387+0000","last_became_peered":"2024-02-05T08:18:34.642387+0000","last_unstale":"2024-02-05T08:22:20.887698+0000","last_undegraded":"2024-02-05T08:22:20.887698+0000","last_fullsized":"2024-02-05T08:22:20.887698+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:53.173888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:53.173888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:58.513833+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"263'19","reported_seq":329,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.896496+0000","last_change":"2024-02-05T08:21:53.427022+0000","last_active":"2024-02-05T08:22:20.896496+0000","last_peered":"2024-02-05T08:22:20.896496+0000","last_clean":"2024-02-05T08:22:20.896496+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:22:20.896496+0000","last_undegraded":"2024-02-05T08:22:20.896496+0000","last_fullsized":"2024-02-05T08:22:20.896496+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:21:53.422885+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_clean_scrub_stamp":"2024-02-05T08:21:53.422885+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:14.476216+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":166,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.897491+0000","last_change":"2024-02-05T08:22:09.096761+0000","last_active":"2024-02-05T08:22:20.897491+0000","last_peered":"2024-02-05T08:22:20.897491+0000","last_clean":"2024-02-05T08:22:20.897491+0000","last_became_active":"2024-02-05T08:22:09.096314+0000","last_became_peered":"2024-02-05T08:22:09.096314+0000","last_unstale":"2024-02-05T08:22:20.897491+0000","last_undegraded":"2024-02-05T08:22:20.897491+0000","last_fullsized":"2024-02-05T08:22:20.897491+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:46.883886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:21:46.883886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:52.876170+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":589,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.896967+0000","last_change":"2024-02-05T08:19:44.903986+0000","last_active":"2024-02-05T08:22:20.896967+0000","last_peered":"2024-02-05T08:22:20.896967+0000","last_clean":"2024-02-05T08:18:29.613784+0000","last_became_active":"2024-02-05T08:19:44.903986+0000","last_became_peered":"2024-02-05T08:19:44.903986+0000","last_unstale":"2024-02-05T08:22:20.896967+0000","last_undegraded":"2024-02-05T08:22:20.896967+0000","last_fullsized":"2024-02-05T08:19:44.899948+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:17.633650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.633650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":681,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.898713+0000","last_change":"2024-02-05T08:21:44.871777+0000","last_active":"2024-02-05T08:22:20.898713+0000","last_peered":"2024-02-05T08:22:20.898713+0000","last_clean":"2024-02-05T08:22:20.898713+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:22:20.898713+0000","last_undegraded":"2024-02-05T08:22:20.898713+0000","last_fullsized":"2024-02-05T08:22:20.898713+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:44.871668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:21:44.871668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:04.896469+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"249'24","reported_seq":724,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.895944+0000","last_change":"2024-02-05T08:19:44.930725+0000","last_active":"2024-02-05T08:22:20.895944+0000","last_peered":"2024-02-05T08:22:20.895944+0000","last_clean":"2024-02-05T08:19:41.830358+0000","last_became_active":"2024-02-05T08:19:44.930725+0000","last_became_peered":"2024-02-05T08:19:44.930725+0000","last_unstale":"2024-02-05T08:22:20.895944+0000","last_undegraded":"2024-02-05T08:19:44.912297+0000","last_fullsized":"2024-02-05T08:19:44.910005+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:19:27.349541+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:19:27.349541+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":513,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.888776+0000","last_change":"2024-02-05T08:22:09.102304+0000","last_active":"2024-02-05T08:22:20.888776+0000","last_peered":"2024-02-05T08:22:20.888776+0000","last_clean":"2024-02-05T08:22:20.888776+0000","last_became_active":"2024-02-05T08:22:09.101652+0000","last_became_peered":"2024-02-05T08:22:09.101652+0000","last_unstale":"2024-02-05T08:22:20.888776+0000","last_undegraded":"2024-02-05T08:22:20.888776+0000","last_fullsized":"2024-02-05T08:22:20.888776+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:29.732422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:29.732422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"355'11","reported_seq":880,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.893640+0000","last_change":"2024-02-05T08:22:10.113309+0000","last_active":"2024-02-05T08:22:20.893640+0000","last_peered":"2024-02-05T08:22:20.893640+0000","last_clean":"2024-02-05T08:22:20.893640+0000","last_became_active":"2024-02-05T08:22:10.100064+0000","last_became_peered":"2024-02-05T08:22:10.100064+0000","last_unstale":"2024-02-05T08:22:20.893640+0000","last_undegraded":"2024-02-05T08:22:20.893640+0000","last_fullsized":"2024-02-05T08:22:20.893640+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:19:12.452237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:12.452237+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,0],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":662,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.888073+0000","last_change":"2024-02-05T08:21:09.098128+0000","last_active":"2024-02-05T08:22:20.888073+0000","last_peered":"2024-02-05T08:22:20.888073+0000","last_clean":"2024-02-05T08:22:20.888073+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:22:20.888073+0000","last_undegraded":"2024-02-05T08:22:20.888073+0000","last_fullsized":"2024-02-05T08:22:20.888073+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:09.098005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:09.098005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:33.909120+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":159,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.898564+0000","last_change":"2024-02-05T08:19:44.899031+0000","last_active":"2024-02-05T08:22:20.898564+0000","last_peered":"2024-02-05T08:22:20.898564+0000","last_clean":"2024-02-05T08:19:41.830913+0000","last_became_active":"2024-02-05T08:19:44.899031+0000","last_became_peered":"2024-02-05T08:19:44.899031+0000","last_unstale":"2024-02-05T08:22:20.898564+0000","last_undegraded":"2024-02-05T08:19:44.887736+0000","last_fullsized":"2024-02-05T08:19:44.887622+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-02-05T08:19:35.981313+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:19:35.981313+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":644,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.895116+0000","last_change":"2024-02-05T08:21:10.125938+0000","last_active":"2024-02-05T08:22:20.895116+0000","last_peered":"2024-02-05T08:22:20.895116+0000","last_clean":"2024-02-05T08:22:20.895116+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:22:20.895116+0000","last_undegraded":"2024-02-05T08:22:20.895116+0000","last_fullsized":"2024-02-05T08:22:20.895116+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:10.125818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:28.519149+0000","last_clean_scrub_stamp":"2024-02-05T08:21:10.125818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:30.234304+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"295'10","reported_seq":203,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.898969+0000","last_change":"2024-02-05T08:22:03.724091+0000","last_active":"2024-02-05T08:22:20.898969+0000","last_peered":"2024-02-05T08:22:20.898969+0000","last_clean":"2024-02-05T08:22:20.898969+0000","last_became_active":"2024-02-05T08:18:34.644517+0000","last_became_peered":"2024-02-05T08:18:34.644517+0000","last_unstale":"2024-02-05T08:22:20.898969+0000","last_undegraded":"2024-02-05T08:22:20.898969+0000","last_fullsized":"2024-02-05T08:22:20.898969+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'10","last_scrub_stamp":"2024-02-05T08:22:03.723970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:03.723970+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:18.050865+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":532,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.887296+0000","last_change":"2024-02-05T08:22:10.098059+0000","last_active":"2024-02-05T08:22:20.887296+0000","last_peered":"2024-02-05T08:22:20.887296+0000","last_clean":"2024-02-05T08:22:20.887296+0000","last_became_active":"2024-02-05T08:22:10.094273+0000","last_became_peered":"2024-02-05T08:22:10.094273+0000","last_unstale":"2024-02-05T08:22:20.887296+0000","last_undegraded":"2024-02-05T08:22:20.887296+0000","last_fullsized":"2024-02-05T08:22:20.887296+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:03.366172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:21:03.366172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,0],"avail_no_missing":["7","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":722,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.893662+0000","last_change":"2024-02-05T08:21:54.447832+0000","last_active":"2024-02-05T08:22:20.893662+0000","last_peered":"2024-02-05T08:22:20.893662+0000","last_clean":"2024-02-05T08:22:20.893662+0000","last_became_active":"2024-02-05T08:18:34.640741+0000","last_became_peered":"2024-02-05T08:18:34.640741+0000","last_unstale":"2024-02-05T08:22:20.893662+0000","last_undegraded":"2024-02-05T08:22:20.893662+0000","last_fullsized":"2024-02-05T08:22:20.893662+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:21:54.447760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:54.447760+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:12.154528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":665,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.883248+0000","last_change":"2024-02-05T08:21:40.074672+0000","last_active":"2024-02-05T08:22:20.883248+0000","last_peered":"2024-02-05T08:22:20.883248+0000","last_clean":"2024-02-05T08:22:20.883248+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:22:20.883248+0000","last_undegraded":"2024-02-05T08:22:20.883248+0000","last_fullsized":"2024-02-05T08:22:20.883248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:40.074544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:40.074544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:40.581670+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":147,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.900355+0000","last_change":"2024-02-05T08:19:44.899825+0000","last_active":"2024-02-05T08:22:20.900355+0000","last_peered":"2024-02-05T08:22:20.900355+0000","last_clean":"2024-02-05T08:19:41.832250+0000","last_became_active":"2024-02-05T08:19:44.899825+0000","last_became_peered":"2024-02-05T08:19:44.899825+0000","last_unstale":"2024-02-05T08:22:20.900355+0000","last_undegraded":"2024-02-05T08:22:20.900355+0000","last_fullsized":"2024-02-05T08:19:44.888998+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:37.028991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:37.028991+0000","last_clean_scrub_stamp":"2024-02-05T08:19:37.028991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":674,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.887319+0000","last_change":"2024-02-05T08:22:05.129978+0000","last_active":"2024-02-05T08:22:20.887319+0000","last_peered":"2024-02-05T08:22:20.887319+0000","last_clean":"2024-02-05T08:22:20.887319+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:22:20.887319+0000","last_undegraded":"2024-02-05T08:22:20.887319+0000","last_fullsized":"2024-02-05T08:22:20.887319+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:05.129584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:22:05.129584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:33.967382+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"278'19","reported_seq":578,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.897371+0000","last_change":"2024-02-05T08:22:09.127927+0000","last_active":"2024-02-05T08:22:20.897371+0000","last_peered":"2024-02-05T08:22:20.897371+0000","last_clean":"2024-02-05T08:22:20.897371+0000","last_became_active":"2024-02-05T08:22:09.110422+0000","last_became_peered":"2024-02-05T08:22:09.110422+0000","last_unstale":"2024-02-05T08:22:20.897371+0000","last_undegraded":"2024-02-05T08:22:20.897371+0000","last_fullsized":"2024-02-05T08:22:20.897371+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-05T08:21:50.244148+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:21:50.244148+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:13.207409+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":662,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.887585+0000","last_change":"2024-02-05T08:21:37.234586+0000","last_active":"2024-02-05T08:22:20.887585+0000","last_peered":"2024-02-05T08:22:20.887585+0000","last_clean":"2024-02-05T08:22:20.887585+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:22:20.887585+0000","last_undegraded":"2024-02-05T08:22:20.887585+0000","last_fullsized":"2024-02-05T08:22:20.887585+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:37.234451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:37.234451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:04.799794+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":660,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.898317+0000","last_change":"2024-02-05T08:19:44.903925+0000","last_active":"2024-02-05T08:22:20.898317+0000","last_peered":"2024-02-05T08:22:20.898317+0000","last_clean":"2024-02-05T08:19:41.832619+0000","last_became_active":"2024-02-05T08:19:44.903925+0000","last_became_peered":"2024-02-05T08:19:44.903925+0000","last_unstale":"2024-02-05T08:22:20.898317+0000","last_undegraded":"2024-02-05T08:22:20.898317+0000","last_fullsized":"2024-02-05T08:19:44.901051+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.486901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.486901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":397,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.888714+0000","last_change":"2024-02-05T08:22:09.098348+0000","last_active":"2024-02-05T08:22:20.888714+0000","last_peered":"2024-02-05T08:22:20.888714+0000","last_clean":"2024-02-05T08:22:20.888714+0000","last_became_active":"2024-02-05T08:22:09.097843+0000","last_became_peered":"2024-02-05T08:22:09.097843+0000","last_unstale":"2024-02-05T08:22:20.888714+0000","last_undegraded":"2024-02-05T08:22:20.888714+0000","last_fullsized":"2024-02-05T08:22:20.888714+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:08.817073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:08.817073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:23.558300+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'15","reported_seq":147,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.900015+0000","last_change":"2024-02-05T08:19:44.899823+0000","last_active":"2024-02-05T08:22:20.900015+0000","last_peered":"2024-02-05T08:22:20.900015+0000","last_clean":"2024-02-05T08:19:41.832108+0000","last_became_active":"2024-02-05T08:19:44.899823+0000","last_became_peered":"2024-02-05T08:19:44.899823+0000","last_unstale":"2024-02-05T08:22:20.900015+0000","last_undegraded":"2024-02-05T08:19:44.888775+0000","last_fullsized":"2024-02-05T08:19:44.888665+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:17:54.101749+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.101749+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"357'14888","reported_seq":15120,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:23.500948+0000","last_change":"2024-02-05T08:22:10.208057+0000","last_active":"2024-02-05T08:22:23.500948+0000","last_peered":"2024-02-05T08:22:23.500948+0000","last_clean":"2024-02-05T08:22:23.500948+0000","last_became_active":"2024-02-05T08:22:10.146665+0000","last_became_peered":"2024-02-05T08:22:10.146665+0000","last_unstale":"2024-02-05T08:22:23.500948+0000","last_undegraded":"2024-02-05T08:22:23.500948+0000","last_fullsized":"2024-02-05T08:22:23.500948+0000","mapping_epoch":353,"log_start":"340'10299","ondisk_log_start":"340'10299","created":309,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'13859","last_scrub_stamp":"2024-02-05T08:22:07.051637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:22:07.051637+0000","objects_scrubbed":2398,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:32.738372+0000","scrub_duration":729,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88473600,"num_objects":1350,"num_object_clones":0,"num_object_copies":4050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1350,"num_objects_unfound":0,"num_objects_dirty":1350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14888,"num_write_kb":519616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,0],"avail_no_missing":["5","0","4"],"object_location_counts":[{"shards":"0,4,5","objects":1350}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":676,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.897063+0000","last_change":"2024-02-05T08:22:09.100651+0000","last_active":"2024-02-05T08:22:20.897063+0000","last_peered":"2024-02-05T08:22:20.897063+0000","last_clean":"2024-02-05T08:22:20.897063+0000","last_became_active":"2024-02-05T08:22:09.100322+0000","last_became_peered":"2024-02-05T08:22:09.100322+0000","last_unstale":"2024-02-05T08:22:20.897063+0000","last_undegraded":"2024-02-05T08:22:20.897063+0000","last_fullsized":"2024-02-05T08:22:20.897063+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:45.532212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:45.532212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:11.328971+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":679,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.888840+0000","last_change":"2024-02-05T08:22:10.109869+0000","last_active":"2024-02-05T08:22:20.888840+0000","last_peered":"2024-02-05T08:22:20.888840+0000","last_clean":"2024-02-05T08:22:20.888840+0000","last_became_active":"2024-02-05T08:22:10.109413+0000","last_became_peered":"2024-02-05T08:22:10.109413+0000","last_unstale":"2024-02-05T08:22:20.888840+0000","last_undegraded":"2024-02-05T08:22:20.888840+0000","last_fullsized":"2024-02-05T08:22:20.888840+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:20:34.173063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:20:34.173063+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4,0],"avail_no_missing":["7","0","4"],"object_location_counts":[{"shards":"0,4,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"262'34","reported_seq":752,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.892882+0000","last_change":"2024-02-05T08:21:44.548331+0000","last_active":"2024-02-05T08:22:20.892882+0000","last_peered":"2024-02-05T08:22:20.892882+0000","last_clean":"2024-02-05T08:22:20.892882+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:22:20.892882+0000","last_undegraded":"2024-02-05T08:22:20.892882+0000","last_fullsized":"2024-02-05T08:22:20.892882+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'34","last_scrub_stamp":"2024-02-05T08:21:44.548195+0000","last_deep_scrub":"262'34","last_deep_scrub_stamp":"2024-02-05T08:19:34.364738+0000","last_clean_scrub_stamp":"2024-02-05T08:21:44.548195+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:04.070224+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":897,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.887603+0000","last_change":"2024-02-05T08:21:46.165287+0000","last_active":"2024-02-05T08:22:20.887603+0000","last_peered":"2024-02-05T08:22:20.887603+0000","last_clean":"2024-02-05T08:22:20.887603+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:22:20.887603+0000","last_undegraded":"2024-02-05T08:22:20.887603+0000","last_fullsized":"2024-02-05T08:22:20.887603+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:21:46.164949+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_clean_scrub_stamp":"2024-02-05T08:21:46.164949+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:52.940791+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"357'14916","reported_seq":15100,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:23.137014+0000","last_change":"2024-02-05T08:21:09.538977+0000","last_active":"2024-02-05T08:22:23.137014+0000","last_peered":"2024-02-05T08:22:23.137014+0000","last_clean":"2024-02-05T08:22:23.137014+0000","last_became_active":"2024-02-05T08:19:07.346046+0000","last_became_peered":"2024-02-05T08:19:07.346046+0000","last_unstale":"2024-02-05T08:22:23.137014+0000","last_undegraded":"2024-02-05T08:22:23.137014+0000","last_fullsized":"2024-02-05T08:22:23.137014+0000","mapping_epoch":309,"log_start":"338'9870","ondisk_log_start":"338'9870","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'9467","last_scrub_stamp":"2024-02-05T08:21:09.538779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:21:09.538779+0000","objects_scrubbed":6962,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:39.064414+0000","scrub_duration":2843,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90832896,"num_objects":1386,"num_object_clones":0,"num_object_copies":4158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14916,"num_write_kb":521664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":139,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.898500+0000","last_change":"2024-02-05T08:19:44.899425+0000","last_active":"2024-02-05T08:22:20.898500+0000","last_peered":"2024-02-05T08:22:20.898500+0000","last_clean":"2024-02-05T08:19:41.833419+0000","last_became_active":"2024-02-05T08:19:44.899425+0000","last_became_peered":"2024-02-05T08:19:44.899425+0000","last_unstale":"2024-02-05T08:22:20.898500+0000","last_undegraded":"2024-02-05T08:22:20.898500+0000","last_fullsized":"2024-02-05T08:19:44.885939+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:31.245736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:18:31.245736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":657,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.889172+0000","last_change":"2024-02-05T08:19:44.926408+0000","last_active":"2024-02-05T08:22:20.889172+0000","last_peered":"2024-02-05T08:22:20.889172+0000","last_clean":"2024-02-05T08:19:41.835482+0000","last_became_active":"2024-02-05T08:19:44.926408+0000","last_became_peered":"2024-02-05T08:19:44.926408+0000","last_unstale":"2024-02-05T08:22:20.889172+0000","last_undegraded":"2024-02-05T08:22:20.889172+0000","last_fullsized":"2024-02-05T08:19:44.901657+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:37.226829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:19:37.226829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":713,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.892204+0000","last_change":"2024-02-05T08:19:44.912589+0000","last_active":"2024-02-05T08:22:20.892204+0000","last_peered":"2024-02-05T08:22:20.892204+0000","last_clean":"2024-02-05T08:19:41.830832+0000","last_became_active":"2024-02-05T08:19:44.912589+0000","last_became_peered":"2024-02-05T08:19:44.912589+0000","last_unstale":"2024-02-05T08:22:20.892204+0000","last_undegraded":"2024-02-05T08:19:44.909517+0000","last_fullsized":"2024-02-05T08:19:44.909410+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-02-05T08:18:02.629366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:02.629366+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":662,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.882220+0000","last_change":"2024-02-05T08:22:10.217919+0000","last_active":"2024-02-05T08:22:20.882220+0000","last_peered":"2024-02-05T08:22:20.882220+0000","last_clean":"2024-02-05T08:22:20.882220+0000","last_became_active":"2024-02-05T08:22:10.217428+0000","last_became_peered":"2024-02-05T08:22:10.217428+0000","last_unstale":"2024-02-05T08:22:20.882220+0000","last_undegraded":"2024-02-05T08:22:20.882220+0000","last_fullsized":"2024-02-05T08:22:20.882220+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:33.725647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:18:33.725647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5,0],"avail_no_missing":["2","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"357'14945","reported_seq":15143,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:22.802738+0000","last_change":"2024-02-05T08:22:10.203857+0000","last_active":"2024-02-05T08:22:22.802738+0000","last_peered":"2024-02-05T08:22:22.802738+0000","last_clean":"2024-02-05T08:22:22.802738+0000","last_became_active":"2024-02-05T08:22:10.109292+0000","last_became_peered":"2024-02-05T08:22:10.109292+0000","last_unstale":"2024-02-05T08:22:22.802738+0000","last_undegraded":"2024-02-05T08:22:22.802738+0000","last_fullsized":"2024-02-05T08:22:22.802738+0000","mapping_epoch":353,"log_start":"341'10800","ondisk_log_start":"341'10800","created":309,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'7352","last_scrub_stamp":"2024-02-05T08:20:35.938230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:35.938230+0000","objects_scrubbed":7270,"log_size":4145,"log_dups_size":0,"ondisk_log_size":4145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94437376,"num_objects":1441,"num_object_clones":0,"num_object_copies":4323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1441,"num_objects_unfound":0,"num_objects_dirty":1441,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14945,"num_write_kb":524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,0],"avail_no_missing":["4","0","3"],"object_location_counts":[{"shards":"0,3,4","objects":1441}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":663,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.886144+0000","last_change":"2024-02-05T08:21:44.100658+0000","last_active":"2024-02-05T08:22:20.886144+0000","last_peered":"2024-02-05T08:22:20.886144+0000","last_clean":"2024-02-05T08:22:20.886144+0000","last_became_active":"2024-02-05T08:18:34.642239+0000","last_became_peered":"2024-02-05T08:18:34.642239+0000","last_unstale":"2024-02-05T08:22:20.886144+0000","last_undegraded":"2024-02-05T08:22:20.886144+0000","last_fullsized":"2024-02-05T08:22:20.886144+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:44.100428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:44.100428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:55.743623+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":667,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.886238+0000","last_change":"2024-02-05T08:22:09.105745+0000","last_active":"2024-02-05T08:22:20.886238+0000","last_peered":"2024-02-05T08:22:20.886238+0000","last_clean":"2024-02-05T08:22:20.886238+0000","last_became_active":"2024-02-05T08:22:09.105439+0000","last_became_peered":"2024-02-05T08:22:09.105439+0000","last_unstale":"2024-02-05T08:22:20.886238+0000","last_undegraded":"2024-02-05T08:22:20.886238+0000","last_fullsized":"2024-02-05T08:22:20.886238+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:37.971622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.971622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":675,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.887908+0000","last_change":"2024-02-05T08:21:40.182453+0000","last_active":"2024-02-05T08:22:20.887908+0000","last_peered":"2024-02-05T08:22:20.887908+0000","last_clean":"2024-02-05T08:22:20.887908+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:22:20.887908+0000","last_undegraded":"2024-02-05T08:22:20.887908+0000","last_fullsized":"2024-02-05T08:22:20.887908+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:40.182334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_clean_scrub_stamp":"2024-02-05T08:21:40.182334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:09.265384+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"236'35","reported_seq":443,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.889881+0000","last_change":"2024-02-05T08:21:49.635918+0000","last_active":"2024-02-05T08:22:20.889881+0000","last_peered":"2024-02-05T08:22:20.889881+0000","last_clean":"2024-02-05T08:22:20.889881+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:22:20.889881+0000","last_undegraded":"2024-02-05T08:22:20.889881+0000","last_fullsized":"2024-02-05T08:22:20.889881+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'35","last_scrub_stamp":"2024-02-05T08:21:49.635790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:49.635790+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:08.866578+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"357'15404","reported_seq":19025,"reported_epoch":357,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:22:23.331371+0000","last_change":"2024-02-05T08:22:10.304224+0000","last_active":"2024-02-05T08:22:23.331371+0000","last_peered":"2024-02-05T08:22:23.331371+0000","last_clean":"2024-02-05T08:22:08.065323+0000","last_became_active":"2024-02-05T08:22:10.228950+0000","last_became_peered":"2024-02-05T08:22:10.228950+0000","last_unstale":"2024-02-05T08:22:23.331371+0000","last_undegraded":"2024-02-05T08:22:23.331371+0000","last_fullsized":"2024-02-05T08:22:23.331371+0000","mapping_epoch":353,"log_start":"340'10559","ondisk_log_start":"340'10559","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"351'14332","last_scrub_stamp":"2024-02-05T08:22:07.026503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:22:07.026503+0000","objects_scrubbed":2490,"log_size":4845,"log_dups_size":0,"ondisk_log_size":4845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90701824,"num_objects":1384,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":108,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15404,"num_write_kb":537216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1695,"num_bytes_recovered":111083520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,0],"avail_no_missing":["2","0","3"],"object_location_counts":[{"shards":"0,2,3","objects":1384}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":20,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.887022+0000","last_change":"2024-02-05T08:22:09.101508+0000","last_active":"2024-02-05T08:22:20.887022+0000","last_peered":"2024-02-05T08:22:20.887022+0000","last_clean":"2024-02-05T08:22:20.887022+0000","last_became_active":"2024-02-05T08:22:09.100986+0000","last_became_peered":"2024-02-05T08:22:09.100986+0000","last_unstale":"2024-02-05T08:22:20.887022+0000","last_undegraded":"2024-02-05T08:22:20.887022+0000","last_fullsized":"2024-02-05T08:22:20.887022+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:37.204550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_clean_scrub_stamp":"2024-02-05T08:21:37.204550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:50.193611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":682,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.882823+0000","last_change":"2024-02-05T08:21:38.087273+0000","last_active":"2024-02-05T08:22:20.882823+0000","last_peered":"2024-02-05T08:22:20.882823+0000","last_clean":"2024-02-05T08:22:20.882823+0000","last_became_active":"2024-02-05T08:18:34.645830+0000","last_became_peered":"2024-02-05T08:18:34.645830+0000","last_unstale":"2024-02-05T08:22:20.882823+0000","last_undegraded":"2024-02-05T08:22:20.882823+0000","last_fullsized":"2024-02-05T08:22:20.882823+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:38.087157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:21:38.087157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:43.004194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":685,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.893335+0000","last_change":"2024-02-05T08:21:15.391166+0000","last_active":"2024-02-05T08:22:20.893335+0000","last_peered":"2024-02-05T08:22:20.893335+0000","last_clean":"2024-02-05T08:22:20.893335+0000","last_became_active":"2024-02-05T08:18:34.633214+0000","last_became_peered":"2024-02-05T08:18:34.633214+0000","last_unstale":"2024-02-05T08:22:20.893335+0000","last_undegraded":"2024-02-05T08:22:20.893335+0000","last_fullsized":"2024-02-05T08:22:20.893335+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:15.391056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:15.391056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:31.413224+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"262'19","reported_seq":489,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.894937+0000","last_change":"2024-02-05T08:22:10.132226+0000","last_active":"2024-02-05T08:22:20.894937+0000","last_peered":"2024-02-05T08:22:20.894937+0000","last_clean":"2024-02-05T08:22:20.894937+0000","last_became_active":"2024-02-05T08:22:10.107689+0000","last_became_peered":"2024-02-05T08:22:10.107689+0000","last_unstale":"2024-02-05T08:22:20.894937+0000","last_undegraded":"2024-02-05T08:22:20.894937+0000","last_fullsized":"2024-02-05T08:22:20.894937+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'19","last_scrub_stamp":"2024-02-05T08:19:17.554161+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.554161+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,0],"avail_no_missing":["4","0","3"],"object_location_counts":[{"shards":"0,3,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.a","version":"357'15055","reported_seq":15295,"reported_epoch":357,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T08:22:23.125934+0000","last_change":"2024-02-05T08:22:10.301068+0000","last_active":"2024-02-05T08:22:23.125934+0000","last_peered":"2024-02-05T08:22:23.125934+0000","last_clean":"2024-02-05T08:22:08.079599+0000","last_became_active":"2024-02-05T08:22:10.217298+0000","last_became_peered":"2024-02-05T08:22:10.217298+0000","last_unstale":"2024-02-05T08:22:23.125934+0000","last_undegraded":"2024-02-05T08:22:23.125934+0000","last_fullsized":"2024-02-05T08:22:23.125934+0000","mapping_epoch":353,"log_start":"338'9985","ondisk_log_start":"338'9985","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"344'12470","last_scrub_stamp":"2024-02-05T08:21:47.018241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:21:47.018241+0000","objects_scrubbed":4021,"log_size":5070,"log_dups_size":0,"ondisk_log_size":5070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1486,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91160576,"num_objects":1391,"num_object_clones":0,"num_object_copies":4173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1391,"num_objects_unfound":0,"num_objects_dirty":1391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15055,"num_write_kb":526272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,0],"avail_no_missing":["6","0","5"],"object_location_counts":[{"shards":"0,5,6","objects":1391}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"82'130","reported_seq":445,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.897477+0000","last_change":"2024-02-05T08:22:12.147142+0000","last_active":"2024-02-05T08:22:20.897477+0000","last_peered":"2024-02-05T08:22:20.897477+0000","last_clean":"2024-02-05T08:22:20.897477+0000","last_became_active":"2024-02-05T08:22:12.146682+0000","last_became_peered":"2024-02-05T08:22:12.146682+0000","last_unstale":"2024-02-05T08:22:20.897477+0000","last_undegraded":"2024-02-05T08:22:20.897477+0000","last_fullsized":"2024-02-05T08:22:20.897477+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:47.509712+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:47.509712+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:52.926942+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":367,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.899647+0000","last_change":"2024-02-05T08:19:44.925889+0000","last_active":"2024-02-05T08:22:20.899647+0000","last_peered":"2024-02-05T08:22:20.899647+0000","last_clean":"2024-02-05T08:19:41.831505+0000","last_became_active":"2024-02-05T08:19:44.925889+0000","last_became_peered":"2024-02-05T08:19:44.925889+0000","last_unstale":"2024-02-05T08:22:20.899647+0000","last_undegraded":"2024-02-05T08:22:20.899647+0000","last_fullsized":"2024-02-05T08:19:44.914970+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:17:54.817641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:17:54.817641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":689,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.896773+0000","last_change":"2024-02-05T08:19:44.903354+0000","last_active":"2024-02-05T08:22:20.896773+0000","last_peered":"2024-02-05T08:22:20.896773+0000","last_clean":"2024-02-05T08:19:41.832449+0000","last_became_active":"2024-02-05T08:19:44.903354+0000","last_became_peered":"2024-02-05T08:19:44.903354+0000","last_unstale":"2024-02-05T08:22:20.896773+0000","last_undegraded":"2024-02-05T08:19:44.873323+0000","last_fullsized":"2024-02-05T08:19:44.873213+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:18:48.543343+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:18:48.543343+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":679,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.895159+0000","last_change":"2024-02-05T08:22:10.146009+0000","last_active":"2024-02-05T08:22:20.895159+0000","last_peered":"2024-02-05T08:22:20.895159+0000","last_clean":"2024-02-05T08:22:20.895159+0000","last_became_active":"2024-02-05T08:22:10.145623+0000","last_became_peered":"2024-02-05T08:22:10.145623+0000","last_unstale":"2024-02-05T08:22:20.895159+0000","last_undegraded":"2024-02-05T08:22:20.895159+0000","last_fullsized":"2024-02-05T08:22:20.895159+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:46.849759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:21:46.849759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:01.373209+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"325'53","reported_seq":712,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.883097+0000","last_change":"2024-02-05T08:19:44.892891+0000","last_active":"2024-02-05T08:22:20.883097+0000","last_peered":"2024-02-05T08:22:20.883097+0000","last_clean":"2024-02-05T08:14:49.167838+0000","last_became_active":"2024-02-05T08:19:44.892891+0000","last_became_peered":"2024-02-05T08:19:44.892891+0000","last_unstale":"2024-02-05T08:22:20.883097+0000","last_undegraded":"2024-02-05T08:19:44.886039+0000","last_fullsized":"2024-02-05T08:19:44.885927+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'52","last_scrub_stamp":"2024-02-05T08:19:15.601598+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.601598+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":10}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":580,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.895808+0000","last_change":"2024-02-05T08:22:10.144886+0000","last_active":"2024-02-05T08:22:20.895808+0000","last_peered":"2024-02-05T08:22:20.895808+0000","last_clean":"2024-02-05T08:22:20.895808+0000","last_became_active":"2024-02-05T08:22:10.144479+0000","last_became_peered":"2024-02-05T08:22:10.144479+0000","last_unstale":"2024-02-05T08:22:20.895808+0000","last_undegraded":"2024-02-05T08:22:20.895808+0000","last_fullsized":"2024-02-05T08:22:20.895808+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:14.615047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.615047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.9","version":"357'15295","reported_seq":15521,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:23.512353+0000","last_change":"2024-02-05T08:21:48.036801+0000","last_active":"2024-02-05T08:22:23.512353+0000","last_peered":"2024-02-05T08:22:23.512353+0000","last_clean":"2024-02-05T08:22:23.512353+0000","last_became_active":"2024-02-05T08:19:07.333532+0000","last_became_peered":"2024-02-05T08:19:07.333532+0000","last_unstale":"2024-02-05T08:22:23.512353+0000","last_undegraded":"2024-02-05T08:22:23.512353+0000","last_fullsized":"2024-02-05T08:22:23.512353+0000","mapping_epoch":309,"log_start":"340'10682","ondisk_log_start":"340'10682","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'12709","last_scrub_stamp":"2024-02-05T08:21:48.036494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:21:48.036494+0000","objects_scrubbed":4030,"log_size":4613,"log_dups_size":0,"ondisk_log_size":4613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:13.805552+0000","scrub_duration":1518,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90374144,"num_objects":1379,"num_object_clones":0,"num_object_copies":4137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15295,"num_write_kb":533568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"82'130","reported_seq":969,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.887732+0000","last_change":"2024-02-05T08:22:12.139134+0000","last_active":"2024-02-05T08:22:20.887732+0000","last_peered":"2024-02-05T08:22:20.887732+0000","last_clean":"2024-02-05T08:22:20.887732+0000","last_became_active":"2024-02-05T08:22:12.138774+0000","last_became_peered":"2024-02-05T08:22:12.138774+0000","last_unstale":"2024-02-05T08:22:20.887732+0000","last_undegraded":"2024-02-05T08:22:20.887732+0000","last_fullsized":"2024-02-05T08:22:20.887732+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:27.164766+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:27.164766+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":675,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.888054+0000","last_change":"2024-02-05T08:22:09.032210+0000","last_active":"2024-02-05T08:22:20.888054+0000","last_peered":"2024-02-05T08:22:20.888054+0000","last_clean":"2024-02-05T08:22:20.888054+0000","last_became_active":"2024-02-05T08:18:34.643109+0000","last_became_peered":"2024-02-05T08:18:34.643109+0000","last_unstale":"2024-02-05T08:22:20.888054+0000","last_undegraded":"2024-02-05T08:22:20.888054+0000","last_fullsized":"2024-02-05T08:22:20.888054+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:09.032112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:09.032112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:21.979012+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":148,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.886363+0000","last_change":"2024-02-05T08:19:44.902069+0000","last_active":"2024-02-05T08:22:20.886363+0000","last_peered":"2024-02-05T08:22:20.886363+0000","last_clean":"2024-02-05T08:06:34.916685+0000","last_became_active":"2024-02-05T08:19:44.902069+0000","last_became_peered":"2024-02-05T08:19:44.902069+0000","last_unstale":"2024-02-05T08:22:20.886363+0000","last_undegraded":"2024-02-05T08:19:44.872249+0000","last_fullsized":"2024-02-05T08:19:44.872099+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:18:43.341831+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:18:43.341831+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":["2","6"],"object_location_counts":[{"shards":"2,6","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":696,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.888099+0000","last_change":"2024-02-05T08:22:10.091231+0000","last_active":"2024-02-05T08:22:20.888099+0000","last_peered":"2024-02-05T08:22:20.888099+0000","last_clean":"2024-02-05T08:22:20.888099+0000","last_became_active":"2024-02-05T08:22:10.090798+0000","last_became_peered":"2024-02-05T08:22:10.090798+0000","last_unstale":"2024-02-05T08:22:20.888099+0000","last_undegraded":"2024-02-05T08:22:20.888099+0000","last_fullsized":"2024-02-05T08:22:20.888099+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_clean_scrub_stamp":"2024-02-05T08:20:29.922794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3,0],"avail_no_missing":["2","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'17","reported_seq":694,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.882584+0000","last_change":"2024-02-05T08:22:09.113942+0000","last_active":"2024-02-05T08:22:20.882584+0000","last_peered":"2024-02-05T08:22:20.882584+0000","last_clean":"2024-02-05T08:22:20.882584+0000","last_became_active":"2024-02-05T08:22:09.103077+0000","last_became_peered":"2024-02-05T08:22:09.103077+0000","last_unstale":"2024-02-05T08:22:20.882584+0000","last_undegraded":"2024-02-05T08:22:20.882584+0000","last_fullsized":"2024-02-05T08:22:20.882584+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'17","last_scrub_stamp":"2024-02-05T08:21:10.019219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:10.019219+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":670,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.885719+0000","last_change":"2024-02-05T08:20:37.014562+0000","last_active":"2024-02-05T08:22:20.885719+0000","last_peered":"2024-02-05T08:22:20.885719+0000","last_clean":"2024-02-05T08:22:20.885719+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:22:20.885719+0000","last_undegraded":"2024-02-05T08:22:20.885719+0000","last_fullsized":"2024-02-05T08:22:20.885719+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:37.014448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.014448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"357'14919","reported_seq":18524,"reported_epoch":357,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-02-05T08:22:23.152929+0000","last_change":"2024-02-05T08:22:10.317624+0000","last_active":"2024-02-05T08:22:23.152929+0000","last_peered":"2024-02-05T08:22:23.152929+0000","last_clean":"2024-02-05T08:19:43.860522+0000","last_became_active":"2024-02-05T08:22:10.243301+0000","last_became_peered":"2024-02-05T08:22:10.243301+0000","last_unstale":"2024-02-05T08:22:23.152929+0000","last_undegraded":"2024-02-05T08:22:10.133651+0000","last_fullsized":"2024-02-05T08:22:10.083620+0000","mapping_epoch":353,"log_start":"338'9906","ondisk_log_start":"338'9906","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":5013,"log_dups_size":0,"ondisk_log_size":5013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91291648,"num_objects":1393,"num_object_clones":0,"num_object_copies":4179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":33,"num_objects_misplaced":1393,"num_objects_unfound":0,"num_objects_dirty":1393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14919,"num_write_kb":521984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1746,"num_bytes_recovered":114425856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":1393}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":662,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.893978+0000","last_change":"2024-02-05T08:21:38.543400+0000","last_active":"2024-02-05T08:22:20.893978+0000","last_peered":"2024-02-05T08:22:20.893978+0000","last_clean":"2024-02-05T08:22:20.893978+0000","last_became_active":"2024-02-05T08:18:34.633450+0000","last_became_peered":"2024-02-05T08:18:34.633450+0000","last_unstale":"2024-02-05T08:22:20.893978+0000","last_undegraded":"2024-02-05T08:22:20.893978+0000","last_fullsized":"2024-02-05T08:22:20.893978+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:38.543288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:38.543288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:59.833175+0000","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":957,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.886338+0000","last_change":"2024-02-05T08:22:10.102822+0000","last_active":"2024-02-05T08:22:20.886338+0000","last_peered":"2024-02-05T08:22:20.886338+0000","last_clean":"2024-02-05T08:22:20.886338+0000","last_became_active":"2024-02-05T08:22:10.102369+0000","last_became_peered":"2024-02-05T08:22:10.102369+0000","last_unstale":"2024-02-05T08:22:20.886338+0000","last_undegraded":"2024-02-05T08:22:20.886338+0000","last_fullsized":"2024-02-05T08:22:20.886338+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_clean_scrub_stamp":"2024-02-05T08:20:33.165371+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,0],"avail_no_missing":["7","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":152,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.897071+0000","last_change":"2024-02-05T08:21:06.644882+0000","last_active":"2024-02-05T08:22:20.897071+0000","last_peered":"2024-02-05T08:22:20.897071+0000","last_clean":"2024-02-05T08:22:20.897071+0000","last_became_active":"2024-02-05T08:18:34.631974+0000","last_became_peered":"2024-02-05T08:18:34.631974+0000","last_unstale":"2024-02-05T08:22:20.897071+0000","last_undegraded":"2024-02-05T08:22:20.897071+0000","last_fullsized":"2024-02-05T08:22:20.897071+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:06.644775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:06.644775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"236'30","reported_seq":546,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.890274+0000","last_change":"2024-02-05T08:21:48.670539+0000","last_active":"2024-02-05T08:22:20.890274+0000","last_peered":"2024-02-05T08:22:20.890274+0000","last_clean":"2024-02-05T08:22:20.890274+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:22:20.890274+0000","last_undegraded":"2024-02-05T08:22:20.890274+0000","last_fullsized":"2024-02-05T08:22:20.890274+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:21:48.670245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:48.670245+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:04.208440+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":655,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.888832+0000","last_change":"2024-02-05T08:21:58.197269+0000","last_active":"2024-02-05T08:22:20.888832+0000","last_peered":"2024-02-05T08:22:20.888832+0000","last_clean":"2024-02-05T08:22:20.888832+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:22:20.888832+0000","last_undegraded":"2024-02-05T08:22:20.888832+0000","last_fullsized":"2024-02-05T08:22:20.888832+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:58.197167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_clean_scrub_stamp":"2024-02-05T08:21:58.197167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:58.414542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"357'14769","reported_seq":18268,"reported_epoch":357,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:22:23.330533+0000","last_change":"2024-02-05T08:22:10.256711+0000","last_active":"2024-02-05T08:22:23.330533+0000","last_peered":"2024-02-05T08:22:23.330533+0000","last_clean":"2024-02-05T08:22:08.063724+0000","last_became_active":"2024-02-05T08:22:10.221028+0000","last_became_peered":"2024-02-05T08:22:10.221028+0000","last_unstale":"2024-02-05T08:22:23.330533+0000","last_undegraded":"2024-02-05T08:22:23.330533+0000","last_fullsized":"2024-02-05T08:22:23.330533+0000","mapping_epoch":353,"log_start":"339'9985","ondisk_log_start":"339'9985","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'7440","last_scrub_stamp":"2024-02-05T08:20:39.198303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:39.198303+0000","objects_scrubbed":7387,"log_size":4784,"log_dups_size":0,"ondisk_log_size":4784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1696,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89718784,"num_objects":1369,"num_object_clones":0,"num_object_copies":4107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":93,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14769,"num_write_kb":516416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1654,"num_bytes_recovered":108396544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,0],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2,7","objects":1369}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"82'130","reported_seq":625,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.896127+0000","last_change":"2024-02-05T08:19:44.916776+0000","last_active":"2024-02-05T08:22:20.896127+0000","last_peered":"2024-02-05T08:22:20.896127+0000","last_clean":"2024-02-05T08:12:23.979378+0000","last_became_active":"2024-02-05T08:19:44.916776+0000","last_became_peered":"2024-02-05T08:19:44.916776+0000","last_unstale":"2024-02-05T08:22:20.896127+0000","last_undegraded":"2024-02-05T08:22:20.896127+0000","last_fullsized":"2024-02-05T08:19:44.897597+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:19:07.594418+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:19:07.594418+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:23.591183+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":586,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.893498+0000","last_change":"2024-02-05T08:19:44.903734+0000","last_active":"2024-02-05T08:22:20.893498+0000","last_peered":"2024-02-05T08:22:20.893498+0000","last_clean":"2024-02-05T08:18:29.610515+0000","last_became_active":"2024-02-05T08:19:44.903734+0000","last_became_peered":"2024-02-05T08:19:44.903734+0000","last_unstale":"2024-02-05T08:22:20.893498+0000","last_undegraded":"2024-02-05T08:22:20.893498+0000","last_fullsized":"2024-02-05T08:19:44.898755+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.646431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.646431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":526,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.892074+0000","last_change":"2024-02-05T08:22:09.103119+0000","last_active":"2024-02-05T08:22:20.892074+0000","last_peered":"2024-02-05T08:22:20.892074+0000","last_clean":"2024-02-05T08:22:20.892074+0000","last_became_active":"2024-02-05T08:22:09.102613+0000","last_became_peered":"2024-02-05T08:22:09.102613+0000","last_unstale":"2024-02-05T08:22:20.892074+0000","last_undegraded":"2024-02-05T08:22:20.892074+0000","last_fullsized":"2024-02-05T08:22:20.892074+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:57.240901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:57.240901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:19.813695+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":1732,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:22.126686+0000","last_change":"2024-02-05T08:22:10.124331+0000","last_active":"2024-02-05T08:22:22.126686+0000","last_peered":"2024-02-05T08:22:22.126686+0000","last_clean":"2024-02-05T08:22:22.126686+0000","last_became_active":"2024-02-05T08:22:10.103659+0000","last_became_peered":"2024-02-05T08:22:10.103659+0000","last_unstale":"2024-02-05T08:22:22.126686+0000","last_undegraded":"2024-02-05T08:22:22.126686+0000","last_fullsized":"2024-02-05T08:22:22.126686+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:19:14.179578+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:19:14.179578+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1046,"num_read_kb":1041,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,0],"avail_no_missing":["7","0","3"],"object_location_counts":[{"shards":"0,3,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":459,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.898650+0000","last_change":"2024-02-05T08:22:09.130703+0000","last_active":"2024-02-05T08:22:20.898650+0000","last_peered":"2024-02-05T08:22:20.898650+0000","last_clean":"2024-02-05T08:22:20.898650+0000","last_became_active":"2024-02-05T08:22:09.106287+0000","last_became_peered":"2024-02-05T08:22:09.106287+0000","last_unstale":"2024-02-05T08:22:20.898650+0000","last_undegraded":"2024-02-05T08:22:20.898650+0000","last_fullsized":"2024-02-05T08:22:20.898650+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:21:52.253002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:52.253002+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:02.481741+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":654,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.896533+0000","last_change":"2024-02-05T08:22:10.108471+0000","last_active":"2024-02-05T08:22:20.896533+0000","last_peered":"2024-02-05T08:22:20.896533+0000","last_clean":"2024-02-05T08:22:20.896533+0000","last_became_active":"2024-02-05T08:22:10.108057+0000","last_became_peered":"2024-02-05T08:22:10.108057+0000","last_unstale":"2024-02-05T08:22:20.896533+0000","last_undegraded":"2024-02-05T08:22:20.896533+0000","last_fullsized":"2024-02-05T08:22:20.896533+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:35.391824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:19:35.391824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,0],"avail_no_missing":["5","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"357'14918","reported_seq":15160,"reported_epoch":357,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T08:22:23.132176+0000","last_change":"2024-02-05T08:22:10.278306+0000","last_active":"2024-02-05T08:22:23.132176+0000","last_peered":"2024-02-05T08:22:23.132176+0000","last_clean":"2024-02-05T08:22:08.038743+0000","last_became_active":"2024-02-05T08:22:10.214694+0000","last_became_peered":"2024-02-05T08:22:10.214694+0000","last_unstale":"2024-02-05T08:22:23.132176+0000","last_undegraded":"2024-02-05T08:22:23.132176+0000","last_fullsized":"2024-02-05T08:22:23.132176+0000","mapping_epoch":353,"log_start":"338'9885","ondisk_log_start":"338'9885","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'9124","last_scrub_stamp":"2024-02-05T08:21:04.237762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:21:04.237762+0000","objects_scrubbed":7297,"log_size":5033,"log_dups_size":0,"ondisk_log_size":5033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92995608,"num_objects":1420,"num_object_clones":0,"num_object_copies":4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1420,"num_objects_unfound":0,"num_objects_dirty":1420,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14918,"num_write_kb":522753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,0],"avail_no_missing":["6","0","4"],"object_location_counts":[{"shards":"0,4,6","objects":1420}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"82'130","reported_seq":706,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.889556+0000","last_change":"2024-02-05T08:22:12.143632+0000","last_active":"2024-02-05T08:22:20.889556+0000","last_peered":"2024-02-05T08:22:20.889556+0000","last_clean":"2024-02-05T08:22:20.889556+0000","last_became_active":"2024-02-05T08:22:12.143283+0000","last_became_peered":"2024-02-05T08:22:12.143283+0000","last_unstale":"2024-02-05T08:22:20.889556+0000","last_undegraded":"2024-02-05T08:22:20.889556+0000","last_fullsized":"2024-02-05T08:22:20.889556+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:56.703057+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:56.703057+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:26.302562+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":353,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.889100+0000","last_change":"2024-02-05T08:19:44.901104+0000","last_active":"2024-02-05T08:22:20.889100+0000","last_peered":"2024-02-05T08:22:20.889100+0000","last_clean":"2024-02-05T08:12:25.219295+0000","last_became_active":"2024-02-05T08:19:44.901104+0000","last_became_peered":"2024-02-05T08:19:44.901104+0000","last_unstale":"2024-02-05T08:22:20.889100+0000","last_undegraded":"2024-02-05T08:22:20.889100+0000","last_fullsized":"2024-02-05T08:19:44.872029+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:27.862543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:27.862543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":74,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.882793+0000","last_change":"2024-02-05T08:19:44.901166+0000","last_active":"2024-02-05T08:22:20.882793+0000","last_peered":"2024-02-05T08:22:20.882793+0000","last_clean":"2024-02-05T08:04:40.511631+0000","last_became_active":"2024-02-05T08:19:44.901166+0000","last_became_peered":"2024-02-05T08:19:44.901166+0000","last_unstale":"2024-02-05T08:22:20.882793+0000","last_undegraded":"2024-02-05T08:22:20.882793+0000","last_fullsized":"2024-02-05T08:19:44.885594+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:37.350622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:18:37.350622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":447,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.893508+0000","last_change":"2024-02-05T08:22:10.097894+0000","last_active":"2024-02-05T08:22:20.893508+0000","last_peered":"2024-02-05T08:22:20.893508+0000","last_clean":"2024-02-05T08:22:20.893508+0000","last_became_active":"2024-02-05T08:22:10.097338+0000","last_became_peered":"2024-02-05T08:22:10.097338+0000","last_unstale":"2024-02-05T08:22:20.893508+0000","last_undegraded":"2024-02-05T08:22:20.893508+0000","last_fullsized":"2024-02-05T08:22:20.893508+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-02-05T08:21:11.323094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:11.323094+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:34.118010+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,0],"avail_no_missing":["5","0","7"],"object_location_counts":[{"shards":"0,5,7","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":397,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.889798+0000","last_change":"2024-02-05T08:22:09.097069+0000","last_active":"2024-02-05T08:22:20.889798+0000","last_peered":"2024-02-05T08:22:20.889798+0000","last_clean":"2024-02-05T08:22:20.889798+0000","last_became_active":"2024-02-05T08:22:09.096719+0000","last_became_peered":"2024-02-05T08:22:09.096719+0000","last_unstale":"2024-02-05T08:22:20.889798+0000","last_undegraded":"2024-02-05T08:22:20.889798+0000","last_fullsized":"2024-02-05T08:22:20.889798+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:07.746072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:21:07.746072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:30.860421+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"357'15101","reported_seq":18561,"reported_epoch":357,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T08:22:23.325404+0000","last_change":"2024-02-05T08:22:10.241168+0000","last_active":"2024-02-05T08:22:23.325404+0000","last_peered":"2024-02-05T08:22:23.325404+0000","last_clean":"2024-02-05T08:22:08.047303+0000","last_became_active":"2024-02-05T08:22:10.196371+0000","last_became_peered":"2024-02-05T08:22:10.196371+0000","last_unstale":"2024-02-05T08:22:23.325404+0000","last_undegraded":"2024-02-05T08:22:23.325404+0000","last_fullsized":"2024-02-05T08:22:23.325404+0000","mapping_epoch":353,"log_start":"340'10453","ondisk_log_start":"340'10453","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"329'7365","last_scrub_stamp":"2024-02-05T08:20:34.584959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:20:34.584959+0000","objects_scrubbed":7281,"log_size":4648,"log_dups_size":0,"ondisk_log_size":4648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84738048,"num_objects":1293,"num_object_clones":0,"num_object_copies":3879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":76,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15101,"num_write_kb":524608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1632,"num_bytes_recovered":106954752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,0],"avail_no_missing":["7","0","4"],"object_location_counts":[{"shards":"0,4,7","objects":1293}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"82'130","reported_seq":697,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.892635+0000","last_change":"2024-02-05T08:22:03.746888+0000","last_active":"2024-02-05T08:22:20.892635+0000","last_peered":"2024-02-05T08:22:20.892635+0000","last_clean":"2024-02-05T08:22:20.892635+0000","last_became_active":"2024-02-05T08:19:01.815446+0000","last_became_peered":"2024-02-05T08:19:01.815446+0000","last_unstale":"2024-02-05T08:22:20.892635+0000","last_undegraded":"2024-02-05T08:22:20.892635+0000","last_fullsized":"2024-02-05T08:22:20.892635+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:22:03.746622+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:35.974462+0000","last_clean_scrub_stamp":"2024-02-05T08:22:03.746622+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:27.722871+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":522,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.896820+0000","last_change":"2024-02-05T08:22:09.100927+0000","last_active":"2024-02-05T08:22:20.896820+0000","last_peered":"2024-02-05T08:22:20.896820+0000","last_clean":"2024-02-05T08:22:20.896820+0000","last_became_active":"2024-02-05T08:22:09.100096+0000","last_became_peered":"2024-02-05T08:22:09.100096+0000","last_unstale":"2024-02-05T08:22:20.896820+0000","last_undegraded":"2024-02-05T08:22:20.896820+0000","last_fullsized":"2024-02-05T08:22:20.896820+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:14.350181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:14.350181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:40.460456+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":674,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.893544+0000","last_change":"2024-02-05T08:21:40.544316+0000","last_active":"2024-02-05T08:22:20.893544+0000","last_peered":"2024-02-05T08:22:20.893544+0000","last_clean":"2024-02-05T08:22:20.893544+0000","last_became_active":"2024-02-05T08:18:34.633263+0000","last_became_peered":"2024-02-05T08:18:34.633263+0000","last_unstale":"2024-02-05T08:22:20.893544+0000","last_undegraded":"2024-02-05T08:22:20.893544+0000","last_fullsized":"2024-02-05T08:22:20.893544+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:40.544202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:40.544202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:46.579918+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'20","reported_seq":483,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.896852+0000","last_change":"2024-02-05T08:19:44.916082+0000","last_active":"2024-02-05T08:22:20.896852+0000","last_peered":"2024-02-05T08:22:20.896852+0000","last_clean":"2024-02-05T08:14:49.165927+0000","last_became_active":"2024-02-05T08:19:44.916082+0000","last_became_peered":"2024-02-05T08:19:44.916082+0000","last_unstale":"2024-02-05T08:22:20.896852+0000","last_undegraded":"2024-02-05T08:19:44.899919+0000","last_fullsized":"2024-02-05T08:19:44.899797+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:19:09.511509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:09.511509+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"355'12","reported_seq":666,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.897414+0000","last_change":"2024-02-05T08:22:09.123548+0000","last_active":"2024-02-05T08:22:20.897414+0000","last_peered":"2024-02-05T08:22:20.897414+0000","last_clean":"2024-02-05T08:22:20.897414+0000","last_became_active":"2024-02-05T08:22:09.099076+0000","last_became_peered":"2024-02-05T08:22:09.099076+0000","last_unstale":"2024-02-05T08:22:20.897414+0000","last_undegraded":"2024-02-05T08:22:20.897414+0000","last_fullsized":"2024-02-05T08:22:20.897414+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:21:39.121890+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_clean_scrub_stamp":"2024-02-05T08:21:39.121890+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:01.677566+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.f","version":"357'15095","reported_seq":15330,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:23.146983+0000","last_change":"2024-02-05T08:21:54.952381+0000","last_active":"2024-02-05T08:22:23.146983+0000","last_peered":"2024-02-05T08:22:23.146983+0000","last_clean":"2024-02-05T08:22:23.146983+0000","last_became_active":"2024-02-05T08:19:07.346024+0000","last_became_peered":"2024-02-05T08:19:07.346024+0000","last_unstale":"2024-02-05T08:22:23.146983+0000","last_undegraded":"2024-02-05T08:22:23.146983+0000","last_fullsized":"2024-02-05T08:22:23.146983+0000","mapping_epoch":309,"log_start":"338'10025","ondisk_log_start":"338'10025","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'13210","last_scrub_stamp":"2024-02-05T08:21:54.952098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:21:54.952098+0000","objects_scrubbed":3372,"log_size":5070,"log_dups_size":0,"ondisk_log_size":5070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:21.291246+0000","scrub_duration":1325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92864512,"num_objects":1417,"num_object_clones":0,"num_object_copies":4251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15095,"num_write_kb":528384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"82'130","reported_seq":428,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.898463+0000","last_change":"2024-02-05T08:20:27.728841+0000","last_active":"2024-02-05T08:22:20.898463+0000","last_peered":"2024-02-05T08:22:20.898463+0000","last_clean":"2024-02-05T08:22:20.898463+0000","last_became_active":"2024-02-05T08:19:01.819856+0000","last_became_peered":"2024-02-05T08:19:01.819856+0000","last_unstale":"2024-02-05T08:22:20.898463+0000","last_undegraded":"2024-02-05T08:22:20.898463+0000","last_fullsized":"2024-02-05T08:22:20.898463+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:20:27.728717+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:20:27.728717+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":655,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.888371+0000","last_change":"2024-02-05T08:20:19.251720+0000","last_active":"2024-02-05T08:22:20.888371+0000","last_peered":"2024-02-05T08:22:20.888371+0000","last_clean":"2024-02-05T08:22:20.888371+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:22:20.888371+0000","last_undegraded":"2024-02-05T08:22:20.888371+0000","last_fullsized":"2024-02-05T08:22:20.888371+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:19.251464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:20:19.251464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":260,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.885210+0000","last_change":"2024-02-05T08:22:10.105894+0000","last_active":"2024-02-05T08:22:20.885210+0000","last_peered":"2024-02-05T08:22:20.885210+0000","last_clean":"2024-02-05T08:22:20.885210+0000","last_became_active":"2024-02-05T08:22:10.092509+0000","last_became_peered":"2024-02-05T08:22:10.092509+0000","last_unstale":"2024-02-05T08:22:20.885210+0000","last_undegraded":"2024-02-05T08:22:20.885210+0000","last_fullsized":"2024-02-05T08:22:20.885210+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:19:11.483052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:11.483052+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7,0],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2,7","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"236'28","reported_seq":736,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.891989+0000","last_change":"2024-02-05T08:21:13.958221+0000","last_active":"2024-02-05T08:22:20.891989+0000","last_peered":"2024-02-05T08:22:20.891989+0000","last_clean":"2024-02-05T08:22:20.891989+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:22:20.891989+0000","last_undegraded":"2024-02-05T08:22:20.891989+0000","last_fullsized":"2024-02-05T08:22:20.891989+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:21:13.957934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:13.957934+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:34.456435+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"325'12","reported_seq":199,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.897737+0000","last_change":"2024-02-05T08:19:44.915079+0000","last_active":"2024-02-05T08:22:20.897737+0000","last_peered":"2024-02-05T08:22:20.897737+0000","last_clean":"2024-02-05T08:19:41.828253+0000","last_became_active":"2024-02-05T08:19:44.915079+0000","last_became_peered":"2024-02-05T08:19:44.915079+0000","last_unstale":"2024-02-05T08:22:20.897737+0000","last_undegraded":"2024-02-05T08:19:44.888020+0000","last_fullsized":"2024-02-05T08:19:44.887894+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'11","last_scrub_stamp":"2024-02-05T08:19:34.968809+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:19:34.968809+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"357'17080","reported_seq":17384,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:22.785477+0000","last_change":"2024-02-05T08:21:06.695781+0000","last_active":"2024-02-05T08:22:22.785477+0000","last_peered":"2024-02-05T08:22:22.785477+0000","last_clean":"2024-02-05T08:22:22.785477+0000","last_became_active":"2024-02-05T08:19:07.331338+0000","last_became_peered":"2024-02-05T08:19:07.331338+0000","last_unstale":"2024-02-05T08:22:22.785477+0000","last_undegraded":"2024-02-05T08:22:22.785477+0000","last_fullsized":"2024-02-05T08:22:22.785477+0000","mapping_epoch":309,"log_start":"340'12942","ondisk_log_start":"340'12942","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"337'11401","last_scrub_stamp":"2024-02-05T08:21:06.695186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:21:06.695186+0000","objects_scrubbed":7153,"log_size":4138,"log_dups_size":0,"ondisk_log_size":4138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:27.523490+0000","scrub_duration":2629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91684864,"num_objects":1399,"num_object_clones":0,"num_object_copies":4197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14997,"num_write_kb":524672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"82'130","reported_seq":428,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.895015+0000","last_change":"2024-02-05T08:21:05.054369+0000","last_active":"2024-02-05T08:22:20.895015+0000","last_peered":"2024-02-05T08:22:20.895015+0000","last_clean":"2024-02-05T08:22:20.895015+0000","last_became_active":"2024-02-05T08:19:01.816923+0000","last_became_peered":"2024-02-05T08:19:01.816923+0000","last_unstale":"2024-02-05T08:22:20.895015+0000","last_undegraded":"2024-02-05T08:22:20.895015+0000","last_fullsized":"2024-02-05T08:22:20.895015+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:05.054231+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:05.054231+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:22.716414+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":670,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.891521+0000","last_change":"2024-02-05T08:22:08.118133+0000","last_active":"2024-02-05T08:22:20.891521+0000","last_peered":"2024-02-05T08:22:20.891521+0000","last_clean":"2024-02-05T08:22:20.891521+0000","last_became_active":"2024-02-05T08:18:34.635908+0000","last_became_peered":"2024-02-05T08:18:34.635908+0000","last_unstale":"2024-02-05T08:22:20.891521+0000","last_undegraded":"2024-02-05T08:22:20.891521+0000","last_fullsized":"2024-02-05T08:22:20.891521+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:08.117925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:08.117925+0000","last_clean_scrub_stamp":"2024-02-05T08:22:08.117925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:19.803772+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":166,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.899914+0000","last_change":"2024-02-05T08:22:09.107982+0000","last_active":"2024-02-05T08:22:20.899914+0000","last_peered":"2024-02-05T08:22:20.899914+0000","last_clean":"2024-02-05T08:22:20.899914+0000","last_became_active":"2024-02-05T08:22:09.101588+0000","last_became_peered":"2024-02-05T08:22:09.101588+0000","last_unstale":"2024-02-05T08:22:20.899914+0000","last_undegraded":"2024-02-05T08:22:20.899914+0000","last_fullsized":"2024-02-05T08:22:20.899914+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:21:08.559493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:08.559493+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"236'38","reported_seq":175,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.896942+0000","last_change":"2024-02-05T08:22:10.105164+0000","last_active":"2024-02-05T08:22:20.896942+0000","last_peered":"2024-02-05T08:22:20.896942+0000","last_clean":"2024-02-05T08:22:20.896942+0000","last_became_active":"2024-02-05T08:22:10.104639+0000","last_became_peered":"2024-02-05T08:22:10.104639+0000","last_unstale":"2024-02-05T08:22:20.896942+0000","last_undegraded":"2024-02-05T08:22:20.896942+0000","last_fullsized":"2024-02-05T08:22:20.896942+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'38","last_scrub_stamp":"2024-02-05T08:20:37.207762+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:20:37.207762+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,0],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3,7","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":666,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.891780+0000","last_change":"2024-02-05T08:22:07.078658+0000","last_active":"2024-02-05T08:22:20.891780+0000","last_peered":"2024-02-05T08:22:20.891780+0000","last_clean":"2024-02-05T08:22:20.891780+0000","last_became_active":"2024-02-05T08:18:34.636667+0000","last_became_peered":"2024-02-05T08:18:34.636667+0000","last_unstale":"2024-02-05T08:22:20.891780+0000","last_undegraded":"2024-02-05T08:22:20.891780+0000","last_fullsized":"2024-02-05T08:22:20.891780+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:07.078318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:31.133206+0000","last_clean_scrub_stamp":"2024-02-05T08:22:07.078318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:19.665445+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"357'14998","reported_seq":15233,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:22.747082+0000","last_change":"2024-02-05T08:22:10.155792+0000","last_active":"2024-02-05T08:22:22.747082+0000","last_peered":"2024-02-05T08:22:22.747082+0000","last_clean":"2024-02-05T08:22:22.747082+0000","last_became_active":"2024-02-05T08:22:10.109596+0000","last_became_peered":"2024-02-05T08:22:10.109596+0000","last_unstale":"2024-02-05T08:22:22.747082+0000","last_undegraded":"2024-02-05T08:22:22.747082+0000","last_fullsized":"2024-02-05T08:22:22.747082+0000","mapping_epoch":353,"log_start":"340'10891","ondisk_log_start":"340'10891","created":309,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'13357","last_scrub_stamp":"2024-02-05T08:21:57.951065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:21:57.951065+0000","objects_scrubbed":3214,"log_size":4107,"log_dups_size":0,"ondisk_log_size":4107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:08.006045+0000","scrub_duration":1035,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100007936,"num_objects":1526,"num_object_clones":0,"num_object_copies":4578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1526,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14998,"num_write_kb":528768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,0],"avail_no_missing":["4","0","7"],"object_location_counts":[{"shards":"0,4,7","objects":1526}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'130","reported_seq":436,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.896501+0000","last_change":"2024-02-05T08:21:48.475705+0000","last_active":"2024-02-05T08:22:20.896501+0000","last_peered":"2024-02-05T08:22:20.896501+0000","last_clean":"2024-02-05T08:22:20.896501+0000","last_became_active":"2024-02-05T08:19:01.817169+0000","last_became_peered":"2024-02-05T08:19:01.817169+0000","last_unstale":"2024-02-05T08:22:20.896501+0000","last_undegraded":"2024-02-05T08:22:20.896501+0000","last_fullsized":"2024-02-05T08:22:20.896501+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:48.475580+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:48.475580+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:03.051786+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":649,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.893349+0000","last_change":"2024-02-05T08:22:10.122045+0000","last_active":"2024-02-05T08:22:20.893349+0000","last_peered":"2024-02-05T08:22:20.893349+0000","last_clean":"2024-02-05T08:22:20.893349+0000","last_became_active":"2024-02-05T08:22:10.121606+0000","last_became_peered":"2024-02-05T08:22:10.121606+0000","last_unstale":"2024-02-05T08:22:20.893349+0000","last_undegraded":"2024-02-05T08:22:20.893349+0000","last_fullsized":"2024-02-05T08:22:20.893349+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:34.613476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:18:34.613476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2,0],"avail_no_missing":["5","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"355'9","reported_seq":884,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.900289+0000","last_change":"2024-02-05T08:22:09.114866+0000","last_active":"2024-02-05T08:22:20.900289+0000","last_peered":"2024-02-05T08:22:20.900289+0000","last_clean":"2024-02-05T08:22:20.900289+0000","last_became_active":"2024-02-05T08:22:09.106117+0000","last_became_peered":"2024-02-05T08:22:09.106117+0000","last_unstale":"2024-02-05T08:22:20.900289+0000","last_undegraded":"2024-02-05T08:22:20.900289+0000","last_fullsized":"2024-02-05T08:22:20.900289+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:22:06.070321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:06.070321+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:31.254858+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":147,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.897176+0000","last_change":"2024-02-05T08:19:44.902070+0000","last_active":"2024-02-05T08:22:20.897176+0000","last_peered":"2024-02-05T08:22:20.897176+0000","last_clean":"2024-02-05T08:19:41.828067+0000","last_became_active":"2024-02-05T08:19:44.902070+0000","last_became_peered":"2024-02-05T08:19:44.902070+0000","last_unstale":"2024-02-05T08:22:20.897176+0000","last_undegraded":"2024-02-05T08:22:20.897176+0000","last_fullsized":"2024-02-05T08:19:44.887805+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:07.027521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:07.027521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"236'27","reported_seq":653,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.899524+0000","last_change":"2024-02-05T08:19:44.912476+0000","last_active":"2024-02-05T08:22:20.899524+0000","last_peered":"2024-02-05T08:22:20.899524+0000","last_clean":"2024-02-05T08:18:29.610409+0000","last_became_active":"2024-02-05T08:19:44.912476+0000","last_became_peered":"2024-02-05T08:19:44.912476+0000","last_unstale":"2024-02-05T08:22:20.899524+0000","last_undegraded":"2024-02-05T08:19:44.888218+0000","last_fullsized":"2024-02-05T08:19:44.888087+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:18:40.384115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:40.384115+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"357'14996","reported_seq":15107,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:23.116941+0000","last_change":"2024-02-05T08:19:44.913164+0000","last_active":"2024-02-05T08:22:23.116941+0000","last_peered":"2024-02-05T08:22:23.116941+0000","last_clean":"2024-02-05T08:19:43.862508+0000","last_became_active":"2024-02-05T08:19:44.913164+0000","last_became_peered":"2024-02-05T08:19:44.913164+0000","last_unstale":"2024-02-05T08:22:23.116941+0000","last_undegraded":"2024-02-05T08:19:44.871956+0000","last_fullsized":"2024-02-05T08:19:44.871842+0000","mapping_epoch":324,"log_start":"338'9982","ondisk_log_start":"338'9982","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":5014,"log_dups_size":0,"ondisk_log_size":5014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88735744,"num_objects":1354,"num_object_clones":0,"num_object_copies":4062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14996,"num_write_kb":523200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":1354}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":152,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.898755+0000","last_change":"2024-02-05T08:20:38.715971+0000","last_active":"2024-02-05T08:22:20.898755+0000","last_peered":"2024-02-05T08:22:20.898755+0000","last_clean":"2024-02-05T08:22:20.898755+0000","last_became_active":"2024-02-05T08:18:34.627261+0000","last_became_peered":"2024-02-05T08:18:34.627261+0000","last_unstale":"2024-02-05T08:22:20.898755+0000","last_undegraded":"2024-02-05T08:22:20.898755+0000","last_fullsized":"2024-02-05T08:22:20.898755+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:20:38.715837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:20:38.715837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":673,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.894353+0000","last_change":"2024-02-05T08:22:02.481157+0000","last_active":"2024-02-05T08:22:20.894353+0000","last_peered":"2024-02-05T08:22:20.894353+0000","last_clean":"2024-02-05T08:22:20.894353+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:22:20.894353+0000","last_undegraded":"2024-02-05T08:22:20.894353+0000","last_fullsized":"2024-02-05T08:22:20.894353+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:02.481045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:02.481045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:20.156078+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":662,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.891858+0000","last_change":"2024-02-05T08:21:07.952159+0000","last_active":"2024-02-05T08:22:20.891858+0000","last_peered":"2024-02-05T08:22:20.891858+0000","last_clean":"2024-02-05T08:22:20.891858+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:22:20.891858+0000","last_undegraded":"2024-02-05T08:22:20.891858+0000","last_fullsized":"2024-02-05T08:22:20.891858+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:07.952031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:21:07.952031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:31.866127+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"236'19","reported_seq":711,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.899897+0000","last_change":"2024-02-05T08:19:44.926373+0000","last_active":"2024-02-05T08:22:20.899897+0000","last_peered":"2024-02-05T08:22:20.899897+0000","last_clean":"2024-02-05T08:19:41.831642+0000","last_became_active":"2024-02-05T08:19:44.926373+0000","last_became_peered":"2024-02-05T08:19:44.926373+0000","last_unstale":"2024-02-05T08:22:20.899897+0000","last_undegraded":"2024-02-05T08:19:44.921915+0000","last_fullsized":"2024-02-05T08:19:44.916232+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:18:36.561837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:18:36.561837+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"357'15182","reported_seq":15415,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:23.315270+0000","last_change":"2024-02-05T08:21:58.973650+0000","last_active":"2024-02-05T08:22:23.315270+0000","last_peered":"2024-02-05T08:22:23.315270+0000","last_clean":"2024-02-05T08:22:23.315270+0000","last_became_active":"2024-02-05T08:19:07.346034+0000","last_became_peered":"2024-02-05T08:19:07.346034+0000","last_unstale":"2024-02-05T08:22:23.315270+0000","last_undegraded":"2024-02-05T08:22:23.315270+0000","last_fullsized":"2024-02-05T08:22:23.315270+0000","mapping_epoch":309,"log_start":"339'10378","ondisk_log_start":"339'10378","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"348'13507","last_scrub_stamp":"2024-02-05T08:21:58.973355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:21:58.973355+0000","objects_scrubbed":3107,"log_size":4804,"log_dups_size":0,"ondisk_log_size":4804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:02.737058+0000","scrub_duration":1011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91226112,"num_objects":1392,"num_object_clones":0,"num_object_copies":4176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15182,"num_write_kb":530368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'130","reported_seq":428,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.893160+0000","last_change":"2024-02-05T08:21:07.292072+0000","last_active":"2024-02-05T08:22:20.893160+0000","last_peered":"2024-02-05T08:22:20.893160+0000","last_clean":"2024-02-05T08:22:20.893160+0000","last_became_active":"2024-02-05T08:19:01.817126+0000","last_became_peered":"2024-02-05T08:19:01.817126+0000","last_unstale":"2024-02-05T08:22:20.893160+0000","last_undegraded":"2024-02-05T08:22:20.893160+0000","last_fullsized":"2024-02-05T08:22:20.893160+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:07.291933+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:07.291933+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:30.946377+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":686,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.882694+0000","last_change":"2024-02-05T08:22:03.196082+0000","last_active":"2024-02-05T08:22:20.882694+0000","last_peered":"2024-02-05T08:22:20.882694+0000","last_clean":"2024-02-05T08:22:20.882694+0000","last_became_active":"2024-02-05T08:18:34.645067+0000","last_became_peered":"2024-02-05T08:18:34.645067+0000","last_unstale":"2024-02-05T08:22:20.882694+0000","last_undegraded":"2024-02-05T08:22:20.882694+0000","last_fullsized":"2024-02-05T08:22:20.882694+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:22:03.195955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:03.195955+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:21.457068+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":80,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.898797+0000","last_change":"2024-02-05T08:22:10.106423+0000","last_active":"2024-02-05T08:22:20.898797+0000","last_peered":"2024-02-05T08:22:20.898797+0000","last_clean":"2024-02-05T08:22:20.898797+0000","last_became_active":"2024-02-05T08:22:10.105785+0000","last_became_peered":"2024-02-05T08:22:10.105785+0000","last_unstale":"2024-02-05T08:22:20.898797+0000","last_undegraded":"2024-02-05T08:22:20.898797+0000","last_fullsized":"2024-02-05T08:22:20.898797+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:18:45.429303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:18:45.429303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":386,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.889981+0000","last_change":"2024-02-05T08:19:44.915255+0000","last_active":"2024-02-05T08:22:20.889981+0000","last_peered":"2024-02-05T08:22:20.889981+0000","last_clean":"2024-02-05T08:19:41.829592+0000","last_became_active":"2024-02-05T08:19:44.915255+0000","last_became_peered":"2024-02-05T08:19:44.915255+0000","last_unstale":"2024-02-05T08:22:20.889981+0000","last_undegraded":"2024-02-05T08:22:20.889981+0000","last_fullsized":"2024-02-05T08:19:44.872335+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:10.372287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:19:10.372287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":["6","4"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"236'31","reported_seq":693,"reported_epoch":357,"state":"active+undersized+degraded","last_fresh":"2024-02-05T08:22:20.895522+0000","last_change":"2024-02-05T08:19:44.922652+0000","last_active":"2024-02-05T08:22:20.895522+0000","last_peered":"2024-02-05T08:22:20.895522+0000","last_clean":"2024-02-05T08:19:41.830877+0000","last_became_active":"2024-02-05T08:19:44.922652+0000","last_became_peered":"2024-02-05T08:19:44.922652+0000","last_unstale":"2024-02-05T08:22:20.895522+0000","last_undegraded":"2024-02-05T08:19:44.913741+0000","last_fullsized":"2024-02-05T08:19:44.913632+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'31","last_scrub_stamp":"2024-02-05T08:19:24.342770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:24.342770+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[{"shards":"2,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"357'15100","reported_seq":15217,"reported_epoch":357,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-02-05T08:22:22.792349+0000","last_change":"2024-02-05T08:22:10.260897+0000","last_active":"2024-02-05T08:22:22.792349+0000","last_peered":"2024-02-05T08:22:22.792349+0000","last_clean":"2024-02-05T08:19:43.841958+0000","last_became_active":"2024-02-05T08:22:10.186921+0000","last_became_peered":"2024-02-05T08:22:10.186921+0000","last_unstale":"2024-02-05T08:22:22.792349+0000","last_undegraded":"2024-02-05T08:22:10.114883+0000","last_fullsized":"2024-02-05T08:22:10.086177+0000","mapping_epoch":353,"log_start":"341'11010","ondisk_log_start":"341'11010","created":309,"last_epoch_clean":310,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:06.306967+0000","last_clean_scrub_stamp":"2024-02-05T08:19:06.306967+0000","objects_scrubbed":0,"log_size":4090,"log_dups_size":0,"ondisk_log_size":4090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94896128,"num_objects":1448,"num_object_clones":0,"num_object_copies":4344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1448,"num_objects_misplaced":1448,"num_objects_unfound":0,"num_objects_dirty":1448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15100,"num_write_kb":529536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1448}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"82'130","reported_seq":960,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.886835+0000","last_change":"2024-02-05T08:21:19.070269+0000","last_active":"2024-02-05T08:22:20.886835+0000","last_peered":"2024-02-05T08:22:20.886835+0000","last_clean":"2024-02-05T08:22:20.886835+0000","last_became_active":"2024-02-05T08:19:01.816744+0000","last_became_peered":"2024-02-05T08:19:01.816744+0000","last_unstale":"2024-02-05T08:22:20.886835+0000","last_undegraded":"2024-02-05T08:22:20.886835+0000","last_fullsized":"2024-02-05T08:22:20.886835+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:19.069725+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:19.069725+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:33.977459+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":636,"reported_epoch":357,"state":"active+undersized","last_fresh":"2024-02-05T08:22:20.900349+0000","last_change":"2024-02-05T08:19:44.927513+0000","last_active":"2024-02-05T08:22:20.900349+0000","last_peered":"2024-02-05T08:22:20.900349+0000","last_clean":"2024-02-05T08:19:41.831929+0000","last_became_active":"2024-02-05T08:19:44.927513+0000","last_became_peered":"2024-02-05T08:19:44.927513+0000","last_unstale":"2024-02-05T08:22:20.900349+0000","last_undegraded":"2024-02-05T08:22:20.900349+0000","last_fullsized":"2024-02-05T08:19:44.916139+0000","mapping_epoch":324,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.282641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.282641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":891,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.883464+0000","last_change":"2024-02-05T08:21:53.992678+0000","last_active":"2024-02-05T08:22:20.883464+0000","last_peered":"2024-02-05T08:22:20.883464+0000","last_clean":"2024-02-05T08:22:20.883464+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:22:20.883464+0000","last_undegraded":"2024-02-05T08:22:20.883464+0000","last_fullsized":"2024-02-05T08:22:20.883464+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:21:53.992536+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:21:53.992536+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:12.669789+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":165,"reported_epoch":357,"state":"active+clean+remapped","last_fresh":"2024-02-05T08:22:20.898658+0000","last_change":"2024-02-05T08:22:10.105915+0000","last_active":"2024-02-05T08:22:20.898658+0000","last_peered":"2024-02-05T08:22:20.898658+0000","last_clean":"2024-02-05T08:22:20.898658+0000","last_became_active":"2024-02-05T08:22:10.105479+0000","last_became_peered":"2024-02-05T08:22:10.105479+0000","last_unstale":"2024-02-05T08:22:20.898658+0000","last_undegraded":"2024-02-05T08:22:20.898658+0000","last_fullsized":"2024-02-05T08:22:20.898658+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:07.643746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:22:07.643746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:29.567240+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,0],"avail_no_missing":["3","0","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"236'20","reported_seq":759,"reported_epoch":357,"state":"active+clean","last_fresh":"2024-02-05T08:22:20.898966+0000","last_change":"2024-02-05T08:21:16.960225+0000","last_active":"2024-02-05T08:22:20.898966+0000","last_peered":"2024-02-05T08:22:20.898966+0000","last_clean":"2024-02-05T08:22:20.898966+0000","last_became_active":"2024-02-05T08:18:34.637413+0000","last_became_peered":"2024-02-05T08:18:34.637413+0000","last_unstale":"2024-02-05T08:22:20.898966+0000","last_undegraded":"2024-02-05T08:22:20.898966+0000","last_fullsized":"2024-02-05T08:22:20.898966+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:21:16.959305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:16.959305+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:22:29.189495+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1464139800,"num_objects":22342,"num_object_clones":0,"num_object_copies":67026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2835,"num_objects_misplaced":10246,"num_objects_unfound":0,"num_objects_dirty":22342,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":240578,"num_write_kb":8413377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6729,"num_bytes_recovered":440860672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5044383744,"data_stored":5044371528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75009,"ondisk_log_size":75009,"up":42,"acting":45,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1046,"num_read_kb":1041,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":6,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":31,"acting":32,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":4414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":80,"acting":87,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":73,"num_objects_misplaced":25,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":315,"num_bytes_recovered":6122,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":766,"ondisk_log_size":766,"up":78,"acting":82,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":81,"acting":89,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":85,"acting":88,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037977,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1035888,"kb_used_data":441064,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92709264,"statfs":{"total":95995035648,"available":94934286336,"internally_reserved":0,"allocated":451649536,"data_stored":451331742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":609089994},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:22:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2410000000000001,"15min":1.625},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.718,"5min":27.376000000000001,"15min":60.127000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.3660000000000001,"15min":1.696},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":5.1580000000000004,"5min":27.456,"15min":58.283000000000001},"last":2.2149999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.544,"15min":1.512},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.22900000000000001},"max":{"1min":2.7810000000000001,"5min":27.518999999999998,"15min":38.399999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.083,"5min":1.4330000000000001,"15min":1.3680000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.22500000000000001},"max":{"1min":4.2409999999999997,"5min":27.503,"15min":38.195999999999998},"last":0.32200000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:22:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":0.99399999999999999,"15min":1.0700000000000001},"min":{"1min":0.26300000000000001,"5min":0.26100000000000001,"15min":0.24099999999999999},"max":{"1min":4.492,"5min":4.492,"15min":7.6779999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.98299999999999998,"15min":1.1910000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.24299999999999999},"max":{"1min":2.7000000000000002,"5min":4.7199999999999998,"15min":8.9100000000000001},"last":0.63700000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:22:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.90500000000000003,"15min":1.373},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":2.7429999999999999,"5min":4.6840000000000002,"15min":58.515000000000001},"last":2.8069999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.97299999999999998,"15min":1.365},"min":{"1min":0.34100000000000003,"5min":0.28699999999999998,"15min":0.27000000000000002},"max":{"1min":1.508,"5min":4.0960000000000001,"15min":58.225999999999999},"last":2.2440000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:22:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.016,"15min":1.034},"min":{"1min":0.26600000000000001,"5min":0.24099999999999999,"15min":0.22},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":49.601999999999997},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.1339999999999999,"15min":1.0920000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":2.7869999999999999,"5min":7.835,"15min":49.795999999999999},"last":2.8410000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:22:00 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":1.016,"15min":1.0860000000000001},"min":{"1min":0.32000000000000001,"5min":0.26800000000000002,"15min":0.24299999999999999},"max":{"1min":2.2930000000000001,"5min":5.1769999999999996,"15min":49.914000000000001},"last":2.855},{"interface":"front","average":{"1min":1.2709999999999999,"5min":1.2669999999999999,"15min":1.131},"min":{"1min":0.33200000000000002,"5min":0.29799999999999999,"15min":0.27000000000000002},"max":{"1min":8.7240000000000002,"5min":9.4459999999999997,"15min":52.256},"last":0.57499999999999996}]},{"osd":6,"last update":"Mon Feb 5 08:22:00 2024","interfaces":[{"interface":"back","average":{"1min":3.0179999999999998,"5min":2.5649999999999999,"15min":1.405},"min":{"1min":0.32800000000000001,"5min":0.28799999999999998,"15min":0.28599999999999998},"max":{"1min":23.800000000000001,"5min":84.254999999999995,"15min":84.254999999999995},"last":2.1739999999999999},{"interface":"front","average":{"1min":3.5230000000000001,"5min":2.6459999999999999,"15min":1.4159999999999999},"min":{"1min":0.38900000000000001,"5min":0.32500000000000001,"15min":0.27300000000000002},"max":{"1min":29.742999999999999,"5min":84.667000000000002,"15min":84.667000000000002},"last":0.82699999999999996}]}]},{"osd":6,"up_from":51,"seq":219043332319,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1128848,"kb_used_data":642312,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92616304,"statfs":{"total":95995035648,"available":94839095296,"internally_reserved":0,"allocated":657727488,"data_stored":657440583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":498203078},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:21:35 2024","interfaces":[{"interface":"back","average":{"1min":2.23,"5min":1.996,"15min":1.889},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":14.836,"5min":76.123000000000005,"15min":102.819},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.80300000000000005,"5min":1.6779999999999999,"15min":1.845},"min":{"1min":0.26800000000000002,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":2.359,"5min":76.037000000000006,"15min":102.75},"last":3.0600000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.90900000000000003,"15min":1.169},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.22700000000000001},"max":{"1min":1.1970000000000001,"5min":6.0229999999999997,"15min":23.773},"last":0.503},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.96299999999999997,"15min":1.1319999999999999},"min":{"1min":0.30299999999999999,"5min":0.26300000000000001,"15min":0.23400000000000001},"max":{"1min":3.0249999999999999,"5min":5.0209999999999999,"15min":23.719000000000001},"last":0.73699999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":1.1479999999999999,"15min":1.1819999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.1539999999999999,"5min":7.3499999999999996,"15min":22.503},"last":0.312},{"interface":"front","average":{"1min":0.69499999999999995,"5min":1.044,"15min":1.0669999999999999},"min":{"1min":0.36399999999999999,"5min":0.27300000000000002,"15min":0.23400000000000001},"max":{"1min":1.494,"5min":6.9720000000000004,"15min":22.466000000000001},"last":3.1499999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:21:35 2024","interfaces":[{"interface":"back","average":{"1min":2.1320000000000001,"5min":1.073,"15min":1.4430000000000001},"min":{"1min":0.29799999999999999,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":14.957000000000001,"5min":14.957000000000001,"15min":102.438},"last":0.35599999999999998},{"interface":"front","average":{"1min":1.083,"5min":1.1399999999999999,"15min":1.452},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29099999999999998},"max":{"1min":3.1899999999999999,"5min":7.4000000000000004,"15min":102.288},"last":0.55000000000000004}]},{"osd":4,"last update":"Mon Feb 5 08:21:35 2024","interfaces":[{"interface":"back","average":{"1min":5.0199999999999996,"5min":1.8660000000000001,"15min":1.042},"min":{"1min":0.34999999999999998,"5min":0.251,"15min":0.223},"max":{"1min":46.125,"5min":46.125,"15min":46.125},"last":3.1269999999999998},{"interface":"front","average":{"1min":5.6689999999999996,"5min":2.0619999999999998,"15min":1.139},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":48.817999999999998,"5min":48.817999999999998,"15min":48.817999999999998},"last":2.0270000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:21:35 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":1.1870000000000001,"15min":1.1100000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.24199999999999999},"max":{"1min":2.8479999999999999,"5min":5.4340000000000002,"15min":43.643000000000001},"last":3.1000000000000001},{"interface":"front","average":{"1min":1.8720000000000001,"5min":1.3580000000000001,"15min":1.1659999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":9.2279999999999998,"5min":9.2279999999999998,"15min":43.893000000000001},"last":1.859}]},{"osd":7,"last update":"Mon Feb 5 08:22:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.246,"15min":1.018},"min":{"1min":0.32700000000000001,"5min":0.314,"15min":0.29999999999999999},"max":{"1min":4.7530000000000001,"5min":9.0229999999999997,"15min":24.728999999999999},"last":3.1829999999999998},{"interface":"front","average":{"1min":2.4900000000000002,"5min":1.782,"15min":1.173},"min":{"1min":0.318,"5min":0.30199999999999999,"15min":0.29299999999999998},"max":{"1min":14.988,"5min":14.988,"15min":21.91},"last":0.41099999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1172192,"kb_used_data":617624,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92572960,"statfs":{"total":95995035648,"available":94794711040,"internally_reserved":0,"allocated":632446976,"data_stored":632002797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":1.1659999999999999,"15min":1.625},"min":{"1min":0.221,"5min":0.183,"15min":0.183},"max":{"1min":1.0660000000000001,"5min":36.462000000000003,"15min":86.105000000000004},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.331,"15min":1.907},"min":{"1min":0.26300000000000001,"5min":0.21099999999999999,"15min":0.20699999999999999},"max":{"1min":8.0340000000000007,"5min":35.878,"15min":85.665000000000006},"last":0.40600000000000003}]},{"osd":2,"last update":"Mon Feb 5 08:18:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.84299999999999997,"15min":1.373},"min":{"1min":0.20899999999999999,"5min":0.20599999999999999,"15min":0.191},"max":{"1min":1.4930000000000001,"5min":4.2859999999999996,"15min":49.030999999999999},"last":2.855},{"interface":"front","average":{"1min":1.2210000000000001,"5min":0.92100000000000004,"15min":1.599},"min":{"1min":0.255,"5min":0.20999999999999999,"15min":0.19900000000000001},"max":{"1min":7.298,"5min":7.298,"15min":48.984000000000002},"last":2.5550000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.754,"15min":1.5189999999999999},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.20599999999999999},"max":{"1min":1.026,"5min":5.3289999999999997,"15min":49.067999999999998},"last":1.252},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.81699999999999995,"15min":1.46},"min":{"1min":0.32100000000000001,"5min":0.254,"15min":0.24199999999999999},"max":{"1min":2.3119999999999998,"5min":4.3529999999999998,"15min":48.734999999999999},"last":1.212}]},{"osd":4,"last update":"Mon Feb 5 08:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.4119999999999999,"15min":1.272},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.26000000000000001},"max":{"1min":1.2030000000000001,"5min":24.196999999999999,"15min":25.952000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":1.3100000000000001,"15min":1.3600000000000001},"min":{"1min":0.33700000000000002,"5min":0.31,"15min":0.27100000000000002},"max":{"1min":0.92100000000000004,"5min":24.215,"15min":25.922000000000001},"last":1.228}]},{"osd":5,"last update":"Mon Feb 5 08:18:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.492,"15min":1.3999999999999999},"min":{"1min":0.40100000000000002,"5min":0.39400000000000002,"15min":0.28699999999999998},"max":{"1min":3.1699999999999999,"5min":9.6820000000000004,"15min":54.552999999999997},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.319,"5min":1.296,"15min":1.4379999999999999},"min":{"1min":0.32000000000000001,"5min":0.311,"15min":0.28999999999999998},"max":{"1min":4.1239999999999997,"5min":6.0259999999999998,"15min":55.811},"last":1.1259999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:19:37 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.97399999999999998,"15min":1.2390000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.5069999999999999,"5min":3.6819999999999999,"15min":54.384},"last":1.196},{"interface":"front","average":{"1min":0.67000000000000004,"5min":1.0920000000000001,"15min":1.2470000000000001},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.34899999999999998},"max":{"1min":1.456,"5min":4.4370000000000003,"15min":55.210999999999999},"last":0.56399999999999995}]},{"osd":7,"last update":"Mon Feb 5 08:18:52 2024","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.45,"15min":1.3089999999999999},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":5.7640000000000002,"5min":6.0579999999999998,"15min":13.749000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.3280000000000001,"15min":1.242},"min":{"1min":0.42399999999999999,"5min":0.379,"15min":0.379},"max":{"1min":2.4020000000000001,"5min":4.6180000000000003,"15min":7.9640000000000004},"last":1.1579999999999999}]}]},{"osd":0,"up_from":0,"seq":34359738621,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411571,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1135280,"kb_used_data":540328,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92609872,"statfs":{"total":95995035648,"available":94832508928,"internally_reserved":0,"allocated":553295872,"data_stored":552977662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":609089990},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:21:59 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":1.177,"15min":2.2280000000000002},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":5.2610000000000001,"5min":30.731999999999999,"15min":107.155},"last":0.185},{"interface":"front","average":{"1min":1.393,"5min":1.4299999999999999,"15min":2.4489999999999998},"min":{"1min":0.23799999999999999,"5min":0.22600000000000001,"15min":0.20799999999999999},"max":{"1min":7.3620000000000001,"5min":30.838999999999999,"15min":107.221},"last":0.40899999999999997}]},{"osd":1,"last update":"Mon Feb 5 08:18:54 2024","interfaces":[{"interface":"back","average":{"1min":2.4540000000000002,"5min":1.167,"15min":1.5860000000000001},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.17999999999999999},"max":{"1min":35.695999999999998,"5min":35.695999999999998,"15min":55.942},"last":0.28299999999999997},{"interface":"front","average":{"1min":2.9209999999999998,"5min":1.581,"15min":1.752},"min":{"1min":0.216,"5min":0.216,"15min":0.193},"max":{"1min":30.765000000000001,"5min":30.765000000000001,"15min":55.960999999999999},"last":0.79700000000000004}]},{"osd":3,"last update":"Mon Feb 5 08:22:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.008,"15min":1.931},"min":{"1min":0.29199999999999998,"5min":0.21199999999999999,"15min":0.20499999999999999},"max":{"1min":5.375,"5min":6.5039999999999996,"15min":68.108000000000004},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.83199999999999996,"15min":1.6830000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":1.6499999999999999,"5min":6.6920000000000002,"15min":67.945999999999998},"last":0.218}]},{"osd":4,"last update":"Mon Feb 5 08:21:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.226,"15min":1.504},"min":{"1min":0.28899999999999998,"5min":0.28100000000000003,"15min":0.27400000000000002},"max":{"1min":7.2549999999999999,"5min":7.6189999999999998,"15min":31.596},"last":0.505},{"interface":"front","average":{"1min":1.089,"5min":1.2490000000000001,"15min":1.343},"min":{"1min":0.41299999999999998,"5min":0.29499999999999998,"15min":0.252},"max":{"1min":2.6739999999999999,"5min":9.5060000000000002,"15min":31.57},"last":0.46400000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:22:12 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.673,"15min":1.7010000000000001},"min":{"1min":0.308,"5min":0.246,"15min":0.246},"max":{"1min":4.2999999999999998,"5min":26.373000000000001,"15min":58.799999999999997},"last":0.52000000000000002},{"interface":"front","average":{"1min":1.744,"5min":1.6830000000000001,"15min":1.6599999999999999},"min":{"1min":0.33600000000000002,"5min":0.28799999999999998,"15min":0.26900000000000002},"max":{"1min":6.0759999999999996,"5min":27.968,"15min":58.615000000000002},"last":0.76200000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:21:48 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":1.266,"15min":1.377},"min":{"1min":0.39300000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":7.1059999999999999,"5min":7.1059999999999999,"15min":15.859},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.252,"15min":1.2070000000000001},"min":{"1min":0.35999999999999999,"5min":0.317,"15min":0.317},"max":{"1min":4.6390000000000002,"5min":9.1739999999999995,"15min":9.1739999999999995},"last":0.64400000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:21:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.218,"15min":1.3440000000000001},"min":{"1min":0.33900000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":6.1109999999999998,"5min":7.7530000000000001,"15min":15.986000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.476,"15min":1.5660000000000001},"min":{"1min":0.35699999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":7.2210000000000001,"5min":11.945,"15min":15.951000000000001},"last":2.7050000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117230,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1093728,"kb_used_data":538936,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92651424,"statfs":{"total":95995035648,"available":94875058176,"internally_reserved":0,"allocated":551870464,"data_stored":551536806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":567867849},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.399,"15min":1.855},"min":{"1min":0.20100000000000001,"5min":0.19700000000000001,"15min":0.17399999999999999},"max":{"1min":1.5960000000000001,"5min":52.831000000000003,"15min":63.43},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":1.546,"15min":1.998},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.4009999999999998,"5min":53.859999999999999,"15min":63.048999999999999},"last":0.997}]},{"osd":1,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.93500000000000005,"15min":1.5109999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.217},"max":{"1min":3.3119999999999998,"5min":23.154,"15min":51.405999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.89200000000000002,"15min":1.8260000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.17399999999999999},"max":{"1min":3.54,"5min":23.207000000000001,"15min":51.189999999999998},"last":2.1379999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:21:27 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.1519999999999999,"15min":1.724},"min":{"1min":0.23000000000000001,"5min":0.22,"15min":0.22},"max":{"1min":2.23,"5min":35.277000000000001,"15min":62.902999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":1.048,"15min":1.5249999999999999},"min":{"1min":0.23400000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.633,"5min":35.442999999999998,"15min":63.468000000000004},"last":0.38}]},{"osd":4,"last update":"Mon Feb 5 08:21:46 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.90700000000000003,"15min":1.1359999999999999},"min":{"1min":0.28999999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":4.7329999999999997,"5min":6.298,"15min":7.7229999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.872,"15min":1.1599999999999999},"min":{"1min":0.34000000000000002,"5min":0.32900000000000001,"15min":0.29599999999999999},"max":{"1min":2.3610000000000002,"5min":5.8159999999999998,"15min":10.539999999999999},"last":2.2360000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:22:11 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.123,"15min":1.875},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.255},"max":{"1min":4.6779999999999999,"5min":11.519,"15min":117.432},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":1.0580000000000001,"15min":1.859},"min":{"1min":0.30599999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.355,"5min":11.071,"15min":114.828},"last":1.337}]},{"osd":6,"last update":"Mon Feb 5 08:21:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":0.98799999999999999,"15min":1.2470000000000001},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.2149999999999999,"5min":3.6360000000000001,"15min":10.622},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.056,"15min":1.2470000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":3.431,"5min":7.0439999999999996,"15min":7.8639999999999999},"last":0.40200000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:21:50 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.1100000000000001,"15min":1.798},"min":{"1min":0.41999999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":4.7629999999999999,"5min":6.3730000000000002,"15min":114.84999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":1.129,"5min":0.96099999999999997,"15min":1.8160000000000001},"min":{"1min":0.38800000000000001,"5min":0.38700000000000001,"15min":0.33600000000000002},"max":{"1min":4.4290000000000003,"5min":4.4290000000000003,"15min":114.871},"last":0.47699999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953705,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1470504,"kb_used_data":835424,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92274648,"statfs":{"total":95995035648,"available":94489239552,"internally_reserved":0,"allocated":855474176,"data_stored":855162869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.3240000000000001,"15min":1.7909999999999999},"min":{"1min":0.26700000000000002,"5min":0.192,"15min":0.192},"max":{"1min":4.1420000000000003,"5min":25.600999999999999,"15min":73.078999999999994},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.585,"5min":1.385,"15min":1.7749999999999999},"min":{"1min":0.27600000000000002,"5min":0.23599999999999999,"15min":0.21099999999999999},"max":{"1min":4.3789999999999996,"5min":23.030999999999999,"15min":73.412000000000006},"last":2.2469999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.97299999999999998,"15min":1.486},"min":{"1min":0.30399999999999999,"5min":0.23300000000000001,"15min":0.21099999999999999},"max":{"1min":1.4590000000000001,"5min":8.3320000000000007,"15min":125.054},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.89300000000000002,"15min":1.387},"min":{"1min":0.309,"5min":0.25700000000000001,"15min":0.23499999999999999},"max":{"1min":1.1910000000000001,"5min":6.2370000000000001,"15min":125.08799999999999},"last":3.1699999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.0660000000000001,"15min":1.3759999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1849999999999996,"5min":4.8159999999999998,"15min":96.849999999999994},"last":0.63},{"interface":"front","average":{"1min":1.599,"5min":1.123,"15min":1.4299999999999999},"min":{"1min":0.371,"5min":0.27100000000000002,"15min":0.26300000000000001},"max":{"1min":4.4290000000000003,"5min":5.6879999999999997,"15min":96.143000000000001},"last":2.1739999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":1.3220000000000001,"15min":1.986},"min":{"1min":0.312,"5min":0.312,"15min":0.24399999999999999},"max":{"1min":4.4550000000000001,"5min":8.9269999999999996,"15min":73.602999999999994},"last":2.145},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1419999999999999,"15min":1.954},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.27200000000000002},"max":{"1min":3.2879999999999998,"5min":4.2599999999999998,"15min":74.817999999999998},"last":0.79000000000000004}]},{"osd":5,"last update":"Mon Feb 5 08:22:19 2024","interfaces":[{"interface":"back","average":{"1min":1.788,"5min":1.411,"15min":1.0960000000000001},"min":{"1min":0.315,"5min":0.30499999999999999,"15min":0.224},"max":{"1min":4.9279999999999999,"5min":6.1909999999999998,"15min":34.167000000000002},"last":1.796},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.3939999999999999,"15min":1.131},"min":{"1min":0.35599999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":5.1120000000000001,"5min":7.851,"15min":36.051000000000002},"last":2.0939999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:21:30 2024","interfaces":[{"interface":"back","average":{"1min":2.194,"5min":1.4039999999999999,"15min":0.91000000000000003},"min":{"1min":0.42899999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.8929999999999998,"5min":5.5910000000000002,"15min":5.5910000000000002},"last":2.2719999999999998},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.3,"15min":0.86099999999999999},"min":{"1min":0.40799999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.7810000000000001,"5min":4.2969999999999997,"15min":5.5049999999999999},"last":1.2869999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:21:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4339999999999999,"15min":0.96899999999999997},"min":{"1min":0.34899999999999998,"5min":0.307,"15min":0.27000000000000002},"max":{"1min":4.282,"5min":12.683999999999999,"15min":12.683999999999999},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.407,"15min":0.94699999999999995},"min":{"1min":0.434,"5min":0.33900000000000002,"15min":0.27900000000000003},"max":{"1min":6.4180000000000001,"5min":6.4180000000000001,"15min":6.4180000000000001},"last":7.6820000000000004}]}]},{"osd":5,"up_from":42,"seq":180388626659,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1161340,"kb_used_data":526036,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92583812,"statfs":{"total":95995035648,"available":94805823488,"internally_reserved":0,"allocated":538660864,"data_stored":538363212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:22:00 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":1.3879999999999999,"15min":2.024},"min":{"1min":0.22,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":3.581,"5min":30.065999999999999,"15min":143.06299999999999},"last":1.218},{"interface":"front","average":{"1min":1.351,"5min":1.4339999999999999,"15min":1.984},"min":{"1min":0.26600000000000001,"5min":0.26400000000000001,"15min":0.20499999999999999},"max":{"1min":4.2779999999999996,"5min":30.173999999999999,"15min":143.09700000000001},"last":0.91900000000000004}]},{"osd":1,"last update":"Mon Feb 5 08:18:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.321,"15min":1.163},"min":{"1min":0.42799999999999999,"5min":0.22500000000000001,"15min":0.215},"max":{"1min":2.8090000000000002,"5min":5.8940000000000001,"15min":46.200000000000003},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.1859999999999999,"15min":1.23},"min":{"1min":0.29699999999999999,"5min":0.246,"15min":0.22900000000000001},"max":{"1min":4.7969999999999997,"5min":5.2210000000000001,"15min":46.343000000000004},"last":0.35099999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:22:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2749999999999999,"15min":1.22},"min":{"1min":0.29699999999999999,"5min":0.245,"15min":0.245},"max":{"1min":3.1800000000000002,"5min":17.09,"15min":17.09},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.018,"5min":1.4059999999999999,"15min":1.331},"min":{"1min":0.26400000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.0739999999999998,"5min":17.515999999999998,"15min":17.515999999999998},"last":1.5609999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:22:00 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.204,"15min":1.823},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":4.6340000000000003,"5min":7.165,"15min":143.03399999999999},"last":1.7729999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.111,"15min":1.905},"min":{"1min":0.28699999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.617,"5min":4.1929999999999996,"15min":143.46799999999999},"last":0.377}]},{"osd":4,"last update":"Mon Feb 5 08:22:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.224,"15min":0.92500000000000004},"min":{"1min":0.29799999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.3609999999999998,"5min":7.7389999999999999,"15min":7.7389999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.26,"15min":0.90500000000000003},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":4.5970000000000004,"5min":7.2380000000000004,"15min":7.2380000000000004},"last":0.68500000000000005}]},{"osd":6,"last update":"Mon Feb 5 08:21:40 2024","interfaces":[{"interface":"back","average":{"1min":3.1179999999999999,"5min":1.6899999999999999,"15min":1.046},"min":{"1min":0.34699999999999998,"5min":0.33200000000000002,"15min":0.26600000000000001},"max":{"1min":30.581,"5min":30.581,"15min":30.581},"last":0.44},{"interface":"front","average":{"1min":3.077,"5min":1.627,"15min":1.04},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.316},"max":{"1min":30.620999999999999,"5min":30.620999999999999,"15min":30.620999999999999},"last":0.47299999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:22:00 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.407,"15min":1.079},"min":{"1min":0.316,"5min":0.28799999999999998,"15min":0.25900000000000001},"max":{"1min":4.2210000000000001,"5min":7.0330000000000004,"15min":9.6940000000000008},"last":34.695},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.415,"15min":1.0349999999999999},"min":{"1min":0.32300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":7.8550000000000004,"5min":9.2360000000000007,"15min":9.2360000000000007},"last":34.514000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":869404672,"data_stored":869400600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":645464064,"data_stored":645464064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":543621120,"data_stored":543621120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":537657344,"data_stored":537657344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":832507904,"data_stored":832503832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":531955712,"data_stored":531955712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":637407232,"data_stored":637403160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":446365696,"data_stored":446365696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:22:29.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool set unique_pool_3 pgp_num 16 2024-02-05T08:22:29.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:29.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:29 smithi196 ceph-mon[33018]: pgmap v793: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 731 B/s rd, 0 B/s wr, 996 op/s; 2595/63684 objects degraded (4.075%); 9274/63684 objects misplaced (14.563%); 23 MiB/s, 367 objects/s recovering 2024-02-05T08:22:29.848 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:29 smithi134 ceph-mon[39278]: pgmap v793: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 82 active+clean, 22 active+undersized, 19 active+undersized+degraded, 28 active+clean+remapped; 1.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 731 B/s rd, 0 B/s wr, 996 op/s; 2595/63684 objects degraded (4.075%); 9274/63684 objects misplaced (14.563%); 23 MiB/s, 367 objects/s recovering 2024-02-05T08:22:29.861 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:29.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:29.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:29.963 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:29.998 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:29.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:29.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:30.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:30.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:30.136 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:30.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:30.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:30.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:30.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:30.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:30.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:30.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:30.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:30.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:30.549 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:30.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:30.584 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:30.639 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:30.652 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:30.687 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:30.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:30.727 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:30 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 2595/63684 objects degraded (4.075%), 21 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:30.727 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:30 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 2595/63684 objects degraded (4.075%), 21 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:30.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:30 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 2595/63684 objects degraded (4.075%), 21 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:30.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:30.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:30.927 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:30.963 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:30.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:31.064 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:31.102 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:31.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:31.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:31.203 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:31.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:31.243 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:31.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:31.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:31.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:22:31.385 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:31.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:31.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:31.528 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:31.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:31.630 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:31.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:31.670 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:31.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:31.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:31.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:31 smithi134 ceph-mon[30086]: pgmap v794: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 1.1 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 2252/52254 objects degraded (4.310%); 7734/52254 objects misplaced (14.801%); 22 MiB/s, 358 objects/s recovering 2024-02-05T08:22:31.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:31 smithi134 ceph-mon[39278]: pgmap v794: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 1.1 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 2252/52254 objects degraded (4.310%); 7734/52254 objects misplaced (14.801%); 22 MiB/s, 358 objects/s recovering 2024-02-05T08:22:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:31 smithi196 ceph-mon[33018]: pgmap v794: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 1.1 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 2252/52254 objects degraded (4.310%); 7734/52254 objects misplaced (14.801%); 22 MiB/s, 358 objects/s recovering 2024-02-05T08:22:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:31.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:31.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:31.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:31.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:31.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:31.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:32.058 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:32.101 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:32.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:32.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:32.203 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:32.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:32.236 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:32.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:32.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:32.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:32.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:32.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:32.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:32.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:32.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:32.605 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:32.641 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:32.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:32.743 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:32.784 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:32.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:32.885 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:32.917 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:32.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:33.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:33.050 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:33.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:33.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:33.193 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:33.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:33.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:33.331 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:33.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:33.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:33.433 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:33.467 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:33.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:33.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:33.604 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:33.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:33.705 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:33.740 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:33.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:33.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:33.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:33.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:33.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:33.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:33.944 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:33 smithi196 ceph-mon[33018]: pgmap v795: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 1.1 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 640 op/s; 2252/52254 objects degraded (4.310%); 7734/52254 objects misplaced (14.801%); 2.3 MiB/s, 36 objects/s recovering 2024-02-05T08:22:33.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:34.012 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:34.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:34.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:33 smithi134 ceph-mon[30086]: pgmap v795: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 1.1 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 640 op/s; 2252/52254 objects degraded (4.310%); 7734/52254 objects misplaced (14.801%); 2.3 MiB/s, 36 objects/s recovering 2024-02-05T08:22:34.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:33 smithi134 ceph-mon[39278]: pgmap v795: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 1.1 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 640 op/s; 2252/52254 objects degraded (4.310%); 7734/52254 objects misplaced (14.801%); 2.3 MiB/s, 36 objects/s recovering 2024-02-05T08:22:34.114 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:34.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:34.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:34.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:34.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:34.291 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:34.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:34.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:34.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:22:33 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:22:34.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:34.430 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:34.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:34.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:34.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:34.566 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:34.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:34.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:34.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:34.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:34.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:34.838 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:34.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:34.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:34.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:34.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:34.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:34.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:35.079 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:35.114 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:35.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:35.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:35.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:35.249 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:35.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:35.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:35.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:35.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:35.393 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:35.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:35.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:35.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:35.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:35.638 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:35.672 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:35.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:35.773 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:35.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:35.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:35.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:35.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:35.948 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:35.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:36.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:35 smithi134 ceph-mon[30086]: pgmap v796: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 679 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.19k op/s; 1457/33147 objects degraded (4.396%); 4933/33147 objects misplaced (14.882%); 1.9 MiB/s, 30 objects/s recovering 2024-02-05T08:22:36.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:35 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:36.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:35 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 1457/33147 objects degraded (4.396%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:36.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:35 smithi134 ceph-mon[39278]: pgmap v796: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 679 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.19k op/s; 1457/33147 objects degraded (4.396%); 4933/33147 objects misplaced (14.882%); 1.9 MiB/s, 30 objects/s recovering 2024-02-05T08:22:36.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:35 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:36.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:35 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 1457/33147 objects degraded (4.396%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:36.050 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:35 smithi196 ceph-mon[33018]: pgmap v796: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 679 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.19k op/s; 1457/33147 objects degraded (4.396%); 4933/33147 objects misplaced (14.882%); 1.9 MiB/s, 30 objects/s recovering 2024-02-05T08:22:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:35 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:36.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:35 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 1457/33147 objects degraded (4.396%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:36.082 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:36.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:36.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:36.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:36.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:36.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:36.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:36.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:36.360 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:36.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:36.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:36.495 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:36.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:36.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:36.597 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:36.626 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:36.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:36.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:36.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:36.770 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:36.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:36.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:36.912 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:36.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:37.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:37.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:36 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:22:37.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:36 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:22:37.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:37.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:37.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:36 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:22:37.152 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:37.191 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:37.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:37.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:37.293 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:37.337 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:37.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:37.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:37.479 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:37.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:37.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:37.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:37.621 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:37.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:37.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:37.723 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:37.760 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:37.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:37.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:37.899 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:37.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:37.981 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:22:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:22:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:22:37.981 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:37 smithi134 ceph-mon[39278]: pgmap v797: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 679 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 961 B/s rd, 0 B/s wr, 1.08k op/s; 1457/33147 objects degraded (4.396%); 4933/33147 objects misplaced (14.882%); 1.7 MiB/s, 27 objects/s recovering 2024-02-05T08:22:37.981 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:37 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:22:37.981 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:37 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:22:37.981 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:37 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:22:37.982 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:37 smithi134 ceph-mon[30086]: pgmap v797: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 679 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 961 B/s rd, 0 B/s wr, 1.08k op/s; 1457/33147 objects degraded (4.396%); 4933/33147 objects misplaced (14.882%); 1.7 MiB/s, 27 objects/s recovering 2024-02-05T08:22:37.983 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:37 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:22:37.983 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:37 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:22:37.983 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:37 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:22:38.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:38.042 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:38.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:38.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:37 smithi196 ceph-mon[33018]: pgmap v797: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 679 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 961 B/s rd, 0 B/s wr, 1.08k op/s; 1457/33147 objects degraded (4.396%); 4933/33147 objects misplaced (14.882%); 1.7 MiB/s, 27 objects/s recovering 2024-02-05T08:22:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:37 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:22:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:37 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:22:38.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:37 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:22:38.144 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:38.183 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:38.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:38.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:38.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:38.322 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:38.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:38.424 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:38.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:38.461 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:38.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:38.562 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:38.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:38.602 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:38.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:38.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:38.746 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:38.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:38.848 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:38.881 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:38.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:38.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:38.891 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:22:38.891 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:22:38.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:39.018 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:39.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:39.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:39.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:22:39.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:39.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:39.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:39.260 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:39.295 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:39.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:39.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:39.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:39.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:39.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:39.522 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:39.577 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:39.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:39.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:39.712 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:39 smithi134 ceph-mon[30086]: pgmap v798: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 580 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 1070/28401 objects degraded (3.767%); 3751/28401 objects misplaced (13.207%); 1.5 MiB/s, 24 objects/s recovering 2024-02-05T08:22:39.713 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:39 smithi134 ceph-mon[39278]: pgmap v798: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 580 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 1070/28401 objects degraded (3.767%); 3751/28401 objects misplaced (13.207%); 1.5 MiB/s, 24 objects/s recovering 2024-02-05T08:22:39.722 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:39.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:39.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:39.857 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:39.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:39.959 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:40.000 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:40.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:40.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:39 smithi196 ceph-mon[33018]: pgmap v798: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 580 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 1070/28401 objects degraded (3.767%); 3751/28401 objects misplaced (13.207%); 1.5 MiB/s, 24 objects/s recovering 2024-02-05T08:22:40.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:40.138 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:40.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:40.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:40.226 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:40.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:40.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:40.274 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:40.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:40.375 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:40.421 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:40.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:40.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:40.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:40.565 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:40.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:40.600 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:40 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 1070/28401 objects degraded (3.767%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:40.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:40.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:40.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:40.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:40.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:40.848 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:40.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:40.860 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:40 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 1070/28401 objects degraded (3.767%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:40.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:40.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:40.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:41.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:40 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 1070/28401 objects degraded (3.767%), 20 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:41.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:41.138 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:41.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:41.240 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:41.275 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:41.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:41.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:41.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:41.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:41.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:41.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:41.518 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:41.553 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:41.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:41.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:41.632 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:22:41.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:41.688 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:41.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:41.752 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:41 smithi134 ceph-mon[30086]: pgmap v799: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 292 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 679/14571 objects degraded (4.660%); 2207/14571 objects misplaced (15.147%); 1.5 MiB/s, 24 objects/s recovering 2024-02-05T08:22:41.752 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:41 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2919646133' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:22:41.753 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:41 smithi134 ceph-mon[39278]: pgmap v799: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 292 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 679/14571 objects degraded (4.660%); 2207/14571 objects misplaced (15.147%); 1.5 MiB/s, 24 objects/s recovering 2024-02-05T08:22:41.753 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:41 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2919646133' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:22:41.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:41.814 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:41.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:41.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:41.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:41.932 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:22:41.932 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:22:41.932 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-02-05T08:22:41.932 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:22:41.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:41.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:42.041 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:41 smithi196 ceph-mon[33018]: pgmap v799: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 292 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 679/14571 objects degraded (4.660%); 2207/14571 objects misplaced (15.147%); 1.5 MiB/s, 24 objects/s recovering 2024-02-05T08:22:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:41 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2919646133' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:22:42.066 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:42.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:42.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:42.168 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:42.192 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:42.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:42.293 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:42.317 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:42.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:42.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:42.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:42.444 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:42.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:42.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:42.570 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:42.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:42.623 INFO:teuthology.orchestra.run.smithi134.stderr:set pool 10 pgp_num to 16 2024-02-05T08:22:42.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:42.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:42.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:42.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:42.809 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:42.841 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:42.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:42.942 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:42.951 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:42 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:22:42.951 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:42 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2919646133' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:22:42.951 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:42 smithi134 ceph-mon[39278]: osdmap e360: 8 total, 7 up, 7 in 2024-02-05T08:22:42.951 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:42 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3461725197' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T08:22:42.951 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:42 smithi134 ceph-mon[39278]: 3.17 scrub starts 2024-02-05T08:22:42.951 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:42 smithi134 ceph-mon[39278]: 3.17 scrub ok 2024-02-05T08:22:42.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:42 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:22:42.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:42 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2919646133' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:22:42.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:42 smithi134 ceph-mon[30086]: osdmap e360: 8 total, 7 up, 7 in 2024-02-05T08:22:42.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:42 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3461725197' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T08:22:42.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:42 smithi134 ceph-mon[30086]: 3.17 scrub starts 2024-02-05T08:22:42.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:42 smithi134 ceph-mon[30086]: 3.17 scrub ok 2024-02-05T08:22:42.978 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:42.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:42.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T08:22:43.018 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Removed 131459 objects 2024-02-05T08:22:43.018 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Clean up completed and total clean up time :111.871 2024-02-05T08:22:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:42 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:22:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:42 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2919646133' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:22:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:42 smithi196 ceph-mon[33018]: osdmap e360: 8 total, 7 up, 7 in 2024-02-05T08:22:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:42 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3461725197' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T08:22:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:42 smithi196 ceph-mon[33018]: 3.17 scrub starts 2024-02-05T08:22:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:42 smithi196 ceph-mon[33018]: 3.17 scrub ok 2024-02-05T08:22:43.080 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:43.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:43.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:43.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:43.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:43.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:43.229 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:43.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:43.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:43.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:43.355 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:43.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:43.456 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:43.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:43.482 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:43.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:43.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:43.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:43.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:43.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:43.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:43.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:43.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:43.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:43.869 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:43.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:43.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:43.918 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[30086]: pgmap v801: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 292 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 679/14571 objects degraded (4.660%); 2207/14571 objects misplaced (15.147%) 2024-02-05T08:22:43.919 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:22:43.919 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:22:43.919 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[30086]: 10.b scrub starts 2024-02-05T08:22:43.919 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[30086]: 10.b scrub ok 2024-02-05T08:22:43.919 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3461725197' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-02-05T08:22:43.919 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[30086]: osdmap e361: 8 total, 7 up, 7 in 2024-02-05T08:22:43.919 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[39278]: pgmap v801: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 292 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 679/14571 objects degraded (4.660%); 2207/14571 objects misplaced (15.147%) 2024-02-05T08:22:43.919 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:22:43.919 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:22:43.919 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[39278]: 10.b scrub starts 2024-02-05T08:22:43.920 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[39278]: 10.b scrub ok 2024-02-05T08:22:43.920 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3461725197' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-02-05T08:22:43.920 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:43 smithi134 ceph-mon[39278]: osdmap e361: 8 total, 7 up, 7 in 2024-02-05T08:22:43.955 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:43 smithi196 ceph-mon[33018]: pgmap v801: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 85 active+clean, 22 active+undersized, 19 active+undersized+degraded, 29 active+clean+remapped; 292 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 679/14571 objects degraded (4.660%); 2207/14571 objects misplaced (15.147%) 2024-02-05T08:22:43.956 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:43 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:22:43.956 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:43 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:22:43.956 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:43 smithi196 ceph-mon[33018]: 10.b scrub starts 2024-02-05T08:22:43.956 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:43 smithi196 ceph-mon[33018]: 10.b scrub ok 2024-02-05T08:22:43.956 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:43 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3461725197' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num", "val": "16"}]': finished 2024-02-05T08:22:43.956 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:43 smithi196 ceph-mon[33018]: osdmap e361: 8 total, 7 up, 7 in 2024-02-05T08:22:43.970 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:43.994 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:43.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:43.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:44.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:44.126 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:44.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:44.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:44.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:44.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:44.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:44.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:44.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:22:43 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:22:44.356 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:44.379 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:44.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:44.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:44.483 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:44.507 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:44.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:44.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:44.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:44.639 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:44.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:44.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:44.765 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:44.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:44.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:44.867 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:44.895 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:44.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:44.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:44.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:45.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:45.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:45.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:45.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:44 smithi134 ceph-mon[30086]: 5.8 scrub starts 2024-02-05T08:22:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:44 smithi134 ceph-mon[30086]: 5.8 scrub ok 2024-02-05T08:22:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:44 smithi134 ceph-mon[30086]: 3.10 scrub starts 2024-02-05T08:22:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:44 smithi134 ceph-mon[30086]: 3.10 scrub ok 2024-02-05T08:22:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:44 smithi134 ceph-mon[30086]: 5.13 scrub starts 2024-02-05T08:22:45.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:44 smithi134 ceph-mon[30086]: 5.13 scrub ok 2024-02-05T08:22:45.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:44 smithi134 ceph-mon[39278]: 5.8 scrub starts 2024-02-05T08:22:45.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:44 smithi134 ceph-mon[39278]: 5.8 scrub ok 2024-02-05T08:22:45.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:44 smithi134 ceph-mon[39278]: 3.10 scrub starts 2024-02-05T08:22:45.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:44 smithi134 ceph-mon[39278]: 3.10 scrub ok 2024-02-05T08:22:45.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:44 smithi134 ceph-mon[39278]: 5.13 scrub starts 2024-02-05T08:22:45.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:44 smithi134 ceph-mon[39278]: 5.13 scrub ok 2024-02-05T08:22:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:44 smithi196 ceph-mon[33018]: 5.8 scrub starts 2024-02-05T08:22:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:44 smithi196 ceph-mon[33018]: 5.8 scrub ok 2024-02-05T08:22:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:44 smithi196 ceph-mon[33018]: 3.10 scrub starts 2024-02-05T08:22:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:44 smithi196 ceph-mon[33018]: 3.10 scrub ok 2024-02-05T08:22:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:44 smithi196 ceph-mon[33018]: 5.13 scrub starts 2024-02-05T08:22:45.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:44 smithi196 ceph-mon[33018]: 5.13 scrub ok 2024-02-05T08:22:45.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:45.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:45.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:45.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:45.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:45.281 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:45.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:45.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:45.382 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:45.406 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:45.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:45.418 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:45.508 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:45.539 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:45.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:45.641 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:45.666 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:45.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:45.767 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:45.793 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:45.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:45.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:45.879 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[30086]: pgmap v803: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 3.4 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 77/729 objects degraded (10.562%); 62/729 objects misplaced (8.505%); 2.3 MiB/s, 37 objects/s recovering 2024-02-05T08:22:45.879 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:45.879 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[30086]: 3.6 scrub starts 2024-02-05T08:22:45.880 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[30086]: 3.6 scrub ok 2024-02-05T08:22:45.880 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[30086]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-02-05T08:22:45.880 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[30086]: 6.18 scrub starts 2024-02-05T08:22:45.880 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[30086]: 6.18 scrub ok 2024-02-05T08:22:45.880 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 77/729 objects degraded (10.562%), 18 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:45.880 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[39278]: pgmap v803: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 3.4 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 77/729 objects degraded (10.562%); 62/729 objects misplaced (8.505%); 2.3 MiB/s, 37 objects/s recovering 2024-02-05T08:22:45.880 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:45.880 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[39278]: 3.6 scrub starts 2024-02-05T08:22:45.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[39278]: 3.6 scrub ok 2024-02-05T08:22:45.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[39278]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-02-05T08:22:45.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[39278]: 6.18 scrub starts 2024-02-05T08:22:45.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[39278]: 6.18 scrub ok 2024-02-05T08:22:45.881 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:45 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 77/729 objects degraded (10.562%), 18 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:45.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:45.920 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:45.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:45.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:45.945 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:22:46.022 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:46.048 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:46.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:45 smithi196 ceph-mon[33018]: pgmap v803: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 3.4 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 77/729 objects degraded (10.562%); 62/729 objects misplaced (8.505%); 2.3 MiB/s, 37 objects/s recovering 2024-02-05T08:22:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:45 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:45 smithi196 ceph-mon[33018]: 3.6 scrub starts 2024-02-05T08:22:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:45 smithi196 ceph-mon[33018]: 3.6 scrub ok 2024-02-05T08:22:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:45 smithi196 ceph-mon[33018]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-02-05T08:22:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:45 smithi196 ceph-mon[33018]: 6.18 scrub starts 2024-02-05T08:22:46.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:45 smithi196 ceph-mon[33018]: 6.18 scrub ok 2024-02-05T08:22:46.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:45 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 77/729 objects degraded (10.562%), 18 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:46.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:46.174 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:46.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:46.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:46.275 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:46.300 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:46.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:46.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:46.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:46.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:46.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:46.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:46.553 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:46.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:46.654 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:46.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:46.679 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:46.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:46.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:46 smithi134 ceph-mon[30086]: 2.1 scrub starts 2024-02-05T08:22:46.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:46 smithi134 ceph-mon[30086]: 2.1 scrub ok 2024-02-05T08:22:46.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:46 smithi134 ceph-mon[30086]: 10.7 scrub starts 2024-02-05T08:22:46.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:46 smithi134 ceph-mon[30086]: 10.7 scrub ok 2024-02-05T08:22:46.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:46 smithi134 ceph-mon[30086]: 3.18 scrub starts 2024-02-05T08:22:46.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:46 smithi134 ceph-mon[30086]: 3.18 scrub ok 2024-02-05T08:22:46.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:46 smithi134 ceph-mon[39278]: 2.1 scrub starts 2024-02-05T08:22:46.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:46 smithi134 ceph-mon[39278]: 2.1 scrub ok 2024-02-05T08:22:46.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:46 smithi134 ceph-mon[39278]: 10.7 scrub starts 2024-02-05T08:22:46.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:46 smithi134 ceph-mon[39278]: 10.7 scrub ok 2024-02-05T08:22:46.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:46 smithi134 ceph-mon[39278]: 3.18 scrub starts 2024-02-05T08:22:46.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:46 smithi134 ceph-mon[39278]: 3.18 scrub ok 2024-02-05T08:22:46.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:46.805 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:46.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:46.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:46.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:46.931 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:46.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:47.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:47.055 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:47.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:47.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:46 smithi196 ceph-mon[33018]: 2.1 scrub starts 2024-02-05T08:22:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:46 smithi196 ceph-mon[33018]: 2.1 scrub ok 2024-02-05T08:22:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:46 smithi196 ceph-mon[33018]: 10.7 scrub starts 2024-02-05T08:22:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:46 smithi196 ceph-mon[33018]: 10.7 scrub ok 2024-02-05T08:22:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:46 smithi196 ceph-mon[33018]: 3.18 scrub starts 2024-02-05T08:22:47.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:46 smithi196 ceph-mon[33018]: 3.18 scrub ok 2024-02-05T08:22:47.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:47.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:47.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:47.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:47.284 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:47.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:47.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:47.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:47.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:47.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:47.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:47.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:47.459 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:22:47.463 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-02-05T08:22:47.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-02-05T08:22:47.479 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:22:47.498 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:22:47.519 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:22:47.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:47.539 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:22:47.558 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:47.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:47.578 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:22:47.598 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:22:47.598 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:22:47.599 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:22:47.599 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:22:47.599 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:22:47.599 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:22:47.599 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:22:47.599 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:22:47.599 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:22:47.661 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:47.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:47.685 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:47.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:47.719 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[30086]: pgmap v804: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 3.4 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 77/729 objects degraded (10.562%); 62/729 objects misplaced (8.505%); 2.3 MiB/s, 37 objects/s recovering 2024-02-05T08:22:47.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[30086]: 10.8 scrub starts 2024-02-05T08:22:47.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[30086]: 10.8 scrub ok 2024-02-05T08:22:47.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[30086]: 6.13 scrub starts 2024-02-05T08:22:47.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[30086]: 6.13 scrub ok 2024-02-05T08:22:47.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[30086]: 6.d scrub starts 2024-02-05T08:22:47.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[30086]: 6.d scrub ok 2024-02-05T08:22:47.720 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/181739118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:22:47.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[39278]: pgmap v804: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 3.4 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 77/729 objects degraded (10.562%); 62/729 objects misplaced (8.505%); 2.3 MiB/s, 37 objects/s recovering 2024-02-05T08:22:47.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[39278]: 10.8 scrub starts 2024-02-05T08:22:47.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[39278]: 10.8 scrub ok 2024-02-05T08:22:47.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[39278]: 6.13 scrub starts 2024-02-05T08:22:47.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[39278]: 6.13 scrub ok 2024-02-05T08:22:47.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[39278]: 6.d scrub starts 2024-02-05T08:22:47.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[39278]: 6.d scrub ok 2024-02-05T08:22:47.721 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/181739118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:22:47.786 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:47.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:47.810 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:47.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:47 smithi196 ceph-mon[33018]: pgmap v804: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 3.4 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 77/729 objects degraded (10.562%); 62/729 objects misplaced (8.505%); 2.3 MiB/s, 37 objects/s recovering 2024-02-05T08:22:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:47 smithi196 ceph-mon[33018]: 10.8 scrub starts 2024-02-05T08:22:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:47 smithi196 ceph-mon[33018]: 10.8 scrub ok 2024-02-05T08:22:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:47 smithi196 ceph-mon[33018]: 6.13 scrub starts 2024-02-05T08:22:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:47 smithi196 ceph-mon[33018]: 6.13 scrub ok 2024-02-05T08:22:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:47 smithi196 ceph-mon[33018]: 6.d scrub starts 2024-02-05T08:22:47.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:47 smithi196 ceph-mon[33018]: 6.d scrub ok 2024-02-05T08:22:47.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:47 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/181739118' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:22:47.911 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:47.934 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:47.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:47.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:47.997 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:22:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:22:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:22:48.036 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:48.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:48.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:48.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:48.120 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:48.129 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:22:48.129 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:22:48.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:48.185 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:48.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:48.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:48.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:48.310 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:48.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:48.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:48.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:48.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:48.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:48.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:48.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:48.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:48.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:48.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:48.687 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:48.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:48.788 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:48.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:48.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:48.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:48.835 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:48 smithi134 ceph-mon[30086]: 1.4 scrub starts 2024-02-05T08:22:48.835 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:48 smithi134 ceph-mon[30086]: 1.4 scrub ok 2024-02-05T08:22:48.835 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:48 smithi134 ceph-mon[30086]: 5.15 scrub starts 2024-02-05T08:22:48.835 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:48 smithi134 ceph-mon[30086]: 5.15 scrub ok 2024-02-05T08:22:48.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:48 smithi134 ceph-mon[39278]: 1.4 scrub starts 2024-02-05T08:22:48.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:48 smithi134 ceph-mon[39278]: 1.4 scrub ok 2024-02-05T08:22:48.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:48 smithi134 ceph-mon[39278]: 5.15 scrub starts 2024-02-05T08:22:48.835 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:48 smithi134 ceph-mon[39278]: 5.15 scrub ok 2024-02-05T08:22:48.862 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:22:48.863 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:22:48.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:48.938 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:48.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:48.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:49.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:48 smithi196 ceph-mon[33018]: 1.4 scrub starts 2024-02-05T08:22:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:48 smithi196 ceph-mon[33018]: 1.4 scrub ok 2024-02-05T08:22:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:48 smithi196 ceph-mon[33018]: 5.15 scrub starts 2024-02-05T08:22:49.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:48 smithi196 ceph-mon[33018]: 5.15 scrub ok 2024-02-05T08:22:49.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:49.065 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:49.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:49.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:49.191 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:49.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:49.292 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:49.317 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:49.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:49.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:49.444 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:49.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:49.546 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:49.570 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:49.570 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:49.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:49.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:49.638 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:49 smithi134 ceph-mon[30086]: pgmap v805: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 582 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%); 2.3 MiB/s, 37 objects/s recovering 2024-02-05T08:22:49.638 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:49 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:22:49.638 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:49 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:22:49.638 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:49 smithi134 ceph-mon[30086]: 4.0 scrub starts 2024-02-05T08:22:49.672 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:49.697 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:49.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:49.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:49.827 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:49.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:49.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:49.930 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:49.952 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:49 smithi134 ceph-mon[30086]: 4.0 scrub ok 2024-02-05T08:22:49.953 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:49 smithi134 ceph-mon[39278]: pgmap v805: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 582 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%); 2.3 MiB/s, 37 objects/s recovering 2024-02-05T08:22:49.953 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:49 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:22:49.953 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:49 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:22:49.953 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:49 smithi134 ceph-mon[39278]: 4.0 scrub starts 2024-02-05T08:22:49.953 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:49 smithi134 ceph-mon[39278]: 4.0 scrub ok 2024-02-05T08:22:49.954 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:49.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:49.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:50.056 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:49 smithi196 ceph-mon[33018]: pgmap v805: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 582 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%); 2.3 MiB/s, 37 objects/s recovering 2024-02-05T08:22:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:49 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:22:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:49 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:22:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:49 smithi196 ceph-mon[33018]: 4.0 scrub starts 2024-02-05T08:22:50.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:49 smithi196 ceph-mon[33018]: 4.0 scrub ok 2024-02-05T08:22:50.084 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:50.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:50.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:50.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:50.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:50.317 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:50.343 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:50.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:50.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:50.445 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:50.470 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:50.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:50.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:50.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:50.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:50.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:50.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:50.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:50.728 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:50.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:50.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:50.808 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 7, 6, 3] out_osds: [0] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-02-05T08:22:50.808 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:22:50.808 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-02-05T08:22:50.809 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-02-05T08:22:50.809 DEBUG:teuthology.orchestra.run.smithi134:osd.1> sudo journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.1.service 2024-02-05T08:22:50.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl start ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.1 2024-02-05T08:22:50.831 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:50.864 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:22:50.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:50.877 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:50 smithi134 ceph-mon[30086]: 5.c scrub starts 2024-02-05T08:22:50.877 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:50 smithi134 ceph-mon[30086]: 5.c scrub ok 2024-02-05T08:22:50.877 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:50 smithi134 ceph-mon[30086]: 5.2 scrub starts 2024-02-05T08:22:50.877 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:50 smithi134 ceph-mon[30086]: 5.2 scrub ok 2024-02-05T08:22:50.877 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:50 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 77/594 objects degraded (12.963%), 18 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:50.877 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:50 smithi134 ceph-mon[39278]: 5.c scrub starts 2024-02-05T08:22:50.878 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:50 smithi134 ceph-mon[39278]: 5.c scrub ok 2024-02-05T08:22:50.878 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:50 smithi134 ceph-mon[39278]: 5.2 scrub starts 2024-02-05T08:22:50.878 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:50 smithi134 ceph-mon[39278]: 5.2 scrub ok 2024-02-05T08:22:50.878 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:50 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 77/594 objects degraded (12.963%), 18 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:50.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:50.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:50.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:51.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:51.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:51.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:50 smithi196 ceph-mon[33018]: 5.c scrub starts 2024-02-05T08:22:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:50 smithi196 ceph-mon[33018]: 5.c scrub ok 2024-02-05T08:22:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:50 smithi196 ceph-mon[33018]: 5.2 scrub starts 2024-02-05T08:22:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:50 smithi196 ceph-mon[33018]: 5.2 scrub ok 2024-02-05T08:22:51.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:50 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 77/594 objects degraded (12.963%), 18 pgs degraded, 41 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:51.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:51.135 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:51.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:51.152 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:50 smithi134 systemd[1]: Starting Ceph osd.1 for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:22:51.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:51.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:51.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:51.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:51.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:51.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:51.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:51.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:51.520 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:51.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:51.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:51.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:51.650 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:51.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:51.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:51.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:51.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:51.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:51.879 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:51.883 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:22:51.892 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[30086]: pgmap v806: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 565 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%); 2.3 MiB/s, 36 objects/s recovering 2024-02-05T08:22:51.892 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[30086]: 10.2 scrub starts 2024-02-05T08:22:51.892 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[30086]: 10.2 scrub ok 2024-02-05T08:22:51.892 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[30086]: 6.10 scrub starts 2024-02-05T08:22:51.892 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[30086]: 6.10 scrub ok 2024-02-05T08:22:51.893 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/930209745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:22:51.893 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:22:51.893 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[39278]: pgmap v806: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 565 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%); 2.3 MiB/s, 36 objects/s recovering 2024-02-05T08:22:51.893 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[39278]: 10.2 scrub starts 2024-02-05T08:22:51.893 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[39278]: 10.2 scrub ok 2024-02-05T08:22:51.893 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[39278]: 6.10 scrub starts 2024-02-05T08:22:51.893 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[39278]: 6.10 scrub ok 2024-02-05T08:22:51.893 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/930209745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:22:51.893 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:51 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:22:51.905 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:51.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:51.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:52.007 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:52.032 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:52.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:52.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:51 smithi196 ceph-mon[33018]: pgmap v806: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 565 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%); 2.3 MiB/s, 36 objects/s recovering 2024-02-05T08:22:52.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:51 smithi196 ceph-mon[33018]: 10.2 scrub starts 2024-02-05T08:22:52.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:51 smithi196 ceph-mon[33018]: 10.2 scrub ok 2024-02-05T08:22:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:51 smithi196 ceph-mon[33018]: 6.10 scrub starts 2024-02-05T08:22:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:51 smithi196 ceph-mon[33018]: 6.10 scrub ok 2024-02-05T08:22:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:51 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/930209745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:22:52.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:51 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:22:52.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T08:22:52.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:52.179 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:52.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:52.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:52.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:52.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:52.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:52.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:52.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:52.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:52.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:52.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:52.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:52.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:52.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:52.689 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:52.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:52.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:52.817 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:52.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:52.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:52.919 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:52.945 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:52.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[30086]: 5.b scrub starts 2024-02-05T08:22:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[30086]: 5.b scrub ok 2024-02-05T08:22:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:22:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:22:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:22:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:22:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[30086]: osdmap e362: 8 total, 7 up, 7 in 2024-02-05T08:22:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[30086]: 6.c scrub starts 2024-02-05T08:22:53.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[30086]: 6.c scrub ok 2024-02-05T08:22:53.028 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:52 smithi134 podman[200776]: 2024-02-05 08:22:52.676123193 +0000 UTC m=+0.021110235 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:22:53.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[39278]: 5.b scrub starts 2024-02-05T08:22:53.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[39278]: 5.b scrub ok 2024-02-05T08:22:53.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:22:53.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:22:53.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:22:53.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:22:53.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[39278]: osdmap e362: 8 total, 7 up, 7 in 2024-02-05T08:22:53.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[39278]: 6.c scrub starts 2024-02-05T08:22:53.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:52 smithi134 ceph-mon[39278]: 6.c scrub ok 2024-02-05T08:22:53.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:52 smithi196 ceph-mon[33018]: 5.b scrub starts 2024-02-05T08:22:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:52 smithi196 ceph-mon[33018]: 5.b scrub ok 2024-02-05T08:22:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:52 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:22:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:52 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:22:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:52 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:22:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:52 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:22:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:52 smithi196 ceph-mon[33018]: osdmap e362: 8 total, 7 up, 7 in 2024-02-05T08:22:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:52 smithi196 ceph-mon[33018]: 6.c scrub starts 2024-02-05T08:22:53.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:52 smithi196 ceph-mon[33018]: 6.c scrub ok 2024-02-05T08:22:53.071 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:53.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:53.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:53.109 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:53.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:53.201 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:53.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:53.303 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:53.318 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 podman[200776]: 2024-02-05 08:22:53.201398732 +0000 UTC m=+0.546385812 container create 815c8cf29987cdb040764008fd3669a8e1dbb0cbf7db94ca2572df973d13102f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T08:22:53.330 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:53.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:53.432 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:53.458 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:53.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:53.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:53.560 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:53.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:53.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:53.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:53.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[30086]: pgmap v808: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 503 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%); 2.0 MiB/s, 32 objects/s recovering 2024-02-05T08:22:53.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[30086]: 5.5 deep-scrub starts 2024-02-05T08:22:53.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[30086]: 5.5 deep-scrub ok 2024-02-05T08:22:53.764 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[30086]: 5.7 scrub starts 2024-02-05T08:22:53.765 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[30086]: 5.7 scrub ok 2024-02-05T08:22:53.765 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:22:53.765 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:22:53.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[39278]: pgmap v808: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 503 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%); 2.0 MiB/s, 32 objects/s recovering 2024-02-05T08:22:53.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[39278]: 5.5 deep-scrub starts 2024-02-05T08:22:53.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[39278]: 5.5 deep-scrub ok 2024-02-05T08:22:53.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[39278]: 5.7 scrub starts 2024-02-05T08:22:53.765 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[39278]: 5.7 scrub ok 2024-02-05T08:22:53.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:22:53.766 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:22:53.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 podman[200776]: 2024-02-05 08:22:53.421906509 +0000 UTC m=+0.766893544 container init 815c8cf29987cdb040764008fd3669a8e1dbb0cbf7db94ca2572df973d13102f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-05T08:22:53.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 podman[200776]: 2024-02-05 08:22:53.42811529 +0000 UTC m=+0.773102331 container start 815c8cf29987cdb040764008fd3669a8e1dbb0cbf7db94ca2572df973d13102f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T08:22:53.766 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 podman[200776]: 2024-02-05 08:22:53.764439841 +0000 UTC m=+1.109426902 container attach 815c8cf29987cdb040764008fd3669a8e1dbb0cbf7db94ca2572df973d13102f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-activate, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-05T08:22:53.830 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:53.849 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:53.875 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:53.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:53.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:53 smithi196 ceph-mon[33018]: pgmap v808: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 503 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%); 2.0 MiB/s, 32 objects/s recovering 2024-02-05T08:22:53.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:53 smithi196 ceph-mon[33018]: 5.5 deep-scrub starts 2024-02-05T08:22:53.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:53 smithi196 ceph-mon[33018]: 5.5 deep-scrub ok 2024-02-05T08:22:53.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:53 smithi196 ceph-mon[33018]: 5.7 scrub starts 2024-02-05T08:22:53.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:53 smithi196 ceph-mon[33018]: 5.7 scrub ok 2024-02-05T08:22:53.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:22:53.965 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:22:53.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:54.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:54.003 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:54.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:54.027 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-activate[200856]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-05T08:22:54.027 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-activate[200856]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-05T08:22:54.027 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 bash[200776]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-05T08:22:54.027 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 bash[200776]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-02-05T08:22:54.027 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 bash[200776]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-05T08:22:54.027 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 bash[200776]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-05T08:22:54.027 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 bash[200776]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-05T08:22:54.027 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 bash[200776]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-05T08:22:54.027 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 bash[200776]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-05T08:22:54.027 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-activate[200856]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-02-05T08:22:54.028 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-activate[200856]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-02-05T08:22:54.028 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-activate[200856]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-02-05T08:22:54.028 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-activate[200856]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-02-05T08:22:54.028 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-activate[200856]: --> ceph-volume raw activate successful for osd ID: 1 2024-02-05T08:22:54.028 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:53 smithi134 podman[200776]: 2024-02-05 08:22:53.968283911 +0000 UTC m=+1.313270951 container died 815c8cf29987cdb040764008fd3669a8e1dbb0cbf7db94ca2572df973d13102f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-activate, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-05T08:22:54.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:54.133 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:54.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:54.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:54.259 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:54.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:54.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:54.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:22:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:22:54.360 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:54.384 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:54.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:54.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:54.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:54.510 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:54.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:54.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:54.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:54.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:54.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:54.657 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'unique_pool_3' removed 2024-02-05T08:22:54.741 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:54.768 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:54.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:54.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:54.900 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:54.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:54.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:55.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:55.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:55.027 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:55.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:54 smithi134 ceph-mon[30086]: 4.b scrub starts 2024-02-05T08:22:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:54 smithi134 ceph-mon[30086]: 4.b scrub ok 2024-02-05T08:22:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:54 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2152257750' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:22:55.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:54 smithi134 ceph-mon[30086]: 3.1f scrub starts 2024-02-05T08:22:55.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:54 smithi134 ceph-mon[30086]: 3.1f scrub ok 2024-02-05T08:22:55.028 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:54 smithi134 podman[201116]: 2024-02-05 08:22:54.846471191 +0000 UTC m=+0.891410722 container remove 815c8cf29987cdb040764008fd3669a8e1dbb0cbf7db94ca2572df973d13102f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-activate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-05T08:22:55.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:54 smithi134 ceph-mon[39278]: 4.b scrub starts 2024-02-05T08:22:55.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:54 smithi134 ceph-mon[39278]: 4.b scrub ok 2024-02-05T08:22:55.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:54 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2152257750' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:22:55.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:54 smithi134 ceph-mon[39278]: 3.1f scrub starts 2024-02-05T08:22:55.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:54 smithi134 ceph-mon[39278]: 3.1f scrub ok 2024-02-05T08:22:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:54 smithi196 ceph-mon[33018]: 4.b scrub starts 2024-02-05T08:22:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:54 smithi196 ceph-mon[33018]: 4.b scrub ok 2024-02-05T08:22:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:54 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2152257750' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:22:55.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:54 smithi196 ceph-mon[33018]: 3.1f scrub starts 2024-02-05T08:22:55.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:54 smithi196 ceph-mon[33018]: 3.1f scrub ok 2024-02-05T08:22:55.089 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:22:55.128 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:55.166 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:55.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:55.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:55.269 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:55.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:55.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:55.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:55.396 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:55.421 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:55.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:55.522 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:55.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:55.546 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:55.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:55.647 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:55.672 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:55.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:55.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:55.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:55.797 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:55.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:55.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:55.870 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:55 smithi134 ceph-mon[30086]: pgmap v809: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 5 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:22:55.870 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:55 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:55.870 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:55 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2152257750' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:22:55.871 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:55 smithi134 ceph-mon[30086]: osdmap e363: 8 total, 7 up, 7 in 2024-02-05T08:22:55.871 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:55 smithi134 ceph-mon[30086]: 3.2 scrub starts 2024-02-05T08:22:55.871 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:55 smithi134 ceph-mon[30086]: 3.2 scrub ok 2024-02-05T08:22:55.871 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:55 smithi134 ceph-mon[39278]: pgmap v809: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 5 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:22:55.871 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:55 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:55.871 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:55 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2152257750' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:22:55.871 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:55 smithi134 ceph-mon[39278]: osdmap e363: 8 total, 7 up, 7 in 2024-02-05T08:22:55.871 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:55 smithi134 ceph-mon[39278]: 3.2 scrub starts 2024-02-05T08:22:55.871 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:55 smithi134 ceph-mon[39278]: 3.2 scrub ok 2024-02-05T08:22:55.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:55.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:55.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:56.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:56.058 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:56.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:56.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:55 smithi196 ceph-mon[33018]: pgmap v809: 158 pgs: 87 active+clean, 23 active+undersized, 18 active+undersized+degraded, 30 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 5 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:22:56.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:55 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:22:56.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:55 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2152257750' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:22:56.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:55 smithi196 ceph-mon[33018]: osdmap e363: 8 total, 7 up, 7 in 2024-02-05T08:22:56.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:55 smithi196 ceph-mon[33018]: 3.2 scrub starts 2024-02-05T08:22:56.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:55 smithi196 ceph-mon[33018]: 3.2 scrub ok 2024-02-05T08:22:56.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:56.174 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T08:22:56.176 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T08:22:56.176 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-02-05T08:22:56.176 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-02-05T08:22:56.185 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:56.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:56.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:56.287 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:56.310 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:56.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:56.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:56.436 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:56.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:56.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:56.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:56.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:56.689 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:56.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:56.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:56.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:56.816 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:56.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:56.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:56.909 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:56 smithi134 ceph-mon[30086]: pgmap v811: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:22:56.909 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:56 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 77/594 objects degraded (12.963%), 18 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:56.909 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:56 smithi134 ceph-mon[39278]: pgmap v811: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:22:56.909 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:56 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 77/594 objects degraded (12.963%), 18 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:56.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:56.942 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:56.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:56.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:57.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:57.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:56 smithi196 ceph-mon[33018]: pgmap v811: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:22:57.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:56 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 77/594 objects degraded (12.963%), 18 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-02-05T08:22:57.068 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:57.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:57.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:57.170 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:57.193 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:57.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:57.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:57.295 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:57.321 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:57.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:57.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:57.445 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:57.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:57.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:57.547 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:57.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:57.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:57.673 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:57.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:57.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:57.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:57.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:57 smithi196 ceph-mon[33018]: 4.7 deep-scrub starts 2024-02-05T08:22:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:57 smithi196 ceph-mon[33018]: 4.7 deep-scrub ok 2024-02-05T08:22:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:57 smithi196 ceph-mon[33018]: 5.1d scrub starts 2024-02-05T08:22:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:57 smithi196 ceph-mon[33018]: 5.1d scrub ok 2024-02-05T08:22:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:57 smithi196 ceph-mon[33018]: 3.7 scrub starts 2024-02-05T08:22:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:57 smithi196 ceph-mon[33018]: 3.7 scrub ok 2024-02-05T08:22:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:57 smithi196 ceph-mon[33018]: 1.8 scrub starts 2024-02-05T08:22:57.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:57 smithi196 ceph-mon[33018]: 1.8 scrub ok 2024-02-05T08:22:57.823 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:57.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:57.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:57.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:57.949 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:57.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:57.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:57.989 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:22:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:22:57.989 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[39278]: 4.7 deep-scrub starts 2024-02-05T08:22:57.989 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[39278]: 4.7 deep-scrub ok 2024-02-05T08:22:57.989 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[39278]: 5.1d scrub starts 2024-02-05T08:22:57.989 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[39278]: 5.1d scrub ok 2024-02-05T08:22:57.989 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[39278]: 3.7 scrub starts 2024-02-05T08:22:57.989 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[39278]: 3.7 scrub ok 2024-02-05T08:22:57.989 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[39278]: 1.8 scrub starts 2024-02-05T08:22:57.990 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[39278]: 1.8 scrub ok 2024-02-05T08:22:57.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[30086]: 4.7 deep-scrub starts 2024-02-05T08:22:57.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[30086]: 4.7 deep-scrub ok 2024-02-05T08:22:57.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[30086]: 5.1d scrub starts 2024-02-05T08:22:57.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[30086]: 5.1d scrub ok 2024-02-05T08:22:57.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[30086]: 3.7 scrub starts 2024-02-05T08:22:57.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[30086]: 3.7 scrub ok 2024-02-05T08:22:57.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[30086]: 1.8 scrub starts 2024-02-05T08:22:57.990 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:57 smithi134 ceph-mon[30086]: 1.8 scrub ok 2024-02-05T08:22:58.051 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:58.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:58.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:58.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:58.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:58.203 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:58.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:58.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:58.328 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:22:58.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:58.329 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:58.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:58.345 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:22:58.369 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:22:58.387 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:22:58.409 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:22:58.428 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:22:58.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:22:58.449 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:22:58.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:58.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:58.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:22:58.469 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:22:58.469 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:22:58.469 INFO:teuthology.orchestra.run.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-02-05T08:22:58.469 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:22:58.469 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:22:58.470 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:22:58.470 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:22:58.470 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:22:58.470 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:22:58.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:58.572 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:22:58.572 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:22:58.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:22:58.572 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:22:58.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.572 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:131949 8.9 8:9948f8f8:::benchmark_data_smithi134_115836_object15289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:22:58.573 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:58.596390+0000", 2024-02-05T08:22:58.573 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 599.97451580400002, 2024-02-05T08:22:58.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16625135899999999, 2024-02-05T08:22:58.573 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.573 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:58.573 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.573 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:58.573 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:58.573 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 131949 2024-02-05T08:22:58.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.573 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596390+0000", 2024-02-05T08:22:58.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596390+0000", 2024-02-05T08:22:58.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596392+0000", 2024-02-05T08:22:58.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7910000000000001e-06 2024-02-05T08:22:58.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596398+0000", 2024-02-05T08:22:58.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8379999999999998e-06 2024-02-05T08:22:58.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596411+0000", 2024-02-05T08:22:58.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2870999999999999e-05 2024-02-05T08:22:58.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596420+0000", 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6859999999999995e-06 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596483+0000", 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3836000000000003e-05 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596624+0000", 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000140941 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5,6", 2024-02-05T08:22:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.596815+0000", 2024-02-05T08:22:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019104500000000001 2024-02-05T08:22:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.599923+0000", 2024-02-05T08:22:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031073260000000001 2024-02-05T08:22:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.603127+0000", 2024-02-05T08:22:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0032045609999999999 2024-02-05T08:22:58.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.603196+0000", 2024-02-05T08:22:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9079999999999996e-05 2024-02-05T08:22:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.762449+0000", 2024-02-05T08:22:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15925238899999999 2024-02-05T08:22:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:58.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.762523+0000", 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4338999999999994e-05 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.762642+0000", 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000118656 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:131951 8.9 8:9d96da9c:::benchmark_data_smithi134_115836_object15291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:12:58.599465+0000", 2024-02-05T08:22:58.579 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 599.97144135500002, 2024-02-05T08:22:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.163446905, 2024-02-05T08:22:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 131951 2024-02-05T08:22:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.580 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.599465+0000", 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.599465+0000", 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.599466+0000", 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.502e-06 2024-02-05T08:22:58.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.599471+0000", 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7879999999999997e-06 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.599482+0000", 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1389000000000001e-05 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.599490+0000", 2024-02-05T08:22:58.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5909999999999998e-06 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.599956+0000", 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00046596000000000002 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600090+0000", 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000134251 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5,6", 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.600249+0000", 2024-02-05T08:22:58.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015838199999999999 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.603175+0000", 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0029261679999999998 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.603212+0000", 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7240999999999998e-05 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.604055+0000", 2024-02-05T08:22:58.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00084325600000000002 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.762689+0000", 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15863354299999999 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.762754+0000", 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5247999999999996e-05 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:12:58.762912+0000", 2024-02-05T08:22:58.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00015758600000000001 2024-02-05T08:22:58.586 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.586 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.586 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.586 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:134711 8.9 8:9c8cde60:::benchmark_data_smithi134_115836_object18051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:22:58.587 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.188436+0000", 2024-02-05T08:22:58.587 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 596.38247017699996, 2024-02-05T08:22:58.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16349269999999999, 2024-02-05T08:22:58.587 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.587 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:58.587 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.587 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:58.587 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:58.587 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 134711 2024-02-05T08:22:58.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.587 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.188436+0000", 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.188436+0000", 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.188438+0000", 2024-02-05T08:22:58.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.79e-06 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.188443+0000", 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2859999999999999e-06 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.188456+0000", 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2670999999999999e-05 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.188464+0000", 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1130000000000001e-06 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.188523+0000", 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9434000000000001e-05 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.188641+0000", 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000117991 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5,6", 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.188753+0000", 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000111921 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.192828+0000", 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0040752519999999997 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.192971+0000", 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014303699999999999 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.194682+0000", 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0017109619999999999 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.351723+0000", 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.157040924 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.351805+0000", 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1957000000000002e-05 2024-02-05T08:22:58.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.351929+0000", 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012336199999999999 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:134924 8.9 8:94405964:::benchmark_data_smithi134_115836_object18264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:02.533391+0000", 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 596.03751525899997, 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.16313651300000001, 2024-02-05T08:22:58.593 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 134924 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.533391+0000", 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.533391+0000", 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.533392+0000", 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4330000000000001e-06 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.533398+0000", 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7300000000000002e-06 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.595 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.533411+0000", 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3045e-05 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.533429+0000", 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7897000000000002e-05 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.533663+0000", 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00023382500000000001 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.533789+0000", 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012593700000000001 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5,6", 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.533919+0000", 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013044700000000001 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538358+0000", 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0044383959999999998 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538886+0000", 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052817800000000002 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.538984+0000", 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7841000000000001e-05 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.696304+0000", 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15732083999999999 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.696414+0000", 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000109783 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:02.696527+0000", 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000113161 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.599 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:161411 8.9 8:970395ee:::benchmark_data_smithi134_115836_object44751:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:35.291027+0000", 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 563.27987881499996, 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15479031700000001, 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 161411 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.291027+0000", 2024-02-05T08:22:58.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.291027+0000", 2024-02-05T08:22:58.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.291029+0000", 2024-02-05T08:22:58.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.733e-06 2024-02-05T08:22:58.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.291035+0000", 2024-02-05T08:22:58.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7790000000000001e-06 2024-02-05T08:22:58.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.291047+0000", 2024-02-05T08:22:58.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2004999999999999e-05 2024-02-05T08:22:58.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.291054+0000", 2024-02-05T08:22:58.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5519999999999999e-06 2024-02-05T08:22:58.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294356+0000", 2024-02-05T08:22:58.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033013040000000001 2024-02-05T08:22:58.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:58.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294519+0000", 2024-02-05T08:22:58.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000163418 2024-02-05T08:22:58.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5,6", 2024-02-05T08:22:58.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.294641+0000", 2024-02-05T08:22:58.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000121983 2024-02-05T08:22:58.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.604 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.299175+0000", 2024-02-05T08:22:58.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0045341970000000002 2024-02-05T08:22:58.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.604 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.299236+0000", 2024-02-05T08:22:58.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0618999999999997e-05 2024-02-05T08:22:58.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.302669+0000", 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0034333929999999999 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.445555+0000", 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.142885858 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.445688+0000", 2024-02-05T08:22:58.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013266600000000001 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:35.445818+0000", 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012981000000000001 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.15087.0:162682 8.9 8:97529ea1:::benchmark_data_smithi134_115836_object46022:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:13:36.820696+0000", 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 561.75021024, 2024-02-05T08:22:58.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.15900682499999999, 2024-02-05T08:22:58.607 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.607 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:58.607 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.607 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.15087", 2024-02-05T08:22:58.607 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:22:58.607 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 162682 2024-02-05T08:22:58.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.607 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.607 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.607 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.820696+0000", 2024-02-05T08:22:58.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.820696+0000", 2024-02-05T08:22:58.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.820698+0000", 2024-02-05T08:22:58.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1629999999999999e-06 2024-02-05T08:22:58.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.820704+0000", 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6370000000000004e-06 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.820717+0000", 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3371e-05 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.820725+0000", 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1710000000000002e-06 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.821489+0000", 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00076381400000000003 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.821606+0000", 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000117067 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,5,6", 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.821740+0000", 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000134339 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:22:58.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.825476+0000", 2024-02-05T08:22:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0037358780000000002 2024-02-05T08:22:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.825513+0000", 2024-02-05T08:22:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6568e-05 2024-02-05T08:22:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.827776+0000", 2024-02-05T08:22:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002263438 2024-02-05T08:22:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.979435+0000", 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.151658986 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.979585+0000", 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014943700000000001 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:13:36.979703+0000", 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00011795599999999999 2024-02-05T08:22:58.612 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24344.0:3013 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94292702904192 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.615963+0000", 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 264.95494274700002, 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.073499212, 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24344", 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1272526039", 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3013 2024-02-05T08:22:58.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.614 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.614 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.614 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.615963+0000", 2024-02-05T08:22:58.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.614 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.614 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.615963+0000", 2024-02-05T08:22:58.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.614 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.615966+0000", 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1390000000000002e-06 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.615974+0000", 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4460000000000001e-06 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.615996+0000", 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.209e-05 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616008+0000", 2024-02-05T08:22:58.616 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1664e-05 2024-02-05T08:22:58.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.616 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.624254+0000", 2024-02-05T08:22:58.616 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0082457539999999992 2024-02-05T08:22:58.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.616 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:58.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.624267+0000", 2024-02-05T08:22:58.616 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3565e-05 2024-02-05T08:22:58.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.664392+0000", 2024-02-05T08:22:58.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.040125111 2024-02-05T08:22:58.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for missing object", 2024-02-05T08:22:58.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.664525+0000", 2024-02-05T08:22:58.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013255499999999999 2024-02-05T08:22:58.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.674568+0000", 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010042822999999999 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.674663+0000", 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5057999999999997e-05 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.677226+0000", 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0025634939999999999 2024-02-05T08:22:58.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.619 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:58.619 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.677318+0000", 2024-02-05T08:22:58.619 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2207999999999995e-05 2024-02-05T08:22:58.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.619 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.619 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.689463+0000", 2024-02-05T08:22:58.619 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012144304999999999 2024-02-05T08:22:58.619 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.619 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.619 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.620 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24344.0:3015 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94292702897280 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:22:58.620 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.616045+0000", 2024-02-05T08:22:58.620 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 264.95486079400001, 2024-02-05T08:22:58.620 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.093509665, 2024-02-05T08:22:58.620 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.620 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:22:58.620 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.620 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24344", 2024-02-05T08:22:58.620 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1272526039", 2024-02-05T08:22:58.620 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3015 2024-02-05T08:22:58.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616045+0000", 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616045+0000", 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.621 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616046+0000", 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8500000000000001e-07 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616049+0000", 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1719999999999999e-06 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616058+0000", 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3389999999999993e-06 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.622 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616064+0000", 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7610000000000002e-06 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.640831+0000", 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.024766535999999999 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.640840+0000", 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4350000000000003e-06 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.645324+0000", 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0044837120000001 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for missing object", 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.645467+0000", 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000142786 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.678676+0000", 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.033208803000000002 2024-02-05T08:22:58.624 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.678770+0000", 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4278999999999996e-05 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.679540+0000", 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00077013699999999995 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.679613+0000", 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3100000000000001e-05 2024-02-05T08:22:58.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.709555+0000", 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.02994192 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259330 9.1 9.2fae0b11 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664160+0000", 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 264.90674632399998, 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95273855299999999, 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259330 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664160+0000", 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664160+0000", 2024-02-05T08:22:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664161+0000", 2024-02-05T08:22:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5880000000000001e-06 2024-02-05T08:22:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664166+0000", 2024-02-05T08:22:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:58.629 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:58.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:58.630 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5800000000000002e-06 2024-02-05T08:22:58.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.630 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.630 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664187+0000", 2024-02-05T08:22:58.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1233000000000002e-05 2024-02-05T08:22:58.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664199+0000", 2024-02-05T08:22:58.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1644000000000001e-05 2024-02-05T08:22:58.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664258+0000", 2024-02-05T08:22:58.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8783000000000003e-05 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664268+0000", 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9580000000000002e-06 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.616887+0000", 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95261980700000004 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.632 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.616898+0000", 2024-02-05T08:22:58.633 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.096e-05 2024-02-05T08:22:58.633 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.633 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.633 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.633 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.633 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259337 9.2 9.d70982e2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:22:58.633 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664222+0000", 2024-02-05T08:22:58.633 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 264.90668405600002, 2024-02-05T08:22:58.633 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95069705999999998, 2024-02-05T08:22:58.633 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.633 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:22:58.633 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259337 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664222+0000", 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664222+0000", 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.634 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.635 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664223+0000", 2024-02-05T08:22:58.635 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3099999999999998e-07 2024-02-05T08:22:58.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.635 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664226+0000", 2024-02-05T08:22:58.635 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0570000000000001e-06 2024-02-05T08:22:58.635 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.635 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.635 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.635 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664234+0000", 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1210000000000007e-06 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664240+0000", 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2790000000000004e-06 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664293+0000", 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2849000000000002e-05 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.636 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664298+0000", 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6940000000000003e-06 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.614904+0000", 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95060682500000004 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.614919+0000", 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4704e-05 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.637 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259339 9.c 9.54971cbc (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664254+0000", 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 264.906651893, 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95406026300000002, 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259339 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.638 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664254+0000", 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664254+0000", 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664255+0000", 2024-02-05T08:22:58.639 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.15e-07 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664257+0000", 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5119999999999998e-06 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664264+0000", 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4409999999999998e-06 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.640 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664269+0000", 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5260000000000001e-06 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664322+0000", 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2709e-05 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664330+0000", 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0220000000000004e-06 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.641 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.618306+0000", 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95397562300000005 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.618315+0000", 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0149999999999992e-06 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259343 9.8 9.8b42e388 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664344+0000", 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 264.90656196899999, 2024-02-05T08:22:58.642 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95628875999999996, 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259343 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664344+0000", 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.643 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664344+0000", 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664345+0000", 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9100000000000004e-07 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664348+0000", 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0170000000000001e-06 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.644 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664355+0000", 2024-02-05T08:22:58.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8719999999999996e-06 2024-02-05T08:22:58.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664361+0000", 2024-02-05T08:22:58.645 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3080000000000004e-06 2024-02-05T08:22:58.645 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.645 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.645 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.645 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664403+0000", 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2247999999999999e-05 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664410+0000", 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6560000000000003e-06 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.620623+0000", 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95621348900000003 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.646 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.620633+0000", 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5789999999999993e-06 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259344 9.1 9.844dc3c1 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664418+0000", 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 264.90648851899999, 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95252151900000004, 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:22:58.647 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259344 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664418+0000", 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664418+0000", 2024-02-05T08:22:58.648 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664418+0000", 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7899999999999999e-07 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664421+0000", 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.711e-06 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664427+0000", 2024-02-05T08:22:58.649 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6279999999999999e-06 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664433+0000", 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6980000000000003e-06 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664468+0000", 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4662000000000001e-05 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:58.650 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664473+0000", 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0100000000000003e-06 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.616924+0000", 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95245070399999998 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.616939+0000", 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5627000000000001e-05 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.651 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259345 9.2 9.50719472 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664445+0000", 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 264.90646141299999, 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95056753100000002, 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259345 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.652 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664445+0000", 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664445+0000", 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664445+0000", 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2899999999999999e-07 2024-02-05T08:22:58.653 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664447+0000", 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.024e-06 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664453+0000", 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9789999999999998e-06 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664458+0000", 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2909999999999998e-06 2024-02-05T08:22:58.654 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664493+0000", 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4397999999999998e-05 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664500+0000", 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3390000000000004e-06 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.614999+0000", 2024-02-05T08:22:58.655 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95049863400000001 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.615012+0000", 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3437e-05 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259347 9.8 9.1b3af678 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664466+0000", 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 264.90643972800001, 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95619838800000001, 2024-02-05T08:22:58.656 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259347 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664466+0000", 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.657 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664466+0000", 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664467+0000", 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.58e-07 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664469+0000", 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4660000000000002e-06 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664475+0000", 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0159999999999997e-06 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664481+0000", 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7300000000000002e-06 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664518+0000", 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6606999999999997e-05 2024-02-05T08:22:58.659 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664524+0000", 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4819999999999999e-06 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.620657+0000", 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95613291199999995 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.620665+0000", 2024-02-05T08:22:58.660 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8169999999999999e-06 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259357 9.1 9.f0ac7411 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664536+0000", 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 264.90637029099997, 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95243644199999999, 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:22:58.661 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259357 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664536+0000", 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664536+0000", 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.662 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664536+0000", 2024-02-05T08:22:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0500000000000004e-07 2024-02-05T08:22:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664539+0000", 2024-02-05T08:22:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5840000000000002e-06 2024-02-05T08:22:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664552+0000", 2024-02-05T08:22:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3419e-05 2024-02-05T08:22:58.663 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664558+0000", 2024-02-05T08:22:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1190000000000002e-06 2024-02-05T08:22:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664592+0000", 2024-02-05T08:22:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3501000000000003e-05 2024-02-05T08:22:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.664 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:22:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664595+0000", 2024-02-05T08:22:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.348e-06 2024-02-05T08:22:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.616963+0000", 2024-02-05T08:22:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95236730199999997 2024-02-05T08:22:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.616972+0000", 2024-02-05T08:22:58.665 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6639999999999993e-06 2024-02-05T08:22:58.666 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.666 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.666 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.666 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.666 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.666 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15714.0:214251 10.f e348/309 10:f26750af:::benchmark_data_smithi134_174861_object82789:head v 348'13527, mlcod=348'13525)", 2024-02-05T08:22:58.666 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:21:58.845845+0000", 2024-02-05T08:22:58.666 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 59.725061355999998, 2024-02-05T08:22:58.666 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.142092944, 2024-02-05T08:22:58.666 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.666 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:58.666 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845845+0000", 2024-02-05T08:22:58.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845845+0000", 2024-02-05T08:22:58.667 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.667 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.667 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.667 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.667 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845846+0000", 2024-02-05T08:22:58.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9100000000000002e-07 2024-02-05T08:22:58.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845855+0000", 2024-02-05T08:22:58.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5170000000000005e-06 2024-02-05T08:22:58.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.668 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.668 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845868+0000", 2024-02-05T08:22:58.668 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3227999999999999e-05 2024-02-05T08:22:58.668 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.668 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845875+0000", 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7619999999999998e-06 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845938+0000", 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3083999999999994e-05 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845956+0000", 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7657000000000001e-05 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.669 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.987898+0000", 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14194200400000001 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.987938+0000", 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9801000000000001e-05 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15714.0:261422 10.4 e359/353 10:2b662601:::benchmark_data_smithi134_174861_object129960:head v 359'16286, mlcod=359'16284)", 2024-02-05T08:22:58.670 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:22:40.740006+0000", 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 17.830899718000001, 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.035030991999999997, 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740006+0000", 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740006+0000", 2024-02-05T08:22:58.671 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740007+0000", 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6899999999999997e-07 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740013+0000", 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7520000000000002e-06 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.672 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740022+0000", 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0620000000000007e-06 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740028+0000", 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4060000000000002e-06 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740071+0000", 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2679000000000003e-05 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:58.673 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740095+0000", 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.3890999999999999e-05 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.775000+0000", 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.034904994000000002 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.775037+0000", 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7539000000000001e-05 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.674 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "pg_backfill(progress 10.2 e 359/359 lb 10:46739161:::benchmark_data_smithi134_174861_object131026:head)", 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:22:40.740795+0000", 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 17.830111433999999, 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.034001994000000001, 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740795+0000", 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.675 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740795+0000", 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740795+0000", 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3000000000000001e-07 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740801+0000", 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9479999999999996e-06 2024-02-05T08:22:58.676 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740807+0000", 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4820000000000002e-06 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.740811+0000", 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0579999999999999e-06 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.774745+0000", 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.033933887000000003 2024-02-05T08:22:58.677 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.774761+0000", 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6412000000000001e-05 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.774797+0000", 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5777e-05 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.678 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15714.0:261420 10.c e359/358 10:387122f1:::benchmark_data_smithi134_174861_object129958:head v 359'16326, mlcod=359'16323)", 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:22:40.742210+0000", 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 17.828696114, 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.038862019999999997, 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.742210+0000", 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:22:58.679 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.742210+0000", 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.742211+0000", 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.7199999999999998e-07 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.742219+0000", 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8909999999999995e-06 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:22:58.680 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.773556+0000", 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.031337377999999999 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.773572+0000", 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6235e-05 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.774952+0000", 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0013796660000000001 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:22:58.681 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.774973+0000", 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0774000000000001e-05 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.780952+0000", 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0059797119999999999 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:22:40.781072+0000", 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000119592 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.682 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:22:58.683 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:22:58.683 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:22:58.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:22:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:58.759 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:58.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:22:58.860 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:58.887 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:58.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:58.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:22:59.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:59.015 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:59.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:22:59.026 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:22:58 smithi134 podman[201702]: 2024-02-05 08:22:58.55273988 +0000 UTC m=+0.650718743 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:22:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:58 smithi134 ceph-mon[30086]: 5.1c scrub starts 2024-02-05T08:22:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:58 smithi134 ceph-mon[30086]: 5.1c scrub ok 2024-02-05T08:22:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:58 smithi134 ceph-mon[30086]: pgmap v812: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:22:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:58 smithi134 ceph-mon[30086]: 6.0 scrub starts 2024-02-05T08:22:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:58 smithi134 ceph-mon[30086]: 6.0 scrub ok 2024-02-05T08:22:59.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:58 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/907773100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:22:59.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:58 smithi134 ceph-mon[39278]: 5.1c scrub starts 2024-02-05T08:22:59.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:58 smithi134 ceph-mon[39278]: 5.1c scrub ok 2024-02-05T08:22:59.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:58 smithi134 ceph-mon[39278]: pgmap v812: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:22:59.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:58 smithi134 ceph-mon[39278]: 6.0 scrub starts 2024-02-05T08:22:59.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:58 smithi134 ceph-mon[39278]: 6.0 scrub ok 2024-02-05T08:22:59.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:58 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/907773100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:22:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:58 smithi196 ceph-mon[33018]: 5.1c scrub starts 2024-02-05T08:22:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:58 smithi196 ceph-mon[33018]: 5.1c scrub ok 2024-02-05T08:22:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:58 smithi196 ceph-mon[33018]: pgmap v812: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:22:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:58 smithi196 ceph-mon[33018]: 6.0 scrub starts 2024-02-05T08:22:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:58 smithi196 ceph-mon[33018]: 6.0 scrub ok 2024-02-05T08:22:59.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:58 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/907773100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:22:59.117 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:59.141 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:59.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:59.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:59.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:59.266 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:59.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:59.368 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:22:59.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:59.392 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:59.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:22:59.493 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:22:59.521 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:59.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:59.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:22:59.623 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:59.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:59.647 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:59.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:59.711 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-02-05T08:22:59.712 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-02-05T08:22:59.747 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:22:59.775 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:22:59.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:59.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:22:59.877 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:22:59.902 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:22:59.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:22:59.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:22:59.939 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:59 smithi134 ceph-mon[30086]: 1.5 scrub starts 2024-02-05T08:22:59.940 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:59 smithi134 ceph-mon[30086]: 1.5 scrub ok 2024-02-05T08:22:59.940 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:59 smithi134 ceph-mon[30086]: 4.1f scrub starts 2024-02-05T08:22:59.940 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:22:59 smithi134 ceph-mon[30086]: 4.1f scrub ok 2024-02-05T08:22:59.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:59 smithi134 ceph-mon[39278]: 1.5 scrub starts 2024-02-05T08:22:59.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:59 smithi134 ceph-mon[39278]: 1.5 scrub ok 2024-02-05T08:22:59.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:59 smithi134 ceph-mon[39278]: 4.1f scrub starts 2024-02-05T08:22:59.940 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:22:59 smithi134 ceph-mon[39278]: 4.1f scrub ok 2024-02-05T08:23:00.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:00.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:00.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:00.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:59 smithi196 ceph-mon[33018]: 1.5 scrub starts 2024-02-05T08:23:00.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:59 smithi196 ceph-mon[33018]: 1.5 scrub ok 2024-02-05T08:23:00.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:59 smithi196 ceph-mon[33018]: 4.1f scrub starts 2024-02-05T08:23:00.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:22:59 smithi196 ceph-mon[33018]: 4.1f scrub ok 2024-02-05T08:23:00.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:00.162 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:00.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:00.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:00.264 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:00.291 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:00.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:00.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:00.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:00.418 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:00.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:00.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:00.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:00.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:00.650 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:00.674 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:00.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:00.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:00.775 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:00.803 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:00.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:00.905 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:00.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:00.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:00.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:00 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:23:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:00 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:23:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:00 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:23:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:00 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:23:01.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:00 smithi134 ceph-mon[39278]: pgmap v813: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:23:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:00 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:23:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:00 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:23:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:00 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:23:01.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:00 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:23:01.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:00 smithi134 ceph-mon[30086]: pgmap v813: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:23:01.028 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:23:00 smithi134 podman[201702]: 2024-02-05 08:23:00.681239689 +0000 UTC m=+2.779218551 container create c8f1a6e0ae3d94c172a1e94b09422e96042dbf575e083b86dc90918bbb540916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T08:23:01.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:01.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:01.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:01.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:01.161 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:01.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:01.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:01.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:01.199 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:01.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:01.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:01.311 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:01.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:00 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:23:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:00 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:23:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:00 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:23:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:00 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:23:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:00 smithi196 ceph-mon[33018]: pgmap v813: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:23:01.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:01.441 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:01.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:01.453 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:23:01 smithi134 podman[201702]: 2024-02-05 08:23:01.345833404 +0000 UTC m=+3.443812266 container init c8f1a6e0ae3d94c172a1e94b09422e96042dbf575e083b86dc90918bbb540916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD) 2024-02-05T08:23:01.453 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:23:01 smithi134 podman[201702]: 2024-02-05 08:23:01.351637936 +0000 UTC m=+3.449616802 container start c8f1a6e0ae3d94c172a1e94b09422e96042dbf575e083b86dc90918bbb540916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True) 2024-02-05T08:23:01.453 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:23:01 smithi134 ceph-osd[202204]: -- 172.21.15.134:0/262479900 <== mon.0 v2:172.21.15.134:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560fd25876c0 con 0x560fd2559000 2024-02-05T08:23:01.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T08:23:01.543 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:01.571 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:01.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:01.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:01.700 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:01.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:01.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:01.705 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:23:01 smithi134 bash[201702]: c8f1a6e0ae3d94c172a1e94b09422e96042dbf575e083b86dc90918bbb540916 2024-02-05T08:23:01.705 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:23:01 smithi134 systemd[1]: Started Ceph osd.1 for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:23:01.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:23:01.802 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:01.828 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:01.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:01.929 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:01.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:01.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:02.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:01 smithi134 ceph-mon[30086]: 6.11 scrub starts 2024-02-05T08:23:02.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:01 smithi134 ceph-mon[30086]: 6.11 scrub ok 2024-02-05T08:23:02.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:01 smithi134 ceph-mon[30086]: 3.c deep-scrub starts 2024-02-05T08:23:02.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:01 smithi134 ceph-mon[30086]: 3.c deep-scrub ok 2024-02-05T08:23:02.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:01 smithi134 ceph-mon[39278]: 6.11 scrub starts 2024-02-05T08:23:02.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:01 smithi134 ceph-mon[39278]: 6.11 scrub ok 2024-02-05T08:23:02.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:01 smithi134 ceph-mon[39278]: 3.c deep-scrub starts 2024-02-05T08:23:02.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:01 smithi134 ceph-mon[39278]: 3.c deep-scrub ok 2024-02-05T08:23:02.028 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:23:01 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1[202098]: 2024-02-05T08:23:01.895+0000 7f53235227c0 -1 Falling back to public interface 2024-02-05T08:23:02.059 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:02.084 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:02.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:02.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:02.186 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:02.210 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:02.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:02.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T08:23:02.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:01 smithi196 ceph-mon[33018]: 6.11 scrub starts 2024-02-05T08:23:02.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:01 smithi196 ceph-mon[33018]: 6.11 scrub ok 2024-02-05T08:23:02.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:01 smithi196 ceph-mon[33018]: 3.c deep-scrub starts 2024-02-05T08:23:02.241 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:01 smithi196 ceph-mon[33018]: 3.c deep-scrub ok 2024-02-05T08:23:02.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:02.336 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:02.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:02.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:02.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:02.463 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:02.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:02.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:02.506 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:23:02.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:02.589 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:02.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:02.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:02.691 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:02.715 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:02.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:02.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:02.843 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:02.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:02.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:02.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:02.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:03.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:02 smithi134 ceph-mon[30086]: pgmap v814: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 921 B/s rd, 0 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:23:03.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:02 smithi134 ceph-mon[30086]: 5.1b deep-scrub starts 2024-02-05T08:23:03.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:02 smithi134 ceph-mon[30086]: 5.1b deep-scrub ok 2024-02-05T08:23:03.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:02 smithi134 ceph-mon[39278]: pgmap v814: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 921 B/s rd, 0 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:23:03.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:02 smithi134 ceph-mon[39278]: 5.1b deep-scrub starts 2024-02-05T08:23:03.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:02 smithi134 ceph-mon[39278]: 5.1b deep-scrub ok 2024-02-05T08:23:03.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:03.094 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:03.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:03.196 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:03.219 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:03.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:02 smithi196 ceph-mon[33018]: pgmap v814: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 921 B/s rd, 0 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:23:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:02 smithi196 ceph-mon[33018]: 5.1b deep-scrub starts 2024-02-05T08:23:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:02 smithi196 ceph-mon[33018]: 5.1b deep-scrub ok 2024-02-05T08:23:03.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:03.345 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:03.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:03.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:03.447 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:03.472 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:03.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:03.573 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:03.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:03.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:03.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:03.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:03.723 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:03.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:03.824 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:03.851 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:03.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:03.853 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:03.854 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:03.900 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:23:03.953 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:03.977 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:03.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:04.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:23:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:23:04.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:03 smithi196 ceph-mon[33018]: 5.3 scrub starts 2024-02-05T08:23:04.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:03 smithi196 ceph-mon[33018]: 5.3 scrub ok 2024-02-05T08:23:04.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:03 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/579138306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:23:04.063 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:03 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:23:04.079 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:04.103 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:04.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:04.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:04.178 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:03 smithi134 ceph-mon[30086]: 5.3 scrub starts 2024-02-05T08:23:04.178 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:03 smithi134 ceph-mon[30086]: 5.3 scrub ok 2024-02-05T08:23:04.178 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:03 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/579138306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:23:04.178 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:03 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:23:04.178 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:03 smithi134 ceph-mon[39278]: 5.3 scrub starts 2024-02-05T08:23:04.178 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:03 smithi134 ceph-mon[39278]: 5.3 scrub ok 2024-02-05T08:23:04.178 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:03 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/579138306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:23:04.178 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:03 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:23:04.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:04.229 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:04.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:04.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:04.323 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:23:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:23:04.323 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 0 2024-02-05T08:23:04.323 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:23:04.331 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:04.356 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:04.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:04.457 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:04.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:04.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:04.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:04.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:04.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:04.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:04.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:04.711 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:04.737 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:04.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:04.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:04.863 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:04.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:04.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:04.988 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:04.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:04.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:05.025 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:04 smithi196 ceph-mon[33018]: 4.c deep-scrub starts 2024-02-05T08:23:05.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:04 smithi196 ceph-mon[33018]: 4.c deep-scrub ok 2024-02-05T08:23:05.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:04 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:23:05.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:04 smithi196 ceph-mon[33018]: pgmap v815: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 921 B/s rd, 0 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:23:05.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:04 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:23:05.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:04 smithi196 ceph-mon[33018]: osdmap e364: 8 total, 7 up, 7 in 2024-02-05T08:23:05.026 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:04 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:05.051 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T08:23:05.090 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:05.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:05.116 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:05.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:05.212 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1[202098]: 2024-02-05T08:23:04.880+0000 7f53235227c0 -1 osd.1 323 log_to_monitors true 2024-02-05T08:23:05.212 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[39278]: 4.c deep-scrub starts 2024-02-05T08:23:05.212 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[39278]: 4.c deep-scrub ok 2024-02-05T08:23:05.212 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:23:05.212 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[39278]: pgmap v815: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 921 B/s rd, 0 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:23:05.212 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:23:05.212 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[39278]: osdmap e364: 8 total, 7 up, 7 in 2024-02-05T08:23:05.213 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:05.213 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[30086]: 4.c deep-scrub starts 2024-02-05T08:23:05.213 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[30086]: 4.c deep-scrub ok 2024-02-05T08:23:05.213 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:23:05.213 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[30086]: pgmap v815: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 921 B/s rd, 0 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:23:05.213 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:23:05.213 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[30086]: osdmap e364: 8 total, 7 up, 7 in 2024-02-05T08:23:05.213 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:04 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:05.217 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:05.242 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:05.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:05.295 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:23:05.344 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:05.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:05.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:05.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:05.470 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:05.496 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:05.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:05.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:05.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:05.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:05.623 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:05.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:05.724 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:05.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:05.749 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:05.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:05.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:05.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:05.876 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:05.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:05.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:06.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:06.002 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:06.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:06.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:05 smithi196 ceph-mon[33018]: from='osd.1 [v2:172.21.15.134:6810/1259179009,v1:172.21.15.134:6811/1259179009]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T08:23:06.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:06.127 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:06.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:06.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:06.229 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:06.252 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:06.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:06.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:06.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:05 smithi134 ceph-mon[30086]: from='osd.1 [v2:172.21.15.134:6810/1259179009,v1:172.21.15.134:6811/1259179009]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T08:23:06.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:05 smithi134 ceph-mon[39278]: from='osd.1 [v2:172.21.15.134:6810/1259179009,v1:172.21.15.134:6811/1259179009]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T08:23:06.277 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:23:05 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1[202098]: 2024-02-05T08:23:05.963+0000 7f5314981700 -1 osd.1 365 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T08:23:06.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:06.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:06.380 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:06.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:06.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:06.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:06.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:06.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:06.606 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:06.631 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:06.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:06.733 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:06.756 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:06.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:06.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:06.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:06.882 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:06.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:06.984 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:07.008 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:07.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:07.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:07.061 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:23:07.061 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:23:07.062 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T08:23:07.062 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T08:23:07.062 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:23:07.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:06 smithi196 ceph-mon[33018]: pgmap v817: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:23:07.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:06 smithi196 ceph-mon[33018]: from='osd.1 [v2:172.21.15.134:6810/1259179009,v1:172.21.15.134:6811/1259179009]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-05T08:23:07.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:06 smithi196 ceph-mon[33018]: osdmap e365: 8 total, 7 up, 7 in 2024-02-05T08:23:07.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:06 smithi196 ceph-mon[33018]: from='osd.1 [v2:172.21.15.134:6810/1259179009,v1:172.21.15.134:6811/1259179009]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:23:07.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:06 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:23:07.062 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:06 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:23:07.102 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:06 smithi134 ceph-mon[30086]: pgmap v817: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:23:07.103 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:06 smithi134 ceph-mon[30086]: from='osd.1 [v2:172.21.15.134:6810/1259179009,v1:172.21.15.134:6811/1259179009]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-05T08:23:07.103 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:06 smithi134 ceph-mon[30086]: osdmap e365: 8 total, 7 up, 7 in 2024-02-05T08:23:07.103 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:06 smithi134 ceph-mon[30086]: from='osd.1 [v2:172.21.15.134:6810/1259179009,v1:172.21.15.134:6811/1259179009]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:23:07.103 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:06 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:23:07.103 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:06 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:23:07.103 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:06 smithi134 ceph-mon[39278]: pgmap v817: 142 pgs: 77 active+clean, 22 active+undersized, 18 active+undersized+degraded, 25 active+clean+remapped; 635 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s; 77/594 objects degraded (12.963%); 30/594 objects misplaced (5.051%) 2024-02-05T08:23:07.103 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:06 smithi134 ceph-mon[39278]: from='osd.1 [v2:172.21.15.134:6810/1259179009,v1:172.21.15.134:6811/1259179009]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-05T08:23:07.103 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:06 smithi134 ceph-mon[39278]: osdmap e365: 8 total, 7 up, 7 in 2024-02-05T08:23:07.103 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:06 smithi134 ceph-mon[39278]: from='osd.1 [v2:172.21.15.134:6810/1259179009,v1:172.21.15.134:6811/1259179009]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi134", "root=default"]}]: dispatch 2024-02-05T08:23:07.103 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:06 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:23:07.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:06 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:23:07.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:07.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:07.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:07.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:07.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:07.259 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:07.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:07.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:07.361 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:07.384 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:07.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:07.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:07.510 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:07.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:07.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:07.612 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:07.635 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:07.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:07.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:07.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:07.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:07.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:07.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T08:23:07.856 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:23:07.856 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:23:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:23:07.865 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:07.875 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:23:07.889 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:07.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:07.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:07.895 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:23:07.935 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:23:07.957 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'unique_pool_4' created 2024-02-05T08:23:07.961 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:23:07.980 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:23:07.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:08.001 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:23:08.014 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:08.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:08.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:08.021 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:23:08.023 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:23:08.023 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:23:08.023 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:23:08.023 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:23:08.023 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:23:08.023 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:23:08.023 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:23:08.023 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:23:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:07 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:23:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:07 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:23:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:07 smithi196 ceph-mon[33018]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-05T08:23:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:07 smithi196 ceph-mon[33018]: osd.1 [v2:172.21.15.134:6810/1259179009,v1:172.21.15.134:6811/1259179009] boot 2024-02-05T08:23:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:07 smithi196 ceph-mon[33018]: osdmap e366: 8 total, 8 up, 7 in 2024-02-05T08:23:08.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:07 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:23:08.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:07 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1957804836' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-05T08:23:08.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:07 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1542721699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:08.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:07 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1957804836' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-05T08:23:08.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:07 smithi196 ceph-mon[33018]: osdmap e367: 8 total, 8 up, 7 in 2024-02-05T08:23:08.101 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:23:08.116 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:08.140 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:08.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:08.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:08.242 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:08.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:08.268 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:08.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:08.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:23:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:23:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[30086]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-05T08:23:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[30086]: osd.1 [v2:172.21.15.134:6810/1259179009,v1:172.21.15.134:6811/1259179009] boot 2024-02-05T08:23:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[30086]: osdmap e366: 8 total, 8 up, 7 in 2024-02-05T08:23:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:23:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1957804836' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-05T08:23:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1542721699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1957804836' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-05T08:23:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[30086]: osdmap e367: 8 total, 8 up, 7 in 2024-02-05T08:23:08.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:23:08.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:23:08.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[39278]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-02-05T08:23:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[39278]: osd.1 [v2:172.21.15.134:6810/1259179009,v1:172.21.15.134:6811/1259179009] boot 2024-02-05T08:23:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[39278]: osdmap e366: 8 total, 8 up, 7 in 2024-02-05T08:23:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T08:23:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1957804836' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-05T08:23:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1542721699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1957804836' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-05T08:23:08.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:07 smithi134 ceph-mon[39278]: osdmap e367: 8 total, 8 up, 7 in 2024-02-05T08:23:08.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:08.394 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:08.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:08.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:08.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:08.520 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:08.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:08.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:08.622 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:08.647 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:08.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:08.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:08.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:08.774 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:08.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:08.875 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:08.901 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:08.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:08.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:08.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:23:09.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:09.027 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:09.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:09.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:09.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:09.147 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:23:09.147 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:23:09.147 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[30086]: pgmap v820: 142 pgs: 11 peering, 6 remapped+peering, 77 active+clean, 17 active+undersized, 12 active+undersized+degraded, 19 active+clean+remapped; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 48/594 objects degraded (8.081%); 26/594 objects misplaced (4.377%) 2024-02-05T08:23:09.147 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[30086]: Health check update: Degraded data redundancy: 48/594 objects degraded (8.081%), 12 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-02-05T08:23:09.148 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:09.148 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:23:09.148 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[30086]: osdmap e368: 8 total, 8 up, 7 in 2024-02-05T08:23:09.148 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:23:09.148 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:23:09.148 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[39278]: pgmap v820: 142 pgs: 11 peering, 6 remapped+peering, 77 active+clean, 17 active+undersized, 12 active+undersized+degraded, 19 active+clean+remapped; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 48/594 objects degraded (8.081%); 26/594 objects misplaced (4.377%) 2024-02-05T08:23:09.148 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[39278]: Health check update: Degraded data redundancy: 48/594 objects degraded (8.081%), 12 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-02-05T08:23:09.148 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:09.148 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:23:09.148 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:08 smithi134 ceph-mon[39278]: osdmap e368: 8 total, 8 up, 7 in 2024-02-05T08:23:09.154 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:09.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:09.257 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:09.281 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:09.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:09.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:08 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:23:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:08 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:23:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:08 smithi196 ceph-mon[33018]: pgmap v820: 142 pgs: 11 peering, 6 remapped+peering, 77 active+clean, 17 active+undersized, 12 active+undersized+degraded, 19 active+clean+remapped; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 48/594 objects degraded (8.081%); 26/594 objects misplaced (4.377%) 2024-02-05T08:23:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:08 smithi196 ceph-mon[33018]: Health check update: Degraded data redundancy: 48/594 objects degraded (8.081%), 12 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-02-05T08:23:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:23:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:08 smithi196 ceph-mon[33018]: osdmap e368: 8 total, 8 up, 7 in 2024-02-05T08:23:09.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:09.408 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:09.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:09.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:09.509 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:09.533 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:09.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:09.635 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:09.661 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:09.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:09.762 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:09.785 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:09.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:09.887 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:09.897 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:23:09.897 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:23:09.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:09.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:10.001 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-02-05T08:23:10.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:10.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:10.039 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:10.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:10.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:10.164 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:10.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:10.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:10.235 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:23:10.236 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:23:10.255 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:10.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:10.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:10.290 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:10.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:10.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:10.414 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:10.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:10.516 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:10.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:10.543 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:10.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:10.557 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T08:23:10.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:10.668 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:10.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:10.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:10.794 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:10.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:10.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:10.836 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:23:10.897 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:10.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:10.923 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:10.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:11.024 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:11.048 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:11.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:11.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:11.175 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:11.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:11.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:11.182 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:10 smithi134 ceph-mon[30086]: osdmap e369: 8 total, 8 up, 7 in 2024-02-05T08:23:11.182 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:10 smithi134 ceph-mon[30086]: pgmap v824: 158 pgs: 13 creating+peering, 3 unknown, 27 peering, 7 remapped+peering, 94 active+clean, 7 active+undersized, 5 active+undersized+degraded, 2 active+clean+remapped; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 768 B/s rd, 0 op/s; 24/594 objects degraded (4.040%); 1/594 objects misplaced (0.168%) 2024-02-05T08:23:11.182 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:10 smithi134 ceph-mon[39278]: osdmap e369: 8 total, 8 up, 7 in 2024-02-05T08:23:11.182 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:10 smithi134 ceph-mon[39278]: pgmap v824: 158 pgs: 13 creating+peering, 3 unknown, 27 peering, 7 remapped+peering, 94 active+clean, 7 active+undersized, 5 active+undersized+degraded, 2 active+clean+remapped; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 768 B/s rd, 0 op/s; 24/594 objects degraded (4.040%); 1/594 objects misplaced (0.168%) 2024-02-05T08:23:11.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:10 smithi196 ceph-mon[33018]: osdmap e369: 8 total, 8 up, 7 in 2024-02-05T08:23:11.191 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:10 smithi196 ceph-mon[33018]: pgmap v824: 158 pgs: 13 creating+peering, 3 unknown, 27 peering, 7 remapped+peering, 94 active+clean, 7 active+undersized, 5 active+undersized+degraded, 2 active+clean+remapped; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 768 B/s rd, 0 op/s; 24/594 objects degraded (4.040%); 1/594 objects misplaced (0.168%) 2024-02-05T08:23:11.191 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:23:11.277 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:11.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:11.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:11.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:11.403 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:11.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:11.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:11.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:11.552 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:11.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:11.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:11.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:11.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:11.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:11.780 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:11.805 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:11.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:11.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:11.907 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:11.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:11.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:11.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:11 smithi196 ceph-mon[33018]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:23:11.962 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:11 smithi196 ceph-mon[33018]: osdmap e370: 8 total, 8 up, 7 in 2024-02-05T08:23:12.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:12.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:12.058 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:12.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:12.159 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:12.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:12.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:12.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:11 smithi134 ceph-mon[30086]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:23:12.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:11 smithi134 ceph-mon[30086]: osdmap e370: 8 total, 8 up, 7 in 2024-02-05T08:23:12.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:11 smithi134 ceph-mon[39278]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T08:23:12.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:11 smithi134 ceph-mon[39278]: osdmap e370: 8 total, 8 up, 7 in 2024-02-05T08:23:12.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:12.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:12.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:12.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:12.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:12.438 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:12.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:12.482 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:12.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:12.564 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:12.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:12.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:12.600 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:23:12.601 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:23:12.666 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:12.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:12.691 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:12.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:12.763 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:12.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:12.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:12.818 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:12.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:12.919 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:12.943 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:12.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:12.987 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:12 smithi134 ceph-mon[30086]: pgmap v826: 158 pgs: 13 creating+peering, 1 unknown, 27 peering, 3 remapped+peering, 114 active+clean; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T08:23:12.988 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:12 smithi134 ceph-mon[39278]: pgmap v826: 158 pgs: 13 creating+peering, 1 unknown, 27 peering, 3 remapped+peering, 114 active+clean; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T08:23:13.044 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:13.070 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:13.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:13.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:13.196 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:13.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:13.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:13.277 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:12 smithi196 ceph-mon[33018]: pgmap v826: 158 pgs: 13 creating+peering, 1 unknown, 27 peering, 3 remapped+peering, 114 active+clean; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T08:23:13.298 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:13.305 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T08:23:13.323 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:13.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:13.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:13.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:13.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:13.449 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:13.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:13.544 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:23:13.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:13.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:13.574 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:13.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:13.675 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:13.699 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:13.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:13.800 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:13.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:13.826 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:13.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:13.896 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:23:13.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:13 smithi196 ceph-mon[33018]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/594 objects degraded (4.040%), 5 pgs degraded, 12 pgs undersized) 2024-02-05T08:23:13.923 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:13 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1950904996' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:23:13.927 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:13.951 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:13.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:13.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:14.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:13 smithi134 ceph-mon[30086]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/594 objects degraded (4.040%), 5 pgs degraded, 12 pgs undersized) 2024-02-05T08:23:14.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:13 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1950904996' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:23:14.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:13 smithi134 ceph-mon[39278]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 24/594 objects degraded (4.040%), 5 pgs degraded, 12 pgs undersized) 2024-02-05T08:23:14.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:13 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1950904996' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:23:14.053 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:14.077 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:14.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:14.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:14.179 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:14.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:14.204 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:14.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:14.305 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:14.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:23:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:23:14.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:14.329 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:14.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:14.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:14.454 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:14.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:14.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:14.556 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:14.582 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:14.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:14.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:14.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:14.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:14.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:14.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:14.833 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:14.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:14.934 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:14.960 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:14.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:15.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:14 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:23:15.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:14 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1950904996' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:23:15.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:14 smithi196 ceph-mon[33018]: pgmap v827: 158 pgs: 158 active+clean; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 3 objects/s recovering 2024-02-05T08:23:15.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:14 smithi196 ceph-mon[33018]: osdmap e371: 8 total, 8 up, 7 in 2024-02-05T08:23:15.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:14 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:15.005 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:14 smithi196 ceph-mon[33018]: osdmap e372: 8 total, 8 up, 7 in 2024-02-05T08:23:15.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:14 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:23:15.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:14 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1950904996' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:23:15.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:14 smithi134 ceph-mon[30086]: pgmap v827: 158 pgs: 158 active+clean; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 3 objects/s recovering 2024-02-05T08:23:15.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:14 smithi134 ceph-mon[30086]: osdmap e371: 8 total, 8 up, 7 in 2024-02-05T08:23:15.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:14 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:15.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:14 smithi134 ceph-mon[30086]: osdmap e372: 8 total, 8 up, 7 in 2024-02-05T08:23:15.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:14 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:23:15.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:14 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1950904996' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:23:15.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:14 smithi134 ceph-mon[39278]: pgmap v827: 158 pgs: 158 active+clean; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 3 objects/s recovering 2024-02-05T08:23:15.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:14 smithi134 ceph-mon[39278]: osdmap e371: 8 total, 8 up, 7 in 2024-02-05T08:23:15.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:14 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:15.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:14 smithi134 ceph-mon[39278]: osdmap e372: 8 total, 8 up, 7 in 2024-02-05T08:23:15.061 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:15.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:15.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:15.188 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:15.213 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:15.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:15.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:15.259 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:23:15.259 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:23:15.314 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:15.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:15.339 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:15.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:15.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:15.467 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:15.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:15.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:15.569 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:15.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:15.594 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:15.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:15.695 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:15.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:15.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:15.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:15.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:15.846 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:15.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:15.948 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:15.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:15.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:15.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:16.074 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:16.098 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:16.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:16.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T08:23:16.201 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:16.207 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 7, 6, 3] out_osds: [0] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-02-05T08:23:16.208 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:23:16.208 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-02-05T08:23:16.208 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-05T08:23:16.208 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-05T08:23:16.208 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-02-05T08:23:16.224 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:16.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:16.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:16.326 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:16.348 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:23:16.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:16.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:16.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:16.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:16.478 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:16.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:16.579 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:16.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:16.603 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:16.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:16.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:16.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:16.729 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:16.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:16.830 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:16.855 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:16.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:16.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:16.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:16 smithi134 ceph-mon[30086]: pgmap v830: 158 pgs: 158 active+clean; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 0 B/s, 3 objects/s recovering 2024-02-05T08:23:16.930 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:16 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/975147770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:16.930 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:16 smithi134 ceph-mon[39278]: pgmap v830: 158 pgs: 158 active+clean; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 0 B/s, 3 objects/s recovering 2024-02-05T08:23:16.930 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:16 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/975147770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:16.957 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:16.982 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:16.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:16.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:17.013 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:23:17.013 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:23:17.014 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:23:17.014 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:23:17.014 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:23:17.014 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:23:17.014 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:23:17.014 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:23:17.014 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:23:17.014 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:23:17.014 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:23:17.014 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:23:17.015 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:23:17.015 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:23:17.015 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:23:17.015 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:23:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:16 smithi196 ceph-mon[33018]: pgmap v830: 158 pgs: 158 active+clean; 635 KiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 0 B/s, 3 objects/s recovering 2024-02-05T08:23:17.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:16 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/975147770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:17.085 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:17.107 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:23:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:17.109 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:17.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:17.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:17.236 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:17.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:17.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:17.338 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:17.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:17.362 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:17.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:17.463 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:17.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:17.492 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:17.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:17.593 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:17.618 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:17.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:17.719 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:17.743 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:17.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:17.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:17.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:17.868 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:17.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:17.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:17.900 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:23:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:23:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:23:17.914 INFO:teuthology.orchestra.run.smithi134.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-02-05T08:23:17.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:17.996 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:17 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/96078045' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:23:17.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:17.997 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:17.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:18.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:18.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:18.123 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:18.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:18.224 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:18.238 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:17 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/96078045' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:23:18.238 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:17 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/96078045' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T08:23:18.239 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:23:18.239 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:23:18.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:23:18.239 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:23:18.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.239 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15087.0:170401 8.8 8:15576685:::benchmark_data_smithi134_115836_object53741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-02-05T08:23:18.240 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:13:45.792841+0000", 2024-02-05T08:23:18.240 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 572.44605235200004, 2024-02-05T08:23:18.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94320472300000002, 2024-02-05T08:23:18.240 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:18.240 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:18.240 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:18.240 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15087", 2024-02-05T08:23:18.240 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/781917829", 2024-02-05T08:23:18.240 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 170401 2024-02-05T08:23:18.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.240 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:18.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:18.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792841+0000", 2024-02-05T08:23:18.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:18.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:18.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792841+0000", 2024-02-05T08:23:18.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:18.241 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.241 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792842+0000", 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2109999999999999e-06 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792846+0000", 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5260000000000002e-06 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792857+0000", 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.135e-05 2024-02-05T08:23:18.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792865+0000", 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8930000000000005e-06 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792926+0000", 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0949000000000002e-05 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:45.792935+0000", 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7050000000000005e-06 2024-02-05T08:23:18.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.733064+0000", 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.94012932299999996 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.733194+0000", 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012970500000000001 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,7", 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.733276+0000", 2024-02-05T08:23:18.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2328000000000002e-05 2024-02-05T08:23:18.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:18.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.735903+0000", 2024-02-05T08:23:18.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0026266029999999999 2024-02-05T08:23:18.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:18.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.735915+0000", 2024-02-05T08:23:18.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1817999999999999e-05 2024-02-05T08:23:18.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.735956+0000", 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1635999999999999e-05 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.736003+0000", 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7089000000000003e-05 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:13:46.736046+0000", 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2586999999999997e-05 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:18.246 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15714.0:225863 10.6 10:6c4cccde:::benchmark_data_smithi134_174861_object94401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:22:08.078103+0000", 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 70.160790810999998, 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1085411509999998, 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15714", 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225863 2024-02-05T08:23:18.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.248 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:18.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:18.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078103+0000", 2024-02-05T08:23:18.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:18.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:18.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078103+0000", 2024-02-05T08:23:18.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:18.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078105+0000", 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1670000000000002e-06 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078110+0000", 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6959999999999996e-06 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078127+0000", 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7450000000000001e-05 2024-02-05T08:23:18.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:18.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078137+0000", 2024-02-05T08:23:18.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8689999999999999e-06 2024-02-05T08:23:18.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078203+0000", 2024-02-05T08:23:18.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6507999999999994e-05 2024-02-05T08:23:18.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:18.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078348+0000", 2024-02-05T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014468599999999999 2024-02-05T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-02-05T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078468+0000", 2024-02-05T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119889 2024-02-05T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.115112+0000", 2024-02-05T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.036644058 2024-02-05T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.115125+0000", 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3023000000000001e-05 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.087270+0000", 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97214539099999997 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.087284+0000", 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3688e-05 2024-02-05T08:23:18.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.186549+0000", 2024-02-05T08:23:18.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.099264815 2024-02-05T08:23:18.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:18.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.186644+0000", 2024-02-05T08:23:18.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4911e-05 2024-02-05T08:23:18.254 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:18.254 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:18.254 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:18.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.255 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15714.0:225864 10.6 10:6e872cbe:::benchmark_data_smithi134_174861_object94402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-02-05T08:23:18.256 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:22:08.078902+0000", 2024-02-05T08:23:18.256 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 70.159991141000006, 2024-02-05T08:23:18.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1287934449999999, 2024-02-05T08:23:18.256 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:18.256 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:23:18.256 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:18.256 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15714", 2024-02-05T08:23:18.256 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:23:18.256 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225864 2024-02-05T08:23:18.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.256 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:18.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078902+0000", 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078902+0000", 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078903+0000", 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2600000000000001e-07 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078907+0000", 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9230000000000004e-06 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078921+0000", 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3709999999999999e-05 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078930+0000", 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6810000000000004e-06 2024-02-05T08:23:18.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.091414+0000", 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012483935999999999 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.091529+0000", 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115661 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.091648+0000", 2024-02-05T08:23:18.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011900600000000001 2024-02-05T08:23:18.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.115176+0000", 2024-02-05T08:23:18.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.023528263000000001 2024-02-05T08:23:18.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:23:18.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.115199+0000", 2024-02-05T08:23:18.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2356000000000001e-05 2024-02-05T08:23:18.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.089214+0000", 2024-02-05T08:23:18.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97401506500000001 2024-02-05T08:23:18.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:23:18.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.089225+0000", 2024-02-05T08:23:18.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1571e-05 2024-02-05T08:23:18.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.186954+0000", 2024-02-05T08:23:18.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.097729011 2024-02-05T08:23:18.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for missing object", 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.187121+0000", 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000166479 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.207626+0000", 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020504887999999999 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.207696+0000", 2024-02-05T08:23:18.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9968999999999994e-05 2024-02-05T08:23:18.263 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:18.263 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:18.263 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:18.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.263 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15714.0:225869 10.6 10:6176b0ec:::benchmark_data_smithi134_174861_object94407:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-02-05T08:23:18.263 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:22:08.079530+0000", 2024-02-05T08:23:18.263 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 70.159363137, 2024-02-05T08:23:18.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1276412310000001, 2024-02-05T08:23:18.263 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:18.263 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:18.263 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:18.263 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15714", 2024-02-05T08:23:18.264 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:23:18.264 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225869 2024-02-05T08:23:18.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.264 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:18.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:18.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.079530+0000", 2024-02-05T08:23:18.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:18.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:18.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.079530+0000", 2024-02-05T08:23:18.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.079531+0000", 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3099999999999998e-07 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.079534+0000", 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9299999999999999e-06 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:18.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.079542+0000", 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8779999999999998e-06 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.079547+0000", 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7629999999999997e-06 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.115251+0000", 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035703603 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:23:18.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.115256+0000", 2024-02-05T08:23:18.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6500000000000004e-06 2024-02-05T08:23:18.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.089356+0000", 2024-02-05T08:23:18.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97410054499999998 2024-02-05T08:23:18.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:23:18.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.089365+0000", 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0329999999999997e-06 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.187236+0000", 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.09787101 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.187360+0000", 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012386 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.268 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.187864+0000", 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050348000000000005 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.195604+0000", 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077400400000000001 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.200859+0000", 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052555379999999997 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:18.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.206992+0000", 2024-02-05T08:23:18.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0061325399999999997 2024-02-05T08:23:18.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:18.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.207062+0000", 2024-02-05T08:23:18.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0358000000000006e-05 2024-02-05T08:23:18.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:18.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.207172+0000", 2024-02-05T08:23:18.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109472 2024-02-05T08:23:18.272 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:18.272 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:18.272 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:18.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.272 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24344.0:3579 5.15 5:a93a5511:::notify.2:head [watch ping cookie 94292702905344 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e364)", 2024-02-05T08:23:18.273 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:23:06.804449+0000", 2024-02-05T08:23:18.273 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 11.434444121, 2024-02-05T08:23:18.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00045584, 2024-02-05T08:23:18.273 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:18.273 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T08:23:18.273 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:18.273 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24344", 2024-02-05T08:23:18.274 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1272526039", 2024-02-05T08:23:18.274 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 3579 2024-02-05T08:23:18.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.274 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:18.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.274 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:18.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:06.804449+0000", 2024-02-05T08:23:18.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:18.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:18.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:06.804449+0000", 2024-02-05T08:23:18.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:18.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:18.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:06.804451+0000", 2024-02-05T08:23:18.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5960000000000001e-06 2024-02-05T08:23:18.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:18.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:06.804461+0000", 2024-02-05T08:23:18.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5079999999999994e-06 2024-02-05T08:23:18.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:18.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:06.804476+0000", 2024-02-05T08:23:18.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5619000000000001e-05 2024-02-05T08:23:18.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:18.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:06.804487+0000", 2024-02-05T08:23:18.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1205e-05 2024-02-05T08:23:18.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:18.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:06.804620+0000", 2024-02-05T08:23:18.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000132187 2024-02-05T08:23:18.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.280 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:18.280 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:06.804712+0000", 2024-02-05T08:23:18.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2937999999999997e-05 2024-02-05T08:23:18.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:18.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:18.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:18.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:06.804905+0000", 2024-02-05T08:23:18.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019278700000000001 2024-02-05T08:23:18.281 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:18.281 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:18.282 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:18.282 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:18.282 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:18.282 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:23:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:18.283 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:18.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:18.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:18.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:18.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:18.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:18.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:18.535 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:18.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:18.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:18.587 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:18.623 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:18.637 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:18.661 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:18.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:18.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:18.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:18.787 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:18.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:18.888 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:18.913 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:18.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:18.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:19.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:19.039 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:19.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:19.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:18 smithi196 ceph-mon[33018]: pgmap v831: 158 pgs: 158 active+clean; 635 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 584 B/s rd, 0 op/s; 0 B/s, 2 objects/s recovering 2024-02-05T08:23:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:18 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/96078045' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:23:19.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:18 smithi196 ceph-mon[33018]: osdmap e373: 8 total, 8 up, 7 in 2024-02-05T08:23:19.115 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:18 smithi134 ceph-mon[30086]: pgmap v831: 158 pgs: 158 active+clean; 635 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 584 B/s rd, 0 op/s; 0 B/s, 2 objects/s recovering 2024-02-05T08:23:19.115 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:18 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/96078045' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:23:19.115 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:18 smithi134 ceph-mon[30086]: osdmap e373: 8 total, 8 up, 7 in 2024-02-05T08:23:19.116 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:18 smithi134 ceph-mon[39278]: pgmap v831: 158 pgs: 158 active+clean; 635 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 584 B/s rd, 0 op/s; 0 B/s, 2 objects/s recovering 2024-02-05T08:23:19.116 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:18 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/96078045' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T08:23:19.116 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:18 smithi134 ceph-mon[39278]: osdmap e373: 8 total, 8 up, 7 in 2024-02-05T08:23:19.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T08:23:19.140 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:19.163 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:19.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:19.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:19.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:19.289 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:19.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:19.391 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:19.411 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:23:19.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:19.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:19.516 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:19.546 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:19.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:19.566 DEBUG:teuthology.orchestra.run.smithi134:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-02-05T08:23:19.568 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T08:23:19.568 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T08:23:19.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:19.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:19.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:19.727 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:hints = 1 2024-02-05T08:23:19.727 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T08:23:19.727 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_205380 2024-02-05T08:23:19.727 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:23:19.727 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-02-05T08:23:19.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:19.810 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:19.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:19.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:19.948 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:19.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:19.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:19.971 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:23:20.051 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:20.101 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:20.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:20.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:20.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:20.236 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:20.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:20.338 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:20.378 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:20.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:20.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:20.516 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:20.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:20.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:20.618 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:20.667 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:20.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:20.728 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 1 16 3518 3502 218.624 218.875 0.00408346 0.00455759 2024-02-05T08:23:20.768 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:20.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:20.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:20.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:20.914 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:20.956 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:20.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:21.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:20 smithi196 ceph-mon[33018]: osdmap e374: 8 total, 8 up, 7 in 2024-02-05T08:23:21.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:20 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:23:21.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:20 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:23:21.012 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:20 smithi196 ceph-mon[33018]: pgmap v834: 158 pgs: 158 active+clean; 635 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:23:21.058 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:21.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:21.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:21.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:20 smithi134 ceph-mon[30086]: osdmap e374: 8 total, 8 up, 7 in 2024-02-05T08:23:21.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:20 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:23:21.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:20 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:23:21.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:20 smithi134 ceph-mon[30086]: pgmap v834: 158 pgs: 158 active+clean; 635 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:23:21.162 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:20 smithi134 ceph-mon[39278]: osdmap e374: 8 total, 8 up, 7 in 2024-02-05T08:23:21.162 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:20 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:23:21.162 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:20 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:23:21.162 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:20 smithi134 ceph-mon[39278]: pgmap v834: 158 pgs: 158 active+clean; 635 KiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:23:21.198 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:21.210 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:21.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:21.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:21.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:21.333 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:21.373 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:21.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:21.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:21.519 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:21.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:21.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:21.620 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:21.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:21.663 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:21.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:21.699 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:23:21.699 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:23:21.699 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.700 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:6169 11.3 11:c8d6f4ac:::benchmark_data_smithi134_205380_object6168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T08:23:21.700 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:23:21.525553+0000", 2024-02-05T08:23:21.700 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.0045391670000000002, 2024-02-05T08:23:21.700 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045751979999999999, 2024-02-05T08:23:21.700 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:21.700 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:23:21.700 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:21.700 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:23:21.700 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:23:21.700 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6169 2024-02-05T08:23:21.700 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.700 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:21.700 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.525553+0000", 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.525553+0000", 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.525554+0000", 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2559999999999999e-06 2024-02-05T08:23:21.701 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:21.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.525582+0000", 2024-02-05T08:23:21.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8411999999999998e-05 2024-02-05T08:23:21.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:21.702 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.525602+0000", 2024-02-05T08:23:21.702 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9471999999999999e-05 2024-02-05T08:23:21.702 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.702 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.702 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.525609+0000", 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4359999999999996e-06 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.525684+0000", 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4639000000000001e-05 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.525803+0000", 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119669 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.703 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,7", 2024-02-05T08:23:21.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.526000+0000", 2024-02-05T08:23:21.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019629900000000001 2024-02-05T08:23:21.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.704 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.704 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:21.704 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.526934+0000", 2024-02-05T08:23:21.704 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000933887 2024-02-05T08:23:21.704 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.705 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.705 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:21.705 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.528970+0000", 2024-02-05T08:23:21.705 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020366030000000001 2024-02-05T08:23:21.705 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:21.705 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:21.705 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:21.706 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.706 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.706 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:6178 11.3 11:c9467338:::benchmark_data_smithi134_205380_object6177:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T08:23:21.706 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:23:21.528722+0000", 2024-02-05T08:23:21.706 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.001369749, 2024-02-05T08:23:21.706 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0015007950000000001, 2024-02-05T08:23:21.706 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:21.706 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:23:21.706 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6178 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.528722+0000", 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.528722+0000", 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:21.707 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:21.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.528724+0000", 2024-02-05T08:23:21.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5549999999999999e-06 2024-02-05T08:23:21.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:21.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.528757+0000", 2024-02-05T08:23:21.708 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3059999999999999e-05 2024-02-05T08:23:21.708 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.708 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.708 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:21.708 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.528767+0000", 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0925e-05 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.528775+0000", 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8390000000000007e-06 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529002+0000", 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022673600000000001 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.709 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529134+0000", 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013151300000000001 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,7", 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529243+0000", 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109802 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:6180 11.1 11:8e7b3d32:::benchmark_data_smithi134_205380_object6179:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:23:21.529269+0000", 2024-02-05T08:23:21.710 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.00082225899999999999, 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010379390000000001, 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6180 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529269+0000", 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.711 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:21.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529269+0000", 2024-02-05T08:23:21.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:21.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:21.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529270+0000", 2024-02-05T08:23:21.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0599999999999999e-07 2024-02-05T08:23:21.712 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.712 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.712 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:21.712 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529559+0000", 2024-02-05T08:23:21.712 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00028849300000000002 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529570+0000", 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1661999999999999e-05 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529578+0000", 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9270000000000005e-06 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529711+0000", 2024-02-05T08:23:21.713 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013264400000000001 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.530265+0000", 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00055385600000000003 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop_reply(client.25820.0:6169 11.3 e375/367)", 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:23:21.529570+0000", 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.00052218499999999999, 2024-02-05T08:23:21.714 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000817963, 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "queued for pg", 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529570+0000", 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529570+0000", 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.715 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529570+0000", 2024-02-05T08:23:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7700000000000005e-07 2024-02-05T08:23:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529574+0000", 2024-02-05T08:23:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0010000000000003e-06 2024-02-05T08:23:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529583+0000", 2024-02-05T08:23:21.716 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7050000000000005e-06 2024-02-05T08:23:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529589+0000", 2024-02-05T08:23:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.404e-06 2024-02-05T08:23:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:6181 11.1 11.19ae8ac1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T08:23:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:23:21.529599+0000", 2024-02-05T08:23:21.717 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.00049285900000000001, 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00084009700000000004, 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "queued for pg", 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 6181 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529599+0000", 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.718 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529599+0000", 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529600+0000", 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3e-07 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529626+0000", 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6483e-05 2024-02-05T08:23:21.719 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.529634+0000", 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5079999999999999e-06 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:21.530423+0000", 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00078972500000000002 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:21.720 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-02-05T08:23:21.721 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 5 2024-02-05T08:23:21.721 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:23:21.729 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 2 16 6701 6685 208.669 198.938 0.00401242 0.00478137 2024-02-05T08:23:21.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:21.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:21.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:21.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:21.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:21.950 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:21.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:22.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:21 smithi134 ceph-mon[30086]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:23:22.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:21 smithi134 ceph-mon[30086]: osdmap e375: 8 total, 8 up, 7 in 2024-02-05T08:23:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:21 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:23:22.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:21 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:23:22.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:21 smithi134 ceph-mon[39278]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:23:22.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:21 smithi134 ceph-mon[39278]: osdmap e375: 8 total, 8 up, 7 in 2024-02-05T08:23:22.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:21 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:23:22.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:21 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:23:22.053 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:21 smithi196 ceph-mon[33018]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T08:23:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:21 smithi196 ceph-mon[33018]: osdmap e375: 8 total, 8 up, 7 in 2024-02-05T08:23:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:21 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:23:22.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:21 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:23:22.093 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:22.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:22.195 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:22.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:22.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:22.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:22.334 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:22.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:22.376 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:22.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:22.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:22.520 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:22.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:22.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T08:23:22.623 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:22.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:22.648 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:22.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:22.729 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 3 15 9401 9386 195.388 168.812 0.00680712 0.00510911 2024-02-05T08:23:22.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:22.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:22.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:22.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:22.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:22 smithi196 ceph-mon[33018]: pgmap v836: 158 pgs: 158 active+clean; 32 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 5.2 MiB/s wr, 83 op/s 2024-02-05T08:23:22.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:22 smithi196 ceph-mon[33018]: osdmap e376: 8 total, 8 up, 7 in 2024-02-05T08:23:22.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:22 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:23:22.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:22 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:23:22.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:22 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/484966886' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:23:22.909 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:22 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:23:22.931 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:23:22.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:22.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:23.034 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:23:23.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:23.079 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:23.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:23.181 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:23.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:23.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:23.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:23.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:22 smithi134 ceph-mon[30086]: pgmap v836: 158 pgs: 158 active+clean; 32 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 5.2 MiB/s wr, 83 op/s 2024-02-05T08:23:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:22 smithi134 ceph-mon[30086]: osdmap e376: 8 total, 8 up, 7 in 2024-02-05T08:23:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:22 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:23:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:22 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:23:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:22 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/484966886' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:23:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:22 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:23:23.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:22 smithi134 ceph-mon[39278]: pgmap v836: 158 pgs: 158 active+clean; 32 MiB data, 4.0 GiB used, 622 GiB / 626 GiB avail; 853 B/s rd, 5.2 MiB/s wr, 83 op/s 2024-02-05T08:23:23.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:22 smithi134 ceph-mon[39278]: osdmap e376: 8 total, 8 up, 7 in 2024-02-05T08:23:23.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:22 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:23:23.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:22 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:23:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:22 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/484966886' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:23:23.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:22 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:23:23.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:23.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:23.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:23.388 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:23:23.388 INFO:teuthology.orchestra.run.smithi134.stdout: "success": "" 2024-02-05T08:23:23.388 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:23:23.469 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:23.509 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:23.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:23.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:23.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:23.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:23.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:23.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:23.729 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 4 15 12433 12418 193.913 189.5 0.00313403 0.00515091 2024-02-05T08:23:23.751 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:23.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:23.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:23.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:23.892 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:23.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:23.936 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:23.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:23:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:23:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[30086]: osdmap e377: 8 total, 8 up, 7 in 2024-02-05T08:23:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[30086]: 3.0 scrub starts 2024-02-05T08:23:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[30086]: 3.0 scrub ok 2024-02-05T08:23:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[30086]: 4.15 scrub starts 2024-02-05T08:23:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[30086]: 4.15 scrub ok 2024-02-05T08:23:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:24.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:23:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:23:24.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:23:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[39278]: osdmap e377: 8 total, 8 up, 7 in 2024-02-05T08:23:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[39278]: 3.0 scrub starts 2024-02-05T08:23:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[39278]: 3.0 scrub ok 2024-02-05T08:23:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[39278]: 4.15 scrub starts 2024-02-05T08:23:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[39278]: 4.15 scrub ok 2024-02-05T08:23:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:24.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:23:24.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:24.072 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:24.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:24.173 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:24.218 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:24.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:24.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:23 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:23:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:23 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:23:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:23 smithi196 ceph-mon[33018]: osdmap e377: 8 total, 8 up, 7 in 2024-02-05T08:23:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:23 smithi196 ceph-mon[33018]: 3.0 scrub starts 2024-02-05T08:23:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:23 smithi196 ceph-mon[33018]: 3.0 scrub ok 2024-02-05T08:23:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:23 smithi196 ceph-mon[33018]: 4.15 scrub starts 2024-02-05T08:23:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:23 smithi196 ceph-mon[33018]: 4.15 scrub ok 2024-02-05T08:23:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:23:24.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:23:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:23:24.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:24.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:24.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:24.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:24.492 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:24.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:24.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:24.594 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:24.632 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:24.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:24.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:24.729 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 5 16 15130 15114 188.83 168.5 0.00316972 0.00528798 2024-02-05T08:23:24.735 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:24.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:24.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:24.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:24.913 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:24.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:24.971 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:24 smithi196 ceph-mon[33018]: pgmap v839: 158 pgs: 158 active+clean; 640 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-02-05T08:23:24.972 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:24 smithi196 ceph-mon[33018]: 6.1b scrub starts 2024-02-05T08:23:24.972 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:24 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:24.972 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:24 smithi196 ceph-mon[33018]: 2.0 scrub starts 2024-02-05T08:23:24.972 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:24 smithi196 ceph-mon[33018]: 2.0 scrub ok 2024-02-05T08:23:24.972 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:24 smithi196 ceph-mon[33018]: 6.1b scrub ok 2024-02-05T08:23:25.014 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:25.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:25.056 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:25.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:25.084 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:23:25.085 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:23:25.085 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T08:23:25.085 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T08:23:25.085 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:23:25.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:25.199 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:25.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:25.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:25.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:24 smithi134 ceph-mon[30086]: pgmap v839: 158 pgs: 158 active+clean; 640 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-02-05T08:23:25.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:24 smithi134 ceph-mon[30086]: 6.1b scrub starts 2024-02-05T08:23:25.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:24 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:25.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:24 smithi134 ceph-mon[30086]: 2.0 scrub starts 2024-02-05T08:23:25.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:24 smithi134 ceph-mon[30086]: 2.0 scrub ok 2024-02-05T08:23:25.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:24 smithi134 ceph-mon[30086]: 6.1b scrub ok 2024-02-05T08:23:25.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:24 smithi134 ceph-mon[39278]: pgmap v839: 158 pgs: 158 active+clean; 640 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-02-05T08:23:25.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:24 smithi134 ceph-mon[39278]: 6.1b scrub starts 2024-02-05T08:23:25.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:24 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:25.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:24 smithi134 ceph-mon[39278]: 2.0 scrub starts 2024-02-05T08:23:25.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:24 smithi134 ceph-mon[39278]: 2.0 scrub ok 2024-02-05T08:23:25.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:24 smithi134 ceph-mon[39278]: 6.1b scrub ok 2024-02-05T08:23:25.301 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:25.337 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:25.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:25.439 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:25.477 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:25.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:25.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:25.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:25.627 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:25.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:25.728 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:25.729 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 6 15 17899 17884 186.211 173.125 0.0057954 0.0053643 2024-02-05T08:23:25.761 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:25.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:25.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:25.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:25.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:25.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:25.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:26.033 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:26.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:25 smithi196 ceph-mon[33018]: 1.3 scrub starts 2024-02-05T08:23:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:25 smithi196 ceph-mon[33018]: 1.3 scrub ok 2024-02-05T08:23:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:25 smithi196 ceph-mon[33018]: 6.19 scrub starts 2024-02-05T08:23:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:25 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:23:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:25 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:23:26.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:25 smithi196 ceph-mon[33018]: 6.19 scrub ok 2024-02-05T08:23:26.136 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:26.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:26.168 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:26.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:26.204 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:25 smithi134 ceph-mon[30086]: 1.3 scrub starts 2024-02-05T08:23:26.204 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:25 smithi134 ceph-mon[30086]: 1.3 scrub ok 2024-02-05T08:23:26.204 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:25 smithi134 ceph-mon[30086]: 6.19 scrub starts 2024-02-05T08:23:26.204 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:25 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:23:26.204 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:25 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:23:26.204 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:25 smithi134 ceph-mon[30086]: 6.19 scrub ok 2024-02-05T08:23:26.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:25 smithi134 ceph-mon[39278]: 1.3 scrub starts 2024-02-05T08:23:26.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:25 smithi134 ceph-mon[39278]: 1.3 scrub ok 2024-02-05T08:23:26.204 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:25 smithi134 ceph-mon[39278]: 6.19 scrub starts 2024-02-05T08:23:26.205 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:25 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:23:26.205 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:25 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:23:26.205 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:25 smithi134 ceph-mon[39278]: 6.19 scrub ok 2024-02-05T08:23:26.269 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:26.311 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:26.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:26.332 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:23:26.332 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:23:26.333 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:23:26.333 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:23:26.359 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:23:26.397 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:23:26.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:26.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:26.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:26.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:26.460 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:23:26.503 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:23:26.507 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:23:26.508 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:23:26.508 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:23:26.508 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:23:26.508 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:23:26.508 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:23:26.508 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:23:26.508 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:23:26.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:26.577 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:26.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:26.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:26.716 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:26.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:26.731 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 7 15 20116 20101 179.404 138.562 0.00603741 0.00556758 2024-02-05T08:23:26.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:26.854 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:26.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:26.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:26.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:26.994 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:26.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:26.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:27.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:27.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:23:27.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:27.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:27.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T08:23:27.170 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:26 smithi134 ceph-mon[30086]: pgmap v840: 158 pgs: 158 active+clean; 640 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-02-05T08:23:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:26 smithi134 ceph-mon[30086]: 4.1d scrub starts 2024-02-05T08:23:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:26 smithi134 ceph-mon[30086]: 4.1d scrub ok 2024-02-05T08:23:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:26 smithi134 ceph-mon[30086]: osdmap e378: 8 total, 8 up, 7 in 2024-02-05T08:23:27.171 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:26 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3181934109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:27.171 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:26 smithi134 ceph-mon[39278]: pgmap v840: 158 pgs: 158 active+clean; 640 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-02-05T08:23:27.171 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:26 smithi134 ceph-mon[39278]: 4.1d scrub starts 2024-02-05T08:23:27.171 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:26 smithi134 ceph-mon[39278]: 4.1d scrub ok 2024-02-05T08:23:27.171 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:26 smithi134 ceph-mon[39278]: osdmap e378: 8 total, 8 up, 7 in 2024-02-05T08:23:27.171 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:26 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3181934109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:27.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:27.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:27.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:26 smithi196 ceph-mon[33018]: pgmap v840: 158 pgs: 158 active+clean; 640 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 107 MiB/s wr, 1.71k op/s 2024-02-05T08:23:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:26 smithi196 ceph-mon[33018]: 4.1d scrub starts 2024-02-05T08:23:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:26 smithi196 ceph-mon[33018]: 4.1d scrub ok 2024-02-05T08:23:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:26 smithi196 ceph-mon[33018]: osdmap e378: 8 total, 8 up, 7 in 2024-02-05T08:23:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:26 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3181934109' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:27.389 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:27.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:27.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:27.522 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:27.564 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:27.567 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:27.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:27.669 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:27.703 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:27.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:27.730 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 8 16 22633 22617 176.634 157.25 0.00534825 0.0056556 2024-02-05T08:23:27.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:27.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:27.847 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:27.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:27.949 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:27.986 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:27.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:27.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:28.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:23:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:23:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[30086]: 4.5 scrub starts 2024-02-05T08:23:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[30086]: 4.5 scrub ok 2024-02-05T08:23:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[30086]: 6.6 scrub starts 2024-02-05T08:23:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[30086]: 6.6 scrub ok 2024-02-05T08:23:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[30086]: 4.17 scrub starts 2024-02-05T08:23:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[30086]: 4.17 scrub ok 2024-02-05T08:23:28.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[30086]: 5.1a scrub starts 2024-02-05T08:23:28.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[30086]: 5.1a scrub ok 2024-02-05T08:23:28.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[30086]: 2.2 scrub starts 2024-02-05T08:23:28.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[30086]: 2.2 scrub ok 2024-02-05T08:23:28.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[30086]: 3.4 scrub starts 2024-02-05T08:23:28.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[30086]: 3.4 scrub ok 2024-02-05T08:23:28.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[39278]: 4.5 scrub starts 2024-02-05T08:23:28.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[39278]: 4.5 scrub ok 2024-02-05T08:23:28.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[39278]: 6.6 scrub starts 2024-02-05T08:23:28.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[39278]: 6.6 scrub ok 2024-02-05T08:23:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[39278]: 4.17 scrub starts 2024-02-05T08:23:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[39278]: 4.17 scrub ok 2024-02-05T08:23:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[39278]: 5.1a scrub starts 2024-02-05T08:23:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[39278]: 5.1a scrub ok 2024-02-05T08:23:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[39278]: 2.2 scrub starts 2024-02-05T08:23:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[39278]: 2.2 scrub ok 2024-02-05T08:23:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[39278]: 3.4 scrub starts 2024-02-05T08:23:28.029 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:27 smithi134 ceph-mon[39278]: 3.4 scrub ok 2024-02-05T08:23:28.068 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:23:28.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:28.135 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:28.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:28.204 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:27 smithi196 ceph-mon[33018]: 4.5 scrub starts 2024-02-05T08:23:28.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:27 smithi196 ceph-mon[33018]: 4.5 scrub ok 2024-02-05T08:23:28.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:27 smithi196 ceph-mon[33018]: 6.6 scrub starts 2024-02-05T08:23:28.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:27 smithi196 ceph-mon[33018]: 6.6 scrub ok 2024-02-05T08:23:28.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:27 smithi196 ceph-mon[33018]: 4.17 scrub starts 2024-02-05T08:23:28.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:27 smithi196 ceph-mon[33018]: 4.17 scrub ok 2024-02-05T08:23:28.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:27 smithi196 ceph-mon[33018]: 5.1a scrub starts 2024-02-05T08:23:28.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:27 smithi196 ceph-mon[33018]: 5.1a scrub ok 2024-02-05T08:23:28.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:27 smithi196 ceph-mon[33018]: 2.2 scrub starts 2024-02-05T08:23:28.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:27 smithi196 ceph-mon[33018]: 2.2 scrub ok 2024-02-05T08:23:28.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:27 smithi196 ceph-mon[33018]: 3.4 scrub starts 2024-02-05T08:23:28.205 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:27 smithi196 ceph-mon[33018]: 3.4 scrub ok 2024-02-05T08:23:28.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:28.278 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:28.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:28.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:28.421 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:28.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:28.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:28.523 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:28.559 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:28.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:28.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:28.661 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:28.685 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:28.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:28.730 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 9 16 24992 24976 173.39 147.438 0.00431703 0.00576177 2024-02-05T08:23:28.787 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:28.826 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:28.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:28.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:28.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:28.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:28.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:28.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:29.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:29.103 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:29.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:29.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:29.205 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:29.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:28 smithi134 ceph-mon[30086]: 4.1b scrub starts 2024-02-05T08:23:29.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:28 smithi134 ceph-mon[30086]: 4.1b scrub ok 2024-02-05T08:23:29.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:28 smithi134 ceph-mon[30086]: pgmap v842: 158 pgs: 158 active+clean; 798 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-02-05T08:23:29.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:28 smithi134 ceph-mon[30086]: 3.16 scrub starts 2024-02-05T08:23:29.242 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:28 smithi134 ceph-mon[30086]: 3.16 scrub ok 2024-02-05T08:23:29.243 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:28 smithi134 ceph-mon[39278]: 4.1b scrub starts 2024-02-05T08:23:29.243 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:28 smithi134 ceph-mon[39278]: 4.1b scrub ok 2024-02-05T08:23:29.243 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:28 smithi134 ceph-mon[39278]: pgmap v842: 158 pgs: 158 active+clean; 798 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-02-05T08:23:29.243 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:28 smithi134 ceph-mon[39278]: 3.16 scrub starts 2024-02-05T08:23:29.243 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:28 smithi134 ceph-mon[39278]: 3.16 scrub ok 2024-02-05T08:23:29.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:29.244 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:29.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:28 smithi196 ceph-mon[33018]: 4.1b scrub starts 2024-02-05T08:23:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:28 smithi196 ceph-mon[33018]: 4.1b scrub ok 2024-02-05T08:23:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:28 smithi196 ceph-mon[33018]: pgmap v842: 158 pgs: 158 active+clean; 798 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 128 MiB/s wr, 2.04k op/s 2024-02-05T08:23:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:28 smithi196 ceph-mon[33018]: 3.16 scrub starts 2024-02-05T08:23:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:28 smithi196 ceph-mon[33018]: 3.16 scrub ok 2024-02-05T08:23:29.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:29.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:29.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:29.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:29.445 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:23:29.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:29.533 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:29.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:29.634 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:29.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:29.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:29.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:29.730 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 10 15 27343 27328 170.748 147 0.00656505 0.00585013 2024-02-05T08:23:29.784 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:29.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:29.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:29.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:29.894 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:29.948 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:29.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:29.992 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:29.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:30.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[30086]: 3.a scrub starts 2024-02-05T08:23:30.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[30086]: 3.a scrub ok 2024-02-05T08:23:30.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[30086]: 6.b scrub starts 2024-02-05T08:23:30.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[30086]: 3.1c scrub starts 2024-02-05T08:23:30.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[30086]: 3.1c scrub ok 2024-02-05T08:23:30.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[30086]: 6.b scrub ok 2024-02-05T08:23:30.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[30086]: osdmap e379: 8 total, 8 up, 7 in 2024-02-05T08:23:30.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[39278]: 3.a scrub starts 2024-02-05T08:23:30.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[39278]: 3.a scrub ok 2024-02-05T08:23:30.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[39278]: 6.b scrub starts 2024-02-05T08:23:30.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[39278]: 3.1c scrub starts 2024-02-05T08:23:30.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[39278]: 3.1c scrub ok 2024-02-05T08:23:30.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[39278]: 6.b scrub ok 2024-02-05T08:23:30.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:29 smithi134 ceph-mon[39278]: osdmap e379: 8 total, 8 up, 7 in 2024-02-05T08:23:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:29 smithi196 ceph-mon[33018]: 3.a scrub starts 2024-02-05T08:23:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:29 smithi196 ceph-mon[33018]: 3.a scrub ok 2024-02-05T08:23:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:29 smithi196 ceph-mon[33018]: 6.b scrub starts 2024-02-05T08:23:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:29 smithi196 ceph-mon[33018]: 3.1c scrub starts 2024-02-05T08:23:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:29 smithi196 ceph-mon[33018]: 3.1c scrub ok 2024-02-05T08:23:30.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:29 smithi196 ceph-mon[33018]: 6.b scrub ok 2024-02-05T08:23:30.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:29 smithi196 ceph-mon[33018]: osdmap e379: 8 total, 8 up, 7 in 2024-02-05T08:23:30.093 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:30.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:30.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:30.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:30.268 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:30.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:30.304 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:23:30.304 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:23:30.369 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:30.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:30.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:30.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:30.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:30.549 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:30.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:30.637 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 7, 6, 3] out_osds: [0] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-02-05T08:23:30.637 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:23:30.638 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-05T08:23:30.638 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T08:23:30.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:30.727 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:30.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:30.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:30.730 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 11 16 29336 29320 166.542 124.5 0.0110743 0.00599894 2024-02-05T08:23:30.830 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:30.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:30.873 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:30.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:30.974 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:31.012 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:31.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:31.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:31.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T08:23:31.149 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:31.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:31.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:31.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:31.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:23:31.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:23:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[30086]: pgmap v844: 158 pgs: 158 active+clean; 1.4 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 199 MiB/s wr, 3.18k op/s 2024-02-05T08:23:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[30086]: 4.1c scrub starts 2024-02-05T08:23:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[30086]: 4.1c scrub ok 2024-02-05T08:23:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[30086]: 3.8 scrub starts 2024-02-05T08:23:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[30086]: 3.8 scrub ok 2024-02-05T08:23:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[30086]: 3.e scrub starts 2024-02-05T08:23:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[30086]: 3.e scrub ok 2024-02-05T08:23:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[30086]: 4.f scrub starts 2024-02-05T08:23:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[30086]: 4.f scrub ok 2024-02-05T08:23:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:23:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:23:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[39278]: pgmap v844: 158 pgs: 158 active+clean; 1.4 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 199 MiB/s wr, 3.18k op/s 2024-02-05T08:23:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[39278]: 4.1c scrub starts 2024-02-05T08:23:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[39278]: 4.1c scrub ok 2024-02-05T08:23:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[39278]: 3.8 scrub starts 2024-02-05T08:23:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[39278]: 3.8 scrub ok 2024-02-05T08:23:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[39278]: 3.e scrub starts 2024-02-05T08:23:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[39278]: 3.e scrub ok 2024-02-05T08:23:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[39278]: 4.f scrub starts 2024-02-05T08:23:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:30 smithi134 ceph-mon[39278]: 4.f scrub ok 2024-02-05T08:23:31.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:31.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:31.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:30 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:23:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:30 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:23:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:30 smithi196 ceph-mon[33018]: pgmap v844: 158 pgs: 158 active+clean; 1.4 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 199 MiB/s wr, 3.18k op/s 2024-02-05T08:23:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:30 smithi196 ceph-mon[33018]: 4.1c scrub starts 2024-02-05T08:23:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:30 smithi196 ceph-mon[33018]: 4.1c scrub ok 2024-02-05T08:23:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:30 smithi196 ceph-mon[33018]: 3.8 scrub starts 2024-02-05T08:23:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:30 smithi196 ceph-mon[33018]: 3.8 scrub ok 2024-02-05T08:23:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:30 smithi196 ceph-mon[33018]: 3.e scrub starts 2024-02-05T08:23:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:30 smithi196 ceph-mon[33018]: 3.e scrub ok 2024-02-05T08:23:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:30 smithi196 ceph-mon[33018]: 4.f scrub starts 2024-02-05T08:23:31.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:30 smithi196 ceph-mon[33018]: 4.f scrub ok 2024-02-05T08:23:31.400 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:31.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:31.445 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:31.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:31.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:31.559 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:23:31.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:31.600 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:31.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:31.701 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:31.730 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 12 16 31382 31366 163.32 127.875 0.00626428 0.00611651 2024-02-05T08:23:31.740 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:31.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:31.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:31.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:31.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:31.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:32.031 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:32.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:32.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:32.133 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:32.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:32.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:32.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:32.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:32.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:31 smithi134 ceph-mon[30086]: 4.9 scrub starts 2024-02-05T08:23:32.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:31 smithi134 ceph-mon[30086]: 4.9 scrub ok 2024-02-05T08:23:32.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:31 smithi134 ceph-mon[30086]: 1.2 scrub starts 2024-02-05T08:23:32.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:31 smithi134 ceph-mon[30086]: 1.2 scrub ok 2024-02-05T08:23:32.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:31 smithi134 ceph-mon[39278]: 4.9 scrub starts 2024-02-05T08:23:32.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:31 smithi134 ceph-mon[39278]: 4.9 scrub ok 2024-02-05T08:23:32.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:31 smithi134 ceph-mon[39278]: 1.2 scrub starts 2024-02-05T08:23:32.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:31 smithi134 ceph-mon[39278]: 1.2 scrub ok 2024-02-05T08:23:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:32.299 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:32.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:31 smithi196 ceph-mon[33018]: 4.9 scrub starts 2024-02-05T08:23:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:31 smithi196 ceph-mon[33018]: 4.9 scrub ok 2024-02-05T08:23:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:31 smithi196 ceph-mon[33018]: 1.2 scrub starts 2024-02-05T08:23:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:31 smithi196 ceph-mon[33018]: 1.2 scrub ok 2024-02-05T08:23:32.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:32.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:32.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:32.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:32.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:32.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:32.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:32.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:32.687 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:32.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:32.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:32.730 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 13 16 32983 32967 158.454 100.062 0.0080785 0.00630516 2024-02-05T08:23:32.789 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:32.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:32.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:32.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:32.923 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:32.961 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:32.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:32.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:33.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:33.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:33.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:33.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:33.218 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:33.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:33.275 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:23:33.275 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[30086]: 4.18 scrub starts 2024-02-05T08:23:33.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[30086]: 4.18 scrub ok 2024-02-05T08:23:33.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[30086]: 5.11 scrub starts 2024-02-05T08:23:33.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[30086]: 5.11 scrub ok 2024-02-05T08:23:33.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[30086]: pgmap v845: 158 pgs: 158 active+clean; 1.5 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-02-05T08:23:33.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[30086]: 4.10 scrub starts 2024-02-05T08:23:33.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[30086]: 4.10 scrub ok 2024-02-05T08:23:33.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[30086]: 5.19 scrub starts 2024-02-05T08:23:33.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[30086]: 5.19 scrub ok 2024-02-05T08:23:33.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2666016509' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:23:33.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:23:33.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[39278]: 4.18 scrub starts 2024-02-05T08:23:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[39278]: 4.18 scrub ok 2024-02-05T08:23:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[39278]: 5.11 scrub starts 2024-02-05T08:23:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[39278]: 5.11 scrub ok 2024-02-05T08:23:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[39278]: pgmap v845: 158 pgs: 158 active+clean; 1.5 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-02-05T08:23:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[39278]: 4.10 scrub starts 2024-02-05T08:23:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[39278]: 4.10 scrub ok 2024-02-05T08:23:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[39278]: 5.19 scrub starts 2024-02-05T08:23:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[39278]: 5.19 scrub ok 2024-02-05T08:23:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2666016509' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:23:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:32 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:23:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:32 smithi196 ceph-mon[33018]: 4.18 scrub starts 2024-02-05T08:23:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:32 smithi196 ceph-mon[33018]: 4.18 scrub ok 2024-02-05T08:23:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:32 smithi196 ceph-mon[33018]: 5.11 scrub starts 2024-02-05T08:23:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:32 smithi196 ceph-mon[33018]: 5.11 scrub ok 2024-02-05T08:23:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:32 smithi196 ceph-mon[33018]: pgmap v845: 158 pgs: 158 active+clean; 1.5 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 113 MiB/s wr, 1.80k op/s 2024-02-05T08:23:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:32 smithi196 ceph-mon[33018]: 4.10 scrub starts 2024-02-05T08:23:33.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:32 smithi196 ceph-mon[33018]: 4.10 scrub ok 2024-02-05T08:23:33.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:32 smithi196 ceph-mon[33018]: 5.19 scrub starts 2024-02-05T08:23:33.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:32 smithi196 ceph-mon[33018]: 5.19 scrub ok 2024-02-05T08:23:33.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:32 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2666016509' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:23:33.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:32 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:23:33.320 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:33.355 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:33.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:33.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:33.457 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:33.496 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:33.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:33.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:33.637 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:33.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:33.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:33.730 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 14 16 34520 34504 153.998 96.0625 0.00890871 0.00648905 2024-02-05T08:23:33.739 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:33.777 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:33.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:33.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:33.917 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:33.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:33.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:34.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:34.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:34.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:34.161 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:34.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:34.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:34.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[30086]: 1.6 scrub starts 2024-02-05T08:23:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[30086]: 1.6 scrub ok 2024-02-05T08:23:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:23:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:23:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:23:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[30086]: osdmap e380: 8 total, 8 up, 7 in 2024-02-05T08:23:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[30086]: 6.16 deep-scrub starts 2024-02-05T08:23:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[30086]: 6.16 deep-scrub ok 2024-02-05T08:23:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[30086]: 6.1d scrub starts 2024-02-05T08:23:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[30086]: 6.1d scrub ok 2024-02-05T08:23:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[39278]: 1.6 scrub starts 2024-02-05T08:23:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[39278]: 1.6 scrub ok 2024-02-05T08:23:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:23:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:23:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:23:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[39278]: osdmap e380: 8 total, 8 up, 7 in 2024-02-05T08:23:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[39278]: 6.16 deep-scrub starts 2024-02-05T08:23:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[39278]: 6.16 deep-scrub ok 2024-02-05T08:23:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[39278]: 6.1d scrub starts 2024-02-05T08:23:34.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:33 smithi134 ceph-mon[39278]: 6.1d scrub ok 2024-02-05T08:23:34.286 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:34.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:33 smithi196 ceph-mon[33018]: 1.6 scrub starts 2024-02-05T08:23:34.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:33 smithi196 ceph-mon[33018]: 1.6 scrub ok 2024-02-05T08:23:34.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:33 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:23:34.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:33 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:23:34.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:33 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:23:34.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:33 smithi196 ceph-mon[33018]: osdmap e380: 8 total, 8 up, 7 in 2024-02-05T08:23:34.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:33 smithi196 ceph-mon[33018]: 6.16 deep-scrub starts 2024-02-05T08:23:34.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:33 smithi196 ceph-mon[33018]: 6.16 deep-scrub ok 2024-02-05T08:23:34.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:33 smithi196 ceph-mon[33018]: 6.1d scrub starts 2024-02-05T08:23:34.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:33 smithi196 ceph-mon[33018]: 6.1d scrub ok 2024-02-05T08:23:34.317 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:23:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:23:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:34.322 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:34.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:34.423 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:34.462 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:34.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:34.563 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:34.599 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:34.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:34.701 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:34.724 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:34.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:34.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:34.730 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 15 16 35676 35660 148.548 72.25 0.00194545 0.00667068 2024-02-05T08:23:34.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:34.862 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:34.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:34.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:34.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:34.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:34.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:35.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:35.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:35.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:35.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:35.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:35.271 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:35.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[30086]: 1.7 scrub starts 2024-02-05T08:23:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[30086]: 1.7 scrub ok 2024-02-05T08:23:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[30086]: pgmap v847: 158 pgs: 158 active+clean; 2.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 167 MiB/s wr, 2.67k op/s 2024-02-05T08:23:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[30086]: 5.0 scrub starts 2024-02-05T08:23:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[30086]: 5.0 scrub ok 2024-02-05T08:23:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[30086]: 1.0 scrub starts 2024-02-05T08:23:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[30086]: 1.0 scrub ok 2024-02-05T08:23:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[30086]: osdmap e381: 8 total, 8 up, 7 in 2024-02-05T08:23:35.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[39278]: 1.7 scrub starts 2024-02-05T08:23:35.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[39278]: 1.7 scrub ok 2024-02-05T08:23:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[39278]: pgmap v847: 158 pgs: 158 active+clean; 2.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 167 MiB/s wr, 2.67k op/s 2024-02-05T08:23:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[39278]: 5.0 scrub starts 2024-02-05T08:23:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[39278]: 5.0 scrub ok 2024-02-05T08:23:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[39278]: 1.0 scrub starts 2024-02-05T08:23:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[39278]: 1.0 scrub ok 2024-02-05T08:23:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:35 smithi134 ceph-mon[39278]: osdmap e381: 8 total, 8 up, 7 in 2024-02-05T08:23:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:35 smithi196 ceph-mon[33018]: 1.7 scrub starts 2024-02-05T08:23:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:35 smithi196 ceph-mon[33018]: 1.7 scrub ok 2024-02-05T08:23:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:35 smithi196 ceph-mon[33018]: pgmap v847: 158 pgs: 158 active+clean; 2.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 167 MiB/s wr, 2.67k op/s 2024-02-05T08:23:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:35 smithi196 ceph-mon[33018]: 5.0 scrub starts 2024-02-05T08:23:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:35 smithi196 ceph-mon[33018]: 5.0 scrub ok 2024-02-05T08:23:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:35 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:35 smithi196 ceph-mon[33018]: 1.0 scrub starts 2024-02-05T08:23:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:35 smithi196 ceph-mon[33018]: 1.0 scrub ok 2024-02-05T08:23:35.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:35 smithi196 ceph-mon[33018]: osdmap e381: 8 total, 8 up, 7 in 2024-02-05T08:23:35.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:35.399 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:35.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:35.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:35.525 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:35.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:35.626 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:35.668 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:35.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:35.730 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 16 16 36852 36836 143.858 73.5 0.00562928 0.00694636 2024-02-05T08:23:35.770 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:35.809 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:35.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:35.911 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:35.933 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:35.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:35.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:36.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:36.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:36.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:36.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:36.189 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:36.226 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:36.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:36.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:36.352 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:36.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:36.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:36.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:36.493 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:36.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:36.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:36.594 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:36.639 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:36.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:36.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:36.732 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 17 16 38636 38620 141.943 111.5 0.00782952 0.00704015 2024-02-05T08:23:36.742 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:36.785 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:36.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:36.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:36.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:36.928 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:36.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:37.030 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:37.068 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:37.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:37.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:37.170 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:37.209 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:37.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:37.274 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:37 smithi134 ceph-mon[30086]: 5.16 scrub starts 2024-02-05T08:23:37.274 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:37 smithi134 ceph-mon[30086]: 5.16 scrub ok 2024-02-05T08:23:37.274 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:37 smithi134 ceph-mon[30086]: pgmap v849: 158 pgs: 158 active+clean; 2.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-05T08:23:37.275 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:37 smithi134 ceph-mon[30086]: 6.3 scrub starts 2024-02-05T08:23:37.275 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:37 smithi134 ceph-mon[30086]: 6.3 scrub ok 2024-02-05T08:23:37.275 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:37 smithi134 ceph-mon[39278]: 5.16 scrub starts 2024-02-05T08:23:37.275 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:37 smithi134 ceph-mon[39278]: 5.16 scrub ok 2024-02-05T08:23:37.275 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:37 smithi134 ceph-mon[39278]: pgmap v849: 158 pgs: 158 active+clean; 2.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-05T08:23:37.275 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:37 smithi134 ceph-mon[39278]: 6.3 scrub starts 2024-02-05T08:23:37.275 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:37 smithi134 ceph-mon[39278]: 6.3 scrub ok 2024-02-05T08:23:37.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:37.317 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:37 smithi196 ceph-mon[33018]: 5.16 scrub starts 2024-02-05T08:23:37.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:37 smithi196 ceph-mon[33018]: 5.16 scrub ok 2024-02-05T08:23:37.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:37 smithi196 ceph-mon[33018]: pgmap v849: 158 pgs: 158 active+clean; 2.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-02-05T08:23:37.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:37 smithi196 ceph-mon[33018]: 6.3 scrub starts 2024-02-05T08:23:37.318 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:37 smithi196 ceph-mon[33018]: 6.3 scrub ok 2024-02-05T08:23:37.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:37.348 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:37.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:37.449 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:37.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:37.492 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:37.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:37.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:37.638 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:37.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:37.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:37.732 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 18 16 40499 40483 140.525 116.438 0.0119352 0.00711125 2024-02-05T08:23:37.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:37.782 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:37.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:37.788 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:23:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:23:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:23:37.884 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:37.918 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:37.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:37.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:38.011 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:23:38.011 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:23:38.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:38 smithi196 ceph-mon[33018]: 4.3 deep-scrub starts 2024-02-05T08:23:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:38 smithi196 ceph-mon[33018]: 5.d scrub starts 2024-02-05T08:23:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:38 smithi196 ceph-mon[33018]: 5.d scrub ok 2024-02-05T08:23:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:38 smithi196 ceph-mon[33018]: 4.3 deep-scrub ok 2024-02-05T08:23:38.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:38 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:23:38.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:38.066 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:38.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:38.115 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:38 smithi134 ceph-mon[30086]: 4.3 deep-scrub starts 2024-02-05T08:23:38.115 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:38 smithi134 ceph-mon[30086]: 5.d scrub starts 2024-02-05T08:23:38.115 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:38 smithi134 ceph-mon[30086]: 5.d scrub ok 2024-02-05T08:23:38.115 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:38 smithi134 ceph-mon[30086]: 4.3 deep-scrub ok 2024-02-05T08:23:38.115 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:38 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:23:38.115 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:38 smithi134 ceph-mon[39278]: 4.3 deep-scrub starts 2024-02-05T08:23:38.115 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:38 smithi134 ceph-mon[39278]: 5.d scrub starts 2024-02-05T08:23:38.115 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:38 smithi134 ceph-mon[39278]: 5.d scrub ok 2024-02-05T08:23:38.116 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:38 smithi134 ceph-mon[39278]: 4.3 deep-scrub ok 2024-02-05T08:23:38.116 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:38 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:23:38.167 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:38.207 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:38.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:38.308 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:38.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:38.354 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:38.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:38.455 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:38.497 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:38.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:38.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:38.630 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:38.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:38.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:38.733 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 19 16 42115 42099 138.445 101 0.010626 0.00721768 2024-02-05T08:23:38.767 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:38.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:38.799 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:23:38.846 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:23:38.869 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:38.890 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:23:38.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T08:23:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:38.910 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:38.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:38.930 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:23:38.970 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:23:39.010 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:23:39.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:39.057 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:39.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:39.060 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:23:39.102 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:23:39.107 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:23:39.107 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:23:39.107 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:23:39.107 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:23:39.107 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:23:39.107 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:23:39.108 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:23:39.108 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:23:39.159 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:23:39.160 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:39.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:39.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:39.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:39.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:39.355 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:23:39.363 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:39.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:39.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:39.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:39.542 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:39.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:39.643 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:39.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:39.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:39.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:39.732 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:23:39.733542+0000 min lat: 0.00126536 max lat: 0.205027 avg lat: 0.00732549 2024-02-05T08:23:39.732 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:23:39.733 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 20 16 43681 43665 136.416 97.875 0.00529058 0.00732549 2024-02-05T08:23:39.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[30086]: pgmap v850: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-05T08:23:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:23:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1376314619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[30086]: 3.5 scrub starts 2024-02-05T08:23:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[30086]: 3.5 scrub ok 2024-02-05T08:23:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:23:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:23:39.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:39.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[39278]: pgmap v850: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-05T08:23:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:23:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1376314619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[39278]: 3.5 scrub starts 2024-02-05T08:23:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[39278]: 3.5 scrub ok 2024-02-05T08:23:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:23:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:23:39.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:39.784 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:39 smithi196 ceph-mon[33018]: pgmap v850: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-05T08:23:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:23:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:39 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1376314619' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:39 smithi196 ceph-mon[33018]: 3.5 scrub starts 2024-02-05T08:23:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:39 smithi196 ceph-mon[33018]: 3.5 scrub ok 2024-02-05T08:23:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:23:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:23:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:39.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:39.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:39.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:39.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:39.961 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:39.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:40.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:40.097 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:40.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:40.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:40.199 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:40.237 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:40.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:40.339 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:40.373 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:40.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:40.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:40.475 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:40.517 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:40.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:40.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:40.656 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:40.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:40.732 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 21 16 45446 45430 135.173 110.312 0.0102032 0.00739201 2024-02-05T08:23:40.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:40.793 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:40.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:40.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:40.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:40.941 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:40.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:41.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[30086]: 6.4 scrub starts 2024-02-05T08:23:41.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[30086]: 6.4 scrub ok 2024-02-05T08:23:41.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[30086]: 4.e scrub starts 2024-02-05T08:23:41.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[30086]: 4.e scrub ok 2024-02-05T08:23:41.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[30086]: pgmap v851: 158 pgs: 158 active+clean; 2.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.02k op/s 2024-02-05T08:23:41.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[30086]: 6.1c scrub starts 2024-02-05T08:23:41.017 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[30086]: 6.1c scrub ok 2024-02-05T08:23:41.017 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[39278]: 6.4 scrub starts 2024-02-05T08:23:41.017 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[39278]: 6.4 scrub ok 2024-02-05T08:23:41.017 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[39278]: 4.e scrub starts 2024-02-05T08:23:41.017 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[39278]: 4.e scrub ok 2024-02-05T08:23:41.017 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[39278]: pgmap v851: 158 pgs: 158 active+clean; 2.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.02k op/s 2024-02-05T08:23:41.017 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[39278]: 6.1c scrub starts 2024-02-05T08:23:41.017 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:40 smithi134 ceph-mon[39278]: 6.1c scrub ok 2024-02-05T08:23:41.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:41.044 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:41.075 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:41.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:41.176 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:41.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:41.213 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:41.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:41.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:40 smithi196 ceph-mon[33018]: 6.4 scrub starts 2024-02-05T08:23:41.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:40 smithi196 ceph-mon[33018]: 6.4 scrub ok 2024-02-05T08:23:41.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:40 smithi196 ceph-mon[33018]: 4.e scrub starts 2024-02-05T08:23:41.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:40 smithi196 ceph-mon[33018]: 4.e scrub ok 2024-02-05T08:23:41.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:40 smithi196 ceph-mon[33018]: pgmap v851: 158 pgs: 158 active+clean; 2.5 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.02k op/s 2024-02-05T08:23:41.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:40 smithi196 ceph-mon[33018]: 6.1c scrub starts 2024-02-05T08:23:41.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:40 smithi196 ceph-mon[33018]: 6.1c scrub ok 2024-02-05T08:23:41.315 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:41.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:41.381 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:41.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:41.482 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:41.492 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:23:41.495 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:23:41.495 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 381, 2024-02-05T08:23:41.495 INFO:teuthology.orchestra.run.smithi134.stdout: "fsid": "43e13aa4-c3fc-11ee-95b6-87774f69a715", 2024-02-05T08:23:41.496 INFO:teuthology.orchestra.run.smithi134.stdout: "created": "2024-02-05T07:59:09.321262+0000", 2024-02-05T08:23:41.496 INFO:teuthology.orchestra.run.smithi134.stdout: "modified": "2024-02-05T08:23:34.878264+0000", 2024-02-05T08:23:41.496 INFO:teuthology.orchestra.run.smithi134.stdout: "last_up_change": "2024-02-05T08:23:06.892657+0000", 2024-02-05T08:23:41.496 INFO:teuthology.orchestra.run.smithi134.stdout: "last_in_change": "2024-02-05T08:22:08.072805+0000", 2024-02-05T08:23:41.496 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T08:23:41.496 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_num": 5799936, 2024-02-05T08:23:41.496 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_set": [ 2024-02-05T08:23:41.496 INFO:teuthology.orchestra.run.smithi134.stdout: "pglog_hardlimit", 2024-02-05T08:23:41.496 INFO:teuthology.orchestra.run.smithi134.stdout: "purged_snapdirs", 2024-02-05T08:23:41.496 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_deletes", 2024-02-05T08:23:41.496 INFO:teuthology.orchestra.run.smithi134.stdout: "sortbitwise" 2024-02-05T08:23:41.496 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_version": 19, 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_snapshot": "", 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_max": 11, 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: "max_osd": 8, 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: "require_min_compat_client": "luminous", 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: "min_compat_client": "luminous", 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: "require_osd_release": "squid", 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: "allow_crimson": false, 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: "pools": [ 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.497 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 1, 2024-02-05T08:23:41.498 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": ".mgr", 2024-02-05T08:23:41.498 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:02:21.034012+0000", 2024-02-05T08:23:41.498 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:23:41.498 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:23:41.498 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:23:41.498 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:23:41.498 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:23:41.498 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:23:41.498 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:23:41.498 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:23:41.498 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:23:41.498 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:23:41.498 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 11, 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 11, 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 11, 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 11, 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 11, 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:23:41.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "303", 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "301", 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:23:41.500 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:23:41.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.502 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:23:41.502 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:23:41.502 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:23:41.502 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:23:41.502 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:23:41.502 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:23:41.502 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:23:41.502 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:23:41.502 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:23:41.502 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:23:41.502 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:23:41.502 INFO:teuthology.orchestra.run.smithi134.stdout: "options": { 2024-02-05T08:23:41.502 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_max": 32, 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_min": 1 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr": {} 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.1700000762939453, 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.1700000762939453, 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.9099999666213989, 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.9099999666213989, 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:23:41.503 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 2, 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "datapool", 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:39.400746+0000", 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 8193, 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:23:41.504 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 3, 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 3, 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 3, 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 3, 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 3, 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:23:41.505 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "70", 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 2, 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 70, 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:23:41.506 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:23:41.507 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:23:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:23:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:23:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:23:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:23:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:23:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:23:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:23:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:23:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:23:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:23:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:23:41.508 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd": {} 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.6700000762939453, 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.6700000762939453, 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.75, 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 2, 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 2, 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.510 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 3, 2024-02-05T08:23:41.510 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": ".rgw.root", 2024-02-05T08:23:41.510 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:40.005769+0000", 2024-02-05T08:23:41.510 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:23:41.510 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:23:41.510 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:23:41.510 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:23:41.510 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:23:41.510 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:23:41.510 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:23:41.510 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:23:41.510 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:23:41.511 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:23:41.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.512 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "68", 2024-02-05T08:23:41.512 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:23:41.512 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:23:41.512 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:23:41.513 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:23:41.514 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:23:41.515 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.7400000095367432, 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.7400000095367432, 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.5299999713897705, 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.5299999713897705, 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 4, 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.log", 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:41.851891+0000", 2024-02-05T08:23:41.516 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:23:41.517 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "70", 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:23:41.518 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:23:41.519 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:23:41.519 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:23:41.519 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:23:41.519 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:23:41.519 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:23:41.519 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:23:41.519 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:23:41.519 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:23:41.519 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:23:41.519 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:23:41.519 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:23:41.519 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:23:41.519 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:23:41.520 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:23:41.521 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.2400000095367432, 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.2400000095367432, 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.9700000286102295, 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.9700000286102295, 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 5, 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.control", 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:43.793804+0000", 2024-02-05T08:23:41.522 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:23:41.523 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "72", 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:23:41.524 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:23:41.525 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:23:41.526 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.2400000095367432, 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.2400000095367432, 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:23:41.527 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.0900000333786011, 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.0900000333786011, 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 6, 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.meta", 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:45.853999+0000", 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:23:41.528 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:23:41.529 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "74", 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:23:41.530 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:23:41.531 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:23:41.532 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: "options": { 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_bias": 4 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.9900000095367432, 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.9900000095367432, 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:23:41.533 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.75, 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.75, 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 11, 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "unique_pool_4", 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:23:07.682500+0000", 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:23:41.534 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 16, 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 16, 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 16, 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 16, 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 16, 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:23:41.535 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "373", 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:23:41.536 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:23:41.537 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:23:41.538 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: "rados": {} 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.9900000095367432, 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.9900000095367432, 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.75, 2024-02-05T08:23:41.539 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.75, 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 1, 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 1, 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: "osds": [ 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 0, 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "d90a5ea0-9a19-452e-9564-ca7b7a7c45e4", 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 0, 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 0, 2024-02-05T08:23:41.540 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 8, 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 324, 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6802", 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.541 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6803", 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6804", 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.542 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6805", 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6808", 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.543 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.544 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6809", 2024-02-05T08:23:41.544 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:23:41.544 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.544 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.544 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:23:41.544 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.544 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.544 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6806", 2024-02-05T08:23:41.544 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:23:41.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.545 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.545 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6807", 2024-02-05T08:23:41.545 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:23:41.545 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.545 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6803/2540271081", 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6805/2540271081", 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6809/2540271081", 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6807/2540271081", 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 1, 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4", 2024-02-05T08:23:41.546 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 13, 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 323, 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 366, 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 367, 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 324, 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6810", 2024-02-05T08:23:41.547 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6811", 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6812", 2024-02-05T08:23:41.548 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6813", 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6816", 2024-02-05T08:23:41.549 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6817", 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6814", 2024-02-05T08:23:41.550 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6815", 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6811/1259179009", 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6813/1259179009", 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6817/1259179009", 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6815/1259179009", 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:23:41.551 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 2, 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708", 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 18, 2024-02-05T08:23:41.552 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 367, 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6818", 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6819", 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:23:41.553 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6820", 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6821", 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:23:41.554 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6824", 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6825", 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.555 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6822", 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6823", 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6819/153827987", 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6821/153827987", 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6825/153827987", 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6823/153827987", 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 3, 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "6774e330-3202-4671-894e-172cc2f6189d", 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:23:41.557 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 27, 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 367, 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6826", 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:23:41.558 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6827", 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6828", 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.559 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6829", 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6832", 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.560 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6833", 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6830", 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6831", 2024-02-05T08:23:41.561 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:23:41.562 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.562 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.562 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6827/1527507056", 2024-02-05T08:23:41.562 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6829/1527507056", 2024-02-05T08:23:41.562 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6833/1527507056", 2024-02-05T08:23:41.562 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6831/1527507056", 2024-02-05T08:23:41.562 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:23:41.562 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:23:41.562 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:23:41.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:41.563 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:41.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:41.564 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.564 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 4, 2024-02-05T08:23:41.564 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693", 2024-02-05T08:23:41.564 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:23:41.564 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 32, 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 367, 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6800", 2024-02-05T08:23:41.565 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6801", 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6802", 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6803", 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6806", 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6807", 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6804", 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.568 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6805", 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6801/3005156808", 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6803/3005156808", 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3005156808", 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3005156808", 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.569 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 5, 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955", 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 42, 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 367, 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6808", 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6809", 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.571 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6810", 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6811", 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.572 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6814", 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6815", 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.573 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6812", 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6813", 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6809/1603224792", 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6811/1603224792", 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6815/1603224792", 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6813/1603224792", 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:23:41.574 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 6, 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a", 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 51, 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 367, 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:23:41.575 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6816", 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6817", 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:23:41.576 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6818", 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6819", 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.577 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6822", 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6823", 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.578 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6820", 2024-02-05T08:23:41.579 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:23:41.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.579 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.579 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6821", 2024-02-05T08:23:41.579 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:23:41.579 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6817/3727413143", 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6819/3727413143", 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6823/3727413143", 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6821/3727413143", 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 7, 2024-02-05T08:23:41.580 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057", 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 60, 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 369, 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.581 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6824", 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6825", 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.582 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6826", 2024-02-05T08:23:41.583 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:23:41.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.583 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.583 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6827", 2024-02-05T08:23:41.583 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:23:41.583 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.583 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.583 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:23:41.583 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.583 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6830", 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6831", 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.584 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:23:41.585 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6828", 2024-02-05T08:23:41.585 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:23:41.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.585 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:23:41.585 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6829", 2024-02-05T08:23:41.585 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:23:41.585 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.585 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.585 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6825/4206571010", 2024-02-05T08:23:41.585 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6827/4206571010", 2024-02-05T08:23:41.585 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6831/4206571010", 2024-02-05T08:23:41.586 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6829/4206571010", 2024-02-05T08:23:41.586 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:23:41.586 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:23:41.586 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:23:41.586 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.586 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.586 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:23:41.586 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_xinfo": [ 2024-02-05T08:23:41.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.586 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 0, 2024-02-05T08:23:41.586 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:23:41.586 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:23:41.586 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 65536, 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:23:25.118956+0000", 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 1, 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:23:05.902681+0000", 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 323 2024-02-05T08:23:41.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 2, 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:23:20.945217+0000", 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 3, 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:23:41.588 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:23:22.070521+0000", 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 4, 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:23:41.589 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:23:07.458558+0000", 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 5, 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:23:19.891780+0000", 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.590 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 6, 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:23:06.286495+0000", 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 7, 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:23:41.591 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:22:51.716195+0000", 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap": [ 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "6.6", 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: "osds": [ 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: 4, 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: 1, 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: 3 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.592 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap_items": [ 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "3.6", 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 4, 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.6", 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.593 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 5 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.7", 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.a", 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.12", 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:23:41.595 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 5 2024-02-05T08:23:41.596 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.596 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.596 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.1f", 2024-02-05T08:23:41.596 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:23:41.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:23:41.596 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:23:41.596 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:23:41.596 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.596 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:23:41.596 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.596 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap_primaries": [], 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_temp": [], 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_temp": [], 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "blocklist": { 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/788812699": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/1046500326": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/464208089": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/301941854": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/3607168099": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/673876301": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/4225724311": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/1525062017": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/3607168099": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/673876301": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:23:41.597 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3200884805": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:23:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/2952444615": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:23:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3325835814": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:23:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3161126612": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:23:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/69097091": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:23:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/1525062017": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:23:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/1645475987": "2024-02-06T08:05:21.465844+0000" 2024-02-05T08:23:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: "range_blocklist": {}, 2024-02-05T08:23:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profiles": { 2024-02-05T08:23:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: "default": { 2024-02-05T08:23:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: "crush-failure-domain": "osd", 2024-02-05T08:23:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: "k": "2", 2024-02-05T08:23:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: "m": "1", 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: "plugin": "jerasure", 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: "technique": "reed_sol_van" 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps_queue": [], 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: "new_removed_snaps": [], 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: "new_purged_snaps": [], 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_node_flags": {}, 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: "device_class_flags": {}, 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode": { 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode_enabled": false, 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_bucket_count": 0, 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: "degraded_stretch_mode": 0, 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: "recovering_stretch_mode": 0, 2024-02-05T08:23:41.599 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode_bucket": 0 2024-02-05T08:23:41.600 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:23:41.600 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:23:41.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:41.705 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:41.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:41.733 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 22 16 47031 47015 133.532 99.0625 0.00486634 0.00748376 2024-02-05T08:23:41.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:41.851 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:41.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:41.937 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:23:41.937 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:23:41.937 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T08:23:41.938 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T08:23:41.938 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:23:41.953 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:41.993 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:41.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:41 smithi196 ceph-mon[33018]: 3.1d scrub starts 2024-02-05T08:23:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:41 smithi196 ceph-mon[33018]: 3.1d scrub ok 2024-02-05T08:23:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:41 smithi196 ceph-mon[33018]: 4.2 scrub starts 2024-02-05T08:23:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:41 smithi196 ceph-mon[33018]: 4.2 scrub ok 2024-02-05T08:23:42.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:41 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4141841192' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:23:42.095 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:42.133 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:42.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:42.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:42.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:42.273 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:42.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:42.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:42.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:41 smithi134 ceph-mon[30086]: 3.1d scrub starts 2024-02-05T08:23:42.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:41 smithi134 ceph-mon[30086]: 3.1d scrub ok 2024-02-05T08:23:42.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:41 smithi134 ceph-mon[30086]: 4.2 scrub starts 2024-02-05T08:23:42.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:41 smithi134 ceph-mon[30086]: 4.2 scrub ok 2024-02-05T08:23:42.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:41 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4141841192' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:23:42.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:41 smithi134 ceph-mon[39278]: 3.1d scrub starts 2024-02-05T08:23:42.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:41 smithi134 ceph-mon[39278]: 3.1d scrub ok 2024-02-05T08:23:42.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:41 smithi134 ceph-mon[39278]: 4.2 scrub starts 2024-02-05T08:23:42.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:41 smithi134 ceph-mon[39278]: 4.2 scrub ok 2024-02-05T08:23:42.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:41 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4141841192' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:23:42.375 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:42.416 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:42.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:42.460 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:23:42.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:42.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:42.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:42.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:42.691 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:42.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:42.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:42.733 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 23 16 48615 48599 132.03 99 0.0142085 0.00756903 2024-02-05T08:23:42.792 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:42.833 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:42.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:42.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:42.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:42.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:42.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:42 smithi196 ceph-mon[33018]: 5.18 deep-scrub starts 2024-02-05T08:23:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:42 smithi196 ceph-mon[33018]: 5.18 deep-scrub ok 2024-02-05T08:23:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:42 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:23:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:42 smithi196 ceph-mon[33018]: pgmap v852: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-02-05T08:23:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:42 smithi196 ceph-mon[33018]: 5.17 deep-scrub starts 2024-02-05T08:23:43.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:42 smithi196 ceph-mon[33018]: 5.17 deep-scrub ok 2024-02-05T08:23:43.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:42 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:23:43.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:43.105 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:43.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:43.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:43.207 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:43.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T08:23:43.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[30086]: 5.18 deep-scrub starts 2024-02-05T08:23:43.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[30086]: 5.18 deep-scrub ok 2024-02-05T08:23:43.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:23:43.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[30086]: pgmap v852: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-02-05T08:23:43.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[30086]: 5.17 deep-scrub starts 2024-02-05T08:23:43.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[30086]: 5.17 deep-scrub ok 2024-02-05T08:23:43.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:23:43.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[39278]: 5.18 deep-scrub starts 2024-02-05T08:23:43.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[39278]: 5.18 deep-scrub ok 2024-02-05T08:23:43.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:23:43.221 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[39278]: pgmap v852: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.1 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-02-05T08:23:43.221 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[39278]: 5.17 deep-scrub starts 2024-02-05T08:23:43.221 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[39278]: 5.17 deep-scrub ok 2024-02-05T08:23:43.221 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:42 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:23:43.248 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:43.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:43.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:43.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:43.411 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:43.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:43.512 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:43.551 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:43.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:43.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:43.613 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:23:43.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:43.711 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:43.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:43.733 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 24 16 50215 50199 130.695 100 0.0102888 0.00764607 2024-02-05T08:23:43.812 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:43.848 INFO:tasks.thrashosds.thrasher:j is {'epoch': 381, 'fsid': '43e13aa4-c3fc-11ee-95b6-87774f69a715', 'created': '2024-02-05T07:59:09.321262+0000', 'modified': '2024-02-05T08:23:34.878264+0000', 'last_up_change': '2024-02-05T08:23:06.892657+0000', 'last_in_change': '2024-02-05T08:22:08.072805+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T08:02:21.034012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '303', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '301', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1700000762939453, 'score_stable': 2.1700000762939453, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T08:04:39.400746+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T08:04:40.005769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T08:04:41.851891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T08:04:43.793804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.2400000095367432, 'score_stable': 1.2400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.090000033378601, 'raw_score_stable': 1.090000033378601, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T08:04:45.853999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-02-05T08:23:07.682500+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '373', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'd90a5ea0-9a19-452e-9564-ca7b7a7c45e4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 324, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6802', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 2540271081}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6804', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 2540271081}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6808', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2540271081}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6806', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 2540271081}]}, 'public_addr': '172.21.15.134:6803/2540271081', 'cluster_addr': '172.21.15.134:6805/2540271081', 'heartbeat_back_addr': '172.21.15.134:6809/2540271081', 'heartbeat_front_addr': '172.21.15.134:6807/2540271081', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 323, 'up_from': 366, 'up_thru': 367, 'down_at': 324, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6810', 'nonce': 1259179009}, {'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 1259179009}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6812', 'nonce': 1259179009}, {'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 1259179009}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6816', 'nonce': 1259179009}, {'type': 'v1', 'addr': '172.21.15.134:6817', 'nonce': 1259179009}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6814', 'nonce': 1259179009}, {'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 1259179009}]}, 'public_addr': '172.21.15.134:6811/1259179009', 'cluster_addr': '172.21.15.134:6813/1259179009', 'heartbeat_back_addr': '172.21.15.134:6817/1259179009', 'heartbeat_front_addr': '172.21.15.134:6815/1259179009', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '078e578b-855b-49ff-83bd-0548a9c5a708', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 367, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6818', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6819', 'nonce': 153827987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6820', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6821', 'nonce': 153827987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 153827987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6822', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6823', 'nonce': 153827987}]}, 'public_addr': '172.21.15.134:6819/153827987', 'cluster_addr': '172.21.15.134:6821/153827987', 'heartbeat_back_addr': '172.21.15.134:6825/153827987', 'heartbeat_front_addr': '172.21.15.134:6823/153827987', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6774e330-3202-4671-894e-172cc2f6189d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 367, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 1527507056}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6828', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6829', 'nonce': 1527507056}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6832', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6833', 'nonce': 1527507056}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6830', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6831', 'nonce': 1527507056}]}, 'public_addr': '172.21.15.134:6827/1527507056', 'cluster_addr': '172.21.15.134:6829/1527507056', 'heartbeat_back_addr': '172.21.15.134:6833/1527507056', 'heartbeat_front_addr': '172.21.15.134:6831/1527507056', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '52f70cfa-ab0e-4751-a5f8-1045c61c1693', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 367, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3005156808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3005156808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3005156808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3005156808}]}, 'public_addr': '172.21.15.196:6801/3005156808', 'cluster_addr': '172.21.15.196:6803/3005156808', 'heartbeat_back_addr': '172.21.15.196:6807/3005156808', 'heartbeat_front_addr': '172.21.15.196:6805/3005156808', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b4b21769-d40f-43fd-b7e9-2477b74d3955', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 367, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 1603224792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 1603224792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 1603224792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 1603224792}]}, 'public_addr': '172.21.15.196:6809/1603224792', 'cluster_addr': '172.21.15.196:6811/1603224792', 'heartbeat_back_addr': '172.21.15.196:6815/1603224792', 'heartbeat_front_addr': '172.21.15.196:6813/1603224792', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'af1cced4-0015-47f5-b105-27641bfe4f4a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 367, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 3727413143}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 3727413143}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 3727413143}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 3727413143}]}, 'public_addr': '172.21.15.196:6817/3727413143', 'cluster_addr': '172.21.15.196:6819/3727413143', 'heartbeat_back_addr': '172.21.15.196:6823/3727413143', 'heartbeat_front_addr': '172.21.15.196:6821/3727413143', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c54beeae-9f7c-4723-a369-13bb531b4057', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 369, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 4206571010}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 4206571010}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 4206571010}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 4206571010}]}, 'public_addr': '172.21.15.196:6825/4206571010', 'cluster_addr': '172.21.15.196:6827/4206571010', 'heartbeat_back_addr': '172.21.15.196:6831/4206571010', 'heartbeat_front_addr': '172.21.15.196:6829/4206571010', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T08:23:25.118956+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:23:05.902681+0000', 'dead_epoch': 323}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:23:20.945217+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:23:22.070521+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:23:07.458558+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:23:19.891780+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:23:06.286495+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:22:51.716195+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '6.6', 'osds': [4, 1, 3]}], 'pg_upmap_items': [{'pgid': '3.6', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.134:0/788812699': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/1046500326': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/464208089': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/301941854': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:6800/3607168099': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6800/673876301': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/4225724311': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6800/1525062017': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:6801/3607168099': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6801/673876301': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/3200884805': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/2952444615': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:0/3325835814': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/3161126612': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:0/69097091': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:6801/1525062017': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/1645475987': '2024-02-06T08:05:21.465844+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T08:23:43.848 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:23:43.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:43.854 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:43.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:43.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:44.005 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:44.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:44.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:43 smithi196 ceph-mon[33018]: 3.f scrub starts 2024-02-05T08:23:44.019 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:43 smithi196 ceph-mon[33018]: 3.f scrub ok 2024-02-05T08:23:44.108 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:44.142 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:43 smithi134 ceph-mon[30086]: 3.f scrub starts 2024-02-05T08:23:44.142 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:43 smithi134 ceph-mon[30086]: 3.f scrub ok 2024-02-05T08:23:44.142 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:43 smithi134 ceph-mon[39278]: 3.f scrub starts 2024-02-05T08:23:44.142 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:43 smithi134 ceph-mon[39278]: 3.f scrub ok 2024-02-05T08:23:44.154 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:44.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:44.256 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:44.293 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:44.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:44.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:44.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:23:44 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:23:44.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:44.434 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:44.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:44.532 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:44.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:44.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:44.600 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:44.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:44.604 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:44.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:44.733 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 25 16 51750 51734 129.304 95.9375 0.00891156 0.0077282 2024-02-05T08:23:44.736 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:44.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:44.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:44.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:44.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:44.871 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:44.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:44.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:44.985 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[30086]: 1.1 scrub starts 2024-02-05T08:23:44.985 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[30086]: 1.1 scrub ok 2024-02-05T08:23:44.985 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[30086]: pgmap v853: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-05T08:23:44.986 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:44.986 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[30086]: 3.12 scrub starts 2024-02-05T08:23:44.986 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[30086]: 3.12 scrub ok 2024-02-05T08:23:44.986 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/52143207' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:23:44.986 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:23:44.986 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[39278]: 1.1 scrub starts 2024-02-05T08:23:44.986 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[39278]: 1.1 scrub ok 2024-02-05T08:23:44.986 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[39278]: pgmap v853: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-05T08:23:44.986 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:44.986 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[39278]: 3.12 scrub starts 2024-02-05T08:23:44.986 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[39278]: 3.12 scrub ok 2024-02-05T08:23:44.986 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/52143207' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:23:44.986 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:44 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:23:45.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:45.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:45.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:45.112 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:45.157 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:45.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:45.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:45.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:44 smithi196 ceph-mon[33018]: 1.1 scrub starts 2024-02-05T08:23:45.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:44 smithi196 ceph-mon[33018]: 1.1 scrub ok 2024-02-05T08:23:45.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:44 smithi196 ceph-mon[33018]: pgmap v853: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-05T08:23:45.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:44 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:45.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:44 smithi196 ceph-mon[33018]: 3.12 scrub starts 2024-02-05T08:23:45.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:44 smithi196 ceph-mon[33018]: 3.12 scrub ok 2024-02-05T08:23:45.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:44 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/52143207' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:23:45.262 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:44 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:23:45.300 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:45.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:45.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:45.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:45.441 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:45.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:45.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:45.582 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:45.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:45.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:45.727 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:45.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:45.733 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 26 16 53287 53271 128.025 96.0625 0.00849729 0.00780529 2024-02-05T08:23:45.828 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:45.866 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:23:45.866 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:23:45.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:23:45.866 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:23:45.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.866 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:27898 9.e 9:752e73bf:::benchmark_data_smithi134_144349_object27897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-02-05T08:23:45.866 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.030994+0000", 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 500.83183240099999, 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17265594500000001, 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 27898 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.030994+0000", 2024-02-05T08:23:45.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.030994+0000", 2024-02-05T08:23:45.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.030996+0000", 2024-02-05T08:23:45.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.643e-06 2024-02-05T08:23:45.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031038+0000", 2024-02-05T08:23:45.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2579e-05 2024-02-05T08:23:45.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031051+0000", 2024-02-05T08:23:45.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2975e-05 2024-02-05T08:23:45.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.869 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.869 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031057+0000", 2024-02-05T08:23:45.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8409999999999996e-06 2024-02-05T08:23:45.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031115+0000", 2024-02-05T08:23:45.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7326e-05 2024-02-05T08:23:45.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031214+0000", 2024-02-05T08:23:45.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9677000000000001e-05 2024-02-05T08:23:45.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.031358+0000", 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000143454 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032987+0000", 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001629098 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033183+0000", 2024-02-05T08:23:45.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019666400000000001 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.203561+0000", 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.170377946 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.203615+0000", 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3207000000000001e-05 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.203650+0000", 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5534999999999998e-05 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:27895 9.5 e249/239 9:a349bc50:::benchmark_data_smithi134_144349_object27894:head v 249'1711, mlcod=249'1707)", 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.032312+0000", 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 500.83051423900002, 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16973406999999999, 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032312+0000", 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032312+0000", 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032313+0000", 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5400000000000002e-07 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032354+0000", 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1622999999999998e-05 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032365+0000", 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0076e-05 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032371+0000", 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2360000000000001e-06 2024-02-05T08:23:45.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032515+0000", 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000144141 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032533+0000", 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8264e-05 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033555+0000", 2024-02-05T08:23:45.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001022226 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202046+0000", 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16849105 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:27899 9.9 e249/239 9:9d77ddc3:::benchmark_data_smithi134_144349_object27898:head v 249'1791, mlcod=249'1789)", 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.032390+0000", 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 500.83043673200001, 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.169799125, 2024-02-05T08:23:45.877 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.878 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.878 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032390+0000", 2024-02-05T08:23:45.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032390+0000", 2024-02-05T08:23:45.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032390+0000", 2024-02-05T08:23:45.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0600000000000001e-07 2024-02-05T08:23:45.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032423+0000", 2024-02-05T08:23:45.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2428999999999997e-05 2024-02-05T08:23:45.879 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.879 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.879 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032430+0000", 2024-02-05T08:23:45.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6629999999999993e-06 2024-02-05T08:23:45.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032436+0000", 2024-02-05T08:23:45.880 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.02e-06 2024-02-05T08:23:45.880 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.880 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.880 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.880 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032599+0000", 2024-02-05T08:23:45.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016209300000000001 2024-02-05T08:23:45.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.881 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.881 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.881 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032618+0000", 2024-02-05T08:23:45.881 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9505e-05 2024-02-05T08:23:45.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202151+0000", 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.169532718 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202189+0000", 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8291000000000003e-05 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.882 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.883 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:27902 9.d 9:b50a7d0d:::benchmark_data_smithi134_144349_object27901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-02-05T08:23:45.883 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.032502+0000", 2024-02-05T08:23:45.883 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 500.83032441199998, 2024-02-05T08:23:45.883 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17749103499999999, 2024-02-05T08:23:45.883 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.883 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.883 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.883 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.883 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.883 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 27902 2024-02-05T08:23:45.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.883 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.883 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032502+0000", 2024-02-05T08:23:45.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032502+0000", 2024-02-05T08:23:45.884 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.884 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.884 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.884 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032503+0000", 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0999999999999997e-07 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032531+0000", 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7786000000000001e-05 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032541+0000", 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0200999999999999e-05 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.885 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032548+0000", 2024-02-05T08:23:45.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9439999999999999e-06 2024-02-05T08:23:45.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032710+0000", 2024-02-05T08:23:45.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016212600000000001 2024-02-05T08:23:45.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.886 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.886 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.886 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032854+0000", 2024-02-05T08:23:45.886 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014396499999999999 2024-02-05T08:23:45.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-02-05T08:23:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.032987+0000", 2024-02-05T08:23:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000132447 2024-02-05T08:23:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202061+0000", 2024-02-05T08:23:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16907446300000001 2024-02-05T08:23:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.887 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202865+0000", 2024-02-05T08:23:45.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00080343400000000001 2024-02-05T08:23:45.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.209910+0000", 2024-02-05T08:23:45.888 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0070456900000000003 2024-02-05T08:23:45.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.888 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.888 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.888 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.209960+0000", 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0050999999999999e-05 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.209993+0000", 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3018000000000001e-05 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:27907 9.e 9:74a14f65:::benchmark_data_smithi134_144349_object27906:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e249)", 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.033274+0000", 2024-02-05T08:23:45.889 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 500.82955282799998, 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17237598000000001, 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 27907 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033274+0000", 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.890 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033274+0000", 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033275+0000", 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8199999999999998e-07 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033344+0000", 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9425999999999996e-05 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.891 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033355+0000", 2024-02-05T08:23:45.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0975000000000001e-05 2024-02-05T08:23:45.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033362+0000", 2024-02-05T08:23:45.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0539999999999997e-06 2024-02-05T08:23:45.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.892 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.892 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.892 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033400+0000", 2024-02-05T08:23:45.892 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7889000000000003e-05 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.201908+0000", 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.168507921 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202081+0000", 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017288399999999999 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.893 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.204290+0000", 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0022091649999999999 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.204499+0000", 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000208996 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205559+0000", 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010599380000000001 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.894 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205615+0000", 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5642e-05 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205650+0000", 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5207999999999999e-05 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:27903 9.8 e249/239 9:1ece6453:::benchmark_data_smithi134_144349_object27902:head v 249'1662, mlcod=249'1659)", 2024-02-05T08:23:45.895 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:25.033685+0000", 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 500.82914185200002, 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17151197500000001, 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033685+0000", 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033685+0000", 2024-02-05T08:23:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033685+0000", 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4999999999999998e-07 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.033745+0000", 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9487000000000002e-05 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.897 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.201990+0000", 2024-02-05T08:23:45.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16824555199999999 2024-02-05T08:23:45.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202001+0000", 2024-02-05T08:23:45.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0828e-05 2024-02-05T08:23:45.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.898 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.898 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202060+0000", 2024-02-05T08:23:45.898 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8529999999999997e-05 2024-02-05T08:23:45.898 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.898 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.202082+0000", 2024-02-05T08:23:45.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2340999999999998e-05 2024-02-05T08:23:45.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205157+0000", 2024-02-05T08:23:45.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030755219999999998 2024-02-05T08:23:45.899 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.899 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.899 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.899 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:25.205197+0000", 2024-02-05T08:23:45.899 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9264999999999998e-05 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25223.0:35467 9.2 e249/239 9:4b974889:::benchmark_data_smithi134_144349_object35466:head v 249'2223, mlcod=249'2220)", 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:29.630509+0000", 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 496.232317636, 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.154489985, 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.900 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.630509+0000", 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.630509+0000", 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.630509+0000", 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9699999999999996e-07 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.901 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.630570+0000", 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0089e-05 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.765054+0000", 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.134484564 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.765061+0000", 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8900000000000001e-06 2024-02-05T08:23:45.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.765153+0000", 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2193000000000003e-05 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.765228+0000", 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4952999999999999e-05 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.784968+0000", 2024-02-05T08:23:45.903 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019740159 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:29.784999+0000", 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0639999999999998e-05 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35875 9.e 9:7b26ffef:::benchmark_data_smithi134_144349_object35874:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.104276+0000", 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 495.75855109299999, 2024-02-05T08:23:45.904 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.19477491999999999, 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35875 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104276+0000", 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.905 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104276+0000", 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104277+0000", 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0690000000000001e-06 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104323+0000", 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5961999999999998e-05 2024-02-05T08:23:45.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104342+0000", 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9038e-05 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104350+0000", 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3189999999999995e-06 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104571+0000", 2024-02-05T08:23:45.907 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022128600000000001 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.104577+0000", 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4469999999999997e-06 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.112613+0000", 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0080362550000000008 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.112736+0000", 2024-02-05T08:23:45.908 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000123087 2024-02-05T08:23:45.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.909 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.909 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:23:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:45.910 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:45.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:45.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.112906+0000", 2024-02-05T08:23:45.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000169772 2024-02-05T08:23:45.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.117780+0000", 2024-02-05T08:23:45.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0048745769999999997 2024-02-05T08:23:45.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.911 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.911 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.911 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.117817+0000", 2024-02-05T08:23:45.911 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6451e-05 2024-02-05T08:23:45.911 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.298971+0000", 2024-02-05T08:23:45.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18115403999999999 2024-02-05T08:23:45.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.912 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299019+0000", 2024-02-05T08:23:45.912 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8164000000000002e-05 2024-02-05T08:23:45.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.912 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.912 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299050+0000", 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1452999999999999e-05 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35882 9.e 9:79e542b7:::benchmark_data_smithi134_144349_object35881:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.111606+0000", 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 495.75122096799998, 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18759089300000001, 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.913 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.914 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.914 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.914 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35882 2024-02-05T08:23:45.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.914 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111606+0000", 2024-02-05T08:23:45.914 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.914 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.914 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.914 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.914 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111606+0000", 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111607+0000", 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4810000000000001e-06 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111642+0000", 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4699000000000001e-05 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.915 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111652+0000", 2024-02-05T08:23:45.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0574000000000001e-05 2024-02-05T08:23:45.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.111660+0000", 2024-02-05T08:23:45.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2350000000000002e-06 2024-02-05T08:23:45.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.916 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.916 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.114371+0000", 2024-02-05T08:23:45.916 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027112450000000001 2024-02-05T08:23:45.916 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.916 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.114476+0000", 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105139 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.114563+0000", 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7253e-05 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.117828+0000", 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0032642650000000001 2024-02-05T08:23:45.917 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.118007+0000", 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017978199999999999 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299094+0000", 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18108685299999999 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299148+0000", 2024-02-05T08:23:45.918 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3956999999999999e-05 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299197+0000", 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8409999999999999e-05 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35887 9.e 9:724dd1b7:::benchmark_data_smithi134_144349_object35886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.113626+0000", 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 495.74920107999998, 2024-02-05T08:23:45.919 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.18572786199999999, 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35887 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113626+0000", 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.920 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113626+0000", 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113627+0000", 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2300000000000001e-06 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113653+0000", 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6208e-05 2024-02-05T08:23:45.921 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113664+0000", 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0523000000000001e-05 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.113671+0000", 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.4510000000000003e-06 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.116160+0000", 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024895170000000001 2024-02-05T08:23:45.922 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.116289+0000", 2024-02-05T08:23:45.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000128355 2024-02-05T08:23:45.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:23:45.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.116439+0000", 2024-02-05T08:23:45.923 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015028799999999999 2024-02-05T08:23:45.923 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.923 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.923 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.923 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.119562+0000", 2024-02-05T08:23:45.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0031226969999999998 2024-02-05T08:23:45.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.119795+0000", 2024-02-05T08:23:45.924 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000233551 2024-02-05T08:23:45.924 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.924 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.924 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.924 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299259+0000", 2024-02-05T08:23:45.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17946410800000001 2024-02-05T08:23:45.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299310+0000", 2024-02-05T08:23:45.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0439000000000003e-05 2024-02-05T08:23:45.925 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.925 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.925 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.925 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299353+0000", 2024-02-05T08:23:45.925 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3495000000000003e-05 2024-02-05T08:23:45.925 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.925 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.926 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.926 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.926 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.926 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35906 9.e 9:7cbe34f1:::benchmark_data_smithi134_144349_object35905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:23:45.926 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.125518+0000", 2024-02-05T08:23:45.926 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 495.73730900999999, 2024-02-05T08:23:45.926 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17395559999999999, 2024-02-05T08:23:45.926 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.926 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.926 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.926 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.926 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.926 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35906 2024-02-05T08:23:45.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.927 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125518+0000", 2024-02-05T08:23:45.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.927 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.927 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125518+0000", 2024-02-05T08:23:45.927 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.927 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.927 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125519+0000", 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3960000000000001e-06 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125570+0000", 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1084000000000001e-05 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125586+0000", 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5903999999999999e-05 2024-02-05T08:23:45.928 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125594+0000", 2024-02-05T08:23:45.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9899999999999997e-06 2024-02-05T08:23:45.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125651+0000", 2024-02-05T08:23:45.929 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7179000000000003e-05 2024-02-05T08:23:45.929 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.929 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.929 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.929 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125776+0000", 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012447200000000001 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.125916+0000", 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000140013 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.127316+0000", 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014006859999999999 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.930 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.127928+0000", 2024-02-05T08:23:45.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00061120399999999996 2024-02-05T08:23:45.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299393+0000", 2024-02-05T08:23:45.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17146588300000001 2024-02-05T08:23:45.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.931 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.931 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.931 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299440+0000", 2024-02-05T08:23:45.931 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6684999999999998e-05 2024-02-05T08:23:45.931 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299473+0000", 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3104e-05 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35911 9.e 9:7aaf23ea:::benchmark_data_smithi134_144349_object35910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.129684+0000", 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 495.73314305700001, 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16995434200000001, 2024-02-05T08:23:45.932 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35911 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129684+0000", 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.933 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129684+0000", 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129685+0000", 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8589999999999999e-06 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129749+0000", 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3529999999999997e-05 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.934 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129764+0000", 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4793999999999999e-05 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129772+0000", 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2109999999999998e-06 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129822+0000", 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9974e-05 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.935 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.129921+0000", 2024-02-05T08:23:45.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8887000000000004e-05 2024-02-05T08:23:45.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:23:45.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.130017+0000", 2024-02-05T08:23:45.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6364000000000001e-05 2024-02-05T08:23:45.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.936 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.936 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.936 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.131511+0000", 2024-02-05T08:23:45.936 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014936540000000001 2024-02-05T08:23:45.936 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.132132+0000", 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00062069600000000001 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299515+0000", 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.16738375999999999 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299604+0000", 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8363999999999996e-05 2024-02-05T08:23:45.937 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.938 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.938 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299638+0000", 2024-02-05T08:23:45.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4248999999999997e-05 2024-02-05T08:23:45.938 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.938 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.938 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.938 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.938 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.938 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35921 9.e 9:776a85fb:::benchmark_data_smithi134_144349_object35920:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:23:45.938 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.138253+0000", 2024-02-05T08:23:45.938 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 495.72457314000002, 2024-02-05T08:23:45.938 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.161497483, 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35921 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138253+0000", 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.939 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138253+0000", 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138255+0000", 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.576e-06 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138299+0000", 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.4409999999999997e-05 2024-02-05T08:23:45.940 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138310+0000", 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0794e-05 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138318+0000", 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7500000000000003e-06 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138376+0000", 2024-02-05T08:23:45.941 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8372999999999998e-05 2024-02-05T08:23:45.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.942 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.942 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138498+0000", 2024-02-05T08:23:45.942 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012176900000000001 2024-02-05T08:23:45.942 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.942 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:23:45.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.138626+0000", 2024-02-05T08:23:45.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012799 2024-02-05T08:23:45.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.139830+0000", 2024-02-05T08:23:45.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001204199 2024-02-05T08:23:45.943 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.943 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.943 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.943 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140009+0000", 2024-02-05T08:23:45.943 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017819299999999999 2024-02-05T08:23:45.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299676+0000", 2024-02-05T08:23:45.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.159667061 2024-02-05T08:23:45.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.944 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299722+0000", 2024-02-05T08:23:45.944 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6164e-05 2024-02-05T08:23:45.944 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.944 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.944 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299751+0000", 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9204e-05 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:35922 9.e 9:77e64ba8:::benchmark_data_smithi134_144349_object35921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:30.140433+0000", 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 495.72239366600002, 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15941660299999999, 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.945 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 35922 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140433+0000", 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140433+0000", 2024-02-05T08:23:45.946 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140434+0000", 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3099999999999996e-07 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140488+0000", 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4243000000000002e-05 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140498+0000", 2024-02-05T08:23:45.947 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0015e-05 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140503+0000", 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1390000000000003e-06 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140550+0000", 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6978999999999999e-05 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.948 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140649+0000", 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9208000000000003e-05 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.140739+0000", 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9967000000000001e-05 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.141731+0000", 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00099119300000000002 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.949 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.141745+0000", 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4671000000000001e-05 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299783+0000", 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15803773400000001 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299822+0000", 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9428999999999998e-05 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.950 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:30.299850+0000", 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7093999999999998e-05 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58423 9.d 9:b730c3b2:::benchmark_data_smithi134_144349_object58422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.759126+0000", 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 481.103700592, 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14701034599999999, 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.951 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.952 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.952 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.952 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58423 2024-02-05T08:23:45.952 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.952 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.952 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.953 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:23:45.953 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.953 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759126+0000", 2024-02-05T08:23:45.953 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.953 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.953 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759126+0000", 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759128+0000", 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.601e-06 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759180+0000", 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2172999999999997e-05 2024-02-05T08:23:45.954 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759194+0000", 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4131000000000001e-05 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759368+0000", 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017358299999999999 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.759575+0000", 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0002073 2024-02-05T08:23:45.955 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763082+0000", 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003507058 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.766845+0000", 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003762933 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776323+0000", 2024-02-05T08:23:45.956 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0094780709999999994 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776344+0000", 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0774000000000001e-05 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.906047+0000", 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.129703771 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.957 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.906098+0000", 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0495999999999999e-05 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.906136+0000", 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8454999999999997e-05 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58429 9.e 9:74deef27:::benchmark_data_smithi134_144349_object58428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.763196+0000", 2024-02-05T08:23:45.958 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 481.09963051599999, 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14953992599999999, 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58429 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763196+0000", 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.959 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763196+0000", 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763198+0000", 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5960000000000001e-06 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763237+0000", 2024-02-05T08:23:45.960 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9320999999999998e-05 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763247+0000", 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6909999999999992e-06 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.763254+0000", 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.8580000000000002e-06 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.766437+0000", 2024-02-05T08:23:45.961 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003183909 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.766559+0000", 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000121561 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.766696+0000", 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013721 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.778872+0000", 2024-02-05T08:23:45.962 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012176200999999999 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.784715+0000", 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0058425609999999996 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912622+0000", 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.127907411 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.963 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912685+0000", 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2501000000000003e-05 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912736+0000", 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1106000000000002e-05 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58435 9.e 9:7bef458b:::benchmark_data_smithi134_144349_object58434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.765269+0000", 2024-02-05T08:23:45.964 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 481.09755728900001, 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14758520999999999, 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58435 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765269+0000", 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.965 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765269+0000", 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765271+0000", 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.638e-06 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765301+0000", 2024-02-05T08:23:45.966 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0369e-05 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765313+0000", 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1481000000000001e-05 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.765321+0000", 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6720000000000004e-06 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.967 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776013+0000", 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010692657 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776149+0000", 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000135909 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.776445+0000", 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00029550300000000002 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.968 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.778879+0000", 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024342790000000001 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.784792+0000", 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059131610000000001 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912788+0000", 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12799622299999999 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.969 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912832+0000", 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3402000000000002e-05 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.912855+0000", 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2915999999999999e-05 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25223.0:58444 9.3 9:c881298d:::benchmark_data_smithi134_144349_object58443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e257)", 2024-02-05T08:23:45.970 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:15:44.777171+0000", 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 481.08565577000002, 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.14967462100000001, 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25223", 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 58444 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777171+0000", 2024-02-05T08:23:45.971 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777171+0000", 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777172+0000", 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.155e-06 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.972 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777202+0000", 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0267e-05 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777211+0000", 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9730000000000003e-06 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.777218+0000", 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2940000000000002e-06 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.973 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.779461+0000", 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002243423 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.779586+0000", 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000125325 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,5", 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.779714+0000", 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012767599999999999 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.974 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.785089+0000", 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0053749059999999996 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.787028+0000", 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0019386290000000001 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.925411+0000", 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13838302199999999 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.975 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:23:45.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.926806+0000", 2024-02-05T08:23:45.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013959249999999999 2024-02-05T08:23:45.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.976 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.976 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:15:44.926845+0000", 2024-02-05T08:23:45.976 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9026000000000002e-05 2024-02-05T08:23:45.976 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.976 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.976 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.976 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.976 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.976 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15714.0:225861 10.c 10:3bf43d57:::benchmark_data_smithi134_174861_object94399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:22:08.077029+0000", 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 97.785797290000005, 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.163278091, 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15714", 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225861 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077029+0000", 2024-02-05T08:23:45.977 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077029+0000", 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077031+0000", 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6080000000000001e-06 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.978 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077038+0000", 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1720000000000001e-06 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077057+0000", 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8802000000000001e-05 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077066+0000", 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4660000000000005e-06 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.979 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077142+0000", 2024-02-05T08:23:45.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5902000000000006e-05 2024-02-05T08:23:45.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077281+0000", 2024-02-05T08:23:45.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000138897 2024-02-05T08:23:45.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.980 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.980 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-02-05T08:23:45.980 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077415+0000", 2024-02-05T08:23:45.980 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013382800000000001 2024-02-05T08:23:45.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.083926+0000", 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065109119999999998 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.098019+0000", 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014093022 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.098035+0000", 2024-02-05T08:23:45.981 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5625e-05 2024-02-05T08:23:45.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.093969+0000", 2024-02-05T08:23:45.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99593420600000004 2024-02-05T08:23:45.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:23:45.982 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.093985+0000", 2024-02-05T08:23:45.982 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6659000000000001e-05 2024-02-05T08:23:45.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.982 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.982 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.240212+0000", 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.146226985 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.240307+0000", 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5006999999999993e-05 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24388.0:1033 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e351)", 2024-02-05T08:23:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:22:08.178524+0000", 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 97.684302263000006, 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.926063401, 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24388", 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/2822725514", 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1033 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178524+0000", 2024-02-05T08:23:45.984 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178524+0000", 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178526+0000", 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.978e-06 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:23:45.985 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178538+0000", 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2140000000000001e-05 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178555+0000", 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6260999999999999e-05 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178564+0000", 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2550000000000003e-06 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.986 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178645+0000", 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1417999999999999e-05 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178654+0000", 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9819999999999997e-06 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.094408+0000", 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.91575361200000005 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:23:45.987 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.094417+0000", 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2059999999999996e-06 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.104208+0000", 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.009791187 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.104367+0000", 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000158846 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:23:45.988 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:23:45.989 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.104588+0000", 2024-02-05T08:23:45.989 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000220516 2024-02-05T08:23:45.989 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.989 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.989 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.989 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:23:45.989 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:23:45.989 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:23:46.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:46.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:45 smithi134 ceph-mon[30086]: 6.7 scrub starts 2024-02-05T08:23:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:45 smithi134 ceph-mon[30086]: 6.7 scrub ok 2024-02-05T08:23:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:45 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:23:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:45 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:23:46.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:45 smithi134 ceph-mon[30086]: osdmap e382: 8 total, 8 up, 7 in 2024-02-05T08:23:46.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:45 smithi134 ceph-mon[39278]: 6.7 scrub starts 2024-02-05T08:23:46.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:45 smithi134 ceph-mon[39278]: 6.7 scrub ok 2024-02-05T08:23:46.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:45 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:23:46.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:45 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:23:46.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:45 smithi134 ceph-mon[39278]: osdmap e382: 8 total, 8 up, 7 in 2024-02-05T08:23:46.052 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:46.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:46.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:45 smithi196 ceph-mon[33018]: 6.7 scrub starts 2024-02-05T08:23:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:45 smithi196 ceph-mon[33018]: 6.7 scrub ok 2024-02-05T08:23:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:45 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:23:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:45 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:23:46.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:45 smithi196 ceph-mon[33018]: osdmap e382: 8 total, 8 up, 7 in 2024-02-05T08:23:46.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:46.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:46.195 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:46.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:46.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:46.335 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:46.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:46.436 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:46.472 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:46.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:46.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:46.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:46.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:46.619 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:46.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:46.720 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:46.733 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 27 16 54853 54837 126.908 97.875 0.00913304 0.0078748 2024-02-05T08:23:46.757 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:46.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:46.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:46.894 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:46.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:46.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:47.035 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:47.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:47.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:47.179 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:47.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:47.191 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:46 smithi134 ceph-mon[30086]: pgmap v855: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-05T08:23:47.192 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:46 smithi134 ceph-mon[39278]: pgmap v855: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-05T08:23:47.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:47.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:46 smithi196 ceph-mon[33018]: pgmap v855: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-05T08:23:47.318 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:47.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:47.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:47.420 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:47.462 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:47.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:47.564 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:47.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:47.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:47.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:47.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:47.734 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 28 16 56467 56451 125.979 100.875 0.0164075 0.00793249 2024-02-05T08:23:47.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:47.742 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:47.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:47.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:47.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:47.882 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:47.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:47.971 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T08:23:47.983 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:48.005 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:23:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:23:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:23:48.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:48.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:48.122 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:48.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:48.169 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:48.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:48.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:48.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:48.311 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:48.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:48.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:48.453 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:48.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:48.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:48.599 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:48.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:48.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:48.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:48.733 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 29 16 57967 57951 124.867 93.75 0.00434364 0.0080018 2024-02-05T08:23:48.735 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:48.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:48.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:48.837 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:48.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:48.876 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:48.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:48.934 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:23:48.978 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:48.986 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:23:49.014 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:49.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:49.026 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:23:49.067 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:23:49.110 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:23:49.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:49.145 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:49.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:49.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:49.149 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:23:49.193 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:48 smithi134 ceph-mon[30086]: pgmap v856: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-05T08:23:49.193 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:48 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1025538091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:49.193 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:23:49.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:48 smithi134 ceph-mon[39278]: pgmap v856: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-05T08:23:49.194 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:48 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1025538091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:49.205 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:49.242 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:23:49.245 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:23:49.245 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:23:49.246 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:23:49.246 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:23:49.246 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:23:49.246 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:23:49.246 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:23:49.246 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:23:49.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:49.249 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:48 smithi196 ceph-mon[33018]: pgmap v856: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.59k op/s 2024-02-05T08:23:49.250 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:48 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1025538091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:23:49.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:49.292 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:49.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:49.393 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:49.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:49.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:49.529 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:23:49.533 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:23:49.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:49.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:49.618 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:49.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:49.719 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:49.734 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 30 15 59309 59294 123.5 83.9375 0.0137364 0.00809226 2024-02-05T08:23:49.758 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:49.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:49.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:49.790 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:23:49.860 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:49.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:49.903 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:49.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:50.005 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:50.065 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:50.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:50.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:50.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:50.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:50.211 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:50.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:50.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:49 smithi134 ceph-mon[30086]: from='client.16050 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:23:50.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:49 smithi134 ceph-mon[30086]: osdmap e383: 8 total, 8 up, 7 in 2024-02-05T08:23:50.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:49 smithi134 ceph-mon[39278]: from='client.16050 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:23:50.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:49 smithi134 ceph-mon[39278]: osdmap e383: 8 total, 8 up, 7 in 2024-02-05T08:23:50.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:49 smithi196 ceph-mon[33018]: from='client.16050 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:23:50.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:49 smithi196 ceph-mon[33018]: osdmap e383: 8 total, 8 up, 7 in 2024-02-05T08:23:50.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:50.336 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:50.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:50.437 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:50.474 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:50.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:50.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:50.575 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:50.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:50.600 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:50.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:50.701 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:50.724 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:50.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:50.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:50.734 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 31 16 60495 60479 121.906 74.0625 0.00312391 0.00819051 2024-02-05T08:23:50.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:50.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:50.867 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:50.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:50.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:51.005 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:51.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:51.107 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:51.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:51.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:51.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:51.270 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:51.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:51.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:50 smithi134 ceph-mon[30086]: pgmap v858: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-02-05T08:23:51.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:50 smithi134 ceph-mon[39278]: pgmap v858: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-02-05T08:23:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:50 smithi196 ceph-mon[33018]: pgmap v858: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-02-05T08:23:51.372 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:51.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:51.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:51.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:51.516 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:51.553 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:51.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:51.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:51.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:51.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:51.734 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 32 16 61867 61851 120.776 85.75 0.00503809 0.00827451 2024-02-05T08:23:51.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:51.817 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:51.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:51.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:51.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:51.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:51.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:51.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:52.055 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:52.092 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:52.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:52.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:52.194 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:52.228 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:52.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:52.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:52.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:52.366 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:52.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:52.468 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:52.491 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:52.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:52.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:52.593 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:52.636 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:52.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:52.734 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 33 15 63287 63272 119.807 88.8125 0.00889943 0.00834232 2024-02-05T08:23:52.738 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:52.774 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:52.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:52.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:52.914 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:52.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:53.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:53.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:53.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:53.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:53.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:53.199 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:53.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:53.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:53.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:52 smithi134 ceph-mon[30086]: pgmap v859: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T08:23:53.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:52 smithi134 ceph-mon[39278]: pgmap v859: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T08:23:53.301 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:52 smithi196 ceph-mon[33018]: pgmap v859: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T08:23:53.339 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:53.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:53.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:53.441 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:53.477 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:53.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:53.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:53.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:53.612 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:53.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:53.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:53.714 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:53.734 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 34 15 64596 64581 118.689 81.8125 0.105468 0.00841928 2024-02-05T08:23:53.748 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:53.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:53.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:53.891 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:53.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:53.993 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:54.034 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:54.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:54.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:54.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:54.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:54.163 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:54.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:54.265 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:54.303 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:54.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:54.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:23:54 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:23:54.405 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:54.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:54.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:54.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:54.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:54.582 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:54.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:54.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:54.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:54.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:54.735 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 35 16 65985 65969 117.772 86.75 0.0252398 0.00848587 2024-02-05T08:23:54.825 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:54.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:54.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:54.942 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:54.943 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:23:54.943 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:54 smithi134 ceph-mon[30086]: pgmap v860: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-02-05T08:23:54.943 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:54 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:54.943 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:54.943 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:23:54.943 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:54 smithi134 ceph-mon[39278]: pgmap v860: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-02-05T08:23:54.943 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:54 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:54.944 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":856,"stamp":"2024-02-05T08:23:47.902382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3312523612,"num_objects":50734,"num_object_clones":0,"num_object_copies":152202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50734,"num_whiteouts":0,"num_read":2255,"num_read_kb":2007,"num_write":51258,"num_write_kb":3234349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":353,"num_bytes_recovered":8947,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52857,"ondisk_log_size":52857,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":477,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13926916,"kb_used_data":10077064,"kb_used_omap":13,"kb_used_meta":3849778,"kb_avail":642289148,"statfs":{"total":671965249536,"available":657704087552,"internally_reserved":0,"allocated":10318913536,"data_stored":10315812361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14001,"internal_metadata":3942173007},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,7,6,8,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1041629184,"num_objects":15894,"num_object_clones":0,"num_object_copies":47682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15894,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":15894,"num_write_kb":1017216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003306"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":700,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.897690+0000","last_change":"2024-02-05T08:23:07.931957+0000","last_active":"2024-02-05T08:23:45.897690+0000","last_peered":"2024-02-05T08:23:45.897690+0000","last_clean":"2024-02-05T08:23:45.897690+0000","last_became_active":"2024-02-05T08:23:07.931642+0000","last_became_peered":"2024-02-05T08:23:07.931642+0000","last_unstale":"2024-02-05T08:23:45.897690+0000","last_undegraded":"2024-02-05T08:23:45.897690+0000","last_fullsized":"2024-02-05T08:23:45.897690+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:36.296981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:36.296981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":568,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.903032+0000","last_change":"2024-02-05T08:23:32.176306+0000","last_active":"2024-02-05T08:23:45.903032+0000","last_peered":"2024-02-05T08:23:45.903032+0000","last_clean":"2024-02-05T08:23:45.903032+0000","last_became_active":"2024-02-05T08:23:07.938772+0000","last_became_peered":"2024-02-05T08:23:07.938772+0000","last_unstale":"2024-02-05T08:23:45.903032+0000","last_undegraded":"2024-02-05T08:23:45.903032+0000","last_fullsized":"2024-02-05T08:23:45.903032+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:32.176199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:23:32.176199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:39.232013+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":76,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.896955+0000","last_change":"2024-02-05T08:22:54.215968+0000","last_active":"2024-02-05T08:23:34.896955+0000","last_peered":"2024-02-05T08:23:34.896955+0000","last_clean":"2024-02-05T08:23:34.896955+0000","last_became_active":"2024-02-05T08:22:09.101941+0000","last_became_peered":"2024-02-05T08:22:09.101941+0000","last_unstale":"2024-02-05T08:23:34.896955+0000","last_undegraded":"2024-02-05T08:23:34.896955+0000","last_fullsized":"2024-02-05T08:23:34.896955+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:54.215846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:22:54.215846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:17.997283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"236'19","reported_seq":769,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.895639+0000","last_change":"2024-02-05T08:23:31.459738+0000","last_active":"2024-02-05T08:23:45.895639+0000","last_peered":"2024-02-05T08:23:45.895639+0000","last_clean":"2024-02-05T08:23:45.895639+0000","last_became_active":"2024-02-05T08:23:07.937041+0000","last_became_peered":"2024-02-05T08:23:07.937041+0000","last_unstale":"2024-02-05T08:23:45.895639+0000","last_undegraded":"2024-02-05T08:23:45.895639+0000","last_fullsized":"2024-02-05T08:23:45.895639+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:23:31.459611+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:23:31.459611+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:49.340063+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":209,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899432+0000","last_change":"2024-02-05T08:23:24.076782+0000","last_active":"2024-02-05T08:23:34.899432+0000","last_peered":"2024-02-05T08:23:34.899432+0000","last_clean":"2024-02-05T08:23:34.899432+0000","last_became_active":"2024-02-05T08:18:34.633083+0000","last_became_peered":"2024-02-05T08:18:34.633083+0000","last_unstale":"2024-02-05T08:23:34.899432+0000","last_undegraded":"2024-02-05T08:23:34.899432+0000","last_fullsized":"2024-02-05T08:23:34.899432+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:24.076590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:23:24.076590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:41.261868+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":716,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.895828+0000","last_change":"2024-02-05T08:23:41.460609+0000","last_active":"2024-02-05T08:23:45.895828+0000","last_peered":"2024-02-05T08:23:45.895828+0000","last_clean":"2024-02-05T08:23:45.895828+0000","last_became_active":"2024-02-05T08:23:07.937180+0000","last_became_peered":"2024-02-05T08:23:07.937180+0000","last_unstale":"2024-02-05T08:23:45.895828+0000","last_undegraded":"2024-02-05T08:23:45.895828+0000","last_fullsized":"2024-02-05T08:23:45.895828+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:41.460468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:41.460468+0000","last_clean_scrub_stamp":"2024-02-05T08:23:41.460468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:46.860369+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":210,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899944+0000","last_change":"2024-02-05T08:21:45.823410+0000","last_active":"2024-02-05T08:23:34.899944+0000","last_peered":"2024-02-05T08:23:34.899944+0000","last_clean":"2024-02-05T08:23:34.899944+0000","last_became_active":"2024-02-05T08:21:41.880038+0000","last_became_peered":"2024-02-05T08:21:41.880038+0000","last_unstale":"2024-02-05T08:23:34.899944+0000","last_undegraded":"2024-02-05T08:23:34.899944+0000","last_fullsized":"2024-02-05T08:23:34.899944+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:45.823300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:21:45.823300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"236'34","reported_seq":247,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.902377+0000","last_change":"2024-02-05T08:22:09.119504+0000","last_active":"2024-02-05T08:23:34.902377+0000","last_peered":"2024-02-05T08:23:34.902377+0000","last_clean":"2024-02-05T08:23:34.902377+0000","last_became_active":"2024-02-05T08:22:09.095896+0000","last_became_peered":"2024-02-05T08:22:09.095896+0000","last_unstale":"2024-02-05T08:23:34.902377+0000","last_undegraded":"2024-02-05T08:23:34.902377+0000","last_fullsized":"2024-02-05T08:23:34.902377+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:22:04.719477+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:22:04.719477+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":41,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.904058+0000","last_change":"2024-02-05T08:23:08.917569+0000","last_active":"2024-02-05T08:23:45.904058+0000","last_peered":"2024-02-05T08:23:45.904058+0000","last_clean":"2024-02-05T08:23:45.904058+0000","last_became_active":"2024-02-05T08:23:08.917199+0000","last_became_peered":"2024-02-05T08:23:08.917199+0000","last_unstale":"2024-02-05T08:23:45.904058+0000","last_undegraded":"2024-02-05T08:23:45.904058+0000","last_fullsized":"2024-02-05T08:23:45.904058+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:44.814367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:44.814367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:57.379064+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":715,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.895556+0000","last_change":"2024-02-05T08:23:02.193023+0000","last_active":"2024-02-05T08:23:34.895556+0000","last_peered":"2024-02-05T08:23:34.895556+0000","last_clean":"2024-02-05T08:23:34.895556+0000","last_became_active":"2024-02-05T08:22:09.108745+0000","last_became_peered":"2024-02-05T08:22:09.108745+0000","last_unstale":"2024-02-05T08:23:34.895556+0000","last_undegraded":"2024-02-05T08:23:34.895556+0000","last_fullsized":"2024-02-05T08:23:34.895556+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:02.192889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:02.192889+0000","last_clean_scrub_stamp":"2024-02-05T08:23:02.192889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:03.238021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":737,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:40.814061+0000","last_change":"2024-02-05T08:23:40.814097+0000","last_active":"2024-02-05T08:23:40.814061+0000","last_peered":"2024-02-05T08:23:40.814061+0000","last_clean":"2024-02-05T08:23:40.814061+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:23:40.814061+0000","last_undegraded":"2024-02-05T08:23:40.814061+0000","last_fullsized":"2024-02-05T08:23:40.814061+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:40.813940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_clean_scrub_stamp":"2024-02-05T08:23:40.813940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:00.487871+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"263'19","reported_seq":790,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.897828+0000","last_change":"2024-02-05T08:22:09.118822+0000","last_active":"2024-02-05T08:23:45.897828+0000","last_peered":"2024-02-05T08:23:45.897828+0000","last_clean":"2024-02-05T08:23:45.897828+0000","last_became_active":"2024-02-05T08:22:09.102371+0000","last_became_peered":"2024-02-05T08:22:09.102371+0000","last_unstale":"2024-02-05T08:23:45.897828+0000","last_undegraded":"2024-02-05T08:23:45.897828+0000","last_fullsized":"2024-02-05T08:23:45.897828+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:21:43.861060+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:21:43.861060+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":710,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.895401+0000","last_change":"2024-02-05T08:23:25.177775+0000","last_active":"2024-02-05T08:23:34.895401+0000","last_peered":"2024-02-05T08:23:34.895401+0000","last_clean":"2024-02-05T08:23:34.895401+0000","last_became_active":"2024-02-05T08:23:07.932548+0000","last_became_peered":"2024-02-05T08:23:07.932548+0000","last_unstale":"2024-02-05T08:23:34.895401+0000","last_undegraded":"2024-02-05T08:23:34.895401+0000","last_fullsized":"2024-02-05T08:23:34.895401+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:25.177650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:25.177650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:44.333065+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":711,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899415+0000","last_change":"2024-02-05T08:23:26.986054+0000","last_active":"2024-02-05T08:23:34.899415+0000","last_peered":"2024-02-05T08:23:34.899415+0000","last_clean":"2024-02-05T08:23:34.899415+0000","last_became_active":"2024-02-05T08:23:07.939630+0000","last_became_peered":"2024-02-05T08:23:07.939630+0000","last_unstale":"2024-02-05T08:23:34.899415+0000","last_undegraded":"2024-02-05T08:23:34.899415+0000","last_fullsized":"2024-02-05T08:23:34.899415+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:26.985938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:23:26.985938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:35.576590+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":718,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.895718+0000","last_change":"2024-02-05T08:23:29.002501+0000","last_active":"2024-02-05T08:23:34.895718+0000","last_peered":"2024-02-05T08:23:34.895718+0000","last_clean":"2024-02-05T08:23:34.895718+0000","last_became_active":"2024-02-05T08:23:07.932530+0000","last_became_peered":"2024-02-05T08:23:07.932530+0000","last_unstale":"2024-02-05T08:23:34.895718+0000","last_undegraded":"2024-02-05T08:23:34.895718+0000","last_fullsized":"2024-02-05T08:23:34.895718+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:29.002380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.002380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:36.585001+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":745,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.899576+0000","last_change":"2024-02-05T08:23:27.578462+0000","last_active":"2024-02-05T08:23:45.899576+0000","last_peered":"2024-02-05T08:23:45.899576+0000","last_clean":"2024-02-05T08:23:45.899576+0000","last_became_active":"2024-02-05T08:23:07.938087+0000","last_became_peered":"2024-02-05T08:23:07.938087+0000","last_unstale":"2024-02-05T08:23:45.899576+0000","last_undegraded":"2024-02-05T08:23:45.899576+0000","last_fullsized":"2024-02-05T08:23:45.899576+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-02-05T08:23:27.578370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:27.578370+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:47.356834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":713,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.899436+0000","last_change":"2024-02-05T08:21:42.867663+0000","last_active":"2024-02-05T08:23:45.899436+0000","last_peered":"2024-02-05T08:23:45.899436+0000","last_clean":"2024-02-05T08:23:45.899436+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:23:45.899436+0000","last_undegraded":"2024-02-05T08:23:45.899436+0000","last_fullsized":"2024-02-05T08:23:45.899436+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:42.867555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:21:42.867555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":609,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.904145+0000","last_change":"2024-02-05T08:23:08.916098+0000","last_active":"2024-02-05T08:23:45.904145+0000","last_peered":"2024-02-05T08:23:45.904145+0000","last_clean":"2024-02-05T08:23:45.904145+0000","last_became_active":"2024-02-05T08:23:08.914258+0000","last_became_peered":"2024-02-05T08:23:08.914258+0000","last_unstale":"2024-02-05T08:23:45.904145+0000","last_undegraded":"2024-02-05T08:23:45.904145+0000","last_fullsized":"2024-02-05T08:23:45.904145+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:56.260200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_clean_scrub_stamp":"2024-02-05T08:22:56.260200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:08.741648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":74,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.899481+0000","last_change":"2024-02-05T08:23:08.921340+0000","last_active":"2024-02-05T08:23:45.899481+0000","last_peered":"2024-02-05T08:23:45.899481+0000","last_clean":"2024-02-05T08:23:45.899481+0000","last_became_active":"2024-02-05T08:23:08.920902+0000","last_became_peered":"2024-02-05T08:23:08.920902+0000","last_unstale":"2024-02-05T08:23:45.899481+0000","last_undegraded":"2024-02-05T08:23:45.899481+0000","last_fullsized":"2024-02-05T08:23:45.899481+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:04.290125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:04.290125+0000","last_clean_scrub_stamp":"2024-02-05T08:22:04.290125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"236'34","reported_seq":824,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.892842+0000","last_change":"2024-02-05T08:23:29.912005+0000","last_active":"2024-02-05T08:23:34.892842+0000","last_peered":"2024-02-05T08:23:34.892842+0000","last_clean":"2024-02-05T08:23:34.892842+0000","last_became_active":"2024-02-05T08:23:07.930731+0000","last_became_peered":"2024-02-05T08:23:07.930731+0000","last_unstale":"2024-02-05T08:23:34.892842+0000","last_undegraded":"2024-02-05T08:23:34.892842+0000","last_fullsized":"2024-02-05T08:23:34.892842+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:23:29.911720+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.911720+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:59.611514+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":201,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.902489+0000","last_change":"2024-02-05T08:21:59.021213+0000","last_active":"2024-02-05T08:23:34.902489+0000","last_peered":"2024-02-05T08:23:34.902489+0000","last_clean":"2024-02-05T08:23:34.902489+0000","last_became_active":"2024-02-05T08:18:34.642494+0000","last_became_peered":"2024-02-05T08:18:34.642494+0000","last_unstale":"2024-02-05T08:23:34.902489+0000","last_undegraded":"2024-02-05T08:23:34.902489+0000","last_fullsized":"2024-02-05T08:23:34.902489+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:59.021074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_clean_scrub_stamp":"2024-02-05T08:21:59.021074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":723,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.899161+0000","last_change":"2024-02-05T08:22:57.230710+0000","last_active":"2024-02-05T08:23:45.899161+0000","last_peered":"2024-02-05T08:23:45.899161+0000","last_clean":"2024-02-05T08:23:45.899161+0000","last_became_active":"2024-02-05T08:18:34.638652+0000","last_became_peered":"2024-02-05T08:18:34.638652+0000","last_unstale":"2024-02-05T08:23:45.899161+0000","last_undegraded":"2024-02-05T08:23:45.899161+0000","last_fullsized":"2024-02-05T08:23:45.899161+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:57.230639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:22:57.230639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:06.998438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":704,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.897861+0000","last_change":"2024-02-05T08:23:07.938371+0000","last_active":"2024-02-05T08:23:45.897861+0000","last_peered":"2024-02-05T08:23:45.897861+0000","last_clean":"2024-02-05T08:23:45.897861+0000","last_became_active":"2024-02-05T08:23:07.937958+0000","last_became_peered":"2024-02-05T08:23:07.937958+0000","last_unstale":"2024-02-05T08:23:45.897861+0000","last_undegraded":"2024-02-05T08:23:45.897861+0000","last_fullsized":"2024-02-05T08:23:45.897861+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:19.338699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:19:19.338699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"236'23","reported_seq":214,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899550+0000","last_change":"2024-02-05T08:23:25.937497+0000","last_active":"2024-02-05T08:23:34.899550+0000","last_peered":"2024-02-05T08:23:34.899550+0000","last_clean":"2024-02-05T08:23:34.899550+0000","last_became_active":"2024-02-05T08:23:07.925083+0000","last_became_peered":"2024-02-05T08:23:07.925083+0000","last_unstale":"2024-02-05T08:23:34.899550+0000","last_undegraded":"2024-02-05T08:23:34.899550+0000","last_fullsized":"2024-02-05T08:23:34.899550+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'23","last_scrub_stamp":"2024-02-05T08:23:25.935815+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:23:25.935815+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:29.899652+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":716,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:40.021989+0000","last_change":"2024-02-05T08:23:40.022017+0000","last_active":"2024-02-05T08:23:40.021989+0000","last_peered":"2024-02-05T08:23:40.021989+0000","last_clean":"2024-02-05T08:23:40.021989+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:23:40.021989+0000","last_undegraded":"2024-02-05T08:23:40.021989+0000","last_fullsized":"2024-02-05T08:23:40.021989+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:23:40.021926+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:23:40.021926+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:09.922432+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":595,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:36.810657+0000","last_change":"2024-02-05T08:22:59.503306+0000","last_active":"2024-02-05T08:23:36.810657+0000","last_peered":"2024-02-05T08:23:36.810657+0000","last_clean":"2024-02-05T08:23:36.810657+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:23:36.810657+0000","last_undegraded":"2024-02-05T08:23:36.810657+0000","last_fullsized":"2024-02-05T08:23:36.810657+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:22:59.503185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:59.503185+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:01.636046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":638,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.903205+0000","last_change":"2024-02-05T08:23:07.940716+0000","last_active":"2024-02-05T08:23:45.903205+0000","last_peered":"2024-02-05T08:23:45.903205+0000","last_clean":"2024-02-05T08:23:45.903205+0000","last_became_active":"2024-02-05T08:23:07.940135+0000","last_became_peered":"2024-02-05T08:23:07.940135+0000","last_unstale":"2024-02-05T08:23:45.903205+0000","last_undegraded":"2024-02-05T08:23:45.903205+0000","last_fullsized":"2024-02-05T08:23:45.903205+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:19:28.743054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:19:28.743054+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"236'20","reported_seq":538,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.898767+0000","last_change":"2024-02-05T08:23:08.935899+0000","last_active":"2024-02-05T08:23:34.898767+0000","last_peered":"2024-02-05T08:23:34.898767+0000","last_clean":"2024-02-05T08:23:34.898767+0000","last_became_active":"2024-02-05T08:23:08.920330+0000","last_became_peered":"2024-02-05T08:23:08.920330+0000","last_unstale":"2024-02-05T08:23:34.898767+0000","last_undegraded":"2024-02-05T08:23:34.898767+0000","last_fullsized":"2024-02-05T08:23:34.898767+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:21:49.240369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:49.240369+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":572,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.904505+0000","last_change":"2024-02-05T08:23:33.200652+0000","last_active":"2024-02-05T08:23:45.904505+0000","last_peered":"2024-02-05T08:23:45.904505+0000","last_clean":"2024-02-05T08:23:45.904505+0000","last_became_active":"2024-02-05T08:23:07.941095+0000","last_became_peered":"2024-02-05T08:23:07.941095+0000","last_unstale":"2024-02-05T08:23:45.904505+0000","last_undegraded":"2024-02-05T08:23:45.904505+0000","last_fullsized":"2024-02-05T08:23:45.904505+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:33.200557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:33.200557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:36.583846+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":72,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899071+0000","last_change":"2024-02-05T08:23:08.911460+0000","last_active":"2024-02-05T08:23:34.899071+0000","last_peered":"2024-02-05T08:23:34.899071+0000","last_clean":"2024-02-05T08:23:34.899071+0000","last_became_active":"2024-02-05T08:23:08.911020+0000","last_became_peered":"2024-02-05T08:23:08.911020+0000","last_unstale":"2024-02-05T08:23:34.899071+0000","last_undegraded":"2024-02-05T08:23:34.899071+0000","last_fullsized":"2024-02-05T08:23:34.899071+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:19.107555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:19.107555+0000","last_clean_scrub_stamp":"2024-02-05T08:21:19.107555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":214,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.898529+0000","last_change":"2024-02-05T08:23:08.927552+0000","last_active":"2024-02-05T08:23:34.898529+0000","last_peered":"2024-02-05T08:23:34.898529+0000","last_clean":"2024-02-05T08:23:34.898529+0000","last_became_active":"2024-02-05T08:23:08.927256+0000","last_became_peered":"2024-02-05T08:23:08.927256+0000","last_unstale":"2024-02-05T08:23:34.898529+0000","last_undegraded":"2024-02-05T08:23:34.898529+0000","last_fullsized":"2024-02-05T08:23:34.898529+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:46.026840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:22:46.026840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:46.162001+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"236'19","reported_seq":472,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.904485+0000","last_change":"2024-02-05T08:22:58.484904+0000","last_active":"2024-02-05T08:23:34.904485+0000","last_peered":"2024-02-05T08:23:34.904485+0000","last_clean":"2024-02-05T08:23:34.904485+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:23:34.904485+0000","last_undegraded":"2024-02-05T08:23:34.904485+0000","last_fullsized":"2024-02-05T08:23:34.904485+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:22:58.484809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:22:58.484809+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:16.300430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":691,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899688+0000","last_change":"2024-02-05T08:21:14.159752+0000","last_active":"2024-02-05T08:23:34.899688+0000","last_peered":"2024-02-05T08:23:34.899688+0000","last_clean":"2024-02-05T08:23:34.899688+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:23:34.899688+0000","last_undegraded":"2024-02-05T08:23:34.899688+0000","last_fullsized":"2024-02-05T08:23:34.899688+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:14.159628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:14.159628+0000","last_clean_scrub_stamp":"2024-02-05T08:21:14.159628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":450,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.897336+0000","last_change":"2024-02-05T08:23:31.538869+0000","last_active":"2024-02-05T08:23:34.897336+0000","last_peered":"2024-02-05T08:23:34.897336+0000","last_clean":"2024-02-05T08:23:34.897336+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:23:34.897336+0000","last_undegraded":"2024-02-05T08:23:34.897336+0000","last_fullsized":"2024-02-05T08:23:34.897336+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:31.538744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:23:31.538744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:42.776710+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":589,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.897043+0000","last_change":"2024-02-05T08:22:42.347338+0000","last_active":"2024-02-05T08:23:34.897043+0000","last_peered":"2024-02-05T08:23:34.897043+0000","last_clean":"2024-02-05T08:23:34.897043+0000","last_became_active":"2024-02-05T08:22:09.104064+0000","last_became_peered":"2024-02-05T08:22:09.104064+0000","last_unstale":"2024-02-05T08:23:34.897043+0000","last_undegraded":"2024-02-05T08:23:34.897043+0000","last_fullsized":"2024-02-05T08:23:34.897043+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:42.347255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:34.697248+0000","last_clean_scrub_stamp":"2024-02-05T08:22:42.347255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:45.200515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":216,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899112+0000","last_change":"2024-02-05T08:23:31.939827+0000","last_active":"2024-02-05T08:23:34.899112+0000","last_peered":"2024-02-05T08:23:34.899112+0000","last_clean":"2024-02-05T08:23:34.899112+0000","last_became_active":"2024-02-05T08:23:07.937911+0000","last_became_peered":"2024-02-05T08:23:07.937911+0000","last_unstale":"2024-02-05T08:23:34.899112+0000","last_undegraded":"2024-02-05T08:23:34.899112+0000","last_fullsized":"2024-02-05T08:23:34.899112+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:23:31.939578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:31.939578+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:36.879693+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":215,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899287+0000","last_change":"2024-02-05T08:22:47.060160+0000","last_active":"2024-02-05T08:23:34.899287+0000","last_peered":"2024-02-05T08:23:34.899287+0000","last_clean":"2024-02-05T08:23:34.899287+0000","last_became_active":"2024-02-05T08:22:09.095805+0000","last_became_peered":"2024-02-05T08:22:09.095805+0000","last_unstale":"2024-02-05T08:23:34.899287+0000","last_undegraded":"2024-02-05T08:23:34.899287+0000","last_fullsized":"2024-02-05T08:23:34.899287+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:47.060036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:26.891040+0000","last_clean_scrub_stamp":"2024-02-05T08:22:47.060036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:01.352644+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":723,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.900172+0000","last_change":"2024-02-05T08:21:54.156544+0000","last_active":"2024-02-05T08:23:34.900172+0000","last_peered":"2024-02-05T08:23:34.900172+0000","last_clean":"2024-02-05T08:23:34.900172+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:23:34.900172+0000","last_undegraded":"2024-02-05T08:23:34.900172+0000","last_fullsized":"2024-02-05T08:23:34.900172+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:54.156080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:54.156080+0000","last_clean_scrub_stamp":"2024-02-05T08:21:54.156080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":718,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.895725+0000","last_change":"2024-02-05T08:23:28.001765+0000","last_active":"2024-02-05T08:23:34.895725+0000","last_peered":"2024-02-05T08:23:34.895725+0000","last_clean":"2024-02-05T08:23:34.895725+0000","last_became_active":"2024-02-05T08:23:07.932374+0000","last_became_peered":"2024-02-05T08:23:07.932374+0000","last_unstale":"2024-02-05T08:23:34.895725+0000","last_undegraded":"2024-02-05T08:23:34.895725+0000","last_fullsized":"2024-02-05T08:23:34.895725+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:28.001523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:23:28.001523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:40.938375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"236'24","reported_seq":213,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.902972+0000","last_change":"2024-02-05T08:22:01.724542+0000","last_active":"2024-02-05T08:23:34.902972+0000","last_peered":"2024-02-05T08:23:34.902972+0000","last_clean":"2024-02-05T08:23:34.902972+0000","last_became_active":"2024-02-05T08:18:34.631912+0000","last_became_peered":"2024-02-05T08:18:34.631912+0000","last_unstale":"2024-02-05T08:23:34.902972+0000","last_undegraded":"2024-02-05T08:23:34.902972+0000","last_fullsized":"2024-02-05T08:23:34.902972+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:22:01.724412+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:22:01.724412+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":644,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.894075+0000","last_change":"2024-02-05T08:23:08.918458+0000","last_active":"2024-02-05T08:23:34.894075+0000","last_peered":"2024-02-05T08:23:34.894075+0000","last_clean":"2024-02-05T08:23:34.894075+0000","last_became_active":"2024-02-05T08:23:08.918104+0000","last_became_peered":"2024-02-05T08:23:08.918104+0000","last_unstale":"2024-02-05T08:23:34.894075+0000","last_undegraded":"2024-02-05T08:23:34.894075+0000","last_fullsized":"2024-02-05T08:23:34.894075+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:51.235476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:22:51.235476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:51.896350+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":214,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899201+0000","last_change":"2024-02-05T08:23:08.918137+0000","last_active":"2024-02-05T08:23:34.899201+0000","last_peered":"2024-02-05T08:23:34.899201+0000","last_clean":"2024-02-05T08:23:34.899201+0000","last_became_active":"2024-02-05T08:23:08.917587+0000","last_became_peered":"2024-02-05T08:23:08.917587+0000","last_unstale":"2024-02-05T08:23:34.899201+0000","last_undegraded":"2024-02-05T08:23:34.899201+0000","last_fullsized":"2024-02-05T08:23:34.899201+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:44.050117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:44.050117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:57.446512+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":722,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899318+0000","last_change":"2024-02-05T08:21:53.174425+0000","last_active":"2024-02-05T08:23:34.899318+0000","last_peered":"2024-02-05T08:23:34.899318+0000","last_clean":"2024-02-05T08:23:34.899318+0000","last_became_active":"2024-02-05T08:18:34.642387+0000","last_became_peered":"2024-02-05T08:18:34.642387+0000","last_unstale":"2024-02-05T08:23:34.899318+0000","last_undegraded":"2024-02-05T08:23:34.899318+0000","last_fullsized":"2024-02-05T08:23:34.899318+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:53.173888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:53.173888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"263'19","reported_seq":377,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.895361+0000","last_change":"2024-02-05T08:21:53.427022+0000","last_active":"2024-02-05T08:23:34.895361+0000","last_peered":"2024-02-05T08:23:34.895361+0000","last_clean":"2024-02-05T08:23:34.895361+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:23:34.895361+0000","last_undegraded":"2024-02-05T08:23:34.895361+0000","last_fullsized":"2024-02-05T08:23:34.895361+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:21:53.422885+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_clean_scrub_stamp":"2024-02-05T08:21:53.422885+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":223,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.902722+0000","last_change":"2024-02-05T08:23:00.825681+0000","last_active":"2024-02-05T08:23:34.902722+0000","last_peered":"2024-02-05T08:23:34.902722+0000","last_clean":"2024-02-05T08:23:34.902722+0000","last_became_active":"2024-02-05T08:22:09.096314+0000","last_became_peered":"2024-02-05T08:22:09.096314+0000","last_unstale":"2024-02-05T08:23:34.902722+0000","last_undegraded":"2024-02-05T08:23:34.902722+0000","last_fullsized":"2024-02-05T08:23:34.902722+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:00.825605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:23:00.825605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:28.304848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":615,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.900134+0000","last_change":"2024-02-05T08:23:07.938030+0000","last_active":"2024-02-05T08:23:45.900134+0000","last_peered":"2024-02-05T08:23:45.900134+0000","last_clean":"2024-02-05T08:23:45.900134+0000","last_became_active":"2024-02-05T08:23:07.936401+0000","last_became_peered":"2024-02-05T08:23:07.936401+0000","last_unstale":"2024-02-05T08:23:45.900134+0000","last_undegraded":"2024-02-05T08:23:45.900134+0000","last_fullsized":"2024-02-05T08:23:45.900134+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:17.633650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:19:17.633650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":732,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.898924+0000","last_change":"2024-02-05T08:21:44.871777+0000","last_active":"2024-02-05T08:23:45.898924+0000","last_peered":"2024-02-05T08:23:45.898924+0000","last_clean":"2024-02-05T08:23:45.898924+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:23:45.898924+0000","last_undegraded":"2024-02-05T08:23:45.898924+0000","last_fullsized":"2024-02-05T08:23:45.898924+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:44.871668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:21:44.871668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"249'24","reported_seq":781,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.900114+0000","last_change":"2024-02-05T08:23:07.938874+0000","last_active":"2024-02-05T08:23:45.900114+0000","last_peered":"2024-02-05T08:23:45.900114+0000","last_clean":"2024-02-05T08:23:45.900114+0000","last_became_active":"2024-02-05T08:23:07.938523+0000","last_became_peered":"2024-02-05T08:23:07.938523+0000","last_unstale":"2024-02-05T08:23:45.900114+0000","last_undegraded":"2024-02-05T08:23:45.900114+0000","last_fullsized":"2024-02-05T08:23:45.900114+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:19:27.349541+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:19:27.349541+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":569,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.898855+0000","last_change":"2024-02-05T08:23:33.061246+0000","last_active":"2024-02-05T08:23:34.898855+0000","last_peered":"2024-02-05T08:23:34.898855+0000","last_clean":"2024-02-05T08:23:34.898855+0000","last_became_active":"2024-02-05T08:22:09.101652+0000","last_became_peered":"2024-02-05T08:22:09.101652+0000","last_unstale":"2024-02-05T08:23:34.898855+0000","last_undegraded":"2024-02-05T08:23:34.898855+0000","last_fullsized":"2024-02-05T08:23:34.898855+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:33.061104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:33.061104+0000","last_clean_scrub_stamp":"2024-02-05T08:23:33.061104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:40.201454+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"370'13","reported_seq":959,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:36.811657+0000","last_change":"2024-02-05T08:23:08.918239+0000","last_active":"2024-02-05T08:23:36.811657+0000","last_peered":"2024-02-05T08:23:36.811657+0000","last_clean":"2024-02-05T08:23:36.811657+0000","last_became_active":"2024-02-05T08:23:08.917858+0000","last_became_peered":"2024-02-05T08:23:08.917858+0000","last_unstale":"2024-02-05T08:23:36.811657+0000","last_undegraded":"2024-02-05T08:23:36.811657+0000","last_fullsized":"2024-02-05T08:23:36.811657+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'11","last_scrub_stamp":"2024-02-05T08:22:48.261980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:48.261980+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:48.313434+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":711,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899200+0000","last_change":"2024-02-05T08:21:09.098128+0000","last_active":"2024-02-05T08:23:34.899200+0000","last_peered":"2024-02-05T08:23:34.899200+0000","last_clean":"2024-02-05T08:23:34.899200+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:23:34.899200+0000","last_undegraded":"2024-02-05T08:23:34.899200+0000","last_fullsized":"2024-02-05T08:23:34.899200+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:09.098005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:09.098005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":214,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.898864+0000","last_change":"2024-02-05T08:23:07.929317+0000","last_active":"2024-02-05T08:23:34.898864+0000","last_peered":"2024-02-05T08:23:34.898864+0000","last_clean":"2024-02-05T08:23:34.898864+0000","last_became_active":"2024-02-05T08:23:07.928966+0000","last_became_peered":"2024-02-05T08:23:07.928966+0000","last_unstale":"2024-02-05T08:23:34.898864+0000","last_undegraded":"2024-02-05T08:23:34.898864+0000","last_fullsized":"2024-02-05T08:23:34.898864+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-02-05T08:19:35.981313+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:19:35.981313+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":694,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.899802+0000","last_change":"2024-02-05T08:21:10.125938+0000","last_active":"2024-02-05T08:23:45.899802+0000","last_peered":"2024-02-05T08:23:45.899802+0000","last_clean":"2024-02-05T08:23:45.899802+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:23:45.899802+0000","last_undegraded":"2024-02-05T08:23:45.899802+0000","last_fullsized":"2024-02-05T08:23:45.899802+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:10.125818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:28.519149+0000","last_clean_scrub_stamp":"2024-02-05T08:21:10.125818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"295'10","reported_seq":268,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:36.810916+0000","last_change":"2024-02-05T08:22:03.724091+0000","last_active":"2024-02-05T08:23:36.810916+0000","last_peered":"2024-02-05T08:23:36.810916+0000","last_clean":"2024-02-05T08:23:36.810916+0000","last_became_active":"2024-02-05T08:18:34.644517+0000","last_became_peered":"2024-02-05T08:18:34.644517+0000","last_unstale":"2024-02-05T08:23:36.810916+0000","last_undegraded":"2024-02-05T08:23:36.810916+0000","last_fullsized":"2024-02-05T08:23:36.810916+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'10","last_scrub_stamp":"2024-02-05T08:22:03.723970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:03.723970+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":585,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.898532+0000","last_change":"2024-02-05T08:23:08.920515+0000","last_active":"2024-02-05T08:23:34.898532+0000","last_peered":"2024-02-05T08:23:34.898532+0000","last_clean":"2024-02-05T08:23:34.898532+0000","last_became_active":"2024-02-05T08:23:08.920193+0000","last_became_peered":"2024-02-05T08:23:08.920193+0000","last_unstale":"2024-02-05T08:23:34.898532+0000","last_undegraded":"2024-02-05T08:23:34.898532+0000","last_fullsized":"2024-02-05T08:23:34.898532+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:03.366172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:21:03.366172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":778,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.896497+0000","last_change":"2024-02-05T08:23:23.049946+0000","last_active":"2024-02-05T08:23:34.896497+0000","last_peered":"2024-02-05T08:23:34.896497+0000","last_clean":"2024-02-05T08:23:34.896497+0000","last_became_active":"2024-02-05T08:18:34.640741+0000","last_became_peered":"2024-02-05T08:18:34.640741+0000","last_unstale":"2024-02-05T08:23:34.896497+0000","last_undegraded":"2024-02-05T08:23:34.896497+0000","last_fullsized":"2024-02-05T08:23:34.896497+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:23:23.049797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:23.049797+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:28.469864+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":722,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.894158+0000","last_change":"2024-02-05T08:22:42.141091+0000","last_active":"2024-02-05T08:23:34.894158+0000","last_peered":"2024-02-05T08:23:34.894158+0000","last_clean":"2024-02-05T08:23:34.894158+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:23:34.894158+0000","last_undegraded":"2024-02-05T08:23:34.894158+0000","last_fullsized":"2024-02-05T08:23:34.894158+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:42.141016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:42.141016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:46.560455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":210,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:42.025631+0000","last_change":"2024-02-05T08:23:42.025667+0000","last_active":"2024-02-05T08:23:42.025631+0000","last_peered":"2024-02-05T08:23:42.025631+0000","last_clean":"2024-02-05T08:23:42.025631+0000","last_became_active":"2024-02-05T08:23:07.928375+0000","last_became_peered":"2024-02-05T08:23:07.928375+0000","last_unstale":"2024-02-05T08:23:42.025631+0000","last_undegraded":"2024-02-05T08:23:42.025631+0000","last_fullsized":"2024-02-05T08:23:42.025631+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:42.025471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:42.025471+0000","last_clean_scrub_stamp":"2024-02-05T08:23:42.025471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:53.919766+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":723,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.898117+0000","last_change":"2024-02-05T08:22:05.129978+0000","last_active":"2024-02-05T08:23:34.898117+0000","last_peered":"2024-02-05T08:23:34.898117+0000","last_clean":"2024-02-05T08:23:34.898117+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:23:34.898117+0000","last_undegraded":"2024-02-05T08:23:34.898117+0000","last_fullsized":"2024-02-05T08:23:34.898117+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:05.129584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:22:05.129584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"278'19","reported_seq":626,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.901934+0000","last_change":"2024-02-05T08:22:09.127927+0000","last_active":"2024-02-05T08:23:34.901934+0000","last_peered":"2024-02-05T08:23:34.901934+0000","last_clean":"2024-02-05T08:23:34.901934+0000","last_became_active":"2024-02-05T08:22:09.110422+0000","last_became_peered":"2024-02-05T08:22:09.110422+0000","last_unstale":"2024-02-05T08:23:34.901934+0000","last_undegraded":"2024-02-05T08:23:34.901934+0000","last_fullsized":"2024-02-05T08:23:34.901934+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-05T08:21:50.244148+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:21:50.244148+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":710,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899226+0000","last_change":"2024-02-05T08:21:37.234586+0000","last_active":"2024-02-05T08:23:34.899226+0000","last_peered":"2024-02-05T08:23:34.899226+0000","last_clean":"2024-02-05T08:23:34.899226+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:23:34.899226+0000","last_undegraded":"2024-02-05T08:23:34.899226+0000","last_fullsized":"2024-02-05T08:23:34.899226+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:37.234451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:37.234451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":722,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:35.889618+0000","last_change":"2024-02-05T08:23:35.889657+0000","last_active":"2024-02-05T08:23:35.889618+0000","last_peered":"2024-02-05T08:23:35.889618+0000","last_clean":"2024-02-05T08:23:35.889618+0000","last_became_active":"2024-02-05T08:23:07.932741+0000","last_became_peered":"2024-02-05T08:23:07.932741+0000","last_unstale":"2024-02-05T08:23:35.889618+0000","last_undegraded":"2024-02-05T08:23:35.889618+0000","last_fullsized":"2024-02-05T08:23:35.889618+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:35.889538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:23:35.889538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:47.168827+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":453,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.896943+0000","last_change":"2024-02-05T08:22:43.515671+0000","last_active":"2024-02-05T08:23:34.896943+0000","last_peered":"2024-02-05T08:23:34.896943+0000","last_clean":"2024-02-05T08:23:34.896943+0000","last_became_active":"2024-02-05T08:22:09.097843+0000","last_became_peered":"2024-02-05T08:22:09.097843+0000","last_unstale":"2024-02-05T08:23:34.896943+0000","last_undegraded":"2024-02-05T08:23:34.896943+0000","last_fullsized":"2024-02-05T08:23:34.896943+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:43.515578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:22:43.515578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:44.669281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'15","reported_seq":210,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.902106+0000","last_change":"2024-02-05T08:23:26.922363+0000","last_active":"2024-02-05T08:23:34.902106+0000","last_peered":"2024-02-05T08:23:34.902106+0000","last_clean":"2024-02-05T08:23:34.902106+0000","last_became_active":"2024-02-05T08:23:07.927922+0000","last_became_peered":"2024-02-05T08:23:07.927922+0000","last_unstale":"2024-02-05T08:23:34.902106+0000","last_undegraded":"2024-02-05T08:23:34.902106+0000","last_fullsized":"2024-02-05T08:23:34.902106+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:23:26.922227+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:23:26.922227+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:29.792548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"381'3161","reported_seq":3199,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.561672+0000","last_change":"2024-02-05T08:23:08.913493+0000","last_active":"2024-02-05T08:23:43.561672+0000","last_peered":"2024-02-05T08:23:43.561672+0000","last_clean":"2024-02-05T08:23:43.561672+0000","last_became_active":"2024-02-05T08:23:08.913227+0000","last_became_peered":"2024-02-05T08:23:08.913227+0000","last_unstale":"2024-02-05T08:23:43.561672+0000","last_undegraded":"2024-02-05T08:23:43.561672+0000","last_fullsized":"2024-02-05T08:23:43.561672+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3161,"log_dups_size":0,"ondisk_log_size":3161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:26.063441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207159296,"num_objects":3161,"num_object_clones":0,"num_object_copies":9483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3161,"num_write_kb":202304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":724,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.896504+0000","last_change":"2024-02-05T08:22:09.100651+0000","last_active":"2024-02-05T08:23:34.896504+0000","last_peered":"2024-02-05T08:23:34.896504+0000","last_clean":"2024-02-05T08:23:34.896504+0000","last_became_active":"2024-02-05T08:22:09.100322+0000","last_became_peered":"2024-02-05T08:22:09.100322+0000","last_unstale":"2024-02-05T08:23:34.896504+0000","last_undegraded":"2024-02-05T08:23:34.896504+0000","last_fullsized":"2024-02-05T08:23:34.896504+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:45.532212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:45.532212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":747,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.066578+0000","last_change":"2024-02-05T08:23:43.066628+0000","last_active":"2024-02-05T08:23:43.066578+0000","last_peered":"2024-02-05T08:23:43.066578+0000","last_clean":"2024-02-05T08:23:43.066578+0000","last_became_active":"2024-02-05T08:23:08.920320+0000","last_became_peered":"2024-02-05T08:23:08.920320+0000","last_unstale":"2024-02-05T08:23:43.066578+0000","last_undegraded":"2024-02-05T08:23:43.066578+0000","last_fullsized":"2024-02-05T08:23:43.066578+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:23:43.066380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:23:43.066380+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:53.616786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"262'34","reported_seq":800,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.893989+0000","last_change":"2024-02-05T08:21:44.548331+0000","last_active":"2024-02-05T08:23:34.893989+0000","last_peered":"2024-02-05T08:23:34.893989+0000","last_clean":"2024-02-05T08:23:34.893989+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:23:34.893989+0000","last_undegraded":"2024-02-05T08:23:34.893989+0000","last_fullsized":"2024-02-05T08:23:34.893989+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'34","last_scrub_stamp":"2024-02-05T08:21:44.548195+0000","last_deep_scrub":"262'34","last_deep_scrub_stamp":"2024-02-05T08:19:34.364738+0000","last_clean_scrub_stamp":"2024-02-05T08:21:44.548195+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":961,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:36.811195+0000","last_change":"2024-02-05T08:21:46.165287+0000","last_active":"2024-02-05T08:23:36.811195+0000","last_peered":"2024-02-05T08:23:36.811195+0000","last_clean":"2024-02-05T08:23:36.811195+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:23:36.811195+0000","last_undegraded":"2024-02-05T08:23:36.811195+0000","last_fullsized":"2024-02-05T08:23:36.811195+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:21:46.164949+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_clean_scrub_stamp":"2024-02-05T08:21:46.164949+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"381'3139","reported_seq":3177,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.317750+0000","last_change":"2024-02-05T08:23:08.910972+0000","last_active":"2024-02-05T08:23:43.317750+0000","last_peered":"2024-02-05T08:23:43.317750+0000","last_clean":"2024-02-05T08:23:43.317750+0000","last_became_active":"2024-02-05T08:23:08.910680+0000","last_became_peered":"2024-02-05T08:23:08.910680+0000","last_unstale":"2024-02-05T08:23:43.317750+0000","last_undegraded":"2024-02-05T08:23:43.317750+0000","last_fullsized":"2024-02-05T08:23:43.317750+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3139,"log_dups_size":0,"ondisk_log_size":3139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:12.860531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205717504,"num_objects":3139,"num_object_clones":0,"num_object_copies":9417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3139,"num_write_kb":200896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":202,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899483+0000","last_change":"2024-02-05T08:23:29.086552+0000","last_active":"2024-02-05T08:23:34.899483+0000","last_peered":"2024-02-05T08:23:34.899483+0000","last_clean":"2024-02-05T08:23:34.899483+0000","last_became_active":"2024-02-05T08:23:07.929942+0000","last_became_peered":"2024-02-05T08:23:07.929942+0000","last_unstale":"2024-02-05T08:23:34.899483+0000","last_undegraded":"2024-02-05T08:23:34.899483+0000","last_fullsized":"2024-02-05T08:23:34.899483+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:29.086293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.086293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:52.169995+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":720,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.898988+0000","last_change":"2024-02-05T08:23:30.010159+0000","last_active":"2024-02-05T08:23:34.898988+0000","last_peered":"2024-02-05T08:23:34.898988+0000","last_clean":"2024-02-05T08:23:34.898988+0000","last_became_active":"2024-02-05T08:23:07.937863+0000","last_became_peered":"2024-02-05T08:23:07.937863+0000","last_unstale":"2024-02-05T08:23:34.898988+0000","last_undegraded":"2024-02-05T08:23:34.898988+0000","last_fullsized":"2024-02-05T08:23:34.898988+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:30.010028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:23:30.010028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:57.900947+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":777,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.897535+0000","last_change":"2024-02-05T08:23:30.492935+0000","last_active":"2024-02-05T08:23:45.897535+0000","last_peered":"2024-02-05T08:23:45.897535+0000","last_clean":"2024-02-05T08:23:45.897535+0000","last_became_active":"2024-02-05T08:23:07.929900+0000","last_became_peered":"2024-02-05T08:23:07.929900+0000","last_unstale":"2024-02-05T08:23:45.897535+0000","last_undegraded":"2024-02-05T08:23:45.897535+0000","last_fullsized":"2024-02-05T08:23:45.897535+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-02-05T08:23:30.492582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:30.492582+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:58.313450+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":723,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.889968+0000","last_change":"2024-02-05T08:23:08.918621+0000","last_active":"2024-02-05T08:23:34.889968+0000","last_peered":"2024-02-05T08:23:34.889968+0000","last_clean":"2024-02-05T08:23:34.889968+0000","last_became_active":"2024-02-05T08:23:08.918318+0000","last_became_peered":"2024-02-05T08:23:08.918318+0000","last_unstale":"2024-02-05T08:23:34.889968+0000","last_undegraded":"2024-02-05T08:23:34.889968+0000","last_fullsized":"2024-02-05T08:23:34.889968+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:43.173767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:22:43.173767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:52.653360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"381'3260","reported_seq":3296,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.373776+0000","last_change":"2024-02-05T08:23:08.911955+0000","last_active":"2024-02-05T08:23:43.373776+0000","last_peered":"2024-02-05T08:23:43.373776+0000","last_clean":"2024-02-05T08:23:43.373776+0000","last_became_active":"2024-02-05T08:23:08.911591+0000","last_became_peered":"2024-02-05T08:23:08.911591+0000","last_unstale":"2024-02-05T08:23:43.373776+0000","last_undegraded":"2024-02-05T08:23:43.373776+0000","last_fullsized":"2024-02-05T08:23:43.373776+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3260,"log_dups_size":0,"ondisk_log_size":3260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:26.070704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213647360,"num_objects":3260,"num_object_clones":0,"num_object_copies":9780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3260,"num_write_kb":208640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":711,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899065+0000","last_change":"2024-02-05T08:21:44.100658+0000","last_active":"2024-02-05T08:23:34.899065+0000","last_peered":"2024-02-05T08:23:34.899065+0000","last_clean":"2024-02-05T08:23:34.899065+0000","last_became_active":"2024-02-05T08:18:34.642239+0000","last_became_peered":"2024-02-05T08:18:34.642239+0000","last_unstale":"2024-02-05T08:23:34.899065+0000","last_undegraded":"2024-02-05T08:23:34.899065+0000","last_fullsized":"2024-02-05T08:23:34.899065+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:44.100428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:44.100428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":724,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.892345+0000","last_change":"2024-02-05T08:22:51.099386+0000","last_active":"2024-02-05T08:23:34.892345+0000","last_peered":"2024-02-05T08:23:34.892345+0000","last_clean":"2024-02-05T08:23:34.892345+0000","last_became_active":"2024-02-05T08:22:09.105439+0000","last_became_peered":"2024-02-05T08:22:09.105439+0000","last_unstale":"2024-02-05T08:23:34.892345+0000","last_undegraded":"2024-02-05T08:23:34.892345+0000","last_fullsized":"2024-02-05T08:23:34.892345+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:51.099279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:22:51.099279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:04.733308+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":724,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899751+0000","last_change":"2024-02-05T08:21:40.182453+0000","last_active":"2024-02-05T08:23:34.899751+0000","last_peered":"2024-02-05T08:23:34.899751+0000","last_clean":"2024-02-05T08:23:34.899751+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:23:34.899751+0000","last_undegraded":"2024-02-05T08:23:34.899751+0000","last_fullsized":"2024-02-05T08:23:34.899751+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:40.182334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_clean_scrub_stamp":"2024-02-05T08:21:40.182334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"236'35","reported_seq":500,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.897167+0000","last_change":"2024-02-05T08:23:29.598019+0000","last_active":"2024-02-05T08:23:34.897167+0000","last_peered":"2024-02-05T08:23:34.897167+0000","last_clean":"2024-02-05T08:23:34.897167+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:23:34.897167+0000","last_undegraded":"2024-02-05T08:23:34.897167+0000","last_fullsized":"2024-02-05T08:23:34.897167+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'35","last_scrub_stamp":"2024-02-05T08:23:29.597891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.597891+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:35.763221+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"381'3054","reported_seq":3092,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.359801+0000","last_change":"2024-02-05T08:23:08.912293+0000","last_active":"2024-02-05T08:23:43.359801+0000","last_peered":"2024-02-05T08:23:43.359801+0000","last_clean":"2024-02-05T08:23:43.359801+0000","last_became_active":"2024-02-05T08:23:08.911987+0000","last_became_peered":"2024-02-05T08:23:08.911987+0000","last_unstale":"2024-02-05T08:23:43.359801+0000","last_undegraded":"2024-02-05T08:23:43.359801+0000","last_fullsized":"2024-02-05T08:23:43.359801+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3054,"log_dups_size":0,"ondisk_log_size":3054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:21.353748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200146944,"num_objects":3054,"num_object_clones":0,"num_object_copies":9162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3054,"num_write_kb":195456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":68,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.897952+0000","last_change":"2024-02-05T08:22:09.101508+0000","last_active":"2024-02-05T08:23:34.897952+0000","last_peered":"2024-02-05T08:23:34.897952+0000","last_clean":"2024-02-05T08:23:34.897952+0000","last_became_active":"2024-02-05T08:22:09.100986+0000","last_became_peered":"2024-02-05T08:22:09.100986+0000","last_unstale":"2024-02-05T08:23:34.897952+0000","last_undegraded":"2024-02-05T08:23:34.897952+0000","last_fullsized":"2024-02-05T08:23:34.897952+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:37.204550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_clean_scrub_stamp":"2024-02-05T08:21:37.204550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":738,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.893448+0000","last_change":"2024-02-05T08:22:59.883429+0000","last_active":"2024-02-05T08:23:34.893448+0000","last_peered":"2024-02-05T08:23:34.893448+0000","last_clean":"2024-02-05T08:23:34.893448+0000","last_became_active":"2024-02-05T08:18:34.645830+0000","last_became_peered":"2024-02-05T08:18:34.645830+0000","last_unstale":"2024-02-05T08:23:34.893448+0000","last_undegraded":"2024-02-05T08:23:34.893448+0000","last_fullsized":"2024-02-05T08:23:34.893448+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:59.883347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:59.883347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:22.202744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":742,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.895809+0000","last_change":"2024-02-05T08:23:01.230170+0000","last_active":"2024-02-05T08:23:34.895809+0000","last_peered":"2024-02-05T08:23:34.895809+0000","last_clean":"2024-02-05T08:23:34.895809+0000","last_became_active":"2024-02-05T08:18:34.633214+0000","last_became_peered":"2024-02-05T08:18:34.633214+0000","last_unstale":"2024-02-05T08:23:34.895809+0000","last_undegraded":"2024-02-05T08:23:34.895809+0000","last_fullsized":"2024-02-05T08:23:34.895809+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:01.230038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:01.230038+0000","last_clean_scrub_stamp":"2024-02-05T08:23:01.230038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:15.812612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"262'19","reported_seq":501,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.900544+0000","last_change":"2024-02-05T08:23:08.921585+0000","last_active":"2024-02-05T08:23:45.900544+0000","last_peered":"2024-02-05T08:23:45.900544+0000","last_clean":"2024-02-05T08:23:45.900544+0000","last_became_active":"2024-02-05T08:23:08.920885+0000","last_became_peered":"2024-02-05T08:23:08.920885+0000","last_unstale":"2024-02-05T08:23:45.900544+0000","last_undegraded":"2024-02-05T08:23:45.900544+0000","last_fullsized":"2024-02-05T08:23:45.900544+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'19","last_scrub_stamp":"2024-02-05T08:22:53.159489+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:22:53.159489+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:05.107488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"381'3045","reported_seq":3082,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.264929+0000","last_change":"2024-02-05T08:23:08.917492+0000","last_active":"2024-02-05T08:23:43.264929+0000","last_peered":"2024-02-05T08:23:43.264929+0000","last_clean":"2024-02-05T08:23:43.264929+0000","last_became_active":"2024-02-05T08:23:08.917212+0000","last_became_peered":"2024-02-05T08:23:08.917212+0000","last_unstale":"2024-02-05T08:23:43.264929+0000","last_undegraded":"2024-02-05T08:23:43.264929+0000","last_fullsized":"2024-02-05T08:23:43.264929+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3045,"log_dups_size":0,"ondisk_log_size":3045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:20.012089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199557120,"num_objects":3045,"num_object_clones":0,"num_object_copies":9135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3045,"num_write_kb":194880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'130","reported_seq":494,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.895117+0000","last_change":"2024-02-05T08:22:12.147142+0000","last_active":"2024-02-05T08:23:34.895117+0000","last_peered":"2024-02-05T08:23:34.895117+0000","last_clean":"2024-02-05T08:23:34.895117+0000","last_became_active":"2024-02-05T08:22:12.146682+0000","last_became_peered":"2024-02-05T08:22:12.146682+0000","last_unstale":"2024-02-05T08:23:34.895117+0000","last_undegraded":"2024-02-05T08:23:34.895117+0000","last_fullsized":"2024-02-05T08:23:34.895117+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:47.509712+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:47.509712+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":432,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.899090+0000","last_change":"2024-02-05T08:23:26.660184+0000","last_active":"2024-02-05T08:23:45.899090+0000","last_peered":"2024-02-05T08:23:45.899090+0000","last_clean":"2024-02-05T08:23:45.899090+0000","last_became_active":"2024-02-05T08:23:07.936660+0000","last_became_peered":"2024-02-05T08:23:07.936660+0000","last_unstale":"2024-02-05T08:23:45.899090+0000","last_undegraded":"2024-02-05T08:23:45.899090+0000","last_fullsized":"2024-02-05T08:23:45.899090+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:26.660108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:26.660108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:46.099841+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":752,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.896032+0000","last_change":"2024-02-05T08:23:27.050401+0000","last_active":"2024-02-05T08:23:34.896032+0000","last_peered":"2024-02-05T08:23:34.896032+0000","last_clean":"2024-02-05T08:23:34.896032+0000","last_became_active":"2024-02-05T08:23:07.937565+0000","last_became_peered":"2024-02-05T08:23:07.937565+0000","last_unstale":"2024-02-05T08:23:34.896032+0000","last_undegraded":"2024-02-05T08:23:34.896032+0000","last_fullsized":"2024-02-05T08:23:34.896032+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:23:27.050262+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:23:27.050262+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:42.760480+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":735,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.899319+0000","last_change":"2024-02-05T08:23:08.921073+0000","last_active":"2024-02-05T08:23:45.899319+0000","last_peered":"2024-02-05T08:23:45.899319+0000","last_clean":"2024-02-05T08:23:45.899319+0000","last_became_active":"2024-02-05T08:23:08.920421+0000","last_became_peered":"2024-02-05T08:23:08.920421+0000","last_unstale":"2024-02-05T08:23:45.899319+0000","last_undegraded":"2024-02-05T08:23:45.899319+0000","last_fullsized":"2024-02-05T08:23:45.899319+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:46.849759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:21:46.849759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"370'54","reported_seq":920,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.900219+0000","last_change":"2024-02-05T08:23:07.932963+0000","last_active":"2024-02-05T08:23:45.900219+0000","last_peered":"2024-02-05T08:23:45.900219+0000","last_clean":"2024-02-05T08:23:45.900219+0000","last_became_active":"2024-02-05T08:23:07.932513+0000","last_became_peered":"2024-02-05T08:23:07.932513+0000","last_unstale":"2024-02-05T08:23:45.900219+0000","last_undegraded":"2024-02-05T08:23:45.900219+0000","last_fullsized":"2024-02-05T08:23:45.900219+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'52","last_scrub_stamp":"2024-02-05T08:19:15.601598+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:19:15.601598+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":41,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.903933+0000","last_change":"2024-02-05T08:23:08.915936+0000","last_active":"2024-02-05T08:23:45.903933+0000","last_peered":"2024-02-05T08:23:45.903933+0000","last_clean":"2024-02-05T08:23:45.903933+0000","last_became_active":"2024-02-05T08:23:08.914563+0000","last_became_peered":"2024-02-05T08:23:08.914563+0000","last_unstale":"2024-02-05T08:23:45.903933+0000","last_undegraded":"2024-02-05T08:23:45.903933+0000","last_fullsized":"2024-02-05T08:23:45.903933+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:52.171116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:52.171116+0000","last_clean_scrub_stamp":"2024-02-05T08:22:52.171116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:52.307791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"381'3143","reported_seq":3181,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.273462+0000","last_change":"2024-02-05T08:23:08.915803+0000","last_active":"2024-02-05T08:23:43.273462+0000","last_peered":"2024-02-05T08:23:43.273462+0000","last_clean":"2024-02-05T08:23:43.273462+0000","last_became_active":"2024-02-05T08:23:08.915520+0000","last_became_peered":"2024-02-05T08:23:08.915520+0000","last_unstale":"2024-02-05T08:23:43.273462+0000","last_undegraded":"2024-02-05T08:23:43.273462+0000","last_fullsized":"2024-02-05T08:23:43.273462+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3143,"log_dups_size":0,"ondisk_log_size":3143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:23.993298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205979648,"num_objects":3143,"num_object_clones":0,"num_object_copies":9429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3143,"num_write_kb":201152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"82'130","reported_seq":1026,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899503+0000","last_change":"2024-02-05T08:23:30.984623+0000","last_active":"2024-02-05T08:23:34.899503+0000","last_peered":"2024-02-05T08:23:34.899503+0000","last_clean":"2024-02-05T08:23:34.899503+0000","last_became_active":"2024-02-05T08:22:12.138774+0000","last_became_peered":"2024-02-05T08:22:12.138774+0000","last_unstale":"2024-02-05T08:23:34.899503+0000","last_undegraded":"2024-02-05T08:23:34.899503+0000","last_fullsized":"2024-02-05T08:23:34.899503+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:30.984495+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:23:30.984495+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:35.070753+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":724,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.901974+0000","last_change":"2024-02-05T08:22:09.032210+0000","last_active":"2024-02-05T08:23:34.901974+0000","last_peered":"2024-02-05T08:23:34.901974+0000","last_clean":"2024-02-05T08:23:34.901974+0000","last_became_active":"2024-02-05T08:18:34.643109+0000","last_became_peered":"2024-02-05T08:18:34.643109+0000","last_unstale":"2024-02-05T08:23:34.901974+0000","last_undegraded":"2024-02-05T08:23:34.901974+0000","last_fullsized":"2024-02-05T08:23:34.901974+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:09.032112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:09.032112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":604,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.904384+0000","last_change":"2024-02-05T08:23:23.016964+0000","last_active":"2024-02-05T08:23:45.904384+0000","last_peered":"2024-02-05T08:23:45.904384+0000","last_clean":"2024-02-05T08:23:45.904384+0000","last_became_active":"2024-02-05T08:23:07.933830+0000","last_became_peered":"2024-02-05T08:23:07.933830+0000","last_unstale":"2024-02-05T08:23:45.904384+0000","last_undegraded":"2024-02-05T08:23:45.904384+0000","last_fullsized":"2024-02-05T08:23:45.904384+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:23:23.016855+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:23:23.016855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:52.654197+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":758,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.890514+0000","last_change":"2024-02-05T08:23:08.913232+0000","last_active":"2024-02-05T08:23:34.890514+0000","last_peered":"2024-02-05T08:23:34.890514+0000","last_clean":"2024-02-05T08:23:34.890514+0000","last_became_active":"2024-02-05T08:23:08.912881+0000","last_became_peered":"2024-02-05T08:23:08.912881+0000","last_unstale":"2024-02-05T08:23:34.890514+0000","last_undegraded":"2024-02-05T08:23:34.890514+0000","last_fullsized":"2024-02-05T08:23:34.890514+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:45.172411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_clean_scrub_stamp":"2024-02-05T08:22:45.172411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:48.066998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"363'22","reported_seq":755,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.892655+0000","last_change":"2024-02-05T08:22:56.041017+0000","last_active":"2024-02-05T08:23:34.892655+0000","last_peered":"2024-02-05T08:23:34.892655+0000","last_clean":"2024-02-05T08:23:34.892655+0000","last_became_active":"2024-02-05T08:22:09.103077+0000","last_became_peered":"2024-02-05T08:22:09.103077+0000","last_unstale":"2024-02-05T08:23:34.892655+0000","last_undegraded":"2024-02-05T08:23:34.892655+0000","last_fullsized":"2024-02-05T08:23:34.892655+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'22","last_scrub_stamp":"2024-02-05T08:22:56.032522+0000","last_deep_scrub":"363'22","last_deep_scrub_stamp":"2024-02-05T08:22:56.032522+0000","last_clean_scrub_stamp":"2024-02-05T08:22:56.032522+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:06.070024+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":726,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.893282+0000","last_change":"2024-02-05T08:22:48.168664+0000","last_active":"2024-02-05T08:23:34.893282+0000","last_peered":"2024-02-05T08:23:34.893282+0000","last_clean":"2024-02-05T08:23:34.893282+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:23:34.893282+0000","last_undegraded":"2024-02-05T08:23:34.893282+0000","last_fullsized":"2024-02-05T08:23:34.893282+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:48.168575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:48.168575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:10.143173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"381'3076","reported_seq":3114,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.256942+0000","last_change":"2024-02-05T08:23:08.914881+0000","last_active":"2024-02-05T08:23:43.256942+0000","last_peered":"2024-02-05T08:23:43.256942+0000","last_clean":"2024-02-05T08:23:43.256942+0000","last_became_active":"2024-02-05T08:23:08.914611+0000","last_became_peered":"2024-02-05T08:23:08.914611+0000","last_unstale":"2024-02-05T08:23:43.256942+0000","last_undegraded":"2024-02-05T08:23:43.256942+0000","last_fullsized":"2024-02-05T08:23:43.256942+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3076,"log_dups_size":0,"ondisk_log_size":3076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:12.499888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201588736,"num_objects":3076,"num_object_clones":0,"num_object_copies":9228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3076,"num_write_kb":196864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":710,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.896864+0000","last_change":"2024-02-05T08:21:38.543400+0000","last_active":"2024-02-05T08:23:34.896864+0000","last_peered":"2024-02-05T08:23:34.896864+0000","last_clean":"2024-02-05T08:23:34.896864+0000","last_became_active":"2024-02-05T08:18:34.633450+0000","last_became_peered":"2024-02-05T08:18:34.633450+0000","last_unstale":"2024-02-05T08:23:34.896864+0000","last_undegraded":"2024-02-05T08:23:34.896864+0000","last_fullsized":"2024-02-05T08:23:34.896864+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:38.543288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:38.543288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":201,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":1023,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899622+0000","last_change":"2024-02-05T08:23:34.058622+0000","last_active":"2024-02-05T08:23:34.899622+0000","last_peered":"2024-02-05T08:23:34.899622+0000","last_clean":"2024-02-05T08:23:34.899622+0000","last_became_active":"2024-02-05T08:23:10.886936+0000","last_became_peered":"2024-02-05T08:23:10.886936+0000","last_unstale":"2024-02-05T08:23:34.899622+0000","last_undegraded":"2024-02-05T08:23:34.899622+0000","last_fullsized":"2024-02-05T08:23:34.899622+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:34.058210+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_clean_scrub_stamp":"2024-02-05T08:23:34.058210+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:03.029013+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":209,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.902196+0000","last_change":"2024-02-05T08:22:54.937959+0000","last_active":"2024-02-05T08:23:34.902196+0000","last_peered":"2024-02-05T08:23:34.902196+0000","last_clean":"2024-02-05T08:23:34.902196+0000","last_became_active":"2024-02-05T08:18:34.631974+0000","last_became_peered":"2024-02-05T08:18:34.631974+0000","last_unstale":"2024-02-05T08:23:34.902196+0000","last_undegraded":"2024-02-05T08:23:34.902196+0000","last_fullsized":"2024-02-05T08:23:34.902196+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:54.937874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:54.937874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:55.087362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"236'30","reported_seq":619,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:36.811599+0000","last_change":"2024-02-05T08:23:26.566491+0000","last_active":"2024-02-05T08:23:36.811599+0000","last_peered":"2024-02-05T08:23:36.811599+0000","last_clean":"2024-02-05T08:23:36.811599+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:23:36.811599+0000","last_undegraded":"2024-02-05T08:23:36.811599+0000","last_fullsized":"2024-02-05T08:23:36.811599+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:23:26.566358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:26.566358+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:53.814998+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":704,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899762+0000","last_change":"2024-02-05T08:21:58.197269+0000","last_active":"2024-02-05T08:23:34.899762+0000","last_peered":"2024-02-05T08:23:34.899762+0000","last_clean":"2024-02-05T08:23:34.899762+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:23:34.899762+0000","last_undegraded":"2024-02-05T08:23:34.899762+0000","last_fullsized":"2024-02-05T08:23:34.899762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:58.197167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_clean_scrub_stamp":"2024-02-05T08:21:58.197167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"381'3017","reported_seq":3055,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.260459+0000","last_change":"2024-02-05T08:23:08.916824+0000","last_active":"2024-02-05T08:23:43.260459+0000","last_peered":"2024-02-05T08:23:43.260459+0000","last_clean":"2024-02-05T08:23:43.260459+0000","last_became_active":"2024-02-05T08:23:08.916452+0000","last_became_peered":"2024-02-05T08:23:08.916452+0000","last_unstale":"2024-02-05T08:23:43.260459+0000","last_undegraded":"2024-02-05T08:23:43.260459+0000","last_fullsized":"2024-02-05T08:23:43.260459+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3017,"log_dups_size":0,"ondisk_log_size":3017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:33.630773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197722112,"num_objects":3017,"num_object_clones":0,"num_object_copies":9051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3017,"num_write_kb":193088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"82'130","reported_seq":375,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.900633+0000","last_change":"2024-02-05T08:23:25.057316+0000","last_active":"2024-02-05T08:23:45.900633+0000","last_peered":"2024-02-05T08:23:45.900633+0000","last_clean":"2024-02-05T08:23:45.900633+0000","last_became_active":"2024-02-05T08:23:07.941142+0000","last_became_peered":"2024-02-05T08:23:07.941142+0000","last_unstale":"2024-02-05T08:23:45.900633+0000","last_undegraded":"2024-02-05T08:23:45.900633+0000","last_fullsized":"2024-02-05T08:23:45.900633+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:25.057181+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:23:25.057181+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:26.634639+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":617,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.903692+0000","last_change":"2024-02-05T08:23:39.136581+0000","last_active":"2024-02-05T08:23:45.903692+0000","last_peered":"2024-02-05T08:23:45.903692+0000","last_clean":"2024-02-05T08:23:45.903692+0000","last_became_active":"2024-02-05T08:23:07.935640+0000","last_became_peered":"2024-02-05T08:23:07.935640+0000","last_unstale":"2024-02-05T08:23:45.903692+0000","last_undegraded":"2024-02-05T08:23:45.903692+0000","last_fullsized":"2024-02-05T08:23:45.903692+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:39.136473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:39.136473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:50.220045+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":576,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.897411+0000","last_change":"2024-02-05T08:22:09.103119+0000","last_active":"2024-02-05T08:23:45.897411+0000","last_peered":"2024-02-05T08:23:45.897411+0000","last_clean":"2024-02-05T08:23:45.897411+0000","last_became_active":"2024-02-05T08:22:09.102613+0000","last_became_peered":"2024-02-05T08:22:09.102613+0000","last_unstale":"2024-02-05T08:23:45.897411+0000","last_undegraded":"2024-02-05T08:23:45.897411+0000","last_fullsized":"2024-02-05T08:23:45.897411+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:57.240901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:57.240901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":1872,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:42.019382+0000","last_change":"2024-02-05T08:23:24.028818+0000","last_active":"2024-02-05T08:23:42.019382+0000","last_peered":"2024-02-05T08:23:42.019382+0000","last_clean":"2024-02-05T08:23:42.019382+0000","last_became_active":"2024-02-05T08:23:08.921709+0000","last_became_peered":"2024-02-05T08:23:08.921709+0000","last_unstale":"2024-02-05T08:23:42.019382+0000","last_undegraded":"2024-02-05T08:23:42.019382+0000","last_fullsized":"2024-02-05T08:23:42.019382+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:23:24.028705+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:23:24.028705+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:37.647929+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1124,"num_read_kb":1119,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":507,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.896287+0000","last_change":"2024-02-05T08:22:09.130703+0000","last_active":"2024-02-05T08:23:34.896287+0000","last_peered":"2024-02-05T08:23:34.896287+0000","last_clean":"2024-02-05T08:23:34.896287+0000","last_became_active":"2024-02-05T08:22:09.106287+0000","last_became_peered":"2024-02-05T08:22:09.106287+0000","last_unstale":"2024-02-05T08:23:34.896287+0000","last_undegraded":"2024-02-05T08:23:34.896287+0000","last_fullsized":"2024-02-05T08:23:34.896287+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:21:52.253002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:52.253002+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":715,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.897321+0000","last_change":"2024-02-05T08:23:08.914768+0000","last_active":"2024-02-05T08:23:34.897321+0000","last_peered":"2024-02-05T08:23:34.897321+0000","last_clean":"2024-02-05T08:23:34.897321+0000","last_became_active":"2024-02-05T08:23:08.914353+0000","last_became_peered":"2024-02-05T08:23:08.914353+0000","last_unstale":"2024-02-05T08:23:34.897321+0000","last_undegraded":"2024-02-05T08:23:34.897321+0000","last_fullsized":"2024-02-05T08:23:34.897321+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:53.288625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:22:53.288625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:10.871064+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"382'3232","reported_seq":3271,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.926749+0000","last_change":"2024-02-05T08:23:08.921308+0000","last_active":"2024-02-05T08:23:45.926749+0000","last_peered":"2024-02-05T08:23:45.926749+0000","last_clean":"2024-02-05T08:23:45.926749+0000","last_became_active":"2024-02-05T08:23:08.920532+0000","last_became_peered":"2024-02-05T08:23:08.920532+0000","last_unstale":"2024-02-05T08:23:45.926749+0000","last_undegraded":"2024-02-05T08:23:45.926749+0000","last_fullsized":"2024-02-05T08:23:45.926749+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3232,"log_dups_size":0,"ondisk_log_size":3232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:35.331168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211812352,"num_objects":3232,"num_object_clones":0,"num_object_copies":9696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3232,"num_write_kb":206848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'130","reported_seq":763,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.903842+0000","last_change":"2024-02-05T08:23:32.482420+0000","last_active":"2024-02-05T08:23:34.903842+0000","last_peered":"2024-02-05T08:23:34.903842+0000","last_clean":"2024-02-05T08:23:34.903842+0000","last_became_active":"2024-02-05T08:22:12.143283+0000","last_became_peered":"2024-02-05T08:22:12.143283+0000","last_unstale":"2024-02-05T08:23:34.903842+0000","last_undegraded":"2024-02-05T08:23:34.903842+0000","last_fullsized":"2024-02-05T08:23:34.903842+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:32.482297+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:23:32.482297+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:35.475259+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":618,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.903114+0000","last_change":"2024-02-05T08:23:07.929555+0000","last_active":"2024-02-05T08:23:45.903114+0000","last_peered":"2024-02-05T08:23:45.903114+0000","last_clean":"2024-02-05T08:23:45.903114+0000","last_became_active":"2024-02-05T08:23:07.928514+0000","last_became_peered":"2024-02-05T08:23:07.928514+0000","last_unstale":"2024-02-05T08:23:45.903114+0000","last_undegraded":"2024-02-05T08:23:45.903114+0000","last_fullsized":"2024-02-05T08:23:45.903114+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:27.862543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:27.862543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":581,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.905820+0000","last_change":"2024-02-05T08:23:27.086997+0000","last_active":"2024-02-05T08:23:45.905820+0000","last_peered":"2024-02-05T08:23:45.905820+0000","last_clean":"2024-02-05T08:23:45.905820+0000","last_became_active":"2024-02-05T08:23:07.935692+0000","last_became_peered":"2024-02-05T08:23:07.935692+0000","last_unstale":"2024-02-05T08:23:45.905820+0000","last_undegraded":"2024-02-05T08:23:45.905820+0000","last_fullsized":"2024-02-05T08:23:45.905820+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:27.086862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:23:27.086862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:27.202163+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"381'37","reported_seq":534,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:36.866398+0000","last_change":"2024-02-05T08:23:36.866473+0000","last_active":"2024-02-05T08:23:36.866398+0000","last_peered":"2024-02-05T08:23:36.866398+0000","last_clean":"2024-02-05T08:23:36.866398+0000","last_became_active":"2024-02-05T08:23:08.917913+0000","last_became_peered":"2024-02-05T08:23:08.917913+0000","last_unstale":"2024-02-05T08:23:36.866398+0000","last_undegraded":"2024-02-05T08:23:36.866398+0000","last_fullsized":"2024-02-05T08:23:36.866398+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'37","last_scrub_stamp":"2024-02-05T08:23:36.866312+0000","last_deep_scrub":"381'37","last_deep_scrub_stamp":"2024-02-05T08:23:36.866312+0000","last_clean_scrub_stamp":"2024-02-05T08:23:36.866312+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:54.708384+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":454,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.897083+0000","last_change":"2024-02-05T08:22:49.513009+0000","last_active":"2024-02-05T08:23:34.897083+0000","last_peered":"2024-02-05T08:23:34.897083+0000","last_clean":"2024-02-05T08:23:34.897083+0000","last_became_active":"2024-02-05T08:22:09.096719+0000","last_became_peered":"2024-02-05T08:22:09.096719+0000","last_unstale":"2024-02-05T08:23:34.897083+0000","last_undegraded":"2024-02-05T08:23:34.897083+0000","last_fullsized":"2024-02-05T08:23:34.897083+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:49.512928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:22:49.512928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:19.214480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"381'3051","reported_seq":3089,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.332654+0000","last_change":"2024-02-05T08:23:08.911393+0000","last_active":"2024-02-05T08:23:43.332654+0000","last_peered":"2024-02-05T08:23:43.332654+0000","last_clean":"2024-02-05T08:23:43.332654+0000","last_became_active":"2024-02-05T08:23:08.911093+0000","last_became_peered":"2024-02-05T08:23:08.911093+0000","last_unstale":"2024-02-05T08:23:43.332654+0000","last_undegraded":"2024-02-05T08:23:43.332654+0000","last_fullsized":"2024-02-05T08:23:43.332654+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3051,"log_dups_size":0,"ondisk_log_size":3051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:10.600129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199950336,"num_objects":3051,"num_object_clones":0,"num_object_copies":9153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3051,"num_write_kb":195264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"82'130","reported_seq":754,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.904213+0000","last_change":"2024-02-05T08:23:33.451461+0000","last_active":"2024-02-05T08:23:34.904213+0000","last_peered":"2024-02-05T08:23:34.904213+0000","last_clean":"2024-02-05T08:23:34.904213+0000","last_became_active":"2024-02-05T08:19:01.815446+0000","last_became_peered":"2024-02-05T08:19:01.815446+0000","last_unstale":"2024-02-05T08:23:34.904213+0000","last_undegraded":"2024-02-05T08:23:34.904213+0000","last_fullsized":"2024-02-05T08:23:34.904213+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:33.451336+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:35.974462+0000","last_clean_scrub_stamp":"2024-02-05T08:23:33.451336+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:33.667706+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":578,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.898433+0000","last_change":"2024-02-05T08:22:57.918423+0000","last_active":"2024-02-05T08:23:34.898433+0000","last_peered":"2024-02-05T08:23:34.898433+0000","last_clean":"2024-02-05T08:23:34.898433+0000","last_became_active":"2024-02-05T08:22:09.100096+0000","last_became_peered":"2024-02-05T08:22:09.100096+0000","last_unstale":"2024-02-05T08:23:34.898433+0000","last_undegraded":"2024-02-05T08:23:34.898433+0000","last_fullsized":"2024-02-05T08:23:34.898433+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:57.918346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:57.918346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:23.039067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":731,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:38.820610+0000","last_change":"2024-02-05T08:23:38.820649+0000","last_active":"2024-02-05T08:23:38.820610+0000","last_peered":"2024-02-05T08:23:38.820610+0000","last_clean":"2024-02-05T08:23:38.820610+0000","last_became_active":"2024-02-05T08:18:34.633263+0000","last_became_peered":"2024-02-05T08:18:34.633263+0000","last_unstale":"2024-02-05T08:23:38.820610+0000","last_undegraded":"2024-02-05T08:23:38.820610+0000","last_fullsized":"2024-02-05T08:23:38.820610+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:38.820534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:23:38.820534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:42.910692+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'20","reported_seq":637,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.900229+0000","last_change":"2024-02-05T08:23:41.072770+0000","last_active":"2024-02-05T08:23:45.900229+0000","last_peered":"2024-02-05T08:23:45.900229+0000","last_clean":"2024-02-05T08:23:45.900229+0000","last_became_active":"2024-02-05T08:23:07.941561+0000","last_became_peered":"2024-02-05T08:23:07.941561+0000","last_unstale":"2024-02-05T08:23:45.900229+0000","last_undegraded":"2024-02-05T08:23:45.900229+0000","last_fullsized":"2024-02-05T08:23:45.900229+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:23:41.072637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:41.072637+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:42.721873+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"355'12","reported_seq":738,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:36.810680+0000","last_change":"2024-02-05T08:23:02.552120+0000","last_active":"2024-02-05T08:23:36.810680+0000","last_peered":"2024-02-05T08:23:36.810680+0000","last_clean":"2024-02-05T08:23:36.810680+0000","last_became_active":"2024-02-05T08:22:09.099076+0000","last_became_peered":"2024-02-05T08:22:09.099076+0000","last_unstale":"2024-02-05T08:23:36.810680+0000","last_undegraded":"2024-02-05T08:23:36.810680+0000","last_fullsized":"2024-02-05T08:23:36.810680+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'12","last_scrub_stamp":"2024-02-05T08:23:02.552025+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_clean_scrub_stamp":"2024-02-05T08:23:02.552025+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:27.568029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.e","version":"382'3563","reported_seq":3603,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:47.866040+0000","last_change":"2024-02-05T08:23:08.910748+0000","last_active":"2024-02-05T08:23:47.866040+0000","last_peered":"2024-02-05T08:23:47.866040+0000","last_clean":"2024-02-05T08:23:47.866040+0000","last_became_active":"2024-02-05T08:23:08.910360+0000","last_became_peered":"2024-02-05T08:23:08.910360+0000","last_unstale":"2024-02-05T08:23:47.866040+0000","last_undegraded":"2024-02-05T08:23:47.866040+0000","last_fullsized":"2024-02-05T08:23:47.866040+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3563,"log_dups_size":0,"ondisk_log_size":3563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:11.948868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233504768,"num_objects":3563,"num_object_clones":0,"num_object_copies":10689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3563,"num_write_kb":228032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"82'130","reported_seq":485,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.898774+0000","last_change":"2024-02-05T08:22:47.999978+0000","last_active":"2024-02-05T08:23:34.898774+0000","last_peered":"2024-02-05T08:23:34.898774+0000","last_clean":"2024-02-05T08:23:34.898774+0000","last_became_active":"2024-02-05T08:19:01.819856+0000","last_became_peered":"2024-02-05T08:19:01.819856+0000","last_unstale":"2024-02-05T08:23:34.898774+0000","last_undegraded":"2024-02-05T08:23:34.898774+0000","last_fullsized":"2024-02-05T08:23:34.898774+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:22:47.999878+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:22:47.999878+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:03.970178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":712,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:36.150748+0000","last_change":"2024-02-05T08:23:36.150803+0000","last_active":"2024-02-05T08:23:36.150748+0000","last_peered":"2024-02-05T08:23:36.150748+0000","last_clean":"2024-02-05T08:23:36.150748+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:23:36.150748+0000","last_undegraded":"2024-02-05T08:23:36.150748+0000","last_fullsized":"2024-02-05T08:23:36.150748+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:36.150674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:36.150674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:56.661921+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":49,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.904769+0000","last_change":"2024-02-05T08:23:08.914105+0000","last_active":"2024-02-05T08:23:45.904769+0000","last_peered":"2024-02-05T08:23:45.904769+0000","last_clean":"2024-02-05T08:23:45.904769+0000","last_became_active":"2024-02-05T08:23:08.913263+0000","last_became_peered":"2024-02-05T08:23:08.913263+0000","last_unstale":"2024-02-05T08:23:45.904769+0000","last_undegraded":"2024-02-05T08:23:45.904769+0000","last_fullsized":"2024-02-05T08:23:45.904769+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:22:44.187232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:22:44.187232+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:49.914710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"236'28","reported_seq":787,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.897319+0000","last_change":"2024-02-05T08:21:13.958221+0000","last_active":"2024-02-05T08:23:45.897319+0000","last_peered":"2024-02-05T08:23:45.897319+0000","last_clean":"2024-02-05T08:23:45.897319+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:23:45.897319+0000","last_undegraded":"2024-02-05T08:23:45.897319+0000","last_fullsized":"2024-02-05T08:23:45.897319+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:21:13.957934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:13.957934+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"370'13","reported_seq":279,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:36.810867+0000","last_change":"2024-02-05T08:23:33.927361+0000","last_active":"2024-02-05T08:23:36.810867+0000","last_peered":"2024-02-05T08:23:36.810867+0000","last_clean":"2024-02-05T08:23:36.810867+0000","last_became_active":"2024-02-05T08:23:07.936347+0000","last_became_peered":"2024-02-05T08:23:07.936347+0000","last_unstale":"2024-02-05T08:23:36.810867+0000","last_undegraded":"2024-02-05T08:23:36.810867+0000","last_fullsized":"2024-02-05T08:23:36.810867+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'13","last_scrub_stamp":"2024-02-05T08:23:33.927148+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:23:33.927148+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:43.180187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"381'2999","reported_seq":3037,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.360021+0000","last_change":"2024-02-05T08:23:08.921447+0000","last_active":"2024-02-05T08:23:43.360021+0000","last_peered":"2024-02-05T08:23:43.360021+0000","last_clean":"2024-02-05T08:23:43.360021+0000","last_became_active":"2024-02-05T08:23:08.921121+0000","last_became_peered":"2024-02-05T08:23:08.921121+0000","last_unstale":"2024-02-05T08:23:43.360021+0000","last_undegraded":"2024-02-05T08:23:43.360021+0000","last_fullsized":"2024-02-05T08:23:43.360021+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":2999,"log_dups_size":0,"ondisk_log_size":2999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:27.163049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196542464,"num_objects":2999,"num_object_clones":0,"num_object_copies":8997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2999,"num_write_kb":191936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"82'130","reported_seq":487,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.900189+0000","last_change":"2024-02-05T08:22:58.247977+0000","last_active":"2024-02-05T08:23:45.900189+0000","last_peered":"2024-02-05T08:23:45.900189+0000","last_clean":"2024-02-05T08:23:45.900189+0000","last_became_active":"2024-02-05T08:19:01.816923+0000","last_became_peered":"2024-02-05T08:19:01.816923+0000","last_unstale":"2024-02-05T08:23:45.900189+0000","last_undegraded":"2024-02-05T08:23:45.900189+0000","last_fullsized":"2024-02-05T08:23:45.900189+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:22:58.247896+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:22:58.247896+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:07.970282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":721,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.897761+0000","last_change":"2024-02-05T08:22:08.118133+0000","last_active":"2024-02-05T08:23:45.897761+0000","last_peered":"2024-02-05T08:23:45.897761+0000","last_clean":"2024-02-05T08:23:45.897761+0000","last_became_active":"2024-02-05T08:18:34.635908+0000","last_became_peered":"2024-02-05T08:18:34.635908+0000","last_unstale":"2024-02-05T08:23:45.897761+0000","last_undegraded":"2024-02-05T08:23:45.897761+0000","last_fullsized":"2024-02-05T08:23:45.897761+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:08.117925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:08.117925+0000","last_clean_scrub_stamp":"2024-02-05T08:22:08.117925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":223,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.902012+0000","last_change":"2024-02-05T08:22:56.880110+0000","last_active":"2024-02-05T08:23:34.902012+0000","last_peered":"2024-02-05T08:23:34.902012+0000","last_clean":"2024-02-05T08:23:34.902012+0000","last_became_active":"2024-02-05T08:22:09.101588+0000","last_became_peered":"2024-02-05T08:22:09.101588+0000","last_unstale":"2024-02-05T08:23:34.902012+0000","last_undegraded":"2024-02-05T08:23:34.902012+0000","last_fullsized":"2024-02-05T08:23:34.902012+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:22:56.880011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:22:56.880011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:23:57.419246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"236'38","reported_seq":255,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.898616+0000","last_change":"2024-02-05T08:23:08.944358+0000","last_active":"2024-02-05T08:23:34.898616+0000","last_peered":"2024-02-05T08:23:34.898616+0000","last_clean":"2024-02-05T08:23:34.898616+0000","last_became_active":"2024-02-05T08:23:08.927776+0000","last_became_peered":"2024-02-05T08:23:08.927776+0000","last_unstale":"2024-02-05T08:23:34.898616+0000","last_undegraded":"2024-02-05T08:23:34.898616+0000","last_fullsized":"2024-02-05T08:23:34.898616+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'38","last_scrub_stamp":"2024-02-05T08:22:48.999388+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:22:48.999388+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:00.979252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":717,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.897111+0000","last_change":"2024-02-05T08:22:07.078658+0000","last_active":"2024-02-05T08:23:45.897111+0000","last_peered":"2024-02-05T08:23:45.897111+0000","last_clean":"2024-02-05T08:23:45.897111+0000","last_became_active":"2024-02-05T08:18:34.636667+0000","last_became_peered":"2024-02-05T08:18:34.636667+0000","last_unstale":"2024-02-05T08:23:45.897111+0000","last_undegraded":"2024-02-05T08:23:45.897111+0000","last_fullsized":"2024-02-05T08:23:45.897111+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:07.078318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:31.133206+0000","last_clean_scrub_stamp":"2024-02-05T08:22:07.078318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"382'3330","reported_seq":3370,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.900537+0000","last_change":"2024-02-05T08:23:08.918166+0000","last_active":"2024-02-05T08:23:45.900537+0000","last_peered":"2024-02-05T08:23:45.900537+0000","last_clean":"2024-02-05T08:23:45.900537+0000","last_became_active":"2024-02-05T08:23:08.916901+0000","last_became_peered":"2024-02-05T08:23:08.916901+0000","last_unstale":"2024-02-05T08:23:45.900537+0000","last_undegraded":"2024-02-05T08:23:45.900537+0000","last_fullsized":"2024-02-05T08:23:45.900537+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3330,"log_dups_size":0,"ondisk_log_size":3330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:26.596617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218234880,"num_objects":3330,"num_object_clones":0,"num_object_copies":9990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3330,"num_write_kb":213120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'130","reported_seq":485,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.895739+0000","last_change":"2024-02-05T08:21:48.475705+0000","last_active":"2024-02-05T08:23:34.895739+0000","last_peered":"2024-02-05T08:23:34.895739+0000","last_clean":"2024-02-05T08:23:34.895739+0000","last_became_active":"2024-02-05T08:19:01.817169+0000","last_became_peered":"2024-02-05T08:19:01.817169+0000","last_unstale":"2024-02-05T08:23:34.895739+0000","last_undegraded":"2024-02-05T08:23:34.895739+0000","last_fullsized":"2024-02-05T08:23:34.895739+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:48.475580+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:48.475580+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":711,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.896154+0000","last_change":"2024-02-05T08:23:08.914041+0000","last_active":"2024-02-05T08:23:34.896154+0000","last_peered":"2024-02-05T08:23:34.896154+0000","last_clean":"2024-02-05T08:23:34.896154+0000","last_became_active":"2024-02-05T08:23:08.913735+0000","last_became_peered":"2024-02-05T08:23:08.913735+0000","last_unstale":"2024-02-05T08:23:34.896154+0000","last_undegraded":"2024-02-05T08:23:34.896154+0000","last_fullsized":"2024-02-05T08:23:34.896154+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:47.385093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:47.385093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:15.192707+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"355'9","reported_seq":952,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.899562+0000","last_change":"2024-02-05T08:22:09.114866+0000","last_active":"2024-02-05T08:23:45.899562+0000","last_peered":"2024-02-05T08:23:45.899562+0000","last_clean":"2024-02-05T08:23:45.899562+0000","last_became_active":"2024-02-05T08:22:09.106117+0000","last_became_peered":"2024-02-05T08:22:09.106117+0000","last_unstale":"2024-02-05T08:23:45.899562+0000","last_undegraded":"2024-02-05T08:23:45.899562+0000","last_fullsized":"2024-02-05T08:23:45.899562+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:22:06.070321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:06.070321+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":210,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.898766+0000","last_change":"2024-02-05T08:23:29.957871+0000","last_active":"2024-02-05T08:23:34.898766+0000","last_peered":"2024-02-05T08:23:34.898766+0000","last_clean":"2024-02-05T08:23:34.898766+0000","last_became_active":"2024-02-05T08:23:07.937468+0000","last_became_peered":"2024-02-05T08:23:07.937468+0000","last_unstale":"2024-02-05T08:23:34.898766+0000","last_undegraded":"2024-02-05T08:23:34.898766+0000","last_fullsized":"2024-02-05T08:23:34.898766+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:29.957655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.957655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:30.740349+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"236'27","reported_seq":668,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.909901+0000","last_change":"2024-02-05T08:23:30.166109+0000","last_active":"2024-02-05T08:23:45.909901+0000","last_peered":"2024-02-05T08:23:45.909901+0000","last_clean":"2024-02-05T08:23:45.909901+0000","last_became_active":"2024-02-05T08:23:07.939747+0000","last_became_peered":"2024-02-05T08:23:07.939747+0000","last_unstale":"2024-02-05T08:23:45.909901+0000","last_undegraded":"2024-02-05T08:23:45.909901+0000","last_fullsized":"2024-02-05T08:23:45.909901+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:23:30.165977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:30.165977+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:34.736139+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"381'3099","reported_seq":3137,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.199337+0000","last_change":"2024-02-05T08:23:08.914212+0000","last_active":"2024-02-05T08:23:43.199337+0000","last_peered":"2024-02-05T08:23:43.199337+0000","last_clean":"2024-02-05T08:23:43.199337+0000","last_became_active":"2024-02-05T08:23:08.913844+0000","last_became_peered":"2024-02-05T08:23:08.913844+0000","last_unstale":"2024-02-05T08:23:43.199337+0000","last_undegraded":"2024-02-05T08:23:43.199337+0000","last_fullsized":"2024-02-05T08:23:43.199337+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3099,"log_dups_size":0,"ondisk_log_size":3099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:16.096600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203096064,"num_objects":3099,"num_object_clones":0,"num_object_copies":9297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3099,"num_write_kb":198336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":209,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.898950+0000","last_change":"2024-02-05T08:22:51.956578+0000","last_active":"2024-02-05T08:23:34.898950+0000","last_peered":"2024-02-05T08:23:34.898950+0000","last_clean":"2024-02-05T08:23:34.898950+0000","last_became_active":"2024-02-05T08:18:34.627261+0000","last_became_peered":"2024-02-05T08:18:34.627261+0000","last_unstale":"2024-02-05T08:23:34.898950+0000","last_undegraded":"2024-02-05T08:23:34.898950+0000","last_fullsized":"2024-02-05T08:23:34.898950+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:51.956504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:22:51.956504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:16.868847+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":722,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.897238+0000","last_change":"2024-02-05T08:22:02.481157+0000","last_active":"2024-02-05T08:23:34.897238+0000","last_peered":"2024-02-05T08:23:34.897238+0000","last_clean":"2024-02-05T08:23:34.897238+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:23:34.897238+0000","last_undegraded":"2024-02-05T08:23:34.897238+0000","last_fullsized":"2024-02-05T08:23:34.897238+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:02.481045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:02.481045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":712,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.897194+0000","last_change":"2024-02-05T08:21:07.952159+0000","last_active":"2024-02-05T08:23:45.897194+0000","last_peered":"2024-02-05T08:23:45.897194+0000","last_clean":"2024-02-05T08:23:45.897194+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:23:45.897194+0000","last_undegraded":"2024-02-05T08:23:45.897194+0000","last_fullsized":"2024-02-05T08:23:45.897194+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:07.952031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:21:07.952031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"236'19","reported_seq":776,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.899271+0000","last_change":"2024-02-05T08:23:39.449120+0000","last_active":"2024-02-05T08:23:45.899271+0000","last_peered":"2024-02-05T08:23:45.899271+0000","last_clean":"2024-02-05T08:23:45.899271+0000","last_became_active":"2024-02-05T08:23:07.937017+0000","last_became_peered":"2024-02-05T08:23:07.937017+0000","last_unstale":"2024-02-05T08:23:45.899271+0000","last_undegraded":"2024-02-05T08:23:45.899271+0000","last_fullsized":"2024-02-05T08:23:45.899271+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:23:39.449015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:39.449015+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:50.639202+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"381'3205","reported_seq":3243,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.325070+0000","last_change":"2024-02-05T08:23:08.911747+0000","last_active":"2024-02-05T08:23:43.325070+0000","last_peered":"2024-02-05T08:23:43.325070+0000","last_clean":"2024-02-05T08:23:43.325070+0000","last_became_active":"2024-02-05T08:23:08.910712+0000","last_became_peered":"2024-02-05T08:23:08.910712+0000","last_unstale":"2024-02-05T08:23:43.325070+0000","last_undegraded":"2024-02-05T08:23:43.325070+0000","last_fullsized":"2024-02-05T08:23:43.325070+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3205,"log_dups_size":0,"ondisk_log_size":3205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:14.120525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210042880,"num_objects":3205,"num_object_clones":0,"num_object_copies":9615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3205,"num_write_kb":205120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'130","reported_seq":485,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.894165+0000","last_change":"2024-02-05T08:22:57.194319+0000","last_active":"2024-02-05T08:23:34.894165+0000","last_peered":"2024-02-05T08:23:34.894165+0000","last_clean":"2024-02-05T08:23:34.894165+0000","last_became_active":"2024-02-05T08:19:01.817126+0000","last_became_peered":"2024-02-05T08:19:01.817126+0000","last_unstale":"2024-02-05T08:23:34.894165+0000","last_undegraded":"2024-02-05T08:23:34.894165+0000","last_fullsized":"2024-02-05T08:23:34.894165+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:22:57.194237+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:22:57.194237+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:27.135405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":743,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:42.090754+0000","last_change":"2024-02-05T08:23:42.090797+0000","last_active":"2024-02-05T08:23:42.090754+0000","last_peered":"2024-02-05T08:23:42.090754+0000","last_clean":"2024-02-05T08:23:42.090754+0000","last_became_active":"2024-02-05T08:18:34.645067+0000","last_became_peered":"2024-02-05T08:18:34.645067+0000","last_unstale":"2024-02-05T08:23:42.090754+0000","last_undegraded":"2024-02-05T08:23:42.090754+0000","last_fullsized":"2024-02-05T08:23:42.090754+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:23:42.090680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:42.090680+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:43.764626+0000","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":558,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.900358+0000","last_change":"2024-02-05T08:23:08.915640+0000","last_active":"2024-02-05T08:23:45.900358+0000","last_peered":"2024-02-05T08:23:45.900358+0000","last_clean":"2024-02-05T08:23:45.900358+0000","last_became_active":"2024-02-05T08:23:08.914502+0000","last_became_peered":"2024-02-05T08:23:08.914502+0000","last_unstale":"2024-02-05T08:23:45.900358+0000","last_undegraded":"2024-02-05T08:23:45.900358+0000","last_fullsized":"2024-02-05T08:23:45.900358+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:49.228660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:49.228660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:18.915051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":449,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.897256+0000","last_change":"2024-02-05T08:23:28.612625+0000","last_active":"2024-02-05T08:23:34.897256+0000","last_peered":"2024-02-05T08:23:34.897256+0000","last_clean":"2024-02-05T08:23:34.897256+0000","last_became_active":"2024-02-05T08:23:07.929749+0000","last_became_peered":"2024-02-05T08:23:07.929749+0000","last_unstale":"2024-02-05T08:23:34.897256+0000","last_undegraded":"2024-02-05T08:23:34.897256+0000","last_fullsized":"2024-02-05T08:23:34.897256+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:28.612512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:23:28.612512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:40.525324+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"236'31","reported_seq":750,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.900389+0000","last_change":"2024-02-05T08:23:07.930476+0000","last_active":"2024-02-05T08:23:45.900389+0000","last_peered":"2024-02-05T08:23:45.900389+0000","last_clean":"2024-02-05T08:23:45.900389+0000","last_became_active":"2024-02-05T08:23:07.930130+0000","last_became_peered":"2024-02-05T08:23:07.930130+0000","last_unstale":"2024-02-05T08:23:45.900389+0000","last_undegraded":"2024-02-05T08:23:45.900389+0000","last_fullsized":"2024-02-05T08:23:45.900389+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'31","last_scrub_stamp":"2024-02-05T08:19:24.342770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:19:24.342770+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"381'3162","reported_seq":3200,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:43.203585+0000","last_change":"2024-02-05T08:23:08.920424+0000","last_active":"2024-02-05T08:23:43.203585+0000","last_peered":"2024-02-05T08:23:43.203585+0000","last_clean":"2024-02-05T08:23:43.203585+0000","last_became_active":"2024-02-05T08:23:08.920062+0000","last_became_peered":"2024-02-05T08:23:08.920062+0000","last_unstale":"2024-02-05T08:23:43.203585+0000","last_undegraded":"2024-02-05T08:23:43.203585+0000","last_fullsized":"2024-02-05T08:23:43.203585+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3162,"log_dups_size":0,"ondisk_log_size":3162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:30.032294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207224832,"num_objects":3162,"num_object_clones":0,"num_object_copies":9486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3162,"num_write_kb":202368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"82'130","reported_seq":1009,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.897856+0000","last_change":"2024-02-05T08:21:19.070269+0000","last_active":"2024-02-05T08:23:34.897856+0000","last_peered":"2024-02-05T08:23:34.897856+0000","last_clean":"2024-02-05T08:23:34.897856+0000","last_became_active":"2024-02-05T08:19:01.816744+0000","last_became_peered":"2024-02-05T08:19:01.816744+0000","last_unstale":"2024-02-05T08:23:34.897856+0000","last_undegraded":"2024-02-05T08:23:34.897856+0000","last_fullsized":"2024-02-05T08:23:34.897856+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:21:19.069725+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:21:19.069725+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":693,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.899607+0000","last_change":"2024-02-05T08:23:07.937930+0000","last_active":"2024-02-05T08:23:45.899607+0000","last_peered":"2024-02-05T08:23:45.899607+0000","last_clean":"2024-02-05T08:23:45.899607+0000","last_became_active":"2024-02-05T08:23:07.937641+0000","last_became_peered":"2024-02-05T08:23:07.937641+0000","last_unstale":"2024-02-05T08:23:45.899607+0000","last_undegraded":"2024-02-05T08:23:45.899607+0000","last_fullsized":"2024-02-05T08:23:45.899607+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:16.282641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:19:16.282641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":964,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:36.866535+0000","last_change":"2024-02-05T08:23:36.866535+0000","last_active":"2024-02-05T08:23:36.866535+0000","last_peered":"2024-02-05T08:23:36.866535+0000","last_clean":"2024-02-05T08:23:36.866535+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:23:36.866535+0000","last_undegraded":"2024-02-05T08:23:36.866535+0000","last_fullsized":"2024-02-05T08:23:36.866535+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:23:36.866292+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:23:36.866292+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:04.936923+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":219,"reported_epoch":381,"state":"active+clean","last_fresh":"2024-02-05T08:23:34.899579+0000","last_change":"2024-02-05T08:23:08.929039+0000","last_active":"2024-02-05T08:23:34.899579+0000","last_peered":"2024-02-05T08:23:34.899579+0000","last_clean":"2024-02-05T08:23:34.899579+0000","last_became_active":"2024-02-05T08:23:08.928775+0000","last_became_peered":"2024-02-05T08:23:08.928775+0000","last_unstale":"2024-02-05T08:23:34.899579+0000","last_undegraded":"2024-02-05T08:23:34.899579+0000","last_fullsized":"2024-02-05T08:23:34.899579+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:07.643746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:22:07.643746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"363'25","reported_seq":823,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-02-05T08:23:45.899064+0000","last_change":"2024-02-05T08:23:03.301185+0000","last_active":"2024-02-05T08:23:45.899064+0000","last_peered":"2024-02-05T08:23:45.899064+0000","last_clean":"2024-02-05T08:23:45.899064+0000","last_became_active":"2024-02-05T08:18:34.637413+0000","last_became_peered":"2024-02-05T08:18:34.637413+0000","last_unstale":"2024-02-05T08:23:45.899064+0000","last_undegraded":"2024-02-05T08:23:45.899064+0000","last_fullsized":"2024-02-05T08:23:45.899064+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'25","last_scrub_stamp":"2024-02-05T08:23:03.301090+0000","last_deep_scrub":"363'25","last_deep_scrub_stamp":"2024-02-05T08:23:03.301090+0000","last_clean_scrub_stamp":"2024-02-05T08:23:03.301090+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:04.111161+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3311927296,"num_objects":50536,"num_object_clones":0,"num_object_copies":151608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50536,"num_write_kb":3234304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10067312640,"data_stored":10067312640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50536,"ondisk_log_size":50536,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1124,"num_read_kb":1119,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":6562,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037993,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2023288,"kb_used_data":1388208,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":91721864,"statfs":{"total":95995035648,"available":93923188736,"internally_reserved":0,"allocated":1421524992,"data_stored":1421080198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":650312138},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:23:00 2024","interfaces":[{"interface":"back","average":{"1min":1.869,"5min":1.1930000000000001,"15min":1.665},"min":{"1min":0.23200000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":20.344999999999999,"5min":20.344999999999999,"15min":60.127000000000002},"last":1.5860000000000001},{"interface":"front","average":{"1min":2.0409999999999999,"5min":1.218,"15min":1.7609999999999999},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.23200000000000001},"max":{"1min":21.347000000000001,"5min":21.347000000000001,"15min":58.283000000000001},"last":1.667}]},{"osd":1,"last update":"Mon Feb 5 08:18:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.544,"15min":1.512},"min":{"1min":0.25600000000000001,"5min":0.23699999999999999,"15min":0.22900000000000001},"max":{"1min":2.7810000000000001,"5min":27.518999999999998,"15min":38.399999999999999},"last":0.40400000000000003},{"interface":"front","average":{"1min":1.083,"5min":1.4330000000000001,"15min":1.3680000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.22500000000000001},"max":{"1min":4.2409999999999997,"5min":27.503,"15min":38.195999999999998},"last":0.64600000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:23:00 2024","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":1.1519999999999999,"15min":1.149},"min":{"1min":0.28899999999999998,"5min":0.26300000000000001,"15min":0.24099999999999999},"max":{"1min":20.288,"5min":20.288,"15min":20.288},"last":2.5779999999999998},{"interface":"front","average":{"1min":1.9019999999999999,"5min":1.157,"15min":1.2290000000000001},"min":{"1min":0.28399999999999997,"5min":0.248,"15min":0.24299999999999999},"max":{"1min":21.247,"5min":21.247,"15min":21.247},"last":2.3079999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:23:00 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.91000000000000003,"15min":1.343},"min":{"1min":0.33100000000000002,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":3.7869999999999999,"5min":4.6840000000000002,"15min":58.515000000000001},"last":1.9630000000000001},{"interface":"front","average":{"1min":1.081,"5min":0.95799999999999996,"15min":1.363},"min":{"1min":0.309,"5min":0.28699999999999998,"15min":0.27000000000000002},"max":{"1min":3.8239999999999998,"5min":3.8239999999999998,"15min":58.225999999999999},"last":2.7959999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:23:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.014,"15min":1.0700000000000001},"min":{"1min":0.24399999999999999,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":3.004,"5min":4.4509999999999996,"15min":49.601999999999997},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.145,"15min":1.1379999999999999},"min":{"1min":0.32600000000000001,"5min":0.249,"15min":0.249},"max":{"1min":4.0679999999999996,"5min":7.835,"15min":49.795999999999999},"last":0.315}]},{"osd":5,"last update":"Mon Feb 5 08:23:00 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":0.98199999999999998,"15min":1.1359999999999999},"min":{"1min":0.27600000000000002,"5min":0.26800000000000002,"15min":0.24299999999999999},"max":{"1min":3.9929999999999999,"5min":4.5579999999999998,"15min":49.914000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.1240000000000001,"15min":1.1599999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.27000000000000002},"max":{"1min":2.3900000000000001,"5min":9.4459999999999997,"15min":52.256},"last":0.85499999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:23:00 2024","interfaces":[{"interface":"back","average":{"1min":1.048,"5min":2.5529999999999999,"15min":1.4410000000000001},"min":{"1min":0.34999999999999998,"5min":0.29199999999999998,"15min":0.28599999999999998},"max":{"1min":3.7909999999999999,"5min":84.254999999999995,"15min":84.254999999999995},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":2.6949999999999998,"15min":1.472},"min":{"1min":0.42499999999999999,"5min":0.36499999999999999,"15min":0.27300000000000002},"max":{"1min":4.5510000000000002,"5min":84.667000000000002,"15min":84.667000000000002},"last":0.47899999999999998}]}]},{"osd":6,"up_from":51,"seq":219043332335,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2091732,"kb_used_data":1546892,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":91653420,"statfs":{"total":95995035648,"available":93853102080,"internally_reserved":0,"allocated":1584017408,"data_stored":1583607575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":557906374},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,6,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.3180000000000001,"15min":1.8819999999999999},"min":{"1min":0.34300000000000003,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.4089999999999998,"5min":14.836,"15min":102.819},"last":1.1619999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.90800000000000003,"15min":1.7809999999999999},"min":{"1min":0.312,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":1.375,"5min":4.9480000000000004,"15min":102.75},"last":1.375}]},{"osd":1,"last update":"Mon Feb 5 08:19:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.90900000000000003,"15min":1.169},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.22700000000000001},"max":{"1min":1.1970000000000001,"5min":6.0229999999999997,"15min":23.773},"last":1.74},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.96299999999999997,"15min":1.1319999999999999},"min":{"1min":0.30299999999999999,"5min":0.26300000000000001,"15min":0.23400000000000001},"max":{"1min":3.0249999999999999,"5min":5.0209999999999999,"15min":23.719000000000001},"last":1.1910000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:23:42 2024","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.377,"15min":1.2350000000000001},"min":{"1min":0.27000000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":25.751000000000001,"5min":25.751000000000001,"15min":25.751000000000001},"last":1.9279999999999999},{"interface":"front","average":{"1min":1.78,"5min":1.222,"15min":1.1379999999999999},"min":{"1min":0.34699999999999998,"5min":0.28199999999999997,"15min":0.23400000000000001},"max":{"1min":26.210999999999999,"5min":26.210999999999999,"15min":26.210999999999999},"last":1.7909999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:23:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":1.3839999999999999,"15min":1.514},"min":{"1min":0.32300000000000001,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":27.113,"5min":27.113,"15min":102.438},"last":1.24},{"interface":"front","average":{"1min":1.8500000000000001,"5min":1.379,"15min":1.472},"min":{"1min":0.32300000000000001,"5min":0.29299999999999998,"15min":0.29099999999999998},"max":{"1min":27.209,"5min":27.209,"15min":102.288},"last":1.9710000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":1.841,"15min":1.137},"min":{"1min":0.317,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.8089999999999999,"5min":46.125,"15min":46.125},"last":1.5640000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.1960000000000002,"15min":1.2709999999999999},"min":{"1min":0.27900000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.3639999999999999,"5min":48.817999999999998,"15min":48.817999999999998},"last":1.1299999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":1.1160000000000001,"15min":1.216},"min":{"1min":0.26700000000000002,"5min":0.26000000000000001,"15min":0.24199999999999999},"max":{"1min":1.4490000000000001,"5min":7.6760000000000002,"15min":43.643000000000001},"last":1.4490000000000001},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.264,"15min":1.2509999999999999},"min":{"1min":0.33700000000000002,"5min":0.246,"15min":0.246},"max":{"1min":4.5999999999999996,"5min":9.2279999999999998,"15min":43.893000000000001},"last":1.097}]},{"osd":7,"last update":"Mon Feb 5 08:23:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.228,"15min":1.0589999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.29999999999999999},"max":{"1min":3.1829999999999998,"5min":9.0229999999999997,"15min":24.728999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.226,"5min":1.6459999999999999,"15min":1.22},"min":{"1min":0.34300000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":6.0629999999999997,"5min":14.988,"15min":21.91},"last":1.2170000000000001}]}]},{"osd":1,"up_from":366,"seq":1571958030345,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2388056,"kb_used_data":2283472,"kb_used_omap":4,"kb_used_meta":104571,"kb_avail":91357096,"statfs":{"total":95995035648,"available":93549666304,"internally_reserved":0,"allocated":2338275328,"data_stored":2337797630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":107081382},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.5739999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6339999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5880000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.792}]}]},{"osd":0,"up_from":0,"seq":34359738637,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411587,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2014264,"kb_used_data":1355568,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":91730888,"statfs":{"total":95995035648,"available":93932429312,"internally_reserved":0,"allocated":1388101632,"data_stored":1387656422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":674494918},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.038,"15min":2.1520000000000001},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2.77,"5min":30.731999999999999,"15min":107.155},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":1.28,"15min":2.3740000000000001},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.6800000000000002,"5min":30.838999999999999,"15min":107.221},"last":0.83099999999999996}]},{"osd":1,"last update":"Mon Feb 5 08:18:54 2024","interfaces":[{"interface":"back","average":{"1min":2.4540000000000002,"5min":1.167,"15min":1.5860000000000001},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.17999999999999999},"max":{"1min":35.695999999999998,"5min":35.695999999999998,"15min":55.942},"last":2.327},{"interface":"front","average":{"1min":2.9209999999999998,"5min":1.581,"15min":1.752},"min":{"1min":0.216,"5min":0.216,"15min":0.193},"max":{"1min":30.765000000000001,"5min":30.765000000000001,"15min":55.960999999999999},"last":2.1230000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.96699999999999997,"15min":1.7809999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":2.444,"5min":6.5039999999999996,"15min":68.108000000000004},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.80000000000000004,"15min":1.5680000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.377,"5min":6.6920000000000002,"15min":67.945999999999998},"last":2.0739999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:22:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5660000000000001,"5min":1.3160000000000001,"15min":1.556},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":5.4809999999999999,"5min":7.6189999999999998,"15min":31.596},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.3380000000000001,"15min":1.385},"min":{"1min":0.36799999999999999,"5min":0.29499999999999998,"15min":0.252},"max":{"1min":4.1349999999999998,"5min":9.5060000000000002,"15min":31.57},"last":1.004}]},{"osd":5,"last update":"Mon Feb 5 08:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":1.238,"15min":1.663},"min":{"1min":0.316,"5min":0.27300000000000002,"15min":0.246},"max":{"1min":1.958,"5min":9.5500000000000007,"15min":58.799999999999997},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":1.355,"15min":1.613},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26900000000000002},"max":{"1min":2.7229999999999999,"5min":12.925000000000001,"15min":58.615000000000002},"last":1.206}]},{"osd":6,"last update":"Mon Feb 5 08:22:50 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.323,"15min":1.4059999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.29199999999999998},"max":{"1min":5.444,"5min":7.1059999999999999,"15min":15.859},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.214,"5min":1.2949999999999999,"15min":1.2330000000000001},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":4.1059999999999999,"5min":9.1739999999999995,"15min":9.1739999999999995},"last":0.70199999999999996}]},{"osd":7,"last update":"Mon Feb 5 08:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":1.087,"15min":1.3340000000000001},"min":{"1min":0.34799999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":2.1789999999999998,"5min":6.1109999999999998,"15min":15.986000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.23,"5min":1.3919999999999999,"15min":1.573},"min":{"1min":0.36599999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":3.8959999999999999,"5min":11.945,"15min":15.951000000000001},"last":1.1699999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117246,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1969864,"kb_used_data":1375040,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91775288,"statfs":{"total":95995035648,"available":93977894912,"internally_reserved":0,"allocated":1408040960,"data_stored":1407580302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":609089993},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.90100000000000002,"15min":1.6339999999999999},"min":{"1min":0.216,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":3.3319999999999999,"5min":11.747999999999999,"15min":52.831000000000003},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.376,"5min":0.86599999999999999,"15min":1.6899999999999999},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.78100000000000003,"5min":11.618,"15min":53.859999999999999},"last":0.46700000000000003}]},{"osd":1,"last update":"Mon Feb 5 08:19:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.93500000000000005,"15min":1.5109999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.217},"max":{"1min":3.3119999999999998,"5min":23.154,"15min":51.405999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.89200000000000002,"15min":1.8260000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.17399999999999999},"max":{"1min":3.54,"5min":23.207000000000001,"15min":51.189999999999998},"last":0.435}]},{"osd":2,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":1.3520000000000001,"15min":1.5129999999999999},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.9529999999999998,"5min":35.277000000000001,"15min":35.277000000000001},"last":2.3940000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":1.2749999999999999,"15min":1.3300000000000001},"min":{"1min":0.27200000000000002,"5min":0.23300000000000001,"15min":0.23200000000000001},"max":{"1min":3.778,"5min":35.442999999999998,"15min":35.442999999999998},"last":0.73199999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.95499999999999996,"15min":1.135},"min":{"1min":0.29699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.1949999999999998,"5min":6.298,"15min":7.7229999999999999},"last":1.98},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.91600000000000004,"15min":1.129},"min":{"1min":0.33300000000000002,"5min":0.32900000000000001,"15min":0.29599999999999999},"max":{"1min":3.153,"5min":5.8159999999999998,"15min":10.539999999999999},"last":1.0509999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:23:13 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":1.143,"15min":1.8620000000000001},"min":{"1min":0.32000000000000001,"5min":0.28899999999999998,"15min":0.28000000000000003},"max":{"1min":3.7730000000000001,"5min":11.519,"15min":117.432},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.99099999999999999,"15min":1.7949999999999999},"min":{"1min":0.33500000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.044,"5min":11.071,"15min":114.828},"last":0.96299999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":1.5309999999999999,"15min":1.3420000000000001},"min":{"1min":0.36299999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.585,"5min":54.219999999999999,"15min":54.219999999999999},"last":9.6129999999999995},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.7010000000000001,"15min":1.3959999999999999},"min":{"1min":0.41999999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.9889999999999999,"5min":54.530999999999999,"15min":54.530999999999999},"last":2.4340000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:22:53 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.111,"15min":1.8029999999999999},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":3.7250000000000001,"5min":6.3730000000000002,"15min":114.84999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.96299999999999997,"15min":1.7649999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.8849999999999998,"5min":4.4290000000000003,"15min":114.871},"last":0.91800000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953722,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1997052,"kb_used_data":1334192,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":91748100,"statfs":{"total":95995035648,"available":93950054400,"internally_reserved":0,"allocated":1366212608,"data_stored":1365774278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.1140000000000001,"15min":1.4590000000000001},"min":{"1min":0.23300000000000001,"5min":0.192,"15min":0.192},"max":{"1min":2.093,"5min":8.6470000000000002,"15min":39.552},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.1950000000000001,"15min":1.5149999999999999},"min":{"1min":0.30499999999999999,"5min":0.25,"15min":0.21099999999999999},"max":{"1min":1.304,"5min":5.165,"15min":39.350999999999999},"last":0.40000000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:19:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.97299999999999998,"15min":1.486},"min":{"1min":0.30399999999999999,"5min":0.23300000000000001,"15min":0.21099999999999999},"max":{"1min":1.4590000000000001,"5min":8.3320000000000007,"15min":125.054},"last":1.028},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.89300000000000002,"15min":1.387},"min":{"1min":0.309,"5min":0.25700000000000001,"15min":0.23499999999999999},"max":{"1min":1.1910000000000001,"5min":6.2370000000000001,"15min":125.08799999999999},"last":1.1499999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.1639999999999999,"15min":1.069},"min":{"1min":0.33100000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1559999999999997,"5min":5.9160000000000004,"15min":5.9640000000000004},"last":1.8300000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.107,"15min":1.1359999999999999},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.26300000000000001},"max":{"1min":3.6589999999999998,"5min":4.4290000000000003,"15min":7.032},"last":1.3320000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.3120000000000001,"15min":1.748},"min":{"1min":0.315,"5min":0.312,"15min":0.24399999999999999},"max":{"1min":1.2509999999999999,"5min":8.9269999999999996,"15min":57.204000000000001},"last":2.1800000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.1830000000000001,"15min":1.675},"min":{"1min":0.32700000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.4209999999999998,"5min":4.2599999999999998,"15min":57.606999999999999},"last":2.8250000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:23:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":1.3260000000000001,"15min":1.123},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.224},"max":{"1min":3.1379999999999999,"5min":6.1909999999999998,"15min":34.167000000000002},"last":0.34399999999999997},{"interface":"front","average":{"1min":1.087,"5min":1.294,"15min":1.179},"min":{"1min":0.28599999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":6.3360000000000003,"5min":7.851,"15min":36.051000000000002},"last":0.434}]},{"osd":6,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":1.4019999999999999,"15min":1.032},"min":{"1min":0.32600000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.3360000000000001,"5min":6.375,"15min":6.375},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":1.2749999999999999,"15min":0.94999999999999996},"min":{"1min":0.35699999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.4630000000000001,"5min":6.9690000000000003,"15min":6.9690000000000003},"last":0.378}]},{"osd":7,"last update":"Mon Feb 5 08:22:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.357,"15min":1.012},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.27000000000000002},"max":{"1min":3.226,"5min":12.683999999999999,"15min":12.683999999999999},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.466,"15min":1.016},"min":{"1min":0.35099999999999998,"5min":0.33900000000000002,"15min":0.27900000000000003},"max":{"1min":7.6820000000000004,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.997}]}]},{"osd":5,"up_from":42,"seq":180388626675,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1442660,"kb_used_data":793692,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92302492,"statfs":{"total":95995035648,"available":94517751808,"internally_reserved":0,"allocated":812740608,"data_stored":812315956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:23:01 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.0389999999999999,"15min":1.6439999999999999},"min":{"1min":0.26600000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":6.4660000000000002,"5min":6.4660000000000002,"15min":46.962000000000003},"last":1.079},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0880000000000001,"15min":1.575},"min":{"1min":0.33700000000000002,"5min":0.26400000000000001,"15min":0.20499999999999999},"max":{"1min":5.0819999999999999,"5min":5.0819999999999999,"15min":47.067},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Feb 5 08:18:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.321,"15min":1.163},"min":{"1min":0.42799999999999999,"5min":0.22500000000000001,"15min":0.215},"max":{"1min":2.8090000000000002,"5min":5.8940000000000001,"15min":46.200000000000003},"last":0.65000000000000002},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.1859999999999999,"15min":1.23},"min":{"1min":0.29699999999999999,"5min":0.246,"15min":0.22900000000000001},"max":{"1min":4.7969999999999997,"5min":5.2210000000000001,"15min":46.343000000000004},"last":1.01}]},{"osd":2,"last update":"Mon Feb 5 08:23:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.258,"15min":1.264},"min":{"1min":0.32000000000000001,"5min":0.245,"15min":0.245},"max":{"1min":7.6600000000000001,"5min":17.09,"15min":17.09},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.268,"5min":1.258,"15min":1.3740000000000001},"min":{"1min":0.26100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.6440000000000001,"5min":17.515999999999998,"15min":17.515999999999998},"last":1.208}]},{"osd":3,"last update":"Mon Feb 5 08:23:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.1379999999999999,"15min":1.4159999999999999},"min":{"1min":0.35299999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":7.6950000000000003,"5min":7.6950000000000003,"15min":46.569000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.048,"15min":1.5349999999999999},"min":{"1min":0.29399999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":6.5519999999999996,"5min":6.5519999999999996,"15min":47.820999999999998},"last":1.0489999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:23:01 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.071,"15min":0.96899999999999997},"min":{"1min":0.25900000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.173,"5min":7.7389999999999999,"15min":7.7389999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.2270000000000001,"15min":0.97499999999999998},"min":{"1min":0.34699999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":7.609,"5min":7.609,"15min":7.609},"last":6.0979999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:22:43 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.6419999999999999,"15min":1.1060000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.26600000000000001},"max":{"1min":8.734,"5min":30.581,"15min":30.581},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.601,"15min":1.101},"min":{"1min":0.38200000000000001,"5min":0.32900000000000001,"15min":0.316},"max":{"1min":7.7229999999999999,"5min":30.620999999999999,"15min":30.620999999999999},"last":1.248}]},{"osd":7,"last update":"Mon Feb 5 08:23:01 2024","interfaces":[{"interface":"back","average":{"1min":3.2440000000000002,"5min":1.5820000000000001,"15min":1.2649999999999999},"min":{"1min":0.33000000000000002,"5min":0.28799999999999998,"15min":0.25900000000000001},"max":{"1min":34.695,"5min":34.695,"15min":34.695},"last":3.0459999999999998},{"interface":"front","average":{"1min":2.843,"5min":1.5289999999999999,"15min":1.1930000000000001},"min":{"1min":0.38200000000000001,"5min":0.308,"15min":0.308},"max":{"1min":34.514000000000003,"5min":34.514000000000003,"15min":34.514000000000003},"last":0.61599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1960509440,"data_stored":1960509440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1412956160,"data_stored":1412956160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1419116544,"data_stored":1419116544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1381761024,"data_stored":1381761024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":833814528,"data_stored":833814528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1625423872,"data_stored":1625423872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1433731072,"data_stored":1433731072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:23:54.953 INFO:tasks.thrashosds.thrasher:Setting 3.0 to [3, 1, 5] 2024-02-05T08:23:54.953 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.0', '3', '1', '5'] 2024-02-05T08:23:54.953 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pg-upmap 3.0 3 1 5 2024-02-05T08:23:54.962 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:54.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:54.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:23:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:23:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:54 smithi196 ceph-mon[33018]: pgmap v860: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-02-05T08:23:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:54 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:23:55.098 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:55.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:55.136 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:55.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:55.237 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:55.282 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:55.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:55.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:55.384 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:55.430 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:55.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:55.481 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:55.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:55.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:55.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:55.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:55.710 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:55.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:55.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:55.736 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 36 16 67375 67359 116.914 86.875 0.0203218 0.00854852 2024-02-05T08:23:55.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:55.848 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:55.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:55.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:55.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:55.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:55.987 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:55.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:56.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:56.141 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:56.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:56.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:56.243 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:56.286 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:56.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:56.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:56.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:56.436 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:56.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:56.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:56.538 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:56.578 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:56.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:56.679 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:56.718 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:56.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:56.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:56.736 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 37 15 68796 68781 116.156 88.875 0.00599043 0.0086047 2024-02-05T08:23:56.756 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:23:56.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:56.855 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:56.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:56.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:56.993 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:56.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:56.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:57.095 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:57.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:57.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:57.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:57.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:57.269 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:57.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:57.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:56 smithi134 ceph-mon[30086]: pgmap v861: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T08:23:57.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:56 smithi134 ceph-mon[39278]: pgmap v861: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T08:23:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:56 smithi196 ceph-mon[33018]: pgmap v861: 158 pgs: 158 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T08:23:57.370 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:23:57.416 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:57.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:23:57.517 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:57.555 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:57.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:57.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:57.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:57.696 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:57.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:57.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:57.736 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 38 16 70218 70202 115.437 88.8125 0.00971455 0.00865789 2024-02-05T08:23:57.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:57.834 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:57.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:57.935 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:23:57.971 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:57.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:57.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:23:58.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:23:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:23:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:23:58.073 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:58.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:58.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:58.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:58.214 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:58.253 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:58.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:58.355 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:58.394 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:58.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:58.497 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:58.530 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:58.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:58.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:58.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:23:58.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:58.669 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:58.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:23:58.736 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 39 16 71575 71559 114.651 84.8125 0.00472108 0.00871753 2024-02-05T08:23:58.771 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:58.807 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:58.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:58.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:23:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:58.945 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:58.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:23:59.046 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:59.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:59.073 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:59.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:59.174 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:59.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:59.210 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:59.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:59.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:23:58 smithi134 ceph-mon[30086]: pgmap v862: 158 pgs: 158 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-02-05T08:23:59.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:23:58 smithi134 ceph-mon[39278]: pgmap v862: 158 pgs: 158 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-02-05T08:23:59.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:23:58 smithi196 ceph-mon[33018]: pgmap v862: 158 pgs: 158 active+clean; 4.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-02-05T08:23:59.350 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:59.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:59.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:59.453 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:23:59.492 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:59.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:59.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:23:59.594 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:23:59.630 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:23:59.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:59.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:23:59.732 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:23:59.736 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:23:59.737327+0000 min lat: 0.00126536 max lat: 0.205027 avg lat: 0.00878097 2024-02-05T08:23:59.736 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:23:59.736 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 40 16 72876 72860 113.818 81.3125 0.012511 0.00878097 2024-02-05T08:23:59.766 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:59.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:59.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:23:59.867 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:23:59.905 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:23:59.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:23:59.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:23:59.961 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:23:59.974 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:23:59.974 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:24:00.007 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:00.052 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:00.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:00.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:00.192 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:00.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:00.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:00.334 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:00.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:00.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:00.474 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:00.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:00.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:00.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:00.624 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:00.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:00.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:00.737 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 41 16 74241 74225 113.122 85.3125 0.0108459 0.00883503 2024-02-05T08:24:00.761 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:00.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:00.864 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:00.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:00.911 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:00.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:01.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:01.045 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:01.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:01.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:01.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:01.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:00 smithi134 ceph-mon[30086]: pgmap v863: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1022 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-02-05T08:24:01.163 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:00 smithi134 ceph-mon[39278]: pgmap v863: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1022 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-02-05T08:24:01.183 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:00 smithi196 ceph-mon[33018]: pgmap v863: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1022 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-02-05T08:24:01.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:01.185 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:01.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:01.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:01.332 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:01.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:01.434 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:01.474 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:01.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:01.576 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:01.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:01.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:01.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:01.638 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T08:24:01.712 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:01.736 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 42 16 75560 75544 112.392 82.4375 0.0121956 0.00889122 2024-02-05T08:24:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:01.760 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:01.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:01.861 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:01.914 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:01.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:01.947 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:24:02.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:02.046 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:02.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:02.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:02.148 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:02.191 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:02.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:02.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:02.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:01 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1456629615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:24:02.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:01 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:24:02.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:01 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1456629615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:24:02.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:01 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:24:02.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:01 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1456629615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:24:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:01 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:24:02.332 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:02.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:02.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:02.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:02.470 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:02.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:02.571 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:02.611 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:02.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:02.677 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:02.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:02.739 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 43 16 76784 76768 111.556 76.5 0.00899203 0.00895957 2024-02-05T08:24:02.748 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:02.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:02.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:02.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:02.889 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:02.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:02.991 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:03.028 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:03.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:03.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:03.130 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:03.164 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:03.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:03.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:02 smithi134 ceph-mon[30086]: pgmap v864: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:24:03.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:02 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:24:03.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:02 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:24:03.225 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:02 smithi134 ceph-mon[30086]: osdmap e384: 8 total, 8 up, 7 in 2024-02-05T08:24:03.225 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:02 smithi134 ceph-mon[39278]: pgmap v864: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:24:03.225 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:02 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:24:03.225 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:02 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:24:03.225 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:02 smithi134 ceph-mon[39278]: osdmap e384: 8 total, 8 up, 7 in 2024-02-05T08:24:03.266 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:03.300 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:03.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:02 smithi196 ceph-mon[33018]: pgmap v864: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T08:24:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:02 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:24:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:02 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:24:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:02 smithi196 ceph-mon[33018]: osdmap e384: 8 total, 8 up, 7 in 2024-02-05T08:24:03.402 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:03.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:03.443 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:03.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:03.544 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:03.587 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:03.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:03.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:03.727 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:03.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:03.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:03.737 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 44 16 78155 78139 110.968 85.6875 0.00976272 0.00900689 2024-02-05T08:24:03.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:03.867 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:03.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:03.968 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:04.030 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:04.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:04.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:04.171 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:04.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:04.197 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:03 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:24:04.198 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:03 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:24:04.198 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:03 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:24:04.198 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:03 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:24:04.274 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:04.314 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:04.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:04.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:03 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:24:04.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:03 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:24:04.316 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:24:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:24:04.416 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:04.464 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:04.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:04.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:04.566 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:04.603 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:04.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:04.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:04.705 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:04.737 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 45 16 79533 79517 110.416 86.125 0.0128327 0.00905198 2024-02-05T08:24:04.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:04.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:04.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:04.887 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:04.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:04.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:04.988 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:05.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:05.041 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:05.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:05.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:05.231 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:05.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:05.234 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:04 smithi134 ceph-mon[39278]: pgmap v866: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-05T08:24:05.234 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:04 smithi134 ceph-mon[39278]: osdmap e385: 8 total, 8 up, 7 in 2024-02-05T08:24:05.234 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:04 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:05.234 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:04 smithi134 ceph-mon[30086]: pgmap v866: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-05T08:24:05.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:04 smithi134 ceph-mon[30086]: osdmap e385: 8 total, 8 up, 7 in 2024-02-05T08:24:05.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:04 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:05.274 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:24:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:04 smithi196 ceph-mon[33018]: pgmap v866: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-02-05T08:24:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:04 smithi196 ceph-mon[33018]: osdmap e385: 8 total, 8 up, 7 in 2024-02-05T08:24:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:04 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:05.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:05.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:05.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:05.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:05.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:05.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:05.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:05.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:05.613 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:05.657 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:05.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:05.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:05.737 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 46 15 80882 80867 109.85 84.375 0.0133232 0.0090989 2024-02-05T08:24:05.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:05.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:05.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:05.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:05.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:05.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:05.929 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:05.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:05.982 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:24:06.011 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:06.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:06.045 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:24:06.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:06.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:06.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:06.080 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:24:06.087 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:06.131 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:05 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1280516841' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:24:06.131 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:24:06.132 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:05 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1280516841' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:24:06.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:06.173 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:24:06.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:06.199 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:06.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:06.215 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:24:06.262 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:24:06.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:06.304 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:24:06.309 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:24:06.309 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:24:06.309 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:24:06.309 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:24:06.309 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:24:06.309 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:24:06.309 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:24:06.309 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:24:06.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:05 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1280516841' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:24:06.336 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:06.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:06.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:06.473 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:06.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:06.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:06.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:06.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:06.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:06.738 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 47 16 82082 82066 109.106 74.9375 0.0131904 0.00916075 2024-02-05T08:24:06.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:06.751 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:06.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:06.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:06.887 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:06.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:06.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:set 3.0 pg_upmap mapping to [3,1,5] 2024-02-05T08:24:06.978 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:06 smithi134 ceph-mon[30086]: pgmap v868: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-02-05T08:24:06.978 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:06 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3019753155' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.0", "id": [3, 1, 5]}]: dispatch 2024-02-05T08:24:06.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:06 smithi134 ceph-mon[39278]: pgmap v868: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-02-05T08:24:06.978 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:06 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3019753155' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.0", "id": [3, 1, 5]}]: dispatch 2024-02-05T08:24:06.989 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:07.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:07.037 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:07.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:07.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:07.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:07.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:07.275 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:06 smithi196 ceph-mon[33018]: pgmap v868: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-02-05T08:24:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:06 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3019753155' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.0", "id": [3, 1, 5]}]: dispatch 2024-02-05T08:24:07.315 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:07.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:07.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:07.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:07.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:07.452 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:07.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:07.553 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:07.596 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:07.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:07.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:07.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:07.739 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 48 16 83443 83427 108.605 85.0625 0.00681107 0.00920309 2024-02-05T08:24:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:07.741 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:07.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:07.776 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:24:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:24:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:24:07.842 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:07.879 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:07.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:07.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:07.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:08.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:08.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:08.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:08.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:08.162 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:08.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:08.264 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:08.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:07 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3019753155' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.0", "id": [3, 1, 5]}]': finished 2024-02-05T08:24:08.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:07 smithi134 ceph-mon[30086]: osdmap e386: 8 total, 8 up, 7 in 2024-02-05T08:24:08.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:07 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3019753155' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.0", "id": [3, 1, 5]}]': finished 2024-02-05T08:24:08.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:07 smithi134 ceph-mon[39278]: osdmap e386: 8 total, 8 up, 7 in 2024-02-05T08:24:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:08.304 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:08.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:07 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3019753155' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.0", "id": [3, 1, 5]}]': finished 2024-02-05T08:24:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:07 smithi196 ceph-mon[33018]: osdmap e386: 8 total, 8 up, 7 in 2024-02-05T08:24:08.405 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:08.446 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:08.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:08.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:08.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:08.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:08.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:08.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:08.738 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:08.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:08.738 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 49 16 84801 84785 108.12 84.875 0.00906167 0.00924434 2024-02-05T08:24:08.839 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:08.881 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:08.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:08.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:08.983 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:09.023 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:09.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:09.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:09.125 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:09.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:09.165 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:09.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:09.219 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:08 smithi134 ceph-mon[30086]: pgmap v870: 158 pgs: 158 active+clean; 4.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-05T08:24:09.220 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:08 smithi134 ceph-mon[30086]: osdmap e387: 8 total, 8 up, 7 in 2024-02-05T08:24:09.220 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:24:09.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:08 smithi134 ceph-mon[39278]: pgmap v870: 158 pgs: 158 active+clean; 4.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-05T08:24:09.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:08 smithi134 ceph-mon[39278]: osdmap e387: 8 total, 8 up, 7 in 2024-02-05T08:24:09.220 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:08 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:24:09.267 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:08 smithi196 ceph-mon[33018]: pgmap v870: 158 pgs: 158 active+clean; 4.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.16k op/s 2024-02-05T08:24:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:08 smithi196 ceph-mon[33018]: osdmap e387: 8 total, 8 up, 7 in 2024-02-05T08:24:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:24:09.314 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:09.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:09.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:09.416 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:09.456 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:09.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:09.558 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:09.602 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:09.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:09.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:09.738 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 50 16 86173 86157 107.673 85.75 0.00881251 0.00928299 2024-02-05T08:24:09.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:09.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:09.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:09.881 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:09.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:09.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:09.984 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:10.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:10.028 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:10.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:10.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:10.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:10.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:10.272 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:10.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:10.311 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:10.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:10.412 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:10.456 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:10.456 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:24:10.456 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:24:10.456 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-02-05T08:24:10.457 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-02-05T08:24:10.457 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:24:10.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:10.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:10.560 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:10.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:10.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:10.598 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:24:10.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:10.738 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 51 15 87409 87394 107.077 77.3125 0.0116533 0.0093337 2024-02-05T08:24:10.758 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:10.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:10.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:10.860 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:10.902 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:10.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:10.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:11.003 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:11.045 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:11.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:11.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:11.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:11.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:11.185 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:11.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:11.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:10 smithi134 ceph-mon[30086]: pgmap v872: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s; 66 B/s, 0 objects/s recovering 2024-02-05T08:24:11.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:10 smithi134 ceph-mon[39278]: pgmap v872: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s; 66 B/s, 0 objects/s recovering 2024-02-05T08:24:11.286 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:10 smithi196 ceph-mon[33018]: pgmap v872: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s; 66 B/s, 0 objects/s recovering 2024-02-05T08:24:11.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:11.327 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:11.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:11.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:11.462 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:11.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:11.564 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:11.606 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:11.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:11.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:11.708 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:11.738 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 52 16 88737 88721 106.613 82.9375 0.0146728 0.00937535 2024-02-05T08:24:11.747 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:11.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:11.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:11.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:11.884 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:11.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:11.986 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:12.026 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:12.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:12.103 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T08:24:12.128 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:12.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:12.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:12.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:12.260 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:12.307 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:12.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:12.409 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:12.440 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:12.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:12.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:12.542 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:12.588 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:12.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:12.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:12.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:12.723 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:12.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:12.737 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:12.740 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 53 16 89992 89976 106.078 78.4375 0.0261247 0.00942173 2024-02-05T08:24:12.791 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:12.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:12.856 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:12.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:12.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:12.958 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:12.995 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:12.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:12.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:13.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:12 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:24:13.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:12 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:24:13.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:12 smithi134 ceph-mon[30086]: pgmap v873: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 641 B/s rd, 59 MiB/s wr, 939 op/s; 49 B/s, 0 objects/s recovering 2024-02-05T08:24:13.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:12 smithi134 ceph-mon[30086]: osdmap e388: 8 total, 8 up, 7 in 2024-02-05T08:24:13.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:12 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:24:13.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:12 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:24:13.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:12 smithi134 ceph-mon[39278]: pgmap v873: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 641 B/s rd, 59 MiB/s wr, 939 op/s; 49 B/s, 0 objects/s recovering 2024-02-05T08:24:13.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:12 smithi134 ceph-mon[39278]: osdmap e388: 8 total, 8 up, 7 in 2024-02-05T08:24:13.077 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 7, 6, 3] out_osds: [0] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-02-05T08:24:13.077 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:24:13.077 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-05T08:24:13.077 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-05T08:24:13.077 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-05T08:24:13.078 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-05T08:24:13.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:13.149 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:13.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:13.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:13.251 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:13.286 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:13.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:13.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:12 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:24:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:12 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:24:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:12 smithi196 ceph-mon[33018]: pgmap v873: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 641 B/s rd, 59 MiB/s wr, 939 op/s; 49 B/s, 0 objects/s recovering 2024-02-05T08:24:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:12 smithi196 ceph-mon[33018]: osdmap e388: 8 total, 8 up, 7 in 2024-02-05T08:24:13.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:13.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:13.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:13.492 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:24:13.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:13.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:13.569 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:13.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:13.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:13.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:13.713 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:13.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:13.740 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 54 16 91187 91171 105.496 74.6875 0.00456732 0.00947409 2024-02-05T08:24:13.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:13.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:13.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:13.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:13.936 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:24:13.957 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:13.990 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:13.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:13.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:14.091 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:14.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:14.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:14.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:14.193 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:13 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3047537501' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:24:14.193 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:13 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:24:14.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:13 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3047537501' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:24:14.193 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:13 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:24:14.229 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:14.274 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:14.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:14.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:13 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3047537501' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:24:14.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:13 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:24:14.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:24:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:24:14.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:14.413 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:14.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:14.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:14.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:14.555 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:14.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:14.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:14.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:14.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:14.740 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 55 16 92533 92517 105.108 84.125 0.00734906 0.00950955 2024-02-05T08:24:14.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:14.830 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:14.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:14.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:14.932 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:14.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:14.979 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:14.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:15.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:15.103 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:14 smithi134 ceph-mon[39278]: pgmap v875: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 57 B/s, 0 objects/s recovering 2024-02-05T08:24:15.103 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:14 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:24:15.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:14 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:24:15.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:14 smithi134 ceph-mon[39278]: osdmap e389: 8 total, 8 up, 7 in 2024-02-05T08:24:15.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:14 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:15.104 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:14 smithi134 ceph-mon[30086]: pgmap v875: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 57 B/s, 0 objects/s recovering 2024-02-05T08:24:15.104 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:14 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:24:15.104 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:14 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:24:15.104 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:14 smithi134 ceph-mon[30086]: osdmap e389: 8 total, 8 up, 7 in 2024-02-05T08:24:15.104 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:14 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:15.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:15.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:15.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:15.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:14 smithi196 ceph-mon[33018]: pgmap v875: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 110 MiB/s wr, 1.76k op/s; 57 B/s, 0 objects/s recovering 2024-02-05T08:24:15.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:14 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:24:15.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:14 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:24:15.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:14 smithi196 ceph-mon[33018]: osdmap e389: 8 total, 8 up, 7 in 2024-02-05T08:24:15.214 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:14 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:15.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:15.265 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:15.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:15.367 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:15.402 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:15.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:15.504 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:15.536 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:15.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:15.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:15.679 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:15.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:15.724 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:24:15.724 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-05T08:24:15.724 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:24:15.740 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 56 16 93814 93798 104.66 80.0625 0.0122515 0.00955022 2024-02-05T08:24:15.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:15.817 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:15.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:15.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:15.919 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:15.962 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:15.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:15.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:16.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:16.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:16.105 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:16.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:16.195 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:15 smithi134 ceph-mon[39278]: 11.2 scrub starts 2024-02-05T08:24:16.196 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:15 smithi134 ceph-mon[30086]: 11.2 scrub starts 2024-02-05T08:24:16.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:16.253 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:16.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:16.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:15 smithi196 ceph-mon[33018]: 11.2 scrub starts 2024-02-05T08:24:16.355 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:16.392 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:16.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:16.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:16.494 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:16.529 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:16.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:16.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:16.667 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:16.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:16.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:16.740 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 57 15 95077 95062 104.21 79 0.016786 0.00959115 2024-02-05T08:24:16.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:16.808 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:16.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:16.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:16.953 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:16.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:17.054 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:17.091 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:17.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:17.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:17.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:17.226 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:17.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:17.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:16 smithi134 ceph-mon[30086]: pgmap v877: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 50 B/s, 0 objects/s recovering 2024-02-05T08:24:17.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:16 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:24:17.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:16 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:24:17.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:16 smithi134 ceph-mon[39278]: pgmap v877: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 50 B/s, 0 objects/s recovering 2024-02-05T08:24:17.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:16 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:24:17.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:16 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:24:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:16 smithi196 ceph-mon[33018]: pgmap v877: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 93 MiB/s wr, 1.49k op/s; 50 B/s, 0 objects/s recovering 2024-02-05T08:24:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:16 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:24:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:16 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:24:17.328 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:17.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:17.376 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:17.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:17.424 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:24:17.464 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:24:17.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:17.509 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:24:17.512 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:17.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:17.544 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:24:17.587 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:24:17.614 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:17.633 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:24:17.658 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:17.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:17.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:17.681 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:24:17.724 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:24:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:24:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:24:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:24:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:24:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:24:17.735 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:24:17.736 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:24:17.736 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:24:17.740 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 58 16 96314 96298 103.745 77.25 0.0388506 0.00963349 2024-02-05T08:24:17.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:17.810 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:17.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:17.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:17.836 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:24:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:24:17.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:17.942 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:24:17.942 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:24:17.955 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:17.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:17.987 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:24:18.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:18.097 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[30086]: 4.1 scrub starts 2024-02-05T08:24:18.097 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[30086]: 4.1 scrub ok 2024-02-05T08:24:18.097 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[30086]: 6.12 scrub starts 2024-02-05T08:24:18.097 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[30086]: 6.12 scrub ok 2024-02-05T08:24:18.097 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2281776065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:24:18.097 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[30086]: 2.1 scrub starts 2024-02-05T08:24:18.097 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[30086]: 2.1 scrub ok 2024-02-05T08:24:18.097 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[39278]: 4.1 scrub starts 2024-02-05T08:24:18.097 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[39278]: 4.1 scrub ok 2024-02-05T08:24:18.097 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[39278]: 6.12 scrub starts 2024-02-05T08:24:18.098 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[39278]: 6.12 scrub ok 2024-02-05T08:24:18.098 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2281776065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:24:18.098 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[39278]: 2.1 scrub starts 2024-02-05T08:24:18.098 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:17 smithi134 ceph-mon[39278]: 2.1 scrub ok 2024-02-05T08:24:18.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:18.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:18.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:18.200 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:18.251 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:18.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:17 smithi196 ceph-mon[33018]: 4.1 scrub starts 2024-02-05T08:24:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:17 smithi196 ceph-mon[33018]: 4.1 scrub ok 2024-02-05T08:24:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:17 smithi196 ceph-mon[33018]: 6.12 scrub starts 2024-02-05T08:24:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:17 smithi196 ceph-mon[33018]: 6.12 scrub ok 2024-02-05T08:24:18.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:17 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2281776065' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:24:18.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:17 smithi196 ceph-mon[33018]: 2.1 scrub starts 2024-02-05T08:24:18.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:17 smithi196 ceph-mon[33018]: 2.1 scrub ok 2024-02-05T08:24:18.352 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:18.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:18.390 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:18.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:18.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:18.529 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:18.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:18.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:18.632 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:18.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:18.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:18.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:18.740 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 59 15 97552 97537 103.299 77.4375 0.013718 0.00967611 2024-02-05T08:24:18.773 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:18.808 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:18.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:18.910 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:18.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T08:24:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:18.954 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:18.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:19.055 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:19.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:19.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:19.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:19.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:19.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:19.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:19.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:19.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:18 smithi134 ceph-mon[30086]: 11.2 scrub ok 2024-02-05T08:24:19.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:18 smithi134 ceph-mon[30086]: pgmap v878: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 908 op/s 2024-02-05T08:24:19.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:18 smithi134 ceph-mon[30086]: 4.4 scrub starts 2024-02-05T08:24:19.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:18 smithi134 ceph-mon[30086]: 4.4 scrub ok 2024-02-05T08:24:19.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:18 smithi134 ceph-mon[30086]: 5.8 scrub starts 2024-02-05T08:24:19.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:18 smithi134 ceph-mon[30086]: 5.8 scrub ok 2024-02-05T08:24:19.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:18 smithi134 ceph-mon[39278]: 11.2 scrub ok 2024-02-05T08:24:19.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:18 smithi134 ceph-mon[39278]: pgmap v878: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 908 op/s 2024-02-05T08:24:19.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:18 smithi134 ceph-mon[39278]: 4.4 scrub starts 2024-02-05T08:24:19.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:18 smithi134 ceph-mon[39278]: 4.4 scrub ok 2024-02-05T08:24:19.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:18 smithi134 ceph-mon[39278]: 5.8 scrub starts 2024-02-05T08:24:19.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:18 smithi134 ceph-mon[39278]: 5.8 scrub ok 2024-02-05T08:24:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:18 smithi196 ceph-mon[33018]: 11.2 scrub ok 2024-02-05T08:24:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:18 smithi196 ceph-mon[33018]: pgmap v878: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 908 op/s 2024-02-05T08:24:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:18 smithi196 ceph-mon[33018]: 4.4 scrub starts 2024-02-05T08:24:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:18 smithi196 ceph-mon[33018]: 4.4 scrub ok 2024-02-05T08:24:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:18 smithi196 ceph-mon[33018]: 5.8 scrub starts 2024-02-05T08:24:19.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:18 smithi196 ceph-mon[33018]: 5.8 scrub ok 2024-02-05T08:24:19.332 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:19.367 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:19.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:19.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:19.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:19.506 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:19.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:19.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:19.543 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:19.608 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:19.644 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:19.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:19.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:19.741 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:24:19.741937+0000 min lat: 0.00126536 max lat: 0.205027 avg lat: 0.00971529 2024-02-05T08:24:19.742 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:24:19.742 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 60 15 98808 98793 102.886 78.5 0.0114762 0.00971529 2024-02-05T08:24:19.746 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:19.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:19.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:19.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:19.927 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:19.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:19.984 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:19 smithi134 ceph-mon[30086]: 4.14 scrub starts 2024-02-05T08:24:19.984 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:19 smithi134 ceph-mon[30086]: 4.14 scrub ok 2024-02-05T08:24:19.984 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:19 smithi134 ceph-mon[39278]: 4.14 scrub starts 2024-02-05T08:24:19.984 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:19 smithi134 ceph-mon[39278]: 4.14 scrub ok 2024-02-05T08:24:20.029 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:20.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:20.067 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:20.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:20.168 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:20.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:20.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:20.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:19 smithi196 ceph-mon[33018]: 4.14 scrub starts 2024-02-05T08:24:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:19 smithi196 ceph-mon[33018]: 4.14 scrub ok 2024-02-05T08:24:20.339 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:20.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:20.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:20.406 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:20.441 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:20.483 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:20.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:20.584 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:20.636 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:20.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:20.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:20.738 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:20.741 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 61 16 100029 100013 102.449 76.25 0.0224507 0.00975129 2024-02-05T08:24:20.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:20.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:20.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:20.907 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:20.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:20.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:21.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:21.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:21.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:21.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:21.146 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:21.182 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:21.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:21.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[30086]: 6.e scrub starts 2024-02-05T08:24:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[30086]: 6.e scrub ok 2024-02-05T08:24:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[30086]: 3.10 scrub starts 2024-02-05T08:24:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[30086]: 3.10 scrub ok 2024-02-05T08:24:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[30086]: 4.6 scrub starts 2024-02-05T08:24:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[30086]: 4.6 scrub ok 2024-02-05T08:24:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[30086]: 3.13 scrub starts 2024-02-05T08:24:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[30086]: 3.13 scrub ok 2024-02-05T08:24:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[30086]: pgmap v879: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-02-05T08:24:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[30086]: 3.1e scrub starts 2024-02-05T08:24:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[30086]: 3.1e scrub ok 2024-02-05T08:24:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[30086]: 5.b scrub starts 2024-02-05T08:24:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[30086]: 5.b scrub ok 2024-02-05T08:24:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[39278]: 6.e scrub starts 2024-02-05T08:24:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[39278]: 6.e scrub ok 2024-02-05T08:24:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[39278]: 3.10 scrub starts 2024-02-05T08:24:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[39278]: 3.10 scrub ok 2024-02-05T08:24:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[39278]: 4.6 scrub starts 2024-02-05T08:24:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[39278]: 4.6 scrub ok 2024-02-05T08:24:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[39278]: 3.13 scrub starts 2024-02-05T08:24:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[39278]: 3.13 scrub ok 2024-02-05T08:24:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[39278]: pgmap v879: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-02-05T08:24:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[39278]: 3.1e scrub starts 2024-02-05T08:24:21.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[39278]: 3.1e scrub ok 2024-02-05T08:24:21.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[39278]: 5.b scrub starts 2024-02-05T08:24:21.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:20 smithi134 ceph-mon[39278]: 5.b scrub ok 2024-02-05T08:24:21.284 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:21.307 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:21.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:21.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:20 smithi196 ceph-mon[33018]: 6.e scrub starts 2024-02-05T08:24:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:20 smithi196 ceph-mon[33018]: 6.e scrub ok 2024-02-05T08:24:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:20 smithi196 ceph-mon[33018]: 3.10 scrub starts 2024-02-05T08:24:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:20 smithi196 ceph-mon[33018]: 3.10 scrub ok 2024-02-05T08:24:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:20 smithi196 ceph-mon[33018]: 4.6 scrub starts 2024-02-05T08:24:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:20 smithi196 ceph-mon[33018]: 4.6 scrub ok 2024-02-05T08:24:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:20 smithi196 ceph-mon[33018]: 3.13 scrub starts 2024-02-05T08:24:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:20 smithi196 ceph-mon[33018]: 3.13 scrub ok 2024-02-05T08:24:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:20 smithi196 ceph-mon[33018]: pgmap v879: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.50k op/s 2024-02-05T08:24:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:20 smithi196 ceph-mon[33018]: 3.1e scrub starts 2024-02-05T08:24:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:20 smithi196 ceph-mon[33018]: 3.1e scrub ok 2024-02-05T08:24:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:20 smithi196 ceph-mon[33018]: 5.b scrub starts 2024-02-05T08:24:21.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:20 smithi196 ceph-mon[33018]: 5.b scrub ok 2024-02-05T08:24:21.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:21.447 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:21.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:21.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:21.533 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:24:21.544 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 7, 6, 3] out_osds: [0] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-02-05T08:24:21.544 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:24:21.545 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-02-05T08:24:21.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:24:21.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:21.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:21.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:21.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:21.688 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:21.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:21.742 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:21.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:21.742 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 62 16 101256 101240 102.033 76.6875 0.0232629 0.00979418 2024-02-05T08:24:21.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:21.894 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:21.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:21.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:21.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:22.029 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:22.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:22.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:22.131 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:22.169 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:22.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:22.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:22.271 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:22.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:21 smithi134 ceph-mon[30086]: 4.7 scrub starts 2024-02-05T08:24:22.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:21 smithi134 ceph-mon[30086]: 4.7 scrub ok 2024-02-05T08:24:22.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:21 smithi134 ceph-mon[39278]: 4.7 scrub starts 2024-02-05T08:24:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:21 smithi134 ceph-mon[39278]: 4.7 scrub ok 2024-02-05T08:24:22.295 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:22.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:21 smithi196 ceph-mon[33018]: 4.7 scrub starts 2024-02-05T08:24:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:21 smithi196 ceph-mon[33018]: 4.7 scrub ok 2024-02-05T08:24:22.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:22.435 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:22.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:22.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:22.537 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:22.574 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:22.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:22.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:22.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:22.712 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:22.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:22.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:22.741 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 63 16 102472 102456 101.619 76 0.00283724 0.00983313 2024-02-05T08:24:22.814 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:22.846 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:22.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:22.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:22.948 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:22.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:22.983 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:22.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:23.084 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:23.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:23.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:23.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:23.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:23.258 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:23.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:23.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:23.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:22 smithi134 ceph-mon[30086]: pgmap v880: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-02-05T08:24:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:22 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:24:23.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:22 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:24:23.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:22 smithi134 ceph-mon[39278]: pgmap v880: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-02-05T08:24:23.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:22 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:24:23.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:22 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:24:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:22 smithi196 ceph-mon[33018]: pgmap v880: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.27k op/s 2024-02-05T08:24:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:22 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:24:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:22 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:24:23.361 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:23.393 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:23.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:23.494 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:23.538 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:23.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:23.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:23.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:23.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:23.741 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 64 16 103634 103618 101.166 72.625 0.0191644 0.00988023 2024-02-05T08:24:23.778 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:23.816 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:23.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:23.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:23.918 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:23.956 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:23.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:23.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:24.058 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:24.093 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:24.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:24.195 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:24.218 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:24.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:24.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:24.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[30086]: 3.1a scrub starts 2024-02-05T08:24:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[30086]: 3.1a scrub ok 2024-02-05T08:24:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[30086]: 1.a scrub starts 2024-02-05T08:24:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[30086]: 1.a scrub ok 2024-02-05T08:24:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[30086]: 1.9 scrub starts 2024-02-05T08:24:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[30086]: 1.9 scrub ok 2024-02-05T08:24:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:24:24.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[39278]: 3.1a scrub starts 2024-02-05T08:24:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[39278]: 3.1a scrub ok 2024-02-05T08:24:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[39278]: 1.a scrub starts 2024-02-05T08:24:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[39278]: 1.a scrub ok 2024-02-05T08:24:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[39278]: 1.9 scrub starts 2024-02-05T08:24:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[39278]: 1.9 scrub ok 2024-02-05T08:24:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:24 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:24:24.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:24:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:24:24.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:24 smithi196 ceph-mon[33018]: 3.1a scrub starts 2024-02-05T08:24:24.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:24 smithi196 ceph-mon[33018]: 3.1a scrub ok 2024-02-05T08:24:24.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:24 smithi196 ceph-mon[33018]: 1.a scrub starts 2024-02-05T08:24:24.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:24 smithi196 ceph-mon[33018]: 1.a scrub ok 2024-02-05T08:24:24.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:24 smithi196 ceph-mon[33018]: 1.9 scrub starts 2024-02-05T08:24:24.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:24 smithi196 ceph-mon[33018]: 1.9 scrub ok 2024-02-05T08:24:24.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:24 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:24:24.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:24.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:24.360 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:24.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:24.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:24.501 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:24.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:24.603 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:24.626 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:24.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:24.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:24.728 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:24.741 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 65 16 104754 104738 100.687 70 0.00652435 0.00992723 2024-02-05T08:24:24.764 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:24.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:24.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:24.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:24.890 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:24.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:24.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:24.991 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:25.031 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:25.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:25.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:25.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:25.158 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:25.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:25.260 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:25.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:25 smithi134 ceph-mon[30086]: pgmap v881: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T08:24:25.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:25 smithi134 ceph-mon[30086]: 5.1e scrub starts 2024-02-05T08:24:25.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:25 smithi134 ceph-mon[30086]: 5.1e scrub ok 2024-02-05T08:24:25.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:25 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:25.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:25 smithi134 ceph-mon[30086]: 3.19 scrub starts 2024-02-05T08:24:25.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:25 smithi134 ceph-mon[30086]: 3.19 scrub ok 2024-02-05T08:24:25.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:25 smithi134 ceph-mon[39278]: pgmap v881: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T08:24:25.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:25 smithi134 ceph-mon[39278]: 5.1e scrub starts 2024-02-05T08:24:25.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:25 smithi134 ceph-mon[39278]: 5.1e scrub ok 2024-02-05T08:24:25.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:25 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:25.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:25 smithi134 ceph-mon[39278]: 3.19 scrub starts 2024-02-05T08:24:25.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:25 smithi134 ceph-mon[39278]: 3.19 scrub ok 2024-02-05T08:24:25.299 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:25.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:25.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:25 smithi196 ceph-mon[33018]: pgmap v881: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T08:24:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:25 smithi196 ceph-mon[33018]: 5.1e scrub starts 2024-02-05T08:24:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:25 smithi196 ceph-mon[33018]: 5.1e scrub ok 2024-02-05T08:24:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:25 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:25 smithi196 ceph-mon[33018]: 3.19 scrub starts 2024-02-05T08:24:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:25 smithi196 ceph-mon[33018]: 3.19 scrub ok 2024-02-05T08:24:25.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:25.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:25.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:25.542 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:25.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:25.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:25.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:25.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:25.772 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 66 16 105830 105814 100.18 67.25 0.00645291 0.00997775 2024-02-05T08:24:25.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:25.807 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:25.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:25.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:25.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:25.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:26.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:26.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:26.089 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:26.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:26.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:26.237 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:26.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:26.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:26 smithi134 ceph-mon[30086]: 4.8 scrub starts 2024-02-05T08:24:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:26 smithi134 ceph-mon[30086]: 4.8 scrub ok 2024-02-05T08:24:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:26 smithi134 ceph-mon[30086]: 6.9 scrub starts 2024-02-05T08:24:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:26 smithi134 ceph-mon[30086]: 6.9 scrub ok 2024-02-05T08:24:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:26 smithi134 ceph-mon[30086]: 5.12 scrub starts 2024-02-05T08:24:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:26 smithi134 ceph-mon[30086]: 5.12 scrub ok 2024-02-05T08:24:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:26 smithi134 ceph-mon[39278]: 4.8 scrub starts 2024-02-05T08:24:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:26 smithi134 ceph-mon[39278]: 4.8 scrub ok 2024-02-05T08:24:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:26 smithi134 ceph-mon[39278]: 6.9 scrub starts 2024-02-05T08:24:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:26 smithi134 ceph-mon[39278]: 6.9 scrub ok 2024-02-05T08:24:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:26 smithi134 ceph-mon[39278]: 5.12 scrub starts 2024-02-05T08:24:26.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:26 smithi134 ceph-mon[39278]: 5.12 scrub ok 2024-02-05T08:24:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:26 smithi196 ceph-mon[33018]: 4.8 scrub starts 2024-02-05T08:24:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:26 smithi196 ceph-mon[33018]: 4.8 scrub ok 2024-02-05T08:24:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:26 smithi196 ceph-mon[33018]: 6.9 scrub starts 2024-02-05T08:24:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:26 smithi196 ceph-mon[33018]: 6.9 scrub ok 2024-02-05T08:24:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:26 smithi196 ceph-mon[33018]: 5.12 scrub starts 2024-02-05T08:24:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:26 smithi196 ceph-mon[33018]: 5.12 scrub ok 2024-02-05T08:24:26.340 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:26.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:26.379 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:26.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:26.480 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:26.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:26.504 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:26.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:26.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:26.629 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:26.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:26.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:26.731 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:26.741 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 67 15 106865 106850 99.6515 64.75 0.0147244 0.0100308 2024-02-05T08:24:26.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:26.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:26.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:26.873 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:26.910 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:26.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:26.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:27.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:27.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:27.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:27.152 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:27.187 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:27.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:27.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[30086]: 6.a scrub starts 2024-02-05T08:24:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[30086]: 6.a scrub ok 2024-02-05T08:24:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[30086]: pgmap v882: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 854 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-02-05T08:24:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[30086]: 5.9 scrub starts 2024-02-05T08:24:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[30086]: 5.9 scrub ok 2024-02-05T08:24:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[30086]: 6.1f scrub starts 2024-02-05T08:24:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[30086]: 6.1f scrub ok 2024-02-05T08:24:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[39278]: 6.a scrub starts 2024-02-05T08:24:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[39278]: 6.a scrub ok 2024-02-05T08:24:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[39278]: pgmap v882: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 854 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-02-05T08:24:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[39278]: 5.9 scrub starts 2024-02-05T08:24:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[39278]: 5.9 scrub ok 2024-02-05T08:24:27.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[39278]: 6.1f scrub starts 2024-02-05T08:24:27.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-mon[39278]: 6.1f scrub ok 2024-02-05T08:24:27.289 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:27 smithi196 ceph-mon[33018]: 6.a scrub starts 2024-02-05T08:24:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:27 smithi196 ceph-mon[33018]: 6.a scrub ok 2024-02-05T08:24:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:27 smithi196 ceph-mon[33018]: pgmap v882: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 854 B/s rd, 65 MiB/s wr, 1.05k op/s 2024-02-05T08:24:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:27 smithi196 ceph-mon[33018]: 5.9 scrub starts 2024-02-05T08:24:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:27 smithi196 ceph-mon[33018]: 5.9 scrub ok 2024-02-05T08:24:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:27 smithi196 ceph-mon[33018]: 6.1f scrub starts 2024-02-05T08:24:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:27 smithi196 ceph-mon[33018]: 6.1f scrub ok 2024-02-05T08:24:27.329 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:27.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:27.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:27.468 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:27.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:27.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:27.570 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:27.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:27.610 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:27.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:27.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:27.742 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 68 16 108069 108053 99.2917 75.1875 0.0205216 0.0100668 2024-02-05T08:24:27.746 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:27.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:27.848 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:27.888 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:27.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:27.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:28.024 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:24:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:24:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:24:28.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:28.025 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:28.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:28.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:28.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:28.163 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:28.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:28.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:28 smithi134 ceph-mon[30086]: 4.d scrub starts 2024-02-05T08:24:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:28 smithi134 ceph-mon[30086]: 4.d scrub ok 2024-02-05T08:24:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:28 smithi134 ceph-mon[30086]: 6.8 scrub starts 2024-02-05T08:24:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:28 smithi134 ceph-mon[30086]: 6.8 scrub ok 2024-02-05T08:24:28.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:28 smithi134 ceph-mon[39278]: 4.d scrub starts 2024-02-05T08:24:28.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:28 smithi134 ceph-mon[39278]: 4.d scrub ok 2024-02-05T08:24:28.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:28 smithi134 ceph-mon[39278]: 6.8 scrub starts 2024-02-05T08:24:28.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:28 smithi134 ceph-mon[39278]: 6.8 scrub ok 2024-02-05T08:24:28.299 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:28.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:28.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:28 smithi196 ceph-mon[33018]: 4.d scrub starts 2024-02-05T08:24:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:28 smithi196 ceph-mon[33018]: 4.d scrub ok 2024-02-05T08:24:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:28 smithi196 ceph-mon[33018]: 6.8 scrub starts 2024-02-05T08:24:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:28 smithi196 ceph-mon[33018]: 6.8 scrub ok 2024-02-05T08:24:28.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:28.440 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:28.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:28.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:28.542 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:28.588 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:28.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:28.690 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:28.727 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:28.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:28.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:28.742 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 69 15 109422 109407 99.079 84.625 0.00944313 0.0100887 2024-02-05T08:24:28.829 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:28.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:28.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:28.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:28.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:29.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:29.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:29.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:29.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:29.145 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:29.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:29.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:29.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:29.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:29.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:29 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:24:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:29 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:24:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:29 smithi196 ceph-mon[33018]: pgmap v883: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-02-05T08:24:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:29 smithi196 ceph-mon[33018]: 5.4 scrub starts 2024-02-05T08:24:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:29 smithi196 ceph-mon[33018]: 5.4 scrub ok 2024-02-05T08:24:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:29 smithi196 ceph-mon[33018]: 4.11 scrub starts 2024-02-05T08:24:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:29 smithi196 ceph-mon[33018]: 4.11 scrub ok 2024-02-05T08:24:29.386 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:29.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:29.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:29.433 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:24:29.433 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:24:29.433 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[30086]: pgmap v883: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-02-05T08:24:29.433 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[30086]: 5.4 scrub starts 2024-02-05T08:24:29.434 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[30086]: 5.4 scrub ok 2024-02-05T08:24:29.434 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[30086]: 4.11 scrub starts 2024-02-05T08:24:29.434 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[30086]: 4.11 scrub ok 2024-02-05T08:24:29.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:24:29.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:24:29.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[39278]: pgmap v883: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-02-05T08:24:29.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[39278]: 5.4 scrub starts 2024-02-05T08:24:29.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[39278]: 5.4 scrub ok 2024-02-05T08:24:29.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[39278]: 4.11 scrub starts 2024-02-05T08:24:29.434 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:29 smithi134 ceph-mon[39278]: 4.11 scrub ok 2024-02-05T08:24:29.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:29.560 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:29.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:29.586 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:29.586 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:29.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:29.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:29.715 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:29.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:29.742 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 70 16 110727 110711 98.8274 81.5 0.00942323 0.010114 2024-02-05T08:24:29.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:29.852 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:29.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:29.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:29.992 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:29.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:30.094 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:30.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:30.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:30.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:30.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:30.273 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:30.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:30 smithi196 ceph-mon[33018]: 4.12 scrub starts 2024-02-05T08:24:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:30 smithi196 ceph-mon[33018]: 4.12 scrub ok 2024-02-05T08:24:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:30 smithi196 ceph-mon[33018]: 6.1 scrub starts 2024-02-05T08:24:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:30 smithi196 ceph-mon[33018]: 6.1 scrub ok 2024-02-05T08:24:30.363 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:30 smithi134 ceph-mon[30086]: 4.12 scrub starts 2024-02-05T08:24:30.363 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:30 smithi134 ceph-mon[30086]: 4.12 scrub ok 2024-02-05T08:24:30.363 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:30 smithi134 ceph-mon[30086]: 6.1 scrub starts 2024-02-05T08:24:30.363 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:30 smithi134 ceph-mon[30086]: 6.1 scrub ok 2024-02-05T08:24:30.363 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:30 smithi134 ceph-mon[39278]: 4.12 scrub starts 2024-02-05T08:24:30.364 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:30 smithi134 ceph-mon[39278]: 4.12 scrub ok 2024-02-05T08:24:30.364 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:30 smithi134 ceph-mon[39278]: 6.1 scrub starts 2024-02-05T08:24:30.364 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:30 smithi134 ceph-mon[39278]: 6.1 scrub ok 2024-02-05T08:24:30.374 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:30.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:30.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:30.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:30.515 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:30.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:30.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:30.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:30.653 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:30.694 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:30.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:30.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:30.742 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 71 16 111946 111930 98.5085 76.1875 0.00974148 0.0101465 2024-02-05T08:24:30.796 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:30.834 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:30.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:30.937 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:30.971 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:30.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:30.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:31.066 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:24:31.076 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:31.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:31.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:31.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:31.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:31.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:31.293 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[30086]: 11.1 scrub starts 2024-02-05T08:24:31.293 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[30086]: pgmap v884: 158 pgs: 158 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-02-05T08:24:31.293 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/433125909' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:24:31.293 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[30086]: 4.19 scrub starts 2024-02-05T08:24:31.293 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[30086]: 4.19 scrub ok 2024-02-05T08:24:31.293 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[30086]: 3.6 scrub starts 2024-02-05T08:24:31.293 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[30086]: 3.6 scrub ok 2024-02-05T08:24:31.293 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[30086]: 11.d scrub starts 2024-02-05T08:24:31.294 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[39278]: 11.1 scrub starts 2024-02-05T08:24:31.294 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[39278]: pgmap v884: 158 pgs: 158 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-02-05T08:24:31.294 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/433125909' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:24:31.294 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[39278]: 4.19 scrub starts 2024-02-05T08:24:31.294 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[39278]: 4.19 scrub ok 2024-02-05T08:24:31.294 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[39278]: 3.6 scrub starts 2024-02-05T08:24:31.294 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[39278]: 3.6 scrub ok 2024-02-05T08:24:31.294 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:31 smithi134 ceph-mon[39278]: 11.d scrub starts 2024-02-05T08:24:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:31 smithi196 ceph-mon[33018]: 11.1 scrub starts 2024-02-05T08:24:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:31 smithi196 ceph-mon[33018]: pgmap v884: 158 pgs: 158 active+clean; 6.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-02-05T08:24:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:31 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/433125909' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:24:31.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:31 smithi196 ceph-mon[33018]: 4.19 scrub starts 2024-02-05T08:24:31.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:31 smithi196 ceph-mon[33018]: 4.19 scrub ok 2024-02-05T08:24:31.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:31 smithi196 ceph-mon[33018]: 3.6 scrub starts 2024-02-05T08:24:31.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:31 smithi196 ceph-mon[33018]: 3.6 scrub ok 2024-02-05T08:24:31.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:31 smithi196 ceph-mon[33018]: 11.d scrub starts 2024-02-05T08:24:31.348 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:31.392 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:31.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:31.494 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:31.530 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:31.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:31.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:31.634 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:31.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:31.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:31.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:31.743 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 72 15 113136 113121 98.1742 74.4375 0.0112247 0.0101808 2024-02-05T08:24:31.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:31.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:31.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:31.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:31.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:31.938 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:31.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:31.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:32.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:32.070 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:24:32.070 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:24:32.086 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:32.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:32.190 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:32.222 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:32.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[30086]: 4.1f scrub starts 2024-02-05T08:24:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[30086]: 4.1f scrub ok 2024-02-05T08:24:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:24:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:24:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[30086]: 5.f scrub starts 2024-02-05T08:24:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[30086]: 5.f scrub ok 2024-02-05T08:24:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:24:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:24:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/433125909' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:24:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[30086]: osdmap e390: 8 total, 8 up, 7 in 2024-02-05T08:24:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[30086]: 11.6 scrub starts 2024-02-05T08:24:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[39278]: 4.1f scrub starts 2024-02-05T08:24:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[39278]: 4.1f scrub ok 2024-02-05T08:24:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:24:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:24:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[39278]: 5.f scrub starts 2024-02-05T08:24:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[39278]: 5.f scrub ok 2024-02-05T08:24:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:24:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:24:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/433125909' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:24:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[39278]: osdmap e390: 8 total, 8 up, 7 in 2024-02-05T08:24:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:32 smithi134 ceph-mon[39278]: 11.6 scrub starts 2024-02-05T08:24:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:32 smithi196 ceph-mon[33018]: 4.1f scrub starts 2024-02-05T08:24:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:32 smithi196 ceph-mon[33018]: 4.1f scrub ok 2024-02-05T08:24:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:32 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:24:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:32 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:24:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:32 smithi196 ceph-mon[33018]: 5.f scrub starts 2024-02-05T08:24:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:32 smithi196 ceph-mon[33018]: 5.f scrub ok 2024-02-05T08:24:32.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:32 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:24:32.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:32 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:24:32.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:32 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/433125909' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:24:32.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:32 smithi196 ceph-mon[33018]: osdmap e390: 8 total, 8 up, 7 in 2024-02-05T08:24:32.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:32 smithi196 ceph-mon[33018]: 11.6 scrub starts 2024-02-05T08:24:32.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:32.363 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:32.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:32.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:32.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:32.507 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:32.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:32.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:32.609 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:32.654 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:32.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:32.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:32.742 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 73 16 114335 114319 97.8549 74.875 0.0118239 0.0102146 2024-02-05T08:24:32.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:32.791 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:32.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:32.893 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:32.950 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:32.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:32.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:33.052 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:33.085 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:33.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:33.187 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:33.211 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:33.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:33.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:33.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:33 smithi196 ceph-mon[33018]: 11.1 scrub ok 2024-02-05T08:24:33.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:33 smithi196 ceph-mon[33018]: 3.17 deep-scrub starts 2024-02-05T08:24:33.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:33 smithi196 ceph-mon[33018]: 3.17 deep-scrub ok 2024-02-05T08:24:33.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:33 smithi196 ceph-mon[33018]: pgmap v886: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-05T08:24:33.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:33 smithi196 ceph-mon[33018]: 5.5 scrub starts 2024-02-05T08:24:33.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:33 smithi196 ceph-mon[33018]: 5.5 scrub ok 2024-02-05T08:24:33.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:33.348 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:33.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:33.365 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:33 smithi134 ceph-mon[39278]: 11.1 scrub ok 2024-02-05T08:24:33.365 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:33 smithi134 ceph-mon[39278]: 3.17 deep-scrub starts 2024-02-05T08:24:33.365 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:33 smithi134 ceph-mon[39278]: 3.17 deep-scrub ok 2024-02-05T08:24:33.365 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:33 smithi134 ceph-mon[39278]: pgmap v886: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-05T08:24:33.365 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:33 smithi134 ceph-mon[39278]: 5.5 scrub starts 2024-02-05T08:24:33.365 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:33 smithi134 ceph-mon[39278]: 5.5 scrub ok 2024-02-05T08:24:33.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:33 smithi134 ceph-mon[30086]: 11.1 scrub ok 2024-02-05T08:24:33.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:33 smithi134 ceph-mon[30086]: 3.17 deep-scrub starts 2024-02-05T08:24:33.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:33 smithi134 ceph-mon[30086]: 3.17 deep-scrub ok 2024-02-05T08:24:33.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:33 smithi134 ceph-mon[30086]: pgmap v886: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-05T08:24:33.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:33 smithi134 ceph-mon[30086]: 5.5 scrub starts 2024-02-05T08:24:33.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:33 smithi134 ceph-mon[30086]: 5.5 scrub ok 2024-02-05T08:24:33.449 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:33.473 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:33.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:33.574 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:33.614 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:33.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:33.716 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:33.745 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 74 16 115399 115383 97.4309 66.5 0.00878977 0.0102588 2024-02-05T08:24:33.754 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:33.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:33.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:33.856 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:33.894 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:33.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:33.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:33.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:34.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:34.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:34.129 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:34.168 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:34.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:34.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:34.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:34 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:24:34.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:34 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:24:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:34 smithi134 ceph-mon[30086]: osdmap e391: 8 total, 8 up, 7 in 2024-02-05T08:24:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:34 smithi134 ceph-mon[30086]: 11.d scrub ok 2024-02-05T08:24:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:34 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:24:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:34 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:24:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:34 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:24:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:34 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:24:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:34 smithi134 ceph-mon[39278]: osdmap e391: 8 total, 8 up, 7 in 2024-02-05T08:24:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:34 smithi134 ceph-mon[39278]: 11.d scrub ok 2024-02-05T08:24:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:34 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:24:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:34 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:24:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:34.307 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:34.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:34.408 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:34.448 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:34.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:34.549 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:34 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:24:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:34 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:24:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:34 smithi196 ceph-mon[33018]: osdmap e391: 8 total, 8 up, 7 in 2024-02-05T08:24:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:34 smithi196 ceph-mon[33018]: 11.d scrub ok 2024-02-05T08:24:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:34 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:24:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:34 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:24:34.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:24:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:24:34.585 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:34.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:34.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:34.660 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T08:24:34.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:34.719 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:34.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:34.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:34.743 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 75 16 116722 116706 97.2342 82.6875 0.0141953 0.01028 2024-02-05T08:24:34.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:34.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:34.869 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:34.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:34.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:35.008 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:35.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:35.109 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:35.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:35.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:35.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:35.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[30086]: 5.15 deep-scrub starts 2024-02-05T08:24:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[30086]: 5.15 deep-scrub ok 2024-02-05T08:24:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[30086]: 3.15 scrub starts 2024-02-05T08:24:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[30086]: 3.15 scrub ok 2024-02-05T08:24:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[30086]: pgmap v888: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-02-05T08:24:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[30086]: 4.13 scrub starts 2024-02-05T08:24:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[30086]: 4.13 scrub ok 2024-02-05T08:24:35.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[30086]: 6.18 scrub starts 2024-02-05T08:24:35.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[30086]: 6.18 scrub ok 2024-02-05T08:24:35.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[30086]: 11.6 scrub ok 2024-02-05T08:24:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[39278]: 5.15 deep-scrub starts 2024-02-05T08:24:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[39278]: 5.15 deep-scrub ok 2024-02-05T08:24:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[39278]: 3.15 scrub starts 2024-02-05T08:24:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[39278]: 3.15 scrub ok 2024-02-05T08:24:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[39278]: pgmap v888: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-02-05T08:24:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[39278]: 4.13 scrub starts 2024-02-05T08:24:35.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[39278]: 4.13 scrub ok 2024-02-05T08:24:35.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[39278]: 6.18 scrub starts 2024-02-05T08:24:35.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[39278]: 6.18 scrub ok 2024-02-05T08:24:35.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:35 smithi134 ceph-mon[39278]: 11.6 scrub ok 2024-02-05T08:24:35.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:35.287 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:35.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:35.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:35.425 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:35.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:35.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:35.527 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:35 smithi196 ceph-mon[33018]: 5.15 deep-scrub starts 2024-02-05T08:24:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:35 smithi196 ceph-mon[33018]: 5.15 deep-scrub ok 2024-02-05T08:24:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:35 smithi196 ceph-mon[33018]: 3.15 scrub starts 2024-02-05T08:24:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:35 smithi196 ceph-mon[33018]: 3.15 scrub ok 2024-02-05T08:24:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:35 smithi196 ceph-mon[33018]: pgmap v888: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-02-05T08:24:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:35 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:35 smithi196 ceph-mon[33018]: 4.13 scrub starts 2024-02-05T08:24:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:35 smithi196 ceph-mon[33018]: 4.13 scrub ok 2024-02-05T08:24:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:35 smithi196 ceph-mon[33018]: 6.18 scrub starts 2024-02-05T08:24:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:35 smithi196 ceph-mon[33018]: 6.18 scrub ok 2024-02-05T08:24:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:35 smithi196 ceph-mon[33018]: 11.6 scrub ok 2024-02-05T08:24:35.567 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:35.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:35.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:35.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:35.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:35.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:35.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:35.743 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 76 16 118071 118055 97.0641 84.3125 0.0185681 0.0102981 2024-02-05T08:24:35.806 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:35.840 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:35.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:35.941 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:35.979 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:35.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:36.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:24:36.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:36.110 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:36.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:36.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:36.213 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:36.297 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:36.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:36.398 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:36.436 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:24:36.442 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:36.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:36.457 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:24:36.545 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:36.578 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:36.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:36.672 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:24:36.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:36.718 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:24:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:36.725 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:36.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:36.743 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 77 16 119160 119144 96.6875 68.0625 0.00688353 0.0103382 2024-02-05T08:24:36.755 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:24:36.799 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:24:36.816 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:36.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:36.850 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:24:36.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:36.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:36.890 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:24:36.926 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:24:36.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:36.971 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:24:36.974 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:24:36.975 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:24:36.975 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:24:36.975 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:24:36.975 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:24:36.975 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:24:36.975 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:24:36.975 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:24:37.000 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:37.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:37.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:37.102 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:37.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:37.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:37.239 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:37.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:37.273 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:37.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:37.374 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:37.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:37.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:37.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:37.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:37.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[30086]: pgmap v889: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T08:24:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[30086]: 6.1a scrub starts 2024-02-05T08:24:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[30086]: 6.1a scrub ok 2024-02-05T08:24:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[30086]: 4.b scrub starts 2024-02-05T08:24:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[30086]: 4.b scrub ok 2024-02-05T08:24:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[30086]: 5.2 scrub starts 2024-02-05T08:24:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[30086]: from='client.25892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:24:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2810050942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:24:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[39278]: pgmap v889: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T08:24:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[39278]: 6.1a scrub starts 2024-02-05T08:24:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[39278]: 6.1a scrub ok 2024-02-05T08:24:37.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[39278]: 4.b scrub starts 2024-02-05T08:24:37.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[39278]: 4.b scrub ok 2024-02-05T08:24:37.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[39278]: 5.2 scrub starts 2024-02-05T08:24:37.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[39278]: from='client.25892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:24:37.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2810050942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:24:37.546 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:37.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:37.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:37 smithi196 ceph-mon[33018]: pgmap v889: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T08:24:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:37 smithi196 ceph-mon[33018]: 6.1a scrub starts 2024-02-05T08:24:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:37 smithi196 ceph-mon[33018]: 6.1a scrub ok 2024-02-05T08:24:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:37 smithi196 ceph-mon[33018]: 4.b scrub starts 2024-02-05T08:24:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:37 smithi196 ceph-mon[33018]: 4.b scrub ok 2024-02-05T08:24:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:37 smithi196 ceph-mon[33018]: 5.2 scrub starts 2024-02-05T08:24:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:37 smithi196 ceph-mon[33018]: from='client.25892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:24:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:37 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2810050942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:24:37.645 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":889,"stamp":"2024-02-05T08:24:35.920194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7514364252,"num_objects":114849,"num_object_clones":0,"num_object_copies":344547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114849,"num_whiteouts":0,"num_read":2305,"num_read_kb":2057,"num_write":115373,"num_write_kb":7337709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":354,"num_bytes_recovered":9345,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73641,"ondisk_log_size":73641,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":477,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":26255780,"kb_used_data":22003300,"kb_used_omap":13,"kb_used_meta":4252402,"kb_avail":629960284,"statfs":{"total":671965249536,"available":645079330816,"internally_reserved":0,"allocated":22531379200,"data_stored":22528030639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14001,"internal_metadata":4354459983},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":729219072,"num_objects":11127,"num_object_clones":0,"num_object_copies":33381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11127,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11127,"num_write_kb":712128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002865"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":716,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065415+0000","last_change":"2024-02-05T08:23:07.931957+0000","last_active":"2024-02-05T08:24:31.065415+0000","last_peered":"2024-02-05T08:24:31.065415+0000","last_clean":"2024-02-05T08:24:31.065415+0000","last_became_active":"2024-02-05T08:23:07.931642+0000","last_became_peered":"2024-02-05T08:23:07.931642+0000","last_unstale":"2024-02-05T08:24:31.065415+0000","last_undegraded":"2024-02-05T08:24:31.065415+0000","last_fullsized":"2024-02-05T08:24:31.065415+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:19:36.296981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:19:36.296981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":582,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:13.931706+0000","last_change":"2024-02-05T08:23:32.176306+0000","last_active":"2024-02-05T08:24:13.931706+0000","last_peered":"2024-02-05T08:24:13.931706+0000","last_clean":"2024-02-05T08:24:13.931706+0000","last_became_active":"2024-02-05T08:23:07.938772+0000","last_became_peered":"2024-02-05T08:23:07.938772+0000","last_unstale":"2024-02-05T08:24:13.931706+0000","last_undegraded":"2024-02-05T08:24:13.931706+0000","last_fullsized":"2024-02-05T08:24:13.931706+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:32.176199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:23:32.176199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:39.232013+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":97,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.092651+0000","last_change":"2024-02-05T08:22:54.215968+0000","last_active":"2024-02-05T08:24:33.092651+0000","last_peered":"2024-02-05T08:24:33.092651+0000","last_clean":"2024-02-05T08:24:33.092651+0000","last_became_active":"2024-02-05T08:22:09.101941+0000","last_became_peered":"2024-02-05T08:22:09.101941+0000","last_unstale":"2024-02-05T08:24:33.092651+0000","last_undegraded":"2024-02-05T08:24:33.092651+0000","last_fullsized":"2024-02-05T08:24:33.092651+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:54.215846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:22:54.215846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"236'19","reported_seq":786,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065147+0000","last_change":"2024-02-05T08:23:31.459738+0000","last_active":"2024-02-05T08:24:31.065147+0000","last_peered":"2024-02-05T08:24:31.065147+0000","last_clean":"2024-02-05T08:24:31.065147+0000","last_became_active":"2024-02-05T08:23:07.937041+0000","last_became_peered":"2024-02-05T08:23:07.937041+0000","last_unstale":"2024-02-05T08:24:31.065147+0000","last_undegraded":"2024-02-05T08:24:31.065147+0000","last_fullsized":"2024-02-05T08:24:31.065147+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:23:31.459611+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:23:31.459611+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:49.340063+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":229,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.083682+0000","last_change":"2024-02-05T08:23:24.076782+0000","last_active":"2024-02-05T08:24:33.083682+0000","last_peered":"2024-02-05T08:24:33.083682+0000","last_clean":"2024-02-05T08:24:33.083682+0000","last_became_active":"2024-02-05T08:18:34.633083+0000","last_became_peered":"2024-02-05T08:18:34.633083+0000","last_unstale":"2024-02-05T08:24:33.083682+0000","last_undegraded":"2024-02-05T08:24:33.083682+0000","last_fullsized":"2024-02-05T08:24:33.083682+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:24.076590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:23:24.076590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:41.261868+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":733,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065265+0000","last_change":"2024-02-05T08:23:41.460609+0000","last_active":"2024-02-05T08:24:31.065265+0000","last_peered":"2024-02-05T08:24:31.065265+0000","last_clean":"2024-02-05T08:24:31.065265+0000","last_became_active":"2024-02-05T08:23:07.937180+0000","last_became_peered":"2024-02-05T08:23:07.937180+0000","last_unstale":"2024-02-05T08:24:31.065265+0000","last_undegraded":"2024-02-05T08:24:31.065265+0000","last_fullsized":"2024-02-05T08:24:31.065265+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:41.460468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:41.460468+0000","last_clean_scrub_stamp":"2024-02-05T08:23:41.460468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:46.860369+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":238,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.085744+0000","last_change":"2024-02-05T08:24:20.030502+0000","last_active":"2024-02-05T08:24:33.085744+0000","last_peered":"2024-02-05T08:24:33.085744+0000","last_clean":"2024-02-05T08:24:33.085744+0000","last_became_active":"2024-02-05T08:21:41.880038+0000","last_became_peered":"2024-02-05T08:21:41.880038+0000","last_unstale":"2024-02-05T08:24:33.085744+0000","last_undegraded":"2024-02-05T08:24:33.085744+0000","last_fullsized":"2024-02-05T08:24:33.085744+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:20.030382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:24:20.030382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:31.271518+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"236'34","reported_seq":275,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.086208+0000","last_change":"2024-02-05T08:24:30.076306+0000","last_active":"2024-02-05T08:24:33.086208+0000","last_peered":"2024-02-05T08:24:33.086208+0000","last_clean":"2024-02-05T08:24:33.086208+0000","last_became_active":"2024-02-05T08:22:09.095896+0000","last_became_peered":"2024-02-05T08:22:09.095896+0000","last_unstale":"2024-02-05T08:24:33.086208+0000","last_undegraded":"2024-02-05T08:24:33.086208+0000","last_fullsized":"2024-02-05T08:24:33.086208+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:24:30.076161+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:24:30.076161+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:45.272166+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":55,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:13.933169+0000","last_change":"2024-02-05T08:23:08.917569+0000","last_active":"2024-02-05T08:24:13.933169+0000","last_peered":"2024-02-05T08:24:13.933169+0000","last_clean":"2024-02-05T08:24:13.933169+0000","last_became_active":"2024-02-05T08:23:08.917199+0000","last_became_peered":"2024-02-05T08:23:08.917199+0000","last_unstale":"2024-02-05T08:24:13.933169+0000","last_undegraded":"2024-02-05T08:24:13.933169+0000","last_fullsized":"2024-02-05T08:24:13.933169+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:44.814367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:44.814367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:14.131780+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":735,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.092233+0000","last_change":"2024-02-05T08:23:02.193023+0000","last_active":"2024-02-05T08:24:33.092233+0000","last_peered":"2024-02-05T08:24:33.092233+0000","last_clean":"2024-02-05T08:24:33.092233+0000","last_became_active":"2024-02-05T08:22:09.108745+0000","last_became_peered":"2024-02-05T08:22:09.108745+0000","last_unstale":"2024-02-05T08:24:33.092233+0000","last_undegraded":"2024-02-05T08:24:33.092233+0000","last_fullsized":"2024-02-05T08:24:33.092233+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:02.192889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:02.192889+0000","last_clean_scrub_stamp":"2024-02-05T08:23:02.192889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":757,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093341+0000","last_change":"2024-02-05T08:23:40.814097+0000","last_active":"2024-02-05T08:24:33.093341+0000","last_peered":"2024-02-05T08:24:33.093341+0000","last_clean":"2024-02-05T08:24:33.093341+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:24:33.093341+0000","last_undegraded":"2024-02-05T08:24:33.093341+0000","last_fullsized":"2024-02-05T08:24:33.093341+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:40.813940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_clean_scrub_stamp":"2024-02-05T08:23:40.813940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:00.487871+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"263'19","reported_seq":806,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065634+0000","last_change":"2024-02-05T08:22:09.118822+0000","last_active":"2024-02-05T08:24:31.065634+0000","last_peered":"2024-02-05T08:24:31.065634+0000","last_clean":"2024-02-05T08:24:31.065634+0000","last_became_active":"2024-02-05T08:22:09.102371+0000","last_became_peered":"2024-02-05T08:22:09.102371+0000","last_unstale":"2024-02-05T08:24:31.065634+0000","last_undegraded":"2024-02-05T08:24:31.065634+0000","last_fullsized":"2024-02-05T08:24:31.065634+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:21:43.861060+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:21:43.861060+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":731,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.092031+0000","last_change":"2024-02-05T08:23:25.177775+0000","last_active":"2024-02-05T08:24:33.092031+0000","last_peered":"2024-02-05T08:24:33.092031+0000","last_clean":"2024-02-05T08:24:33.092031+0000","last_became_active":"2024-02-05T08:23:07.932548+0000","last_became_peered":"2024-02-05T08:23:07.932548+0000","last_unstale":"2024-02-05T08:24:33.092031+0000","last_undegraded":"2024-02-05T08:24:33.092031+0000","last_fullsized":"2024-02-05T08:24:33.092031+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:25.177650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:25.177650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:44.333065+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":731,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.091117+0000","last_change":"2024-02-05T08:23:26.986054+0000","last_active":"2024-02-05T08:24:33.091117+0000","last_peered":"2024-02-05T08:24:33.091117+0000","last_clean":"2024-02-05T08:24:33.091117+0000","last_became_active":"2024-02-05T08:23:07.939630+0000","last_became_peered":"2024-02-05T08:23:07.939630+0000","last_unstale":"2024-02-05T08:24:33.091117+0000","last_undegraded":"2024-02-05T08:24:33.091117+0000","last_fullsized":"2024-02-05T08:24:33.091117+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:26.985938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:23:26.985938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:35.576590+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":738,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.092686+0000","last_change":"2024-02-05T08:23:29.002501+0000","last_active":"2024-02-05T08:24:33.092686+0000","last_peered":"2024-02-05T08:24:33.092686+0000","last_clean":"2024-02-05T08:24:33.092686+0000","last_became_active":"2024-02-05T08:23:07.932530+0000","last_became_peered":"2024-02-05T08:23:07.932530+0000","last_unstale":"2024-02-05T08:24:33.092686+0000","last_undegraded":"2024-02-05T08:24:33.092686+0000","last_fullsized":"2024-02-05T08:24:33.092686+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:29.002380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.002380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:36.585001+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":761,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.066590+0000","last_change":"2024-02-05T08:23:27.578462+0000","last_active":"2024-02-05T08:24:31.066590+0000","last_peered":"2024-02-05T08:24:31.066590+0000","last_clean":"2024-02-05T08:24:31.066590+0000","last_became_active":"2024-02-05T08:23:07.938087+0000","last_became_peered":"2024-02-05T08:23:07.938087+0000","last_unstale":"2024-02-05T08:24:31.066590+0000","last_undegraded":"2024-02-05T08:24:31.066590+0000","last_fullsized":"2024-02-05T08:24:31.066590+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-02-05T08:23:27.578370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:27.578370+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:47.356834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":729,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065745+0000","last_change":"2024-02-05T08:21:42.867663+0000","last_active":"2024-02-05T08:24:31.065745+0000","last_peered":"2024-02-05T08:24:31.065745+0000","last_clean":"2024-02-05T08:24:31.065745+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:24:31.065745+0000","last_undegraded":"2024-02-05T08:24:31.065745+0000","last_fullsized":"2024-02-05T08:24:31.065745+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:42.867555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:21:42.867555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":623,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:13.932169+0000","last_change":"2024-02-05T08:23:08.916098+0000","last_active":"2024-02-05T08:24:13.932169+0000","last_peered":"2024-02-05T08:24:13.932169+0000","last_clean":"2024-02-05T08:24:13.932169+0000","last_became_active":"2024-02-05T08:23:08.914258+0000","last_became_peered":"2024-02-05T08:23:08.914258+0000","last_unstale":"2024-02-05T08:24:13.932169+0000","last_undegraded":"2024-02-05T08:24:13.932169+0000","last_fullsized":"2024-02-05T08:24:13.932169+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:56.260200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_clean_scrub_stamp":"2024-02-05T08:22:56.260200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:14.131804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":90,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.066485+0000","last_change":"2024-02-05T08:23:08.921340+0000","last_active":"2024-02-05T08:24:31.066485+0000","last_peered":"2024-02-05T08:24:31.066485+0000","last_clean":"2024-02-05T08:24:31.066485+0000","last_became_active":"2024-02-05T08:23:08.920902+0000","last_became_peered":"2024-02-05T08:23:08.920902+0000","last_unstale":"2024-02-05T08:24:31.066485+0000","last_undegraded":"2024-02-05T08:24:31.066485+0000","last_fullsized":"2024-02-05T08:24:31.066485+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:04.290125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:04.290125+0000","last_clean_scrub_stamp":"2024-02-05T08:22:04.290125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"236'34","reported_seq":844,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.081420+0000","last_change":"2024-02-05T08:23:29.912005+0000","last_active":"2024-02-05T08:24:33.081420+0000","last_peered":"2024-02-05T08:24:33.081420+0000","last_clean":"2024-02-05T08:24:33.081420+0000","last_became_active":"2024-02-05T08:23:07.930731+0000","last_became_peered":"2024-02-05T08:23:07.930731+0000","last_unstale":"2024-02-05T08:24:33.081420+0000","last_undegraded":"2024-02-05T08:24:33.081420+0000","last_fullsized":"2024-02-05T08:24:33.081420+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:23:29.911720+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.911720+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:59.611514+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":229,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.085223+0000","last_change":"2024-02-05T08:24:26.342293+0000","last_active":"2024-02-05T08:24:33.085223+0000","last_peered":"2024-02-05T08:24:33.085223+0000","last_clean":"2024-02-05T08:24:33.085223+0000","last_became_active":"2024-02-05T08:18:34.642494+0000","last_became_peered":"2024-02-05T08:18:34.642494+0000","last_unstale":"2024-02-05T08:24:33.085223+0000","last_undegraded":"2024-02-05T08:24:33.085223+0000","last_fullsized":"2024-02-05T08:24:33.085223+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:26.342171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_clean_scrub_stamp":"2024-02-05T08:24:26.342171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:41.547144+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":739,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065817+0000","last_change":"2024-02-05T08:22:57.230710+0000","last_active":"2024-02-05T08:24:31.065817+0000","last_peered":"2024-02-05T08:24:31.065817+0000","last_clean":"2024-02-05T08:24:31.065817+0000","last_became_active":"2024-02-05T08:18:34.638652+0000","last_became_peered":"2024-02-05T08:18:34.638652+0000","last_unstale":"2024-02-05T08:24:31.065817+0000","last_undegraded":"2024-02-05T08:24:31.065817+0000","last_fullsized":"2024-02-05T08:24:31.065817+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:57.230639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:22:57.230639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":728,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065723+0000","last_change":"2024-02-05T08:24:22.154106+0000","last_active":"2024-02-05T08:24:31.065723+0000","last_peered":"2024-02-05T08:24:31.065723+0000","last_clean":"2024-02-05T08:24:31.065723+0000","last_became_active":"2024-02-05T08:23:07.937958+0000","last_became_peered":"2024-02-05T08:23:07.937958+0000","last_unstale":"2024-02-05T08:24:31.065723+0000","last_undegraded":"2024-02-05T08:24:31.065723+0000","last_fullsized":"2024-02-05T08:24:31.065723+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:22.153964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:24:22.153964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:25.670574+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"236'23","reported_seq":234,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.084547+0000","last_change":"2024-02-05T08:23:25.937497+0000","last_active":"2024-02-05T08:24:33.084547+0000","last_peered":"2024-02-05T08:24:33.084547+0000","last_clean":"2024-02-05T08:24:33.084547+0000","last_became_active":"2024-02-05T08:23:07.925083+0000","last_became_peered":"2024-02-05T08:23:07.925083+0000","last_unstale":"2024-02-05T08:24:33.084547+0000","last_undegraded":"2024-02-05T08:24:33.084547+0000","last_fullsized":"2024-02-05T08:24:33.084547+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'23","last_scrub_stamp":"2024-02-05T08:23:25.935815+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:23:25.935815+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":736,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.089791+0000","last_change":"2024-02-05T08:23:40.022017+0000","last_active":"2024-02-05T08:24:33.089791+0000","last_peered":"2024-02-05T08:24:33.089791+0000","last_clean":"2024-02-05T08:24:33.089791+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:24:33.089791+0000","last_undegraded":"2024-02-05T08:24:33.089791+0000","last_fullsized":"2024-02-05T08:24:33.089791+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:23:40.021926+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:23:40.021926+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:09.922432+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":633,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.096770+0000","last_change":"2024-02-05T08:24:27.484484+0000","last_active":"2024-02-05T08:24:33.096770+0000","last_peered":"2024-02-05T08:24:33.096770+0000","last_clean":"2024-02-05T08:24:33.096770+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:24:33.096770+0000","last_undegraded":"2024-02-05T08:24:33.096770+0000","last_fullsized":"2024-02-05T08:24:33.096770+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:24:27.484356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:24:27.484356+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:38.333363+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":660,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:24.293619+0000","last_change":"2024-02-05T08:24:24.293664+0000","last_active":"2024-02-05T08:24:24.293619+0000","last_peered":"2024-02-05T08:24:24.293619+0000","last_clean":"2024-02-05T08:24:24.293619+0000","last_became_active":"2024-02-05T08:23:07.940135+0000","last_became_peered":"2024-02-05T08:23:07.940135+0000","last_unstale":"2024-02-05T08:24:24.293619+0000","last_undegraded":"2024-02-05T08:24:24.293619+0000","last_fullsized":"2024-02-05T08:24:24.293619+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:24:24.293517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:24.293517+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:48.981702+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"236'20","reported_seq":559,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.089665+0000","last_change":"2024-02-05T08:23:08.935899+0000","last_active":"2024-02-05T08:24:33.089665+0000","last_peered":"2024-02-05T08:24:33.089665+0000","last_clean":"2024-02-05T08:24:33.089665+0000","last_became_active":"2024-02-05T08:23:08.920330+0000","last_became_peered":"2024-02-05T08:23:08.920330+0000","last_unstale":"2024-02-05T08:24:33.089665+0000","last_undegraded":"2024-02-05T08:24:33.089665+0000","last_fullsized":"2024-02-05T08:24:33.089665+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:21:49.240369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:21:49.240369+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":586,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:13.933127+0000","last_change":"2024-02-05T08:23:33.200652+0000","last_active":"2024-02-05T08:24:13.933127+0000","last_peered":"2024-02-05T08:24:13.933127+0000","last_clean":"2024-02-05T08:24:13.933127+0000","last_became_active":"2024-02-05T08:23:07.941095+0000","last_became_peered":"2024-02-05T08:23:07.941095+0000","last_unstale":"2024-02-05T08:24:13.933127+0000","last_undegraded":"2024-02-05T08:24:13.933127+0000","last_fullsized":"2024-02-05T08:24:13.933127+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:33.200557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:33.200557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:36.583846+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":101,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.089983+0000","last_change":"2024-02-05T08:24:23.992394+0000","last_active":"2024-02-05T08:24:33.089983+0000","last_peered":"2024-02-05T08:24:33.089983+0000","last_clean":"2024-02-05T08:24:33.089983+0000","last_became_active":"2024-02-05T08:23:08.911020+0000","last_became_peered":"2024-02-05T08:23:08.911020+0000","last_unstale":"2024-02-05T08:24:33.089983+0000","last_undegraded":"2024-02-05T08:24:33.089983+0000","last_fullsized":"2024-02-05T08:24:33.089983+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:23.992280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:19.107555+0000","last_clean_scrub_stamp":"2024-02-05T08:24:23.992280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:47.731830+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":234,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.085312+0000","last_change":"2024-02-05T08:23:08.927552+0000","last_active":"2024-02-05T08:24:33.085312+0000","last_peered":"2024-02-05T08:24:33.085312+0000","last_clean":"2024-02-05T08:24:33.085312+0000","last_became_active":"2024-02-05T08:23:08.927256+0000","last_became_peered":"2024-02-05T08:23:08.927256+0000","last_unstale":"2024-02-05T08:24:33.085312+0000","last_undegraded":"2024-02-05T08:24:33.085312+0000","last_fullsized":"2024-02-05T08:24:33.085312+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:46.026840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:22:46.026840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"236'19","reported_seq":500,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.096635+0000","last_change":"2024-02-05T08:24:30.419919+0000","last_active":"2024-02-05T08:24:33.096635+0000","last_peered":"2024-02-05T08:24:33.096635+0000","last_clean":"2024-02-05T08:24:33.096635+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:24:33.096635+0000","last_undegraded":"2024-02-05T08:24:33.096635+0000","last_fullsized":"2024-02-05T08:24:33.096635+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:24:30.419791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:24:30.419791+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:54.385059+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":719,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.091273+0000","last_change":"2024-02-05T08:24:16.984051+0000","last_active":"2024-02-05T08:24:33.091273+0000","last_peered":"2024-02-05T08:24:33.091273+0000","last_clean":"2024-02-05T08:24:33.091273+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:24:33.091273+0000","last_undegraded":"2024-02-05T08:24:33.091273+0000","last_fullsized":"2024-02-05T08:24:33.091273+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:16.983946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:14.159628+0000","last_clean_scrub_stamp":"2024-02-05T08:24:16.983946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:41.637025+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":471,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.090585+0000","last_change":"2024-02-05T08:23:31.538869+0000","last_active":"2024-02-05T08:24:33.090585+0000","last_peered":"2024-02-05T08:24:33.090585+0000","last_clean":"2024-02-05T08:24:33.090585+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:24:33.090585+0000","last_undegraded":"2024-02-05T08:24:33.090585+0000","last_fullsized":"2024-02-05T08:24:33.090585+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:31.538744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:23:31.538744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:42.776710+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":618,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093592+0000","last_change":"2024-02-05T08:24:31.862924+0000","last_active":"2024-02-05T08:24:33.093592+0000","last_peered":"2024-02-05T08:24:33.093592+0000","last_clean":"2024-02-05T08:24:33.093592+0000","last_became_active":"2024-02-05T08:22:09.104064+0000","last_became_peered":"2024-02-05T08:22:09.104064+0000","last_unstale":"2024-02-05T08:24:33.093592+0000","last_undegraded":"2024-02-05T08:24:33.093592+0000","last_fullsized":"2024-02-05T08:24:33.093592+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:31.862841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:31.862841+0000","last_clean_scrub_stamp":"2024-02-05T08:24:31.862841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:48.896461+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":236,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.085924+0000","last_change":"2024-02-05T08:23:31.939827+0000","last_active":"2024-02-05T08:24:33.085924+0000","last_peered":"2024-02-05T08:24:33.085924+0000","last_clean":"2024-02-05T08:24:33.085924+0000","last_became_active":"2024-02-05T08:23:07.937911+0000","last_became_peered":"2024-02-05T08:23:07.937911+0000","last_unstale":"2024-02-05T08:24:33.085924+0000","last_undegraded":"2024-02-05T08:24:33.085924+0000","last_fullsized":"2024-02-05T08:24:33.085924+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:23:31.939578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:31.939578+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:36.879693+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":235,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.083584+0000","last_change":"2024-02-05T08:22:47.060160+0000","last_active":"2024-02-05T08:24:33.083584+0000","last_peered":"2024-02-05T08:24:33.083584+0000","last_clean":"2024-02-05T08:24:33.083584+0000","last_became_active":"2024-02-05T08:22:09.095805+0000","last_became_peered":"2024-02-05T08:22:09.095805+0000","last_unstale":"2024-02-05T08:24:33.083584+0000","last_undegraded":"2024-02-05T08:24:33.083584+0000","last_fullsized":"2024-02-05T08:24:33.083584+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:47.060036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:26.891040+0000","last_clean_scrub_stamp":"2024-02-05T08:22:47.060036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":743,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.088749+0000","last_change":"2024-02-05T08:21:54.156544+0000","last_active":"2024-02-05T08:24:33.088749+0000","last_peered":"2024-02-05T08:24:33.088749+0000","last_clean":"2024-02-05T08:24:33.088749+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:24:33.088749+0000","last_undegraded":"2024-02-05T08:24:33.088749+0000","last_fullsized":"2024-02-05T08:24:33.088749+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:54.156080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:54.156080+0000","last_clean_scrub_stamp":"2024-02-05T08:21:54.156080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":739,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093379+0000","last_change":"2024-02-05T08:23:28.001765+0000","last_active":"2024-02-05T08:24:33.093379+0000","last_peered":"2024-02-05T08:24:33.093379+0000","last_clean":"2024-02-05T08:24:33.093379+0000","last_became_active":"2024-02-05T08:23:07.932374+0000","last_became_peered":"2024-02-05T08:23:07.932374+0000","last_unstale":"2024-02-05T08:24:33.093379+0000","last_undegraded":"2024-02-05T08:24:33.093379+0000","last_fullsized":"2024-02-05T08:24:33.093379+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:28.001523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:23:28.001523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:40.938375+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"236'24","reported_seq":241,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.086530+0000","last_change":"2024-02-05T08:24:28.130743+0000","last_active":"2024-02-05T08:24:33.086530+0000","last_peered":"2024-02-05T08:24:33.086530+0000","last_clean":"2024-02-05T08:24:33.086530+0000","last_became_active":"2024-02-05T08:18:34.631912+0000","last_became_peered":"2024-02-05T08:18:34.631912+0000","last_unstale":"2024-02-05T08:24:33.086530+0000","last_undegraded":"2024-02-05T08:24:33.086530+0000","last_fullsized":"2024-02-05T08:24:33.086530+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:24:28.130616+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:24:28.130616+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:51.228990+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":665,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.091601+0000","last_change":"2024-02-05T08:23:08.918458+0000","last_active":"2024-02-05T08:24:33.091601+0000","last_peered":"2024-02-05T08:24:33.091601+0000","last_clean":"2024-02-05T08:24:33.091601+0000","last_became_active":"2024-02-05T08:23:08.918104+0000","last_became_peered":"2024-02-05T08:23:08.918104+0000","last_unstale":"2024-02-05T08:24:33.091601+0000","last_undegraded":"2024-02-05T08:24:33.091601+0000","last_fullsized":"2024-02-05T08:24:33.091601+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:51.235476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:22:51.235476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":234,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.083498+0000","last_change":"2024-02-05T08:23:08.918137+0000","last_active":"2024-02-05T08:24:33.083498+0000","last_peered":"2024-02-05T08:24:33.083498+0000","last_clean":"2024-02-05T08:24:33.083498+0000","last_became_active":"2024-02-05T08:23:08.917587+0000","last_became_peered":"2024-02-05T08:23:08.917587+0000","last_unstale":"2024-02-05T08:24:33.083498+0000","last_undegraded":"2024-02-05T08:24:33.083498+0000","last_fullsized":"2024-02-05T08:24:33.083498+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:44.050117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:44.050117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":743,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.092870+0000","last_change":"2024-02-05T08:21:53.174425+0000","last_active":"2024-02-05T08:24:33.092870+0000","last_peered":"2024-02-05T08:24:33.092870+0000","last_clean":"2024-02-05T08:24:33.092870+0000","last_became_active":"2024-02-05T08:18:34.642387+0000","last_became_peered":"2024-02-05T08:18:34.642387+0000","last_unstale":"2024-02-05T08:24:33.092870+0000","last_undegraded":"2024-02-05T08:24:33.092870+0000","last_fullsized":"2024-02-05T08:24:33.092870+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:53.173888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:53.173888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"263'19","reported_seq":406,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093682+0000","last_change":"2024-02-05T08:24:28.876345+0000","last_active":"2024-02-05T08:24:33.093682+0000","last_peered":"2024-02-05T08:24:33.093682+0000","last_clean":"2024-02-05T08:24:33.093682+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:24:33.093682+0000","last_undegraded":"2024-02-05T08:24:33.093682+0000","last_fullsized":"2024-02-05T08:24:33.093682+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:24:28.876211+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_clean_scrub_stamp":"2024-02-05T08:24:28.876211+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:29.962110+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":243,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.086374+0000","last_change":"2024-02-05T08:23:00.825681+0000","last_active":"2024-02-05T08:24:33.086374+0000","last_peered":"2024-02-05T08:24:33.086374+0000","last_clean":"2024-02-05T08:24:33.086374+0000","last_became_active":"2024-02-05T08:22:09.096314+0000","last_became_peered":"2024-02-05T08:22:09.096314+0000","last_unstale":"2024-02-05T08:24:33.086374+0000","last_undegraded":"2024-02-05T08:24:33.086374+0000","last_fullsized":"2024-02-05T08:24:33.086374+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:00.825605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:23:00.825605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":637,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:25.279720+0000","last_change":"2024-02-05T08:24:25.279757+0000","last_active":"2024-02-05T08:24:25.279720+0000","last_peered":"2024-02-05T08:24:25.279720+0000","last_clean":"2024-02-05T08:24:25.279720+0000","last_became_active":"2024-02-05T08:23:07.936401+0000","last_became_peered":"2024-02-05T08:23:07.936401+0000","last_unstale":"2024-02-05T08:24:25.279720+0000","last_undegraded":"2024-02-05T08:24:25.279720+0000","last_fullsized":"2024-02-05T08:24:25.279720+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:25.279647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:24:25.279647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:41.285794+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":748,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065569+0000","last_change":"2024-02-05T08:21:44.871777+0000","last_active":"2024-02-05T08:24:31.065569+0000","last_peered":"2024-02-05T08:24:31.065569+0000","last_clean":"2024-02-05T08:24:31.065569+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:24:31.065569+0000","last_undegraded":"2024-02-05T08:24:31.065569+0000","last_fullsized":"2024-02-05T08:24:31.065569+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:44.871668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:21:44.871668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"249'24","reported_seq":797,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.066743+0000","last_change":"2024-02-05T08:23:07.938874+0000","last_active":"2024-02-05T08:24:31.066743+0000","last_peered":"2024-02-05T08:24:31.066743+0000","last_clean":"2024-02-05T08:24:31.066743+0000","last_became_active":"2024-02-05T08:23:07.938523+0000","last_became_peered":"2024-02-05T08:23:07.938523+0000","last_unstale":"2024-02-05T08:24:31.066743+0000","last_undegraded":"2024-02-05T08:24:31.066743+0000","last_fullsized":"2024-02-05T08:24:31.066743+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:19:27.349541+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:19:27.349541+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":590,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.089800+0000","last_change":"2024-02-05T08:23:33.061246+0000","last_active":"2024-02-05T08:24:33.089800+0000","last_peered":"2024-02-05T08:24:33.089800+0000","last_clean":"2024-02-05T08:24:33.089800+0000","last_became_active":"2024-02-05T08:22:09.101652+0000","last_became_peered":"2024-02-05T08:22:09.101652+0000","last_unstale":"2024-02-05T08:24:33.089800+0000","last_undegraded":"2024-02-05T08:24:33.089800+0000","last_fullsized":"2024-02-05T08:24:33.089800+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:33.061104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:33.061104+0000","last_clean_scrub_stamp":"2024-02-05T08:23:33.061104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:40.201454+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"370'13","reported_seq":990,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093392+0000","last_change":"2024-02-05T08:23:08.918239+0000","last_active":"2024-02-05T08:24:33.093392+0000","last_peered":"2024-02-05T08:24:33.093392+0000","last_clean":"2024-02-05T08:24:33.093392+0000","last_became_active":"2024-02-05T08:23:08.917858+0000","last_became_peered":"2024-02-05T08:23:08.917858+0000","last_unstale":"2024-02-05T08:24:33.093392+0000","last_undegraded":"2024-02-05T08:24:33.093392+0000","last_fullsized":"2024-02-05T08:24:33.093392+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'11","last_scrub_stamp":"2024-02-05T08:22:48.261980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:48.261980+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":739,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.089378+0000","last_change":"2024-02-05T08:24:19.912012+0000","last_active":"2024-02-05T08:24:33.089378+0000","last_peered":"2024-02-05T08:24:33.089378+0000","last_clean":"2024-02-05T08:24:33.089378+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:24:33.089378+0000","last_undegraded":"2024-02-05T08:24:33.089378+0000","last_fullsized":"2024-02-05T08:24:33.089378+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:19.911782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:19.911782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:28.277541+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":242,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.083908+0000","last_change":"2024-02-05T08:24:19.039429+0000","last_active":"2024-02-05T08:24:33.083908+0000","last_peered":"2024-02-05T08:24:33.083908+0000","last_clean":"2024-02-05T08:24:33.083908+0000","last_became_active":"2024-02-05T08:23:07.928966+0000","last_became_peered":"2024-02-05T08:23:07.928966+0000","last_unstale":"2024-02-05T08:24:33.083908+0000","last_undegraded":"2024-02-05T08:24:33.083908+0000","last_fullsized":"2024-02-05T08:24:33.083908+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-02-05T08:24:19.039293+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:24:19.039293+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:33.178419+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":711,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.066487+0000","last_change":"2024-02-05T08:21:10.125938+0000","last_active":"2024-02-05T08:24:31.066487+0000","last_peered":"2024-02-05T08:24:31.066487+0000","last_clean":"2024-02-05T08:24:31.066487+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:24:31.066487+0000","last_undegraded":"2024-02-05T08:24:31.066487+0000","last_fullsized":"2024-02-05T08:24:31.066487+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:10.125818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:28.519149+0000","last_clean_scrub_stamp":"2024-02-05T08:21:10.125818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"295'10","reported_seq":298,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.084060+0000","last_change":"2024-02-05T08:22:03.724091+0000","last_active":"2024-02-05T08:24:33.084060+0000","last_peered":"2024-02-05T08:24:33.084060+0000","last_clean":"2024-02-05T08:24:33.084060+0000","last_became_active":"2024-02-05T08:18:34.644517+0000","last_became_peered":"2024-02-05T08:18:34.644517+0000","last_unstale":"2024-02-05T08:24:33.084060+0000","last_undegraded":"2024-02-05T08:24:33.084060+0000","last_fullsized":"2024-02-05T08:24:33.084060+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'10","last_scrub_stamp":"2024-02-05T08:22:03.723970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:03.723970+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":614,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.090981+0000","last_change":"2024-02-05T08:23:44.038746+0000","last_active":"2024-02-05T08:24:33.090981+0000","last_peered":"2024-02-05T08:24:33.090981+0000","last_clean":"2024-02-05T08:24:33.090981+0000","last_became_active":"2024-02-05T08:23:08.920193+0000","last_became_peered":"2024-02-05T08:23:08.920193+0000","last_unstale":"2024-02-05T08:24:33.090981+0000","last_undegraded":"2024-02-05T08:24:33.090981+0000","last_fullsized":"2024-02-05T08:24:33.090981+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:44.038564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:23:44.038564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:50.333109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":799,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093574+0000","last_change":"2024-02-05T08:23:23.049946+0000","last_active":"2024-02-05T08:24:33.093574+0000","last_peered":"2024-02-05T08:24:33.093574+0000","last_clean":"2024-02-05T08:24:33.093574+0000","last_became_active":"2024-02-05T08:18:34.640741+0000","last_became_peered":"2024-02-05T08:18:34.640741+0000","last_unstale":"2024-02-05T08:24:33.093574+0000","last_undegraded":"2024-02-05T08:24:33.093574+0000","last_fullsized":"2024-02-05T08:24:33.093574+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:23:23.049797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:23.049797+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":750,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.081746+0000","last_change":"2024-02-05T08:24:16.703350+0000","last_active":"2024-02-05T08:24:33.081746+0000","last_peered":"2024-02-05T08:24:33.081746+0000","last_clean":"2024-02-05T08:24:33.081746+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:24:33.081746+0000","last_undegraded":"2024-02-05T08:24:33.081746+0000","last_fullsized":"2024-02-05T08:24:33.081746+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:16.702946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:16.702946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:32.608708+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":230,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.085122+0000","last_change":"2024-02-05T08:23:42.025667+0000","last_active":"2024-02-05T08:24:33.085122+0000","last_peered":"2024-02-05T08:24:33.085122+0000","last_clean":"2024-02-05T08:24:33.085122+0000","last_became_active":"2024-02-05T08:23:07.928375+0000","last_became_peered":"2024-02-05T08:23:07.928375+0000","last_unstale":"2024-02-05T08:24:33.085122+0000","last_undegraded":"2024-02-05T08:24:33.085122+0000","last_fullsized":"2024-02-05T08:24:33.085122+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:42.025471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:42.025471+0000","last_clean_scrub_stamp":"2024-02-05T08:23:42.025471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:53.919766+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":743,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.088923+0000","last_change":"2024-02-05T08:22:05.129978+0000","last_active":"2024-02-05T08:24:33.088923+0000","last_peered":"2024-02-05T08:24:33.088923+0000","last_clean":"2024-02-05T08:24:33.088923+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:24:33.088923+0000","last_undegraded":"2024-02-05T08:24:33.088923+0000","last_fullsized":"2024-02-05T08:24:33.088923+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:05.129584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:22:05.129584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"278'19","reported_seq":647,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.085448+0000","last_change":"2024-02-05T08:22:09.127927+0000","last_active":"2024-02-05T08:24:33.085448+0000","last_peered":"2024-02-05T08:24:33.085448+0000","last_clean":"2024-02-05T08:24:33.085448+0000","last_became_active":"2024-02-05T08:22:09.110422+0000","last_became_peered":"2024-02-05T08:22:09.110422+0000","last_unstale":"2024-02-05T08:24:33.085448+0000","last_undegraded":"2024-02-05T08:24:33.085448+0000","last_fullsized":"2024-02-05T08:24:33.085448+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-05T08:21:50.244148+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:21:50.244148+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":731,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.092776+0000","last_change":"2024-02-05T08:21:37.234586+0000","last_active":"2024-02-05T08:24:33.092776+0000","last_peered":"2024-02-05T08:24:33.092776+0000","last_clean":"2024-02-05T08:24:33.092776+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:24:33.092776+0000","last_undegraded":"2024-02-05T08:24:33.092776+0000","last_fullsized":"2024-02-05T08:24:33.092776+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:37.234451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:21:37.234451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":743,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093656+0000","last_change":"2024-02-05T08:23:35.889657+0000","last_active":"2024-02-05T08:24:33.093656+0000","last_peered":"2024-02-05T08:24:33.093656+0000","last_clean":"2024-02-05T08:24:33.093656+0000","last_became_active":"2024-02-05T08:23:07.932741+0000","last_became_peered":"2024-02-05T08:23:07.932741+0000","last_unstale":"2024-02-05T08:24:33.093656+0000","last_undegraded":"2024-02-05T08:24:33.093656+0000","last_fullsized":"2024-02-05T08:24:33.093656+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:35.889538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:23:35.889538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:47.168827+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":482,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.088656+0000","last_change":"2024-02-05T08:24:19.508139+0000","last_active":"2024-02-05T08:24:33.088656+0000","last_peered":"2024-02-05T08:24:33.088656+0000","last_clean":"2024-02-05T08:24:33.088656+0000","last_became_active":"2024-02-05T08:22:09.097843+0000","last_became_peered":"2024-02-05T08:22:09.097843+0000","last_unstale":"2024-02-05T08:24:33.088656+0000","last_undegraded":"2024-02-05T08:24:33.088656+0000","last_fullsized":"2024-02-05T08:24:33.088656+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:19.508059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:19.508059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:26.904195+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'15","reported_seq":230,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.084971+0000","last_change":"2024-02-05T08:23:26.922363+0000","last_active":"2024-02-05T08:24:33.084971+0000","last_peered":"2024-02-05T08:24:33.084971+0000","last_clean":"2024-02-05T08:24:33.084971+0000","last_became_active":"2024-02-05T08:23:07.927922+0000","last_became_peered":"2024-02-05T08:23:07.927922+0000","last_unstale":"2024-02-05T08:24:33.084971+0000","last_undegraded":"2024-02-05T08:24:33.084971+0000","last_fullsized":"2024-02-05T08:24:33.084971+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:23:26.922227+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:23:26.922227+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"391'7223","reported_seq":7281,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.606242+0000","last_change":"2024-02-05T08:23:08.913493+0000","last_active":"2024-02-05T08:24:33.606242+0000","last_peered":"2024-02-05T08:24:33.606242+0000","last_clean":"2024-02-05T08:24:33.606242+0000","last_became_active":"2024-02-05T08:23:08.913227+0000","last_became_peered":"2024-02-05T08:23:08.913227+0000","last_unstale":"2024-02-05T08:24:33.606242+0000","last_undegraded":"2024-02-05T08:24:33.606242+0000","last_fullsized":"2024-02-05T08:24:33.606242+0000","mapping_epoch":367,"log_start":"381'2600","ondisk_log_start":"381'2600","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473366528,"num_objects":7223,"num_object_clones":0,"num_object_copies":21669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7223,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":753,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.094217+0000","last_change":"2024-02-05T08:24:25.887052+0000","last_active":"2024-02-05T08:24:33.094217+0000","last_peered":"2024-02-05T08:24:33.094217+0000","last_clean":"2024-02-05T08:24:33.094217+0000","last_became_active":"2024-02-05T08:22:09.100322+0000","last_became_peered":"2024-02-05T08:22:09.100322+0000","last_unstale":"2024-02-05T08:24:33.094217+0000","last_undegraded":"2024-02-05T08:24:33.094217+0000","last_fullsized":"2024-02-05T08:24:33.094217+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:25.886939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:25.886939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:51.587172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":767,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.089809+0000","last_change":"2024-02-05T08:23:43.066628+0000","last_active":"2024-02-05T08:24:33.089809+0000","last_peered":"2024-02-05T08:24:33.089809+0000","last_clean":"2024-02-05T08:24:33.089809+0000","last_became_active":"2024-02-05T08:23:08.920320+0000","last_became_peered":"2024-02-05T08:23:08.920320+0000","last_unstale":"2024-02-05T08:24:33.089809+0000","last_undegraded":"2024-02-05T08:24:33.089809+0000","last_fullsized":"2024-02-05T08:24:33.089809+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:23:43.066380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:23:43.066380+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:53.616786+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"262'34","reported_seq":829,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.091508+0000","last_change":"2024-02-05T08:24:24.865550+0000","last_active":"2024-02-05T08:24:33.091508+0000","last_peered":"2024-02-05T08:24:33.091508+0000","last_clean":"2024-02-05T08:24:33.091508+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:24:33.091508+0000","last_undegraded":"2024-02-05T08:24:33.091508+0000","last_fullsized":"2024-02-05T08:24:33.091508+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'34","last_scrub_stamp":"2024-02-05T08:24:24.865290+0000","last_deep_scrub":"262'34","last_deep_scrub_stamp":"2024-02-05T08:19:34.364738+0000","last_clean_scrub_stamp":"2024-02-05T08:24:24.865290+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:50.095788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":1000,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.089146+0000","last_change":"2024-02-05T08:24:26.044610+0000","last_active":"2024-02-05T08:24:33.089146+0000","last_peered":"2024-02-05T08:24:33.089146+0000","last_clean":"2024-02-05T08:24:33.089146+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:24:33.089146+0000","last_undegraded":"2024-02-05T08:24:33.089146+0000","last_fullsized":"2024-02-05T08:24:33.089146+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:24:26.044511+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_clean_scrub_stamp":"2024-02-05T08:24:26.044511+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:33.058094+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"391'7117","reported_seq":7223,"reported_epoch":391,"state":"active+clean+scrubbing","last_fresh":"2024-02-05T08:24:33.393586+0000","last_change":"2024-02-05T08:24:31.814835+0000","last_active":"2024-02-05T08:24:33.393586+0000","last_peered":"2024-02-05T08:24:33.393586+0000","last_clean":"2024-02-05T08:24:33.393586+0000","last_became_active":"2024-02-05T08:23:08.910680+0000","last_became_peered":"2024-02-05T08:23:08.910680+0000","last_unstale":"2024-02-05T08:24:33.393586+0000","last_undegraded":"2024-02-05T08:24:33.393586+0000","last_fullsized":"2024-02-05T08:24:33.393586+0000","mapping_epoch":367,"log_start":"380'2200","ondisk_log_start":"380'2200","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":4349,"log_size":4917,"log_dups_size":0,"ondisk_log_size":4917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466419712,"num_objects":7117,"num_object_clones":0,"num_object_copies":21351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7117,"num_write_kb":455488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":222,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.083773+0000","last_change":"2024-02-05T08:23:29.086552+0000","last_active":"2024-02-05T08:24:33.083773+0000","last_peered":"2024-02-05T08:24:33.083773+0000","last_clean":"2024-02-05T08:24:33.083773+0000","last_became_active":"2024-02-05T08:23:07.929942+0000","last_became_peered":"2024-02-05T08:23:07.929942+0000","last_unstale":"2024-02-05T08:24:33.083773+0000","last_undegraded":"2024-02-05T08:24:33.083773+0000","last_fullsized":"2024-02-05T08:24:33.083773+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:29.086293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.086293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:52.169995+0000","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":740,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.089904+0000","last_change":"2024-02-05T08:23:30.010159+0000","last_active":"2024-02-05T08:24:33.089904+0000","last_peered":"2024-02-05T08:24:33.089904+0000","last_clean":"2024-02-05T08:24:33.089904+0000","last_became_active":"2024-02-05T08:23:07.937863+0000","last_became_peered":"2024-02-05T08:23:07.937863+0000","last_unstale":"2024-02-05T08:24:33.089904+0000","last_undegraded":"2024-02-05T08:24:33.089904+0000","last_fullsized":"2024-02-05T08:24:33.089904+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:30.010028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:23:30.010028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:57.900947+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":794,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.066467+0000","last_change":"2024-02-05T08:23:30.492935+0000","last_active":"2024-02-05T08:24:31.066467+0000","last_peered":"2024-02-05T08:24:31.066467+0000","last_clean":"2024-02-05T08:24:31.066467+0000","last_became_active":"2024-02-05T08:23:07.929900+0000","last_became_peered":"2024-02-05T08:23:07.929900+0000","last_unstale":"2024-02-05T08:24:31.066467+0000","last_undegraded":"2024-02-05T08:24:31.066467+0000","last_fullsized":"2024-02-05T08:24:31.066467+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-02-05T08:23:30.492582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:30.492582+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:58.313450+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":752,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.081182+0000","last_change":"2024-02-05T08:24:18.737115+0000","last_active":"2024-02-05T08:24:33.081182+0000","last_peered":"2024-02-05T08:24:33.081182+0000","last_clean":"2024-02-05T08:24:33.081182+0000","last_became_active":"2024-02-05T08:23:08.918318+0000","last_became_peered":"2024-02-05T08:23:08.918318+0000","last_unstale":"2024-02-05T08:24:33.081182+0000","last_undegraded":"2024-02-05T08:24:33.081182+0000","last_fullsized":"2024-02-05T08:24:33.081182+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:18.736973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:24:18.736973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:26.663379+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"391'7211","reported_seq":7269,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.398718+0000","last_change":"2024-02-05T08:23:08.911955+0000","last_active":"2024-02-05T08:24:33.398718+0000","last_peered":"2024-02-05T08:24:33.398718+0000","last_clean":"2024-02-05T08:24:33.398718+0000","last_became_active":"2024-02-05T08:23:08.911591+0000","last_became_peered":"2024-02-05T08:23:08.911591+0000","last_unstale":"2024-02-05T08:24:33.398718+0000","last_undegraded":"2024-02-05T08:24:33.398718+0000","last_fullsized":"2024-02-05T08:24:33.398718+0000","mapping_epoch":367,"log_start":"381'2700","ondisk_log_start":"381'2700","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4511,"log_dups_size":0,"ondisk_log_size":4511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472580096,"num_objects":7211,"num_object_clones":0,"num_object_copies":21633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7211,"num_write_kb":461504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":740,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.088396+0000","last_change":"2024-02-05T08:24:27.030998+0000","last_active":"2024-02-05T08:24:33.088396+0000","last_peered":"2024-02-05T08:24:33.088396+0000","last_clean":"2024-02-05T08:24:33.088396+0000","last_became_active":"2024-02-05T08:18:34.642239+0000","last_became_peered":"2024-02-05T08:18:34.642239+0000","last_unstale":"2024-02-05T08:24:33.088396+0000","last_undegraded":"2024-02-05T08:24:33.088396+0000","last_fullsized":"2024-02-05T08:24:33.088396+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:27.030762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:27.030762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:39.215089+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":752,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.080969+0000","last_change":"2024-02-05T08:24:20.763701+0000","last_active":"2024-02-05T08:24:33.080969+0000","last_peered":"2024-02-05T08:24:33.080969+0000","last_clean":"2024-02-05T08:24:33.080969+0000","last_became_active":"2024-02-05T08:22:09.105439+0000","last_became_peered":"2024-02-05T08:22:09.105439+0000","last_unstale":"2024-02-05T08:24:33.080969+0000","last_undegraded":"2024-02-05T08:24:33.080969+0000","last_fullsized":"2024-02-05T08:24:33.080969+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:20.763558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:24:20.763558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:34.872761+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":744,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.095458+0000","last_change":"2024-02-05T08:21:40.182453+0000","last_active":"2024-02-05T08:24:33.095458+0000","last_peered":"2024-02-05T08:24:33.095458+0000","last_clean":"2024-02-05T08:24:33.095458+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:24:33.095458+0000","last_undegraded":"2024-02-05T08:24:33.095458+0000","last_fullsized":"2024-02-05T08:24:33.095458+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:40.182334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_clean_scrub_stamp":"2024-02-05T08:21:40.182334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"236'35","reported_seq":520,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.089386+0000","last_change":"2024-02-05T08:23:29.598019+0000","last_active":"2024-02-05T08:24:33.089386+0000","last_peered":"2024-02-05T08:24:33.089386+0000","last_clean":"2024-02-05T08:24:33.089386+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:24:33.089386+0000","last_undegraded":"2024-02-05T08:24:33.089386+0000","last_fullsized":"2024-02-05T08:24:33.089386+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'35","last_scrub_stamp":"2024-02-05T08:23:29.597891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.597891+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:35.763221+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"391'7214","reported_seq":7271,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.358808+0000","last_change":"2024-02-05T08:23:08.912293+0000","last_active":"2024-02-05T08:24:33.358808+0000","last_peered":"2024-02-05T08:24:33.358808+0000","last_clean":"2024-02-05T08:24:33.358808+0000","last_became_active":"2024-02-05T08:23:08.911987+0000","last_became_peered":"2024-02-05T08:23:08.911987+0000","last_unstale":"2024-02-05T08:24:33.358808+0000","last_undegraded":"2024-02-05T08:24:33.358808+0000","last_fullsized":"2024-02-05T08:24:33.358808+0000","mapping_epoch":367,"log_start":"381'2700","ondisk_log_start":"381'2700","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472776704,"num_objects":7214,"num_object_clones":0,"num_object_copies":21642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7214,"num_write_kb":461696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":97,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.088457+0000","last_change":"2024-02-05T08:24:25.220957+0000","last_active":"2024-02-05T08:24:33.088457+0000","last_peered":"2024-02-05T08:24:33.088457+0000","last_clean":"2024-02-05T08:24:33.088457+0000","last_became_active":"2024-02-05T08:22:09.100986+0000","last_became_peered":"2024-02-05T08:22:09.100986+0000","last_unstale":"2024-02-05T08:24:33.088457+0000","last_undegraded":"2024-02-05T08:24:33.088457+0000","last_fullsized":"2024-02-05T08:24:33.088457+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:25.220831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_clean_scrub_stamp":"2024-02-05T08:24:25.220831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:37.504652+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":759,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.084319+0000","last_change":"2024-02-05T08:22:59.883429+0000","last_active":"2024-02-05T08:24:33.084319+0000","last_peered":"2024-02-05T08:24:33.084319+0000","last_clean":"2024-02-05T08:24:33.084319+0000","last_became_active":"2024-02-05T08:18:34.645830+0000","last_became_peered":"2024-02-05T08:18:34.645830+0000","last_unstale":"2024-02-05T08:24:33.084319+0000","last_undegraded":"2024-02-05T08:24:33.084319+0000","last_fullsized":"2024-02-05T08:24:33.084319+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:59.883347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:59.883347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":762,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093065+0000","last_change":"2024-02-05T08:23:01.230170+0000","last_active":"2024-02-05T08:24:33.093065+0000","last_peered":"2024-02-05T08:24:33.093065+0000","last_clean":"2024-02-05T08:24:33.093065+0000","last_became_active":"2024-02-05T08:18:34.633214+0000","last_became_peered":"2024-02-05T08:18:34.633214+0000","last_unstale":"2024-02-05T08:24:33.093065+0000","last_undegraded":"2024-02-05T08:24:33.093065+0000","last_fullsized":"2024-02-05T08:24:33.093065+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:01.230038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:01.230038+0000","last_clean_scrub_stamp":"2024-02-05T08:23:01.230038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"262'19","reported_seq":515,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:13.934769+0000","last_change":"2024-02-05T08:23:08.921585+0000","last_active":"2024-02-05T08:24:13.934769+0000","last_peered":"2024-02-05T08:24:13.934769+0000","last_clean":"2024-02-05T08:24:13.934769+0000","last_became_active":"2024-02-05T08:23:08.920885+0000","last_became_peered":"2024-02-05T08:23:08.920885+0000","last_unstale":"2024-02-05T08:24:13.934769+0000","last_undegraded":"2024-02-05T08:24:13.934769+0000","last_fullsized":"2024-02-05T08:24:13.934769+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'19","last_scrub_stamp":"2024-02-05T08:22:53.159489+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:22:53.159489+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"391'7118","reported_seq":7176,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.354192+0000","last_change":"2024-02-05T08:23:08.917492+0000","last_active":"2024-02-05T08:24:33.354192+0000","last_peered":"2024-02-05T08:24:33.354192+0000","last_clean":"2024-02-05T08:24:33.354192+0000","last_became_active":"2024-02-05T08:23:08.917212+0000","last_became_peered":"2024-02-05T08:23:08.917212+0000","last_unstale":"2024-02-05T08:24:33.354192+0000","last_undegraded":"2024-02-05T08:24:33.354192+0000","last_fullsized":"2024-02-05T08:24:33.354192+0000","mapping_epoch":367,"log_start":"381'3100","ondisk_log_start":"381'3100","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466485248,"num_objects":7118,"num_object_clones":0,"num_object_copies":21354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7118,"num_write_kb":455552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'130","reported_seq":522,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.091843+0000","last_change":"2024-02-05T08:23:43.752206+0000","last_active":"2024-02-05T08:24:33.091843+0000","last_peered":"2024-02-05T08:24:33.091843+0000","last_clean":"2024-02-05T08:24:33.091843+0000","last_became_active":"2024-02-05T08:22:12.146682+0000","last_became_peered":"2024-02-05T08:22:12.146682+0000","last_unstale":"2024-02-05T08:24:33.091843+0000","last_undegraded":"2024-02-05T08:24:33.091843+0000","last_fullsized":"2024-02-05T08:24:33.091843+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:43.752076+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:23:43.752076+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:59.361461+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":448,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065421+0000","last_change":"2024-02-05T08:23:26.660184+0000","last_active":"2024-02-05T08:24:31.065421+0000","last_peered":"2024-02-05T08:24:31.065421+0000","last_clean":"2024-02-05T08:24:31.065421+0000","last_became_active":"2024-02-05T08:23:07.936660+0000","last_became_peered":"2024-02-05T08:23:07.936660+0000","last_unstale":"2024-02-05T08:24:31.065421+0000","last_undegraded":"2024-02-05T08:24:31.065421+0000","last_fullsized":"2024-02-05T08:24:31.065421+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:26.660108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:26.660108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:46.099841+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":772,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.094000+0000","last_change":"2024-02-05T08:23:27.050401+0000","last_active":"2024-02-05T08:24:33.094000+0000","last_peered":"2024-02-05T08:24:33.094000+0000","last_clean":"2024-02-05T08:24:33.094000+0000","last_became_active":"2024-02-05T08:23:07.937565+0000","last_became_peered":"2024-02-05T08:23:07.937565+0000","last_unstale":"2024-02-05T08:24:33.094000+0000","last_undegraded":"2024-02-05T08:24:33.094000+0000","last_fullsized":"2024-02-05T08:24:33.094000+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:23:27.050262+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:23:27.050262+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:42.760480+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":751,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.066337+0000","last_change":"2024-02-05T08:23:08.921073+0000","last_active":"2024-02-05T08:24:31.066337+0000","last_peered":"2024-02-05T08:24:31.066337+0000","last_clean":"2024-02-05T08:24:31.066337+0000","last_became_active":"2024-02-05T08:23:08.920421+0000","last_became_peered":"2024-02-05T08:23:08.920421+0000","last_unstale":"2024-02-05T08:24:31.066337+0000","last_undegraded":"2024-02-05T08:24:31.066337+0000","last_fullsized":"2024-02-05T08:24:31.066337+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:46.849759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:10.946904+0000","last_clean_scrub_stamp":"2024-02-05T08:21:46.849759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"370'54","reported_seq":951,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:21.819409+0000","last_change":"2024-02-05T08:24:18.160994+0000","last_active":"2024-02-05T08:24:21.819409+0000","last_peered":"2024-02-05T08:24:21.819409+0000","last_clean":"2024-02-05T08:24:21.819409+0000","last_became_active":"2024-02-05T08:23:07.932513+0000","last_became_peered":"2024-02-05T08:23:07.932513+0000","last_unstale":"2024-02-05T08:24:21.819409+0000","last_undegraded":"2024-02-05T08:24:21.819409+0000","last_fullsized":"2024-02-05T08:24:21.819409+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'54","last_scrub_stamp":"2024-02-05T08:24:18.160865+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:24:18.160865+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:41.199814+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":55,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:13.932053+0000","last_change":"2024-02-05T08:23:08.915936+0000","last_active":"2024-02-05T08:24:13.932053+0000","last_peered":"2024-02-05T08:24:13.932053+0000","last_clean":"2024-02-05T08:24:13.932053+0000","last_became_active":"2024-02-05T08:23:08.914563+0000","last_became_peered":"2024-02-05T08:23:08.914563+0000","last_unstale":"2024-02-05T08:24:13.932053+0000","last_undegraded":"2024-02-05T08:24:13.932053+0000","last_fullsized":"2024-02-05T08:24:13.932053+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:52.171116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:52.171116+0000","last_clean_scrub_stamp":"2024-02-05T08:22:52.171116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:14.131755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"391'7258","reported_seq":7316,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.389914+0000","last_change":"2024-02-05T08:23:08.915803+0000","last_active":"2024-02-05T08:24:33.389914+0000","last_peered":"2024-02-05T08:24:33.389914+0000","last_clean":"2024-02-05T08:24:33.389914+0000","last_became_active":"2024-02-05T08:23:08.915520+0000","last_became_peered":"2024-02-05T08:23:08.915520+0000","last_unstale":"2024-02-05T08:24:33.389914+0000","last_undegraded":"2024-02-05T08:24:33.389914+0000","last_fullsized":"2024-02-05T08:24:33.389914+0000","mapping_epoch":367,"log_start":"381'3207","ondisk_log_start":"381'3207","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4051,"log_dups_size":0,"ondisk_log_size":4051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475660288,"num_objects":7258,"num_object_clones":0,"num_object_copies":21774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7258,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7258,"num_write_kb":464512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"82'130","reported_seq":1046,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.091192+0000","last_change":"2024-02-05T08:23:30.984623+0000","last_active":"2024-02-05T08:24:33.091192+0000","last_peered":"2024-02-05T08:24:33.091192+0000","last_clean":"2024-02-05T08:24:33.091192+0000","last_became_active":"2024-02-05T08:22:12.138774+0000","last_became_peered":"2024-02-05T08:22:12.138774+0000","last_unstale":"2024-02-05T08:24:33.091192+0000","last_undegraded":"2024-02-05T08:24:33.091192+0000","last_fullsized":"2024-02-05T08:24:33.091192+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:30.984495+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:23:30.984495+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:35.070753+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":744,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.095604+0000","last_change":"2024-02-05T08:22:09.032210+0000","last_active":"2024-02-05T08:24:33.095604+0000","last_peered":"2024-02-05T08:24:33.095604+0000","last_clean":"2024-02-05T08:24:33.095604+0000","last_became_active":"2024-02-05T08:18:34.643109+0000","last_became_peered":"2024-02-05T08:18:34.643109+0000","last_unstale":"2024-02-05T08:24:33.095604+0000","last_undegraded":"2024-02-05T08:24:33.095604+0000","last_fullsized":"2024-02-05T08:24:33.095604+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:09.032112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:09.032112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":24,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.085734+0000","last_change":"2024-02-05T08:24:08.030313+0000","last_active":"2024-02-05T08:24:33.085734+0000","last_peered":"2024-02-05T08:24:33.085734+0000","last_clean":"2024-02-05T08:24:33.085734+0000","last_became_active":"2024-02-05T08:24:08.005056+0000","last_became_peered":"2024-02-05T08:24:08.005056+0000","last_unstale":"2024-02-05T08:24:33.085734+0000","last_undegraded":"2024-02-05T08:24:33.085734+0000","last_fullsized":"2024-02-05T08:24:33.085734+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:23:23.016855+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:23:23.016855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:34.581244+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":786,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.080929+0000","last_change":"2024-02-05T08:24:17.745325+0000","last_active":"2024-02-05T08:24:33.080929+0000","last_peered":"2024-02-05T08:24:33.080929+0000","last_clean":"2024-02-05T08:24:33.080929+0000","last_became_active":"2024-02-05T08:23:08.912881+0000","last_became_peered":"2024-02-05T08:23:08.912881+0000","last_unstale":"2024-02-05T08:24:33.080929+0000","last_undegraded":"2024-02-05T08:24:33.080929+0000","last_fullsized":"2024-02-05T08:24:33.080929+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:17.745200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_clean_scrub_stamp":"2024-02-05T08:24:17.745200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:18.434153+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"363'22","reported_seq":784,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.082678+0000","last_change":"2024-02-05T08:24:21.745236+0000","last_active":"2024-02-05T08:24:33.082678+0000","last_peered":"2024-02-05T08:24:33.082678+0000","last_clean":"2024-02-05T08:24:33.082678+0000","last_became_active":"2024-02-05T08:22:09.103077+0000","last_became_peered":"2024-02-05T08:22:09.103077+0000","last_unstale":"2024-02-05T08:24:33.082678+0000","last_undegraded":"2024-02-05T08:24:33.082678+0000","last_fullsized":"2024-02-05T08:24:33.082678+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'22","last_scrub_stamp":"2024-02-05T08:24:21.745112+0000","last_deep_scrub":"363'22","last_deep_scrub_stamp":"2024-02-05T08:22:56.032522+0000","last_clean_scrub_stamp":"2024-02-05T08:24:21.745112+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:51.473685+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":755,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.083983+0000","last_change":"2024-02-05T08:24:22.757410+0000","last_active":"2024-02-05T08:24:33.083983+0000","last_peered":"2024-02-05T08:24:33.083983+0000","last_clean":"2024-02-05T08:24:33.083983+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:24:33.083983+0000","last_undegraded":"2024-02-05T08:24:33.083983+0000","last_fullsized":"2024-02-05T08:24:33.083983+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:22.757290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:24:22.757290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:38.124802+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"391'7246","reported_seq":7304,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.393548+0000","last_change":"2024-02-05T08:23:08.914881+0000","last_active":"2024-02-05T08:24:33.393548+0000","last_peered":"2024-02-05T08:24:33.393548+0000","last_clean":"2024-02-05T08:24:33.393548+0000","last_became_active":"2024-02-05T08:23:08.914611+0000","last_became_peered":"2024-02-05T08:23:08.914611+0000","last_unstale":"2024-02-05T08:24:33.393548+0000","last_undegraded":"2024-02-05T08:24:33.393548+0000","last_fullsized":"2024-02-05T08:24:33.393548+0000","mapping_epoch":367,"log_start":"381'3200","ondisk_log_start":"381'3200","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4046,"log_dups_size":0,"ondisk_log_size":4046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474873856,"num_objects":7246,"num_object_clones":0,"num_object_copies":21738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7246,"num_write_kb":463744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":739,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.092569+0000","last_change":"2024-02-05T08:23:44.881991+0000","last_active":"2024-02-05T08:24:33.092569+0000","last_peered":"2024-02-05T08:24:33.092569+0000","last_clean":"2024-02-05T08:24:33.092569+0000","last_became_active":"2024-02-05T08:18:34.633450+0000","last_became_peered":"2024-02-05T08:18:34.633450+0000","last_unstale":"2024-02-05T08:24:33.092569+0000","last_undegraded":"2024-02-05T08:24:33.092569+0000","last_fullsized":"2024-02-05T08:24:33.092569+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:44.881864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:44.881864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:48.073687+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":1043,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.088493+0000","last_change":"2024-02-05T08:23:34.058622+0000","last_active":"2024-02-05T08:24:33.088493+0000","last_peered":"2024-02-05T08:24:33.088493+0000","last_clean":"2024-02-05T08:24:33.088493+0000","last_became_active":"2024-02-05T08:23:10.886936+0000","last_became_peered":"2024-02-05T08:23:10.886936+0000","last_unstale":"2024-02-05T08:24:33.088493+0000","last_undegraded":"2024-02-05T08:24:33.088493+0000","last_fullsized":"2024-02-05T08:24:33.088493+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:34.058210+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_clean_scrub_stamp":"2024-02-05T08:23:34.058210+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:03.029013+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":229,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.089047+0000","last_change":"2024-02-05T08:22:54.937959+0000","last_active":"2024-02-05T08:24:33.089047+0000","last_peered":"2024-02-05T08:24:33.089047+0000","last_clean":"2024-02-05T08:24:33.089047+0000","last_became_active":"2024-02-05T08:18:34.631974+0000","last_became_peered":"2024-02-05T08:18:34.631974+0000","last_unstale":"2024-02-05T08:24:33.089047+0000","last_undegraded":"2024-02-05T08:24:33.089047+0000","last_fullsized":"2024-02-05T08:24:33.089047+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:54.937874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:54.937874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"236'30","reported_seq":649,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.090292+0000","last_change":"2024-02-05T08:23:26.566491+0000","last_active":"2024-02-05T08:24:33.090292+0000","last_peered":"2024-02-05T08:24:33.090292+0000","last_clean":"2024-02-05T08:24:33.090292+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:24:33.090292+0000","last_undegraded":"2024-02-05T08:24:33.090292+0000","last_fullsized":"2024-02-05T08:24:33.090292+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:23:26.566358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:26.566358+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:53.814998+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":732,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.089946+0000","last_change":"2024-02-05T08:24:27.990755+0000","last_active":"2024-02-05T08:24:33.089946+0000","last_peered":"2024-02-05T08:24:33.089946+0000","last_clean":"2024-02-05T08:24:33.089946+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:24:33.089946+0000","last_undegraded":"2024-02-05T08:24:33.089946+0000","last_fullsized":"2024-02-05T08:24:33.089946+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:27.990653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_clean_scrub_stamp":"2024-02-05T08:24:27.990653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:34.928370+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"391'7147","reported_seq":7205,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.385014+0000","last_change":"2024-02-05T08:23:08.916824+0000","last_active":"2024-02-05T08:24:33.385014+0000","last_peered":"2024-02-05T08:24:33.385014+0000","last_clean":"2024-02-05T08:24:33.385014+0000","last_became_active":"2024-02-05T08:23:08.916452+0000","last_became_peered":"2024-02-05T08:23:08.916452+0000","last_unstale":"2024-02-05T08:24:33.385014+0000","last_undegraded":"2024-02-05T08:24:33.385014+0000","last_fullsized":"2024-02-05T08:24:33.385014+0000","mapping_epoch":367,"log_start":"381'3124","ondisk_log_start":"381'3124","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4023,"log_dups_size":0,"ondisk_log_size":4023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:33.630773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468385792,"num_objects":7147,"num_object_clones":0,"num_object_copies":21441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7147,"num_write_kb":457408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"82'130","reported_seq":389,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:13.934860+0000","last_change":"2024-02-05T08:23:25.057316+0000","last_active":"2024-02-05T08:24:13.934860+0000","last_peered":"2024-02-05T08:24:13.934860+0000","last_clean":"2024-02-05T08:24:13.934860+0000","last_became_active":"2024-02-05T08:23:07.941142+0000","last_became_peered":"2024-02-05T08:23:07.941142+0000","last_unstale":"2024-02-05T08:24:13.934860+0000","last_undegraded":"2024-02-05T08:24:13.934860+0000","last_fullsized":"2024-02-05T08:24:13.934860+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:25.057181+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:23:25.057181+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:26.634639+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":631,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:13.933211+0000","last_change":"2024-02-05T08:23:39.136581+0000","last_active":"2024-02-05T08:24:13.933211+0000","last_peered":"2024-02-05T08:24:13.933211+0000","last_clean":"2024-02-05T08:24:13.933211+0000","last_became_active":"2024-02-05T08:23:07.935640+0000","last_became_peered":"2024-02-05T08:23:07.935640+0000","last_unstale":"2024-02-05T08:24:13.933211+0000","last_undegraded":"2024-02-05T08:24:13.933211+0000","last_fullsized":"2024-02-05T08:24:13.933211+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:39.136473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:39.136473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:50.220045+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":593,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.066318+0000","last_change":"2024-02-05T08:22:09.103119+0000","last_active":"2024-02-05T08:24:31.066318+0000","last_peered":"2024-02-05T08:24:31.066318+0000","last_clean":"2024-02-05T08:24:31.066318+0000","last_became_active":"2024-02-05T08:22:09.102613+0000","last_became_peered":"2024-02-05T08:22:09.102613+0000","last_unstale":"2024-02-05T08:24:31.066318+0000","last_undegraded":"2024-02-05T08:24:31.066318+0000","last_fullsized":"2024-02-05T08:24:31.066318+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:57.240901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:57.240901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":1942,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.088503+0000","last_change":"2024-02-05T08:23:24.028818+0000","last_active":"2024-02-05T08:24:33.088503+0000","last_peered":"2024-02-05T08:24:33.088503+0000","last_clean":"2024-02-05T08:24:33.088503+0000","last_became_active":"2024-02-05T08:23:08.921709+0000","last_became_peered":"2024-02-05T08:23:08.921709+0000","last_unstale":"2024-02-05T08:24:33.088503+0000","last_undegraded":"2024-02-05T08:24:33.088503+0000","last_fullsized":"2024-02-05T08:24:33.088503+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:23:24.028705+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:23:24.028705+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:37.647929+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1174,"num_read_kb":1169,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":536,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093742+0000","last_change":"2024-02-05T08:24:19.820651+0000","last_active":"2024-02-05T08:24:33.093742+0000","last_peered":"2024-02-05T08:24:33.093742+0000","last_clean":"2024-02-05T08:24:33.093742+0000","last_became_active":"2024-02-05T08:22:09.106287+0000","last_became_peered":"2024-02-05T08:22:09.106287+0000","last_unstale":"2024-02-05T08:24:33.093742+0000","last_undegraded":"2024-02-05T08:24:33.093742+0000","last_fullsized":"2024-02-05T08:24:33.093742+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:24:19.820518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:24:19.820518+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:41.780807+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":736,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093643+0000","last_change":"2024-02-05T08:23:08.914768+0000","last_active":"2024-02-05T08:24:33.093643+0000","last_peered":"2024-02-05T08:24:33.093643+0000","last_clean":"2024-02-05T08:24:33.093643+0000","last_became_active":"2024-02-05T08:23:08.914353+0000","last_became_peered":"2024-02-05T08:23:08.914353+0000","last_unstale":"2024-02-05T08:24:33.093643+0000","last_undegraded":"2024-02-05T08:24:33.093643+0000","last_fullsized":"2024-02-05T08:24:33.093643+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:53.288625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:22:53.288625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"389'6945","reported_seq":6999,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:30.975381+0000","last_change":"2024-02-05T08:23:08.921308+0000","last_active":"2024-02-05T08:24:30.975381+0000","last_peered":"2024-02-05T08:24:30.975381+0000","last_clean":"2024-02-05T08:24:30.975381+0000","last_became_active":"2024-02-05T08:23:08.920532+0000","last_became_peered":"2024-02-05T08:23:08.920532+0000","last_unstale":"2024-02-05T08:24:30.975381+0000","last_undegraded":"2024-02-05T08:24:30.975381+0000","last_fullsized":"2024-02-05T08:24:30.975381+0000","mapping_epoch":367,"log_start":"381'3000","ondisk_log_start":"381'3000","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:35.331168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455147520,"num_objects":6945,"num_object_clones":0,"num_object_copies":20835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6945,"num_write_kb":444480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'130","reported_seq":783,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.090026+0000","last_change":"2024-02-05T08:23:32.482420+0000","last_active":"2024-02-05T08:24:33.090026+0000","last_peered":"2024-02-05T08:24:33.090026+0000","last_clean":"2024-02-05T08:24:33.090026+0000","last_became_active":"2024-02-05T08:22:12.143283+0000","last_became_peered":"2024-02-05T08:22:12.143283+0000","last_unstale":"2024-02-05T08:24:33.090026+0000","last_undegraded":"2024-02-05T08:24:33.090026+0000","last_fullsized":"2024-02-05T08:24:33.090026+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:32.482297+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:23:32.482297+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:35.475259+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":640,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:29.342439+0000","last_change":"2024-02-05T08:24:29.342475+0000","last_active":"2024-02-05T08:24:29.342439+0000","last_peered":"2024-02-05T08:24:29.342439+0000","last_clean":"2024-02-05T08:24:29.342439+0000","last_became_active":"2024-02-05T08:23:07.928514+0000","last_became_peered":"2024-02-05T08:23:07.928514+0000","last_unstale":"2024-02-05T08:24:29.342439+0000","last_undegraded":"2024-02-05T08:24:29.342439+0000","last_fullsized":"2024-02-05T08:24:29.342439+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:29.342365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:29.342365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:33.382311+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":595,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:13.933476+0000","last_change":"2024-02-05T08:23:27.086997+0000","last_active":"2024-02-05T08:24:13.933476+0000","last_peered":"2024-02-05T08:24:13.933476+0000","last_clean":"2024-02-05T08:24:13.933476+0000","last_became_active":"2024-02-05T08:23:07.935692+0000","last_became_peered":"2024-02-05T08:23:07.935692+0000","last_unstale":"2024-02-05T08:24:13.933476+0000","last_undegraded":"2024-02-05T08:24:13.933476+0000","last_fullsized":"2024-02-05T08:24:13.933476+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:27.086862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:23:27.086862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:27.202163+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"381'37","reported_seq":555,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.092322+0000","last_change":"2024-02-05T08:23:36.866473+0000","last_active":"2024-02-05T08:24:33.092322+0000","last_peered":"2024-02-05T08:24:33.092322+0000","last_clean":"2024-02-05T08:24:33.092322+0000","last_became_active":"2024-02-05T08:23:08.917913+0000","last_became_peered":"2024-02-05T08:23:08.917913+0000","last_unstale":"2024-02-05T08:24:33.092322+0000","last_undegraded":"2024-02-05T08:24:33.092322+0000","last_fullsized":"2024-02-05T08:24:33.092322+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'37","last_scrub_stamp":"2024-02-05T08:23:36.866312+0000","last_deep_scrub":"381'37","last_deep_scrub_stamp":"2024-02-05T08:23:36.866312+0000","last_clean_scrub_stamp":"2024-02-05T08:23:36.866312+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:54.708384+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":474,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.089303+0000","last_change":"2024-02-05T08:22:49.513009+0000","last_active":"2024-02-05T08:24:33.089303+0000","last_peered":"2024-02-05T08:24:33.089303+0000","last_clean":"2024-02-05T08:24:33.089303+0000","last_became_active":"2024-02-05T08:22:09.096719+0000","last_became_peered":"2024-02-05T08:22:09.096719+0000","last_unstale":"2024-02-05T08:24:33.089303+0000","last_undegraded":"2024-02-05T08:24:33.089303+0000","last_fullsized":"2024-02-05T08:24:33.089303+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:49.512928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:22:49.512928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"391'7173","reported_seq":7310,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.480639+0000","last_change":"2024-02-05T08:24:33.152720+0000","last_active":"2024-02-05T08:24:33.480639+0000","last_peered":"2024-02-05T08:24:33.480639+0000","last_clean":"2024-02-05T08:24:33.480639+0000","last_became_active":"2024-02-05T08:23:08.911093+0000","last_became_peered":"2024-02-05T08:23:08.911093+0000","last_unstale":"2024-02-05T08:24:33.480639+0000","last_undegraded":"2024-02-05T08:24:33.480639+0000","last_fullsized":"2024-02-05T08:24:33.480639+0000","mapping_epoch":367,"log_start":"381'2300","ondisk_log_start":"381'2300","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'7149","last_scrub_stamp":"2024-02-05T08:24:33.152473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:24:33.152473+0000","objects_scrubbed":7060,"log_size":4873,"log_dups_size":0,"ondisk_log_size":4873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:55.944976+0000","scrub_duration":2346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470089728,"num_objects":7173,"num_object_clones":0,"num_object_copies":21519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7173,"num_write_kb":459072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"82'130","reported_seq":774,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.090339+0000","last_change":"2024-02-05T08:23:33.451461+0000","last_active":"2024-02-05T08:24:33.090339+0000","last_peered":"2024-02-05T08:24:33.090339+0000","last_clean":"2024-02-05T08:24:33.090339+0000","last_became_active":"2024-02-05T08:19:01.815446+0000","last_became_peered":"2024-02-05T08:19:01.815446+0000","last_unstale":"2024-02-05T08:24:33.090339+0000","last_undegraded":"2024-02-05T08:24:33.090339+0000","last_fullsized":"2024-02-05T08:24:33.090339+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:33.451336+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:35.974462+0000","last_clean_scrub_stamp":"2024-02-05T08:23:33.451336+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:33.667706+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":599,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.085220+0000","last_change":"2024-02-05T08:22:57.918423+0000","last_active":"2024-02-05T08:24:33.085220+0000","last_peered":"2024-02-05T08:24:33.085220+0000","last_clean":"2024-02-05T08:24:33.085220+0000","last_became_active":"2024-02-05T08:22:09.100096+0000","last_became_peered":"2024-02-05T08:22:09.100096+0000","last_unstale":"2024-02-05T08:24:33.085220+0000","last_undegraded":"2024-02-05T08:24:33.085220+0000","last_fullsized":"2024-02-05T08:24:33.085220+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:57.918346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:57.918346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":751,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093423+0000","last_change":"2024-02-05T08:23:38.820649+0000","last_active":"2024-02-05T08:24:33.093423+0000","last_peered":"2024-02-05T08:24:33.093423+0000","last_clean":"2024-02-05T08:24:33.093423+0000","last_became_active":"2024-02-05T08:18:34.633263+0000","last_became_peered":"2024-02-05T08:18:34.633263+0000","last_unstale":"2024-02-05T08:24:33.093423+0000","last_undegraded":"2024-02-05T08:24:33.093423+0000","last_fullsized":"2024-02-05T08:24:33.093423+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:38.820534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:23:38.820534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:42.910692+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'20","reported_seq":651,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:13.931929+0000","last_change":"2024-02-05T08:23:41.072770+0000","last_active":"2024-02-05T08:24:13.931929+0000","last_peered":"2024-02-05T08:24:13.931929+0000","last_clean":"2024-02-05T08:24:13.931929+0000","last_became_active":"2024-02-05T08:23:07.941561+0000","last_became_peered":"2024-02-05T08:23:07.941561+0000","last_unstale":"2024-02-05T08:24:13.931929+0000","last_undegraded":"2024-02-05T08:24:13.931929+0000","last_fullsized":"2024-02-05T08:24:13.931929+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:23:41.072637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:41.072637+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:42.721873+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"355'12","reported_seq":769,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.091690+0000","last_change":"2024-02-05T08:23:02.552120+0000","last_active":"2024-02-05T08:24:33.091690+0000","last_peered":"2024-02-05T08:24:33.091690+0000","last_clean":"2024-02-05T08:24:33.091690+0000","last_became_active":"2024-02-05T08:22:09.099076+0000","last_became_peered":"2024-02-05T08:22:09.099076+0000","last_unstale":"2024-02-05T08:24:33.091690+0000","last_undegraded":"2024-02-05T08:24:33.091690+0000","last_fullsized":"2024-02-05T08:24:33.091690+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'12","last_scrub_stamp":"2024-02-05T08:23:02.552025+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:17:07.973171+0000","last_clean_scrub_stamp":"2024-02-05T08:23:02.552025+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.e","version":"390'7125","reported_seq":7181,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:32.888954+0000","last_change":"2024-02-05T08:23:08.910748+0000","last_active":"2024-02-05T08:24:32.888954+0000","last_peered":"2024-02-05T08:24:32.888954+0000","last_clean":"2024-02-05T08:24:32.888954+0000","last_became_active":"2024-02-05T08:23:08.910360+0000","last_became_peered":"2024-02-05T08:23:08.910360+0000","last_unstale":"2024-02-05T08:24:32.888954+0000","last_undegraded":"2024-02-05T08:24:32.888954+0000","last_fullsized":"2024-02-05T08:24:32.888954+0000","mapping_epoch":367,"log_start":"381'2800","ondisk_log_start":"381'2800","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4325,"log_dups_size":0,"ondisk_log_size":4325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466944000,"num_objects":7125,"num_object_clones":0,"num_object_copies":21375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7125,"num_write_kb":456000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"82'130","reported_seq":505,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.083822+0000","last_change":"2024-02-05T08:22:47.999978+0000","last_active":"2024-02-05T08:24:33.083822+0000","last_peered":"2024-02-05T08:24:33.083822+0000","last_clean":"2024-02-05T08:24:33.083822+0000","last_became_active":"2024-02-05T08:19:01.819856+0000","last_became_peered":"2024-02-05T08:19:01.819856+0000","last_unstale":"2024-02-05T08:24:33.083822+0000","last_undegraded":"2024-02-05T08:24:33.083822+0000","last_fullsized":"2024-02-05T08:24:33.083822+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:22:47.999878+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:22:47.999878+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":732,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.090614+0000","last_change":"2024-02-05T08:23:36.150803+0000","last_active":"2024-02-05T08:24:33.090614+0000","last_peered":"2024-02-05T08:24:33.090614+0000","last_clean":"2024-02-05T08:24:33.090614+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:24:33.090614+0000","last_undegraded":"2024-02-05T08:24:33.090614+0000","last_fullsized":"2024-02-05T08:24:33.090614+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:36.150674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:36.150674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:56.661921+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":71,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:30.256625+0000","last_change":"2024-02-05T08:24:30.256669+0000","last_active":"2024-02-05T08:24:30.256625+0000","last_peered":"2024-02-05T08:24:30.256625+0000","last_clean":"2024-02-05T08:24:30.256625+0000","last_became_active":"2024-02-05T08:23:08.913263+0000","last_became_peered":"2024-02-05T08:23:08.913263+0000","last_unstale":"2024-02-05T08:24:30.256625+0000","last_undegraded":"2024-02-05T08:24:30.256625+0000","last_fullsized":"2024-02-05T08:24:30.256625+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:24:30.256524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:30.256524+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:30.313107+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"236'28","reported_seq":811,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.066222+0000","last_change":"2024-02-05T08:24:16.241737+0000","last_active":"2024-02-05T08:24:31.066222+0000","last_peered":"2024-02-05T08:24:31.066222+0000","last_clean":"2024-02-05T08:24:31.066222+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:24:31.066222+0000","last_undegraded":"2024-02-05T08:24:31.066222+0000","last_fullsized":"2024-02-05T08:24:31.066222+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:24:16.241606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:24:16.241606+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:34.065074+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"370'13","reported_seq":309,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.086007+0000","last_change":"2024-02-05T08:23:33.927361+0000","last_active":"2024-02-05T08:24:33.086007+0000","last_peered":"2024-02-05T08:24:33.086007+0000","last_clean":"2024-02-05T08:24:33.086007+0000","last_became_active":"2024-02-05T08:23:07.936347+0000","last_became_peered":"2024-02-05T08:23:07.936347+0000","last_unstale":"2024-02-05T08:24:33.086007+0000","last_undegraded":"2024-02-05T08:24:33.086007+0000","last_fullsized":"2024-02-05T08:24:33.086007+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'13","last_scrub_stamp":"2024-02-05T08:23:33.927148+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:23:33.927148+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:43.180187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"391'7090","reported_seq":7147,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.390462+0000","last_change":"2024-02-05T08:23:08.921447+0000","last_active":"2024-02-05T08:24:33.390462+0000","last_peered":"2024-02-05T08:24:33.390462+0000","last_clean":"2024-02-05T08:24:33.390462+0000","last_became_active":"2024-02-05T08:23:08.921121+0000","last_became_peered":"2024-02-05T08:23:08.921121+0000","last_unstale":"2024-02-05T08:24:33.390462+0000","last_undegraded":"2024-02-05T08:24:33.390462+0000","last_fullsized":"2024-02-05T08:24:33.390462+0000","mapping_epoch":367,"log_start":"381'2600","ondisk_log_start":"381'2600","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464650240,"num_objects":7090,"num_object_clones":0,"num_object_copies":21270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7090,"num_write_kb":453760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"82'130","reported_seq":503,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.066699+0000","last_change":"2024-02-05T08:22:58.247977+0000","last_active":"2024-02-05T08:24:31.066699+0000","last_peered":"2024-02-05T08:24:31.066699+0000","last_clean":"2024-02-05T08:24:31.066699+0000","last_became_active":"2024-02-05T08:19:01.816923+0000","last_became_peered":"2024-02-05T08:19:01.816923+0000","last_unstale":"2024-02-05T08:24:31.066699+0000","last_undegraded":"2024-02-05T08:24:31.066699+0000","last_fullsized":"2024-02-05T08:24:31.066699+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:22:58.247896+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:22:58.247896+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":737,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065496+0000","last_change":"2024-02-05T08:22:08.118133+0000","last_active":"2024-02-05T08:24:31.065496+0000","last_peered":"2024-02-05T08:24:31.065496+0000","last_clean":"2024-02-05T08:24:31.065496+0000","last_became_active":"2024-02-05T08:18:34.635908+0000","last_became_peered":"2024-02-05T08:18:34.635908+0000","last_unstale":"2024-02-05T08:24:31.065496+0000","last_undegraded":"2024-02-05T08:24:31.065496+0000","last_fullsized":"2024-02-05T08:24:31.065496+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:08.117925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:08.117925+0000","last_clean_scrub_stamp":"2024-02-05T08:22:08.117925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":243,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.084875+0000","last_change":"2024-02-05T08:22:56.880110+0000","last_active":"2024-02-05T08:24:33.084875+0000","last_peered":"2024-02-05T08:24:33.084875+0000","last_clean":"2024-02-05T08:24:33.084875+0000","last_became_active":"2024-02-05T08:22:09.101588+0000","last_became_peered":"2024-02-05T08:22:09.101588+0000","last_unstale":"2024-02-05T08:24:33.084875+0000","last_undegraded":"2024-02-05T08:24:33.084875+0000","last_fullsized":"2024-02-05T08:24:33.084875+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:22:56.880011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:22:56.880011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"236'38","reported_seq":275,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.085407+0000","last_change":"2024-02-05T08:23:08.944358+0000","last_active":"2024-02-05T08:24:33.085407+0000","last_peered":"2024-02-05T08:24:33.085407+0000","last_clean":"2024-02-05T08:24:33.085407+0000","last_became_active":"2024-02-05T08:23:08.927776+0000","last_became_peered":"2024-02-05T08:23:08.927776+0000","last_unstale":"2024-02-05T08:24:33.085407+0000","last_undegraded":"2024-02-05T08:24:33.085407+0000","last_fullsized":"2024-02-05T08:24:33.085407+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'38","last_scrub_stamp":"2024-02-05T08:22:48.999388+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:22:48.999388+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":733,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.066106+0000","last_change":"2024-02-05T08:22:07.078658+0000","last_active":"2024-02-05T08:24:31.066106+0000","last_peered":"2024-02-05T08:24:31.066106+0000","last_clean":"2024-02-05T08:24:31.066106+0000","last_became_active":"2024-02-05T08:18:34.636667+0000","last_became_peered":"2024-02-05T08:18:34.636667+0000","last_unstale":"2024-02-05T08:24:31.066106+0000","last_undegraded":"2024-02-05T08:24:31.066106+0000","last_fullsized":"2024-02-05T08:24:31.066106+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:07.078318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:31.133206+0000","last_clean_scrub_stamp":"2024-02-05T08:22:07.078318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"389'7000","reported_seq":7054,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:30.980510+0000","last_change":"2024-02-05T08:23:08.918166+0000","last_active":"2024-02-05T08:24:30.980510+0000","last_peered":"2024-02-05T08:24:30.980510+0000","last_clean":"2024-02-05T08:24:30.980510+0000","last_became_active":"2024-02-05T08:23:08.916901+0000","last_became_peered":"2024-02-05T08:23:08.916901+0000","last_unstale":"2024-02-05T08:24:30.980510+0000","last_undegraded":"2024-02-05T08:24:30.980510+0000","last_fullsized":"2024-02-05T08:24:30.980510+0000","mapping_epoch":367,"log_start":"381'3100","ondisk_log_start":"381'3100","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752000,"num_objects":7000,"num_object_clones":0,"num_object_copies":21000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7000,"num_write_kb":448000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'130","reported_seq":513,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093775+0000","last_change":"2024-02-05T08:24:22.852362+0000","last_active":"2024-02-05T08:24:33.093775+0000","last_peered":"2024-02-05T08:24:33.093775+0000","last_clean":"2024-02-05T08:24:33.093775+0000","last_became_active":"2024-02-05T08:19:01.817169+0000","last_became_peered":"2024-02-05T08:19:01.817169+0000","last_unstale":"2024-02-05T08:24:33.093775+0000","last_undegraded":"2024-02-05T08:24:33.093775+0000","last_fullsized":"2024-02-05T08:24:33.093775+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:24:22.852081+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:24:22.852081+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:38.827043+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":731,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093238+0000","last_change":"2024-02-05T08:23:08.914041+0000","last_active":"2024-02-05T08:24:33.093238+0000","last_peered":"2024-02-05T08:24:33.093238+0000","last_clean":"2024-02-05T08:24:33.093238+0000","last_became_active":"2024-02-05T08:23:08.913735+0000","last_became_peered":"2024-02-05T08:23:08.913735+0000","last_unstale":"2024-02-05T08:24:33.093238+0000","last_undegraded":"2024-02-05T08:24:33.093238+0000","last_fullsized":"2024-02-05T08:24:33.093238+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:47.385093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:47.385093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"355'9","reported_seq":977,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065869+0000","last_change":"2024-02-05T08:22:09.114866+0000","last_active":"2024-02-05T08:24:31.065869+0000","last_peered":"2024-02-05T08:24:31.065869+0000","last_clean":"2024-02-05T08:24:31.065869+0000","last_became_active":"2024-02-05T08:22:09.106117+0000","last_became_peered":"2024-02-05T08:22:09.106117+0000","last_unstale":"2024-02-05T08:24:31.065869+0000","last_undegraded":"2024-02-05T08:24:31.065869+0000","last_fullsized":"2024-02-05T08:24:31.065869+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:22:06.070321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:06.070321+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":230,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.085641+0000","last_change":"2024-02-05T08:23:29.957871+0000","last_active":"2024-02-05T08:24:33.085641+0000","last_peered":"2024-02-05T08:24:33.085641+0000","last_clean":"2024-02-05T08:24:33.085641+0000","last_became_active":"2024-02-05T08:23:07.937468+0000","last_became_peered":"2024-02-05T08:23:07.937468+0000","last_unstale":"2024-02-05T08:24:33.085641+0000","last_undegraded":"2024-02-05T08:24:33.085641+0000","last_fullsized":"2024-02-05T08:24:33.085641+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:29.957655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.957655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"236'27","reported_seq":682,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:13.933209+0000","last_change":"2024-02-05T08:23:30.166109+0000","last_active":"2024-02-05T08:24:13.933209+0000","last_peered":"2024-02-05T08:24:13.933209+0000","last_clean":"2024-02-05T08:24:13.933209+0000","last_became_active":"2024-02-05T08:23:07.939747+0000","last_became_peered":"2024-02-05T08:23:07.939747+0000","last_unstale":"2024-02-05T08:24:13.933209+0000","last_undegraded":"2024-02-05T08:24:13.933209+0000","last_fullsized":"2024-02-05T08:24:13.933209+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:23:30.165977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:30.165977+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:34.736139+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"391'7163","reported_seq":7298,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.241131+0000","last_change":"2024-02-05T08:24:31.613860+0000","last_active":"2024-02-05T08:24:33.241131+0000","last_peered":"2024-02-05T08:24:33.241131+0000","last_clean":"2024-02-05T08:24:33.241131+0000","last_became_active":"2024-02-05T08:23:08.913844+0000","last_became_peered":"2024-02-05T08:23:08.913844+0000","last_unstale":"2024-02-05T08:24:33.241131+0000","last_undegraded":"2024-02-05T08:24:33.241131+0000","last_fullsized":"2024-02-05T08:24:33.241131+0000","mapping_epoch":367,"log_start":"379'2100","ondisk_log_start":"379'2100","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'7028","last_scrub_stamp":"2024-02-05T08:24:31.613639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:24:31.613639+0000","objects_scrubbed":6944,"log_size":5063,"log_dups_size":0,"ondisk_log_size":5063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:40.017217+0000","scrub_duration":2182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":469434368,"num_objects":7163,"num_object_clones":0,"num_object_copies":21489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7163,"num_write_kb":458432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":229,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.084007+0000","last_change":"2024-02-05T08:22:51.956578+0000","last_active":"2024-02-05T08:24:33.084007+0000","last_peered":"2024-02-05T08:24:33.084007+0000","last_clean":"2024-02-05T08:24:33.084007+0000","last_became_active":"2024-02-05T08:18:34.627261+0000","last_became_peered":"2024-02-05T08:18:34.627261+0000","last_unstale":"2024-02-05T08:24:33.084007+0000","last_undegraded":"2024-02-05T08:24:33.084007+0000","last_fullsized":"2024-02-05T08:24:33.084007+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:51.956504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:22:51.956504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":750,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.093503+0000","last_change":"2024-02-05T08:24:30.868452+0000","last_active":"2024-02-05T08:24:33.093503+0000","last_peered":"2024-02-05T08:24:33.093503+0000","last_clean":"2024-02-05T08:24:33.093503+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:24:33.093503+0000","last_undegraded":"2024-02-05T08:24:33.093503+0000","last_fullsized":"2024-02-05T08:24:33.093503+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:30.867869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:24:30.867869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:49.702075+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":729,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.066109+0000","last_change":"2024-02-05T08:21:07.952159+0000","last_active":"2024-02-05T08:24:31.066109+0000","last_peered":"2024-02-05T08:24:31.066109+0000","last_clean":"2024-02-05T08:24:31.066109+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:24:31.066109+0000","last_undegraded":"2024-02-05T08:24:31.066109+0000","last_fullsized":"2024-02-05T08:24:31.066109+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:07.952031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:31.201062+0000","last_clean_scrub_stamp":"2024-02-05T08:21:07.952031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"236'19","reported_seq":792,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065599+0000","last_change":"2024-02-05T08:23:39.449120+0000","last_active":"2024-02-05T08:24:31.065599+0000","last_peered":"2024-02-05T08:24:31.065599+0000","last_clean":"2024-02-05T08:24:31.065599+0000","last_became_active":"2024-02-05T08:23:07.937017+0000","last_became_peered":"2024-02-05T08:23:07.937017+0000","last_unstale":"2024-02-05T08:24:31.065599+0000","last_undegraded":"2024-02-05T08:24:31.065599+0000","last_fullsized":"2024-02-05T08:24:31.065599+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:23:39.449015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:39.449015+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:50.639202+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"391'7326","reported_seq":7453,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.376096+0000","last_change":"2024-02-05T08:24:17.908639+0000","last_active":"2024-02-05T08:24:33.376096+0000","last_peered":"2024-02-05T08:24:33.376096+0000","last_clean":"2024-02-05T08:24:33.376096+0000","last_became_active":"2024-02-05T08:23:08.910712+0000","last_became_peered":"2024-02-05T08:23:08.910712+0000","last_unstale":"2024-02-05T08:24:33.376096+0000","last_undegraded":"2024-02-05T08:24:33.376096+0000","last_fullsized":"2024-02-05T08:24:33.376096+0000","mapping_epoch":367,"log_start":"381'2400","ondisk_log_start":"381'2400","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'6207","last_scrub_stamp":"2024-02-05T08:24:17.908422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:24:17.908422+0000","objects_scrubbed":6107,"log_size":4926,"log_dups_size":0,"ondisk_log_size":4926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:30.909954+0000","scrub_duration":2250,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480116736,"num_objects":7326,"num_object_clones":0,"num_object_copies":21978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7326,"num_write_kb":468864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"82'130","reported_seq":505,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.091732+0000","last_change":"2024-02-05T08:22:57.194319+0000","last_active":"2024-02-05T08:24:33.091732+0000","last_peered":"2024-02-05T08:24:33.091732+0000","last_clean":"2024-02-05T08:24:33.091732+0000","last_became_active":"2024-02-05T08:19:01.817126+0000","last_became_peered":"2024-02-05T08:19:01.817126+0000","last_unstale":"2024-02-05T08:24:33.091732+0000","last_undegraded":"2024-02-05T08:24:33.091732+0000","last_fullsized":"2024-02-05T08:24:33.091732+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:22:57.194237+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:22:57.194237+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":763,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.082939+0000","last_change":"2024-02-05T08:23:42.090797+0000","last_active":"2024-02-05T08:24:33.082939+0000","last_peered":"2024-02-05T08:24:33.082939+0000","last_clean":"2024-02-05T08:24:33.082939+0000","last_became_active":"2024-02-05T08:18:34.645067+0000","last_became_peered":"2024-02-05T08:18:34.645067+0000","last_unstale":"2024-02-05T08:24:33.082939+0000","last_undegraded":"2024-02-05T08:24:33.082939+0000","last_fullsized":"2024-02-05T08:24:33.082939+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:23:42.090680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:42.090680+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:43.764626+0000","scrub_duration":274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":572,"reported_epoch":389,"state":"active+clean","last_fresh":"2024-02-05T08:24:13.932717+0000","last_change":"2024-02-05T08:23:08.915640+0000","last_active":"2024-02-05T08:24:13.932717+0000","last_peered":"2024-02-05T08:24:13.932717+0000","last_clean":"2024-02-05T08:24:13.932717+0000","last_became_active":"2024-02-05T08:23:08.914502+0000","last_became_peered":"2024-02-05T08:23:08.914502+0000","last_unstale":"2024-02-05T08:24:13.932717+0000","last_undegraded":"2024-02-05T08:24:13.932717+0000","last_fullsized":"2024-02-05T08:24:13.932717+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:49.228660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:49.228660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:18.915051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":469,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.089464+0000","last_change":"2024-02-05T08:23:28.612625+0000","last_active":"2024-02-05T08:24:33.089464+0000","last_peered":"2024-02-05T08:24:33.089464+0000","last_clean":"2024-02-05T08:24:33.089464+0000","last_became_active":"2024-02-05T08:23:07.929749+0000","last_became_peered":"2024-02-05T08:23:07.929749+0000","last_unstale":"2024-02-05T08:24:33.089464+0000","last_undegraded":"2024-02-05T08:24:33.089464+0000","last_fullsized":"2024-02-05T08:24:33.089464+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:28.612512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:23:28.612512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:24:40.525324+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"236'31","reported_seq":774,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.067041+0000","last_change":"2024-02-05T08:24:26.158100+0000","last_active":"2024-02-05T08:24:31.067041+0000","last_peered":"2024-02-05T08:24:31.067041+0000","last_clean":"2024-02-05T08:24:31.067041+0000","last_became_active":"2024-02-05T08:23:07.930130+0000","last_became_peered":"2024-02-05T08:23:07.930130+0000","last_unstale":"2024-02-05T08:24:31.067041+0000","last_undegraded":"2024-02-05T08:24:31.067041+0000","last_fullsized":"2024-02-05T08:24:31.067041+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'31","last_scrub_stamp":"2024-02-05T08:24:26.157978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:24:26.157978+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:34.179674+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"391'7295","reported_seq":7352,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.267968+0000","last_change":"2024-02-05T08:23:08.920424+0000","last_active":"2024-02-05T08:24:33.267968+0000","last_peered":"2024-02-05T08:24:33.267968+0000","last_clean":"2024-02-05T08:24:33.267968+0000","last_became_active":"2024-02-05T08:23:08.920062+0000","last_became_peered":"2024-02-05T08:23:08.920062+0000","last_unstale":"2024-02-05T08:24:33.267968+0000","last_undegraded":"2024-02-05T08:24:33.267968+0000","last_fullsized":"2024-02-05T08:24:33.267968+0000","mapping_epoch":367,"log_start":"380'2200","ondisk_log_start":"380'2200","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":478085120,"num_objects":7295,"num_object_clones":0,"num_object_copies":21885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7295,"num_write_kb":466880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"82'130","reported_seq":1037,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.088355+0000","last_change":"2024-02-05T08:24:22.941847+0000","last_active":"2024-02-05T08:24:33.088355+0000","last_peered":"2024-02-05T08:24:33.088355+0000","last_clean":"2024-02-05T08:24:33.088355+0000","last_became_active":"2024-02-05T08:19:01.816744+0000","last_became_peered":"2024-02-05T08:19:01.816744+0000","last_unstale":"2024-02-05T08:24:33.088355+0000","last_undegraded":"2024-02-05T08:24:33.088355+0000","last_fullsized":"2024-02-05T08:24:33.088355+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:24:22.941728+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:24:22.941728+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:35.303846+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":717,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065922+0000","last_change":"2024-02-05T08:24:19.346422+0000","last_active":"2024-02-05T08:24:31.065922+0000","last_peered":"2024-02-05T08:24:31.065922+0000","last_clean":"2024-02-05T08:24:31.065922+0000","last_became_active":"2024-02-05T08:23:07.937641+0000","last_became_peered":"2024-02-05T08:23:07.937641+0000","last_unstale":"2024-02-05T08:24:31.065922+0000","last_undegraded":"2024-02-05T08:24:31.065922+0000","last_fullsized":"2024-02-05T08:24:31.065922+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:19.346083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:24:19.346083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:42.751303+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"129'9","reported_seq":994,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.082291+0000","last_change":"2024-02-05T08:23:36.866535+0000","last_active":"2024-02-05T08:24:33.082291+0000","last_peered":"2024-02-05T08:24:33.082291+0000","last_clean":"2024-02-05T08:24:33.082291+0000","last_became_active":"2024-02-05T08:04:45.566558+0000","last_became_peered":"2024-02-05T08:04:45.566558+0000","last_unstale":"2024-02-05T08:24:33.082291+0000","last_undegraded":"2024-02-05T08:24:33.082291+0000","last_fullsized":"2024-02-05T08:24:33.082291+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:23:36.866292+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:23:36.866292+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:04.936923+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":239,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-02-05T08:24:33.083861+0000","last_change":"2024-02-05T08:23:08.929039+0000","last_active":"2024-02-05T08:24:33.083861+0000","last_peered":"2024-02-05T08:24:33.083861+0000","last_clean":"2024-02-05T08:24:33.083861+0000","last_became_active":"2024-02-05T08:23:08.928775+0000","last_became_peered":"2024-02-05T08:23:08.928775+0000","last_unstale":"2024-02-05T08:24:33.083861+0000","last_undegraded":"2024-02-05T08:24:33.083861+0000","last_fullsized":"2024-02-05T08:24:33.083861+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:07.643746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:22:07.643746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"363'25","reported_seq":839,"reported_epoch":390,"state":"active+clean","last_fresh":"2024-02-05T08:24:31.065716+0000","last_change":"2024-02-05T08:23:03.301185+0000","last_active":"2024-02-05T08:24:31.065716+0000","last_peered":"2024-02-05T08:24:31.065716+0000","last_clean":"2024-02-05T08:24:31.065716+0000","last_became_active":"2024-02-05T08:18:34.637413+0000","last_became_peered":"2024-02-05T08:18:34.637413+0000","last_unstale":"2024-02-05T08:24:31.065716+0000","last_undegraded":"2024-02-05T08:24:31.065716+0000","last_fullsized":"2024-02-05T08:24:31.065716+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'25","last_scrub_stamp":"2024-02-05T08:23:03.301090+0000","last_deep_scrub":"363'25","last_deep_scrub_stamp":"2024-02-05T08:23:03.301090+0000","last_clean_scrub_stamp":"2024-02-05T08:23:03.301090+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":7513767936,"num_objects":114651,"num_object_clones":0,"num_object_copies":343953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114651,"num_write_kb":7337664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22500278272,"data_stored":22500278272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71320,"ondisk_log_size":71320,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1174,"num_read_kb":1169,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":6562,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038003,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3932132,"kb_used_data":3219292,"kb_used_omap":1,"kb_used_meta":712830,"kb_avail":89813020,"statfs":{"total":95995035648,"available":91968532480,"internally_reserved":0,"allocated":3296555008,"data_stored":3296073715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":729938378},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":1.1559999999999999,"15min":1.637},"min":{"1min":0.26100000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.6899999999999999,"5min":20.344999999999999,"15min":60.127000000000002},"last":2.1469999999999998},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.2030000000000001,"15min":1.736},"min":{"1min":0.28999999999999998,"5min":0.24099999999999999,"15min":0.23200000000000001},"max":{"1min":4.633,"5min":21.347000000000001,"15min":58.283000000000001},"last":0.46300000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:24:07 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":4.9560000000000004,"5min":4.9560000000000004,"15min":4.9560000000000004},"last":1.639},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.5960000000000001,"5min":4.5960000000000001,"15min":4.5960000000000001},"last":1.6739999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.1359999999999999,"15min":1.129},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.6920000000000002,"5min":20.288,"15min":20.288},"last":2.7360000000000002},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.2490000000000001,"15min":1.222},"min":{"1min":0.28999999999999998,"5min":0.248,"15min":0.248},"max":{"1min":4.8849999999999998,"5min":21.247,"15min":21.247},"last":0.371}]},{"osd":3,"last update":"Mon Feb 5 08:24:02 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":0.93600000000000005,"15min":1.3320000000000001},"min":{"1min":0.41999999999999998,"5min":0.28499999999999998,"15min":0.249},"max":{"1min":4.9809999999999999,"5min":4.9809999999999999,"15min":58.515000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.92400000000000004,"15min":1.363},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.7770000000000001,"5min":3.8239999999999998,"15min":58.225999999999999},"last":1.1399999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:24:02 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.0329999999999999,"15min":1.1220000000000001},"min":{"1min":0.28199999999999997,"5min":0.24399999999999999,"15min":0.23300000000000001},"max":{"1min":4.5369999999999999,"5min":4.5369999999999999,"15min":49.601999999999997},"last":2.1709999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.2110000000000001,"15min":1.1930000000000001},"min":{"1min":0.315,"5min":0.249,"15min":0.249},"max":{"1min":4.9290000000000003,"5min":7.835,"15min":49.795999999999999},"last":0.32200000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.91400000000000003,"15min":1.161},"min":{"1min":0.30599999999999999,"5min":0.26800000000000002,"15min":0.24299999999999999},"max":{"1min":3.802,"5min":3.9929999999999999,"15min":49.914000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.99099999999999999,"15min":1.1970000000000001},"min":{"1min":0.32400000000000001,"5min":0.28999999999999998,"15min":0.27000000000000002},"max":{"1min":3.734,"5min":8.7240000000000002,"15min":52.256},"last":2.105}]},{"osd":6,"last update":"Mon Feb 5 08:24:02 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":2.5859999999999999,"15min":1.488},"min":{"1min":0.316,"5min":0.316,"15min":0.28599999999999998},"max":{"1min":4.4610000000000003,"5min":84.254999999999995,"15min":84.254999999999995},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.208,"5min":2.7530000000000001,"15min":1.5229999999999999},"min":{"1min":0.40799999999999997,"5min":0.36499999999999999,"15min":0.27300000000000002},"max":{"1min":5.0570000000000004,"5min":84.667000000000002,"15min":84.667000000000002},"last":0.34999999999999998}]}]},{"osd":6,"up_from":51,"seq":219043332345,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4293884,"kb_used_data":3658804,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":89451268,"statfs":{"total":95995035648,"available":91598098432,"internally_reserved":0,"allocated":3746615296,"data_stored":3746172662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":650312134},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":1.3180000000000001,"15min":1.8819999999999999},"min":{"1min":0.34300000000000003,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.4089999999999998,"5min":14.836,"15min":102.819},"last":0.871},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.90800000000000003,"15min":1.7809999999999999},"min":{"1min":0.312,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":1.375,"5min":4.9480000000000004,"15min":102.75},"last":0.42799999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:24:09 2024","interfaces":[{"interface":"back","average":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":25.898,"5min":25.898,"15min":25.898},"last":0.40300000000000002},{"interface":"front","average":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":25.884,"5min":25.884,"15min":25.884},"last":0.80100000000000005}]},{"osd":2,"last update":"Mon Feb 5 08:23:42 2024","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.377,"15min":1.2350000000000001},"min":{"1min":0.27000000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":25.751000000000001,"5min":25.751000000000001,"15min":25.751000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.78,"5min":1.222,"15min":1.1379999999999999},"min":{"1min":0.34699999999999998,"5min":0.28199999999999997,"15min":0.23400000000000001},"max":{"1min":26.210999999999999,"5min":26.210999999999999,"15min":26.210999999999999},"last":1.831}]},{"osd":3,"last update":"Mon Feb 5 08:23:42 2024","interfaces":[{"interface":"back","average":{"1min":1.8839999999999999,"5min":1.3839999999999999,"15min":1.514},"min":{"1min":0.32300000000000001,"5min":0.28199999999999997,"15min":0.27100000000000002},"max":{"1min":27.113,"5min":27.113,"15min":102.438},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.8500000000000001,"5min":1.379,"15min":1.472},"min":{"1min":0.32300000000000001,"5min":0.29299999999999998,"15min":0.29099999999999998},"max":{"1min":27.209,"5min":27.209,"15min":102.288},"last":0.97999999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":1.841,"15min":1.137},"min":{"1min":0.317,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.8089999999999999,"5min":46.125,"15min":46.125},"last":0.374},{"interface":"front","average":{"1min":0.76600000000000001,"5min":2.1960000000000002,"15min":1.2709999999999999},"min":{"1min":0.27900000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.3639999999999999,"5min":48.817999999999998,"15min":48.817999999999998},"last":0.82699999999999996}]},{"osd":5,"last update":"Mon Feb 5 08:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":1.1160000000000001,"15min":1.216},"min":{"1min":0.26700000000000002,"5min":0.26000000000000001,"15min":0.24199999999999999},"max":{"1min":1.4490000000000001,"5min":7.6760000000000002,"15min":43.643000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.264,"15min":1.2509999999999999},"min":{"1min":0.33700000000000002,"5min":0.246,"15min":0.246},"max":{"1min":4.5999999999999996,"5min":9.2279999999999998,"15min":43.893000000000001},"last":0.754}]},{"osd":7,"last update":"Mon Feb 5 08:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.999,"15min":1.079},"min":{"1min":0.38300000000000001,"5min":0.307,"15min":0.307},"max":{"1min":1.643,"5min":4.7530000000000001,"15min":24.728999999999999},"last":1.069},{"interface":"front","average":{"1min":1.151,"5min":1.498,"15min":1.266},"min":{"1min":0.437,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.6420000000000003,"5min":14.988,"15min":21.91},"last":1.4410000000000001}]}]},{"osd":1,"up_from":366,"seq":1571958030354,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4313208,"kb_used_data":4154480,"kb_used_omap":4,"kb_used_meta":158715,"kb_avail":89431944,"statfs":{"total":95995035648,"available":91578310656,"internally_reserved":0,"allocated":4254187520,"data_stored":4253677335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":162524838},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:24:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":3.0550000000000002,"5min":3.0550000000000002,"15min":3.0550000000000002},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":7.6029999999999998,"5min":7.6029999999999998,"15min":7.6029999999999998},"last":1.4950000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:24:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.5739999999999998,"5min":5.5739999999999998,"15min":5.5739999999999998},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":4.2640000000000002,"5min":4.2640000000000002,"15min":4.2640000000000002},"last":1.0569999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:24:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":3.8149999999999999,"5min":3.8149999999999999,"15min":3.8149999999999999},"last":1.585},{"interface":"front","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.9300000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":1.6899999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:24:09 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.4590000000000001,"5min":3.4590000000000001,"15min":3.4590000000000001},"last":1.958},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":3.782,"5min":3.782,"15min":3.782},"last":2.516}]},{"osd":5,"last update":"Mon Feb 5 08:24:09 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.099,"15min":1.099},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":2.4590000000000001,"5min":2.4590000000000001,"15min":2.4590000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":3.593,"5min":3.593,"15min":3.593},"last":1.917}]},{"osd":6,"last update":"Mon Feb 5 08:24:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.6179999999999999,"5min":2.6179999999999999,"15min":2.6179999999999999},"last":1.27},{"interface":"front","average":{"1min":1.224,"5min":1.224,"15min":1.224},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":3.5339999999999998,"5min":3.5339999999999998,"15min":3.5339999999999998},"last":2.5459999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:24:09 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.196,"15min":1.196},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.351,"5min":3.351,"15min":3.351},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.387,"5min":1.387,"15min":1.387},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":6.9189999999999996,"5min":6.9189999999999996,"15min":6.9189999999999996},"last":0.998}]}]},{"osd":0,"up_from":0,"seq":34359738647,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411597,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3912032,"kb_used_data":3195032,"kb_used_omap":1,"kb_used_meta":716990,"kb_avail":89833120,"statfs":{"total":95995035648,"available":91989114880,"internally_reserved":0,"allocated":3271712768,"data_stored":3271234757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":734198214},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:24:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.69699999999999995,"15min":1.946},"min":{"1min":0.248,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.47,"5min":6.181,"15min":107.155},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.93600000000000005,"15min":2.1259999999999999},"min":{"1min":0.25800000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.2000000000000002,"5min":7.3620000000000001,"15min":107.221},"last":0.88700000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:24:09 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":2.327,"5min":2.327,"15min":2.327},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.8120000000000003,"5min":4.8120000000000003,"15min":4.8120000000000003},"last":3.1669999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:24:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.018,"15min":1.4850000000000001},"min":{"1min":0.26400000000000001,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":3.5750000000000002,"5min":6.5039999999999996,"15min":68.108000000000004},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.2889999999999999,"5min":0.88700000000000001,"15min":1.351},"min":{"1min":0.317,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.0459999999999998,"5min":6.101,"15min":67.945999999999998},"last":2.968}]},{"osd":4,"last update":"Mon Feb 5 08:23:54 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":1.2090000000000001,"15min":1.4339999999999999},"min":{"1min":0.27600000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.3460000000000001,"5min":7.6189999999999998,"15min":31.596},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.81999999999999995,"5min":1.1990000000000001,"15min":1.3520000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.252},"max":{"1min":2.774,"5min":7.6509999999999998,"15min":31.57},"last":0.81599999999999995}]},{"osd":5,"last update":"Mon Feb 5 08:24:14 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":1.073,"15min":1.5720000000000001},"min":{"1min":0.34799999999999998,"5min":0.27300000000000002,"15min":0.246},"max":{"1min":1.9450000000000001,"5min":9.516,"15min":58.799999999999997},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.248,"5min":1.3140000000000001,"15min":1.6020000000000001},"min":{"1min":0.33000000000000002,"5min":0.27300000000000002,"15min":0.26900000000000002},"max":{"1min":3.8029999999999999,"5min":9.3209999999999997,"15min":58.615000000000002},"last":3.2639999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.8660000000000001,"5min":1.46,"15min":1.339},"min":{"1min":0.32700000000000001,"5min":0.307,"15min":0.29199999999999998},"max":{"1min":21.885000000000002,"5min":21.885000000000002,"15min":21.885000000000002},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.8440000000000001,"5min":1.3420000000000001,"15min":1.2609999999999999},"min":{"1min":0.40600000000000003,"5min":0.317,"15min":0.317},"max":{"1min":22.158000000000001,"5min":22.158000000000001,"15min":22.158000000000001},"last":0.87}]},{"osd":7,"last update":"Mon Feb 5 08:24:04 2024","interfaces":[{"interface":"back","average":{"1min":5.4790000000000001,"5min":1.9350000000000001,"15min":1.5009999999999999},"min":{"1min":0.376,"5min":0.33900000000000002,"15min":0.28899999999999998},"max":{"1min":72.334999999999994,"5min":72.334999999999994,"15min":72.334999999999994},"last":3.2949999999999999},{"interface":"front","average":{"1min":5.2770000000000001,"5min":2.0659999999999998,"15min":1.7170000000000001},"min":{"1min":0.48899999999999999,"5min":0.35699999999999998,"15min":0.32700000000000001},"max":{"1min":72.222999999999999,"5min":72.222999999999999,"15min":72.222999999999999},"last":1.1379999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117256,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3855996,"kb_used_data":3220912,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":89889156,"statfs":{"total":95995035648,"available":92046495744,"internally_reserved":0,"allocated":3298213888,"data_stored":3297713033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":650312137},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.90100000000000002,"15min":1.6339999999999999},"min":{"1min":0.216,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":3.3319999999999999,"5min":11.747999999999999,"15min":52.831000000000003},"last":1.496},{"interface":"front","average":{"1min":0.376,"5min":0.86599999999999999,"15min":1.6899999999999999},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.78100000000000003,"5min":11.618,"15min":53.859999999999999},"last":1.365}]},{"osd":1,"last update":"Mon Feb 5 08:24:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":5.7149999999999999,"5min":5.7149999999999999,"15min":5.7149999999999999},"last":1.53},{"interface":"front","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":5.2699999999999996,"5min":5.2699999999999996,"15min":5.2699999999999996},"last":1.4059999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":1.3520000000000001,"15min":1.5129999999999999},"min":{"1min":0.24099999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.9529999999999998,"5min":35.277000000000001,"15min":35.277000000000001},"last":5.4340000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":1.2749999999999999,"15min":1.3300000000000001},"min":{"1min":0.27200000000000002,"5min":0.23300000000000001,"15min":0.23200000000000001},"max":{"1min":3.778,"5min":35.442999999999998,"15min":35.442999999999998},"last":1.879}]},{"osd":4,"last update":"Mon Feb 5 08:23:47 2024","interfaces":[{"interface":"back","average":{"1min":3.927,"5min":1.5580000000000001,"15min":1.2629999999999999},"min":{"1min":0.433,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":54.502000000000002,"5min":54.502000000000002,"15min":54.502000000000002},"last":1.925},{"interface":"front","average":{"1min":3.6949999999999998,"5min":1.4330000000000001,"15min":1.244},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":54.540999999999997,"5min":54.540999999999997,"15min":54.540999999999997},"last":1.4339999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:24:13 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.113,"15min":1.778},"min":{"1min":0.44500000000000001,"5min":0.28899999999999998,"15min":0.28000000000000003},"max":{"1min":5.5060000000000002,"5min":6.343,"15min":117.432},"last":1.6240000000000001},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.85199999999999998,"15min":1.7230000000000001},"min":{"1min":0.40799999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.3180000000000001,"5min":3.3180000000000001,"15min":114.828},"last":1.462}]},{"osd":6,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":1.5309999999999999,"15min":1.3420000000000001},"min":{"1min":0.36299999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.585,"5min":54.219999999999999,"15min":54.219999999999999},"last":4.5519999999999996},{"interface":"front","average":{"1min":0.81200000000000006,"5min":1.7010000000000001,"15min":1.3959999999999999},"min":{"1min":0.41999999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.9889999999999999,"5min":54.530999999999999,"15min":54.530999999999999},"last":2.1669999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.05,"15min":1.7170000000000001},"min":{"1min":0.441,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":3.6299999999999999,"5min":6.3730000000000002,"15min":114.84999999999999},"last":1.5900000000000001},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.94799999999999995,"15min":1.6919999999999999},"min":{"1min":0.376,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.3580000000000001,"5min":4.4290000000000003,"15min":114.871},"last":4.6120000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953731,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3405400,"kb_used_data":2710608,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":90339752,"statfs":{"total":95995035648,"available":92507906048,"internally_reserved":0,"allocated":2775662592,"data_stored":2775191858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":1.1140000000000001,"15min":1.4590000000000001},"min":{"1min":0.23300000000000001,"5min":0.192,"15min":0.192},"max":{"1min":2.093,"5min":8.6470000000000002,"15min":39.552},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":1.1950000000000001,"15min":1.5149999999999999},"min":{"1min":0.30499999999999999,"5min":0.25,"15min":0.21099999999999999},"max":{"1min":1.304,"5min":5.165,"15min":39.350999999999999},"last":0.68300000000000005}]},{"osd":1,"last update":"Mon Feb 5 08:24:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":3.915,"5min":3.915,"15min":3.915},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":2.899,"5min":2.899,"15min":2.899},"last":2.2709999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.1639999999999999,"15min":1.069},"min":{"1min":0.33100000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.1559999999999997,"5min":5.9160000000000004,"15min":5.9640000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.107,"15min":1.1359999999999999},"min":{"1min":0.34100000000000003,"5min":0.33700000000000002,"15min":0.26300000000000001},"max":{"1min":3.6589999999999998,"5min":4.4290000000000003,"15min":7.032},"last":2.4550000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":1.3120000000000001,"15min":1.748},"min":{"1min":0.315,"5min":0.312,"15min":0.24399999999999999},"max":{"1min":1.2509999999999999,"5min":8.9269999999999996,"15min":57.204000000000001},"last":2.4249999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":1.1830000000000001,"15min":1.675},"min":{"1min":0.32700000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.4209999999999998,"5min":4.2599999999999998,"15min":57.606999999999999},"last":0.34799999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:24:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.3280000000000001,"15min":1.171},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.224},"max":{"1min":3.4039999999999999,"5min":6.1909999999999998,"15min":34.167000000000002},"last":0.307},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.2509999999999999,"15min":1.2010000000000001},"min":{"1min":0.375,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.1099999999999999,"5min":7.851,"15min":36.051000000000002},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":1.4019999999999999,"15min":1.032},"min":{"1min":0.32600000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":1.3360000000000001,"5min":6.375,"15min":6.375},"last":2.3700000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":1.2749999999999999,"15min":0.94999999999999996},"min":{"1min":0.35699999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.4630000000000001,"5min":6.9690000000000003,"15min":6.9690000000000003},"last":2.0579999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:23:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.331,"15min":1.0529999999999999},"min":{"1min":0.38900000000000001,"5min":0.34300000000000003,"15min":0.27000000000000002},"max":{"1min":6.1280000000000001,"5min":12.683999999999999,"15min":12.683999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.83999999999999997,"5min":1.333,"15min":1.042},"min":{"1min":0.39000000000000001,"5min":0.34100000000000003,"15min":0.27900000000000003},"max":{"1min":3.3199999999999998,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.40200000000000002}]}]},{"osd":5,"up_from":42,"seq":180388626685,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2543128,"kb_used_data":1844172,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":91202024,"statfs":{"total":95995035648,"available":93390872576,"internally_reserved":0,"allocated":1888432128,"data_stored":1887967279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.999,"15min":1.605},"min":{"1min":0.28599999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.5220000000000002,"5min":6.4660000000000002,"15min":46.962000000000003},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.91400000000000003,"5min":1.083,"15min":1.5569999999999999},"min":{"1min":0.32400000000000001,"5min":0.26600000000000001,"15min":0.218},"max":{"1min":4.4340000000000002,"5min":5.0819999999999999,"15min":47.067},"last":0.40100000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:24:07 2024","interfaces":[{"interface":"back","average":{"1min":2.6600000000000001,"5min":2.6600000000000001,"15min":2.6600000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":24.606999999999999,"5min":24.606999999999999,"15min":24.606999999999999},"last":0.318},{"interface":"front","average":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":24.861000000000001,"5min":24.861000000000001,"15min":24.861000000000001},"last":0.45300000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.169,"15min":1.2030000000000001},"min":{"1min":0.39700000000000002,"5min":0.245,"15min":0.245},"max":{"1min":2.3839999999999999,"5min":17.09,"15min":17.09},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.077,"5min":1.2589999999999999,"15min":1.306},"min":{"1min":0.316,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.6859999999999999,"5min":17.515999999999998,"15min":17.515999999999998},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Feb 5 08:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.0720000000000001,"15min":1.387},"min":{"1min":0.34999999999999998,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.427,"5min":7.6950000000000003,"15min":46.569000000000003},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":1,"15min":1.5},"min":{"1min":0.34899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.4139999999999999,"5min":6.5519999999999996,"15min":47.820999999999998},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":1.056,"15min":0.995},"min":{"1min":0.28000000000000003,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.4870000000000001,"5min":7.7389999999999999,"15min":7.7389999999999999},"last":0.435},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.169,"15min":1.032},"min":{"1min":0.36899999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":6.0979999999999999,"5min":7.609,"15min":7.609},"last":0.54300000000000004}]},{"osd":6,"last update":"Mon Feb 5 08:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":1.464,"15min":1.131},"min":{"1min":0.33200000000000002,"5min":0.32300000000000001,"15min":0.26600000000000001},"max":{"1min":2.2269999999999999,"5min":30.581,"15min":30.581},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":1.397,"15min":1.1279999999999999},"min":{"1min":0.439,"5min":0.32900000000000001,"15min":0.316},"max":{"1min":2.2010000000000001,"5min":30.620999999999999,"15min":30.620999999999999},"last":0.75}]},{"osd":7,"last update":"Mon Feb 5 08:24:01 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":1.47,"15min":1.2949999999999999},"min":{"1min":0.37,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.0459999999999998,"5min":34.695,"15min":34.695},"last":0.64400000000000002},{"interface":"front","average":{"1min":1.119,"5min":1.423,"15min":1.232},"min":{"1min":0.33300000000000002,"5min":0.308,"15min":0.308},"max":{"1min":4.4619999999999997,"5min":34.514000000000003,"15min":34.514000000000003},"last":0.875}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4119134208,"data_stored":4119134208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3290300416,"data_stored":3290300416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3306946560,"data_stored":3306946560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2796945408,"data_stored":2796945408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1898971136,"data_stored":1898971136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3778084864,"data_stored":3778084864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3309895680,"data_stored":3309895680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:24:37.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-02-05T08:24:37.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:37.697 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:37.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:37.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:37.744 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 78 16 120444 120428 96.4757 80.25 0.0140813 0.0103607 2024-02-05T08:24:37.799 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:37.851 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:37.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:37.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:37.953 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:37.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:37.997 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:37.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:38.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:24:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:24:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:24:38.099 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:38.135 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:38.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:38.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:38.237 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:38.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:38.279 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:38.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:38.363 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:38 smithi134 ceph-mon[30086]: 5.2 scrub ok 2024-02-05T08:24:38.364 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:38 smithi134 ceph-mon[30086]: 6.17 scrub starts 2024-02-05T08:24:38.364 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:38 smithi134 ceph-mon[30086]: 6.17 scrub ok 2024-02-05T08:24:38.364 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:38 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:24:38.364 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:38 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:24:38.364 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:38 smithi134 ceph-mon[39278]: 5.2 scrub ok 2024-02-05T08:24:38.364 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:38 smithi134 ceph-mon[39278]: 6.17 scrub starts 2024-02-05T08:24:38.364 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:38 smithi134 ceph-mon[39278]: 6.17 scrub ok 2024-02-05T08:24:38.365 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:38 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:24:38.365 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:38 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:24:38.380 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:38.418 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:38.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:38.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:38.554 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:38.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:38.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:38 smithi196 ceph-mon[33018]: 5.2 scrub ok 2024-02-05T08:24:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:38 smithi196 ceph-mon[33018]: 6.17 scrub starts 2024-02-05T08:24:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:38 smithi196 ceph-mon[33018]: 6.17 scrub ok 2024-02-05T08:24:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:38 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:24:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:38 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:24:38.656 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:38.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:38.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:38.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:38.744 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 79 16 121686 121670 96.2371 77.625 0.0149133 0.010386 2024-02-05T08:24:38.791 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:38.866 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:38.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:38.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:39.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:39.005 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:39.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:39.106 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:39.151 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:39.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:39.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:39.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:39.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[30086]: 6.10 scrub starts 2024-02-05T08:24:39.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[30086]: 6.10 scrub ok 2024-02-05T08:24:39.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[30086]: pgmap v890: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 864 op/s 2024-02-05T08:24:39.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[30086]: osdmap e392: 8 total, 8 up, 7 in 2024-02-05T08:24:39.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[30086]: 3.9 deep-scrub starts 2024-02-05T08:24:39.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[30086]: 3.9 deep-scrub ok 2024-02-05T08:24:39.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[30086]: 5.3 deep-scrub starts 2024-02-05T08:24:39.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[30086]: 5.3 deep-scrub ok 2024-02-05T08:24:39.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:24:39.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:24:39.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[30086]: 4.1e scrub starts 2024-02-05T08:24:39.280 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[30086]: 4.1e scrub ok 2024-02-05T08:24:39.280 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[39278]: 6.10 scrub starts 2024-02-05T08:24:39.280 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[39278]: 6.10 scrub ok 2024-02-05T08:24:39.280 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[39278]: pgmap v890: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 864 op/s 2024-02-05T08:24:39.280 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[39278]: osdmap e392: 8 total, 8 up, 7 in 2024-02-05T08:24:39.281 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[39278]: 3.9 deep-scrub starts 2024-02-05T08:24:39.281 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[39278]: 3.9 deep-scrub ok 2024-02-05T08:24:39.281 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[39278]: 5.3 deep-scrub starts 2024-02-05T08:24:39.281 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[39278]: 5.3 deep-scrub ok 2024-02-05T08:24:39.281 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:24:39.281 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:24:39.281 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[39278]: 4.1e scrub starts 2024-02-05T08:24:39.281 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:39 smithi134 ceph-mon[39278]: 4.1e scrub ok 2024-02-05T08:24:39.290 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:39.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:39.318 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:39.391 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:39.443 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:39.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:39.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:39 smithi196 ceph-mon[33018]: 6.10 scrub starts 2024-02-05T08:24:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:39 smithi196 ceph-mon[33018]: 6.10 scrub ok 2024-02-05T08:24:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:39 smithi196 ceph-mon[33018]: pgmap v890: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 864 op/s 2024-02-05T08:24:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:39 smithi196 ceph-mon[33018]: osdmap e392: 8 total, 8 up, 7 in 2024-02-05T08:24:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:39 smithi196 ceph-mon[33018]: 3.9 deep-scrub starts 2024-02-05T08:24:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:39 smithi196 ceph-mon[33018]: 3.9 deep-scrub ok 2024-02-05T08:24:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:39 smithi196 ceph-mon[33018]: 5.3 deep-scrub starts 2024-02-05T08:24:39.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:39 smithi196 ceph-mon[33018]: 5.3 deep-scrub ok 2024-02-05T08:24:39.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:24:39.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:24:39.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:39 smithi196 ceph-mon[33018]: 4.1e scrub starts 2024-02-05T08:24:39.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:39 smithi196 ceph-mon[33018]: 4.1e scrub ok 2024-02-05T08:24:39.585 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:39.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:39.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:39.732 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:39.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:39.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:39.744 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:2024-02-05T08:24:39.745368+0000 min lat: 0.00126536 max lat: 0.205027 avg lat: 0.0104136 2024-02-05T08:24:39.745 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T08:24:39.745 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 80 16 122910 122894 95.9903 76.5 0.011141 0.0104136 2024-02-05T08:24:39.835 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:39.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:39.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:39.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:40.020 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:40.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:40.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:40.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:40.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:40.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:40.192 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:40.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:40.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:40.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:40.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:40.363 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:40 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:24:40.363 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:40 smithi134 ceph-mon[30086]: osdmap e393: 8 total, 8 up, 7 in 2024-02-05T08:24:40.363 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:40 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:24:40.363 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:40 smithi134 ceph-mon[39278]: osdmap e393: 8 total, 8 up, 7 in 2024-02-05T08:24:40.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:40.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:40.449 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:40.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:40.550 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:40 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:24:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:40 smithi196 ceph-mon[33018]: osdmap e393: 8 total, 8 up, 7 in 2024-02-05T08:24:40.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:40.594 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:40.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:40.695 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:40.733 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:40.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:40.744 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 81 16 124047 124031 95.6825 71.0625 0.00765857 0.010447 2024-02-05T08:24:40.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:24:40.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:40.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:40.881 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:40.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:40.982 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:41.027 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:41.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:41.130 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:41.168 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:41.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:41.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:41.306 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:41.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:41.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:41.449 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:41.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:41.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:41 smithi134 ceph-mon[30086]: pgmap v893: 158 pgs: 158 active+clean; 7.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.5 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-02-05T08:24:41.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:41 smithi134 ceph-mon[30086]: 3.3 deep-scrub starts 2024-02-05T08:24:41.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:41 smithi134 ceph-mon[30086]: 3.3 deep-scrub ok 2024-02-05T08:24:41.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:41 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:24:41.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:41 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:24:41.492 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:41 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:24:41.492 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:41 smithi134 ceph-mon[39278]: pgmap v893: 158 pgs: 158 active+clean; 7.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.5 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-02-05T08:24:41.492 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:41 smithi134 ceph-mon[39278]: 3.3 deep-scrub starts 2024-02-05T08:24:41.492 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:41 smithi134 ceph-mon[39278]: 3.3 deep-scrub ok 2024-02-05T08:24:41.493 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:41 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:24:41.493 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:41 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:24:41.493 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:41 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:24:41.549 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:41 smithi196 ceph-mon[33018]: pgmap v893: 158 pgs: 158 active+clean; 7.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.5 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-02-05T08:24:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:41 smithi196 ceph-mon[33018]: 3.3 deep-scrub starts 2024-02-05T08:24:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:41 smithi196 ceph-mon[33018]: 3.3 deep-scrub ok 2024-02-05T08:24:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:41 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:24:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:41 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:24:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:41 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:24:41.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:41.594 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:41.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:41.695 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:41.737 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:41.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:41.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:41.744 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 82 16 125223 125207 95.412 73.5 0.0108936 0.0104766 2024-02-05T08:24:41.839 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:41.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:41.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:41.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:41.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:42.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:42.020 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:42.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:42.121 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:42.159 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:42.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:42.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:42.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:42 smithi134 ceph-mon[30086]: 11.3 scrub starts 2024-02-05T08:24:42.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:42 smithi134 ceph-mon[30086]: 6.1e scrub starts 2024-02-05T08:24:42.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:42 smithi134 ceph-mon[30086]: 6.1e scrub ok 2024-02-05T08:24:42.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:42 smithi134 ceph-mon[39278]: 11.3 scrub starts 2024-02-05T08:24:42.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:42 smithi134 ceph-mon[39278]: 6.1e scrub starts 2024-02-05T08:24:42.279 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:42 smithi134 ceph-mon[39278]: 6.1e scrub ok 2024-02-05T08:24:42.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:42.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:42.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:42.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:42.428 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:42.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:42.530 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:42.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:42 smithi196 ceph-mon[33018]: 11.3 scrub starts 2024-02-05T08:24:42.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:42 smithi196 ceph-mon[33018]: 6.1e scrub starts 2024-02-05T08:24:42.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:42 smithi196 ceph-mon[33018]: 6.1e scrub ok 2024-02-05T08:24:42.566 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:42.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:42.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:42.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:42.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:42.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:42.744 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 83 16 126409 126393 95.1555 74.125 0.0059347 0.0105041 2024-02-05T08:24:42.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:42.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:42.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:42.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:42.925 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:24:42.925 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:24:42.925 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-02-05T08:24:42.925 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-02-05T08:24:42.925 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:24:42.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:42.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:42.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:43.079 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:43.114 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:43.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:43.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:43.216 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:43.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:43.257 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:43.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:43.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:43 smithi134 ceph-mon[30086]: 5.1b scrub starts 2024-02-05T08:24:43.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:43 smithi134 ceph-mon[30086]: 5.1b scrub ok 2024-02-05T08:24:43.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:43 smithi134 ceph-mon[30086]: pgmap v894: 158 pgs: 158 active+clean; 7.4 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 867 op/s 2024-02-05T08:24:43.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:43 smithi134 ceph-mon[39278]: 5.1b scrub starts 2024-02-05T08:24:43.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:43 smithi134 ceph-mon[39278]: 5.1b scrub ok 2024-02-05T08:24:43.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:43 smithi134 ceph-mon[39278]: pgmap v894: 158 pgs: 158 active+clean; 7.4 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 867 op/s 2024-02-05T08:24:43.359 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:43.398 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:43.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:43.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:43.500 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:43.537 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:43.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:43 smithi196 ceph-mon[33018]: 5.1b scrub starts 2024-02-05T08:24:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:43 smithi196 ceph-mon[33018]: 5.1b scrub ok 2024-02-05T08:24:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:43 smithi196 ceph-mon[33018]: pgmap v894: 158 pgs: 158 active+clean; 7.4 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 54 MiB/s wr, 867 op/s 2024-02-05T08:24:43.639 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:43.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:43.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:43.745 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 84 16 127753 127737 95.0225 84 0.00657915 0.0105191 2024-02-05T08:24:43.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:43.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:43.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:43.916 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:43.964 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:43.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:44.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:44.103 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:44.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:44.204 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:44.240 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:44.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:44.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:44.342 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:44.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:44.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:44.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:44.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:44 smithi196 ceph-mon[33018]: 11.3 scrub ok 2024-02-05T08:24:44.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:44 smithi196 ceph-mon[33018]: 1.7 scrub starts 2024-02-05T08:24:44.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:44 smithi196 ceph-mon[33018]: 1.7 scrub ok 2024-02-05T08:24:44.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:24:44 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:24:44.484 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:44.523 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:44.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:44.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:44.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:44 smithi134 ceph-mon[30086]: 11.3 scrub ok 2024-02-05T08:24:44.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:44 smithi134 ceph-mon[30086]: 1.7 scrub starts 2024-02-05T08:24:44.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:44 smithi134 ceph-mon[30086]: 1.7 scrub ok 2024-02-05T08:24:44.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:44 smithi134 ceph-mon[39278]: 11.3 scrub ok 2024-02-05T08:24:44.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:44 smithi134 ceph-mon[39278]: 1.7 scrub starts 2024-02-05T08:24:44.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:44 smithi134 ceph-mon[39278]: 1.7 scrub ok 2024-02-05T08:24:44.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:44.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:44.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:44.745 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 85 16 129061 129045 94.8663 81.75 0.00904285 0.0105368 2024-02-05T08:24:44.767 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:44.810 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:44.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:44.912 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:44.963 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:44.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:45.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:45.110 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:45.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:45.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:45.212 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T08:24:45.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:45.249 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:45.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:45.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:45.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:45.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:45.455 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[30086]: pgmap v895: 158 pgs: 158 active+clean; 7.7 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-02-05T08:24:45.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:24:45.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:24:45.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:45.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[30086]: osdmap e394: 8 total, 8 up, 7 in 2024-02-05T08:24:45.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[30086]: 1.6 scrub starts 2024-02-05T08:24:45.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[30086]: 1.6 scrub ok 2024-02-05T08:24:45.456 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:24:45.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[30086]: osdmap e395: 8 total, 8 up, 7 in 2024-02-05T08:24:45.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[39278]: pgmap v895: 158 pgs: 158 active+clean; 7.7 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-02-05T08:24:45.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:24:45.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:24:45.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:45.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[39278]: osdmap e394: 8 total, 8 up, 7 in 2024-02-05T08:24:45.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[39278]: 1.6 scrub starts 2024-02-05T08:24:45.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[39278]: 1.6 scrub ok 2024-02-05T08:24:45.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:24:45.457 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:45 smithi134 ceph-mon[39278]: osdmap e395: 8 total, 8 up, 7 in 2024-02-05T08:24:45.470 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:45.472 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:45 smithi196 ceph-mon[33018]: pgmap v895: 158 pgs: 158 active+clean; 7.7 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s 2024-02-05T08:24:45.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:45 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:24:45.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:45 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:24:45.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:45 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:45.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:45 smithi196 ceph-mon[33018]: osdmap e394: 8 total, 8 up, 7 in 2024-02-05T08:24:45.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:45 smithi196 ceph-mon[33018]: 1.6 scrub starts 2024-02-05T08:24:45.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:45 smithi196 ceph-mon[33018]: 1.6 scrub ok 2024-02-05T08:24:45.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:45 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:24:45.473 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:45 smithi196 ceph-mon[33018]: osdmap e395: 8 total, 8 up, 7 in 2024-02-05T08:24:45.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:45.524 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:45.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:45.625 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:45.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:45.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:45.745 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 86 16 130139 130123 94.5466 67.375 0.00790861 0.0105724 2024-02-05T08:24:45.766 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:45.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:45.823 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:45.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:45.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:45.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:45.967 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:45.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:46.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:46.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:46.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:46.162 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:24:46.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:46.257 INFO:teuthology.orchestra.run.smithi134.stderr:set pool 11 pgp_num to 16 2024-02-05T08:24:46.300 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:46.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:46.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:46 smithi196 ceph-mon[33018]: 5.7 scrub starts 2024-02-05T08:24:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:46 smithi196 ceph-mon[33018]: 5.7 scrub ok 2024-02-05T08:24:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:46 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:24:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:46 smithi196 ceph-mon[33018]: 4.16 scrub starts 2024-02-05T08:24:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:46 smithi196 ceph-mon[33018]: 4.16 scrub ok 2024-02-05T08:24:46.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:46 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2487053028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:24:46.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:46 smithi196 ceph-mon[33018]: 5.1d scrub starts 2024-02-05T08:24:46.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:46 smithi196 ceph-mon[33018]: 5.1d scrub ok 2024-02-05T08:24:46.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:46 smithi196 ceph-mon[33018]: 6.15 scrub starts 2024-02-05T08:24:46.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:46 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3418844321' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T08:24:46.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:46.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:46.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:46.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[30086]: 5.7 scrub starts 2024-02-05T08:24:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[30086]: 5.7 scrub ok 2024-02-05T08:24:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:24:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[30086]: 4.16 scrub starts 2024-02-05T08:24:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[30086]: 4.16 scrub ok 2024-02-05T08:24:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2487053028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:24:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[30086]: 5.1d scrub starts 2024-02-05T08:24:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[30086]: 5.1d scrub ok 2024-02-05T08:24:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[30086]: 6.15 scrub starts 2024-02-05T08:24:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3418844321' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T08:24:46.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[39278]: 5.7 scrub starts 2024-02-05T08:24:46.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[39278]: 5.7 scrub ok 2024-02-05T08:24:46.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:24:46.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[39278]: 4.16 scrub starts 2024-02-05T08:24:46.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[39278]: 4.16 scrub ok 2024-02-05T08:24:46.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2487053028' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:24:46.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[39278]: 5.1d scrub starts 2024-02-05T08:24:46.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[39278]: 5.1d scrub ok 2024-02-05T08:24:46.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[39278]: 6.15 scrub starts 2024-02-05T08:24:46.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:46 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3418844321' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T08:24:46.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:46.591 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:46.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:46.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:46.693 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:46.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:46.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:46.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:46.745 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 87 16 131329 131313 94.3147 74.375 0.0104703 0.0105985 2024-02-05T08:24:46.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:46.869 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:46.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:46.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:47.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:47.009 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:47.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:47.110 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:47.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:47.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:47.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:47.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:47.287 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:47.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:47.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:47.388 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:47.428 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:47.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:47.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:47.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[30086]: 6.d deep-scrub starts 2024-02-05T08:24:47.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[30086]: pgmap v898: 158 pgs: 158 active+clean; 7.7 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 850 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-02-05T08:24:47.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[30086]: 6.15 scrub ok 2024-02-05T08:24:47.465 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[30086]: 6.d deep-scrub ok 2024-02-05T08:24:47.466 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[30086]: 3.b scrub starts 2024-02-05T08:24:47.466 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[30086]: 3.b scrub ok 2024-02-05T08:24:47.466 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:24:47.466 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2487053028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:24:47.466 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3418844321' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-02-05T08:24:47.468 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[30086]: osdmap e396: 8 total, 8 up, 7 in 2024-02-05T08:24:47.468 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:24:47.468 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:24:47.468 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[39278]: 6.d deep-scrub starts 2024-02-05T08:24:47.468 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[39278]: pgmap v898: 158 pgs: 158 active+clean; 7.7 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 850 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-02-05T08:24:47.468 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[39278]: 6.15 scrub ok 2024-02-05T08:24:47.469 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[39278]: 6.d deep-scrub ok 2024-02-05T08:24:47.469 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[39278]: 3.b scrub starts 2024-02-05T08:24:47.469 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[39278]: 3.b scrub ok 2024-02-05T08:24:47.469 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:24:47.469 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2487053028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:24:47.469 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3418844321' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-02-05T08:24:47.469 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[39278]: osdmap e396: 8 total, 8 up, 7 in 2024-02-05T08:24:47.469 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:24:47.469 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:24:47.530 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:47 smithi196 ceph-mon[33018]: 6.d deep-scrub starts 2024-02-05T08:24:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:47 smithi196 ceph-mon[33018]: pgmap v898: 158 pgs: 158 active+clean; 7.7 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 850 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-02-05T08:24:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:47 smithi196 ceph-mon[33018]: 6.15 scrub ok 2024-02-05T08:24:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:47 smithi196 ceph-mon[33018]: 6.d deep-scrub ok 2024-02-05T08:24:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:47 smithi196 ceph-mon[33018]: 3.b scrub starts 2024-02-05T08:24:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:47 smithi196 ceph-mon[33018]: 3.b scrub ok 2024-02-05T08:24:47.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:47 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:24:47.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:47 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2487053028' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:24:47.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:47 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3418844321' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-02-05T08:24:47.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:47 smithi196 ceph-mon[33018]: osdmap e396: 8 total, 8 up, 7 in 2024-02-05T08:24:47.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:47 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:24:47.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:47 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:24:47.570 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:47.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:47.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:47.672 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:47.710 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:47.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:47.745 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 88 16 132627 132611 94.1647 81.125 0.0130944 0.0106151 2024-02-05T08:24:47.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:47.847 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:47.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:47.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:47.985 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:47.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:47.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:48.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:24:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:24:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:24:48.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:48.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:48.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:48.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:48.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:48.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:48.277 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:48.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:48.379 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:48.414 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:48.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:48.516 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:48.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:48 smithi134 ceph-mon[30086]: osdmap e397: 8 total, 8 up, 7 in 2024-02-05T08:24:48.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:48 smithi134 ceph-mon[39278]: osdmap e397: 8 total, 8 up, 7 in 2024-02-05T08:24:48.541 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:48.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:48.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:48 smithi196 ceph-mon[33018]: osdmap e397: 8 total, 8 up, 7 in 2024-02-05T08:24:48.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:48.681 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:48.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:48.745 INFO:tasks.radosbench.radosbench.0.smithi134.stdout: 89 16 133804 133788 93.9332 73.5625 0.00860925 0.0106414 2024-02-05T08:24:48.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:48.820 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:48.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:48.921 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:48.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:48.964 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:48.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:49.065 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:49.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:49.104 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:49.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:49.205 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:49.234 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:49.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:49.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:49.336 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:49.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:49.372 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:49.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:49.473 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:49.513 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:49.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:49.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:49 smithi134 ceph-mon[30086]: pgmap v901: 158 pgs: 158 active+clean; 7.8 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 12 MiB/s wr, 193 op/s 2024-02-05T08:24:49.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:49 smithi134 ceph-mon[39278]: pgmap v901: 158 pgs: 158 active+clean; 7.8 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 12 MiB/s wr, 193 op/s 2024-02-05T08:24:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:49 smithi196 ceph-mon[33018]: pgmap v901: 158 pgs: 158 active+clean; 7.8 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 12 MiB/s wr, 193 op/s 2024-02-05T08:24:49.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:49.659 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:49.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:49.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:49.745 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Total time run: 90.0059 2024-02-05T08:24:49.745 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Total writes made: 135099 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Write size: 65536 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Object size: 65536 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Bandwidth (MB/sec): 93.8126 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev Bandwidth: 31.0405 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max bandwidth (MB/sec): 218.875 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min bandwidth (MB/sec): 64.75 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Average IOPS: 1501 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev IOPS: 496.648 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max IOPS: 3502 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min IOPS: 1036 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Average Latency(s): 0.0106557 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Stddev Latency(s): 0.00823356 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Max latency(s): 0.205027 2024-02-05T08:24:49.746 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Min latency(s): 0.00126536 2024-02-05T08:24:49.752 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T08:24:49.760 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:49.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:49.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:49.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:49.868 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:24:49.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:49.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:49.936 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:49.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:50.037 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:50.073 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:50.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:50.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:50.175 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:50.211 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:50.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:50.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:50.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:50.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:50.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:50.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:50.479 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:50.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:50.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:50.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:50.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:50.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:50.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:50.739 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:50.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:50.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:50.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:50.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:50.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:50.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:51.002 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:51.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:51.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:51.142 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:51.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:51.244 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:51.262 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:50 smithi134 ceph-mon[30086]: pgmap v902: 158 pgs: 158 active+clean; 8.1 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 884 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-05T08:24:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:50 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:24:51.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:50 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:24:51.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:50 smithi134 ceph-mon[39278]: pgmap v902: 158 pgs: 158 active+clean; 8.1 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 884 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-05T08:24:51.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:50 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:24:51.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:50 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:24:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:51.314 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:51.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:51.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:50 smithi196 ceph-mon[33018]: pgmap v902: 158 pgs: 158 active+clean; 8.1 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 884 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-05T08:24:51.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:50 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:24:51.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:50 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:24:51.415 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:51.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:51.466 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:51.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:51.567 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:51.604 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:51.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:51.705 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:51.740 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:51.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:51.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:51.842 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:51.875 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:51.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:51.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:52.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:52.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:52.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:52.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:52.136 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 7, 6, 3] out_osds: [0] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-02-05T08:24:52.136 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:24:52.137 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-05T08:24:52.137 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.482624 2024-02-05T08:24:52.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 2 0.4826235733218951 2024-02-05T08:24:52.144 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:52.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:52.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:52.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:52.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:51 smithi134 ceph-mon[30086]: osdmap e398: 8 total, 8 up, 7 in 2024-02-05T08:24:52.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:51 smithi134 ceph-mon[39278]: osdmap e398: 8 total, 8 up, 7 in 2024-02-05T08:24:52.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:52.286 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:52.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:51 smithi196 ceph-mon[33018]: osdmap e398: 8 total, 8 up, 7 in 2024-02-05T08:24:52.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:52.420 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:52.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:52.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:52.522 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:52.559 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:52.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:52.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:52.661 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:52.700 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:52.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:52.802 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:52.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:52.837 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:52.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:52.938 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:52.996 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:52.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:52.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:53.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:53.113 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:52 smithi134 ceph-mon[39278]: pgmap v904: 158 pgs: 158 active+clean; 8.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 11 KiB/s rd, 70 MiB/s wr, 1.17k op/s 2024-02-05T08:24:53.115 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:52 smithi134 ceph-mon[30086]: pgmap v904: 158 pgs: 158 active+clean; 8.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 11 KiB/s rd, 70 MiB/s wr, 1.17k op/s 2024-02-05T08:24:53.125 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:53.147 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:53.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:53.248 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:53.308 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:53.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:53.313 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:52 smithi196 ceph-mon[33018]: pgmap v904: 158 pgs: 158 active+clean; 8.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 11 KiB/s rd, 70 MiB/s wr, 1.17k op/s 2024-02-05T08:24:53.410 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:53.450 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:53.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:53.552 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:53.587 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:53.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:53.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:53.689 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:53.724 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:53.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:53.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:53.826 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:53.872 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:53.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:53.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:53.974 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:54.013 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:54.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:54.114 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:54.151 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:54.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:54.253 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:54.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:54.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:54.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:54.390 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:54.426 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:54.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:54.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:54.561 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:54.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:54.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:24:54 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:24:54.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:54.701 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:54.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:54.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:54.803 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:54.836 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:54.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:54.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:54.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:54.973 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:54.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:54.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:55.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:24:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:24:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:54 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1466934613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:24:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:54 smithi134 ceph-mon[30086]: pgmap v905: 158 pgs: 158 active+clean; 8.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 57 KiB/s rd, 60 MiB/s wr, 1.60k op/s 2024-02-05T08:24:55.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:54 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:24:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:24:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:54 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1466934613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:24:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:54 smithi134 ceph-mon[39278]: pgmap v905: 158 pgs: 158 active+clean; 8.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 57 KiB/s rd, 60 MiB/s wr, 1.60k op/s 2024-02-05T08:24:55.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:54 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:24:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:24:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:54 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1466934613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:24:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:54 smithi196 ceph-mon[33018]: pgmap v905: 158 pgs: 158 active+clean; 8.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 57 KiB/s rd, 60 MiB/s wr, 1.60k op/s 2024-02-05T08:24:55.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:54 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:24:55.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:55.112 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:55.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:55.214 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:55.247 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:55.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:55.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:55.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:55.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:55.388 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:55.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:55.489 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:55.524 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:55.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:55.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:55.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:55.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:55.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:55.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:55.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:55.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:55.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:55.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:55.933 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:55.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:56.034 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:56.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:56.082 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:56.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:56.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:56.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:56.223 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:56.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:56.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:56.378 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:56.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:56.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:56.480 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:56.516 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:56.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:56.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:56.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:56.657 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:56.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:56.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:56.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:56.792 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:56.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:56.895 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:56.930 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:56.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:56.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:57.032 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:57.069 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:57.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:57.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:57.092 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:56 smithi134 ceph-mon[30086]: pgmap v906: 158 pgs: 158 active+clean; 8.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 50 KiB/s rd, 53 MiB/s wr, 1.42k op/s 2024-02-05T08:24:57.092 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:56 smithi134 ceph-mon[39278]: pgmap v906: 158 pgs: 158 active+clean; 8.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 50 KiB/s rd, 53 MiB/s wr, 1.42k op/s 2024-02-05T08:24:57.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:57.208 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:57.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:57.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:56 smithi196 ceph-mon[33018]: pgmap v906: 158 pgs: 158 active+clean; 8.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 50 KiB/s rd, 53 MiB/s wr, 1.42k op/s 2024-02-05T08:24:57.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:57.346 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:57.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:57.447 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:57.495 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:57.496 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:24:57.496 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:24:57.497 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:24:57.497 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:24:57.497 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:24:57.497 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:24:57.498 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:24:57.498 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:24:57.498 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:24:57.498 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:24:57.499 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:24:57.499 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:24:57.499 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:24:57.499 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:24:57.499 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:24:57.499 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:24:57.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:57.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:57.601 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:57.635 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:57.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:57.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:57.775 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:57.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:57.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:24:57.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:57.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:24:58.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:58.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:24:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:24:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:24:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:58.057 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:58.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:58.157 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:58.194 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:58.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:58.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:58.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:58.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:58.334 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:58.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:58.435 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:58.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:58.479 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:58.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:58.581 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:24:58.621 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:24:58.621 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:24:58.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:58.633 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:58.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:24:58.734 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:58.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:58.770 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:58.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:24:58.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:24:58.911 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:58.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:58.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:24:59.013 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:59.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:59.059 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:59.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:59.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:59.195 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:59.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:59.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:59.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:24:58 smithi134 ceph-mon[30086]: pgmap v907: 158 pgs: 158 active+clean; 7.9 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 44 KiB/s rd, 46 MiB/s wr, 1.33k op/s 2024-02-05T08:24:59.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:24:58 smithi134 ceph-mon[39278]: pgmap v907: 158 pgs: 158 active+clean; 7.9 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 44 KiB/s rd, 46 MiB/s wr, 1.33k op/s 2024-02-05T08:24:59.297 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:24:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:24:58 smithi196 ceph-mon[33018]: pgmap v907: 158 pgs: 158 active+clean; 7.9 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 44 KiB/s rd, 46 MiB/s wr, 1.33k op/s 2024-02-05T08:24:59.321 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:59.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:24:59.423 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:24:59.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:24:59.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:59.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:24:59.559 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:24:59.598 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:59.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:24:59.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T08:24:59.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:24:59.750 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:59.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:59.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:24:59.852 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:24:59.857 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:24:59.903 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:24:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:24:59.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:00.005 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:00.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:00.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:00.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:00.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:00.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:00.194 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:00.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:00.293 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:25:00.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:00.341 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:00.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:00.442 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:00.487 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:00.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:00.589 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:00.636 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:00.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:00.737 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:00.745 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:00.804 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:00.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:00.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:00.969 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:00.970 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:25:00.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:00.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:01.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:01.105 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:01.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:01.207 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:01.246 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:01.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:01.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:00 smithi134 ceph-mon[30086]: pgmap v908: 158 pgs: 158 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-02-05T08:25:01.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:00 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1858130557' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:25:01.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:00 smithi134 ceph-mon[39278]: pgmap v908: 158 pgs: 158 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-02-05T08:25:01.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:00 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1858130557' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:25:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:00 smithi196 ceph-mon[33018]: pgmap v908: 158 pgs: 158 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 44 KiB/s rd, 13 MiB/s wr, 1.42k op/s 2024-02-05T08:25:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:00 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1858130557' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:25:01.347 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:01.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:01.382 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:01.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:01.483 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:01.533 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:01.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:01.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:01.635 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:01.670 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:01.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:01.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:01.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:01.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:01.807 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:01.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:01.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:01.945 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:01.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:02.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:01 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:25:02.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:01 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1858130557' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:25:02.016 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:01 smithi134 ceph-mon[30086]: osdmap e399: 8 total, 8 up, 7 in 2024-02-05T08:25:02.016 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:01 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:25:02.017 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:01 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1858130557' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:25:02.017 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:01 smithi134 ceph-mon[39278]: osdmap e399: 8 total, 8 up, 7 in 2024-02-05T08:25:02.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:02.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:02.087 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:02.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:02.188 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:02.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:02.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:02.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:01 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:25:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:01 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1858130557' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:25:02.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:01 smithi196 ceph-mon[33018]: osdmap e399: 8 total, 8 up, 7 in 2024-02-05T08:25:02.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:02.360 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:02.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:02.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:02.500 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:02.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:02.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:02.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:02.633 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:02.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:02.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:02.735 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:02.771 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:02.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:02.872 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:02.907 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:02.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:03.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:03.045 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:03.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:03.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:03.068 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:02 smithi134 ceph-mon[30086]: pgmap v910: 158 pgs: 158 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 38 KiB/s rd, 9.0 MiB/s wr, 1.42k op/s 2024-02-05T08:25:03.068 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:02 smithi134 ceph-mon[39278]: pgmap v910: 158 pgs: 158 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 38 KiB/s rd, 9.0 MiB/s wr, 1.42k op/s 2024-02-05T08:25:03.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:03.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:03.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:03.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:03.302 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:02 smithi196 ceph-mon[33018]: pgmap v910: 158 pgs: 158 active+clean; 7.5 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 38 KiB/s rd, 9.0 MiB/s wr, 1.42k op/s 2024-02-05T08:25:03.337 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:03.338 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:03.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:03.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:03.441 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:03.466 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:03.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:03.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:03.568 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:03.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:03.602 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:03.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:03.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:03.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:03.743 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:03.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:03.844 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:03.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:03.870 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:03.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:03.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:03.974 INFO:teuthology.orchestra.run.smithi134.stderr:set osd.2 primary-affinity to 0.482624 (8316292) 2024-02-05T08:25:04.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:04.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:04.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:04.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:04.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:04.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:04.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:04.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:04.254 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:04.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:04.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:03 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2509031693' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.4826235733218951}]: dispatch 2024-02-05T08:25:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:03 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2509031693' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.4826235733218951}]: dispatch 2024-02-05T08:25:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:03 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2509031693' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.4826235733218951}]: dispatch 2024-02-05T08:25:04.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:25:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:25:04.356 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:04.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:04.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:04.481 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:04.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:04.506 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:04.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:04.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:04.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:04.635 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:04.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:04.737 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:04.761 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:04.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:04.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:04.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:04.887 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:04.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:04.988 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:05.021 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:05.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:05.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:04 smithi134 ceph-mon[30086]: pgmap v911: 158 pgs: 158 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s 2024-02-05T08:25:05.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:04 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2509031693' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.4826235733218951}]': finished 2024-02-05T08:25:05.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:04 smithi134 ceph-mon[30086]: osdmap e400: 8 total, 8 up, 7 in 2024-02-05T08:25:05.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:04 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:05.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:04 smithi134 ceph-mon[39278]: pgmap v911: 158 pgs: 158 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s 2024-02-05T08:25:05.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:04 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2509031693' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.4826235733218951}]': finished 2024-02-05T08:25:05.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:04 smithi134 ceph-mon[39278]: osdmap e400: 8 total, 8 up, 7 in 2024-02-05T08:25:05.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:04 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:05.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:05.159 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:05.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:05.261 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:05.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:05.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:04 smithi196 ceph-mon[33018]: pgmap v911: 158 pgs: 158 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s 2024-02-05T08:25:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:04 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2509031693' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.4826235733218951}]': finished 2024-02-05T08:25:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:04 smithi196 ceph-mon[33018]: osdmap e400: 8 total, 8 up, 7 in 2024-02-05T08:25:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:04 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:05.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:05.433 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:05.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:05.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:05.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:05.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:05.573 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:05.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:05.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:05.714 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:05.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:05.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:05.851 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:05.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:05.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:05.953 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:05.990 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:05.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:06.092 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:06.129 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:06.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:06.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:06.159 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:05 smithi134 ceph-mon[30086]: osdmap e401: 8 total, 8 up, 7 in 2024-02-05T08:25:06.160 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:05 smithi134 ceph-mon[39278]: osdmap e401: 8 total, 8 up, 7 in 2024-02-05T08:25:06.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:06.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:06.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:05 smithi196 ceph-mon[33018]: osdmap e401: 8 total, 8 up, 7 in 2024-02-05T08:25:06.363 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:06.366 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:25:06.366 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:25:06.401 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:06.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:06.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:06.542 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:06.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:06.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:06.678 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:06.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:06.779 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:06.782 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:25:06.812 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:06.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:06.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:06.913 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:06.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:06.961 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:06.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:07.062 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:07.099 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:06 smithi134 ceph-mon[30086]: pgmap v914: 158 pgs: 158 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-05T08:25:07.099 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:06 smithi134 ceph-mon[39278]: pgmap v914: 158 pgs: 158 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-05T08:25:07.118 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:07.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:07.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:07.220 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:07.262 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:07.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:07.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:06 smithi196 ceph-mon[33018]: pgmap v914: 158 pgs: 158 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.11k op/s 2024-02-05T08:25:07.363 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:07.401 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:07.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:07.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:07.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:07.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:07.544 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:07.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:07.645 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:07.692 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:07.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:07.794 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:07.826 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:07.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:07.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:07.966 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:07.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:08.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:25:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:25:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:25:08.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:08.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:08.105 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:08.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:08.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:08.242 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:08.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:08.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:08.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:08.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:08.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:08.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:08.479 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:08.517 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:08.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:08.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:08.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:08.624 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T08:25:08.661 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:08.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:08.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:08.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:08.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:08.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:08.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:08.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:08.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:08.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:09.033 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:09.074 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:09.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:09.176 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:09.187 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:08 smithi134 ceph-mon[30086]: pgmap v915: 158 pgs: 158 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 587 B/s rd, 0 B/s wr, 1.08k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:09.188 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:08 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:25:09.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:08 smithi134 ceph-mon[39278]: pgmap v915: 158 pgs: 158 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 587 B/s rd, 0 B/s wr, 1.08k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:09.188 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:09 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:25:09.198 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:09.205 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:09.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:09.302 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:25:09.307 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:08 smithi196 ceph-mon[33018]: pgmap v915: 158 pgs: 158 active+clean; 7.1 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 587 B/s rd, 0 B/s wr, 1.08k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:08 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:25:09.339 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:25:09.351 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:09.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:09.381 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:25:09.421 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:25:09.454 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:09.457 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:25:09.485 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:09.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:09.497 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:25:09.533 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:25:09.574 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:25:09.580 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:25:09.580 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:25:09.580 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:25:09.580 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:25:09.580 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:25:09.580 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:25:09.580 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:25:09.580 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:25:09.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:09.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:09.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:09.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:09.718 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:09.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:09.753 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:09.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:09.854 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:09.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:09.892 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:09.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:09.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:10.029 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:10.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:10.131 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:10.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:10.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:10.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:10.267 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:10.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:10 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/77324985' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:25:10.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:10 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/77324985' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:25:10.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:10.305 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:10.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:10 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/77324985' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:25:10.405 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:10.453 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:10.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:10.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:10.555 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:10.584 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:10.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:10.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:10.588 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:10.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:10.724 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:10.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:10.826 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:10.862 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:10.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:10.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:10.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:11.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:11.001 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:11.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:11.102 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:11.137 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:11.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:11.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:11.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:11.263 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:11 smithi134 ceph-mon[30086]: pgmap v916: 158 pgs: 158 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:11.263 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:11 smithi134 ceph-mon[39278]: pgmap v916: 158 pgs: 158 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:11.280 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:11.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:11.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:11 smithi196 ceph-mon[33018]: pgmap v916: 158 pgs: 158 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.40k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:11.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:11.417 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:11.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:11.519 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:11.560 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:11.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:11.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:11.662 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:11.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:11.697 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:11.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:11.798 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:11.835 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:11.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:11.936 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:11.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:11.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:12.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:12.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:12.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:12.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:12.101 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 7, 6, 3] out_osds: [0] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-02-05T08:25:12.101 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:25:12.101 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-02-05T08:25:12.101 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-02-05T08:25:12.101 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:25:12.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:12.235 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:12.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:12.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:12.363 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:12.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:12.465 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:12.505 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:12.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:12.606 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:12.645 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:12.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:12.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:12.747 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:12.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:12.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:12.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:12.886 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:12.931 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:12.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:12.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:13.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:13 smithi134 ceph-mon[30086]: pgmap v917: 158 pgs: 158 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 832 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:13.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:13 smithi134 ceph-mon[39278]: pgmap v917: 158 pgs: 158 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 832 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:13.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:13.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:13.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:13.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:13.215 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:13.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:13 smithi196 ceph-mon[33018]: pgmap v917: 158 pgs: 158 active+clean; 6.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 832 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:13.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:13.353 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:13.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:13.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:13.455 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:13.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:13.479 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:13.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:13.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:13.613 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:13.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:13.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:13.714 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:13.747 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:13.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:13.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:13.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:13.887 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:13.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:13.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:14.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:14.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:14.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:25:14.130 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:14.167 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:14.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:14.269 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:14.308 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:14.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:14.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:14.450 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:14.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:14.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:14.552 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:14.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:25:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:25:14.590 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:14.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:14.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:14.692 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:14.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:14.722 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:14.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:14.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:14.862 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:14.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:14.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:14.994 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:14.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:15.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:15.134 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:15.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:15.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:15.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:15.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:15.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:15.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:15 smithi134 ceph-mon[30086]: pgmap v918: 158 pgs: 158 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 924 B/s rd, 0 B/s wr, 1.22k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:15.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:15 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:15 smithi134 ceph-mon[39278]: pgmap v918: 158 pgs: 158 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 924 B/s rd, 0 B/s wr, 1.22k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:15 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:15 smithi196 ceph-mon[33018]: pgmap v918: 158 pgs: 158 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 924 B/s rd, 0 B/s wr, 1.22k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:15 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:15.371 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:15.407 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:15.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:15.509 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:15.540 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:15.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:15.642 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:15.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:15.680 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:15.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:15.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:15.832 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:15.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:15.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:15.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:15.981 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:15.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:15.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:16.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:16.120 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:16.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:16.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:16.222 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:16.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:16.278 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:16.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:16.379 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:16.421 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:16.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:16.522 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:16.573 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:16.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:16.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:16.706 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:16.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:16.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:16.807 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:16.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:16.838 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:16.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:16.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:16.867 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:25:16.917 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:25:16.917 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-02-05T08:25:16.918 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-02-05T08:25:16.918 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:25:16.940 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:16.977 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:16.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:16.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:17.039 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:25:17.079 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:17.113 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:17.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:17.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:17.215 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:17.249 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:17.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:17.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:17 smithi134 ceph-mon[30086]: pgmap v919: 158 pgs: 158 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 840 B/s rd, 0 B/s wr, 1.11k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:17.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:17 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2874749191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:25:17.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:17 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:25:17.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:17 smithi134 ceph-mon[39278]: pgmap v919: 158 pgs: 158 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 840 B/s rd, 0 B/s wr, 1.11k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:17.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:17 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2874749191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:25:17.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:17 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:25:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:17 smithi196 ceph-mon[33018]: pgmap v919: 158 pgs: 158 active+clean; 6.4 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 840 B/s rd, 0 B/s wr, 1.11k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T08:25:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:17 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2874749191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:25:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:17 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:25:17.352 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:17.390 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:17.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:17.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:17.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:17.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:17.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:17.628 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:17.667 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:17.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:17.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:17.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:17.806 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:17.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:17.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:17.909 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:17.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:17.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:17.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:18.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:25:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:25:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:25:18.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:18.083 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:18.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:18.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:18.185 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:18.221 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:18.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:18 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:25:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:18 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:25:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:18 smithi196 ceph-mon[33018]: osdmap e402: 8 total, 8 up, 7 in 2024-02-05T08:25:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:18 smithi196 ceph-mon[33018]: 5.10 scrub starts 2024-02-05T08:25:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:18 smithi196 ceph-mon[33018]: 5.10 scrub ok 2024-02-05T08:25:18.322 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:18.362 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:18.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:18.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:18.476 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:18 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:25:18.476 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:18 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:25:18.476 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:18 smithi134 ceph-mon[30086]: osdmap e402: 8 total, 8 up, 7 in 2024-02-05T08:25:18.476 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:18 smithi134 ceph-mon[30086]: 5.10 scrub starts 2024-02-05T08:25:18.477 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:18 smithi134 ceph-mon[30086]: 5.10 scrub ok 2024-02-05T08:25:18.477 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:18 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:25:18.477 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:18 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:25:18.477 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:18 smithi134 ceph-mon[39278]: osdmap e402: 8 total, 8 up, 7 in 2024-02-05T08:25:18.477 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:18 smithi134 ceph-mon[39278]: 5.10 scrub starts 2024-02-05T08:25:18.477 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:18 smithi134 ceph-mon[39278]: 5.10 scrub ok 2024-02-05T08:25:18.501 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:18.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:18.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:18.603 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:18.637 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:18.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:18.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:18.772 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:18.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:18.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:18.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:18.910 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:18.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:18.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:19.012 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:19.049 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:19.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:19.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:19.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:19.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:19.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:19.286 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:19 smithi196 ceph-mon[33018]: pgmap v921: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T08:25:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:19 smithi196 ceph-mon[33018]: 11.0 scrub starts 2024-02-05T08:25:19.319 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:19.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:19.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:19.421 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:19.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:19.460 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:19.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:19.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:19 smithi134 ceph-mon[30086]: pgmap v921: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T08:25:19.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:19 smithi134 ceph-mon[30086]: 11.0 scrub starts 2024-02-05T08:25:19.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:19 smithi134 ceph-mon[39278]: pgmap v921: 158 pgs: 158 active+clean; 6.3 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s 2024-02-05T08:25:19.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:19 smithi134 ceph-mon[39278]: 11.0 scrub starts 2024-02-05T08:25:19.561 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:19.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:19.595 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:19.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:19.696 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:19.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:19.737 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:19.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:19.838 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:19.874 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:19.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:19.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:20.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:20.014 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:20.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:20.115 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:20.154 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:20.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:20.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:20.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:20.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:20.290 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:20.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:20.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:20.429 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:20.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:20.530 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:20.565 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:20.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:20.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:20.667 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:20.704 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:20.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:20.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:20.841 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:20.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:20.943 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:20.978 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:20.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:20 smithi134 ceph-mon[30086]: pgmap v922: 158 pgs: 158 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-02-05T08:25:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:20 smithi134 ceph-mon[30086]: 2.1 scrub starts 2024-02-05T08:25:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:20 smithi134 ceph-mon[30086]: 2.1 scrub ok 2024-02-05T08:25:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:20 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:25:21.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:20 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:25:21.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:20 smithi134 ceph-mon[39278]: pgmap v922: 158 pgs: 158 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-02-05T08:25:21.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:20 smithi134 ceph-mon[39278]: 2.1 scrub starts 2024-02-05T08:25:21.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:20 smithi134 ceph-mon[39278]: 2.1 scrub ok 2024-02-05T08:25:21.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:20 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:25:21.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:20 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:25:21.081 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:21.117 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:21.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:21.137 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:21.219 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:21.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:21.264 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:21.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:20 smithi196 ceph-mon[33018]: pgmap v922: 158 pgs: 158 active+clean; 6.0 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-02-05T08:25:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:20 smithi196 ceph-mon[33018]: 2.1 scrub starts 2024-02-05T08:25:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:20 smithi196 ceph-mon[33018]: 2.1 scrub ok 2024-02-05T08:25:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:20 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:25:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:20 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:25:21.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:21.397 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:21.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:21.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:21.499 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:21.535 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:21.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:21.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:21.638 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:21.672 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:21.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:21.775 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:21.815 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:21.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:21.917 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:21.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:21.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:22.056 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:22.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:22.096 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:22.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:22.197 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:22.233 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:22.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:21 smithi134 ceph-mon[30086]: 5.c scrub starts 2024-02-05T08:25:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:21 smithi134 ceph-mon[30086]: 5.c scrub ok 2024-02-05T08:25:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:21 smithi134 ceph-mon[30086]: 11.0 scrub ok 2024-02-05T08:25:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:21 smithi134 ceph-mon[30086]: 3.d scrub starts 2024-02-05T08:25:22.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:21 smithi134 ceph-mon[30086]: 3.d scrub ok 2024-02-05T08:25:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:21 smithi134 ceph-mon[39278]: 5.c scrub starts 2024-02-05T08:25:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:21 smithi134 ceph-mon[39278]: 5.c scrub ok 2024-02-05T08:25:22.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:21 smithi134 ceph-mon[39278]: 11.0 scrub ok 2024-02-05T08:25:22.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:21 smithi134 ceph-mon[39278]: 3.d scrub starts 2024-02-05T08:25:22.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:21 smithi134 ceph-mon[39278]: 3.d scrub ok 2024-02-05T08:25:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:21 smithi196 ceph-mon[33018]: 5.c scrub starts 2024-02-05T08:25:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:21 smithi196 ceph-mon[33018]: 5.c scrub ok 2024-02-05T08:25:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:21 smithi196 ceph-mon[33018]: 11.0 scrub ok 2024-02-05T08:25:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:21 smithi196 ceph-mon[33018]: 3.d scrub starts 2024-02-05T08:25:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:21 smithi196 ceph-mon[33018]: 3.d scrub ok 2024-02-05T08:25:22.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:22.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:22.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:22.389 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T08:25:22.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:22.499 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:22.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:22.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:22.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:22.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:22.656 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:22.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:22.757 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:22.786 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:25:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:22.805 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:22.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:22.906 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:22.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:22.953 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:22.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:23.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:23.086 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:23.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:23.188 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:23.207 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:22 smithi134 ceph-mon[30086]: 3.14 scrub starts 2024-02-05T08:25:23.207 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:22 smithi134 ceph-mon[30086]: 3.14 scrub ok 2024-02-05T08:25:23.207 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:22 smithi134 ceph-mon[30086]: 11.c scrub starts 2024-02-05T08:25:23.207 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:22 smithi134 ceph-mon[30086]: pgmap v923: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-02-05T08:25:23.207 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:22 smithi134 ceph-mon[30086]: 6.5 scrub starts 2024-02-05T08:25:23.208 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:22 smithi134 ceph-mon[39278]: 3.14 scrub starts 2024-02-05T08:25:23.208 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:22 smithi134 ceph-mon[39278]: 3.14 scrub ok 2024-02-05T08:25:23.208 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:22 smithi134 ceph-mon[39278]: 11.c scrub starts 2024-02-05T08:25:23.208 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:22 smithi134 ceph-mon[39278]: pgmap v923: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-02-05T08:25:23.208 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:22 smithi134 ceph-mon[39278]: 6.5 scrub starts 2024-02-05T08:25:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:23.222 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:23.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:22 smithi196 ceph-mon[33018]: 3.14 scrub starts 2024-02-05T08:25:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:22 smithi196 ceph-mon[33018]: 3.14 scrub ok 2024-02-05T08:25:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:22 smithi196 ceph-mon[33018]: 11.c scrub starts 2024-02-05T08:25:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:22 smithi196 ceph-mon[33018]: pgmap v923: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-02-05T08:25:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:22 smithi196 ceph-mon[33018]: 6.5 scrub starts 2024-02-05T08:25:23.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:23.360 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:23.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:23.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:23.496 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:23.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:23.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:23.597 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:23.625 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:23.627 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:23.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:23.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:23.703 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:25:23.707 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:25:23.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:23.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:23.774 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:23.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:23.875 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:23.914 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:23.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:23.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:24.016 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:24.051 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:24.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:24.153 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:24.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:24.201 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:24.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:24.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:23 smithi134 ceph-mon[30086]: 4.1a scrub starts 2024-02-05T08:25:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:23 smithi134 ceph-mon[30086]: 4.1a scrub ok 2024-02-05T08:25:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:23 smithi134 ceph-mon[30086]: 6.5 scrub ok 2024-02-05T08:25:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:23 smithi134 ceph-mon[30086]: from='client.16119 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:25:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:23 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:25:24.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:23 smithi134 ceph-mon[39278]: 4.1a scrub starts 2024-02-05T08:25:24.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:23 smithi134 ceph-mon[39278]: 4.1a scrub ok 2024-02-05T08:25:24.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:23 smithi134 ceph-mon[39278]: 6.5 scrub ok 2024-02-05T08:25:24.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:23 smithi134 ceph-mon[39278]: from='client.16119 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:25:24.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:23 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:25:24.303 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:24.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:25:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:25:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:23 smithi196 ceph-mon[33018]: 4.1a scrub starts 2024-02-05T08:25:24.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:23 smithi196 ceph-mon[33018]: 4.1a scrub ok 2024-02-05T08:25:24.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:23 smithi196 ceph-mon[33018]: 6.5 scrub ok 2024-02-05T08:25:24.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:23 smithi196 ceph-mon[33018]: from='client.16119 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:25:24.316 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:23 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:25:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:24.342 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:24.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:24.392 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:24.443 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:24.482 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:24.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:24.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:24.623 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:24.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:24.725 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:24.759 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:24.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:24.860 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:24.873 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":923,"stamp":"2024-02-05T08:25:21.938913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6344808820,"num_objects":97004,"num_object_clones":0,"num_object_copies":291012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97004,"num_whiteouts":0,"num_read":2779,"num_read_kb":2531,"num_write":174402,"num_write_kb":8646382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":355,"num_bytes_recovered":9332,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72562,"ondisk_log_size":72562,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":23224832,"kb_used_data":18752888,"kb_used_omap":13,"kb_used_meta":4471730,"kb_avail":632991232,"statfs":{"total":671965249536,"available":648183021568,"internally_reserved":0,"allocated":19202957312,"data_stored":19200404348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14001,"internal_metadata":4579051855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,8,11,11,9,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-882966528,"num_objects":-13473,"num_object_clones":0,"num_object_copies":-40419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-13473,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13473,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003170"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":749,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027270+0000","last_change":"2024-02-05T08:24:36.101074+0000","last_active":"2024-02-05T08:25:17.027270+0000","last_peered":"2024-02-05T08:25:17.027270+0000","last_clean":"2024-02-05T08:25:17.027270+0000","last_became_active":"2024-02-05T08:23:07.931642+0000","last_became_peered":"2024-02-05T08:23:07.931642+0000","last_unstale":"2024-02-05T08:25:17.027270+0000","last_undegraded":"2024-02-05T08:25:17.027270+0000","last_fullsized":"2024-02-05T08:25:17.027270+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:36.100961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:36.100961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:53.204567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":608,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.030409+0000","last_change":"2024-02-05T08:23:32.176306+0000","last_active":"2024-02-05T08:25:17.030409+0000","last_peered":"2024-02-05T08:25:17.030409+0000","last_clean":"2024-02-05T08:25:17.030409+0000","last_became_active":"2024-02-05T08:23:07.938772+0000","last_became_peered":"2024-02-05T08:23:07.938772+0000","last_unstale":"2024-02-05T08:25:17.030409+0000","last_undegraded":"2024-02-05T08:25:17.030409+0000","last_fullsized":"2024-02-05T08:25:17.030409+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:32.176199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:23:32.176199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":119,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.039516+0000","last_change":"2024-02-05T08:22:54.215968+0000","last_active":"2024-02-05T08:25:17.039516+0000","last_peered":"2024-02-05T08:25:17.039516+0000","last_clean":"2024-02-05T08:25:17.039516+0000","last_became_active":"2024-02-05T08:22:09.101941+0000","last_became_peered":"2024-02-05T08:22:09.101941+0000","last_unstale":"2024-02-05T08:25:17.039516+0000","last_undegraded":"2024-02-05T08:25:17.039516+0000","last_fullsized":"2024-02-05T08:25:17.039516+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:54.215846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:22:54.215846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"397'25","reported_seq":819,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.026868+0000","last_change":"2024-02-05T08:23:31.459738+0000","last_active":"2024-02-05T08:25:17.026868+0000","last_peered":"2024-02-05T08:25:17.026868+0000","last_clean":"2024-02-05T08:25:17.026868+0000","last_became_active":"2024-02-05T08:23:07.937041+0000","last_became_peered":"2024-02-05T08:23:07.937041+0000","last_unstale":"2024-02-05T08:25:17.026868+0000","last_undegraded":"2024-02-05T08:25:17.026868+0000","last_fullsized":"2024-02-05T08:25:17.026868+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:23:31.459611+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:23:31.459611+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174907+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":251,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041225+0000","last_change":"2024-02-05T08:23:24.076782+0000","last_active":"2024-02-05T08:25:17.041225+0000","last_peered":"2024-02-05T08:25:17.041225+0000","last_clean":"2024-02-05T08:25:17.041225+0000","last_became_active":"2024-02-05T08:18:34.633083+0000","last_became_peered":"2024-02-05T08:18:34.633083+0000","last_unstale":"2024-02-05T08:25:17.041225+0000","last_undegraded":"2024-02-05T08:25:17.041225+0000","last_fullsized":"2024-02-05T08:25:17.041225+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:24.076590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:23:24.076590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":757,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027035+0000","last_change":"2024-02-05T08:23:41.460609+0000","last_active":"2024-02-05T08:25:17.027035+0000","last_peered":"2024-02-05T08:25:17.027035+0000","last_clean":"2024-02-05T08:25:17.027035+0000","last_became_active":"2024-02-05T08:23:07.937180+0000","last_became_peered":"2024-02-05T08:23:07.937180+0000","last_unstale":"2024-02-05T08:25:17.027035+0000","last_undegraded":"2024-02-05T08:25:17.027035+0000","last_fullsized":"2024-02-05T08:25:17.027035+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:41.460468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:41.460468+0000","last_clean_scrub_stamp":"2024-02-05T08:23:41.460468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174851+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":261,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041980+0000","last_change":"2024-02-05T08:24:20.030502+0000","last_active":"2024-02-05T08:25:17.041980+0000","last_peered":"2024-02-05T08:25:17.041980+0000","last_clean":"2024-02-05T08:25:17.041980+0000","last_became_active":"2024-02-05T08:21:41.880038+0000","last_became_peered":"2024-02-05T08:21:41.880038+0000","last_unstale":"2024-02-05T08:25:17.041980+0000","last_undegraded":"2024-02-05T08:25:17.041980+0000","last_fullsized":"2024-02-05T08:25:17.041980+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:20.030382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:24:20.030382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:31.271518+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"397'46","reported_seq":315,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040881+0000","last_change":"2024-02-05T08:24:30.076306+0000","last_active":"2024-02-05T08:25:17.040881+0000","last_peered":"2024-02-05T08:25:17.040881+0000","last_clean":"2024-02-05T08:25:17.040881+0000","last_became_active":"2024-02-05T08:22:09.095896+0000","last_became_peered":"2024-02-05T08:22:09.095896+0000","last_unstale":"2024-02-05T08:25:17.040881+0000","last_undegraded":"2024-02-05T08:25:17.040881+0000","last_fullsized":"2024-02-05T08:25:17.040881+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:24:30.076161+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:24:30.076161+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:45.272166+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":89,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.030478+0000","last_change":"2024-02-05T08:24:34.239724+0000","last_active":"2024-02-05T08:25:17.030478+0000","last_peered":"2024-02-05T08:25:17.030478+0000","last_clean":"2024-02-05T08:25:17.030478+0000","last_became_active":"2024-02-05T08:23:08.917199+0000","last_became_peered":"2024-02-05T08:23:08.917199+0000","last_unstale":"2024-02-05T08:25:17.030478+0000","last_undegraded":"2024-02-05T08:25:17.030478+0000","last_fullsized":"2024-02-05T08:25:17.030478+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:34.238451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:34.238451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:52.462779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":766,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.038665+0000","last_change":"2024-02-05T08:24:41.918729+0000","last_active":"2024-02-05T08:25:17.038665+0000","last_peered":"2024-02-05T08:25:17.038665+0000","last_clean":"2024-02-05T08:25:17.038665+0000","last_became_active":"2024-02-05T08:22:09.108745+0000","last_became_peered":"2024-02-05T08:22:09.108745+0000","last_unstale":"2024-02-05T08:25:17.038665+0000","last_undegraded":"2024-02-05T08:25:17.038665+0000","last_fullsized":"2024-02-05T08:25:17.038665+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:41.918616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:02.192889+0000","last_clean_scrub_stamp":"2024-02-05T08:24:41.918616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:52.898764+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":780,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040529+0000","last_change":"2024-02-05T08:23:40.814097+0000","last_active":"2024-02-05T08:25:17.040529+0000","last_peered":"2024-02-05T08:25:17.040529+0000","last_clean":"2024-02-05T08:25:17.040529+0000","last_became_active":"2024-02-05T08:12:29.684174+0000","last_became_peered":"2024-02-05T08:12:29.684174+0000","last_unstale":"2024-02-05T08:25:17.040529+0000","last_undegraded":"2024-02-05T08:25:17.040529+0000","last_fullsized":"2024-02-05T08:25:17.040529+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:40.813940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_clean_scrub_stamp":"2024-02-05T08:23:40.813940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887530+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"397'25","reported_seq":840,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027549+0000","last_change":"2024-02-05T08:22:09.118822+0000","last_active":"2024-02-05T08:25:17.027549+0000","last_peered":"2024-02-05T08:25:17.027549+0000","last_clean":"2024-02-05T08:25:17.027549+0000","last_became_active":"2024-02-05T08:22:09.102371+0000","last_became_peered":"2024-02-05T08:22:09.102371+0000","last_unstale":"2024-02-05T08:25:17.027549+0000","last_undegraded":"2024-02-05T08:25:17.027549+0000","last_fullsized":"2024-02-05T08:25:17.027549+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:21:43.861060+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:21:43.861060+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174568+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":753,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.038404+0000","last_change":"2024-02-05T08:23:25.177775+0000","last_active":"2024-02-05T08:25:17.038404+0000","last_peered":"2024-02-05T08:25:17.038404+0000","last_clean":"2024-02-05T08:25:17.038404+0000","last_became_active":"2024-02-05T08:23:07.932548+0000","last_became_peered":"2024-02-05T08:23:07.932548+0000","last_unstale":"2024-02-05T08:25:17.038404+0000","last_undegraded":"2024-02-05T08:25:17.038404+0000","last_fullsized":"2024-02-05T08:25:17.038404+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:25.177650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:25.177650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887398+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":754,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.034900+0000","last_change":"2024-02-05T08:23:26.986054+0000","last_active":"2024-02-05T08:25:17.034900+0000","last_peered":"2024-02-05T08:25:17.034900+0000","last_clean":"2024-02-05T08:25:17.034900+0000","last_became_active":"2024-02-05T08:23:07.939630+0000","last_became_peered":"2024-02-05T08:23:07.939630+0000","last_unstale":"2024-02-05T08:25:17.034900+0000","last_undegraded":"2024-02-05T08:25:17.034900+0000","last_fullsized":"2024-02-05T08:25:17.034900+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:26.985938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:23:26.985938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:17.784188+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":761,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.039049+0000","last_change":"2024-02-05T08:23:29.002501+0000","last_active":"2024-02-05T08:25:17.039049+0000","last_peered":"2024-02-05T08:25:17.039049+0000","last_clean":"2024-02-05T08:25:17.039049+0000","last_became_active":"2024-02-05T08:23:07.932530+0000","last_became_peered":"2024-02-05T08:23:07.932530+0000","last_unstale":"2024-02-05T08:25:17.039049+0000","last_undegraded":"2024-02-05T08:25:17.039049+0000","last_fullsized":"2024-02-05T08:25:17.039049+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:29.002380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.002380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887281+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"397'9","reported_seq":789,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027942+0000","last_change":"2024-02-05T08:23:27.578462+0000","last_active":"2024-02-05T08:25:17.027942+0000","last_peered":"2024-02-05T08:25:17.027942+0000","last_clean":"2024-02-05T08:25:17.027942+0000","last_became_active":"2024-02-05T08:23:07.938087+0000","last_became_peered":"2024-02-05T08:23:07.938087+0000","last_unstale":"2024-02-05T08:25:17.027942+0000","last_undegraded":"2024-02-05T08:25:17.027942+0000","last_fullsized":"2024-02-05T08:25:17.027942+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-02-05T08:23:27.578370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:27.578370+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":762,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.028392+0000","last_change":"2024-02-05T08:24:41.278446+0000","last_active":"2024-02-05T08:25:17.028392+0000","last_peered":"2024-02-05T08:25:17.028392+0000","last_clean":"2024-02-05T08:25:17.028392+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:25:17.028392+0000","last_undegraded":"2024-02-05T08:25:17.028392+0000","last_fullsized":"2024-02-05T08:25:17.028392+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:41.278322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:24:41.278322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:06.296142+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":657,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.032118+0000","last_change":"2024-02-05T08:24:45.270707+0000","last_active":"2024-02-05T08:25:17.032118+0000","last_peered":"2024-02-05T08:25:17.032118+0000","last_clean":"2024-02-05T08:25:17.032118+0000","last_became_active":"2024-02-05T08:23:08.914258+0000","last_became_peered":"2024-02-05T08:23:08.914258+0000","last_unstale":"2024-02-05T08:25:17.032118+0000","last_undegraded":"2024-02-05T08:25:17.032118+0000","last_fullsized":"2024-02-05T08:25:17.032118+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:45.270592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_clean_scrub_stamp":"2024-02-05T08:24:45.270592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:57.698185+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":115,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027820+0000","last_change":"2024-02-05T08:23:08.921340+0000","last_active":"2024-02-05T08:25:17.027820+0000","last_peered":"2024-02-05T08:25:17.027820+0000","last_clean":"2024-02-05T08:25:17.027820+0000","last_became_active":"2024-02-05T08:23:08.920902+0000","last_became_peered":"2024-02-05T08:23:08.920902+0000","last_unstale":"2024-02-05T08:25:17.027820+0000","last_undegraded":"2024-02-05T08:25:17.027820+0000","last_fullsized":"2024-02-05T08:25:17.027820+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:04.290125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:04.290125+0000","last_clean_scrub_stamp":"2024-02-05T08:22:04.290125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"397'46","reported_seq":887,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.031863+0000","last_change":"2024-02-05T08:25:04.980660+0000","last_active":"2024-02-05T08:25:17.031863+0000","last_peered":"2024-02-05T08:25:17.031863+0000","last_clean":"2024-02-05T08:25:17.031863+0000","last_became_active":"2024-02-05T08:25:04.978627+0000","last_became_peered":"2024-02-05T08:25:04.978627+0000","last_unstale":"2024-02-05T08:25:17.031863+0000","last_undegraded":"2024-02-05T08:25:17.031863+0000","last_fullsized":"2024-02-05T08:25:17.031863+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:23:29.911720+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.911720+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":252,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.042671+0000","last_change":"2024-02-05T08:24:26.342293+0000","last_active":"2024-02-05T08:25:17.042671+0000","last_peered":"2024-02-05T08:25:17.042671+0000","last_clean":"2024-02-05T08:25:17.042671+0000","last_became_active":"2024-02-05T08:18:34.642494+0000","last_became_peered":"2024-02-05T08:18:34.642494+0000","last_unstale":"2024-02-05T08:25:17.042671+0000","last_undegraded":"2024-02-05T08:25:17.042671+0000","last_fullsized":"2024-02-05T08:25:17.042671+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:26.342171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_clean_scrub_stamp":"2024-02-05T08:24:26.342171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:41.547144+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":764,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027978+0000","last_change":"2024-02-05T08:22:57.230710+0000","last_active":"2024-02-05T08:25:17.027978+0000","last_peered":"2024-02-05T08:25:17.027978+0000","last_clean":"2024-02-05T08:25:17.027978+0000","last_became_active":"2024-02-05T08:18:34.638652+0000","last_became_peered":"2024-02-05T08:18:34.638652+0000","last_unstale":"2024-02-05T08:25:17.027978+0000","last_undegraded":"2024-02-05T08:25:17.027978+0000","last_fullsized":"2024-02-05T08:25:17.027978+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:57.230639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:22:57.230639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":753,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027535+0000","last_change":"2024-02-05T08:24:22.154106+0000","last_active":"2024-02-05T08:25:17.027535+0000","last_peered":"2024-02-05T08:25:17.027535+0000","last_clean":"2024-02-05T08:25:17.027535+0000","last_became_active":"2024-02-05T08:23:07.937958+0000","last_became_peered":"2024-02-05T08:23:07.937958+0000","last_unstale":"2024-02-05T08:25:17.027535+0000","last_undegraded":"2024-02-05T08:25:17.027535+0000","last_fullsized":"2024-02-05T08:25:17.027535+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:22.153964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:24:22.153964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:25.670574+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"397'29","reported_seq":265,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041463+0000","last_change":"2024-02-05T08:23:25.937497+0000","last_active":"2024-02-05T08:25:17.041463+0000","last_peered":"2024-02-05T08:25:17.041463+0000","last_clean":"2024-02-05T08:25:17.041463+0000","last_became_active":"2024-02-05T08:23:07.925083+0000","last_became_peered":"2024-02-05T08:23:07.925083+0000","last_unstale":"2024-02-05T08:25:17.041463+0000","last_undegraded":"2024-02-05T08:25:17.041463+0000","last_fullsized":"2024-02-05T08:25:17.041463+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'23","last_scrub_stamp":"2024-02-05T08:23:25.935815+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:23:25.935815+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":759,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.036109+0000","last_change":"2024-02-05T08:23:40.022017+0000","last_active":"2024-02-05T08:25:17.036109+0000","last_peered":"2024-02-05T08:25:17.036109+0000","last_clean":"2024-02-05T08:25:17.036109+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:25:17.036109+0000","last_undegraded":"2024-02-05T08:25:17.036109+0000","last_fullsized":"2024-02-05T08:25:17.036109+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:23:40.021926+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:23:40.021926+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:17.784499+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":665,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.038015+0000","last_change":"2024-02-05T08:24:27.484484+0000","last_active":"2024-02-05T08:25:17.038015+0000","last_peered":"2024-02-05T08:25:17.038015+0000","last_clean":"2024-02-05T08:25:17.038015+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:25:17.038015+0000","last_undegraded":"2024-02-05T08:25:17.038015+0000","last_fullsized":"2024-02-05T08:25:17.038015+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:24:27.484356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:24:27.484356+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:38.333363+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":686,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.030614+0000","last_change":"2024-02-05T08:24:24.293664+0000","last_active":"2024-02-05T08:25:17.030614+0000","last_peered":"2024-02-05T08:25:17.030614+0000","last_clean":"2024-02-05T08:25:17.030614+0000","last_became_active":"2024-02-05T08:23:07.940135+0000","last_became_peered":"2024-02-05T08:23:07.940135+0000","last_unstale":"2024-02-05T08:25:17.030614+0000","last_undegraded":"2024-02-05T08:25:17.030614+0000","last_fullsized":"2024-02-05T08:25:17.030614+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:24:24.293517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:24.293517+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:48.981702+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"397'30","reported_seq":604,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.036004+0000","last_change":"2024-02-05T08:24:38.874884+0000","last_active":"2024-02-05T08:25:17.036004+0000","last_peered":"2024-02-05T08:25:17.036004+0000","last_clean":"2024-02-05T08:25:17.036004+0000","last_became_active":"2024-02-05T08:23:08.920330+0000","last_became_peered":"2024-02-05T08:23:08.920330+0000","last_unstale":"2024-02-05T08:25:17.036004+0000","last_undegraded":"2024-02-05T08:25:17.036004+0000","last_fullsized":"2024-02-05T08:25:17.036004+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:24:38.872158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:24:38.872158+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:07.167994+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":612,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.032042+0000","last_change":"2024-02-05T08:23:33.200652+0000","last_active":"2024-02-05T08:25:17.032042+0000","last_peered":"2024-02-05T08:25:17.032042+0000","last_clean":"2024-02-05T08:25:17.032042+0000","last_became_active":"2024-02-05T08:23:07.941095+0000","last_became_peered":"2024-02-05T08:23:07.941095+0000","last_unstale":"2024-02-05T08:25:17.032042+0000","last_undegraded":"2024-02-05T08:25:17.032042+0000","last_fullsized":"2024-02-05T08:25:17.032042+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:33.200557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:33.200557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":123,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.036353+0000","last_change":"2024-02-05T08:24:23.992394+0000","last_active":"2024-02-05T08:25:17.036353+0000","last_peered":"2024-02-05T08:25:17.036353+0000","last_clean":"2024-02-05T08:25:17.036353+0000","last_became_active":"2024-02-05T08:23:08.911020+0000","last_became_peered":"2024-02-05T08:23:08.911020+0000","last_unstale":"2024-02-05T08:25:17.036353+0000","last_undegraded":"2024-02-05T08:25:17.036353+0000","last_fullsized":"2024-02-05T08:25:17.036353+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:23.992280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:19.107555+0000","last_clean_scrub_stamp":"2024-02-05T08:24:23.992280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:47.731830+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":256,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.035984+0000","last_change":"2024-02-05T08:23:08.927552+0000","last_active":"2024-02-05T08:25:17.035984+0000","last_peered":"2024-02-05T08:25:17.035984+0000","last_clean":"2024-02-05T08:25:17.035984+0000","last_became_active":"2024-02-05T08:23:08.927256+0000","last_became_peered":"2024-02-05T08:23:08.927256+0000","last_unstale":"2024-02-05T08:25:17.035984+0000","last_undegraded":"2024-02-05T08:25:17.035984+0000","last_fullsized":"2024-02-05T08:25:17.035984+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:46.026840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:22:46.026840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"397'27","reported_seq":535,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.037886+0000","last_change":"2024-02-05T08:24:30.419919+0000","last_active":"2024-02-05T08:25:17.037886+0000","last_peered":"2024-02-05T08:25:17.037886+0000","last_clean":"2024-02-05T08:25:17.037886+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:25:17.037886+0000","last_undegraded":"2024-02-05T08:25:17.037886+0000","last_fullsized":"2024-02-05T08:25:17.037886+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:24:30.419791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:24:30.419791+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:54.385059+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":741,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.035078+0000","last_change":"2024-02-05T08:24:16.984051+0000","last_active":"2024-02-05T08:25:17.035078+0000","last_peered":"2024-02-05T08:25:17.035078+0000","last_clean":"2024-02-05T08:25:17.035078+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:25:17.035078+0000","last_undegraded":"2024-02-05T08:25:17.035078+0000","last_fullsized":"2024-02-05T08:25:17.035078+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:16.983946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:14.159628+0000","last_clean_scrub_stamp":"2024-02-05T08:24:16.983946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:41.637025+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":493,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041940+0000","last_change":"2024-02-05T08:23:31.538869+0000","last_active":"2024-02-05T08:25:17.041940+0000","last_peered":"2024-02-05T08:25:17.041940+0000","last_clean":"2024-02-05T08:25:17.041940+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:25:17.041940+0000","last_undegraded":"2024-02-05T08:25:17.041940+0000","last_fullsized":"2024-02-05T08:25:17.041940+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:31.538744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:23:31.538744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.112823+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":640,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.039542+0000","last_change":"2024-02-05T08:24:31.862924+0000","last_active":"2024-02-05T08:25:17.039542+0000","last_peered":"2024-02-05T08:25:17.039542+0000","last_clean":"2024-02-05T08:25:17.039542+0000","last_became_active":"2024-02-05T08:22:09.104064+0000","last_became_peered":"2024-02-05T08:22:09.104064+0000","last_unstale":"2024-02-05T08:25:17.039542+0000","last_undegraded":"2024-02-05T08:25:17.039542+0000","last_fullsized":"2024-02-05T08:25:17.039542+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:31.862841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:31.862841+0000","last_clean_scrub_stamp":"2024-02-05T08:24:31.862841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:48.896461+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"397'12","reported_seq":264,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.042485+0000","last_change":"2024-02-05T08:23:31.939827+0000","last_active":"2024-02-05T08:25:17.042485+0000","last_peered":"2024-02-05T08:25:17.042485+0000","last_clean":"2024-02-05T08:25:17.042485+0000","last_became_active":"2024-02-05T08:23:07.937911+0000","last_became_peered":"2024-02-05T08:23:07.937911+0000","last_unstale":"2024-02-05T08:25:17.042485+0000","last_undegraded":"2024-02-05T08:25:17.042485+0000","last_fullsized":"2024-02-05T08:25:17.042485+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:23:31.939578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:31.939578+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":257,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040945+0000","last_change":"2024-02-05T08:22:47.060160+0000","last_active":"2024-02-05T08:25:17.040945+0000","last_peered":"2024-02-05T08:25:17.040945+0000","last_clean":"2024-02-05T08:25:17.040945+0000","last_became_active":"2024-02-05T08:22:09.095805+0000","last_became_peered":"2024-02-05T08:22:09.095805+0000","last_unstale":"2024-02-05T08:25:17.040945+0000","last_undegraded":"2024-02-05T08:25:17.040945+0000","last_fullsized":"2024-02-05T08:25:17.040945+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:47.060036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:26.891040+0000","last_clean_scrub_stamp":"2024-02-05T08:22:47.060036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":773,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.893400+0000","last_change":"2024-02-05T08:25:17.893419+0000","last_active":"2024-02-05T08:25:17.893400+0000","last_peered":"2024-02-05T08:25:17.893400+0000","last_clean":"2024-02-05T08:25:17.893400+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:25:17.893400+0000","last_undegraded":"2024-02-05T08:25:17.893400+0000","last_fullsized":"2024-02-05T08:25:17.893400+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:17.893349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:54.156080+0000","last_clean_scrub_stamp":"2024-02-05T08:25:17.893349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:30.835433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":761,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.038752+0000","last_change":"2024-02-05T08:23:28.001765+0000","last_active":"2024-02-05T08:25:17.038752+0000","last_peered":"2024-02-05T08:25:17.038752+0000","last_clean":"2024-02-05T08:25:17.038752+0000","last_became_active":"2024-02-05T08:23:07.932374+0000","last_became_peered":"2024-02-05T08:23:07.932374+0000","last_unstale":"2024-02-05T08:25:17.038752+0000","last_undegraded":"2024-02-05T08:25:17.038752+0000","last_fullsized":"2024-02-05T08:25:17.038752+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:28.001523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:23:28.001523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"397'32","reported_seq":275,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041912+0000","last_change":"2024-02-05T08:24:28.130743+0000","last_active":"2024-02-05T08:25:17.041912+0000","last_peered":"2024-02-05T08:25:17.041912+0000","last_clean":"2024-02-05T08:25:17.041912+0000","last_became_active":"2024-02-05T08:18:34.631912+0000","last_became_peered":"2024-02-05T08:18:34.631912+0000","last_unstale":"2024-02-05T08:25:17.041912+0000","last_undegraded":"2024-02-05T08:25:17.041912+0000","last_fullsized":"2024-02-05T08:25:17.041912+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:24:28.130616+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:24:28.130616+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:51.228990+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":695,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040209+0000","last_change":"2024-02-05T08:24:37.823706+0000","last_active":"2024-02-05T08:25:17.040209+0000","last_peered":"2024-02-05T08:25:17.040209+0000","last_clean":"2024-02-05T08:25:17.040209+0000","last_became_active":"2024-02-05T08:23:08.918104+0000","last_became_peered":"2024-02-05T08:23:08.918104+0000","last_unstale":"2024-02-05T08:25:17.040209+0000","last_undegraded":"2024-02-05T08:25:17.040209+0000","last_fullsized":"2024-02-05T08:25:17.040209+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:37.823577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:24:37.823577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:53.743111+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":256,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040860+0000","last_change":"2024-02-05T08:23:08.918137+0000","last_active":"2024-02-05T08:25:17.040860+0000","last_peered":"2024-02-05T08:25:17.040860+0000","last_clean":"2024-02-05T08:25:17.040860+0000","last_became_active":"2024-02-05T08:23:08.917587+0000","last_became_peered":"2024-02-05T08:23:08.917587+0000","last_unstale":"2024-02-05T08:25:17.040860+0000","last_undegraded":"2024-02-05T08:25:17.040860+0000","last_fullsized":"2024-02-05T08:25:17.040860+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:44.050117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:44.050117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":773,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.035742+0000","last_change":"2024-02-05T08:24:33.916510+0000","last_active":"2024-02-05T08:25:17.035742+0000","last_peered":"2024-02-05T08:25:17.035742+0000","last_clean":"2024-02-05T08:25:17.035742+0000","last_became_active":"2024-02-05T08:18:34.642387+0000","last_became_peered":"2024-02-05T08:18:34.642387+0000","last_unstale":"2024-02-05T08:25:17.035742+0000","last_undegraded":"2024-02-05T08:25:17.035742+0000","last_fullsized":"2024-02-05T08:25:17.035742+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:33.916402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:33.916402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:59.834709+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"397'25","reported_seq":437,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.038205+0000","last_change":"2024-02-05T08:24:28.876345+0000","last_active":"2024-02-05T08:25:17.038205+0000","last_peered":"2024-02-05T08:25:17.038205+0000","last_clean":"2024-02-05T08:25:17.038205+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:25:17.038205+0000","last_undegraded":"2024-02-05T08:25:17.038205+0000","last_fullsized":"2024-02-05T08:25:17.038205+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'19","last_scrub_stamp":"2024-02-05T08:24:28.876211+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_clean_scrub_stamp":"2024-02-05T08:24:28.876211+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:29.962110+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":265,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041780+0000","last_change":"2024-02-05T08:23:00.825681+0000","last_active":"2024-02-05T08:25:17.041780+0000","last_peered":"2024-02-05T08:25:17.041780+0000","last_clean":"2024-02-05T08:25:17.041780+0000","last_became_active":"2024-02-05T08:22:09.096314+0000","last_became_peered":"2024-02-05T08:22:09.096314+0000","last_unstale":"2024-02-05T08:25:17.041780+0000","last_undegraded":"2024-02-05T08:25:17.041780+0000","last_fullsized":"2024-02-05T08:25:17.041780+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:00.825605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:23:00.825605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":663,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.031326+0000","last_change":"2024-02-05T08:24:25.279757+0000","last_active":"2024-02-05T08:25:17.031326+0000","last_peered":"2024-02-05T08:25:17.031326+0000","last_clean":"2024-02-05T08:25:17.031326+0000","last_became_active":"2024-02-05T08:23:07.936401+0000","last_became_peered":"2024-02-05T08:23:07.936401+0000","last_unstale":"2024-02-05T08:25:17.031326+0000","last_undegraded":"2024-02-05T08:25:17.031326+0000","last_fullsized":"2024-02-05T08:25:17.031326+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:25.279647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:24:25.279647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:41.285794+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":772,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027838+0000","last_change":"2024-02-05T08:21:44.871777+0000","last_active":"2024-02-05T08:25:17.027838+0000","last_peered":"2024-02-05T08:25:17.027838+0000","last_clean":"2024-02-05T08:25:17.027838+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:25:17.027838+0000","last_undegraded":"2024-02-05T08:25:17.027838+0000","last_fullsized":"2024-02-05T08:25:17.027838+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:44.871668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:21:44.871668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174536+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"397'32","reported_seq":841,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.028041+0000","last_change":"2024-02-05T08:24:34.133027+0000","last_active":"2024-02-05T08:25:17.028041+0000","last_peered":"2024-02-05T08:25:17.028041+0000","last_clean":"2024-02-05T08:25:17.028041+0000","last_became_active":"2024-02-05T08:23:07.938523+0000","last_became_peered":"2024-02-05T08:23:07.938523+0000","last_unstale":"2024-02-05T08:25:17.028041+0000","last_undegraded":"2024-02-05T08:25:17.028041+0000","last_fullsized":"2024-02-05T08:25:17.028041+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:24:34.132899+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:24:34.132899+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:59.813089+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":612,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.036132+0000","last_change":"2024-02-05T08:23:33.061246+0000","last_active":"2024-02-05T08:25:17.036132+0000","last_peered":"2024-02-05T08:25:17.036132+0000","last_clean":"2024-02-05T08:25:17.036132+0000","last_became_active":"2024-02-05T08:22:09.101652+0000","last_became_peered":"2024-02-05T08:22:09.101652+0000","last_unstale":"2024-02-05T08:25:17.036132+0000","last_undegraded":"2024-02-05T08:25:17.036132+0000","last_fullsized":"2024-02-05T08:25:17.036132+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:33.061104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:33.061104+0000","last_clean_scrub_stamp":"2024-02-05T08:23:33.061104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:17.784270+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"391'14","reported_seq":1030,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040549+0000","last_change":"2024-02-05T08:24:33.874083+0000","last_active":"2024-02-05T08:25:17.040549+0000","last_peered":"2024-02-05T08:25:17.040549+0000","last_clean":"2024-02-05T08:25:17.040549+0000","last_became_active":"2024-02-05T08:23:08.917858+0000","last_became_peered":"2024-02-05T08:23:08.917858+0000","last_unstale":"2024-02-05T08:25:17.040549+0000","last_undegraded":"2024-02-05T08:25:17.040549+0000","last_fullsized":"2024-02-05T08:25:17.040549+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'14","last_scrub_stamp":"2024-02-05T08:24:33.873828+0000","last_deep_scrub":"391'14","last_deep_scrub_stamp":"2024-02-05T08:24:33.873828+0000","last_clean_scrub_stamp":"2024-02-05T08:24:33.873828+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:54.784202+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":761,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.035303+0000","last_change":"2024-02-05T08:24:19.912012+0000","last_active":"2024-02-05T08:25:17.035303+0000","last_peered":"2024-02-05T08:25:17.035303+0000","last_clean":"2024-02-05T08:25:17.035303+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:25:17.035303+0000","last_undegraded":"2024-02-05T08:25:17.035303+0000","last_fullsized":"2024-02-05T08:25:17.035303+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:19.911782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:19.911782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:28.277541+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"397'35","reported_seq":279,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041316+0000","last_change":"2024-02-05T08:24:19.039429+0000","last_active":"2024-02-05T08:25:17.041316+0000","last_peered":"2024-02-05T08:25:17.041316+0000","last_clean":"2024-02-05T08:25:17.041316+0000","last_became_active":"2024-02-05T08:23:07.928966+0000","last_became_peered":"2024-02-05T08:23:07.928966+0000","last_unstale":"2024-02-05T08:25:17.041316+0000","last_undegraded":"2024-02-05T08:25:17.041316+0000","last_fullsized":"2024-02-05T08:25:17.041316+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-02-05T08:24:19.039293+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:24:19.039293+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:33.178419+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":743,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.028335+0000","last_change":"2024-02-05T08:24:37.328751+0000","last_active":"2024-02-05T08:25:17.028335+0000","last_peered":"2024-02-05T08:25:17.028335+0000","last_clean":"2024-02-05T08:25:17.028335+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:25:17.028335+0000","last_undegraded":"2024-02-05T08:25:17.028335+0000","last_fullsized":"2024-02-05T08:25:17.028335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:37.328629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:28.519149+0000","last_clean_scrub_stamp":"2024-02-05T08:24:37.328629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:05.469206+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"295'10","reported_seq":329,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041535+0000","last_change":"2024-02-05T08:22:03.724091+0000","last_active":"2024-02-05T08:25:17.041535+0000","last_peered":"2024-02-05T08:25:17.041535+0000","last_clean":"2024-02-05T08:25:17.041535+0000","last_became_active":"2024-02-05T08:18:34.644517+0000","last_became_peered":"2024-02-05T08:18:34.644517+0000","last_unstale":"2024-02-05T08:25:17.041535+0000","last_undegraded":"2024-02-05T08:25:17.041535+0000","last_fullsized":"2024-02-05T08:25:17.041535+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'10","last_scrub_stamp":"2024-02-05T08:22:03.723970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:03.723970+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":636,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.034796+0000","last_change":"2024-02-05T08:23:44.038746+0000","last_active":"2024-02-05T08:25:17.034796+0000","last_peered":"2024-02-05T08:25:17.034796+0000","last_clean":"2024-02-05T08:25:17.034796+0000","last_became_active":"2024-02-05T08:23:08.920193+0000","last_became_peered":"2024-02-05T08:23:08.920193+0000","last_unstale":"2024-02-05T08:25:17.034796+0000","last_undegraded":"2024-02-05T08:25:17.034796+0000","last_fullsized":"2024-02-05T08:25:17.034796+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:44.038564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:23:44.038564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:17.784322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"397'21","reported_seq":830,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040714+0000","last_change":"2024-02-05T08:23:23.049946+0000","last_active":"2024-02-05T08:25:17.040714+0000","last_peered":"2024-02-05T08:25:17.040714+0000","last_clean":"2024-02-05T08:25:17.040714+0000","last_became_active":"2024-02-05T08:18:34.640741+0000","last_became_peered":"2024-02-05T08:18:34.640741+0000","last_unstale":"2024-02-05T08:25:17.040714+0000","last_undegraded":"2024-02-05T08:25:17.040714+0000","last_fullsized":"2024-02-05T08:25:17.040714+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:23:23.049797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:23.049797+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887257+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":772,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.029935+0000","last_change":"2024-02-05T08:24:16.703350+0000","last_active":"2024-02-05T08:25:17.029935+0000","last_peered":"2024-02-05T08:25:17.029935+0000","last_clean":"2024-02-05T08:25:17.029935+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:25:17.029935+0000","last_undegraded":"2024-02-05T08:25:17.029935+0000","last_fullsized":"2024-02-05T08:25:17.029935+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:16.702946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:16.702946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:32.608708+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":252,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.042582+0000","last_change":"2024-02-05T08:23:42.025667+0000","last_active":"2024-02-05T08:25:17.042582+0000","last_peered":"2024-02-05T08:25:17.042582+0000","last_clean":"2024-02-05T08:25:17.042582+0000","last_became_active":"2024-02-05T08:23:07.928375+0000","last_became_peered":"2024-02-05T08:23:07.928375+0000","last_unstale":"2024-02-05T08:25:17.042582+0000","last_undegraded":"2024-02-05T08:25:17.042582+0000","last_fullsized":"2024-02-05T08:25:17.042582+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:42.025471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:42.025471+0000","last_clean_scrub_stamp":"2024-02-05T08:23:42.025471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":765,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.034529+0000","last_change":"2024-02-05T08:22:05.129978+0000","last_active":"2024-02-05T08:25:17.034529+0000","last_peered":"2024-02-05T08:25:17.034529+0000","last_clean":"2024-02-05T08:25:17.034529+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:25:17.034529+0000","last_undegraded":"2024-02-05T08:25:17.034529+0000","last_fullsized":"2024-02-05T08:25:17.034529+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:05.129584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:22:05.129584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:17.784076+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"397'25","reported_seq":686,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041631+0000","last_change":"2024-02-05T08:24:45.102697+0000","last_active":"2024-02-05T08:25:17.041631+0000","last_peered":"2024-02-05T08:25:17.041631+0000","last_clean":"2024-02-05T08:25:17.041631+0000","last_became_active":"2024-02-05T08:22:09.110422+0000","last_became_peered":"2024-02-05T08:22:09.110422+0000","last_unstale":"2024-02-05T08:25:17.041631+0000","last_undegraded":"2024-02-05T08:25:17.041631+0000","last_fullsized":"2024-02-05T08:25:17.041631+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-05T08:24:45.102570+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:24:45.102570+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:51.311290+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":761,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.035658+0000","last_change":"2024-02-05T08:24:45.975121+0000","last_active":"2024-02-05T08:25:17.035658+0000","last_peered":"2024-02-05T08:25:17.035658+0000","last_clean":"2024-02-05T08:25:17.035658+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:25:17.035658+0000","last_undegraded":"2024-02-05T08:25:17.035658+0000","last_fullsized":"2024-02-05T08:25:17.035658+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:45.974690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:45.974690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:53.341271+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":765,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.039344+0000","last_change":"2024-02-05T08:23:35.889657+0000","last_active":"2024-02-05T08:25:17.039344+0000","last_peered":"2024-02-05T08:25:17.039344+0000","last_clean":"2024-02-05T08:25:17.039344+0000","last_became_active":"2024-02-05T08:23:07.932741+0000","last_became_peered":"2024-02-05T08:23:07.932741+0000","last_unstale":"2024-02-05T08:25:17.039344+0000","last_undegraded":"2024-02-05T08:25:17.039344+0000","last_fullsized":"2024-02-05T08:25:17.039344+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:35.889538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:23:35.889538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887423+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":504,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040860+0000","last_change":"2024-02-05T08:24:19.508139+0000","last_active":"2024-02-05T08:25:17.040860+0000","last_peered":"2024-02-05T08:25:17.040860+0000","last_clean":"2024-02-05T08:25:17.040860+0000","last_became_active":"2024-02-05T08:22:09.097843+0000","last_became_peered":"2024-02-05T08:22:09.097843+0000","last_unstale":"2024-02-05T08:25:17.040860+0000","last_undegraded":"2024-02-05T08:25:17.040860+0000","last_fullsized":"2024-02-05T08:25:17.040860+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:19.508059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:19.508059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:26.904195+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"397'21","reported_seq":261,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.042444+0000","last_change":"2024-02-05T08:23:26.922363+0000","last_active":"2024-02-05T08:25:17.042444+0000","last_peered":"2024-02-05T08:25:17.042444+0000","last_clean":"2024-02-05T08:25:17.042444+0000","last_became_active":"2024-02-05T08:23:07.927922+0000","last_became_peered":"2024-02-05T08:23:07.927922+0000","last_unstale":"2024-02-05T08:25:17.042444+0000","last_undegraded":"2024-02-05T08:25:17.042444+0000","last_fullsized":"2024-02-05T08:25:17.042444+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:23:26.922227+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:23:26.922227+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"402'10868","reported_seq":10948,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:18.623044+0000","last_change":"2024-02-05T08:23:08.913493+0000","last_active":"2024-02-05T08:25:18.623044+0000","last_peered":"2024-02-05T08:25:18.623044+0000","last_clean":"2024-02-05T08:25:18.623044+0000","last_became_active":"2024-02-05T08:23:08.913227+0000","last_became_peered":"2024-02-05T08:23:08.913227+0000","last_unstale":"2024-02-05T08:25:18.623044+0000","last_undegraded":"2024-02-05T08:25:18.623044+0000","last_fullsized":"2024-02-05T08:25:18.623044+0000","mapping_epoch":367,"log_start":"389'6200","ondisk_log_start":"389'6200","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4668,"log_dups_size":0,"ondisk_log_size":4668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396361728,"num_objects":6048,"num_object_clones":0,"num_object_copies":18144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10868,"num_write_kb":541312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":775,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.039039+0000","last_change":"2024-02-05T08:24:25.887052+0000","last_active":"2024-02-05T08:25:17.039039+0000","last_peered":"2024-02-05T08:25:17.039039+0000","last_clean":"2024-02-05T08:25:17.039039+0000","last_became_active":"2024-02-05T08:22:09.100322+0000","last_became_peered":"2024-02-05T08:22:09.100322+0000","last_unstale":"2024-02-05T08:25:17.039039+0000","last_undegraded":"2024-02-05T08:25:17.039039+0000","last_fullsized":"2024-02-05T08:25:17.039039+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:25.886939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:25.886939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:51.587172+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":790,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.036699+0000","last_change":"2024-02-05T08:23:43.066628+0000","last_active":"2024-02-05T08:25:17.036699+0000","last_peered":"2024-02-05T08:25:17.036699+0000","last_clean":"2024-02-05T08:25:17.036699+0000","last_became_active":"2024-02-05T08:23:08.920320+0000","last_became_peered":"2024-02-05T08:23:08.920320+0000","last_unstale":"2024-02-05T08:25:17.036699+0000","last_undegraded":"2024-02-05T08:25:17.036699+0000","last_fullsized":"2024-02-05T08:25:17.036699+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:23:43.066380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:23:43.066380+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:17.784394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"397'46","reported_seq":869,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040176+0000","last_change":"2024-02-05T08:24:24.865550+0000","last_active":"2024-02-05T08:25:17.040176+0000","last_peered":"2024-02-05T08:25:17.040176+0000","last_clean":"2024-02-05T08:25:17.040176+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:25:17.040176+0000","last_undegraded":"2024-02-05T08:25:17.040176+0000","last_fullsized":"2024-02-05T08:25:17.040176+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'34","last_scrub_stamp":"2024-02-05T08:24:24.865290+0000","last_deep_scrub":"262'34","last_deep_scrub_stamp":"2024-02-05T08:19:34.364738+0000","last_clean_scrub_stamp":"2024-02-05T08:24:24.865290+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:50.095788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":1031,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.034782+0000","last_change":"2024-02-05T08:24:26.044610+0000","last_active":"2024-02-05T08:25:17.034782+0000","last_peered":"2024-02-05T08:25:17.034782+0000","last_clean":"2024-02-05T08:25:17.034782+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:25:17.034782+0000","last_undegraded":"2024-02-05T08:25:17.034782+0000","last_fullsized":"2024-02-05T08:25:17.034782+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:24:26.044511+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_clean_scrub_stamp":"2024-02-05T08:24:26.044511+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:33.058094+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"402'10973","reported_seq":11147,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:21.038257+0000","last_change":"2024-02-05T08:25:05.035022+0000","last_active":"2024-02-05T08:25:21.038257+0000","last_peered":"2024-02-05T08:25:21.038257+0000","last_clean":"2024-02-05T08:25:21.038257+0000","last_became_active":"2024-02-05T08:25:04.989417+0000","last_became_peered":"2024-02-05T08:25:04.989417+0000","last_unstale":"2024-02-05T08:25:21.038257+0000","last_undegraded":"2024-02-05T08:25:21.038257+0000","last_fullsized":"2024-02-05T08:25:21.038257+0000","mapping_epoch":400,"log_start":"389'6844","ondisk_log_start":"389'6844","created":367,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'7188","last_scrub_stamp":"2024-02-05T08:24:34.355597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:24:34.355597+0000","objects_scrubbed":7095,"log_size":4129,"log_dups_size":0,"ondisk_log_size":4129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:58.347800+0000","scrub_duration":2540,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376373248,"num_objects":5743,"num_object_clones":0,"num_object_copies":17229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10973,"num_write_kb":534912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":244,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041322+0000","last_change":"2024-02-05T08:23:29.086552+0000","last_active":"2024-02-05T08:25:17.041322+0000","last_peered":"2024-02-05T08:25:17.041322+0000","last_clean":"2024-02-05T08:25:17.041322+0000","last_became_active":"2024-02-05T08:23:07.929942+0000","last_became_peered":"2024-02-05T08:23:07.929942+0000","last_unstale":"2024-02-05T08:25:17.041322+0000","last_undegraded":"2024-02-05T08:25:17.041322+0000","last_fullsized":"2024-02-05T08:25:17.041322+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:29.086293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.086293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":763,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.036270+0000","last_change":"2024-02-05T08:23:30.010159+0000","last_active":"2024-02-05T08:25:17.036270+0000","last_peered":"2024-02-05T08:25:17.036270+0000","last_clean":"2024-02-05T08:25:17.036270+0000","last_became_active":"2024-02-05T08:23:07.937863+0000","last_became_peered":"2024-02-05T08:23:07.937863+0000","last_unstale":"2024-02-05T08:25:17.036270+0000","last_undegraded":"2024-02-05T08:25:17.036270+0000","last_fullsized":"2024-02-05T08:25:17.036270+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:30.010028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:23:30.010028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:17.784442+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"397'24","reported_seq":827,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027319+0000","last_change":"2024-02-05T08:23:30.492935+0000","last_active":"2024-02-05T08:25:17.027319+0000","last_peered":"2024-02-05T08:25:17.027319+0000","last_clean":"2024-02-05T08:25:17.027319+0000","last_became_active":"2024-02-05T08:23:07.929900+0000","last_became_peered":"2024-02-05T08:23:07.929900+0000","last_unstale":"2024-02-05T08:25:17.027319+0000","last_undegraded":"2024-02-05T08:25:17.027319+0000","last_fullsized":"2024-02-05T08:25:17.027319+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-02-05T08:23:30.492582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:30.492582+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174963+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":702,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.030395+0000","last_change":"2024-02-05T08:25:04.981442+0000","last_active":"2024-02-05T08:25:17.030395+0000","last_peered":"2024-02-05T08:25:17.030395+0000","last_clean":"2024-02-05T08:25:17.030395+0000","last_became_active":"2024-02-05T08:25:04.980723+0000","last_became_peered":"2024-02-05T08:25:04.980723+0000","last_unstale":"2024-02-05T08:25:17.030395+0000","last_undegraded":"2024-02-05T08:25:17.030395+0000","last_fullsized":"2024-02-05T08:25:17.030395+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:18.736973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:24:18.736973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:39.985515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.5","version":"402'10966","reported_seq":11047,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:18.504757+0000","last_change":"2024-02-05T08:23:08.911955+0000","last_active":"2024-02-05T08:25:18.504757+0000","last_peered":"2024-02-05T08:25:18.504757+0000","last_clean":"2024-02-05T08:25:18.504757+0000","last_became_active":"2024-02-05T08:23:08.911591+0000","last_became_peered":"2024-02-05T08:23:08.911591+0000","last_unstale":"2024-02-05T08:25:18.504757+0000","last_undegraded":"2024-02-05T08:25:18.504757+0000","last_fullsized":"2024-02-05T08:25:18.504757+0000","mapping_epoch":367,"log_start":"389'6400","ondisk_log_start":"389'6400","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4566,"log_dups_size":0,"ondisk_log_size":4566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389808128,"num_objects":5948,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10966,"num_write_kb":541248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":762,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.034306+0000","last_change":"2024-02-05T08:24:27.030998+0000","last_active":"2024-02-05T08:25:17.034306+0000","last_peered":"2024-02-05T08:25:17.034306+0000","last_clean":"2024-02-05T08:25:17.034306+0000","last_became_active":"2024-02-05T08:18:34.642239+0000","last_became_peered":"2024-02-05T08:18:34.642239+0000","last_unstale":"2024-02-05T08:25:17.034306+0000","last_undegraded":"2024-02-05T08:25:17.034306+0000","last_fullsized":"2024-02-05T08:25:17.034306+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:27.030762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:27.030762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:39.215089+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":775,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027273+0000","last_change":"2024-02-05T08:24:20.763701+0000","last_active":"2024-02-05T08:25:17.027273+0000","last_peered":"2024-02-05T08:25:17.027273+0000","last_clean":"2024-02-05T08:25:17.027273+0000","last_became_active":"2024-02-05T08:22:09.105439+0000","last_became_peered":"2024-02-05T08:22:09.105439+0000","last_unstale":"2024-02-05T08:25:17.027273+0000","last_undegraded":"2024-02-05T08:25:17.027273+0000","last_fullsized":"2024-02-05T08:25:17.027273+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:20.763558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:24:20.763558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:34.872761+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":766,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.039833+0000","last_change":"2024-02-05T08:21:40.182453+0000","last_active":"2024-02-05T08:25:17.039833+0000","last_peered":"2024-02-05T08:25:17.039833+0000","last_clean":"2024-02-05T08:25:17.039833+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:25:17.039833+0000","last_undegraded":"2024-02-05T08:25:17.039833+0000","last_fullsized":"2024-02-05T08:25:17.039833+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:40.182334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_clean_scrub_stamp":"2024-02-05T08:21:40.182334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:17.784022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"397'51","reported_seq":566,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040946+0000","last_change":"2024-02-05T08:23:29.598019+0000","last_active":"2024-02-05T08:25:17.040946+0000","last_peered":"2024-02-05T08:25:17.040946+0000","last_clean":"2024-02-05T08:25:17.040946+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:25:17.040946+0000","last_undegraded":"2024-02-05T08:25:17.040946+0000","last_fullsized":"2024-02-05T08:25:17.040946+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'35","last_scrub_stamp":"2024-02-05T08:23:29.597891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.597891+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.112767+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"402'10844","reported_seq":10926,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:18.488801+0000","last_change":"2024-02-05T08:23:08.912293+0000","last_active":"2024-02-05T08:25:18.488801+0000","last_peered":"2024-02-05T08:25:18.488801+0000","last_clean":"2024-02-05T08:25:18.488801+0000","last_became_active":"2024-02-05T08:23:08.911987+0000","last_became_peered":"2024-02-05T08:23:08.911987+0000","last_unstale":"2024-02-05T08:25:18.488801+0000","last_undegraded":"2024-02-05T08:25:18.488801+0000","last_fullsized":"2024-02-05T08:25:18.488801+0000","mapping_epoch":367,"log_start":"389'6300","ondisk_log_start":"389'6300","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4544,"log_dups_size":0,"ondisk_log_size":4544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406585344,"num_objects":6204,"num_object_clones":0,"num_object_copies":18612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10844,"num_write_kb":545536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":119,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.034323+0000","last_change":"2024-02-05T08:24:25.220957+0000","last_active":"2024-02-05T08:25:17.034323+0000","last_peered":"2024-02-05T08:25:17.034323+0000","last_clean":"2024-02-05T08:25:17.034323+0000","last_became_active":"2024-02-05T08:22:09.100986+0000","last_became_peered":"2024-02-05T08:22:09.100986+0000","last_unstale":"2024-02-05T08:25:17.034323+0000","last_undegraded":"2024-02-05T08:25:17.034323+0000","last_fullsized":"2024-02-05T08:25:17.034323+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:25.220831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_clean_scrub_stamp":"2024-02-05T08:24:25.220831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:37.504652+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":537,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027685+0000","last_change":"2024-02-05T08:25:04.987081+0000","last_active":"2024-02-05T08:25:17.027685+0000","last_peered":"2024-02-05T08:25:17.027685+0000","last_clean":"2024-02-05T08:25:17.027685+0000","last_became_active":"2024-02-05T08:25:04.986710+0000","last_became_peered":"2024-02-05T08:25:04.986710+0000","last_unstale":"2024-02-05T08:25:17.027685+0000","last_undegraded":"2024-02-05T08:25:17.027685+0000","last_fullsized":"2024-02-05T08:25:17.027685+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:33.808037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:24:33.808037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:37.597137+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":785,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.039391+0000","last_change":"2024-02-05T08:23:01.230170+0000","last_active":"2024-02-05T08:25:17.039391+0000","last_peered":"2024-02-05T08:25:17.039391+0000","last_clean":"2024-02-05T08:25:17.039391+0000","last_became_active":"2024-02-05T08:18:34.633214+0000","last_became_peered":"2024-02-05T08:18:34.633214+0000","last_unstale":"2024-02-05T08:25:17.039391+0000","last_undegraded":"2024-02-05T08:25:17.039391+0000","last_fullsized":"2024-02-05T08:25:17.039391+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:01.230038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:01.230038+0000","last_clean_scrub_stamp":"2024-02-05T08:23:01.230038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"397'25","reported_seq":558,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.032044+0000","last_change":"2024-02-05T08:24:36.258882+0000","last_active":"2024-02-05T08:25:17.032044+0000","last_peered":"2024-02-05T08:25:17.032044+0000","last_clean":"2024-02-05T08:25:17.032044+0000","last_became_active":"2024-02-05T08:23:08.920885+0000","last_became_peered":"2024-02-05T08:23:08.920885+0000","last_unstale":"2024-02-05T08:25:17.032044+0000","last_undegraded":"2024-02-05T08:25:17.032044+0000","last_fullsized":"2024-02-05T08:25:17.032044+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'19","last_scrub_stamp":"2024-02-05T08:24:36.258646+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:24:36.258646+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:53.321334+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.b","version":"402'10606","reported_seq":10687,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:18.422244+0000","last_change":"2024-02-05T08:23:08.917492+0000","last_active":"2024-02-05T08:25:18.422244+0000","last_peered":"2024-02-05T08:25:18.422244+0000","last_clean":"2024-02-05T08:25:18.422244+0000","last_became_active":"2024-02-05T08:23:08.917212+0000","last_became_peered":"2024-02-05T08:23:08.917212+0000","last_unstale":"2024-02-05T08:25:18.422244+0000","last_undegraded":"2024-02-05T08:25:18.422244+0000","last_fullsized":"2024-02-05T08:25:18.422244+0000","mapping_epoch":367,"log_start":"389'6600","ondisk_log_start":"389'6600","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4006,"log_dups_size":0,"ondisk_log_size":4006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397279232,"num_objects":6062,"num_object_clones":0,"num_object_copies":18186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10606,"num_write_kb":533376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'130","reported_seq":545,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.038244+0000","last_change":"2024-02-05T08:23:43.752206+0000","last_active":"2024-02-05T08:25:17.038244+0000","last_peered":"2024-02-05T08:25:17.038244+0000","last_clean":"2024-02-05T08:25:17.038244+0000","last_became_active":"2024-02-05T08:22:12.146682+0000","last_became_peered":"2024-02-05T08:22:12.146682+0000","last_unstale":"2024-02-05T08:25:17.038244+0000","last_undegraded":"2024-02-05T08:25:17.038244+0000","last_fullsized":"2024-02-05T08:25:17.038244+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:43.752076+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:23:43.752076+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887505+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":472,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.028205+0000","last_change":"2024-02-05T08:23:26.660184+0000","last_active":"2024-02-05T08:25:17.028205+0000","last_peered":"2024-02-05T08:25:17.028205+0000","last_clean":"2024-02-05T08:25:17.028205+0000","last_became_active":"2024-02-05T08:23:07.936660+0000","last_became_peered":"2024-02-05T08:23:07.936660+0000","last_unstale":"2024-02-05T08:25:17.028205+0000","last_undegraded":"2024-02-05T08:25:17.028205+0000","last_fullsized":"2024-02-05T08:25:17.028205+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:26.660108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:26.660108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174827+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":794,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.038752+0000","last_change":"2024-02-05T08:23:27.050401+0000","last_active":"2024-02-05T08:25:17.038752+0000","last_peered":"2024-02-05T08:25:17.038752+0000","last_clean":"2024-02-05T08:25:17.038752+0000","last_became_active":"2024-02-05T08:23:07.937565+0000","last_became_peered":"2024-02-05T08:23:07.937565+0000","last_unstale":"2024-02-05T08:25:17.038752+0000","last_undegraded":"2024-02-05T08:25:17.038752+0000","last_fullsized":"2024-02-05T08:25:17.038752+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:23:27.050262+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:23:27.050262+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887348+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":783,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027672+0000","last_change":"2024-02-05T08:24:40.136041+0000","last_active":"2024-02-05T08:25:17.027672+0000","last_peered":"2024-02-05T08:25:17.027672+0000","last_clean":"2024-02-05T08:25:17.027672+0000","last_became_active":"2024-02-05T08:23:08.920421+0000","last_became_peered":"2024-02-05T08:23:08.920421+0000","last_unstale":"2024-02-05T08:25:17.027672+0000","last_undegraded":"2024-02-05T08:25:17.027672+0000","last_fullsized":"2024-02-05T08:25:17.027672+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:40.135374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:40.135374+0000","last_clean_scrub_stamp":"2024-02-05T08:24:40.135374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:00.424011+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"397'66","reported_seq":1006,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.031457+0000","last_change":"2024-02-05T08:24:18.160994+0000","last_active":"2024-02-05T08:25:17.031457+0000","last_peered":"2024-02-05T08:25:17.031457+0000","last_clean":"2024-02-05T08:25:17.031457+0000","last_became_active":"2024-02-05T08:23:07.932513+0000","last_became_peered":"2024-02-05T08:23:07.932513+0000","last_unstale":"2024-02-05T08:25:17.031457+0000","last_undegraded":"2024-02-05T08:25:17.031457+0000","last_fullsized":"2024-02-05T08:25:17.031457+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'54","last_scrub_stamp":"2024-02-05T08:24:18.160865+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:24:18.160865+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:41.199814+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":90,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.031736+0000","last_change":"2024-02-05T08:24:32.208227+0000","last_active":"2024-02-05T08:25:17.031736+0000","last_peered":"2024-02-05T08:25:17.031736+0000","last_clean":"2024-02-05T08:25:17.031736+0000","last_became_active":"2024-02-05T08:23:08.914563+0000","last_became_peered":"2024-02-05T08:23:08.914563+0000","last_unstale":"2024-02-05T08:25:17.031736+0000","last_undegraded":"2024-02-05T08:25:17.031736+0000","last_fullsized":"2024-02-05T08:25:17.031736+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:32.208069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:52.171116+0000","last_clean_scrub_stamp":"2024-02-05T08:24:32.208069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:55.410624+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"402'10889","reported_seq":10971,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:18.408500+0000","last_change":"2024-02-05T08:23:08.915803+0000","last_active":"2024-02-05T08:25:18.408500+0000","last_peered":"2024-02-05T08:25:18.408500+0000","last_clean":"2024-02-05T08:25:18.408500+0000","last_became_active":"2024-02-05T08:23:08.915520+0000","last_became_peered":"2024-02-05T08:23:08.915520+0000","last_unstale":"2024-02-05T08:25:18.408500+0000","last_undegraded":"2024-02-05T08:25:18.408500+0000","last_fullsized":"2024-02-05T08:25:18.408500+0000","mapping_epoch":367,"log_start":"389'6807","ondisk_log_start":"389'6807","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4082,"log_dups_size":0,"ondisk_log_size":4082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393543680,"num_objects":6005,"num_object_clones":0,"num_object_copies":18015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10889,"num_write_kb":540608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"82'130","reported_seq":1069,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.034990+0000","last_change":"2024-02-05T08:23:30.984623+0000","last_active":"2024-02-05T08:25:17.034990+0000","last_peered":"2024-02-05T08:25:17.034990+0000","last_clean":"2024-02-05T08:25:17.034990+0000","last_became_active":"2024-02-05T08:22:12.138774+0000","last_became_peered":"2024-02-05T08:22:12.138774+0000","last_unstale":"2024-02-05T08:25:17.034990+0000","last_undegraded":"2024-02-05T08:25:17.034990+0000","last_fullsized":"2024-02-05T08:25:17.034990+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:30.984495+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:23:30.984495+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:17.784164+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":766,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.037134+0000","last_change":"2024-02-05T08:22:09.032210+0000","last_active":"2024-02-05T08:25:17.037134+0000","last_peered":"2024-02-05T08:25:17.037134+0000","last_clean":"2024-02-05T08:25:17.037134+0000","last_became_active":"2024-02-05T08:18:34.643109+0000","last_became_peered":"2024-02-05T08:18:34.643109+0000","last_unstale":"2024-02-05T08:25:17.037134+0000","last_undegraded":"2024-02-05T08:25:17.037134+0000","last_fullsized":"2024-02-05T08:25:17.037134+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:09.032112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:09.032112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:17.784048+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":46,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.036309+0000","last_change":"2024-02-05T08:24:08.030313+0000","last_active":"2024-02-05T08:25:17.036309+0000","last_peered":"2024-02-05T08:25:17.036309+0000","last_clean":"2024-02-05T08:25:17.036309+0000","last_became_active":"2024-02-05T08:24:08.005056+0000","last_became_peered":"2024-02-05T08:24:08.005056+0000","last_unstale":"2024-02-05T08:25:17.036309+0000","last_undegraded":"2024-02-05T08:25:17.036309+0000","last_fullsized":"2024-02-05T08:25:17.036309+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:23:23.016855+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:23:23.016855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":808,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.028694+0000","last_change":"2024-02-05T08:24:17.745325+0000","last_active":"2024-02-05T08:25:17.028694+0000","last_peered":"2024-02-05T08:25:17.028694+0000","last_clean":"2024-02-05T08:25:17.028694+0000","last_became_active":"2024-02-05T08:23:08.912881+0000","last_became_peered":"2024-02-05T08:23:08.912881+0000","last_unstale":"2024-02-05T08:25:17.028694+0000","last_undegraded":"2024-02-05T08:25:17.028694+0000","last_fullsized":"2024-02-05T08:25:17.028694+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:17.745200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_clean_scrub_stamp":"2024-02-05T08:24:17.745200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:18.434153+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"397'26","reported_seq":812,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027575+0000","last_change":"2024-02-05T08:24:21.745236+0000","last_active":"2024-02-05T08:25:17.027575+0000","last_peered":"2024-02-05T08:25:17.027575+0000","last_clean":"2024-02-05T08:25:17.027575+0000","last_became_active":"2024-02-05T08:22:09.103077+0000","last_became_peered":"2024-02-05T08:22:09.103077+0000","last_unstale":"2024-02-05T08:25:17.027575+0000","last_undegraded":"2024-02-05T08:25:17.027575+0000","last_fullsized":"2024-02-05T08:25:17.027575+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'22","last_scrub_stamp":"2024-02-05T08:24:21.745112+0000","last_deep_scrub":"363'22","last_deep_scrub_stamp":"2024-02-05T08:22:56.032522+0000","last_clean_scrub_stamp":"2024-02-05T08:24:21.745112+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:51.473685+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":777,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.028510+0000","last_change":"2024-02-05T08:24:22.757410+0000","last_active":"2024-02-05T08:25:17.028510+0000","last_peered":"2024-02-05T08:25:17.028510+0000","last_clean":"2024-02-05T08:25:17.028510+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:25:17.028510+0000","last_undegraded":"2024-02-05T08:25:17.028510+0000","last_fullsized":"2024-02-05T08:25:17.028510+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:22.757290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:24:22.757290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:38.124802+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"402'10920","reported_seq":11001,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:18.425772+0000","last_change":"2024-02-05T08:23:08.914881+0000","last_active":"2024-02-05T08:25:18.425772+0000","last_peered":"2024-02-05T08:25:18.425772+0000","last_clean":"2024-02-05T08:25:18.425772+0000","last_became_active":"2024-02-05T08:23:08.914611+0000","last_became_peered":"2024-02-05T08:23:08.914611+0000","last_unstale":"2024-02-05T08:25:18.425772+0000","last_undegraded":"2024-02-05T08:25:18.425772+0000","last_fullsized":"2024-02-05T08:25:18.425772+0000","mapping_epoch":367,"log_start":"389'6900","ondisk_log_start":"389'6900","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4020,"log_dups_size":0,"ondisk_log_size":4020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405798912,"num_objects":6192,"num_object_clones":0,"num_object_copies":18576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10920,"num_write_kb":547584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":761,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.039430+0000","last_change":"2024-02-05T08:23:44.881991+0000","last_active":"2024-02-05T08:25:17.039430+0000","last_peered":"2024-02-05T08:25:17.039430+0000","last_clean":"2024-02-05T08:25:17.039430+0000","last_became_active":"2024-02-05T08:18:34.633450+0000","last_became_peered":"2024-02-05T08:18:34.633450+0000","last_unstale":"2024-02-05T08:25:17.039430+0000","last_undegraded":"2024-02-05T08:25:17.039430+0000","last_fullsized":"2024-02-05T08:25:17.039430+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:44.881864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:44.881864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887448+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":1066,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.034507+0000","last_change":"2024-02-05T08:23:34.058622+0000","last_active":"2024-02-05T08:25:17.034507+0000","last_peered":"2024-02-05T08:25:17.034507+0000","last_clean":"2024-02-05T08:25:17.034507+0000","last_became_active":"2024-02-05T08:23:10.886936+0000","last_became_peered":"2024-02-05T08:23:10.886936+0000","last_unstale":"2024-02-05T08:25:17.034507+0000","last_undegraded":"2024-02-05T08:25:17.034507+0000","last_fullsized":"2024-02-05T08:25:17.034507+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:34.058210+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_clean_scrub_stamp":"2024-02-05T08:23:34.058210+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:17.784470+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":252,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040929+0000","last_change":"2024-02-05T08:22:54.937959+0000","last_active":"2024-02-05T08:25:17.040929+0000","last_peered":"2024-02-05T08:25:17.040929+0000","last_clean":"2024-02-05T08:25:17.040929+0000","last_became_active":"2024-02-05T08:18:34.631974+0000","last_became_peered":"2024-02-05T08:18:34.631974+0000","last_unstale":"2024-02-05T08:25:17.040929+0000","last_undegraded":"2024-02-05T08:25:17.040929+0000","last_fullsized":"2024-02-05T08:25:17.040929+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:54.937874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:54.937874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"397'42","reported_seq":699,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041640+0000","last_change":"2024-02-05T08:23:26.566491+0000","last_active":"2024-02-05T08:25:17.041640+0000","last_peered":"2024-02-05T08:25:17.041640+0000","last_clean":"2024-02-05T08:25:17.041640+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:25:17.041640+0000","last_undegraded":"2024-02-05T08:25:17.041640+0000","last_fullsized":"2024-02-05T08:25:17.041640+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'30","last_scrub_stamp":"2024-02-05T08:23:26.566358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:26.566358+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.112854+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":754,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.036163+0000","last_change":"2024-02-05T08:24:27.990755+0000","last_active":"2024-02-05T08:25:17.036163+0000","last_peered":"2024-02-05T08:25:17.036163+0000","last_clean":"2024-02-05T08:25:17.036163+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:25:17.036163+0000","last_undegraded":"2024-02-05T08:25:17.036163+0000","last_fullsized":"2024-02-05T08:25:17.036163+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:27.990653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_clean_scrub_stamp":"2024-02-05T08:24:27.990653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:34.928370+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"402'10692","reported_seq":10773,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:18.437500+0000","last_change":"2024-02-05T08:23:08.916824+0000","last_active":"2024-02-05T08:25:18.437500+0000","last_peered":"2024-02-05T08:25:18.437500+0000","last_clean":"2024-02-05T08:25:18.437500+0000","last_became_active":"2024-02-05T08:23:08.916452+0000","last_became_peered":"2024-02-05T08:23:08.916452+0000","last_unstale":"2024-02-05T08:25:18.437500+0000","last_undegraded":"2024-02-05T08:25:18.437500+0000","last_fullsized":"2024-02-05T08:25:18.437500+0000","mapping_epoch":367,"log_start":"389'6624","ondisk_log_start":"389'6624","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402784256,"num_objects":6146,"num_object_clones":0,"num_object_copies":18438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10692,"num_write_kb":538816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"82'130","reported_seq":416,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.032416+0000","last_change":"2024-02-05T08:23:25.057316+0000","last_active":"2024-02-05T08:25:17.032416+0000","last_peered":"2024-02-05T08:25:17.032416+0000","last_clean":"2024-02-05T08:25:17.032416+0000","last_became_active":"2024-02-05T08:23:07.941142+0000","last_became_peered":"2024-02-05T08:23:07.941142+0000","last_unstale":"2024-02-05T08:25:17.032416+0000","last_undegraded":"2024-02-05T08:25:17.032416+0000","last_fullsized":"2024-02-05T08:25:17.032416+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:23:25.057181+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:23:25.057181+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":658,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.031941+0000","last_change":"2024-02-05T08:23:39.136581+0000","last_active":"2024-02-05T08:25:17.031941+0000","last_peered":"2024-02-05T08:25:17.031941+0000","last_clean":"2024-02-05T08:25:17.031941+0000","last_became_active":"2024-02-05T08:23:07.935640+0000","last_became_peered":"2024-02-05T08:23:07.935640+0000","last_unstale":"2024-02-05T08:25:17.031941+0000","last_undegraded":"2024-02-05T08:25:17.031941+0000","last_fullsized":"2024-02-05T08:25:17.031941+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:39.136473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:39.136473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":617,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027260+0000","last_change":"2024-02-05T08:22:09.103119+0000","last_active":"2024-02-05T08:25:17.027260+0000","last_peered":"2024-02-05T08:25:17.027260+0000","last_clean":"2024-02-05T08:25:17.027260+0000","last_became_active":"2024-02-05T08:22:09.102613+0000","last_became_peered":"2024-02-05T08:22:09.102613+0000","last_unstale":"2024-02-05T08:25:17.027260+0000","last_undegraded":"2024-02-05T08:25:17.027260+0000","last_fullsized":"2024-02-05T08:25:17.027260+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:21:57.240901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:21:57.240901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174666+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":2010,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.204477+0000","last_change":"2024-02-05T08:23:24.028818+0000","last_active":"2024-02-05T08:25:17.204477+0000","last_peered":"2024-02-05T08:25:17.204477+0000","last_clean":"2024-02-05T08:25:17.204477+0000","last_became_active":"2024-02-05T08:23:08.921709+0000","last_became_peered":"2024-02-05T08:23:08.921709+0000","last_unstale":"2024-02-05T08:25:17.204477+0000","last_undegraded":"2024-02-05T08:25:17.204477+0000","last_fullsized":"2024-02-05T08:25:17.204477+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:23:24.028705+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:23:24.028705+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1219,"num_read_kb":1214,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"397'36","reported_seq":576,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.039436+0000","last_change":"2024-02-05T08:24:19.820651+0000","last_active":"2024-02-05T08:25:17.039436+0000","last_peered":"2024-02-05T08:25:17.039436+0000","last_clean":"2024-02-05T08:25:17.039436+0000","last_became_active":"2024-02-05T08:22:09.106287+0000","last_became_peered":"2024-02-05T08:22:09.106287+0000","last_unstale":"2024-02-05T08:25:17.039436+0000","last_undegraded":"2024-02-05T08:25:17.039436+0000","last_fullsized":"2024-02-05T08:25:17.039436+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:24:19.820518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:24:19.820518+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:41.780807+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":766,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.039677+0000","last_change":"2024-02-05T08:24:44.871482+0000","last_active":"2024-02-05T08:25:17.039677+0000","last_peered":"2024-02-05T08:25:17.039677+0000","last_clean":"2024-02-05T08:25:17.039677+0000","last_became_active":"2024-02-05T08:23:08.914353+0000","last_became_peered":"2024-02-05T08:23:08.914353+0000","last_unstale":"2024-02-05T08:25:17.039677+0000","last_undegraded":"2024-02-05T08:25:17.039677+0000","last_fullsized":"2024-02-05T08:25:17.039677+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:44.871105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:24:44.871105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:08.941424+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"402'10873","reported_seq":10954,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:21.052437+0000","last_change":"2024-02-05T08:23:08.921308+0000","last_active":"2024-02-05T08:25:21.052437+0000","last_peered":"2024-02-05T08:25:21.052437+0000","last_clean":"2024-02-05T08:25:21.052437+0000","last_became_active":"2024-02-05T08:23:08.920532+0000","last_became_peered":"2024-02-05T08:23:08.920532+0000","last_unstale":"2024-02-05T08:25:21.052437+0000","last_undegraded":"2024-02-05T08:25:21.052437+0000","last_fullsized":"2024-02-05T08:25:21.052437+0000","mapping_epoch":367,"log_start":"389'6800","ondisk_log_start":"389'6800","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4073,"log_dups_size":0,"ondisk_log_size":4073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385024000,"num_objects":5875,"num_object_clones":0,"num_object_copies":17625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10873,"num_write_kb":535936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'130","reported_seq":813,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040323+0000","last_change":"2024-02-05T08:24:44.389866+0000","last_active":"2024-02-05T08:25:17.040323+0000","last_peered":"2024-02-05T08:25:17.040323+0000","last_clean":"2024-02-05T08:25:17.040323+0000","last_became_active":"2024-02-05T08:22:12.143283+0000","last_became_peered":"2024-02-05T08:22:12.143283+0000","last_unstale":"2024-02-05T08:25:17.040323+0000","last_undegraded":"2024-02-05T08:25:17.040323+0000","last_fullsized":"2024-02-05T08:25:17.040323+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:24:44.389747+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:24:44.389747+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:44.913688+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":667,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.031294+0000","last_change":"2024-02-05T08:24:29.342475+0000","last_active":"2024-02-05T08:25:17.031294+0000","last_peered":"2024-02-05T08:25:17.031294+0000","last_clean":"2024-02-05T08:25:17.031294+0000","last_became_active":"2024-02-05T08:23:07.928514+0000","last_became_peered":"2024-02-05T08:23:07.928514+0000","last_unstale":"2024-02-05T08:25:17.031294+0000","last_undegraded":"2024-02-05T08:25:17.031294+0000","last_fullsized":"2024-02-05T08:25:17.031294+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:29.342365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:29.342365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:33.382311+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":622,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.031783+0000","last_change":"2024-02-05T08:23:27.086997+0000","last_active":"2024-02-05T08:25:17.031783+0000","last_peered":"2024-02-05T08:25:17.031783+0000","last_clean":"2024-02-05T08:25:17.031783+0000","last_became_active":"2024-02-05T08:23:07.935692+0000","last_became_peered":"2024-02-05T08:23:07.935692+0000","last_unstale":"2024-02-05T08:25:17.031783+0000","last_undegraded":"2024-02-05T08:25:17.031783+0000","last_fullsized":"2024-02-05T08:25:17.031783+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:27.086862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:23:27.086862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"397'47","reported_seq":592,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.038765+0000","last_change":"2024-02-05T08:23:36.866473+0000","last_active":"2024-02-05T08:25:17.038765+0000","last_peered":"2024-02-05T08:25:17.038765+0000","last_clean":"2024-02-05T08:25:17.038765+0000","last_became_active":"2024-02-05T08:23:08.917913+0000","last_became_peered":"2024-02-05T08:23:08.917913+0000","last_unstale":"2024-02-05T08:25:17.038765+0000","last_undegraded":"2024-02-05T08:25:17.038765+0000","last_fullsized":"2024-02-05T08:25:17.038765+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"381'37","last_scrub_stamp":"2024-02-05T08:23:36.866312+0000","last_deep_scrub":"381'37","last_deep_scrub_stamp":"2024-02-05T08:23:36.866312+0000","last_clean_scrub_stamp":"2024-02-05T08:23:36.866312+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887476+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":505,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.034380+0000","last_change":"2024-02-05T08:24:36.394771+0000","last_active":"2024-02-05T08:25:17.034380+0000","last_peered":"2024-02-05T08:25:17.034380+0000","last_clean":"2024-02-05T08:25:17.034380+0000","last_became_active":"2024-02-05T08:22:09.096719+0000","last_became_peered":"2024-02-05T08:22:09.096719+0000","last_unstale":"2024-02-05T08:25:17.034380+0000","last_undegraded":"2024-02-05T08:25:17.034380+0000","last_fullsized":"2024-02-05T08:25:17.034380+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:36.394540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:24:36.394540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:48.929784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"402'10741","reported_seq":10902,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:18.608607+0000","last_change":"2024-02-05T08:24:33.152720+0000","last_active":"2024-02-05T08:25:18.608607+0000","last_peered":"2024-02-05T08:25:18.608607+0000","last_clean":"2024-02-05T08:25:18.608607+0000","last_became_active":"2024-02-05T08:23:08.911093+0000","last_became_peered":"2024-02-05T08:23:08.911093+0000","last_unstale":"2024-02-05T08:25:18.608607+0000","last_undegraded":"2024-02-05T08:25:18.608607+0000","last_fullsized":"2024-02-05T08:25:18.608607+0000","mapping_epoch":367,"log_start":"389'5900","ondisk_log_start":"389'5900","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'7149","last_scrub_stamp":"2024-02-05T08:24:33.152473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:24:33.152473+0000","objects_scrubbed":7060,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:36.147401+0000","scrub_duration":2346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397148184,"num_objects":6061,"num_object_clones":0,"num_object_copies":18183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10741,"num_write_kb":537601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"82'130","reported_seq":804,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.037742+0000","last_change":"2024-02-05T08:24:43.359798+0000","last_active":"2024-02-05T08:25:17.037742+0000","last_peered":"2024-02-05T08:25:17.037742+0000","last_clean":"2024-02-05T08:25:17.037742+0000","last_became_active":"2024-02-05T08:19:01.815446+0000","last_became_peered":"2024-02-05T08:19:01.815446+0000","last_unstale":"2024-02-05T08:25:17.037742+0000","last_undegraded":"2024-02-05T08:25:17.037742+0000","last_fullsized":"2024-02-05T08:25:17.037742+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:24:43.359678+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:35.974462+0000","last_clean_scrub_stamp":"2024-02-05T08:24:43.359678+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:08.116373+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":621,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.035904+0000","last_change":"2024-02-05T08:22:57.918423+0000","last_active":"2024-02-05T08:25:17.035904+0000","last_peered":"2024-02-05T08:25:17.035904+0000","last_clean":"2024-02-05T08:25:17.035904+0000","last_became_active":"2024-02-05T08:22:09.100096+0000","last_became_peered":"2024-02-05T08:22:09.100096+0000","last_unstale":"2024-02-05T08:25:17.035904+0000","last_undegraded":"2024-02-05T08:25:17.035904+0000","last_fullsized":"2024-02-05T08:25:17.035904+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:57.918346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:22:57.918346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":773,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040611+0000","last_change":"2024-02-05T08:23:38.820649+0000","last_active":"2024-02-05T08:25:17.040611+0000","last_peered":"2024-02-05T08:25:17.040611+0000","last_clean":"2024-02-05T08:25:17.040611+0000","last_became_active":"2024-02-05T08:18:34.633263+0000","last_became_peered":"2024-02-05T08:18:34.633263+0000","last_unstale":"2024-02-05T08:25:17.040611+0000","last_undegraded":"2024-02-05T08:25:17.040611+0000","last_fullsized":"2024-02-05T08:25:17.040611+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:38.820534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:23:38.820534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887373+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"397'30","reported_seq":693,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.031423+0000","last_change":"2024-02-05T08:23:41.072770+0000","last_active":"2024-02-05T08:25:17.031423+0000","last_peered":"2024-02-05T08:25:17.031423+0000","last_clean":"2024-02-05T08:25:17.031423+0000","last_became_active":"2024-02-05T08:23:07.941561+0000","last_became_peered":"2024-02-05T08:23:07.941561+0000","last_unstale":"2024-02-05T08:25:17.031423+0000","last_undegraded":"2024-02-05T08:25:17.031423+0000","last_fullsized":"2024-02-05T08:25:17.031423+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'20","last_scrub_stamp":"2024-02-05T08:23:41.072637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:41.072637+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"355'12","reported_seq":808,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041261+0000","last_change":"2024-02-05T08:24:38.385360+0000","last_active":"2024-02-05T08:25:17.041261+0000","last_peered":"2024-02-05T08:25:17.041261+0000","last_clean":"2024-02-05T08:25:17.041261+0000","last_became_active":"2024-02-05T08:22:09.099076+0000","last_became_peered":"2024-02-05T08:22:09.099076+0000","last_unstale":"2024-02-05T08:25:17.041261+0000","last_undegraded":"2024-02-05T08:25:17.041261+0000","last_fullsized":"2024-02-05T08:25:17.041261+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'12","last_scrub_stamp":"2024-02-05T08:24:38.385240+0000","last_deep_scrub":"355'12","last_deep_scrub_stamp":"2024-02-05T08:24:38.385240+0000","last_clean_scrub_stamp":"2024-02-05T08:24:38.385240+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:05.218735+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.e","version":"402'10687","reported_seq":10768,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.900392+0000","last_change":"2024-02-05T08:23:08.910748+0000","last_active":"2024-02-05T08:25:17.900392+0000","last_peered":"2024-02-05T08:25:17.900392+0000","last_clean":"2024-02-05T08:25:17.900392+0000","last_became_active":"2024-02-05T08:23:08.910360+0000","last_became_peered":"2024-02-05T08:23:08.910360+0000","last_unstale":"2024-02-05T08:25:17.900392+0000","last_undegraded":"2024-02-05T08:25:17.900392+0000","last_fullsized":"2024-02-05T08:25:17.900392+0000","mapping_epoch":367,"log_start":"389'6400","ondisk_log_start":"389'6400","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399048704,"num_objects":6089,"num_object_clones":0,"num_object_copies":18267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10687,"num_write_kb":536832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"82'130","reported_seq":528,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041226+0000","last_change":"2024-02-05T08:22:47.999978+0000","last_active":"2024-02-05T08:25:17.041226+0000","last_peered":"2024-02-05T08:25:17.041226+0000","last_clean":"2024-02-05T08:25:17.041226+0000","last_became_active":"2024-02-05T08:19:01.819856+0000","last_became_peered":"2024-02-05T08:19:01.819856+0000","last_unstale":"2024-02-05T08:25:17.041226+0000","last_undegraded":"2024-02-05T08:25:17.041226+0000","last_fullsized":"2024-02-05T08:25:17.041226+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:22:47.999878+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:22:47.999878+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":754,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.035712+0000","last_change":"2024-02-05T08:23:36.150803+0000","last_active":"2024-02-05T08:25:17.035712+0000","last_peered":"2024-02-05T08:25:17.035712+0000","last_clean":"2024-02-05T08:25:17.035712+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:25:17.035712+0000","last_undegraded":"2024-02-05T08:25:17.035712+0000","last_fullsized":"2024-02-05T08:25:17.035712+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:36.150674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:23:36.150674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:17.784419+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":98,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.032154+0000","last_change":"2024-02-05T08:24:30.256669+0000","last_active":"2024-02-05T08:25:17.032154+0000","last_peered":"2024-02-05T08:25:17.032154+0000","last_clean":"2024-02-05T08:25:17.032154+0000","last_became_active":"2024-02-05T08:23:08.913263+0000","last_became_peered":"2024-02-05T08:23:08.913263+0000","last_unstale":"2024-02-05T08:25:17.032154+0000","last_undegraded":"2024-02-05T08:25:17.032154+0000","last_fullsized":"2024-02-05T08:25:17.032154+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:24:30.256524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:30.256524+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:30.313107+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"397'42","reported_seq":856,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027215+0000","last_change":"2024-02-05T08:24:16.241737+0000","last_active":"2024-02-05T08:25:17.027215+0000","last_peered":"2024-02-05T08:25:17.027215+0000","last_clean":"2024-02-05T08:25:17.027215+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:25:17.027215+0000","last_undegraded":"2024-02-05T08:25:17.027215+0000","last_fullsized":"2024-02-05T08:25:17.027215+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:24:16.241606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:24:16.241606+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:34.065074+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"370'13","reported_seq":341,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.042769+0000","last_change":"2024-02-05T08:23:33.927361+0000","last_active":"2024-02-05T08:25:17.042769+0000","last_peered":"2024-02-05T08:25:17.042769+0000","last_clean":"2024-02-05T08:25:17.042769+0000","last_became_active":"2024-02-05T08:23:07.936347+0000","last_became_peered":"2024-02-05T08:23:07.936347+0000","last_unstale":"2024-02-05T08:25:17.042769+0000","last_undegraded":"2024-02-05T08:25:17.042769+0000","last_fullsized":"2024-02-05T08:25:17.042769+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'13","last_scrub_stamp":"2024-02-05T08:23:33.927148+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:23:33.927148+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"402'10614","reported_seq":10695,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:18.464542+0000","last_change":"2024-02-05T08:23:08.921447+0000","last_active":"2024-02-05T08:25:18.464542+0000","last_peered":"2024-02-05T08:25:18.464542+0000","last_clean":"2024-02-05T08:25:18.464542+0000","last_became_active":"2024-02-05T08:23:08.921121+0000","last_became_peered":"2024-02-05T08:23:08.921121+0000","last_unstale":"2024-02-05T08:25:18.464542+0000","last_undegraded":"2024-02-05T08:25:18.464542+0000","last_fullsized":"2024-02-05T08:25:18.464542+0000","mapping_epoch":367,"log_start":"389'6100","ondisk_log_start":"389'6100","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:23:07.894546+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397672448,"num_objects":6068,"num_object_clones":0,"num_object_copies":18204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10614,"num_write_kb":533824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"82'130","reported_seq":527,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027874+0000","last_change":"2024-02-05T08:22:58.247977+0000","last_active":"2024-02-05T08:25:17.027874+0000","last_peered":"2024-02-05T08:25:17.027874+0000","last_clean":"2024-02-05T08:25:17.027874+0000","last_became_active":"2024-02-05T08:19:01.816923+0000","last_became_peered":"2024-02-05T08:19:01.816923+0000","last_unstale":"2024-02-05T08:25:17.027874+0000","last_undegraded":"2024-02-05T08:25:17.027874+0000","last_fullsized":"2024-02-05T08:25:17.027874+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:22:58.247896+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:22:58.247896+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":761,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027375+0000","last_change":"2024-02-05T08:22:08.118133+0000","last_active":"2024-02-05T08:25:17.027375+0000","last_peered":"2024-02-05T08:25:17.027375+0000","last_clean":"2024-02-05T08:25:17.027375+0000","last_became_active":"2024-02-05T08:18:34.635908+0000","last_became_peered":"2024-02-05T08:18:34.635908+0000","last_unstale":"2024-02-05T08:25:17.027375+0000","last_undegraded":"2024-02-05T08:25:17.027375+0000","last_fullsized":"2024-02-05T08:25:17.027375+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:08.117925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:08.117925+0000","last_clean_scrub_stamp":"2024-02-05T08:22:08.117925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174641+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":266,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.042357+0000","last_change":"2024-02-05T08:22:56.880110+0000","last_active":"2024-02-05T08:25:17.042357+0000","last_peered":"2024-02-05T08:25:17.042357+0000","last_clean":"2024-02-05T08:25:17.042357+0000","last_became_active":"2024-02-05T08:22:09.101588+0000","last_became_peered":"2024-02-05T08:22:09.101588+0000","last_unstale":"2024-02-05T08:25:17.042357+0000","last_undegraded":"2024-02-05T08:25:17.042357+0000","last_fullsized":"2024-02-05T08:25:17.042357+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:22:56.880011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:22:56.880011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"397'50","reported_seq":316,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.036081+0000","last_change":"2024-02-05T08:23:08.944358+0000","last_active":"2024-02-05T08:25:17.036081+0000","last_peered":"2024-02-05T08:25:17.036081+0000","last_clean":"2024-02-05T08:25:17.036081+0000","last_became_active":"2024-02-05T08:23:08.927776+0000","last_became_peered":"2024-02-05T08:23:08.927776+0000","last_unstale":"2024-02-05T08:25:17.036081+0000","last_undegraded":"2024-02-05T08:25:17.036081+0000","last_fullsized":"2024-02-05T08:25:17.036081+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'38","last_scrub_stamp":"2024-02-05T08:22:48.999388+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:22:48.999388+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":757,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027132+0000","last_change":"2024-02-05T08:22:07.078658+0000","last_active":"2024-02-05T08:25:17.027132+0000","last_peered":"2024-02-05T08:25:17.027132+0000","last_clean":"2024-02-05T08:25:17.027132+0000","last_became_active":"2024-02-05T08:18:34.636667+0000","last_became_peered":"2024-02-05T08:18:34.636667+0000","last_unstale":"2024-02-05T08:25:17.027132+0000","last_undegraded":"2024-02-05T08:25:17.027132+0000","last_fullsized":"2024-02-05T08:25:17.027132+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:07.078318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:31.133206+0000","last_clean_scrub_stamp":"2024-02-05T08:22:07.078318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174617+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"402'10962","reported_seq":11113,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:21.020624+0000","last_change":"2024-02-05T08:25:20.245674+0000","last_active":"2024-02-05T08:25:21.020624+0000","last_peered":"2024-02-05T08:25:21.020624+0000","last_clean":"2024-02-05T08:25:21.020624+0000","last_became_active":"2024-02-05T08:23:08.916901+0000","last_became_peered":"2024-02-05T08:23:08.916901+0000","last_unstale":"2024-02-05T08:25:21.020624+0000","last_undegraded":"2024-02-05T08:25:21.020624+0000","last_fullsized":"2024-02-05T08:25:21.020624+0000","mapping_epoch":367,"log_start":"389'6900","ondisk_log_start":"389'6900","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'10893","last_scrub_stamp":"2024-02-05T08:25:20.245455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:25:20.245455+0000","objects_scrubbed":6069,"log_size":4062,"log_dups_size":0,"ondisk_log_size":4062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:43.636796+0000","scrub_duration":2064,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387317760,"num_objects":5910,"num_object_clones":0,"num_object_copies":17730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10962,"num_write_kb":539904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'130","reported_seq":535,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.038302+0000","last_change":"2024-02-05T08:24:22.852362+0000","last_active":"2024-02-05T08:25:17.038302+0000","last_peered":"2024-02-05T08:25:17.038302+0000","last_clean":"2024-02-05T08:25:17.038302+0000","last_became_active":"2024-02-05T08:19:01.817169+0000","last_became_peered":"2024-02-05T08:19:01.817169+0000","last_unstale":"2024-02-05T08:25:17.038302+0000","last_undegraded":"2024-02-05T08:25:17.038302+0000","last_fullsized":"2024-02-05T08:25:17.038302+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:24:22.852081+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:24:22.852081+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:38.827043+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":762,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040460+0000","last_change":"2024-02-05T08:24:46.039199+0000","last_active":"2024-02-05T08:25:17.040460+0000","last_peered":"2024-02-05T08:25:17.040460+0000","last_clean":"2024-02-05T08:25:17.040460+0000","last_became_active":"2024-02-05T08:23:08.913735+0000","last_became_peered":"2024-02-05T08:23:08.913735+0000","last_unstale":"2024-02-05T08:25:17.040460+0000","last_undegraded":"2024-02-05T08:25:17.040460+0000","last_fullsized":"2024-02-05T08:25:17.040460+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:46.039077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:46.039077+0000","last_clean_scrub_stamp":"2024-02-05T08:24:46.039077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:58.558223+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"355'9","reported_seq":1011,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.028402+0000","last_change":"2024-02-05T08:22:09.114866+0000","last_active":"2024-02-05T08:25:17.028402+0000","last_peered":"2024-02-05T08:25:17.028402+0000","last_clean":"2024-02-05T08:25:17.028402+0000","last_became_active":"2024-02-05T08:22:09.106117+0000","last_became_peered":"2024-02-05T08:22:09.106117+0000","last_unstale":"2024-02-05T08:25:17.028402+0000","last_undegraded":"2024-02-05T08:25:17.028402+0000","last_fullsized":"2024-02-05T08:25:17.028402+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-02-05T08:22:06.070321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:22:06.070321+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174694+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":253,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.036233+0000","last_change":"2024-02-05T08:23:29.957871+0000","last_active":"2024-02-05T08:25:17.036233+0000","last_peered":"2024-02-05T08:25:17.036233+0000","last_clean":"2024-02-05T08:25:17.036233+0000","last_became_active":"2024-02-05T08:23:07.937468+0000","last_became_peered":"2024-02-05T08:23:07.937468+0000","last_unstale":"2024-02-05T08:25:17.036233+0000","last_undegraded":"2024-02-05T08:25:17.036233+0000","last_fullsized":"2024-02-05T08:25:17.036233+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:29.957655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.957655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"397'39","reported_seq":727,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.034399+0000","last_change":"2024-02-05T08:23:30.166109+0000","last_active":"2024-02-05T08:25:17.034399+0000","last_peered":"2024-02-05T08:25:17.034399+0000","last_clean":"2024-02-05T08:25:17.034399+0000","last_became_active":"2024-02-05T08:23:07.939747+0000","last_became_peered":"2024-02-05T08:23:07.939747+0000","last_unstale":"2024-02-05T08:25:17.034399+0000","last_undegraded":"2024-02-05T08:25:17.034399+0000","last_fullsized":"2024-02-05T08:25:17.034399+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-02-05T08:23:30.165977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:30.165977+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"402'10807","reported_seq":10966,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:18.301666+0000","last_change":"2024-02-05T08:24:31.613860+0000","last_active":"2024-02-05T08:25:18.301666+0000","last_peered":"2024-02-05T08:25:18.301666+0000","last_clean":"2024-02-05T08:25:18.301666+0000","last_became_active":"2024-02-05T08:23:08.913844+0000","last_became_peered":"2024-02-05T08:23:08.913844+0000","last_unstale":"2024-02-05T08:25:18.301666+0000","last_undegraded":"2024-02-05T08:25:18.301666+0000","last_fullsized":"2024-02-05T08:25:18.301666+0000","mapping_epoch":367,"log_start":"389'5800","ondisk_log_start":"389'5800","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'7028","last_scrub_stamp":"2024-02-05T08:24:31.613639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:24:31.613639+0000","objects_scrubbed":6944,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:52.989084+0000","scrub_duration":2182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398262272,"num_objects":6077,"num_object_clones":0,"num_object_copies":18231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10807,"num_write_kb":540288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":251,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041400+0000","last_change":"2024-02-05T08:22:51.956578+0000","last_active":"2024-02-05T08:25:17.041400+0000","last_peered":"2024-02-05T08:25:17.041400+0000","last_clean":"2024-02-05T08:25:17.041400+0000","last_became_active":"2024-02-05T08:18:34.627261+0000","last_became_peered":"2024-02-05T08:18:34.627261+0000","last_unstale":"2024-02-05T08:25:17.041400+0000","last_undegraded":"2024-02-05T08:25:17.041400+0000","last_fullsized":"2024-02-05T08:25:17.041400+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:22:51.956504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:22:51.956504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":773,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.039653+0000","last_change":"2024-02-05T08:24:30.868452+0000","last_active":"2024-02-05T08:25:17.039653+0000","last_peered":"2024-02-05T08:25:17.039653+0000","last_clean":"2024-02-05T08:25:17.039653+0000","last_became_active":"2024-02-05T08:12:29.685769+0000","last_became_peered":"2024-02-05T08:12:29.685769+0000","last_unstale":"2024-02-05T08:25:17.039653+0000","last_undegraded":"2024-02-05T08:25:17.039653+0000","last_fullsized":"2024-02-05T08:25:17.039653+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:30.867869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:24:30.867869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:49.702075+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":761,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.027169+0000","last_change":"2024-02-05T08:24:38.141660+0000","last_active":"2024-02-05T08:25:17.027169+0000","last_peered":"2024-02-05T08:25:17.027169+0000","last_clean":"2024-02-05T08:25:17.027169+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:25:17.027169+0000","last_undegraded":"2024-02-05T08:25:17.027169+0000","last_fullsized":"2024-02-05T08:25:17.027169+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:38.141536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:38.141536+0000","last_clean_scrub_stamp":"2024-02-05T08:24:38.141536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:57.395156+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"397'27","reported_seq":829,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.028293+0000","last_change":"2024-02-05T08:23:39.449120+0000","last_active":"2024-02-05T08:25:17.028293+0000","last_peered":"2024-02-05T08:25:17.028293+0000","last_clean":"2024-02-05T08:25:17.028293+0000","last_became_active":"2024-02-05T08:23:07.937017+0000","last_became_peered":"2024-02-05T08:23:07.937017+0000","last_unstale":"2024-02-05T08:25:17.028293+0000","last_undegraded":"2024-02-05T08:25:17.028293+0000","last_fullsized":"2024-02-05T08:25:17.028293+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'19","last_scrub_stamp":"2024-02-05T08:23:39.449015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:39.449015+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174935+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"402'11024","reported_seq":11184,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:18.387363+0000","last_change":"2024-02-05T08:25:05.046745+0000","last_active":"2024-02-05T08:25:18.387363+0000","last_peered":"2024-02-05T08:25:18.387363+0000","last_clean":"2024-02-05T08:25:18.387363+0000","last_became_active":"2024-02-05T08:25:04.994287+0000","last_became_peered":"2024-02-05T08:25:04.994287+0000","last_unstale":"2024-02-05T08:25:18.387363+0000","last_undegraded":"2024-02-05T08:25:18.387363+0000","last_fullsized":"2024-02-05T08:25:18.387363+0000","mapping_epoch":400,"log_start":"389'6966","ondisk_log_start":"389'6966","created":367,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'6207","last_scrub_stamp":"2024-02-05T08:24:17.908422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:24:17.908422+0000","objects_scrubbed":6107,"log_size":4058,"log_dups_size":0,"ondisk_log_size":4058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:41.683930+0000","scrub_duration":2250,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406061056,"num_objects":6196,"num_object_clones":0,"num_object_copies":18588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11024,"num_write_kb":551040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"82'130","reported_seq":527,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.040274+0000","last_change":"2024-02-05T08:22:57.194319+0000","last_active":"2024-02-05T08:25:17.040274+0000","last_peered":"2024-02-05T08:25:17.040274+0000","last_clean":"2024-02-05T08:25:17.040274+0000","last_became_active":"2024-02-05T08:19:01.817126+0000","last_became_peered":"2024-02-05T08:19:01.817126+0000","last_unstale":"2024-02-05T08:25:17.040274+0000","last_undegraded":"2024-02-05T08:25:17.040274+0000","last_fullsized":"2024-02-05T08:25:17.040274+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:22:57.194237+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:22:57.194237+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:21.887227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":538,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.042159+0000","last_change":"2024-02-05T08:25:04.981918+0000","last_active":"2024-02-05T08:25:17.042159+0000","last_peered":"2024-02-05T08:25:17.042159+0000","last_clean":"2024-02-05T08:25:17.042159+0000","last_became_active":"2024-02-05T08:25:04.981516+0000","last_became_peered":"2024-02-05T08:25:04.981516+0000","last_unstale":"2024-02-05T08:25:17.042159+0000","last_undegraded":"2024-02-05T08:25:17.042159+0000","last_fullsized":"2024-02-05T08:25:17.042159+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:24:45.091300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:45.091300+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:45.277573+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":607,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:20.244771+0000","last_change":"2024-02-05T08:25:20.244771+0000","last_active":"2024-02-05T08:25:20.244771+0000","last_peered":"2024-02-05T08:25:20.244771+0000","last_clean":"2024-02-05T08:25:20.244771+0000","last_became_active":"2024-02-05T08:23:08.914502+0000","last_became_peered":"2024-02-05T08:23:08.914502+0000","last_unstale":"2024-02-05T08:25:20.244771+0000","last_undegraded":"2024-02-05T08:25:20.244771+0000","last_fullsized":"2024-02-05T08:25:20.244771+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:20.244668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:25:20.244668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:23.679654+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":491,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041041+0000","last_change":"2024-02-05T08:23:28.612625+0000","last_active":"2024-02-05T08:25:17.041041+0000","last_peered":"2024-02-05T08:25:17.041041+0000","last_clean":"2024-02-05T08:25:17.041041+0000","last_became_active":"2024-02-05T08:23:07.929749+0000","last_became_peered":"2024-02-05T08:23:07.929749+0000","last_unstale":"2024-02-05T08:25:17.041041+0000","last_undegraded":"2024-02-05T08:25:17.041041+0000","last_fullsized":"2024-02-05T08:25:17.041041+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:28.612512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:23:28.612512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.112796+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"397'45","reported_seq":820,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.028374+0000","last_change":"2024-02-05T08:24:26.158100+0000","last_active":"2024-02-05T08:25:17.028374+0000","last_peered":"2024-02-05T08:25:17.028374+0000","last_clean":"2024-02-05T08:25:17.028374+0000","last_became_active":"2024-02-05T08:23:07.930130+0000","last_became_peered":"2024-02-05T08:23:07.930130+0000","last_unstale":"2024-02-05T08:25:17.028374+0000","last_undegraded":"2024-02-05T08:25:17.028374+0000","last_fullsized":"2024-02-05T08:25:17.028374+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'31","last_scrub_stamp":"2024-02-05T08:24:26.157978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:24:26.157978+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:34.179674+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"402'10928","reported_seq":11098,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:18.277952+0000","last_change":"2024-02-05T08:24:42.683152+0000","last_active":"2024-02-05T08:25:18.277952+0000","last_peered":"2024-02-05T08:25:18.277952+0000","last_clean":"2024-02-05T08:25:18.277952+0000","last_became_active":"2024-02-05T08:23:08.920062+0000","last_became_peered":"2024-02-05T08:23:08.920062+0000","last_unstale":"2024-02-05T08:25:18.277952+0000","last_undegraded":"2024-02-05T08:25:18.277952+0000","last_fullsized":"2024-02-05T08:25:18.277952+0000","mapping_epoch":367,"log_start":"389'5900","ondisk_log_start":"389'5900","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'7992","last_scrub_stamp":"2024-02-05T08:24:42.682996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:24:42.682996+0000","objects_scrubbed":7922,"log_size":5028,"log_dups_size":0,"ondisk_log_size":5028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:44.041356+0000","scrub_duration":2276,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405143552,"num_objects":6182,"num_object_clones":0,"num_object_copies":18546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10928,"num_write_kb":547520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"82'130","reported_seq":1059,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.034212+0000","last_change":"2024-02-05T08:24:22.941847+0000","last_active":"2024-02-05T08:25:17.034212+0000","last_peered":"2024-02-05T08:25:17.034212+0000","last_clean":"2024-02-05T08:25:17.034212+0000","last_became_active":"2024-02-05T08:19:01.816744+0000","last_became_peered":"2024-02-05T08:19:01.816744+0000","last_unstale":"2024-02-05T08:25:17.034212+0000","last_undegraded":"2024-02-05T08:25:17.034212+0000","last_fullsized":"2024-02-05T08:25:17.034212+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:24:22.941728+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:24:22.941728+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:35.303846+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":742,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.028475+0000","last_change":"2024-02-05T08:24:19.346422+0000","last_active":"2024-02-05T08:25:17.028475+0000","last_peered":"2024-02-05T08:25:17.028475+0000","last_clean":"2024-02-05T08:25:17.028475+0000","last_became_active":"2024-02-05T08:23:07.937641+0000","last_became_peered":"2024-02-05T08:23:07.937641+0000","last_unstale":"2024-02-05T08:25:17.028475+0000","last_undegraded":"2024-02-05T08:25:17.028475+0000","last_fullsized":"2024-02-05T08:25:17.028475+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:19.346083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:24:19.346083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:42.751303+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"401'10","reported_seq":221,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.035872+0000","last_change":"2024-02-05T08:25:04.979307+0000","last_active":"2024-02-05T08:25:17.035872+0000","last_peered":"2024-02-05T08:25:17.035872+0000","last_clean":"2024-02-05T08:25:17.035872+0000","last_became_active":"2024-02-05T08:25:04.978603+0000","last_became_peered":"2024-02-05T08:25:04.978603+0000","last_unstale":"2024-02-05T08:25:17.035872+0000","last_undegraded":"2024-02-05T08:25:17.035872+0000","last_fullsized":"2024-02-05T08:25:17.035872+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'9","last_scrub_stamp":"2024-02-05T08:23:36.866292+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:23:36.866292+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:17.784298+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":269,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.041412+0000","last_change":"2024-02-05T08:24:46.140620+0000","last_active":"2024-02-05T08:25:17.041412+0000","last_peered":"2024-02-05T08:25:17.041412+0000","last_clean":"2024-02-05T08:25:17.041412+0000","last_became_active":"2024-02-05T08:23:08.928775+0000","last_became_peered":"2024-02-05T08:23:08.928775+0000","last_unstale":"2024-02-05T08:25:17.041412+0000","last_undegraded":"2024-02-05T08:25:17.041412+0000","last_fullsized":"2024-02-05T08:25:17.041412+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:46.140359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:46.140359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:15.077248+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"397'35","reported_seq":879,"reported_epoch":402,"state":"active+clean","last_fresh":"2024-02-05T08:25:17.028103+0000","last_change":"2024-02-05T08:23:03.301185+0000","last_active":"2024-02-05T08:25:17.028103+0000","last_peered":"2024-02-05T08:25:17.028103+0000","last_clean":"2024-02-05T08:25:17.028103+0000","last_became_active":"2024-02-05T08:18:34.637413+0000","last_became_peered":"2024-02-05T08:18:34.637413+0000","last_unstale":"2024-02-05T08:25:17.028103+0000","last_undegraded":"2024-02-05T08:25:17.028103+0000","last_fullsized":"2024-02-05T08:25:17.028103+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"363'25","last_scrub_stamp":"2024-02-05T08:23:03.301090+0000","last_deep_scrub":"363'25","last_deep_scrub_stamp":"2024-02-05T08:23:03.301090+0000","last_clean_scrub_stamp":"2024-02-05T08:23:03.301090+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:25:20.174722+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":6344212504,"num_objects":96806,"num_object_clones":0,"num_object_copies":290418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96806,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173394,"num_write_kb":8646337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19034681344,"data_stored":19034669128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69953,"ondisk_log_size":69953,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1219,"num_read_kb":1214,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1542,"num_read_kb":1299,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":6562,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1071,"ondisk_log_size":1071,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038012,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3518604,"kb_used_data":2761344,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":90226548,"statfs":{"total":95995035648,"available":92391985152,"internally_reserved":0,"allocated":2827616256,"data_stored":2827250938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":775420362},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:25:05 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.2609999999999999,"15min":1.4490000000000001},"min":{"1min":0.28799999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.5650000000000004,"5min":20.344999999999999,"15min":52.488999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":1.208,"15min":1.508},"min":{"1min":0.27500000000000002,"5min":0.24099999999999999,"15min":0.23200000000000001},"max":{"1min":4.423,"5min":21.347000000000001,"15min":52.335000000000001},"last":0.89100000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:25:13 2024","interfaces":[{"interface":"back","average":{"1min":2.464,"5min":1.4390000000000001,"15min":1.268},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":31.495999999999999,"5min":31.495999999999999,"15min":31.495999999999999},"last":1.339},{"interface":"front","average":{"1min":2.75,"5min":1.5640000000000001,"15min":1.3660000000000001},"min":{"1min":0.309,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":31.547000000000001,"5min":31.547000000000001,"15min":31.547000000000001},"last":1.286}]},{"osd":2,"last update":"Mon Feb 5 08:25:05 2024","interfaces":[{"interface":"back","average":{"1min":3.5030000000000001,"5min":1.718,"15min":1.3009999999999999},"min":{"1min":0.248,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":31.585999999999999,"5min":31.585999999999999,"15min":31.585999999999999},"last":1.714},{"interface":"front","average":{"1min":3.5529999999999999,"5min":1.7709999999999999,"15min":1.3380000000000001},"min":{"1min":0.371,"5min":0.248,"15min":0.248},"max":{"1min":31.614999999999998,"5min":31.614999999999998,"15min":31.614999999999998},"last":1.3029999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:25:05 2024","interfaces":[{"interface":"back","average":{"1min":3.3780000000000001,"5min":1.4670000000000001,"15min":1.1899999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.249},"max":{"1min":31.780000000000001,"5min":31.780000000000001,"15min":31.780000000000001},"last":1.054},{"interface":"front","average":{"1min":3.2269999999999999,"5min":1.4339999999999999,"15min":1.2130000000000001},"min":{"1min":0.31900000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":31.795999999999999,"5min":31.795999999999999,"15min":31.795999999999999},"last":1.0069999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:25:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.2030000000000001,"15min":1.1990000000000001},"min":{"1min":0.28899999999999998,"5min":0.24399999999999999,"15min":0.23300000000000001},"max":{"1min":6.5289999999999999,"5min":6.5289999999999999,"15min":49.601999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.204,"15min":1.2350000000000001},"min":{"1min":0.31,"5min":0.249,"15min":0.249},"max":{"1min":3.9279999999999999,"5min":4.9290000000000003,"15min":49.795999999999999},"last":0.97499999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:25:05 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.099,"15min":1.236},"min":{"1min":0.308,"5min":0.26800000000000002,"15min":0.24299999999999999},"max":{"1min":9.6980000000000004,"5min":9.6980000000000004,"15min":49.914000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.333,"5min":1.121,"15min":1.256},"min":{"1min":0.36799999999999999,"5min":0.28999999999999998,"15min":0.27000000000000002},"max":{"1min":6.5410000000000004,"5min":8.7240000000000002,"15min":52.256},"last":0.38400000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:25:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.726,"15min":1.546},"min":{"1min":0.42499999999999999,"5min":0.316,"15min":0.28599999999999998},"max":{"1min":2.9980000000000002,"5min":23.800000000000001,"15min":84.254999999999995},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.831,"15min":1.577},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.27300000000000002},"max":{"1min":3.7250000000000001,"5min":29.742999999999999,"15min":84.667000000000002},"last":1.089}]}]},{"osd":6,"up_from":51,"seq":219043332354,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3782116,"kb_used_data":3141468,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":89963036,"statfs":{"total":95995035648,"available":92122148864,"internally_reserved":0,"allocated":3216863232,"data_stored":3216532501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":656013766},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:24:44 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.3320000000000001,"15min":1.8720000000000001},"min":{"1min":0.27100000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":2.2810000000000001,"5min":14.836,"15min":102.819},"last":2.645},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.87,"15min":1.728},"min":{"1min":0.27600000000000002,"5min":0.22500000000000001,"15min":0.214},"max":{"1min":1.9890000000000001,"5min":3.7850000000000001,"15min":102.75},"last":2.7240000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:25:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.905,"15min":1.9570000000000001},"min":{"1min":0.374,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":10.025,"5min":25.898,"15min":25.898},"last":2.6909999999999998},{"interface":"front","average":{"1min":1.7909999999999999,"5min":2.3210000000000002,"15min":2.4089999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":10.236000000000001,"5min":25.884,"15min":25.884},"last":2.7069999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:24:44 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.4159999999999999,"15min":1.2310000000000001},"min":{"1min":0.29399999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":6.6550000000000002,"5min":25.751000000000001,"15min":25.751000000000001},"last":2.5030000000000001},{"interface":"front","average":{"1min":1.202,"5min":1.282,"15min":1.157},"min":{"1min":0.38200000000000001,"5min":0.28199999999999997,"15min":0.23400000000000001},"max":{"1min":6.3949999999999996,"5min":26.210999999999999,"15min":26.210999999999999},"last":2.7530000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:24:44 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.542,"15min":1.496},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":4.2169999999999996,"5min":27.113,"15min":102.438},"last":2.6709999999999998},{"interface":"front","average":{"1min":1.028,"5min":1.3999999999999999,"15min":1.458},"min":{"1min":0.32700000000000001,"5min":0.29299999999999998,"15min":0.29099999999999998},"max":{"1min":2.944,"5min":27.209,"15min":102.288},"last":2.46}]},{"osd":4,"last update":"Mon Feb 5 08:24:44 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":1.849,"15min":1.159},"min":{"1min":0.26700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.2440000000000002,"5min":46.125,"15min":46.125},"last":2.891},{"interface":"front","average":{"1min":1.1319999999999999,"5min":2.2690000000000001,"15min":1.3160000000000001},"min":{"1min":0.314,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":5.5039999999999996,"5min":48.817999999999998,"15min":48.817999999999998},"last":2.617}]},{"osd":5,"last update":"Mon Feb 5 08:24:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":1.042,"15min":1.23},"min":{"1min":0.35199999999999998,"5min":0.26000000000000001,"15min":0.24199999999999999},"max":{"1min":1.085,"5min":7.6760000000000002,"15min":43.643000000000001},"last":3.0840000000000001},{"interface":"front","average":{"1min":1.0269999999999999,"5min":1.3029999999999999,"15min":1.2869999999999999},"min":{"1min":0.36499999999999999,"5min":0.246,"15min":0.246},"max":{"1min":3.2959999999999998,"5min":9.2279999999999998,"15min":43.893000000000001},"last":3.1459999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:24:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.999,"15min":1.079},"min":{"1min":0.38300000000000001,"5min":0.307,"15min":0.307},"max":{"1min":1.643,"5min":4.7530000000000001,"15min":24.728999999999999},"last":2.98},{"interface":"front","average":{"1min":1.151,"5min":1.498,"15min":1.266},"min":{"1min":0.437,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.6420000000000003,"5min":14.988,"15min":21.91},"last":3.1909999999999998}]}]},{"osd":1,"up_from":366,"seq":1571958030364,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3610536,"kb_used_data":3419808,"kb_used_omap":4,"kb_used_meta":190651,"kb_avail":90134616,"statfs":{"total":95995035648,"available":92297846784,"internally_reserved":0,"allocated":3501883392,"data_stored":3501485170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":195227302},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:25:13 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":0.78500000000000003,"15min":0.67400000000000004},"min":{"1min":0.29199999999999998,"5min":0.222,"15min":0.222},"max":{"1min":6.5999999999999996,"5min":6.5999999999999996,"15min":6.5999999999999996},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.96899999999999997,"5min":1.073,"15min":1.091},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":7.1420000000000003,"5min":7.6029999999999998,"15min":7.6029999999999998},"last":2.7919999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:25:13 2024","interfaces":[{"interface":"back","average":{"1min":1.863,"5min":1.6379999999999999,"15min":1.6000000000000001},"min":{"1min":0.40799999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":7.3289999999999997,"5min":7.3289999999999997,"15min":7.3289999999999997},"last":2.4790000000000001},{"interface":"front","average":{"1min":2.2229999999999999,"5min":1.4590000000000001,"15min":1.3320000000000001},"min":{"1min":0.39300000000000002,"5min":0.254,"15min":0.254},"max":{"1min":6.6589999999999998,"5min":6.6589999999999998,"15min":6.6589999999999998},"last":1.1910000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:25:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.2989999999999999,"15min":1.276},"min":{"1min":0.47199999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":7.3719999999999999,"5min":7.3719999999999999,"15min":7.3719999999999999},"last":2.5190000000000001},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.2949999999999999,"15min":1.23},"min":{"1min":0.32700000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":6.5650000000000004,"5min":6.5650000000000004,"15min":6.5650000000000004},"last":5.0590000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:25:13 2024","interfaces":[{"interface":"back","average":{"1min":1.8700000000000001,"5min":1.383,"15min":1.302},"min":{"1min":0.39400000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":6.3920000000000003,"5min":6.3920000000000003,"15min":6.3920000000000003},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2150000000000001,"15min":1.2010000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":7.1890000000000001,"5min":7.1890000000000001,"15min":7.1890000000000001},"last":2.7530000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:25:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.1559999999999999,"15min":1.1180000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":4.7800000000000002,"5min":4.7800000000000002,"15min":4.7800000000000002},"last":4.9749999999999996},{"interface":"front","average":{"1min":1.7589999999999999,"5min":1.2110000000000001,"15min":1.119},"min":{"1min":0.40400000000000003,"5min":0.311,"15min":0.311},"max":{"1min":6.4240000000000004,"5min":6.4240000000000004,"15min":6.4240000000000004},"last":1.22}]},{"osd":6,"last update":"Mon Feb 5 08:25:13 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.26,"15min":1.22},"min":{"1min":0.434,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.0789999999999997,"5min":5.0789999999999997,"15min":5.0789999999999997},"last":5.0220000000000002},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.2989999999999999,"15min":1.2490000000000001},"min":{"1min":0.50700000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":6.3280000000000003,"5min":6.3280000000000003,"15min":6.3280000000000003},"last":3.302}]},{"osd":7,"last update":"Mon Feb 5 08:25:13 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.26,"15min":1.2170000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":7.2160000000000002,"5min":7.2160000000000002,"15min":7.2160000000000002},"last":2.5920000000000001},{"interface":"front","average":{"1min":1.663,"5min":1.4419999999999999,"15min":1.405},"min":{"1min":0.46200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":5.1180000000000003,"5min":6.9189999999999996,"15min":6.9189999999999996},"last":5.0899999999999999}]}]},{"osd":0,"up_from":0,"seq":34359738656,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411606,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3514440,"kb_used_data":2748864,"kb_used_omap":1,"kb_used_meta":765566,"kb_avail":90230712,"statfs":{"total":95995035648,"available":92396249088,"internally_reserved":0,"allocated":2814836736,"data_stored":2814470628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":783940038},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:25:05 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.72299999999999998,"15min":1.8500000000000001},"min":{"1min":0.20100000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2.3370000000000002,"5min":6.181,"15min":107.155},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.92500000000000004,"15min":2.0369999999999999},"min":{"1min":0.27200000000000002,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.1429999999999998,"5min":7.3620000000000001,"15min":107.221},"last":0.41799999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:25:10 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.018,"15min":0.97799999999999998},"min":{"1min":0.27200000000000002,"5min":0.19,"15min":0.19},"max":{"1min":6.1719999999999997,"5min":6.1719999999999997,"15min":6.1719999999999997},"last":1.403},{"interface":"front","average":{"1min":1.591,"5min":1.335,"15min":1.2929999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":7.0709999999999997,"5min":7.0709999999999997,"15min":7.0709999999999997},"last":1.2150000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:24:26 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.018,"15min":1.4850000000000001},"min":{"1min":0.26400000000000001,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":3.5750000000000002,"5min":6.5039999999999996,"15min":68.108000000000004},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.2889999999999999,"5min":0.88700000000000001,"15min":1.351},"min":{"1min":0.317,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.0459999999999998,"5min":6.101,"15min":67.945999999999998},"last":0.95499999999999996}]},{"osd":4,"last update":"Mon Feb 5 08:24:55 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.304,"15min":1.4399999999999999},"min":{"1min":0.47499999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.923,"5min":7.6189999999999998,"15min":31.596},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.296,"5min":1.2889999999999999,"15min":1.3420000000000001},"min":{"1min":0.307,"5min":0.28799999999999998,"15min":0.27200000000000002},"max":{"1min":2.6659999999999999,"5min":7.6509999999999998,"15min":31.57},"last":0.98199999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:25:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.165,"15min":1.575},"min":{"1min":0.33800000000000002,"5min":0.308,"15min":0.246},"max":{"1min":2.71,"5min":9.516,"15min":58.799999999999997},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.327,"15min":1.615},"min":{"1min":0.32900000000000001,"5min":0.27300000000000002,"15min":0.26900000000000002},"max":{"1min":3.339,"5min":9.3209999999999997,"15min":58.615000000000002},"last":1.244}]},{"osd":6,"last update":"Mon Feb 5 08:24:55 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.5569999999999999,"15min":1.339},"min":{"1min":0.35899999999999999,"5min":0.307,"15min":0.29199999999999998},"max":{"1min":4.0140000000000002,"5min":21.885000000000002,"15min":21.885000000000002},"last":0.49399999999999999},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.4990000000000001,"15min":1.2709999999999999},"min":{"1min":0.66200000000000003,"5min":0.317,"15min":0.317},"max":{"1min":6.1230000000000002,"5min":22.158000000000001,"15min":22.158000000000001},"last":1.107}]},{"osd":7,"last update":"Mon Feb 5 08:25:05 2024","interfaces":[{"interface":"back","average":{"1min":1.7290000000000001,"5min":2.0880000000000001,"15min":1.536},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.28899999999999998},"max":{"1min":5.8499999999999996,"5min":72.334999999999994,"15min":72.334999999999994},"last":4.0090000000000003},{"interface":"front","average":{"1min":1.73,"5min":2.2389999999999999,"15min":1.7669999999999999},"min":{"1min":0.47199999999999998,"5min":0.35699999999999998,"15min":0.32700000000000001},"max":{"1min":5.7220000000000004,"5min":72.222999999999999,"15min":72.222999999999999},"last":4.0350000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117265,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3414304,"kb_used_data":2751444,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":90330848,"statfs":{"total":95995035648,"available":92498788352,"internally_reserved":0,"allocated":2817478656,"data_stored":2817093776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":678754761},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:24:34 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.95899999999999996,"15min":1.4770000000000001},"min":{"1min":0.20300000000000001,"5min":0.20100000000000001,"15min":0.19700000000000001},"max":{"1min":1.8779999999999999,"5min":11.747999999999999,"15min":52.831000000000003},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.88100000000000001,"15min":1.5880000000000001},"min":{"1min":0.28599999999999998,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.3540000000000001,"5min":11.618,"15min":53.859999999999999},"last":0.39900000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:25:14 2024","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4359999999999999,"15min":1.4259999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":6.4509999999999996,"5min":6.4509999999999996,"15min":6.4509999999999996},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.552,"15min":1.544},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":6.5789999999999997,"5min":6.5789999999999997,"15min":6.5789999999999997},"last":0.435}]},{"osd":2,"last update":"Mon Feb 5 08:24:34 2024","interfaces":[{"interface":"back","average":{"1min":1.9199999999999999,"5min":1.609,"15min":1.4059999999999999},"min":{"1min":0.28599999999999998,"5min":0.22,"15min":0.22},"max":{"1min":6.5999999999999996,"5min":35.277000000000001,"15min":35.277000000000001},"last":1.611},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.48,"15min":1.2110000000000001},"min":{"1min":0.247,"5min":0.23300000000000001,"15min":0.23200000000000001},"max":{"1min":5.6509999999999998,"5min":35.442999999999998,"15min":35.442999999999998},"last":0.48199999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:24:49 2024","interfaces":[{"interface":"back","average":{"1min":1.8240000000000001,"5min":1.8069999999999999,"15min":1.252},"min":{"1min":0.41399999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":6.391,"5min":54.502000000000002,"15min":54.502000000000002},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.605,"15min":1.212},"min":{"1min":0.41699999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.4320000000000004,"5min":54.540999999999997,"15min":54.540999999999997},"last":1.367}]},{"osd":5,"last update":"Mon Feb 5 08:25:14 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.2290000000000001,"15min":1.7789999999999999},"min":{"1min":0.34200000000000003,"5min":0.28899999999999998,"15min":0.28000000000000003},"max":{"1min":6.4219999999999997,"5min":6.4219999999999997,"15min":117.432},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.135,"5min":0.90600000000000003,"15min":1.671},"min":{"1min":0.39700000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.0499999999999998,"5min":3.3180000000000001,"15min":114.828},"last":1.333}]},{"osd":6,"last update":"Mon Feb 5 08:24:34 2024","interfaces":[{"interface":"back","average":{"1min":1.7629999999999999,"5min":1.7350000000000001,"15min":1.3100000000000001},"min":{"1min":0.45400000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":9.6129999999999995,"5min":54.219999999999999,"15min":54.219999999999999},"last":2.1419999999999999},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.863,"15min":1.397},"min":{"1min":0.32700000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":6.6820000000000004,"5min":54.530999999999999,"15min":54.530999999999999},"last":2.121}]},{"osd":7,"last update":"Mon Feb 5 08:24:58 2024","interfaces":[{"interface":"back","average":{"1min":1.8620000000000001,"5min":1.2869999999999999,"15min":1.726},"min":{"1min":0.36899999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":6.4800000000000004,"5min":6.4800000000000004,"15min":114.84999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":2.2029999999999998,"5min":1.244,"15min":1.6859999999999999},"min":{"1min":0.51500000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":5.891,"5min":5.891,"15min":114.871},"last":1.403}]}]},{"osd":4,"up_from":32,"seq":137438953740,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3093316,"kb_used_data":2363768,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":90651836,"statfs":{"total":95995035648,"available":92827480064,"internally_reserved":0,"allocated":2420498432,"data_stored":2420139601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":746977733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:24:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.131,"15min":1.3149999999999999},"min":{"1min":0.23599999999999999,"5min":0.192,"15min":0.192},"max":{"1min":2.4279999999999999,"5min":8.6470000000000002,"15min":39.552},"last":3.133},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.26,"15min":1.4419999999999999},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.21199999999999999},"max":{"1min":5.5229999999999997,"5min":5.5229999999999997,"15min":39.350999999999999},"last":1.036}]},{"osd":1,"last update":"Mon Feb 5 08:25:07 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0049999999999999,"15min":1.01},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.7869999999999999,"5min":3.915,"15min":3.915},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.95399999999999996,"15min":0.95999999999999996},"min":{"1min":0.26800000000000002,"5min":0.254,"15min":0.254},"max":{"1min":2.2709999999999999,"5min":2.899,"15min":2.899},"last":3.1019999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:24:38 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.3120000000000001,"15min":1.111},"min":{"1min":0.28299999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":6.1749999999999998,"5min":6.1749999999999998,"15min":6.1749999999999998},"last":3.1629999999999998},{"interface":"front","average":{"1min":1.647,"5min":1.298,"15min":1.1419999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.26300000000000001},"max":{"1min":8.8490000000000002,"5min":8.8490000000000002,"15min":8.8490000000000002},"last":3.4390000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:24:38 2024","interfaces":[{"interface":"back","average":{"1min":1.748,"5min":1.526,"15min":1.647},"min":{"1min":0.40799999999999997,"5min":0.312,"15min":0.24399999999999999},"max":{"1min":4.5869999999999997,"5min":8.9269999999999996,"15min":57.204000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3280000000000001,"15min":1.6020000000000001},"min":{"1min":0.32500000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.266,"5min":4.2599999999999998,"15min":57.606999999999999},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:24:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.3280000000000001,"15min":1.171},"min":{"1min":0.26100000000000001,"5min":0.24199999999999999,"15min":0.224},"max":{"1min":3.4039999999999999,"5min":6.1909999999999998,"15min":34.167000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.71299999999999997,"5min":1.2509999999999999,"15min":1.2010000000000001},"min":{"1min":0.375,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.1099999999999999,"5min":7.851,"15min":36.051000000000002},"last":1.006}]},{"osd":6,"last update":"Mon Feb 5 08:24:38 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.5429999999999999,"15min":1.097},"min":{"1min":0.46700000000000003,"5min":0.32600000000000001,"15min":0.26900000000000002},"max":{"1min":3.2709999999999999,"5min":6.375,"15min":6.375},"last":3.476},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.466,"15min":1.03},"min":{"1min":0.378,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":6.0179999999999998,"5min":6.9690000000000003,"15min":6.9690000000000003},"last":0.64800000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:24:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.345,"15min":1.0940000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.27000000000000002},"max":{"1min":2.863,"5min":12.683999999999999,"15min":12.683999999999999},"last":3.0640000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.3859999999999999,"15min":1.085},"min":{"1min":0.40200000000000002,"5min":0.35099999999999998,"15min":0.27900000000000003},"max":{"1min":2.472,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":0.627}]}]},{"osd":5,"up_from":42,"seq":180388626694,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2291516,"kb_used_data":1566192,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":91453636,"statfs":{"total":95995035648,"available":93648523264,"internally_reserved":0,"allocated":1603780608,"data_stored":1603431734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.97999999999999998,"15min":1.5349999999999999},"min":{"1min":0.25800000000000001,"5min":0.22,"15min":0.20799999999999999},"max":{"1min":4.7290000000000001,"5min":6.4660000000000002,"15min":46.962000000000003},"last":0.27300000000000002},{"interface":"front","average":{"1min":1.012,"5min":1.093,"15min":1.4930000000000001},"min":{"1min":0.26900000000000002,"5min":0.26600000000000001,"15min":0.23699999999999999},"max":{"1min":4.1500000000000004,"5min":5.0819999999999999,"15min":47.067},"last":0.38600000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:25:09 2024","interfaces":[{"interface":"back","average":{"1min":1.167,"5min":2.3610000000000002,"15min":2.5600000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.9580000000000002,"5min":24.606999999999999,"15min":24.606999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":1.0569999999999999,"5min":2.2589999999999999,"15min":2.4590000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.2130000000000001,"5min":24.861000000000001,"15min":24.861000000000001},"last":0.30599999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":1.2330000000000001,"15min":1.2},"min":{"1min":0.29399999999999998,"5min":0.245,"15min":0.245},"max":{"1min":2.7599999999999998,"5min":17.09,"15min":17.09},"last":0.58499999999999996},{"interface":"front","average":{"1min":1.103,"5min":1.325,"15min":1.2949999999999999},"min":{"1min":0.29699999999999999,"5min":0.26100000000000001,"15min":0.25600000000000001},"max":{"1min":3.7320000000000002,"5min":17.515999999999998,"15min":17.515999999999998},"last":0.498}]},{"osd":3,"last update":"Mon Feb 5 08:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.151,"15min":1.3260000000000001},"min":{"1min":0.32900000000000001,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":3.5329999999999999,"5min":7.6950000000000003,"15min":46.569000000000003},"last":0.622},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0609999999999999,"15min":1.4570000000000001},"min":{"1min":0.34899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.843,"5min":6.5519999999999996,"15min":47.820999999999998},"last":0.46200000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":1.0840000000000001,"15min":1.0089999999999999},"min":{"1min":0.26900000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.0999999999999996,"5min":7.7389999999999999,"15min":7.7389999999999999},"last":2.4399999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":1.204,"15min":1.048},"min":{"1min":0.34200000000000003,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.95,"5min":7.609,"15min":7.609},"last":0.36299999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:24:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.5229999999999999,"15min":1.181},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.26600000000000001},"max":{"1min":5.1500000000000004,"5min":30.581,"15min":30.581},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.107,"5min":1.51,"15min":1.1679999999999999},"min":{"1min":0.34200000000000003,"5min":0.32900000000000001,"15min":0.316},"max":{"1min":5.1909999999999998,"5min":30.620999999999999,"15min":30.620999999999999},"last":2.6110000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:25:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":1.478,"15min":1.3160000000000001},"min":{"1min":0.34999999999999998,"5min":0.316,"15min":0.28799999999999998},"max":{"1min":4.5869999999999997,"5min":34.695,"15min":34.695},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.81699999999999995,"5min":1.4610000000000001,"15min":1.252},"min":{"1min":0.33700000000000002,"5min":0.308,"15min":0.308},"max":{"1min":3.9079999999999999,"5min":34.514000000000003,"15min":34.514000000000003},"last":2.4039999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3456434176,"data_stored":3456434176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2788691968,"data_stored":2788687896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2805137408,"data_stored":2805137408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2394230784,"data_stored":2394226712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1583087616,"data_stored":1583087616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3206221824,"data_stored":3206217752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2800877568,"data_stored":2800877568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:25:24.881 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-02-05T08:25:24.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:24.899 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:24.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:25.000 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:25.036 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:25.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:25.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:25.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:24 smithi134 ceph-mon[30086]: 11.c scrub ok 2024-02-05T08:25:25.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:24 smithi134 ceph-mon[30086]: pgmap v924: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-02-05T08:25:25.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:24 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:25.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:24 smithi134 ceph-mon[30086]: 3.a scrub starts 2024-02-05T08:25:25.127 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:24 smithi134 ceph-mon[30086]: 3.a scrub ok 2024-02-05T08:25:25.128 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:24 smithi134 ceph-mon[39278]: 11.c scrub ok 2024-02-05T08:25:25.128 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:24 smithi134 ceph-mon[39278]: pgmap v924: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-02-05T08:25:25.128 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:24 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:25.128 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:24 smithi134 ceph-mon[39278]: 3.a scrub starts 2024-02-05T08:25:25.128 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:24 smithi134 ceph-mon[39278]: 3.a scrub ok 2024-02-05T08:25:25.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:25.174 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:25.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:25.275 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:25.312 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:25.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:24 smithi196 ceph-mon[33018]: 11.c scrub ok 2024-02-05T08:25:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:24 smithi196 ceph-mon[33018]: pgmap v924: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-02-05T08:25:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:24 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:24 smithi196 ceph-mon[33018]: 3.a scrub starts 2024-02-05T08:25:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:24 smithi196 ceph-mon[33018]: 3.a scrub ok 2024-02-05T08:25:25.414 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:25.453 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:25.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:25.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:25.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:25.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:25.592 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:25.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:25.693 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:25.734 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:25.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:25.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:25.836 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:25.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:25.871 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:25.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:25.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:26.013 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:26.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:26.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:26.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:26.155 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:26.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:26.256 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:25 smithi134 ceph-mon[30086]: 3.1b scrub starts 2024-02-05T08:25:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:25 smithi134 ceph-mon[30086]: 3.1b scrub ok 2024-02-05T08:25:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:25 smithi134 ceph-mon[30086]: 3.c scrub starts 2024-02-05T08:25:26.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:25 smithi134 ceph-mon[30086]: 3.c scrub ok 2024-02-05T08:25:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:25 smithi134 ceph-mon[39278]: 3.1b scrub starts 2024-02-05T08:25:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:25 smithi134 ceph-mon[39278]: 3.1b scrub ok 2024-02-05T08:25:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:25 smithi134 ceph-mon[39278]: 3.c scrub starts 2024-02-05T08:25:26.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:25 smithi134 ceph-mon[39278]: 3.c scrub ok 2024-02-05T08:25:26.290 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:26.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:25 smithi196 ceph-mon[33018]: 3.1b scrub starts 2024-02-05T08:25:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:25 smithi196 ceph-mon[33018]: 3.1b scrub ok 2024-02-05T08:25:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:25 smithi196 ceph-mon[33018]: 3.c scrub starts 2024-02-05T08:25:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:25 smithi196 ceph-mon[33018]: 3.c scrub ok 2024-02-05T08:25:26.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:26.430 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:26.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:26.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:26.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:26.568 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:26.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:26.670 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:26.704 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:26.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:26.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:26.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:26.843 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:26.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:26.944 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:26.981 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:26.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:27.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:27.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:27.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:27.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:27.227 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:26 smithi134 ceph-mon[30086]: pgmap v925: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-02-05T08:25:27.227 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:26 smithi134 ceph-mon[30086]: 3.11 scrub starts 2024-02-05T08:25:27.227 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:26 smithi134 ceph-mon[30086]: 3.11 scrub ok 2024-02-05T08:25:27.228 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:26 smithi134 ceph-mon[39278]: pgmap v925: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-02-05T08:25:27.228 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:26 smithi134 ceph-mon[39278]: 3.11 scrub starts 2024-02-05T08:25:27.228 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:26 smithi134 ceph-mon[39278]: 3.11 scrub ok 2024-02-05T08:25:27.253 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:27.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:27.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:26 smithi196 ceph-mon[33018]: pgmap v925: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-02-05T08:25:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:26 smithi196 ceph-mon[33018]: 3.11 scrub starts 2024-02-05T08:25:27.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:26 smithi196 ceph-mon[33018]: 3.11 scrub ok 2024-02-05T08:25:27.355 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:27.399 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:27.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:27.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:27.501 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:27.542 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:27.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:27.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:27.663 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:25:27.663 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:25:27.663 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:25:27.663 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:25:27.663 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:25:27.663 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:25:27.679 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:27.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:27.687 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:25:27.731 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:25:27.734 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:25:27.734 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:25:27.734 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:25:27.734 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:25:27.734 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:25:27.734 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:25:27.734 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:25:27.734 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:25:27.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:27.823 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:27.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:27.925 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:27.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:27.961 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:27.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:27.991 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:25:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:25:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:25:28.062 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:28.098 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:28.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:28.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:28.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:28.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:28.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:27 smithi134 ceph-mon[30086]: 3.1f scrub starts 2024-02-05T08:25:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:27 smithi134 ceph-mon[30086]: 3.1f scrub ok 2024-02-05T08:25:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:27 smithi134 ceph-mon[30086]: 5.14 scrub starts 2024-02-05T08:25:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:27 smithi134 ceph-mon[30086]: 5.14 scrub ok 2024-02-05T08:25:28.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:27 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1234534714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:25:28.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:27 smithi134 ceph-mon[39278]: 3.1f scrub starts 2024-02-05T08:25:28.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:27 smithi134 ceph-mon[39278]: 3.1f scrub ok 2024-02-05T08:25:28.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:27 smithi134 ceph-mon[39278]: 5.14 scrub starts 2024-02-05T08:25:28.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:27 smithi134 ceph-mon[39278]: 5.14 scrub ok 2024-02-05T08:25:28.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:27 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1234534714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:25:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:27 smithi196 ceph-mon[33018]: 3.1f scrub starts 2024-02-05T08:25:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:27 smithi196 ceph-mon[33018]: 3.1f scrub ok 2024-02-05T08:25:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:27 smithi196 ceph-mon[33018]: 5.14 scrub starts 2024-02-05T08:25:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:27 smithi196 ceph-mon[33018]: 5.14 scrub ok 2024-02-05T08:25:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:27 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1234534714' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:25:28.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:28.377 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:28.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:28.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:28.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:28.513 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:28.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:28.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:28.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:28.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:28.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:28.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:28.757 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:28.814 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:28.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:28.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:28.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:28.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:28.959 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:28.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:29.060 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:29.097 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:29.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:29.199 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:29.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:29.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:29.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:29.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:28 smithi134 ceph-mon[30086]: 1.3 scrub starts 2024-02-05T08:25:29.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:28 smithi134 ceph-mon[30086]: 1.3 scrub ok 2024-02-05T08:25:29.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:28 smithi134 ceph-mon[30086]: 11.7 scrub starts 2024-02-05T08:25:29.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:28 smithi134 ceph-mon[30086]: pgmap v926: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 938 B/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T08:25:29.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:28 smithi134 ceph-mon[30086]: 11.5 scrub starts 2024-02-05T08:25:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:28 smithi134 ceph-mon[39278]: 1.3 scrub starts 2024-02-05T08:25:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:28 smithi134 ceph-mon[39278]: 1.3 scrub ok 2024-02-05T08:25:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:28 smithi134 ceph-mon[39278]: 11.7 scrub starts 2024-02-05T08:25:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:28 smithi134 ceph-mon[39278]: pgmap v926: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 938 B/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T08:25:29.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:28 smithi134 ceph-mon[39278]: 11.5 scrub starts 2024-02-05T08:25:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:28 smithi196 ceph-mon[33018]: 1.3 scrub starts 2024-02-05T08:25:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:28 smithi196 ceph-mon[33018]: 1.3 scrub ok 2024-02-05T08:25:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:28 smithi196 ceph-mon[33018]: 11.7 scrub starts 2024-02-05T08:25:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:28 smithi196 ceph-mon[33018]: pgmap v926: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 938 B/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T08:25:29.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:28 smithi196 ceph-mon[33018]: 11.5 scrub starts 2024-02-05T08:25:29.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:29.372 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:25:29.373 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:25:29.379 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:29.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:29.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:29.481 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:29.517 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:29.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:29.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:29.656 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:29.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:29.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:29.791 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:29.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:29.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:29.882 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 7, 6, 3] out_osds: [0] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-02-05T08:25:29.882 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:25:29.882 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-02-05T08:25:29.883 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-05T08:25:29.883 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-05T08:25:29.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-02-05T08:25:29.893 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:29.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:29.976 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:29.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:30.077 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:30.108 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:30.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:30.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:30.211 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:30.212 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:29 smithi134 ceph-mon[30086]: 5.1 scrub starts 2024-02-05T08:25:30.213 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:29 smithi134 ceph-mon[30086]: 5.1 scrub ok 2024-02-05T08:25:30.213 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:29 smithi134 ceph-mon[30086]: 3.18 scrub starts 2024-02-05T08:25:30.213 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:29 smithi134 ceph-mon[30086]: 3.18 scrub ok 2024-02-05T08:25:30.213 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:29 smithi134 ceph-mon[30086]: 11.7 scrub ok 2024-02-05T08:25:30.213 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:29 smithi134 ceph-mon[30086]: 11.f scrub starts 2024-02-05T08:25:30.213 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:29 smithi134 ceph-mon[39278]: 5.1 scrub starts 2024-02-05T08:25:30.213 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:29 smithi134 ceph-mon[39278]: 5.1 scrub ok 2024-02-05T08:25:30.213 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:29 smithi134 ceph-mon[39278]: 3.18 scrub starts 2024-02-05T08:25:30.213 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:29 smithi134 ceph-mon[39278]: 3.18 scrub ok 2024-02-05T08:25:30.213 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:29 smithi134 ceph-mon[39278]: 11.7 scrub ok 2024-02-05T08:25:30.214 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:29 smithi134 ceph-mon[39278]: 11.f scrub starts 2024-02-05T08:25:30.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:30.259 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:30.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:30.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T08:25:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:29 smithi196 ceph-mon[33018]: 5.1 scrub starts 2024-02-05T08:25:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:29 smithi196 ceph-mon[33018]: 5.1 scrub ok 2024-02-05T08:25:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:29 smithi196 ceph-mon[33018]: 3.18 scrub starts 2024-02-05T08:25:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:29 smithi196 ceph-mon[33018]: 3.18 scrub ok 2024-02-05T08:25:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:29 smithi196 ceph-mon[33018]: 11.7 scrub ok 2024-02-05T08:25:30.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:29 smithi196 ceph-mon[33018]: 11.f scrub starts 2024-02-05T08:25:30.361 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:30.384 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:25:30.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:30.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:30.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:30.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:30.578 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:30.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:30.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:30.680 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:30.726 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:30.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:30.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:30.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:30.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:30.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:30.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:31.013 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:31.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:30 smithi196 ceph-mon[33018]: 1.8 scrub starts 2024-02-05T08:25:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:30 smithi196 ceph-mon[33018]: 1.8 scrub ok 2024-02-05T08:25:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:30 smithi196 ceph-mon[33018]: pgmap v927: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-02-05T08:25:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:30 smithi196 ceph-mon[33018]: 3.2 scrub starts 2024-02-05T08:25:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:30 smithi196 ceph-mon[33018]: 3.2 scrub ok 2024-02-05T08:25:31.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:30 smithi196 ceph-mon[33018]: 11.5 scrub ok 2024-02-05T08:25:31.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:30 smithi196 ceph-mon[33018]: 1.2 scrub starts 2024-02-05T08:25:31.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:30 smithi196 ceph-mon[33018]: 1.2 scrub ok 2024-02-05T08:25:31.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:30 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/975631518' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:25:31.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:30 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:25:31.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:31.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:31.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:31.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:31.252 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:31.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[30086]: 1.8 scrub starts 2024-02-05T08:25:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[30086]: 1.8 scrub ok 2024-02-05T08:25:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[30086]: pgmap v927: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-02-05T08:25:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[30086]: 3.2 scrub starts 2024-02-05T08:25:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[30086]: 3.2 scrub ok 2024-02-05T08:25:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[30086]: 11.5 scrub ok 2024-02-05T08:25:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[30086]: 1.2 scrub starts 2024-02-05T08:25:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[30086]: 1.2 scrub ok 2024-02-05T08:25:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/975631518' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:25:31.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:25:31.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[39278]: 1.8 scrub starts 2024-02-05T08:25:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[39278]: 1.8 scrub ok 2024-02-05T08:25:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[39278]: pgmap v927: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-02-05T08:25:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[39278]: 3.2 scrub starts 2024-02-05T08:25:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[39278]: 3.2 scrub ok 2024-02-05T08:25:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[39278]: 11.5 scrub ok 2024-02-05T08:25:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[39278]: 1.2 scrub starts 2024-02-05T08:25:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[39278]: 1.2 scrub ok 2024-02-05T08:25:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/975631518' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:25:31.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:30 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:25:31.289 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:31.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:31.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:31.392 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:31.437 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:31.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:31.538 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:31.574 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:31.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:31.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:31.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:31.712 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:31.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:31.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:31.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:31.845 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:31.862 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:31.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:31.949 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:25:31.965 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:32.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:32.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:32.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:32.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:32 smithi196 ceph-mon[33018]: 4.15 scrub starts 2024-02-05T08:25:32.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:32 smithi196 ceph-mon[33018]: 4.15 scrub ok 2024-02-05T08:25:32.065 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:32 smithi196 ceph-mon[33018]: 3.7 scrub starts 2024-02-05T08:25:32.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:32 smithi196 ceph-mon[33018]: 3.7 scrub ok 2024-02-05T08:25:32.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:32 smithi196 ceph-mon[33018]: 5.11 scrub starts 2024-02-05T08:25:32.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:32 smithi196 ceph-mon[33018]: 5.11 scrub ok 2024-02-05T08:25:32.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:32 smithi196 ceph-mon[33018]: 11.f scrub ok 2024-02-05T08:25:32.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:32 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:25:32.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:32 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:25:32.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:32 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:25:32.066 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:32 smithi196 ceph-mon[33018]: osdmap e403: 8 total, 8 up, 7 in 2024-02-05T08:25:32.102 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:32.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:32.139 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:32.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:32.240 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:32.273 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:32.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:32.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:32.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[30086]: 4.15 scrub starts 2024-02-05T08:25:32.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[30086]: 4.15 scrub ok 2024-02-05T08:25:32.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[30086]: 3.7 scrub starts 2024-02-05T08:25:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[30086]: 3.7 scrub ok 2024-02-05T08:25:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[30086]: 5.11 scrub starts 2024-02-05T08:25:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[30086]: 5.11 scrub ok 2024-02-05T08:25:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[30086]: 11.f scrub ok 2024-02-05T08:25:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:25:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:25:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:25:32.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[30086]: osdmap e403: 8 total, 8 up, 7 in 2024-02-05T08:25:32.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[39278]: 4.15 scrub starts 2024-02-05T08:25:32.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[39278]: 4.15 scrub ok 2024-02-05T08:25:32.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[39278]: 3.7 scrub starts 2024-02-05T08:25:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[39278]: 3.7 scrub ok 2024-02-05T08:25:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[39278]: 5.11 scrub starts 2024-02-05T08:25:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[39278]: 5.11 scrub ok 2024-02-05T08:25:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[39278]: 11.f scrub ok 2024-02-05T08:25:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:25:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:25:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:25:32.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:32 smithi134 ceph-mon[39278]: osdmap e403: 8 total, 8 up, 7 in 2024-02-05T08:25:32.375 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:32.411 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:32.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:32.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:32.513 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:32.551 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:32.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:32.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:32.653 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:32.686 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:25:32.686 INFO:teuthology.orchestra.run.smithi196.stdout: "success": "" 2024-02-05T08:25:32.687 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:25:32.691 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:32.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:32.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:32.829 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:32.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:32.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:32.931 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:32.965 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:32.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:33.066 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:33.104 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:33.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:33.158 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[30086]: 6.2 scrub starts 2024-02-05T08:25:33.158 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[30086]: 6.2 scrub ok 2024-02-05T08:25:33.158 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[30086]: pgmap v929: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T08:25:33.158 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[30086]: 4.5 scrub starts 2024-02-05T08:25:33.158 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[30086]: 4.5 scrub ok 2024-02-05T08:25:33.158 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[30086]: 5.1a deep-scrub starts 2024-02-05T08:25:33.158 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[30086]: 5.1a deep-scrub ok 2024-02-05T08:25:33.158 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[39278]: 6.2 scrub starts 2024-02-05T08:25:33.158 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[39278]: 6.2 scrub ok 2024-02-05T08:25:33.159 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[39278]: pgmap v929: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T08:25:33.159 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[39278]: 4.5 scrub starts 2024-02-05T08:25:33.159 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[39278]: 4.5 scrub ok 2024-02-05T08:25:33.159 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[39278]: 5.1a deep-scrub starts 2024-02-05T08:25:33.159 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:33 smithi134 ceph-mon[39278]: 5.1a deep-scrub ok 2024-02-05T08:25:33.206 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:33.246 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:33.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:33 smithi196 ceph-mon[33018]: 6.2 scrub starts 2024-02-05T08:25:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:33 smithi196 ceph-mon[33018]: 6.2 scrub ok 2024-02-05T08:25:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:33 smithi196 ceph-mon[33018]: pgmap v929: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T08:25:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:33 smithi196 ceph-mon[33018]: 4.5 scrub starts 2024-02-05T08:25:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:33 smithi196 ceph-mon[33018]: 4.5 scrub ok 2024-02-05T08:25:33.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:33 smithi196 ceph-mon[33018]: 5.1a deep-scrub starts 2024-02-05T08:25:33.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:33 smithi196 ceph-mon[33018]: 5.1a deep-scrub ok 2024-02-05T08:25:33.347 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:33.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:25:33.383 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:33.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:33.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:33.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:33.546 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:33.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:33.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:33.683 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:33.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:33.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:33.784 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:33.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:33.823 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:33.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:33.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:33.957 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:33.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:33.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:34.059 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:34.105 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:34.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:34.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:34.137 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:34.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:34.240 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:34.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:34.249 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[30086]: 3.1c scrub starts 2024-02-05T08:25:34.249 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[30086]: 3.1c scrub ok 2024-02-05T08:25:34.249 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[30086]: 5.13 deep-scrub starts 2024-02-05T08:25:34.249 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[30086]: 5.13 deep-scrub ok 2024-02-05T08:25:34.249 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:25:34.249 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:25:34.249 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:25:34.249 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:25:34.250 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[39278]: 3.1c scrub starts 2024-02-05T08:25:34.250 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[39278]: 3.1c scrub ok 2024-02-05T08:25:34.250 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[39278]: 5.13 deep-scrub starts 2024-02-05T08:25:34.250 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[39278]: 5.13 deep-scrub ok 2024-02-05T08:25:34.250 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:25:34.250 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:25:34.250 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:25:34.250 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:34 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:25:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:34 smithi196 ceph-mon[33018]: 3.1c scrub starts 2024-02-05T08:25:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:34 smithi196 ceph-mon[33018]: 3.1c scrub ok 2024-02-05T08:25:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:34 smithi196 ceph-mon[33018]: 5.13 deep-scrub starts 2024-02-05T08:25:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:34 smithi196 ceph-mon[33018]: 5.13 deep-scrub ok 2024-02-05T08:25:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:34 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:25:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:34 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:25:34.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:34 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:25:34.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:34 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:25:34.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:25:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:25:34.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:34.375 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:34.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:34.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:34.512 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:34.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:34.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:34.614 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:34.656 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:34.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:34.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:34.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:34.793 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:34.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:34.895 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:34.936 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:34.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:34.939 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:25:34.939 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:25:34.953 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:25:35.037 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:35.080 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:35.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:35.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:35.227 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:35.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:35.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:35 smithi134 ceph-mon[30086]: 3.4 scrub starts 2024-02-05T08:25:35.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:35 smithi134 ceph-mon[30086]: 3.4 scrub ok 2024-02-05T08:25:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:35 smithi134 ceph-mon[30086]: pgmap v930: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T08:25:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:35 smithi134 ceph-mon[30086]: osdmap e404: 8 total, 8 up, 7 in 2024-02-05T08:25:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:35 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:35.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:35 smithi134 ceph-mon[30086]: 11.4 scrub starts 2024-02-05T08:25:35.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:35 smithi134 ceph-mon[39278]: 3.4 scrub starts 2024-02-05T08:25:35.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:35 smithi134 ceph-mon[39278]: 3.4 scrub ok 2024-02-05T08:25:35.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:35 smithi134 ceph-mon[39278]: pgmap v930: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T08:25:35.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:35 smithi134 ceph-mon[39278]: osdmap e404: 8 total, 8 up, 7 in 2024-02-05T08:25:35.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:35 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:35.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:35 smithi134 ceph-mon[39278]: 11.4 scrub starts 2024-02-05T08:25:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:35 smithi196 ceph-mon[33018]: 3.4 scrub starts 2024-02-05T08:25:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:35 smithi196 ceph-mon[33018]: 3.4 scrub ok 2024-02-05T08:25:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:35 smithi196 ceph-mon[33018]: pgmap v930: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T08:25:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:35 smithi196 ceph-mon[33018]: osdmap e404: 8 total, 8 up, 7 in 2024-02-05T08:25:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:35 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:35.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:35 smithi196 ceph-mon[33018]: 11.4 scrub starts 2024-02-05T08:25:35.329 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:35.369 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:35.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:35.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:35.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:35.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:35.505 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:35.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:35.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:35.634 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:35.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:35.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:35.736 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:35.775 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:35.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:35.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:35.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:35.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:35.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:36.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:36.056 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:36.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:36.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:36.158 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:36.196 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:36.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:36.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:36.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:36 smithi134 ceph-mon[30086]: 3.16 scrub starts 2024-02-05T08:25:36.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:36 smithi134 ceph-mon[30086]: 3.16 scrub ok 2024-02-05T08:25:36.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:36 smithi134 ceph-mon[30086]: 4.0 scrub starts 2024-02-05T08:25:36.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:36 smithi134 ceph-mon[30086]: 4.0 scrub ok 2024-02-05T08:25:36.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:36 smithi134 ceph-mon[39278]: 3.16 scrub starts 2024-02-05T08:25:36.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:36 smithi134 ceph-mon[39278]: 3.16 scrub ok 2024-02-05T08:25:36.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:36 smithi134 ceph-mon[39278]: 4.0 scrub starts 2024-02-05T08:25:36.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:36 smithi134 ceph-mon[39278]: 4.0 scrub ok 2024-02-05T08:25:36.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:36.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:36 smithi196 ceph-mon[33018]: 3.16 scrub starts 2024-02-05T08:25:36.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:36 smithi196 ceph-mon[33018]: 3.16 scrub ok 2024-02-05T08:25:36.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:36 smithi196 ceph-mon[33018]: 4.0 scrub starts 2024-02-05T08:25:36.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:36 smithi196 ceph-mon[33018]: 4.0 scrub ok 2024-02-05T08:25:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:36.336 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:36.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:36.437 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:36.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:36.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:36.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:36.611 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:36.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:36.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:36.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:36.745 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:36.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:36.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:36.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:36.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:36.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:36.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:37.012 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:37.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:37.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:37.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:37.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:37.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:37.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:37 smithi134 ceph-mon[30086]: pgmap v932: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-02-05T08:25:37.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:37 smithi134 ceph-mon[30086]: 11.4 scrub ok 2024-02-05T08:25:37.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:37 smithi134 ceph-mon[39278]: pgmap v932: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-02-05T08:25:37.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:37 smithi134 ceph-mon[39278]: 11.4 scrub ok 2024-02-05T08:25:37.290 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:37.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:37.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:37 smithi196 ceph-mon[33018]: pgmap v932: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s 2024-02-05T08:25:37.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:37 smithi196 ceph-mon[33018]: 11.4 scrub ok 2024-02-05T08:25:37.391 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:37.434 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:37.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:37.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:37.537 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:37.572 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:37.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:37.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:37.707 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:37.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:37.808 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:37.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:37.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:37.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:37.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:37.988 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:37.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:38.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:25:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:25:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:25:38.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:38.130 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:38.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:38.172 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T08:25:38.231 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:38.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:38.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:38.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:38 smithi196 ceph-mon[33018]: 3.1 scrub starts 2024-02-05T08:25:38.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:38 smithi196 ceph-mon[33018]: 3.1 scrub ok 2024-02-05T08:25:38.370 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:38.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:38.413 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:38.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:38.514 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:38.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:38 smithi134 ceph-mon[30086]: 3.1 scrub starts 2024-02-05T08:25:38.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:38 smithi134 ceph-mon[30086]: 3.1 scrub ok 2024-02-05T08:25:38.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:38 smithi134 ceph-mon[39278]: 3.1 scrub starts 2024-02-05T08:25:38.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:38 smithi134 ceph-mon[39278]: 3.1 scrub ok 2024-02-05T08:25:38.534 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 7, 6, 3] out_osds: [0] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-02-05T08:25:38.534 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:25:38.534 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-02-05T08:25:38.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd in 0 2024-02-05T08:25:38.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:38.555 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:38.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:38.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:38.690 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:38.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:38.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:38.792 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:38.822 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:38.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:38.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:38.967 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:38.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:39.068 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:39.107 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:39.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:39.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:39.236 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:39.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:39.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:39.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[30086]: pgmap v933: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 976 op/s 2024-02-05T08:25:39.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[30086]: 6.13 deep-scrub starts 2024-02-05T08:25:39.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[30086]: 6.13 deep-scrub ok 2024-02-05T08:25:39.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[30086]: 2.0 scrub starts 2024-02-05T08:25:39.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[30086]: 2.0 scrub ok 2024-02-05T08:25:39.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:25:39.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:25:39.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[39278]: pgmap v933: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 976 op/s 2024-02-05T08:25:39.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[39278]: 6.13 deep-scrub starts 2024-02-05T08:25:39.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[39278]: 6.13 deep-scrub ok 2024-02-05T08:25:39.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[39278]: 2.0 scrub starts 2024-02-05T08:25:39.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[39278]: 2.0 scrub ok 2024-02-05T08:25:39.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:25:39.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:25:39.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:39 smithi196 ceph-mon[33018]: pgmap v933: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 976 op/s 2024-02-05T08:25:39.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:39 smithi196 ceph-mon[33018]: 6.13 deep-scrub starts 2024-02-05T08:25:39.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:39 smithi196 ceph-mon[33018]: 6.13 deep-scrub ok 2024-02-05T08:25:39.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:39 smithi196 ceph-mon[33018]: 2.0 scrub starts 2024-02-05T08:25:39.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:39 smithi196 ceph-mon[33018]: 2.0 scrub ok 2024-02-05T08:25:39.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:25:39.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:25:39.339 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:39.372 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:39.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:39.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:39.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:39.516 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:39.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:39.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:39.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:39.657 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:39.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:39.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:39.759 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:39.789 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:39.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:39.841 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:39.891 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:39.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:39.969 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:39.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:39.975 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:40.071 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:40.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:40.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:40.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:40.208 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:40.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:40.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:40.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:40.311 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[30086]: 5.e scrub starts 2024-02-05T08:25:40.311 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[30086]: 5.e scrub ok 2024-02-05T08:25:40.311 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:25:40.311 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:25:40.311 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[30086]: 5.b scrub starts 2024-02-05T08:25:40.311 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[30086]: 5.b scrub ok 2024-02-05T08:25:40.311 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[30086]: 1.4 scrub starts 2024-02-05T08:25:40.312 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[30086]: 1.4 scrub ok 2024-02-05T08:25:40.312 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:25:40.312 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:25:40.312 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[30086]: osdmap e405: 8 total, 8 up, 7 in 2024-02-05T08:25:40.312 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[39278]: 5.e scrub starts 2024-02-05T08:25:40.312 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[39278]: 5.e scrub ok 2024-02-05T08:25:40.312 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:25:40.312 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:25:40.312 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[39278]: 5.b scrub starts 2024-02-05T08:25:40.312 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[39278]: 5.b scrub ok 2024-02-05T08:25:40.312 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[39278]: 1.4 scrub starts 2024-02-05T08:25:40.312 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[39278]: 1.4 scrub ok 2024-02-05T08:25:40.313 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:25:40.313 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:25:40.313 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:40 smithi134 ceph-mon[39278]: osdmap e405: 8 total, 8 up, 7 in 2024-02-05T08:25:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:40 smithi196 ceph-mon[33018]: 5.e scrub starts 2024-02-05T08:25:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:40 smithi196 ceph-mon[33018]: 5.e scrub ok 2024-02-05T08:25:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:40 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:25:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:40 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:25:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:40 smithi196 ceph-mon[33018]: 5.b scrub starts 2024-02-05T08:25:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:40 smithi196 ceph-mon[33018]: 5.b scrub ok 2024-02-05T08:25:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:40 smithi196 ceph-mon[33018]: 1.4 scrub starts 2024-02-05T08:25:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:40 smithi196 ceph-mon[33018]: 1.4 scrub ok 2024-02-05T08:25:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:40 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:25:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:40 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:25:40.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:40 smithi196 ceph-mon[33018]: osdmap e405: 8 total, 8 up, 7 in 2024-02-05T08:25:40.348 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:40.389 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:40.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:40.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:40.527 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:40.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:40.629 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:40.676 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:40.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:40.777 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:40.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:40.821 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:40.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:40.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:40.968 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:40.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:40.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:41.070 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:41.100 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:41.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:41.201 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:41.202 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:41 smithi134 ceph-mon[30086]: pgmap v935: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-02-05T08:25:41.202 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:41 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:25:41.202 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:41 smithi134 ceph-mon[39278]: pgmap v935: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-02-05T08:25:41.202 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:41 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:25:41.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:41.252 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:41.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:41.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:41 smithi196 ceph-mon[33018]: pgmap v935: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s 2024-02-05T08:25:41.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:41 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:25:41.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:41.389 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:41.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:41.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:41.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:41.523 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:41.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:41.624 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:41.659 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:41.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:41.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:41.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:41.797 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:41.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:41.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:41.931 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:41.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:42.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:42.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:42.069 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:42.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:42.171 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:42.210 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:42.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:42.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:42.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:42 smithi196 ceph-mon[33018]: 11.9 scrub starts 2024-02-05T08:25:42.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:42 smithi196 ceph-mon[33018]: 4.f scrub starts 2024-02-05T08:25:42.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:42 smithi196 ceph-mon[33018]: 4.f scrub ok 2024-02-05T08:25:42.345 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:42.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:42.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:42.446 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:42.480 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:42.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:42.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:42.565 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:42 smithi134 ceph-mon[30086]: 11.9 scrub starts 2024-02-05T08:25:42.565 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:42 smithi134 ceph-mon[30086]: 4.f scrub starts 2024-02-05T08:25:42.566 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:42 smithi134 ceph-mon[30086]: 4.f scrub ok 2024-02-05T08:25:42.566 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:42 smithi134 ceph-mon[39278]: 11.9 scrub starts 2024-02-05T08:25:42.566 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:42 smithi134 ceph-mon[39278]: 4.f scrub starts 2024-02-05T08:25:42.566 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:42 smithi134 ceph-mon[39278]: 4.f scrub ok 2024-02-05T08:25:42.666 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:42.694 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:42.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:42.794 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:42.830 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:42.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:42.931 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:42.967 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:42.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:42.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:43.068 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:43.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:43.098 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:43.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:43.199 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:43.240 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:43.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:43.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:43.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:43 smithi196 ceph-mon[33018]: 2.2 scrub starts 2024-02-05T08:25:43.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:43 smithi196 ceph-mon[33018]: 2.2 scrub ok 2024-02-05T08:25:43.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:43 smithi196 ceph-mon[33018]: pgmap v936: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.01k op/s 2024-02-05T08:25:43.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:43 smithi196 ceph-mon[33018]: 11.d scrub starts 2024-02-05T08:25:43.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:43 smithi196 ceph-mon[33018]: 6.16 scrub starts 2024-02-05T08:25:43.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:43 smithi196 ceph-mon[33018]: 6.16 scrub ok 2024-02-05T08:25:43.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:43.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:43.378 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:43.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:43.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:43.514 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:43.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:43.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:43 smithi134 ceph-mon[30086]: 2.2 scrub starts 2024-02-05T08:25:43.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:43 smithi134 ceph-mon[30086]: 2.2 scrub ok 2024-02-05T08:25:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:43 smithi134 ceph-mon[30086]: pgmap v936: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.01k op/s 2024-02-05T08:25:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:43 smithi134 ceph-mon[30086]: 11.d scrub starts 2024-02-05T08:25:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:43 smithi134 ceph-mon[30086]: 6.16 scrub starts 2024-02-05T08:25:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:43 smithi134 ceph-mon[30086]: 6.16 scrub ok 2024-02-05T08:25:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:43 smithi134 ceph-mon[39278]: 2.2 scrub starts 2024-02-05T08:25:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:43 smithi134 ceph-mon[39278]: 2.2 scrub ok 2024-02-05T08:25:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:43 smithi134 ceph-mon[39278]: pgmap v936: 158 pgs: 158 active+clean; 4.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.01k op/s 2024-02-05T08:25:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:43 smithi134 ceph-mon[39278]: 11.d scrub starts 2024-02-05T08:25:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:43 smithi134 ceph-mon[39278]: 6.16 scrub starts 2024-02-05T08:25:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:43 smithi134 ceph-mon[39278]: 6.16 scrub ok 2024-02-05T08:25:43.616 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:43.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:43.651 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:43.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:43.752 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:43.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:43.788 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:43.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:43.829 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:43.890 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:43.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:43.937 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:43.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:44.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:44.070 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:44.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:44.171 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:44.211 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:44.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:44.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:44.313 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:44.350 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:44.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:44.451 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:44.488 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:44.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:44.488 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[30086]: 11.9 scrub ok 2024-02-05T08:25:44.488 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[30086]: 6.c scrub starts 2024-02-05T08:25:44.488 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[30086]: 6.c scrub ok 2024-02-05T08:25:44.488 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[30086]: 6.1d scrub starts 2024-02-05T08:25:44.488 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[30086]: 6.1d scrub ok 2024-02-05T08:25:44.489 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[30086]: 5.d scrub starts 2024-02-05T08:25:44.489 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[30086]: 5.d scrub ok 2024-02-05T08:25:44.489 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[39278]: 11.9 scrub ok 2024-02-05T08:25:44.489 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[39278]: 6.c scrub starts 2024-02-05T08:25:44.489 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[39278]: 6.c scrub ok 2024-02-05T08:25:44.489 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[39278]: 6.1d scrub starts 2024-02-05T08:25:44.489 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[39278]: 6.1d scrub ok 2024-02-05T08:25:44.489 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[39278]: 5.d scrub starts 2024-02-05T08:25:44.489 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:44 smithi134 ceph-mon[39278]: 5.d scrub ok 2024-02-05T08:25:44.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:25:44 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:25:44.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:44 smithi196 ceph-mon[33018]: 11.9 scrub ok 2024-02-05T08:25:44.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:44 smithi196 ceph-mon[33018]: 6.c scrub starts 2024-02-05T08:25:44.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:44 smithi196 ceph-mon[33018]: 6.c scrub ok 2024-02-05T08:25:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:44 smithi196 ceph-mon[33018]: 6.1d scrub starts 2024-02-05T08:25:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:44 smithi196 ceph-mon[33018]: 6.1d scrub ok 2024-02-05T08:25:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:44 smithi196 ceph-mon[33018]: 5.d scrub starts 2024-02-05T08:25:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:44 smithi196 ceph-mon[33018]: 5.d scrub ok 2024-02-05T08:25:44.589 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:44.622 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:44.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:44.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:44.724 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:44.758 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:44.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:44.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:44.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:44.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:44.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:44.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:44.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:45.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:45.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:45.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:45.135 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:45.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:45.193 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:45.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:45.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:45.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:45.343 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:45.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:45.357 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[30086]: 4.c deep-scrub starts 2024-02-05T08:25:45.357 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[30086]: 4.c deep-scrub ok 2024-02-05T08:25:45.357 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[30086]: 3.10 scrub starts 2024-02-05T08:25:45.357 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[30086]: 3.10 scrub ok 2024-02-05T08:25:45.357 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[30086]: pgmap v937: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T08:25:45.357 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[30086]: 11.d scrub ok 2024-02-05T08:25:45.357 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:25:45.357 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:25:45.357 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:25:45.357 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:45.358 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[30086]: 3.12 scrub starts 2024-02-05T08:25:45.358 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[30086]: 3.12 scrub ok 2024-02-05T08:25:45.358 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[39278]: 4.c deep-scrub starts 2024-02-05T08:25:45.358 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[39278]: 4.c deep-scrub ok 2024-02-05T08:25:45.358 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[39278]: 3.10 scrub starts 2024-02-05T08:25:45.358 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[39278]: 3.10 scrub ok 2024-02-05T08:25:45.358 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[39278]: pgmap v937: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T08:25:45.358 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[39278]: 11.d scrub ok 2024-02-05T08:25:45.358 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:25:45.359 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:25:45.359 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:25:45.359 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:45.359 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[39278]: 3.12 scrub starts 2024-02-05T08:25:45.359 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:45 smithi134 ceph-mon[39278]: 3.12 scrub ok 2024-02-05T08:25:45.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:45.487 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:45.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:45.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:45 smithi196 ceph-mon[33018]: 4.c deep-scrub starts 2024-02-05T08:25:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:45 smithi196 ceph-mon[33018]: 4.c deep-scrub ok 2024-02-05T08:25:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:45 smithi196 ceph-mon[33018]: 3.10 scrub starts 2024-02-05T08:25:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:45 smithi196 ceph-mon[33018]: 3.10 scrub ok 2024-02-05T08:25:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:45 smithi196 ceph-mon[33018]: pgmap v937: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T08:25:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:45 smithi196 ceph-mon[33018]: 11.d scrub ok 2024-02-05T08:25:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:45 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:25:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:45 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:25:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:45 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:25:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:45 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:45 smithi196 ceph-mon[33018]: 3.12 scrub starts 2024-02-05T08:25:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:45 smithi196 ceph-mon[33018]: 3.12 scrub ok 2024-02-05T08:25:45.588 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:45.626 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:45.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:45.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:45.761 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:45.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:45.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:45.863 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:45.900 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:45.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:46.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:46.037 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:46.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:46.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:46.176 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:46.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:46.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:46.278 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:46.313 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:46.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:46.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:46.415 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:46.467 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:46.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:46.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:46 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:25:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:46 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:25:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:46 smithi134 ceph-mon[30086]: 3.5 scrub starts 2024-02-05T08:25:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:46 smithi134 ceph-mon[30086]: 3.5 scrub ok 2024-02-05T08:25:46.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:46 smithi134 ceph-mon[30086]: 11.a scrub starts 2024-02-05T08:25:46.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:46 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:25:46.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:46 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:25:46.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:46 smithi134 ceph-mon[39278]: 3.5 scrub starts 2024-02-05T08:25:46.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:46 smithi134 ceph-mon[39278]: 3.5 scrub ok 2024-02-05T08:25:46.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:46 smithi134 ceph-mon[39278]: 11.a scrub starts 2024-02-05T08:25:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:46 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:25:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:46 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:25:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:46 smithi196 ceph-mon[33018]: 3.5 scrub starts 2024-02-05T08:25:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:46 smithi196 ceph-mon[33018]: 3.5 scrub ok 2024-02-05T08:25:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:46 smithi196 ceph-mon[33018]: 11.a scrub starts 2024-02-05T08:25:46.569 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:46.607 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:46.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:46.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:46.743 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:46.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:46.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:46.900 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:46.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:47.002 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:47.027 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:25:47.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:47.041 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:47.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:47.071 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:25:47.112 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:25:47.142 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:47.156 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:25:47.177 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:47.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:47.212 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:25:47.263 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:25:47.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:47.309 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:25:47.309 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:47 smithi134 ceph-mon[30086]: 11.3 scrub starts 2024-02-05T08:25:47.310 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:47 smithi134 ceph-mon[30086]: pgmap v938: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s 2024-02-05T08:25:47.310 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:47 smithi134 ceph-mon[30086]: 11.b scrub starts 2024-02-05T08:25:47.310 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:47 smithi134 ceph-mon[30086]: 11.a scrub ok 2024-02-05T08:25:47.310 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:47 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4232747514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:25:47.311 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:47 smithi134 ceph-mon[39278]: 11.3 scrub starts 2024-02-05T08:25:47.311 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:47 smithi134 ceph-mon[39278]: pgmap v938: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s 2024-02-05T08:25:47.311 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:47 smithi134 ceph-mon[39278]: 11.b scrub starts 2024-02-05T08:25:47.311 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:47 smithi134 ceph-mon[39278]: 11.a scrub ok 2024-02-05T08:25:47.311 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:47 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4232747514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:25:47.315 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:47.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:47.357 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:25:47.373 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:25:47.373 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:25:47.373 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:25:47.374 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:25:47.374 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:25:47.374 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:25:47.374 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:25:47.374 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:25:47.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:47.459 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:47.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:47.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:47.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:47 smithi196 ceph-mon[33018]: 11.3 scrub starts 2024-02-05T08:25:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:47 smithi196 ceph-mon[33018]: pgmap v938: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.9 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s 2024-02-05T08:25:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:47 smithi196 ceph-mon[33018]: 11.b scrub starts 2024-02-05T08:25:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:47 smithi196 ceph-mon[33018]: 11.a scrub ok 2024-02-05T08:25:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:47 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4232747514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:25:47.594 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:47.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:47.696 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:47.736 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:47.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:47.837 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:47.873 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:47.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:47.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:47.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:48.017 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:48.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:48.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a[30063]: 2024-02-05T08:25:47.710+0000 7f4523bf3700 -1 mon.a@0(leader).osd e405 definitely_dead 0 2024-02-05T08:25:48.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:25:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:25:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:25:48.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:48.148 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:48.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:48.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:48.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:48.274 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:48.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:48.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:48.376 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:48.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:48.400 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:48.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:48.501 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:48.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:48.525 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:48.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:48.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[30086]: 11.3 scrub ok 2024-02-05T08:25:48.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[30086]: 5.6 deep-scrub starts 2024-02-05T08:25:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[30086]: 5.6 deep-scrub ok 2024-02-05T08:25:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[30086]: 11.b scrub ok 2024-02-05T08:25:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/583590949' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-05T08:25:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-05T08:25:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[30086]: 3.f scrub starts 2024-02-05T08:25:48.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[39278]: 11.3 scrub ok 2024-02-05T08:25:48.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[39278]: 5.6 deep-scrub starts 2024-02-05T08:25:48.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[39278]: 5.6 deep-scrub ok 2024-02-05T08:25:48.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[39278]: 11.b scrub ok 2024-02-05T08:25:48.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/583590949' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-05T08:25:48.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-05T08:25:48.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:48 smithi134 ceph-mon[39278]: 3.f scrub starts 2024-02-05T08:25:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:48 smithi196 ceph-mon[33018]: 11.3 scrub ok 2024-02-05T08:25:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:48 smithi196 ceph-mon[33018]: 5.6 deep-scrub starts 2024-02-05T08:25:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:48 smithi196 ceph-mon[33018]: 5.6 deep-scrub ok 2024-02-05T08:25:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:48 smithi196 ceph-mon[33018]: 11.b scrub ok 2024-02-05T08:25:48.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:48 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/583590949' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-05T08:25:48.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:48 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-02-05T08:25:48.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:48 smithi196 ceph-mon[33018]: 3.f scrub starts 2024-02-05T08:25:48.622 INFO:teuthology.orchestra.run.smithi134.stderr:marked in osd.0. 2024-02-05T08:25:48.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:48.651 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:48.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:48.753 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:48.776 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:48.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:48.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:48.903 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:48.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:49.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:49.029 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:49.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:49.112 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:25:49.131 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:49.159 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:49.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:49.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:49.260 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:49.285 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:49.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:49.385 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:49.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:49.410 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:49.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:49.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:49.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:49 smithi134 ceph-mon[30086]: 3.f scrub ok 2024-02-05T08:25:49.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:49 smithi134 ceph-mon[30086]: pgmap v939: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s 2024-02-05T08:25:49.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:49 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-05T08:25:49.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:49 smithi134 ceph-mon[30086]: osdmap e406: 8 total, 8 up, 8 in 2024-02-05T08:25:49.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:49 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/894853183' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:25:49.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:49 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:25:49.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:49 smithi134 ceph-mon[39278]: 3.f scrub ok 2024-02-05T08:25:49.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:49 smithi134 ceph-mon[39278]: pgmap v939: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s 2024-02-05T08:25:49.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:49 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-05T08:25:49.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:49 smithi134 ceph-mon[39278]: osdmap e406: 8 total, 8 up, 8 in 2024-02-05T08:25:49.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:49 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/894853183' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:25:49.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:49 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:25:49.535 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:49.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:49 smithi196 ceph-mon[33018]: 3.f scrub ok 2024-02-05T08:25:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:49 smithi196 ceph-mon[33018]: pgmap v939: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s 2024-02-05T08:25:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:49 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-02-05T08:25:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:49 smithi196 ceph-mon[33018]: osdmap e406: 8 total, 8 up, 8 in 2024-02-05T08:25:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:49 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/894853183' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:25:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:49 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:25:49.637 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:49.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:49.663 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:49.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:49.764 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:49.790 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:49.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:49.892 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:49.916 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:49.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:49.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:50.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:50.043 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:50.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:50.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:50.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:50.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:50.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:50.271 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:50.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:50.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:50.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:50.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:50.429 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:50.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:50.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:50 smithi134 ceph-mon[30086]: 6.19 scrub starts 2024-02-05T08:25:50.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:50 smithi134 ceph-mon[30086]: 6.19 scrub ok 2024-02-05T08:25:50.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:50 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:25:50.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:50 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:25:50.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:50 smithi134 ceph-mon[30086]: osdmap e407: 8 total, 8 up, 8 in 2024-02-05T08:25:50.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:50 smithi134 ceph-mon[30086]: osdmap e408: 8 total, 8 up, 8 in 2024-02-05T08:25:50.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:50 smithi134 ceph-mon[39278]: 6.19 scrub starts 2024-02-05T08:25:50.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:50 smithi134 ceph-mon[39278]: 6.19 scrub ok 2024-02-05T08:25:50.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:50 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:25:50.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:50 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:25:50.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:50 smithi134 ceph-mon[39278]: osdmap e407: 8 total, 8 up, 8 in 2024-02-05T08:25:50.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:50 smithi134 ceph-mon[39278]: osdmap e408: 8 total, 8 up, 8 in 2024-02-05T08:25:50.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:50 smithi196 ceph-mon[33018]: 6.19 scrub starts 2024-02-05T08:25:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:50 smithi196 ceph-mon[33018]: 6.19 scrub ok 2024-02-05T08:25:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:50 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:25:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:50 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:25:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:50 smithi196 ceph-mon[33018]: osdmap e407: 8 total, 8 up, 8 in 2024-02-05T08:25:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:50 smithi196 ceph-mon[33018]: osdmap e408: 8 total, 8 up, 8 in 2024-02-05T08:25:50.567 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:50.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:50.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:50.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:50.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:50.705 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:50.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:50.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:50.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:50.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:50.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:50.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:50.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:50.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:50.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:51.082 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:51.120 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:51.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:51.222 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:51.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:51.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:51.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:51.348 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:51.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:51.385 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:51.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:51.486 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:51.518 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:51.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:51.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:51 smithi134 ceph-mon[30086]: pgmap v942: 158 pgs: 28 peering, 130 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.38k op/s; 74 B/s, 3 objects/s recovering 2024-02-05T08:25:51.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:51 smithi134 ceph-mon[30086]: 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 405'13276 2024-02-05T08:25:51.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:51 smithi134 ceph-mon[30086]: 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 405'13105 2024-02-05T08:25:51.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:51 smithi134 ceph-mon[30086]: 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 405'13161 2024-02-05T08:25:51.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:51 smithi134 ceph-mon[30086]: 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 405'13208 2024-02-05T08:25:51.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:51 smithi134 ceph-mon[30086]: 11.2 starting backfill to osd.0 from (0'0,0'0] MAX to 405'13525 2024-02-05T08:25:51.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:51 smithi134 ceph-mon[39278]: pgmap v942: 158 pgs: 28 peering, 130 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.38k op/s; 74 B/s, 3 objects/s recovering 2024-02-05T08:25:51.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:51 smithi134 ceph-mon[39278]: 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 405'13276 2024-02-05T08:25:51.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:51 smithi134 ceph-mon[39278]: 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 405'13105 2024-02-05T08:25:51.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:51 smithi134 ceph-mon[39278]: 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 405'13161 2024-02-05T08:25:51.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:51 smithi134 ceph-mon[39278]: 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 405'13208 2024-02-05T08:25:51.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:51 smithi134 ceph-mon[39278]: 11.2 starting backfill to osd.0 from (0'0,0'0] MAX to 405'13525 2024-02-05T08:25:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:51 smithi196 ceph-mon[33018]: pgmap v942: 158 pgs: 28 peering, 130 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.38k op/s; 74 B/s, 3 objects/s recovering 2024-02-05T08:25:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:51 smithi196 ceph-mon[33018]: 11.1 starting backfill to osd.0 from (0'0,0'0] MAX to 405'13276 2024-02-05T08:25:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:51 smithi196 ceph-mon[33018]: 11.b starting backfill to osd.0 from (0'0,0'0] MAX to 405'13105 2024-02-05T08:25:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:51 smithi196 ceph-mon[33018]: 11.6 starting backfill to osd.0 from (0'0,0'0] MAX to 405'13161 2024-02-05T08:25:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:51 smithi196 ceph-mon[33018]: 11.9 starting backfill to osd.0 from (0'0,0'0] MAX to 405'13208 2024-02-05T08:25:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:51 smithi196 ceph-mon[33018]: 11.2 starting backfill to osd.0 from (0'0,0'0] MAX to 405'13525 2024-02-05T08:25:51.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:51.662 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:51.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:51.764 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:51.805 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:51.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:51.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:51.907 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:51.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:51.943 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:51.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:52.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:52.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:52.078 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:52.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:52.179 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:52.211 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:52.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:52.313 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:52.352 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:52.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:52.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:52.400 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-02-05T08:25:52.454 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:52.467 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:52 smithi134 ceph-mon[30086]: osdmap e409: 8 total, 8 up, 8 in 2024-02-05T08:25:52.467 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:52 smithi134 ceph-mon[39278]: osdmap e409: 8 total, 8 up, 8 in 2024-02-05T08:25:52.497 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:52.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:52.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:52 smithi196 ceph-mon[33018]: osdmap e409: 8 total, 8 up, 8 in 2024-02-05T08:25:52.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:52.631 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:52.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:52.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:52.733 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:52.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:52.778 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:52.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:52.879 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:52.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:52.912 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:52.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:53.013 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:53.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:53.044 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:53.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:53.145 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:53.184 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:53.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:53.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:53.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:53.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:53.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:53.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:53.457 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:53.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:53.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:53 smithi134 ceph-mon[30086]: pgmap v945: 158 pgs: 1 active+remapped+backfilling, 28 peering, 129 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s; 3389/174348 objects misplaced (1.944%); 1.6 MiB/s, 32 objects/s recovering 2024-02-05T08:25:53.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:53 smithi134 ceph-mon[30086]: osdmap e410: 8 total, 8 up, 8 in 2024-02-05T08:25:53.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:53 smithi134 ceph-mon[39278]: pgmap v945: 158 pgs: 1 active+remapped+backfilling, 28 peering, 129 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s; 3389/174348 objects misplaced (1.944%); 1.6 MiB/s, 32 objects/s recovering 2024-02-05T08:25:53.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:53 smithi134 ceph-mon[39278]: osdmap e410: 8 total, 8 up, 8 in 2024-02-05T08:25:53.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:53.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:53 smithi196 ceph-mon[33018]: pgmap v945: 158 pgs: 1 active+remapped+backfilling, 28 peering, 129 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 1.34k op/s; 3389/174348 objects misplaced (1.944%); 1.6 MiB/s, 32 objects/s recovering 2024-02-05T08:25:53.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:53 smithi196 ceph-mon[33018]: osdmap e410: 8 total, 8 up, 8 in 2024-02-05T08:25:53.591 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:53.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:53.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:53.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:53.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:53.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:53.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:53.870 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:53.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:53.972 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:54.016 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:54.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:54.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:54.118 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:54.161 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:54.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:54.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:54.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:54.313 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:54.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:54.414 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:54.446 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:25:54.446 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:25:54.446 INFO:teuthology.orchestra.run.smithi134.stdout: "complaint_time": 30, 2024-02-05T08:25:54.446 INFO:teuthology.orchestra.run.smithi134.stdout: "num_blocked_ops": 0 2024-02-05T08:25:54.446 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:25:54.459 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:54.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:54.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:54.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:25:54 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:25:54.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:54.596 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:54.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:54.698 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:54.736 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:54.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:54.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:54.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:54.874 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:54.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:54.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:54.976 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:55.021 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:55.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:55.123 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:55.144 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:25:55.144 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:25:55.144 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:54 smithi134 ceph-mon[30086]: pgmap v947: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 210 B/s rd, 0 B/s wr, 725 op/s; 16412/165738 objects misplaced (9.902%); 9.3 MiB/s, 160 objects/s recovering 2024-02-05T08:25:55.144 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:54 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:55.144 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:25:55.144 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:25:55.144 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:54 smithi134 ceph-mon[39278]: pgmap v947: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 210 B/s rd, 0 B/s wr, 725 op/s; 16412/165738 objects misplaced (9.902%); 9.3 MiB/s, 160 objects/s recovering 2024-02-05T08:25:55.145 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:54 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:55.155 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:55.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:55.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:55.193 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T08:25:55.257 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:25:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:25:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:54 smithi196 ceph-mon[33018]: pgmap v947: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 210 B/s rd, 0 B/s wr, 725 op/s; 16412/165738 objects misplaced (9.902%); 9.3 MiB/s, 160 objects/s recovering 2024-02-05T08:25:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:54 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:25:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:55.315 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:55.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:55.416 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:55.444 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:55.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:55.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:55.546 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:55.597 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:55.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:55.606 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:25:55.670 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:55.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:55.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:55.728 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:55.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:55.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:55.860 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:55.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:55.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:56.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:56.022 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:56.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:56.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:56.170 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:56.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:56.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:56.296 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:25:56.302 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:56.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:56.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:56.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:56.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:56.434 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:56.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:56.535 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:56.563 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:56.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:56.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:56.596 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:56.665 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:56.720 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:56.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:56.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:56.822 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:56.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:56.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:56.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:56.963 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:57.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:57.010 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:57.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:57.111 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:57.150 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:57.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:57.251 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:57.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:57 smithi134 ceph-mon[30086]: pgmap v948: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 587 op/s; 16412/165738 objects misplaced (9.902%); 7.5 MiB/s, 125 objects/s recovering 2024-02-05T08:25:57.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:57 smithi134 ceph-mon[39278]: pgmap v948: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 587 op/s; 16412/165738 objects misplaced (9.902%); 7.5 MiB/s, 125 objects/s recovering 2024-02-05T08:25:57.296 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:57.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:57 smithi196 ceph-mon[33018]: pgmap v948: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 587 op/s; 16412/165738 objects misplaced (9.902%); 7.5 MiB/s, 125 objects/s recovering 2024-02-05T08:25:57.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:57.401 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 7, 6, 3, 0] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-02-05T08:25:57.401 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:25:57.402 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-05T08:25:57.402 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.554926 2024-02-05T08:25:57.402 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 5 0.5549260869338017 2024-02-05T08:25:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:57.432 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:57.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:57.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:25:57.562 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:57.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:57.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:25:57.664 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:57.732 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:57.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:57.741 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:25:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:25:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:25:57.834 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:57.869 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:57.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:57.971 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:58.010 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:58.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:58.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:58.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:25:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:58.156 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:58.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:25:58.257 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:58.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:58 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:25:58.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:58 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:25:58.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:58 smithi134 ceph-mon[30086]: osdmap e411: 8 total, 8 up, 8 in 2024-02-05T08:25:58.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:58 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:25:58.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:58 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:25:58.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:58 smithi134 ceph-mon[39278]: osdmap e411: 8 total, 8 up, 8 in 2024-02-05T08:25:58.288 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:58.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:58.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:58 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:25:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:58 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:25:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:58 smithi196 ceph-mon[33018]: osdmap e411: 8 total, 8 up, 8 in 2024-02-05T08:25:58.390 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:58.436 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:58.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:58.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:58.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:58.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:58.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:25:58.671 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:58.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:58.715 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:58.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:58.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:25:58.849 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:58.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:25:58.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:25:58.981 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:58.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:25:59.083 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:59.132 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:59.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:59.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:25:59.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:59 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:25:59.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:59 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:25:59.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:59 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:25:59.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:59 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:25:59.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:59 smithi134 ceph-mon[39278]: pgmap v950: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 662 op/s; 15633/162777 objects misplaced (9.604%); 11 MiB/s, 185 objects/s recovering 2024-02-05T08:25:59.258 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:25:59 smithi134 ceph-mon[39278]: osdmap e412: 8 total, 8 up, 8 in 2024-02-05T08:25:59.258 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:59 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:25:59.259 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:59 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:25:59.259 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:59 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:25:59.259 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:59 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:25:59.259 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:59 smithi134 ceph-mon[30086]: pgmap v950: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 662 op/s; 15633/162777 objects misplaced (9.604%); 11 MiB/s, 185 objects/s recovering 2024-02-05T08:25:59.259 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:25:59 smithi134 ceph-mon[30086]: osdmap e412: 8 total, 8 up, 8 in 2024-02-05T08:25:59.281 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:59.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:25:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:59 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:25:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:59 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:25:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:59 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:25:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:59 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:25:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:59 smithi196 ceph-mon[33018]: pgmap v950: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 751 B/s rd, 0 B/s wr, 662 op/s; 15633/162777 objects misplaced (9.604%); 11 MiB/s, 185 objects/s recovering 2024-02-05T08:25:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:25:59 smithi196 ceph-mon[33018]: osdmap e412: 8 total, 8 up, 8 in 2024-02-05T08:25:59.383 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:59.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:59.424 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:59.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:59.463 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:59.483 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:25:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:25:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:25:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:25:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24344.0:3013 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94292702904192 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:25:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.615963+0000", 2024-02-05T08:25:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.86283567300001, 2024-02-05T08:25:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.073499212, 2024-02-05T08:25:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.484 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:25:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24344", 2024-02-05T08:25:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1272526039", 2024-02-05T08:25:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3013 2024-02-05T08:25:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.615963+0000", 2024-02-05T08:25:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.615963+0000", 2024-02-05T08:25:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.615966+0000", 2024-02-05T08:25:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1390000000000002e-06 2024-02-05T08:25:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.615974+0000", 2024-02-05T08:25:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4460000000000001e-06 2024-02-05T08:25:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.615996+0000", 2024-02-05T08:25:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.209e-05 2024-02-05T08:25:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616008+0000", 2024-02-05T08:25:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1664e-05 2024-02-05T08:25:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.624254+0000", 2024-02-05T08:25:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0082457539999999992 2024-02-05T08:25:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.624267+0000", 2024-02-05T08:25:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3565e-05 2024-02-05T08:25:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.664392+0000", 2024-02-05T08:25:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.040125111 2024-02-05T08:25:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for missing object", 2024-02-05T08:25:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.664525+0000", 2024-02-05T08:25:59.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013255499999999999 2024-02-05T08:25:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.674568+0000", 2024-02-05T08:25:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.010042822999999999 2024-02-05T08:25:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-02-05T08:25:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.674663+0000", 2024-02-05T08:25:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5057999999999997e-05 2024-02-05T08:25:59.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.677226+0000", 2024-02-05T08:25:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0025634939999999999 2024-02-05T08:25:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:25:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.677318+0000", 2024-02-05T08:25:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2207999999999995e-05 2024-02-05T08:25:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.689463+0000", 2024-02-05T08:25:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.012144304999999999 2024-02-05T08:25:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.24344.0:3015 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94292702897280 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:25:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.616045+0000", 2024-02-05T08:25:59.492 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.86275372, 2024-02-05T08:25:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.093509665, 2024-02-05T08:25:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "started", 2024-02-05T08:25:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.24344", 2024-02-05T08:25:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/1272526039", 2024-02-05T08:25:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 3015 2024-02-05T08:25:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.493 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616045+0000", 2024-02-05T08:25:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616045+0000", 2024-02-05T08:25:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616046+0000", 2024-02-05T08:25:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8500000000000001e-07 2024-02-05T08:25:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616049+0000", 2024-02-05T08:25:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1719999999999999e-06 2024-02-05T08:25:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616058+0000", 2024-02-05T08:25:59.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3389999999999993e-06 2024-02-05T08:25:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.616064+0000", 2024-02-05T08:25:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7610000000000002e-06 2024-02-05T08:25:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.640831+0000", 2024-02-05T08:25:59.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.024766535999999999 2024-02-05T08:25:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.640840+0000", 2024-02-05T08:25:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4350000000000003e-06 2024-02-05T08:25:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.645324+0000", 2024-02-05T08:25:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0044837120000001 2024-02-05T08:25:59.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for missing object", 2024-02-05T08:25:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.645467+0000", 2024-02-05T08:25:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000142786 2024-02-05T08:25:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.678676+0000", 2024-02-05T08:25:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.033208803000000002 2024-02-05T08:25:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for rw locks", 2024-02-05T08:25:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.678770+0000", 2024-02-05T08:25:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4278999999999996e-05 2024-02-05T08:25:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.679540+0000", 2024-02-05T08:25:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00077013699999999995 2024-02-05T08:25:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:25:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.679613+0000", 2024-02-05T08:25:59.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3100000000000001e-05 2024-02-05T08:25:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.709555+0000", 2024-02-05T08:25:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.02994192 2024-02-05T08:25:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259330 9.1 9.2fae0b11 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:25:59.500 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664160+0000", 2024-02-05T08:25:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.81463925000003, 2024-02-05T08:25:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95273855299999999, 2024-02-05T08:25:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:25:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:25:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:25:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259330 2024-02-05T08:25:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664160+0000", 2024-02-05T08:25:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664160+0000", 2024-02-05T08:25:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664161+0000", 2024-02-05T08:25:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5880000000000001e-06 2024-02-05T08:25:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664166+0000", 2024-02-05T08:25:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5800000000000002e-06 2024-02-05T08:25:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664187+0000", 2024-02-05T08:25:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1233000000000002e-05 2024-02-05T08:25:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664199+0000", 2024-02-05T08:25:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1644000000000001e-05 2024-02-05T08:25:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664258+0000", 2024-02-05T08:25:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8783000000000003e-05 2024-02-05T08:25:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664268+0000", 2024-02-05T08:25:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9580000000000002e-06 2024-02-05T08:25:59.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.616887+0000", 2024-02-05T08:25:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95261980700000004 2024-02-05T08:25:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.616898+0000", 2024-02-05T08:25:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.096e-05 2024-02-05T08:25:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.505 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259337 9.2 9.d70982e2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:25:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664222+0000", 2024-02-05T08:25:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.81457698200001, 2024-02-05T08:25:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95069705999999998, 2024-02-05T08:25:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:25:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:25:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:25:59.506 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259337 2024-02-05T08:25:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664222+0000", 2024-02-05T08:25:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664222+0000", 2024-02-05T08:25:59.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664223+0000", 2024-02-05T08:25:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3099999999999998e-07 2024-02-05T08:25:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664226+0000", 2024-02-05T08:25:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0570000000000001e-06 2024-02-05T08:25:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664234+0000", 2024-02-05T08:25:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1210000000000007e-06 2024-02-05T08:25:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664240+0000", 2024-02-05T08:25:59.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2790000000000004e-06 2024-02-05T08:25:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664293+0000", 2024-02-05T08:25:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2849000000000002e-05 2024-02-05T08:25:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664298+0000", 2024-02-05T08:25:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6940000000000003e-06 2024-02-05T08:25:59.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.614904+0000", 2024-02-05T08:25:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95060682500000004 2024-02-05T08:25:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.614919+0000", 2024-02-05T08:25:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4704e-05 2024-02-05T08:25:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.511 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259339 9.c 9.54971cbc (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:25:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664254+0000", 2024-02-05T08:25:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.81454481899999, 2024-02-05T08:25:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95406026300000002, 2024-02-05T08:25:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:25:59.512 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:25:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:25:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259339 2024-02-05T08:25:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664254+0000", 2024-02-05T08:25:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664254+0000", 2024-02-05T08:25:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664255+0000", 2024-02-05T08:25:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.15e-07 2024-02-05T08:25:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664257+0000", 2024-02-05T08:25:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5119999999999998e-06 2024-02-05T08:25:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664264+0000", 2024-02-05T08:25:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4409999999999998e-06 2024-02-05T08:25:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664269+0000", 2024-02-05T08:25:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5260000000000001e-06 2024-02-05T08:25:59.515 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664322+0000", 2024-02-05T08:25:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2709e-05 2024-02-05T08:25:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664330+0000", 2024-02-05T08:25:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0220000000000004e-06 2024-02-05T08:25:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.618306+0000", 2024-02-05T08:25:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95397562300000005 2024-02-05T08:25:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.618315+0000", 2024-02-05T08:25:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.0149999999999992e-06 2024-02-05T08:25:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.517 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259343 9.8 9.8b42e388 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:25:59.518 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664344+0000", 2024-02-05T08:25:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.81445489499998, 2024-02-05T08:25:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95628875999999996, 2024-02-05T08:25:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:25:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:25:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:25:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259343 2024-02-05T08:25:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664344+0000", 2024-02-05T08:25:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664344+0000", 2024-02-05T08:25:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664345+0000", 2024-02-05T08:25:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9100000000000004e-07 2024-02-05T08:25:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664348+0000", 2024-02-05T08:25:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.0170000000000001e-06 2024-02-05T08:25:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664355+0000", 2024-02-05T08:25:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.8719999999999996e-06 2024-02-05T08:25:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664361+0000", 2024-02-05T08:25:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3080000000000004e-06 2024-02-05T08:25:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664403+0000", 2024-02-05T08:25:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2247999999999999e-05 2024-02-05T08:25:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664410+0000", 2024-02-05T08:25:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6560000000000003e-06 2024-02-05T08:25:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.620623+0000", 2024-02-05T08:25:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95621348900000003 2024-02-05T08:25:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.620633+0000", 2024-02-05T08:25:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5789999999999993e-06 2024-02-05T08:25:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.524 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259344 9.1 9.844dc3c1 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:25:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664418+0000", 2024-02-05T08:25:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.81438144499998, 2024-02-05T08:25:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95252151900000004, 2024-02-05T08:25:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:25:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:25:59.525 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:25:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259344 2024-02-05T08:25:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664418+0000", 2024-02-05T08:25:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664418+0000", 2024-02-05T08:25:59.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664418+0000", 2024-02-05T08:25:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7899999999999999e-07 2024-02-05T08:25:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664421+0000", 2024-02-05T08:25:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.711e-06 2024-02-05T08:25:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664427+0000", 2024-02-05T08:25:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6279999999999999e-06 2024-02-05T08:25:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664433+0000", 2024-02-05T08:25:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6980000000000003e-06 2024-02-05T08:25:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.528 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664468+0000", 2024-02-05T08:25:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4662000000000001e-05 2024-02-05T08:25:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664473+0000", 2024-02-05T08:25:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0100000000000003e-06 2024-02-05T08:25:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.616924+0000", 2024-02-05T08:25:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95245070399999998 2024-02-05T08:25:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.616939+0000", 2024-02-05T08:25:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5627000000000001e-05 2024-02-05T08:25:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259345 9.2 9.50719472 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:25:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664445+0000", 2024-02-05T08:25:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.81435433899998, 2024-02-05T08:25:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95056753100000002, 2024-02-05T08:25:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:25:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:25:59.531 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:25:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259345 2024-02-05T08:25:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664445+0000", 2024-02-05T08:25:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664445+0000", 2024-02-05T08:25:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664445+0000", 2024-02-05T08:25:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2899999999999999e-07 2024-02-05T08:25:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664447+0000", 2024-02-05T08:25:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.024e-06 2024-02-05T08:25:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664453+0000", 2024-02-05T08:25:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.9789999999999998e-06 2024-02-05T08:25:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664458+0000", 2024-02-05T08:25:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2909999999999998e-06 2024-02-05T08:25:59.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664493+0000", 2024-02-05T08:25:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4397999999999998e-05 2024-02-05T08:25:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664500+0000", 2024-02-05T08:25:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3390000000000004e-06 2024-02-05T08:25:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.614999+0000", 2024-02-05T08:25:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95049863400000001 2024-02-05T08:25:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.615012+0000", 2024-02-05T08:25:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3437e-05 2024-02-05T08:25:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.536 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259347 9.8 9.1b3af678 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:25:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664466+0000", 2024-02-05T08:25:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.814332654, 2024-02-05T08:25:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95619838800000001, 2024-02-05T08:25:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:25:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:25:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:25:59.537 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259347 2024-02-05T08:25:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664466+0000", 2024-02-05T08:25:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664466+0000", 2024-02-05T08:25:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664467+0000", 2024-02-05T08:25:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.58e-07 2024-02-05T08:25:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664469+0000", 2024-02-05T08:25:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4660000000000002e-06 2024-02-05T08:25:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664475+0000", 2024-02-05T08:25:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0159999999999997e-06 2024-02-05T08:25:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.540 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664481+0000", 2024-02-05T08:25:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7300000000000002e-06 2024-02-05T08:25:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664518+0000", 2024-02-05T08:25:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6606999999999997e-05 2024-02-05T08:25:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.541 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664524+0000", 2024-02-05T08:25:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4819999999999999e-06 2024-02-05T08:25:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.620657+0000", 2024-02-05T08:25:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95613291199999995 2024-02-05T08:25:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.542 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.620665+0000", 2024-02-05T08:25:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8169999999999999e-06 2024-02-05T08:25:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25223.0:259357 9.1 9.f0ac7411 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e294)", 2024-02-05T08:25:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:18:33.664536+0000", 2024-02-05T08:25:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 445.81426321700002, 2024-02-05T08:25:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95243644199999999, 2024-02-05T08:25:59.543 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:25:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25223", 2024-02-05T08:25:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/2581122440", 2024-02-05T08:25:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 259357 2024-02-05T08:25:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664536+0000", 2024-02-05T08:25:59.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664536+0000", 2024-02-05T08:25:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664536+0000", 2024-02-05T08:25:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0500000000000004e-07 2024-02-05T08:25:59.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664539+0000", 2024-02-05T08:25:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5840000000000002e-06 2024-02-05T08:25:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664552+0000", 2024-02-05T08:25:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3419e-05 2024-02-05T08:25:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664558+0000", 2024-02-05T08:25:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1190000000000002e-06 2024-02-05T08:25:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664592+0000", 2024-02-05T08:25:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.3501000000000003e-05 2024-02-05T08:25:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:33.664595+0000", 2024-02-05T08:25:59.547 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.348e-06 2024-02-05T08:25:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.616963+0000", 2024-02-05T08:25:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95236730199999997 2024-02-05T08:25:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:18:34.616972+0000", 2024-02-05T08:25:59.548 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6639999999999993e-06 2024-02-05T08:25:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.15714.0:214251 10.f e348/309 10:f26750af:::benchmark_data_smithi134_174861_object82789:head v 348'13527, mlcod=348'13525)", 2024-02-05T08:25:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:21:58.845845+0000", 2024-02-05T08:25:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 240.63295428200001, 2024-02-05T08:25:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.142092944, 2024-02-05T08:25:59.549 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:25:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845845+0000", 2024-02-05T08:25:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845845+0000", 2024-02-05T08:25:59.550 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845846+0000", 2024-02-05T08:25:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9100000000000002e-07 2024-02-05T08:25:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845855+0000", 2024-02-05T08:25:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.5170000000000005e-06 2024-02-05T08:25:59.551 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845868+0000", 2024-02-05T08:25:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3227999999999999e-05 2024-02-05T08:25:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845875+0000", 2024-02-05T08:25:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7619999999999998e-06 2024-02-05T08:25:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.552 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845938+0000", 2024-02-05T08:25:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3083999999999994e-05 2024-02-05T08:25:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:25:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.845956+0000", 2024-02-05T08:25:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7657000000000001e-05 2024-02-05T08:25:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:25:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.987898+0000", 2024-02-05T08:25:59.553 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.14194200400000001 2024-02-05T08:25:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:21:58.987938+0000", 2024-02-05T08:25:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9801000000000001e-05 2024-02-05T08:25:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.25820.0:136148 11.2 e397/367 11:4ed7b354:::benchmark_data_smithi134_205380_object1046:head v 397'8687, mlcod=397'8680)", 2024-02-05T08:25:59.554 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:24:50.633655+0000", 2024-02-05T08:25:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 68.845144016000006, 2024-02-05T08:25:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13446425300000001, 2024-02-05T08:25:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:25:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.633655+0000", 2024-02-05T08:25:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.555 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.633655+0000", 2024-02-05T08:25:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.633656+0000", 2024-02-05T08:25:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1999999999999999e-07 2024-02-05T08:25:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.556 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.633661+0000", 2024-02-05T08:25:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2599999999999996e-06 2024-02-05T08:25:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.633669+0000", 2024-02-05T08:25:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.4449999999999996e-06 2024-02-05T08:25:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.633675+0000", 2024-02-05T08:25:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1150000000000002e-06 2024-02-05T08:25:59.557 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.635801+0000", 2024-02-05T08:25:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00212678 2024-02-05T08:25:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:25:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.635820+0000", 2024-02-05T08:25:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9020000000000001e-05 2024-02-05T08:25:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.558 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:25:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.768031+0000", 2024-02-05T08:25:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13221049700000001 2024-02-05T08:25:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.768119+0000", 2024-02-05T08:25:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8416000000000002e-05 2024-02-05T08:25:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.559 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:136155 11.a 11:526d38b6:::benchmark_data_smithi134_205380_object1053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-05T08:25:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:24:50.637125+0000", 2024-02-05T08:25:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 68.841673701000005, 2024-02-05T08:25:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.13264741099999999, 2024-02-05T08:25:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:25:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:25:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:25:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 136155 2024-02-05T08:25:59.560 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.637125+0000", 2024-02-05T08:25:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.637125+0000", 2024-02-05T08:25:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.561 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.637126+0000", 2024-02-05T08:25:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7400000000000003e-07 2024-02-05T08:25:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.637129+0000", 2024-02-05T08:25:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1669999999999999e-06 2024-02-05T08:25:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.562 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.637137+0000", 2024-02-05T08:25:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9710000000000004e-06 2024-02-05T08:25:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.637143+0000", 2024-02-05T08:25:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7849999999999997e-06 2024-02-05T08:25:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.637197+0000", 2024-02-05T08:25:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4234000000000003e-05 2024-02-05T08:25:59.563 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:25:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.637301+0000", 2024-02-05T08:25:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000103856 2024-02-05T08:25:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 5,6", 2024-02-05T08:25:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.637424+0000", 2024-02-05T08:25:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012294899999999999 2024-02-05T08:25:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.564 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.760791+0000", 2024-02-05T08:25:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12336678299999999 2024-02-05T08:25:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.763785+0000", 2024-02-05T08:25:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0029942010000000002 2024-02-05T08:25:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:25:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.769694+0000", 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0059093260000000003 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.769737+0000", 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3149999999999999e-05 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.769773+0000", 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5315000000000003e-05 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.566 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212064 11.9 11:945cbbf6:::benchmark_data_smithi134_205380_object76962:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-02-05T08:25:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:48.094587+0000", 2024-02-05T08:25:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 11.384211827, 2024-02-05T08:25:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2479190579999999, 2024-02-05T08:25:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:25:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:25:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:25:59.567 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212064 2024-02-05T08:25:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094587+0000", 2024-02-05T08:25:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094587+0000", 2024-02-05T08:25:59.568 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094589+0000", 2024-02-05T08:25:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.7439999999999999e-06 2024-02-05T08:25:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094595+0000", 2024-02-05T08:25:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0589999999999999e-06 2024-02-05T08:25:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.569 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094613+0000", 2024-02-05T08:25:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8349999999999999e-05 2024-02-05T08:25:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094623+0000", 2024-02-05T08:25:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1200000000000008e-06 2024-02-05T08:25:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.570 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094690+0000", 2024-02-05T08:25:59.571 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:25:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7306999999999998e-05 2024-02-05T08:25:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.571 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:25:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094850+0000", 2024-02-05T08:25:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016055999999999999 2024-02-05T08:25:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,6", 2024-02-05T08:25:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.097341+0000", 2024-02-05T08:25:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002490103 2024-02-05T08:25:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.572 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.123708+0000", 2024-02-05T08:25:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.02636759 2024-02-05T08:25:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:25:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.124213+0000", 2024-02-05T08:25:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00050442399999999995 2024-02-05T08:25:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.573 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.144030+0000", 2024-02-05T08:25:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.019817837000000001 2024-02-05T08:25:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.144042+0000", 2024-02-05T08:25:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2048999999999999e-05 2024-02-05T08:25:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.574 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.120119+0000", 2024-02-05T08:25:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.97607706000000005 2024-02-05T08:25:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.120132+0000", 2024-02-05T08:25:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2498e-05 2024-02-05T08:25:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.575 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.342431+0000", 2024-02-05T08:25:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.2222987540000001 2024-02-05T08:25:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.342506+0000", 2024-02-05T08:25:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5603000000000001e-05 2024-02-05T08:25:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.576 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212065 11.b 11:d069df34:::benchmark_data_smithi134_205380_object76963:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-02-05T08:25:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:48.094647+0000", 2024-02-05T08:25:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 11.384152156000001, 2024-02-05T08:25:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1405962500000002, 2024-02-05T08:25:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:25:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:25:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:25:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212065 2024-02-05T08:25:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.577 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094647+0000", 2024-02-05T08:25:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094647+0000", 2024-02-05T08:25:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.578 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094647+0000", 2024-02-05T08:25:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2499999999999995e-07 2024-02-05T08:25:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094651+0000", 2024-02-05T08:25:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5549999999999998e-06 2024-02-05T08:25:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.579 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094659+0000", 2024-02-05T08:25:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1300000000000001e-06 2024-02-05T08:25:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094669+0000", 2024-02-05T08:25:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0013000000000001e-05 2024-02-05T08:25:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094722+0000", 2024-02-05T08:25:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3029999999999999e-05 2024-02-05T08:25:59.580 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:25:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094858+0000", 2024-02-05T08:25:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000135487 2024-02-05T08:25:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:25:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.094995+0000", 2024-02-05T08:25:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000136947 2024-02-05T08:25:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.581 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.098363+0000", 2024-02-05T08:25:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0033683340000000002 2024-02-05T08:25:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:25:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.102625+0000", 2024-02-05T08:25:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042622169999999996 2024-02-05T08:25:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.142039+0000", 2024-02-05T08:25:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.039413924000000003 2024-02-05T08:25:59.582 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.142052+0000", 2024-02-05T08:25:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3378e-05 2024-02-05T08:25:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.122702+0000", 2024-02-05T08:25:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.98064931200000005 2024-02-05T08:25:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.583 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.122716+0000", 2024-02-05T08:25:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4296e-05 2024-02-05T08:25:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.235161+0000", 2024-02-05T08:25:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1124446100000001 2024-02-05T08:25:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.235243+0000", 2024-02-05T08:25:59.584 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2491999999999996e-05 2024-02-05T08:25:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212069 11.2 11:4dd58955:::benchmark_data_smithi134_205380_object76967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e405)", 2024-02-05T08:25:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:48.100836+0000", 2024-02-05T08:25:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 11.377962601, 2024-02-05T08:25:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1338321210000002, 2024-02-05T08:25:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:25:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.585 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:25:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:25:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212069 2024-02-05T08:25:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.100836+0000", 2024-02-05T08:25:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.586 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.100836+0000", 2024-02-05T08:25:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.100838+0000", 2024-02-05T08:25:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9099999999999999e-06 2024-02-05T08:25:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.587 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.100845+0000", 2024-02-05T08:25:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4479999999999999e-06 2024-02-05T08:25:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.588 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.588 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:25:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.100859+0000", 2024-02-05T08:25:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4202000000000001e-05 2024-02-05T08:25:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.589 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.100869+0000", 2024-02-05T08:25:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.7990000000000005e-06 2024-02-05T08:25:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.101004+0000", 2024-02-05T08:25:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013485299999999999 2024-02-05T08:25:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.590 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:25:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.101139+0000", 2024-02-05T08:25:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013568599999999999 2024-02-05T08:25:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,7", 2024-02-05T08:25:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.101272+0000", 2024-02-05T08:25:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000133034 2024-02-05T08:25:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.591 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.124429+0000", 2024-02-05T08:25:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.023156300000000001 2024-02-05T08:25:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.124445+0000", 2024-02-05T08:25:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6769000000000002e-05 2024-02-05T08:25:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.112990+0000", 2024-02-05T08:25:59.592 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.98854417699999997 2024-02-05T08:25:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.113005+0000", 2024-02-05T08:25:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5464999999999998e-05 2024-02-05T08:25:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.228885+0000", 2024-02-05T08:25:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.115879931 2024-02-05T08:25:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.593 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for missing object", 2024-02-05T08:25:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.229046+0000", 2024-02-05T08:25:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000160841 2024-02-05T08:25:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.234566+0000", 2024-02-05T08:25:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0055204310000000001 2024-02-05T08:25:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.234669+0000", 2024-02-05T08:25:59.594 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010227500000000001 2024-02-05T08:25:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212098 11.b 11:d2b7de44:::benchmark_data_smithi134_205380_object76996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-02-05T08:25:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:48.152350+0000", 2024-02-05T08:25:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 11.326449299, 2024-02-05T08:25:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0974029540000001, 2024-02-05T08:25:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:25:59.595 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:25:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:25:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212098 2024-02-05T08:25:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.152350+0000", 2024-02-05T08:25:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.596 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.152350+0000", 2024-02-05T08:25:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.152352+0000", 2024-02-05T08:25:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8279999999999999e-06 2024-02-05T08:25:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.152359+0000", 2024-02-05T08:25:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.289e-06 2024-02-05T08:25:59.597 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.152381+0000", 2024-02-05T08:25:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1951e-05 2024-02-05T08:25:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.152391+0000", 2024-02-05T08:25:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0152e-05 2024-02-05T08:25:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.598 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.152497+0000", 2024-02-05T08:25:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000106 2024-02-05T08:25:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.152512+0000", 2024-02-05T08:25:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4834000000000001e-05 2024-02-05T08:25:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.599 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.122955+0000", 2024-02-05T08:25:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.97044309900000003 2024-02-05T08:25:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.122965+0000", 2024-02-05T08:25:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0329999999999999e-05 2024-02-05T08:25:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.235724+0000", 2024-02-05T08:25:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1127591610000001 2024-02-05T08:25:59.600 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:25:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.235845+0000", 2024-02-05T08:25:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012051 2024-02-05T08:25:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,1,4", 2024-02-05T08:25:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.235969+0000", 2024-02-05T08:25:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012372800000000001 2024-02-05T08:25:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.601 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:25:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.239148+0000", 2024-02-05T08:25:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0031797100000000001 2024-02-05T08:25:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.242125+0000", 2024-02-05T08:25:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0029763419999999999 2024-02-05T08:25:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.249515+0000", 2024-02-05T08:25:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0073899860000000003 2024-02-05T08:25:59.602 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.249576+0000", 2024-02-05T08:25:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1218999999999998e-05 2024-02-05T08:25:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:25:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.249638+0000", 2024-02-05T08:25:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2342999999999996e-05 2024-02-05T08:25:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.603 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.249753+0000", 2024-02-05T08:25:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000114472 2024-02-05T08:25:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212105 11.9 11:95cea59c:::benchmark_data_smithi134_205380_object77003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-02-05T08:25:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:48.159314+0000", 2024-02-05T08:25:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 11.319484730999999, 2024-02-05T08:25:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.214335036, 2024-02-05T08:25:59.604 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212105 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.159314+0000", 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.605 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.159314+0000", 2024-02-05T08:25:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.159316+0000", 2024-02-05T08:25:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.438e-06 2024-02-05T08:25:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.159321+0000", 2024-02-05T08:25:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6409999999999999e-06 2024-02-05T08:25:59.606 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.159338+0000", 2024-02-05T08:25:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.643e-05 2024-02-05T08:25:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.159345+0000", 2024-02-05T08:25:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.391e-06 2024-02-05T08:25:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.607 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.159426+0000", 2024-02-05T08:25:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1225999999999998e-05 2024-02-05T08:25:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.159434+0000", 2024-02-05T08:25:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4669999999999998e-06 2024-02-05T08:25:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.608 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.120158+0000", 2024-02-05T08:25:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.96072447500000002 2024-02-05T08:25:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.120166+0000", 2024-02-05T08:25:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0660000000000004e-06 2024-02-05T08:25:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.357905+0000", 2024-02-05T08:25:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.237738555 2024-02-05T08:25:59.609 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:25:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.358413+0000", 2024-02-05T08:25:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00050798200000000003 2024-02-05T08:25:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.610 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,1,6", 2024-02-05T08:25:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.358533+0000", 2024-02-05T08:25:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000120128 2024-02-05T08:25:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.366002+0000", 2024-02-05T08:25:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0074693119999999997 2024-02-05T08:25:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.611 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:25:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.366017+0000", 2024-02-05T08:25:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.4697e-05 2024-02-05T08:25:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.366062+0000", 2024-02-05T08:25:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5238000000000003e-05 2024-02-05T08:25:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.612 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.373479+0000", 2024-02-05T08:25:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0074164699999999997 2024-02-05T08:25:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:25:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.373537+0000", 2024-02-05T08:25:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7924000000000003e-05 2024-02-05T08:25:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.613 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.373649+0000", 2024-02-05T08:25:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000112596 2024-02-05T08:25:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212112 11.b 11:df8a64dd:::benchmark_data_smithi134_205380_object77010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-02-05T08:25:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:48.168583+0000", 2024-02-05T08:25:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 11.31021653, 2024-02-05T08:25:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.081426268, 2024-02-05T08:25:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.614 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:25:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:25:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:25:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212112 2024-02-05T08:25:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.168583+0000", 2024-02-05T08:25:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.615 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.168583+0000", 2024-02-05T08:25:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.168584+0000", 2024-02-05T08:25:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.57e-06 2024-02-05T08:25:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.616 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.168591+0000", 2024-02-05T08:25:59.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0929999999999997e-06 2024-02-05T08:25:59.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.168608+0000", 2024-02-05T08:25:59.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.6296000000000001e-05 2024-02-05T08:25:59.617 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.617 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.617 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.617 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.168618+0000", 2024-02-05T08:25:59.617 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0206e-05 2024-02-05T08:25:59.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.618 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.618 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.168708+0000", 2024-02-05T08:25:59.618 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9957000000000006e-05 2024-02-05T08:25:59.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.618 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.618 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.168723+0000", 2024-02-05T08:25:59.618 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5566999999999999e-05 2024-02-05T08:25:59.618 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.618 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.619 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.619 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.123022+0000", 2024-02-05T08:25:59.619 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.95429839699999996 2024-02-05T08:25:59.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.619 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.619 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.123031+0000", 2024-02-05T08:25:59.619 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1600000000000004e-06 2024-02-05T08:25:59.619 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.619 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.619 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.619 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.239239+0000", 2024-02-05T08:25:59.620 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.116208157 2024-02-05T08:25:59.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.620 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:25:59.620 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.239370+0000", 2024-02-05T08:25:59.620 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000130946 2024-02-05T08:25:59.620 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.620 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.620 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,1,4", 2024-02-05T08:25:59.620 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.239521+0000", 2024-02-05T08:25:59.620 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000151521 2024-02-05T08:25:59.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.621 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.621 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.242183+0000", 2024-02-05T08:25:59.621 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002661479 2024-02-05T08:25:59.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.621 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.621 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.249793+0000", 2024-02-05T08:25:59.621 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0076098950000000002 2024-02-05T08:25:59.621 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.621 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.622 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:25:59.622 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.249807+0000", 2024-02-05T08:25:59.622 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3902e-05 2024-02-05T08:25:59.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.622 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.622 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.249845+0000", 2024-02-05T08:25:59.622 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8086000000000001e-05 2024-02-05T08:25:59.622 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.622 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:25:59.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.249900+0000", 2024-02-05T08:25:59.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5649000000000002e-05 2024-02-05T08:25:59.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.623 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.623 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.623 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.250009+0000", 2024-02-05T08:25:59.623 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000108387 2024-02-05T08:25:59.623 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.623 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.623 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.623 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.624 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.624 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212127 11.2 11:4cd4a940:::benchmark_data_smithi134_205380_object77025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-02-05T08:25:59.624 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:48.208237+0000", 2024-02-05T08:25:59.624 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 11.270562403, 2024-02-05T08:25:59.624 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0267604279999998, 2024-02-05T08:25:59.624 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:25:59.624 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:25:59.624 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:25:59.624 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:25:59.624 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:25:59.625 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212127 2024-02-05T08:25:59.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.625 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:25:59.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.625 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:25:59.625 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.208237+0000", 2024-02-05T08:25:59.625 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.625 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.625 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.625 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:25:59.626 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.208237+0000", 2024-02-05T08:25:59.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:25:59.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.626 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:25:59.626 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.208239+0000", 2024-02-05T08:25:59.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9290000000000001e-06 2024-02-05T08:25:59.626 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.626 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.626 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:25:59.626 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.208247+0000", 2024-02-05T08:25:59.626 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.7050000000000005e-06 2024-02-05T08:25:59.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.627 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:25:59.627 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.208270+0000", 2024-02-05T08:25:59.627 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2243e-05 2024-02-05T08:25:59.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.627 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:25:59.627 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.208281+0000", 2024-02-05T08:25:59.627 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1928e-05 2024-02-05T08:25:59.627 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.627 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.628 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.628 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.208352+0000", 2024-02-05T08:25:59.628 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.0607999999999999e-05 2024-02-05T08:25:59.628 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.628 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.628 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.628 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:48.208361+0000", 2024-02-05T08:25:59.628 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8559999999999996e-06 2024-02-05T08:25:59.628 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.628 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.629 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.629 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.113033+0000", 2024-02-05T08:25:59.629 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.90467164700000002 2024-02-05T08:25:59.629 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.629 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.629 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:25:59.629 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.113042+0000", 2024-02-05T08:25:59.629 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1349999999999998e-06 2024-02-05T08:25:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:25:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.229340+0000", 2024-02-05T08:25:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.11629801 2024-02-05T08:25:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:25:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.229540+0000", 2024-02-05T08:25:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020022900000000001 2024-02-05T08:25:59.630 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,2,7", 2024-02-05T08:25:59.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.229683+0000", 2024-02-05T08:25:59.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014279399999999999 2024-02-05T08:25:59.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.631 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:25:59.631 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.232326+0000", 2024-02-05T08:25:59.631 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0026429190000000001 2024-02-05T08:25:59.631 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.631 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.632 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.232817+0000", 2024-02-05T08:25:59.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00049092899999999997 2024-02-05T08:25:59.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.632 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.233267+0000", 2024-02-05T08:25:59.632 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000450087 2024-02-05T08:25:59.632 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.632 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.632 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:25:59.633 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.234771+0000", 2024-02-05T08:25:59.633 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0015041500000000001 2024-02-05T08:25:59.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.633 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.633 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:25:59.633 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.234835+0000", 2024-02-05T08:25:59.633 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4502e-05 2024-02-05T08:25:59.633 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:25:59.633 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:25:59.633 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:25:59.633 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.234997+0000", 2024-02-05T08:25:59.633 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000161757 2024-02-05T08:25:59.634 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.634 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.634 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.634 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:25:59.634 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:25:59.634 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:25:59.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:59.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:59.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:25:59.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:25:59.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:59.822 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:25:59.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:25:59.923 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:25:59.953 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:25:59.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:25:59.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:00.055 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:00.101 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:00.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:00.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:00.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:00.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:00.242 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:00.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:00.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:00 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:26:00.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:00 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:26:00.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:00 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:26:00.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:00 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:26:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:00 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:26:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:00 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:26:00.343 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:00.385 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:00.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:00.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:00.527 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:00.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:00.629 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:00.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:00.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:00.767 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:00.809 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:00.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:00.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:00.911 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:00.951 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:00.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:01.053 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:01.095 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:01.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:01.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:01.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:01.237 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:01.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:01 smithi196 ceph-mon[33018]: pgmap v952: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.03k op/s; 13493/150240 objects misplaced (8.981%); 17 MiB/s, 271 objects/s recovering 2024-02-05T08:26:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:01 smithi196 ceph-mon[33018]: osdmap e413: 8 total, 8 up, 8 in 2024-02-05T08:26:01.337 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:01.358 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:01 smithi134 ceph-mon[30086]: pgmap v952: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.03k op/s; 13493/150240 objects misplaced (8.981%); 17 MiB/s, 271 objects/s recovering 2024-02-05T08:26:01.359 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:01 smithi134 ceph-mon[30086]: osdmap e413: 8 total, 8 up, 8 in 2024-02-05T08:26:01.359 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:01 smithi134 ceph-mon[39278]: pgmap v952: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.03k op/s; 13493/150240 objects misplaced (8.981%); 17 MiB/s, 271 objects/s recovering 2024-02-05T08:26:01.359 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:01 smithi134 ceph-mon[39278]: osdmap e413: 8 total, 8 up, 8 in 2024-02-05T08:26:01.369 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:01.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:01.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:01.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:01.502 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:01.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:01.603 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:01.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:01.644 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:01.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:01.748 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:01.783 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:01.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:01.852 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T08:26:01.885 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:01.936 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:01.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:02.037 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:02.076 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:02.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:02.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:02.206 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:26:02.223 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:02.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:02.326 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:02.371 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:02.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:02.472 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:02.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:02.506 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:02.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:02.607 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:02.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:02.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:02.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:02.778 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:02.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:02.880 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:02.917 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:02.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:03.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:03.049 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:03.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:03.151 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:03.196 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:03.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:03.224 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:03 smithi134 ceph-mon[30086]: pgmap v954: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 12766/147333 objects misplaced (8.665%); 20 MiB/s, 323 objects/s recovering 2024-02-05T08:26:03.224 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:03 smithi134 ceph-mon[39278]: pgmap v954: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 12766/147333 objects misplaced (8.665%); 20 MiB/s, 323 objects/s recovering 2024-02-05T08:26:03.301 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:03 smithi196 ceph-mon[33018]: pgmap v954: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 12766/147333 objects misplaced (8.665%); 20 MiB/s, 323 objects/s recovering 2024-02-05T08:26:03.342 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:03.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:03.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:03.475 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:03.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:03.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:03.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:03.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:03.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:03.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:03.749 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:03.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:03.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:03.851 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:03.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:03.903 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:03.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:04.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:04.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:04.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:04.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:04.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:04.169 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:04.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:04.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:04 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1770536031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:04.179 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:04 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1770536031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:04.270 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:04.308 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:04.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:04 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1770536031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:04.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:26:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:26:04.413 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:04.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:04.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:04.457 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:26:04.457 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:26:04.554 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:04.599 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:04.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:04.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:04.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:04.737 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:04.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:04.839 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:04.873 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:04.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:04.975 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:05.012 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:05.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:05.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:05.105 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:26:05.114 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:05.147 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:05.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:05.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:05.176 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:05 smithi196 ceph-mon[33018]: pgmap v955: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 10697/134409 objects misplaced (7.959%); 22 MiB/s, 344 objects/s recovering 2024-02-05T08:26:05.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:05 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:05.177 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:05 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3334331985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:26:05.221 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T08:26:05.249 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:05.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:05 smithi134 ceph-mon[30086]: pgmap v955: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 10697/134409 objects misplaced (7.959%); 22 MiB/s, 344 objects/s recovering 2024-02-05T08:26:05.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:05 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:05 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3334331985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:26:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:05 smithi134 ceph-mon[39278]: pgmap v955: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 10697/134409 objects misplaced (7.959%); 22 MiB/s, 344 objects/s recovering 2024-02-05T08:26:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:05 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:05 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3334331985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:26:05.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:05.293 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:05.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:05.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:05.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:05.434 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:05.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:05.501 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:26:05.501 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:26:05.502 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:26:05.502 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:26:05.502 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:26:05.502 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:26:05.502 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:26:05.502 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:26:05.502 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:26:05.502 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:26:05.502 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:26:05.502 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:26:05.502 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:26:05.502 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:26:05.503 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:26:05.503 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:26:05.536 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:05.583 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:05.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:05.590 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:26:05.684 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:05.722 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:05.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:05.823 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:05.859 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:05.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:05.961 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:05.986 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:05.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:06.088 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:06.128 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:06.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:06.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:06.229 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:06.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:06.260 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:06.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:06 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:26:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:06 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3334331985' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:26:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:06 smithi196 ceph-mon[33018]: osdmap e414: 8 total, 8 up, 8 in 2024-02-05T08:26:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:06 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:26:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:06 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:26:06.361 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:06.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:06.401 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:06.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:06.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:06.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:06 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:26:06.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:06 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3334331985' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:26:06.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:06 smithi134 ceph-mon[30086]: osdmap e414: 8 total, 8 up, 8 in 2024-02-05T08:26:06.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:06 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:26:06.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:06 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:26:06.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:06 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:26:06.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:06 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3334331985' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:26:06.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:06 smithi134 ceph-mon[39278]: osdmap e414: 8 total, 8 up, 8 in 2024-02-05T08:26:06.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:06 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:26:06.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:06 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:26:06.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:06.546 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:06.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:06.647 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:06.675 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:06.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:06.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:06.777 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:06.813 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:06.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:06.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:06.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:06.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:06.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:07.041 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:07.077 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:07.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:07.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:07.211 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:07.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:07.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:07.313 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:07.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:07.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:07.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:07.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:07 smithi196 ceph-mon[33018]: pgmap v957: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.16k op/s; 10697/134409 objects misplaced (7.959%); 19 MiB/s, 301 objects/s recovering 2024-02-05T08:26:07.386 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:07 smithi196 ceph-mon[33018]: osdmap e415: 8 total, 8 up, 8 in 2024-02-05T08:26:07.446 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:07.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:07.480 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:07.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:07.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:07 smithi134 ceph-mon[30086]: pgmap v957: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.16k op/s; 10697/134409 objects misplaced (7.959%); 19 MiB/s, 301 objects/s recovering 2024-02-05T08:26:07.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:07 smithi134 ceph-mon[30086]: osdmap e415: 8 total, 8 up, 8 in 2024-02-05T08:26:07.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:07 smithi134 ceph-mon[39278]: pgmap v957: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.16k op/s; 10697/134409 objects misplaced (7.959%); 19 MiB/s, 301 objects/s recovering 2024-02-05T08:26:07.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:07 smithi134 ceph-mon[39278]: osdmap e415: 8 total, 8 up, 8 in 2024-02-05T08:26:07.580 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:07.616 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:07.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:07.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:07.747 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:07.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:07.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:07.849 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:07.885 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:07.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:07.987 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:08.024 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:08.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:08.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:08.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:26:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:26:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:26:08.127 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:08.137 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:26:08.137 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:26:08.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:08.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:08.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:08.305 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:08.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:08.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:08.407 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:08.447 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:08.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:08.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:08.590 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:08.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:08.691 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:08.731 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:08.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:08.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:08.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:08.874 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:08.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:08.975 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:09.014 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:09.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:09.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:09.116 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:09.159 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:09.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:09.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:09.261 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:09.305 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:09.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:09.342 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T08:26:09.376 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:09 smithi196 ceph-mon[33018]: pgmap v959: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 791 op/s; 10087/131481 objects misplaced (7.672%); 15 MiB/s, 234 objects/s recovering 2024-02-05T08:26:09.376 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:09 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:26:09.407 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:09.446 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:09.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:09.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:09 smithi134 ceph-mon[30086]: pgmap v959: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 791 op/s; 10087/131481 objects misplaced (7.672%); 15 MiB/s, 234 objects/s recovering 2024-02-05T08:26:09.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:09 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:26:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:09 smithi134 ceph-mon[39278]: pgmap v959: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 791 op/s; 10087/131481 objects misplaced (7.672%); 15 MiB/s, 234 objects/s recovering 2024-02-05T08:26:09.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:09 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:26:09.548 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:09.586 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:09.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:09.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:09.714 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:26:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:09.725 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:09.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:09.827 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:09.862 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:09.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:09.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:09.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:09.990 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:09.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:10.092 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:10.131 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:10.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:10.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:10.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:10.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:10.273 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:10.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:10.374 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:10.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:10.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:10.512 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:10.552 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:10.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:10.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:10.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:10.697 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:10.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:10.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:10.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:10.831 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:10.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:10.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:10.962 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:10.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:11.063 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:11.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:10 smithi196 ceph-mon[33018]: pgmap v960: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 8436/120558 objects misplaced (6.997%); 17 MiB/s, 276 objects/s recovering 2024-02-05T08:26:11.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:11.106 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:11.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:11.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:11.234 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:10 smithi134 ceph-mon[30086]: pgmap v960: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 8436/120558 objects misplaced (6.997%); 17 MiB/s, 276 objects/s recovering 2024-02-05T08:26:11.236 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:10 smithi134 ceph-mon[39278]: pgmap v960: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 8436/120558 objects misplaced (6.997%); 17 MiB/s, 276 objects/s recovering 2024-02-05T08:26:11.246 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:11.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:11.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:11.348 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:11.384 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:11.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:11.486 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:11.521 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:11.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:11.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:11.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:11.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:11.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:11.772 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:11.813 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:11.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:11.915 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:11.944 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:26:11.945 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:26:11.955 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:11.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:11.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:12.057 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:12.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:12.091 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:12.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:12.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:12.225 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:12.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:12.327 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:12.361 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:12.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:12.463 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:12.516 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:12.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:12.618 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:12.659 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:12.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:12.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:12.761 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:12.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T08:26:12.801 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:12.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:12.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:12.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:12.940 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:12.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:13.042 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:13.082 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:13.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:13.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:13.185 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:26:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:13.261 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:13.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:13.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:13 smithi134 ceph-mon[30086]: pgmap v961: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 694 op/s; 7760/117756 objects misplaced (6.590%); 14 MiB/s, 229 objects/s recovering 2024-02-05T08:26:13.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:13 smithi134 ceph-mon[39278]: pgmap v961: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 694 op/s; 7760/117756 objects misplaced (6.590%); 14 MiB/s, 229 objects/s recovering 2024-02-05T08:26:13.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:26:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:13 smithi196 ceph-mon[33018]: pgmap v961: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 694 op/s; 7760/117756 objects misplaced (6.590%); 14 MiB/s, 229 objects/s recovering 2024-02-05T08:26:13.362 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:13.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:13.412 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:13.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:13.513 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:13.555 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:13.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:13.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:13.699 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:13.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:13.800 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:13.837 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:13.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:13.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:13.939 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:13.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:13.989 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:13.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:14.090 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:14.162 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:14.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:14.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:14.293 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:14.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:14.395 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:14.437 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:26:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:26:14.449 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:14.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:14.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:14.579 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:14.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:14.680 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:14.722 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:14.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:14.754 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:14.824 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:14.873 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:14.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:14.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:15.004 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:15.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:15.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:15.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:15.137 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:15.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:15.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:15.218 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:15 smithi134 ceph-mon[30086]: pgmap v962: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.13k op/s; 5822/104457 objects misplaced (5.574%); 20 MiB/s, 322 objects/s recovering 2024-02-05T08:26:15.218 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:15 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:15.218 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:15 smithi134 ceph-mon[39278]: pgmap v962: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.13k op/s; 5822/104457 objects misplaced (5.574%); 20 MiB/s, 322 objects/s recovering 2024-02-05T08:26:15.218 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:15 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:15.239 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:15.263 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:26:15.268 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:15.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:15 smithi196 ceph-mon[33018]: pgmap v962: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.13k op/s; 5822/104457 objects misplaced (5.574%); 20 MiB/s, 322 objects/s recovering 2024-02-05T08:26:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:15 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:15.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:15.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:15.409 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:15.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:15.510 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:15.536 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:15.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:15.638 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:15.681 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:15.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:15.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:15.810 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:15.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:15.912 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:15.946 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:15.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:16.040 INFO:teuthology.orchestra.run.smithi134.stderr:set osd.5 primary-affinity to 0.554926 (8363672) 2024-02-05T08:26:16.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:16.081 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:16.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:16.183 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:16.216 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:16.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:16.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:16.251 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:16 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1418090694' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5549260869338017}]: dispatch 2024-02-05T08:26:16.266 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:26:16.266 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:26:16.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:16 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1418090694' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5549260869338017}]: dispatch 2024-02-05T08:26:16.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:16 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1418090694' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5549260869338017}]: dispatch 2024-02-05T08:26:16.318 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:16.348 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:16.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:16.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:16.450 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:16.478 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:16.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:16.584 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:16.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:16.609 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:16.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:16.710 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:16.734 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:16.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:16.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:16.865 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:16.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:16.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:16.967 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:16.991 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:16.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:16.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:17.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:17.132 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:17.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:17.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:17.234 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:17.262 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:17.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:17.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:17 smithi196 ceph-mon[33018]: pgmap v963: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s; 5822/104457 objects misplaced (5.574%); 18 MiB/s, 285 objects/s recovering 2024-02-05T08:26:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:17 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1418090694' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5549260869338017}]': finished 2024-02-05T08:26:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:17 smithi196 ceph-mon[33018]: osdmap e416: 8 total, 8 up, 8 in 2024-02-05T08:26:17.364 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:17.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T08:26:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:17.390 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:17.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:17.491 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:17.519 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:17.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:17.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:17.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:17 smithi134 ceph-mon[30086]: pgmap v963: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s; 5822/104457 objects misplaced (5.574%); 18 MiB/s, 285 objects/s recovering 2024-02-05T08:26:17.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:17 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1418090694' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5549260869338017}]': finished 2024-02-05T08:26:17.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:17 smithi134 ceph-mon[30086]: osdmap e416: 8 total, 8 up, 8 in 2024-02-05T08:26:17.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:17 smithi134 ceph-mon[39278]: pgmap v963: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 153 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s; 5822/104457 objects misplaced (5.574%); 18 MiB/s, 285 objects/s recovering 2024-02-05T08:26:17.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:17 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1418090694' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.5549260869338017}]': finished 2024-02-05T08:26:17.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:17 smithi134 ceph-mon[39278]: osdmap e416: 8 total, 8 up, 8 in 2024-02-05T08:26:17.621 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:17.649 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:17.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:17.704 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:26:17.751 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:17.759 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:17.776 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:17.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:17.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:17.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:17.903 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:17.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:17.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:18.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:18.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:26:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:26:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:26:18.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:18.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:18.130 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:18.161 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:18.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:18.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:18.263 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:18.296 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:18.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:18.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:18 smithi196 ceph-mon[33018]: osdmap e417: 8 total, 8 up, 8 in 2024-02-05T08:26:18.356 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:18 smithi134 ceph-mon[30086]: osdmap e417: 8 total, 8 up, 8 in 2024-02-05T08:26:18.356 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:18 smithi134 ceph-mon[39278]: osdmap e417: 8 total, 8 up, 8 in 2024-02-05T08:26:18.398 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:18.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:18.442 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:18.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:18.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:18.584 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:18.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:18.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:18.686 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:18.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:18.720 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:18.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:18.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:18.871 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:18.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:18.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:18.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:19.007 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:19.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:19.108 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:19.148 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:19.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:19.250 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:19.289 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:19.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:19.391 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:19.403 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:19 smithi196 ceph-mon[33018]: pgmap v966: 158 pgs: 2 peering, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 151 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 782 op/s; 5246/101784 objects misplaced (5.154%); 17 MiB/s, 265 objects/s recovering 2024-02-05T08:26:19.403 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:19 smithi196 ceph-mon[33018]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-05T08:26:19.403 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:19 smithi196 ceph-mon[33018]: osdmap e418: 8 total, 8 up, 8 in 2024-02-05T08:26:19.419 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:19 smithi134 ceph-mon[30086]: pgmap v966: 158 pgs: 2 peering, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 151 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 782 op/s; 5246/101784 objects misplaced (5.154%); 17 MiB/s, 265 objects/s recovering 2024-02-05T08:26:19.420 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:19 smithi134 ceph-mon[30086]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-05T08:26:19.420 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:19 smithi134 ceph-mon[30086]: osdmap e418: 8 total, 8 up, 8 in 2024-02-05T08:26:19.420 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:19 smithi134 ceph-mon[39278]: pgmap v966: 158 pgs: 2 peering, 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 151 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 782 op/s; 5246/101784 objects misplaced (5.154%); 17 MiB/s, 265 objects/s recovering 2024-02-05T08:26:19.420 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:19 smithi134 ceph-mon[39278]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-05T08:26:19.420 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:19 smithi134 ceph-mon[39278]: osdmap e418: 8 total, 8 up, 8 in 2024-02-05T08:26:19.429 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:19.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:19.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:19.566 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:19.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:19.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:19.668 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:19.703 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:19.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:19.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:19.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:19.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:19.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:19.980 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:19.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:20.082 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:20.090 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:26:20.136 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:20.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:20.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:20.161 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:20 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2130259750' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:26:20.161 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:20 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:26:20.161 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:20 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2130259750' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:26:20.161 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:20 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:26:20.238 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:20.283 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:20.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:20.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:20 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2130259750' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:26:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:20 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:26:20.386 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:20.424 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:20.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:20.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:20.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:20.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:20.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:20.677 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:26:20.679 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:26:20.714 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:20.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:20.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:20.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:20.854 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:20.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:20.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:20.988 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:20.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:20.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:21.090 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:21.119 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:21.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:21.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:21.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:21.265 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:21.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:21.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:21 smithi134 ceph-mon[30086]: pgmap v968: 158 pgs: 2 peering, 2 active+remapped+backfilling, 154 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 9.5 KiB/s rd, 73 KiB/s wr, 609 op/s; 1906/93687 objects misplaced (2.034%); 36 MiB/s, 571 objects/s recovering 2024-02-05T08:26:21.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:21 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:26:21.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:21 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:26:21.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:21 smithi134 ceph-mon[30086]: osdmap e419: 8 total, 8 up, 8 in 2024-02-05T08:26:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:21 smithi134 ceph-mon[39278]: pgmap v968: 158 pgs: 2 peering, 2 active+remapped+backfilling, 154 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 9.5 KiB/s rd, 73 KiB/s wr, 609 op/s; 1906/93687 objects misplaced (2.034%); 36 MiB/s, 571 objects/s recovering 2024-02-05T08:26:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:21 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:26:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:21 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:26:21.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:21 smithi134 ceph-mon[39278]: osdmap e419: 8 total, 8 up, 8 in 2024-02-05T08:26:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:21 smithi196 ceph-mon[33018]: pgmap v968: 158 pgs: 2 peering, 2 active+remapped+backfilling, 154 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 9.5 KiB/s rd, 73 KiB/s wr, 609 op/s; 1906/93687 objects misplaced (2.034%); 36 MiB/s, 571 objects/s recovering 2024-02-05T08:26:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:21 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:26:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:21 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:26:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:21 smithi196 ceph-mon[33018]: osdmap e419: 8 total, 8 up, 8 in 2024-02-05T08:26:21.368 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:21.400 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:21.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:21.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:21.503 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:21.511 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:26:21.511 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:26:21.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T08:26:21.525 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:26:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:21.540 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:21.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:21.570 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:26:21.612 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:26:21.642 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:21.650 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:26:21.679 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:21.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:21.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:21.701 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:26:21.742 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:26:21.753 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:26:21.753 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:26:21.753 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:26:21.753 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:26:21.753 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:26:21.753 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:26:21.753 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:26:21.753 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:26:21.781 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:21.823 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:21.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:21.893 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:26:21.925 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:21.970 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:21.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:22.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:22.095 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 7, 6, 3, 0] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-02-05T08:26:22.095 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:26:22.095 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-05T08:26:22.096 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T08:26:22.106 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:22.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:22.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:22.208 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:22.268 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:22.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:22.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:22 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4177773445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:22.371 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:22.410 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:22.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:22.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:22.512 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:22.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:22 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4177773445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:22.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:22 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4177773445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:22.541 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:22.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:22.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:22.671 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:22.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:22.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:22.684 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:22.773 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:22.826 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:22.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:22.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:22.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:22.959 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:22.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:22.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:23.019 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:26:23.061 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:23.099 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:23.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:23.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:23.227 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:23.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:23.328 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:23.369 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:23.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:23.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:23.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:23.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:23.495 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:23.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:23 smithi134 ceph-mon[30086]: 5.16 scrub starts 2024-02-05T08:26:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:23 smithi134 ceph-mon[30086]: 5.16 scrub ok 2024-02-05T08:26:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:23 smithi134 ceph-mon[30086]: pgmap v970: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 9.6 KiB/s rd, 74 KiB/s wr, 690 op/s; 1906/92364 objects misplaced (2.064%); 36 MiB/s, 578 objects/s recovering 2024-02-05T08:26:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:23 smithi134 ceph-mon[30086]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-05T08:26:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:23 smithi134 ceph-mon[30086]: 1.0 scrub starts 2024-02-05T08:26:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:23 smithi134 ceph-mon[30086]: 1.0 scrub ok 2024-02-05T08:26:23.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:23 smithi134 ceph-mon[39278]: 5.16 scrub starts 2024-02-05T08:26:23.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:23 smithi134 ceph-mon[39278]: 5.16 scrub ok 2024-02-05T08:26:23.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:23 smithi134 ceph-mon[39278]: pgmap v970: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 9.6 KiB/s rd, 74 KiB/s wr, 690 op/s; 1906/92364 objects misplaced (2.064%); 36 MiB/s, 578 objects/s recovering 2024-02-05T08:26:23.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:23 smithi134 ceph-mon[39278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-05T08:26:23.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:23 smithi134 ceph-mon[39278]: 1.0 scrub starts 2024-02-05T08:26:23.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:23 smithi134 ceph-mon[39278]: 1.0 scrub ok 2024-02-05T08:26:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:23 smithi196 ceph-mon[33018]: 5.16 scrub starts 2024-02-05T08:26:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:23 smithi196 ceph-mon[33018]: 5.16 scrub ok 2024-02-05T08:26:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:23 smithi196 ceph-mon[33018]: pgmap v970: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 9.6 KiB/s rd, 74 KiB/s wr, 690 op/s; 1906/92364 objects misplaced (2.064%); 36 MiB/s, 578 objects/s recovering 2024-02-05T08:26:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:23 smithi196 ceph-mon[33018]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-05T08:26:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:23 smithi196 ceph-mon[33018]: 1.0 scrub starts 2024-02-05T08:26:23.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:23 smithi196 ceph-mon[33018]: 1.0 scrub ok 2024-02-05T08:26:23.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:23.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:23.665 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:23.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:23.766 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:23.800 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:23.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:23.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:23.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:23.939 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:23.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:23.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:24.041 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:24.079 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:24.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:24.181 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:24.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:24.215 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:24.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[30086]: 5.1c scrub starts 2024-02-05T08:26:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[30086]: 5.1c scrub ok 2024-02-05T08:26:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[30086]: 6.3 scrub starts 2024-02-05T08:26:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[30086]: 1.a scrub starts 2024-02-05T08:26:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[30086]: 1.a scrub ok 2024-02-05T08:26:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[30086]: 6.3 scrub ok 2024-02-05T08:26:24.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:26:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[39278]: 5.1c scrub starts 2024-02-05T08:26:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[39278]: 5.1c scrub ok 2024-02-05T08:26:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[39278]: 6.3 scrub starts 2024-02-05T08:26:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[39278]: 1.a scrub starts 2024-02-05T08:26:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[39278]: 1.a scrub ok 2024-02-05T08:26:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[39278]: 6.3 scrub ok 2024-02-05T08:26:24.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:24 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:26:24.316 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:24.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:24.358 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:24.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:24.459 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:24.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:24 smithi196 ceph-mon[33018]: 5.1c scrub starts 2024-02-05T08:26:24.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:24 smithi196 ceph-mon[33018]: 5.1c scrub ok 2024-02-05T08:26:24.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:24 smithi196 ceph-mon[33018]: 6.3 scrub starts 2024-02-05T08:26:24.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:24 smithi196 ceph-mon[33018]: 1.a scrub starts 2024-02-05T08:26:24.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:24 smithi196 ceph-mon[33018]: 1.a scrub ok 2024-02-05T08:26:24.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:24 smithi196 ceph-mon[33018]: 6.3 scrub ok 2024-02-05T08:26:24.478 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:24 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:26:24.478 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:26:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:26:24.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:24.504 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:24.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:24.583 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:26:24.605 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:24.652 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:24.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:24.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:24.754 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:24.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:24.785 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:24.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:24.814 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:26:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a[66672]: logger=cleanup t=2024-02-05T08:26:24.478464424Z level=info msg="Completed cleanup jobs" duration=6.05593ms 2024-02-05T08:26:24.887 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:24.928 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:24.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:25.030 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:25.069 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:25.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:25.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:25.171 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:25.210 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:25.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:25.312 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:25 smithi196 ceph-mon[33018]: 1.5 scrub starts 2024-02-05T08:26:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:25 smithi196 ceph-mon[33018]: 1.5 scrub ok 2024-02-05T08:26:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:25 smithi196 ceph-mon[33018]: 6.7 scrub starts 2024-02-05T08:26:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:25 smithi196 ceph-mon[33018]: pgmap v971: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 5.2 KiB/s rd, 32 KiB/s wr, 1.05k op/s; 575/79986 objects misplaced (0.719%); 36 MiB/s, 583 objects/s recovering 2024-02-05T08:26:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:25 smithi196 ceph-mon[33018]: 6.7 scrub ok 2024-02-05T08:26:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:25 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:25.357 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:25.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:25.458 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:25.497 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:25.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:25.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:25 smithi134 ceph-mon[30086]: 1.5 scrub starts 2024-02-05T08:26:25.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:25 smithi134 ceph-mon[30086]: 1.5 scrub ok 2024-02-05T08:26:25.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:25 smithi134 ceph-mon[30086]: 6.7 scrub starts 2024-02-05T08:26:25.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:25 smithi134 ceph-mon[30086]: pgmap v971: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 5.2 KiB/s rd, 32 KiB/s wr, 1.05k op/s; 575/79986 objects misplaced (0.719%); 36 MiB/s, 583 objects/s recovering 2024-02-05T08:26:25.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:25 smithi134 ceph-mon[30086]: 6.7 scrub ok 2024-02-05T08:26:25.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:25 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:25.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:25 smithi134 ceph-mon[39278]: 1.5 scrub starts 2024-02-05T08:26:25.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:25 smithi134 ceph-mon[39278]: 1.5 scrub ok 2024-02-05T08:26:25.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:25 smithi134 ceph-mon[39278]: 6.7 scrub starts 2024-02-05T08:26:25.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:25 smithi134 ceph-mon[39278]: pgmap v971: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 5.2 KiB/s rd, 32 KiB/s wr, 1.05k op/s; 575/79986 objects misplaced (0.719%); 36 MiB/s, 583 objects/s recovering 2024-02-05T08:26:25.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:25 smithi134 ceph-mon[39278]: 6.7 scrub ok 2024-02-05T08:26:25.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:25 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:25.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:25.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:25.634 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:25.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:25.735 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:25.774 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:25.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:25.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:25.875 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:25.918 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:25.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:25.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:26.020 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:26.050 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:26.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:26.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:26.175 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:26.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:26.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:26.179 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:26 smithi196 ceph-mon[33018]: 4.12 scrub starts 2024-02-05T08:26:26.179 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:26 smithi196 ceph-mon[33018]: 4.12 scrub ok 2024-02-05T08:26:26.179 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:26 smithi196 ceph-mon[33018]: 11.e scrub starts 2024-02-05T08:26:26.179 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:26 smithi196 ceph-mon[33018]: osdmap e420: 8 total, 8 up, 8 in 2024-02-05T08:26:26.179 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:26 smithi196 ceph-mon[33018]: 11.e scrub ok 2024-02-05T08:26:26.210 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:26:26.210 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:26:26.210 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:26:26.210 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:26:26.211 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.211 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15714.0:225863 10.6 10:6c4cccde:::benchmark_data_smithi134_174861_object94401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-02-05T08:26:26.211 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:22:08.078103+0000", 2024-02-05T08:26:26.211 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 258.130151641, 2024-02-05T08:26:26.211 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1085411509999998, 2024-02-05T08:26:26.211 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.211 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:26:26.211 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.211 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15714", 2024-02-05T08:26:26.212 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:26:26.212 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225863 2024-02-05T08:26:26.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.212 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078103+0000", 2024-02-05T08:26:26.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.212 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.212 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.212 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.212 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078103+0000", 2024-02-05T08:26:26.212 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078105+0000", 2024-02-05T08:26:26.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1670000000000002e-06 2024-02-05T08:26:26.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.213 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078110+0000", 2024-02-05T08:26:26.213 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6959999999999996e-06 2024-02-05T08:26:26.213 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.213 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.213 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078127+0000", 2024-02-05T08:26:26.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7450000000000001e-05 2024-02-05T08:26:26.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078137+0000", 2024-02-05T08:26:26.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.8689999999999999e-06 2024-02-05T08:26:26.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.214 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.214 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.214 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078203+0000", 2024-02-05T08:26:26.214 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6507999999999994e-05 2024-02-05T08:26:26.214 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078348+0000", 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014468599999999999 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078468+0000", 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000119889 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.115112+0000", 2024-02-05T08:26:26.215 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.036644058 2024-02-05T08:26:26.216 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.216 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.216 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:26:26.216 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.115125+0000", 2024-02-05T08:26:26.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3023000000000001e-05 2024-02-05T08:26:26.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.087270+0000", 2024-02-05T08:26:26.219 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97214539099999997 2024-02-05T08:26:26.219 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.219 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.219 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:26:26.219 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.087284+0000", 2024-02-05T08:26:26.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3688e-05 2024-02-05T08:26:26.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.186549+0000", 2024-02-05T08:26:26.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.099264815 2024-02-05T08:26:26.220 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.220 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.220 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.220 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.186644+0000", 2024-02-05T08:26:26.220 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4911e-05 2024-02-05T08:26:26.220 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.220 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15714.0:225864 10.6 10:6e872cbe:::benchmark_data_smithi134_174861_object94402:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:22:08.078902+0000", 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 258.12935197100001, 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1287934449999999, 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15714", 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225864 2024-02-05T08:26:26.221 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078902+0000", 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078902+0000", 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.222 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078903+0000", 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2600000000000001e-07 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078907+0000", 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9230000000000004e-06 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078921+0000", 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3709999999999999e-05 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.223 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.078930+0000", 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6810000000000004e-06 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.091414+0000", 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012483935999999999 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.091529+0000", 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000115661 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.224 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.091648+0000", 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011900600000000001 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.115176+0000", 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.023528263000000001 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.115199+0000", 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2356000000000001e-05 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.225 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.089214+0000", 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97401506500000001 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.089225+0000", 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1571e-05 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.186954+0000", 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.097729011 2024-02-05T08:26:26.226 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for missing object", 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.187121+0000", 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000166479 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.207626+0000", 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.020504887999999999 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.207696+0000", 2024-02-05T08:26:26.227 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9968999999999994e-05 2024-02-05T08:26:26.228 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.228 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.228 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.228 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.228 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.228 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15714.0:225869 10.6 10:6176b0ec:::benchmark_data_smithi134_174861_object94407:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-02-05T08:26:26.228 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:22:08.079530+0000", 2024-02-05T08:26:26.228 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 258.12872396699998, 2024-02-05T08:26:26.228 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1276412310000001, 2024-02-05T08:26:26.228 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.228 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.228 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15714", 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225869 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.079530+0000", 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.079530+0000", 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.229 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.079531+0000", 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3099999999999998e-07 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.079534+0000", 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9299999999999999e-06 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.079542+0000", 2024-02-05T08:26:26.230 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.8779999999999998e-06 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.079547+0000", 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7629999999999997e-06 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.115251+0000", 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.035703603 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:26:26.231 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.115256+0000", 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6500000000000004e-06 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.089356+0000", 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.97410054499999998 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.089365+0000", 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0329999999999997e-06 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.232 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.187236+0000", 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.09787101 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.187360+0000", 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012386 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.187864+0000", 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00050348000000000005 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.233 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.195604+0000", 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077400400000000001 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.200859+0000", 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052555379999999997 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.206992+0000", 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0061325399999999997 2024-02-05T08:26:26.234 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.207062+0000", 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0358000000000006e-05 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.207172+0000", 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000109472 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.235 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:51736 11.7 11:e0970dfb:::benchmark_data_smithi134_205380_object51735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:23:44.724073+0000", 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 161.484180896, 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.082492602999999998, 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 51736 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.236 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.724073+0000", 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.724073+0000", 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.724075+0000", 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5960000000000001e-06 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.237 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.724126+0000", 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0773999999999999e-05 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.724149+0000", 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2963e-05 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.724158+0000", 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3219999999999999e-06 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.238 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.724228+0000", 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0001999999999998e-05 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.724405+0000", 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017694100000000001 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.724560+0000", 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015536799999999999 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.239 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.729120+0000", 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0045595590000000004 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.732601+0000", 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034809559999999999 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.806457+0000", 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.073855626999999993 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.240 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.241 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.241 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.806527+0000", 2024-02-05T08:26:26.241 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0252000000000004e-05 2024-02-05T08:26:26.242 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.242 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.242 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.242 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.806566+0000", 2024-02-05T08:26:26.242 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9242999999999997e-05 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:51748 11.7 11:e7ec628b:::benchmark_data_smithi134_205380_object51747:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:23:44.732814+0000", 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 161.47543988999999, 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.073857882, 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:26.243 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:26.244 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 51748 2024-02-05T08:26:26.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.244 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.732814+0000", 2024-02-05T08:26:26.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.244 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.244 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.244 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.732814+0000", 2024-02-05T08:26:26.244 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.244 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.732816+0000", 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.454e-06 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.732844+0000", 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.7983e-05 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.732861+0000", 2024-02-05T08:26:26.245 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.696e-05 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.732869+0000", 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6109999999999994e-06 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.733095+0000", 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00022596700000000001 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.733224+0000", 2024-02-05T08:26:26.246 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012830599999999999 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.733356+0000", 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000132285 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.739301+0000", 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0059450309999999999 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.247 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.739955+0000", 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00065388899999999995 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.806580+0000", 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.066624811000000006 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.806639+0000", 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9655999999999999e-05 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.248 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.806672+0000", 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2929000000000003e-05 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:51758 11.7 11:e9f7490e:::benchmark_data_smithi134_205380_object51757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e381)", 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:23:44.740600+0000", 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 161.467654818, 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.066273702000000004, 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.249 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 51758 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.740600+0000", 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.740600+0000", 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.250 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.740601+0000", 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8029999999999999e-06 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.740631+0000", 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9567999999999999e-05 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.740648+0000", 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7455000000000001e-05 2024-02-05T08:26:26.251 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.740657+0000", 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3739999999999998e-06 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.740710+0000", 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2964999999999999e-05 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.740833+0000", 2024-02-05T08:26:26.252 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012314699999999999 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.741081+0000", 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000247741 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.745356+0000", 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0042757489999999997 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.253 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.745393+0000", 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6981999999999999e-05 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.806679+0000", 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.061285775000000001 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.806788+0000", 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108832 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.254 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:23:44.806873+0000", 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5310999999999997e-05 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:119635 11.7 11:e27b13b0:::benchmark_data_smithi134_205380_object119634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:24:37.111857+0000", 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 109.096397709, 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.10303725699999999, 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.255 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.256 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:26.256 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:26.256 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 119635 2024-02-05T08:26:26.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.256 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.256 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.111857+0000", 2024-02-05T08:26:26.256 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.256 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.256 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.256 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.111857+0000", 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.111858+0000", 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.784e-06 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.111901+0000", 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2395999999999999e-05 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.257 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.111920+0000", 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8785999999999999e-05 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.111929+0000", 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5920000000000007e-06 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.112121+0000", 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000191647 2024-02-05T08:26:26.258 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.112489+0000", 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00036811999999999999 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.112659+0000", 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000170283 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.115719+0000", 2024-02-05T08:26:26.259 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0030595230000000002 2024-02-05T08:26:26.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.124411+0000", 2024-02-05T08:26:26.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0086919170000000004 2024-02-05T08:26:26.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.260 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.214729+0000", 2024-02-05T08:26:26.260 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.090318186999999994 2024-02-05T08:26:26.260 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.260 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.260 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.214837+0000", 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000108027 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.214894+0000", 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6994999999999999e-05 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:119741 11.7 11:ea1dba08:::benchmark_data_smithi134_205380_object119740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-02-05T08:26:26.261 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:24:37.202444+0000", 2024-02-05T08:26:26.262 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 109.005810275, 2024-02-05T08:26:26.262 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.126045674, 2024-02-05T08:26:26.262 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.262 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.262 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.262 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:26.262 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:26.262 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 119741 2024-02-05T08:26:26.262 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.262 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.262 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.262 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.262 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202444+0000", 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202444+0000", 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202446+0000", 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.759e-06 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.263 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202485+0000", 2024-02-05T08:26:26.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9104999999999999e-05 2024-02-05T08:26:26.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202502+0000", 2024-02-05T08:26:26.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7427000000000001e-05 2024-02-05T08:26:26.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.264 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.264 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.264 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202512+0000", 2024-02-05T08:26:26.264 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.3279999999999995e-06 2024-02-05T08:26:26.264 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202618+0000", 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106109 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202773+0000", 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015511799999999999 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202927+0000", 2024-02-05T08:26:26.265 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015409899999999999 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.208495+0000", 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0055677089999999997 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.216582+0000", 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0080867790000000005 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.266 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328372+0000", 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.111790263 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328431+0000", 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.906e-05 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328490+0000", 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8918000000000001e-05 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.267 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:119742 11.7 11:e316fad3:::benchmark_data_smithi134_205380_object119741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:24:37.202533+0000", 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 109.005721397, 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.12605560299999999, 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 119742 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.268 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202533+0000", 2024-02-05T08:26:26.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.269 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202533+0000", 2024-02-05T08:26:26.269 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.269 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.269 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.269 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202533+0000", 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.8999999999999997e-07 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202560+0000", 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.6145000000000001e-05 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202567+0000", 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.6199999999999999e-06 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.270 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.202573+0000", 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0929999999999999e-06 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.205626+0000", 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003052524 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.205757+0000", 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000131414 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.271 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.205901+0000", 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014334699999999999 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.209336+0000", 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00343575 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.217355+0000", 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0080189259999999991 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.272 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328499+0000", 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11114405199999999 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328547+0000", 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7549999999999997e-05 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328589+0000", 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1692e-05 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.273 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:119767 11.7 11:ee601d1d:::benchmark_data_smithi134_205380_object119766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:24:37.223557+0000", 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 108.984696984, 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.105167465, 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 119767 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.274 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.223557+0000", 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.223557+0000", 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.223559+0000", 2024-02-05T08:26:26.275 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.671e-06 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.223591+0000", 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1741e-05 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.223608+0000", 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.7187999999999999e-05 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.276 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.223617+0000", 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2609999999999999e-06 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.223954+0000", 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00033646199999999999 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.224246+0000", 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000292224 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:26:26.277 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.224465+0000", 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00021924299999999999 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.232657+0000", 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0081918479999999998 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.232692+0000", 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5472999999999997e-05 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.278 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328596+0000", 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.095903139999999998 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328643+0000", 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7407999999999997e-05 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328725+0000", 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1805999999999996e-05 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.279 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:119779 11.7 11:e767f031:::benchmark_data_smithi134_205380_object119778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:24:37.233973+0000", 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 108.974281198, 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.094847049000000003, 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 119779 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.280 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.233973+0000", 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.233973+0000", 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.233983+0000", 2024-02-05T08:26:26.281 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7769999999999997e-06 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.234015+0000", 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2089999999999999e-05 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.234029+0000", 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4445e-05 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.234038+0000", 2024-02-05T08:26:26.282 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.8529999999999998e-06 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.234197+0000", 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015861400000000001 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.234297+0000", 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.9699999999999998e-05 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:26:26.283 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.234464+0000", 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00016720099999999999 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.240591+0000", 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0061270229999999997 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.248369+0000", 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0077777749999999998 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.284 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328736+0000", 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.080366993999999997 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328785+0000", 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9826e-05 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328820+0000", 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4751e-05 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.285 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:119790 11.7 11:e9ba380c:::benchmark_data_smithi134_205380_object119789:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:24:37.244586+0000", 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 108.963668775, 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.084320952000000005, 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 119790 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.286 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.244586+0000", 2024-02-05T08:26:26.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.244586+0000", 2024-02-05T08:26:26.287 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.287 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.287 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.287 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.287 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.244587+0000", 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.716e-06 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.244646+0000", 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8217000000000002e-05 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.244659+0000", 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3176000000000001e-05 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.288 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.244668+0000", 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2629999999999992e-06 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.244773+0000", 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000105228 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.244909+0000", 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000136274 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.289 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.245059+0000", 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015000699999999999 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.249690+0000", 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.004630777 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.249726+0000", 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6111e-05 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.290 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328828+0000", 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.079101403000000001 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328877+0000", 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9403000000000001e-05 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:37.328907+0000", 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9377e-05 2024-02-05T08:26:26.291 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:129242 11.7 11:eb527c13:::benchmark_data_smithi134_205380_object129241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:24:44.918957+0000", 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 101.28929768899999, 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.094358542000000004, 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:26.292 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 129242 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:44.918957+0000", 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:44.918957+0000", 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.293 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:44.918959+0000", 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.973e-06 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:44.919018+0000", 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9644e-05 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:44.919038+0000", 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9632000000000001e-05 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.294 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:44.919047+0000", 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4369999999999996e-06 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:44.919123+0000", 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5859000000000007e-05 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:44.919306+0000", 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000182778 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.295 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:26:26.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:44.919500+0000", 2024-02-05T08:26:26.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019442 2024-02-05T08:26:26.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:44.922241+0000", 2024-02-05T08:26:26.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027403639999999999 2024-02-05T08:26:26.296 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.296 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.296 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.296 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:44.934747+0000", 2024-02-05T08:26:26.296 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012506549 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.013191+0000", 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.078443888000000003 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.013273+0000", 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1557999999999994e-05 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.013315+0000", 2024-02-05T08:26:26.297 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.244e-05 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25820.0:130318 11.5 e395/367 11:a4f65b7a:::benchmark_data_smithi134_205380_object130317:head v 395'8147, mlcod=395'8144)", 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:24:45.957224+0000", 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 100.251030598, 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.065081545000000005, 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.298 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.957224+0000", 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.957224+0000", 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.957225+0000", 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.47e-07 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.299 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.957261+0000", 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6281e-05 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.957277+0000", 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5749000000000001e-05 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.957290+0000", 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.313e-05 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.300 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.957460+0000", 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00017050499999999999 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.957482+0000", 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1285e-05 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:46.022256+0000", 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.064774440000000003 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.301 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:46.022305+0000", 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9208e-05 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25820.0:130319 11.5 e395/367 11:a5c25ef8:::benchmark_data_smithi134_205380_object130318:head v 395'8148, mlcod=395'8144)", 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:24:45.959911+0000", 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 100.248343713, 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.062440058999999999, 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.302 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.959911+0000", 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.959911+0000", 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.959912+0000", 2024-02-05T08:26:26.303 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.003e-06 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.959934+0000", 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2725e-05 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.959951+0000", 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6724000000000001e-05 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.304 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.959960+0000", 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.115e-06 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.960117+0000", 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015709799999999999 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.960143+0000", 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.5204000000000001e-05 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.305 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:46.022310+0000", 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.062167690999999997 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:46.022351+0000", 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.0499e-05 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25820.0:130326 11.5 e395/367 11:a45c0423:::benchmark_data_smithi134_205380_object130325:head v 395'8149, mlcod=395'8144)", 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:24:45.960632+0000", 2024-02-05T08:26:26.306 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 100.247622319, 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.061758155000000002, 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.960632+0000", 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.960632+0000", 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.307 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.960633+0000", 2024-02-05T08:26:26.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1900000000000002e-07 2024-02-05T08:26:26.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.308 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.960655+0000", 2024-02-05T08:26:26.308 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2302000000000001e-05 2024-02-05T08:26:26.308 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.308 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.308 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.960668+0000", 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2971e-05 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.960675+0000", 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6810000000000001e-06 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.965127+0000", 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00445219 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.309 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:45.965145+0000", 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8278999999999999e-05 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:46.022356+0000", 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.057210817999999997 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:24:46.022390+0000", 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4195e-05 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.310 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:174439 11.7 11:ede50808:::benchmark_data_smithi134_205380_object39337:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:25:19.738524+0000", 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 66.469730738999999, 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.076522649999999998, 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 174439 2024-02-05T08:26:26.311 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.738524+0000", 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.738524+0000", 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.312 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.738525+0000", 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0780000000000001e-06 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.738529+0000", 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6600000000000003e-06 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.757749+0000", 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.019219838 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.313 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.757760+0000", 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0869999999999999e-05 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.758513+0000", 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00075299700000000004 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.758633+0000", 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012013700000000001 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.314 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.758745+0000", 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000112009 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.761554+0000", 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002808732 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.762592+0000", 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001038141 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.315 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.814915+0000", 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.052322717999999997 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.815000+0000", 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5538999999999995e-05 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:19.815046+0000", 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.5930999999999998e-05 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.316 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24344.0:4344 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 94292702905344 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:25:48.097390+0000", 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 38.110864616000001, 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0557349709999999, 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24344", 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/1272526039", 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 4344 2024-02-05T08:26:26.317 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.097390+0000", 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.097390+0000", 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.318 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.097392+0000", 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3180000000000001e-06 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.097398+0000", 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.5430000000000001e-06 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.097418+0000", 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.0307000000000001e-05 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.319 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.097428+0000", 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0488e-05 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.097499+0000", 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1029e-05 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op must wait for map", 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.097505+0000", 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6030000000000004e-06 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.320 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.104474+0000", 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0069685800000000003 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.104486+0000", 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2373e-05 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:49.136149+0000", 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.031663459 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.321 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for degraded object", 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:49.136559+0000", 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00040955499999999998 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:49.149238+0000", 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.012678811 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:49.149706+0000", 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00046836099999999998 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.323 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.323 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:26.324 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:49.153125+0000", 2024-02-05T08:26:26.324 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0034185439999999999 2024-02-05T08:26:26.324 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.324 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.324 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.324 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.324 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.324 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:219737 11.7 11:eb931370:::benchmark_data_smithi134_205380_object84635:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-02-05T08:26:26.324 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:25:57.311647+0000", 2024-02-05T08:26:26.324 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 28.896607724999999, 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.074800957000000001, 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 219737 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.311647+0000", 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.325 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.311647+0000", 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.311648+0000", 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3909999999999999e-06 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.311654+0000", 2024-02-05T08:26:26.326 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9859999999999999e-06 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.311670+0000", 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6201e-05 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.311678+0000", 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0830000000000004e-06 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.327 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.311736+0000", 2024-02-05T08:26:26.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.766e-05 2024-02-05T08:26:26.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.311827+0000", 2024-02-05T08:26:26.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.0755999999999996e-05 2024-02-05T08:26:26.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.328 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:26:26.328 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.311908+0000", 2024-02-05T08:26:26.328 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0857000000000002e-05 2024-02-05T08:26:26.328 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.328 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.314390+0000", 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0024823369999999998 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.315814+0000", 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0014245060000000001 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.386360+0000", 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.070545842999999997 2024-02-05T08:26:26.329 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.386417+0000", 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6369000000000002e-05 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:57.386448+0000", 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0967999999999999e-05 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.330 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:246456 11.7 11:e6ba9e20:::benchmark_data_smithi134_205380_object111354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:25.633209+0000", 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.57504544400000002, 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.091973731000000003, 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 246456 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:26.331 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.633209+0000", 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.633209+0000", 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.633210+0000", 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.581e-06 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.332 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.633216+0000", 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2020000000000003e-06 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.633229+0000", 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3026e-05 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.633237+0000", 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.0220000000000004e-06 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.333 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.633343+0000", 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0001059 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.633457+0000", 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011386499999999999 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,4", 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.634716+0000", 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001259488 2024-02-05T08:26:26.334 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.637376+0000", 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002659748 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.642610+0000", 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0052340889999999999 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.725030+0000", 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.082420486000000001 2024-02-05T08:26:26.335 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.725128+0000", 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.7455999999999997e-05 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.725183+0000", 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.4867999999999997e-05 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:26.336 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:26.337 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:26:26.359 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:26.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:26.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:26.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:26.485 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:26.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:26.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:26 smithi134 ceph-mon[30086]: 4.12 scrub starts 2024-02-05T08:26:26.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:26 smithi134 ceph-mon[30086]: 4.12 scrub ok 2024-02-05T08:26:26.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:26 smithi134 ceph-mon[30086]: 11.e scrub starts 2024-02-05T08:26:26.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:26 smithi134 ceph-mon[30086]: osdmap e420: 8 total, 8 up, 8 in 2024-02-05T08:26:26.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:26 smithi134 ceph-mon[30086]: 11.e scrub ok 2024-02-05T08:26:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:26 smithi134 ceph-mon[39278]: 4.12 scrub starts 2024-02-05T08:26:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:26 smithi134 ceph-mon[39278]: 4.12 scrub ok 2024-02-05T08:26:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:26 smithi134 ceph-mon[39278]: 11.e scrub starts 2024-02-05T08:26:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:26 smithi134 ceph-mon[39278]: osdmap e420: 8 total, 8 up, 8 in 2024-02-05T08:26:26.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:26 smithi134 ceph-mon[39278]: 11.e scrub ok 2024-02-05T08:26:26.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:26.611 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:26.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:26.712 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:26.736 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:26.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:26.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:26.837 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:26.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:26.861 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:26.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:26.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:26.987 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:26.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:27.088 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:27.116 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:27.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:27.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:27.226 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T08:26:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:27.249 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:27.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:27.351 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:27.375 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:27.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:27.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:27.477 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:27.501 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:27.501 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:26:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:27.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[30086]: 5.19 scrub starts 2024-02-05T08:26:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[30086]: 5.19 scrub ok 2024-02-05T08:26:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[30086]: 6.a scrub starts 2024-02-05T08:26:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[30086]: 6.a scrub ok 2024-02-05T08:26:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[30086]: pgmap v973: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 4.6 KiB/s rd, 28 KiB/s wr, 939 op/s; 575/79053 objects misplaced (0.727%); 32 MiB/s, 511 objects/s recovering 2024-02-05T08:26:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[30086]: 6.0 scrub starts 2024-02-05T08:26:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[30086]: 6.0 scrub ok 2024-02-05T08:26:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[30086]: 6.6 scrub starts 2024-02-05T08:26:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[30086]: 6.6 scrub ok 2024-02-05T08:26:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[30086]: 3.e scrub starts 2024-02-05T08:26:27.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[30086]: 3.e scrub ok 2024-02-05T08:26:27.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[30086]: osdmap e421: 8 total, 8 up, 8 in 2024-02-05T08:26:27.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[39278]: 5.19 scrub starts 2024-02-05T08:26:27.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[39278]: 5.19 scrub ok 2024-02-05T08:26:27.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[39278]: 6.a scrub starts 2024-02-05T08:26:27.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[39278]: 6.a scrub ok 2024-02-05T08:26:27.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[39278]: pgmap v973: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 4.6 KiB/s rd, 28 KiB/s wr, 939 op/s; 575/79053 objects misplaced (0.727%); 32 MiB/s, 511 objects/s recovering 2024-02-05T08:26:27.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[39278]: 6.0 scrub starts 2024-02-05T08:26:27.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[39278]: 6.0 scrub ok 2024-02-05T08:26:27.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[39278]: 6.6 scrub starts 2024-02-05T08:26:27.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[39278]: 6.6 scrub ok 2024-02-05T08:26:27.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[39278]: 3.e scrub starts 2024-02-05T08:26:27.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[39278]: 3.e scrub ok 2024-02-05T08:26:27.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-mon[39278]: osdmap e421: 8 total, 8 up, 8 in 2024-02-05T08:26:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:27 smithi196 ceph-mon[33018]: 5.19 scrub starts 2024-02-05T08:26:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:27 smithi196 ceph-mon[33018]: 5.19 scrub ok 2024-02-05T08:26:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:27 smithi196 ceph-mon[33018]: 6.a scrub starts 2024-02-05T08:26:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:27 smithi196 ceph-mon[33018]: 6.a scrub ok 2024-02-05T08:26:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:27 smithi196 ceph-mon[33018]: pgmap v973: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 4.6 KiB/s rd, 28 KiB/s wr, 939 op/s; 575/79053 objects misplaced (0.727%); 32 MiB/s, 511 objects/s recovering 2024-02-05T08:26:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:27 smithi196 ceph-mon[33018]: 6.0 scrub starts 2024-02-05T08:26:27.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:27 smithi196 ceph-mon[33018]: 6.0 scrub ok 2024-02-05T08:26:27.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:27 smithi196 ceph-mon[33018]: 6.6 scrub starts 2024-02-05T08:26:27.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:27 smithi196 ceph-mon[33018]: 6.6 scrub ok 2024-02-05T08:26:27.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:27 smithi196 ceph-mon[33018]: 3.e scrub starts 2024-02-05T08:26:27.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:27 smithi196 ceph-mon[33018]: 3.e scrub ok 2024-02-05T08:26:27.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:27 smithi196 ceph-mon[33018]: osdmap e421: 8 total, 8 up, 8 in 2024-02-05T08:26:27.603 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:27.626 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:27.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:27.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:27.728 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:27.754 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:27.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:27.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:27.856 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:27.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:27.880 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:27.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:27.981 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:28.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:28.018 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:28.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:28.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:26:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:26:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:26:28.119 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:28.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:28.153 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:28.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:28.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:28.289 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:28.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:28 smithi196 ceph-mon[33018]: 4.2 scrub starts 2024-02-05T08:26:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:28 smithi196 ceph-mon[33018]: 4.2 scrub ok 2024-02-05T08:26:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:28 smithi196 ceph-mon[33018]: 5.2 scrub starts 2024-02-05T08:26:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:28 smithi196 ceph-mon[33018]: 5.2 scrub ok 2024-02-05T08:26:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:28 smithi196 ceph-mon[33018]: 4.3 scrub starts 2024-02-05T08:26:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:28 smithi196 ceph-mon[33018]: 4.3 scrub ok 2024-02-05T08:26:28.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:28 smithi196 ceph-mon[33018]: 5.18 scrub starts 2024-02-05T08:26:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:28 smithi196 ceph-mon[33018]: 5.18 scrub ok 2024-02-05T08:26:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:28 smithi196 ceph-mon[33018]: 6.1c scrub starts 2024-02-05T08:26:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:28 smithi196 ceph-mon[33018]: 6.1c scrub ok 2024-02-05T08:26:28.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:28 smithi196 ceph-mon[33018]: osdmap e422: 8 total, 8 up, 8 in 2024-02-05T08:26:28.390 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:28.427 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:28.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:28.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:28.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[30086]: 4.2 scrub starts 2024-02-05T08:26:28.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[30086]: 4.2 scrub ok 2024-02-05T08:26:28.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[30086]: 5.2 scrub starts 2024-02-05T08:26:28.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[30086]: 5.2 scrub ok 2024-02-05T08:26:28.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[30086]: 4.3 scrub starts 2024-02-05T08:26:28.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[30086]: 4.3 scrub ok 2024-02-05T08:26:28.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[30086]: 5.18 scrub starts 2024-02-05T08:26:28.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[30086]: 5.18 scrub ok 2024-02-05T08:26:28.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[30086]: 6.1c scrub starts 2024-02-05T08:26:28.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[30086]: 6.1c scrub ok 2024-02-05T08:26:28.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[30086]: osdmap e422: 8 total, 8 up, 8 in 2024-02-05T08:26:28.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[39278]: 4.2 scrub starts 2024-02-05T08:26:28.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[39278]: 4.2 scrub ok 2024-02-05T08:26:28.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[39278]: 5.2 scrub starts 2024-02-05T08:26:28.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[39278]: 5.2 scrub ok 2024-02-05T08:26:28.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[39278]: 4.3 scrub starts 2024-02-05T08:26:28.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[39278]: 4.3 scrub ok 2024-02-05T08:26:28.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[39278]: 5.18 scrub starts 2024-02-05T08:26:28.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[39278]: 5.18 scrub ok 2024-02-05T08:26:28.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[39278]: 6.1c scrub starts 2024-02-05T08:26:28.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[39278]: 6.1c scrub ok 2024-02-05T08:26:28.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:28 smithi134 ceph-mon[39278]: osdmap e422: 8 total, 8 up, 8 in 2024-02-05T08:26:28.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:28.559 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:28.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:28.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:28.661 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:28.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:28.704 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:28.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:28.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:28.846 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:28.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:28.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:28.954 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:28.998 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:29.021 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:29.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:29.103 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[39278]: 6.4 scrub starts 2024-02-05T08:26:29.103 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[39278]: 6.4 scrub ok 2024-02-05T08:26:29.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[39278]: 11.1 scrub starts 2024-02-05T08:26:29.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[39278]: 5.15 scrub starts 2024-02-05T08:26:29.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[39278]: 5.15 scrub ok 2024-02-05T08:26:29.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[39278]: 11.1 scrub ok 2024-02-05T08:26:29.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[39278]: pgmap v976: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 575/77163 objects misplaced (0.745%); 12 MiB/s, 189 objects/s recovering 2024-02-05T08:26:29.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[39278]: 1.1 scrub starts 2024-02-05T08:26:29.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[39278]: 1.1 scrub ok 2024-02-05T08:26:29.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[39278]: 1.8 scrub starts 2024-02-05T08:26:29.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[39278]: 1.8 scrub ok 2024-02-05T08:26:29.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[39278]: 4.7 scrub starts 2024-02-05T08:26:29.104 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[39278]: 4.7 scrub ok 2024-02-05T08:26:29.122 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:29.156 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:29.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:29.260 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:29.294 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:29.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:29.378 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:29 smithi196 ceph-mon[33018]: 6.4 scrub starts 2024-02-05T08:26:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:29 smithi196 ceph-mon[33018]: 6.4 scrub ok 2024-02-05T08:26:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:29 smithi196 ceph-mon[33018]: 11.1 scrub starts 2024-02-05T08:26:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:29 smithi196 ceph-mon[33018]: 5.15 scrub starts 2024-02-05T08:26:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:29 smithi196 ceph-mon[33018]: 5.15 scrub ok 2024-02-05T08:26:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:29 smithi196 ceph-mon[33018]: 11.1 scrub ok 2024-02-05T08:26:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:29 smithi196 ceph-mon[33018]: pgmap v976: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 575/77163 objects misplaced (0.745%); 12 MiB/s, 189 objects/s recovering 2024-02-05T08:26:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:29 smithi196 ceph-mon[33018]: 1.1 scrub starts 2024-02-05T08:26:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:29 smithi196 ceph-mon[33018]: 1.1 scrub ok 2024-02-05T08:26:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:29 smithi196 ceph-mon[33018]: 1.8 scrub starts 2024-02-05T08:26:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:29 smithi196 ceph-mon[33018]: 1.8 scrub ok 2024-02-05T08:26:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:29 smithi196 ceph-mon[33018]: 4.7 scrub starts 2024-02-05T08:26:29.379 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:29 smithi196 ceph-mon[33018]: 4.7 scrub ok 2024-02-05T08:26:29.396 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:29.399 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[30086]: 6.4 scrub starts 2024-02-05T08:26:29.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[30086]: 6.4 scrub ok 2024-02-05T08:26:29.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[30086]: 11.1 scrub starts 2024-02-05T08:26:29.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[30086]: 5.15 scrub starts 2024-02-05T08:26:29.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[30086]: 5.15 scrub ok 2024-02-05T08:26:29.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[30086]: 11.1 scrub ok 2024-02-05T08:26:29.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[30086]: pgmap v976: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 841 op/s; 575/77163 objects misplaced (0.745%); 12 MiB/s, 189 objects/s recovering 2024-02-05T08:26:29.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[30086]: 1.1 scrub starts 2024-02-05T08:26:29.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[30086]: 1.1 scrub ok 2024-02-05T08:26:29.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[30086]: 1.8 scrub starts 2024-02-05T08:26:29.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[30086]: 1.8 scrub ok 2024-02-05T08:26:29.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[30086]: 4.7 scrub starts 2024-02-05T08:26:29.400 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:29 smithi134 ceph-mon[30086]: 4.7 scrub ok 2024-02-05T08:26:29.426 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:29.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:29.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:29.528 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:29.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:29.566 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:29.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:29.667 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:29.716 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:29.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:29.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:29.818 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:29.862 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:29.863 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:26:29.863 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:26:29.863 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.863 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:249150 11.1 11:82454292:::benchmark_data_smithi134_205380_object114048:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-02-05T08:26:29.863 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:29.844245+0000", 2024-02-05T08:26:29.864 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.018005134999999999, 2024-02-05T08:26:29.864 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.018027228999999999, 2024-02-05T08:26:29.864 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:29.864 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:26:29.864 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:29.864 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:29.864 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:29.864 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 249150 2024-02-05T08:26:29.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.864 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:29.864 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.864 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:29.864 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.844245+0000", 2024-02-05T08:26:29.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:29.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:29.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.844245+0000", 2024-02-05T08:26:29.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:29.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.865 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:29.865 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.844246+0000", 2024-02-05T08:26:29.865 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0070000000000001e-06 2024-02-05T08:26:29.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.865 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.844251+0000", 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.1019999999999996e-06 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.844267+0000", 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5150999999999999e-05 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.844276+0000", 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1430000000000004e-06 2024-02-05T08:26:29.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.844876+0000", 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00060060699999999996 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.844956+0000", 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.9553000000000002e-05 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.845028+0000", 2024-02-05T08:26:29.867 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2523999999999999e-05 2024-02-05T08:26:29.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:29.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.850322+0000", 2024-02-05T08:26:29.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.005293234 2024-02-05T08:26:29.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.868 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.868 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:29.868 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.860355+0000", 2024-02-05T08:26:29.868 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.010033168 2024-02-05T08:26:29.868 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:29.868 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:29.868 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:29.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.869 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.869 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:249151 11.1 11:87080966:::benchmark_data_smithi134_205380_object114049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-02-05T08:26:29.869 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:29.845370+0000", 2024-02-05T08:26:29.869 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.016880216999999999, 2024-02-05T08:26:29.869 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.016964763000000001, 2024-02-05T08:26:29.869 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:29.869 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:26:29.869 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:29.869 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:29.869 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:29.869 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 249151 2024-02-05T08:26:29.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.870 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:29.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:29.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.845370+0000", 2024-02-05T08:26:29.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:29.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.870 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:29.870 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.845370+0000", 2024-02-05T08:26:29.870 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:29.870 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.870 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:29.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.845371+0000", 2024-02-05T08:26:29.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.7199999999999997e-07 2024-02-05T08:26:29.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:29.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.845375+0000", 2024-02-05T08:26:29.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.245e-06 2024-02-05T08:26:29.871 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.871 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.871 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:29.871 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.845389+0000", 2024-02-05T08:26:29.871 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4073000000000001e-05 2024-02-05T08:26:29.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:29.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.845399+0000", 2024-02-05T08:26:29.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.012e-05 2024-02-05T08:26:29.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:29.872 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.846718+0000", 2024-02-05T08:26:29.872 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001318133 2024-02-05T08:26:29.872 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.872 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.872 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:29.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.846849+0000", 2024-02-05T08:26:29.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013096399999999999 2024-02-05T08:26:29.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,3", 2024-02-05T08:26:29.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.846982+0000", 2024-02-05T08:26:29.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013351899999999999 2024-02-05T08:26:29.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.873 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.873 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:29.873 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.852644+0000", 2024-02-05T08:26:29.873 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0056622779999999998 2024-02-05T08:26:29.873 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:249159 11.3 11:cde746ce:::benchmark_data_smithi134_205380_object114057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-02-05T08:26:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:29.849365+0000", 2024-02-05T08:26:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 0.012885765, 2024-02-05T08:26:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.013072208, 2024-02-05T08:26:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "waiting for sub ops", 2024-02-05T08:26:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:29.874 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 249159 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.849365+0000", 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.849365+0000", 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.875 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:29.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.849366+0000", 2024-02-05T08:26:29.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0699999999999999e-06 2024-02-05T08:26:29.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:29.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.849373+0000", 2024-02-05T08:26:29.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5179999999999998e-06 2024-02-05T08:26:29.876 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.876 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.876 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:29.876 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.849389+0000", 2024-02-05T08:26:29.876 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6201e-05 2024-02-05T08:26:29.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:29.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.849398+0000", 2024-02-05T08:26:29.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9919999999999996e-06 2024-02-05T08:26:29.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:29.877 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.849457+0000", 2024-02-05T08:26:29.877 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8136e-05 2024-02-05T08:26:29.877 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.877 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.877 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.849569+0000", 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011257800000000001 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,7", 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.849678+0000", 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00010929699999999999 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.852860+0000", 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0031818749999999998 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:29.878 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:29.879 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:29.854215+0000", 2024-02-05T08:26:29.879 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0013549930000000001 2024-02-05T08:26:29.879 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:29.879 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:29.879 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:29.879 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:29.879 INFO:teuthology.orchestra.run.smithi196.stdout: ], 2024-02-05T08:26:29.879 INFO:teuthology.orchestra.run.smithi196.stdout: "num_ops": 3 2024-02-05T08:26:29.879 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:26:29.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:29.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:29.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:30.010 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:30.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:30.112 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:30.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:30.164 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:30.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:30.265 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:30.298 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:30.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:30.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:30.365 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[39278]: 4.1c scrub starts 2024-02-05T08:26:30.365 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[39278]: 4.1c scrub ok 2024-02-05T08:26:30.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[39278]: 4.8 scrub starts 2024-02-05T08:26:30.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[39278]: 4.8 scrub ok 2024-02-05T08:26:30.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[39278]: 11.8 scrub starts 2024-02-05T08:26:30.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[39278]: 11.6 scrub starts 2024-02-05T08:26:30.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[39278]: 11.8 scrub ok 2024-02-05T08:26:30.367 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[30086]: 4.1c scrub starts 2024-02-05T08:26:30.367 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[30086]: 4.1c scrub ok 2024-02-05T08:26:30.367 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[30086]: 4.8 scrub starts 2024-02-05T08:26:30.367 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[30086]: 4.8 scrub ok 2024-02-05T08:26:30.368 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[30086]: 11.8 scrub starts 2024-02-05T08:26:30.368 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[30086]: 11.6 scrub starts 2024-02-05T08:26:30.368 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:30 smithi134 ceph-mon[30086]: 11.8 scrub ok 2024-02-05T08:26:30.375 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:26:30.375 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:26:30.375 INFO:teuthology.orchestra.run.smithi134.stdout: "epoch": 422, 2024-02-05T08:26:30.375 INFO:teuthology.orchestra.run.smithi134.stdout: "fsid": "43e13aa4-c3fc-11ee-95b6-87774f69a715", 2024-02-05T08:26:30.375 INFO:teuthology.orchestra.run.smithi134.stdout: "created": "2024-02-05T07:59:09.321262+0000", 2024-02-05T08:26:30.375 INFO:teuthology.orchestra.run.smithi134.stdout: "modified": "2024-02-05T08:26:27.944534+0000", 2024-02-05T08:26:30.375 INFO:teuthology.orchestra.run.smithi134.stdout: "last_up_change": "2024-02-05T08:23:06.892657+0000", 2024-02-05T08:26:30.375 INFO:teuthology.orchestra.run.smithi134.stdout: "last_in_change": "2024-02-05T08:25:48.086167+0000", 2024-02-05T08:26:30.375 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T08:26:30.376 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_num": 5804032, 2024-02-05T08:26:30.376 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_set": [ 2024-02-05T08:26:30.376 INFO:teuthology.orchestra.run.smithi134.stdout: "nodeep-scrub", 2024-02-05T08:26:30.376 INFO:teuthology.orchestra.run.smithi134.stdout: "pglog_hardlimit", 2024-02-05T08:26:30.376 INFO:teuthology.orchestra.run.smithi134.stdout: "purged_snapdirs", 2024-02-05T08:26:30.376 INFO:teuthology.orchestra.run.smithi134.stdout: "recovery_deletes", 2024-02-05T08:26:30.376 INFO:teuthology.orchestra.run.smithi134.stdout: "sortbitwise" 2024-02-05T08:26:30.376 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:26:30.376 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_version": 19, 2024-02-05T08:26:30.376 INFO:teuthology.orchestra.run.smithi134.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T08:26:30.377 INFO:teuthology.orchestra.run.smithi134.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T08:26:30.377 INFO:teuthology.orchestra.run.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T08:26:30.377 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_snapshot": "", 2024-02-05T08:26:30.377 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_max": 11, 2024-02-05T08:26:30.377 INFO:teuthology.orchestra.run.smithi134.stdout: "max_osd": 8, 2024-02-05T08:26:30.377 INFO:teuthology.orchestra.run.smithi134.stdout: "require_min_compat_client": "luminous", 2024-02-05T08:26:30.377 INFO:teuthology.orchestra.run.smithi134.stdout: "min_compat_client": "luminous", 2024-02-05T08:26:30.377 INFO:teuthology.orchestra.run.smithi134.stdout: "require_osd_release": "squid", 2024-02-05T08:26:30.377 INFO:teuthology.orchestra.run.smithi134.stdout: "allow_crimson": false, 2024-02-05T08:26:30.377 INFO:teuthology.orchestra.run.smithi134.stdout: "pools": [ 2024-02-05T08:26:30.377 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.377 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 1, 2024-02-05T08:26:30.377 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": ".mgr", 2024-02-05T08:26:30.378 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:02:21.034012+0000", 2024-02-05T08:26:30.378 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:26:30.378 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:26:30.378 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:26:30.378 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:26:30.378 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:26:30.378 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:26:30.378 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:26:30.378 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:26:30.378 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:26:30.378 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:26:30.378 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:26:30.378 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 11, 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 11, 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 11, 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 11, 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 11, 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.379 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "303", 2024-02-05T08:26:30.380 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:26:30.380 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:26:30.380 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "301", 2024-02-05T08:26:30.380 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:26:30.380 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:26:30.380 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:26:30.380 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:26:30.380 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:26:30.380 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:26:30.380 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:26:30.381 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:26:30.381 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:26:30.381 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:26:30.381 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:26:30.381 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:26:30.381 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:26:30.381 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:26:30.381 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:26:30.381 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:26:30.381 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:26:30.381 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:26:30.381 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:26:30.382 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: "options": { 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_max": 32, 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_min": 1 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: "mgr": {} 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.1800000667572021, 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.1800000667572021, 2024-02-05T08:26:30.383 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 1, 2024-02-05T08:26:30.384 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 2.1800000667572021, 2024-02-05T08:26:30.384 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 2.1800000667572021, 2024-02-05T08:26:30.384 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 0.87969207763671875, 2024-02-05T08:26:30.384 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T08:26:30.384 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T08:26:30.384 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.384 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.384 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.384 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 2, 2024-02-05T08:26:30.385 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "datapool", 2024-02-05T08:26:30.385 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:39.400746+0000", 2024-02-05T08:26:30.385 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 8193, 2024-02-05T08:26:30.385 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T08:26:30.385 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:26:30.385 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:26:30.385 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:26:30.385 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:26:30.385 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:26:30.385 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:26:30.385 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:26:30.385 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 3, 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 3, 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 3, 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 3, 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 3, 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:26:30.386 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.387 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "70", 2024-02-05T08:26:30.387 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:26:30.387 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:26:30.387 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:26:30.387 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:26:30.387 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:26:30.387 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 2, 2024-02-05T08:26:30.387 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 70, 2024-02-05T08:26:30.387 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:26:30.387 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:26:30.387 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:26:30.387 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:26:30.387 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:26:30.388 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:26:30.388 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:26:30.388 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:26:30.388 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:26:30.388 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:26:30.388 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:26:30.388 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:26:30.388 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:26:30.388 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:26:30.388 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:26:30.388 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:26:30.388 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:26:30.388 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:26:30.389 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: "rbd": {} 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.6500000953674316, 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.6500000953674316, 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 2.3299999237060547, 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 2.3299999237060547, 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 0.87969207763671875, 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.390 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.391 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.391 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 3, 2024-02-05T08:26:30.391 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": ".rgw.root", 2024-02-05T08:26:30.391 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:40.005769+0000", 2024-02-05T08:26:30.391 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:26:30.391 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:26:30.391 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:26:30.391 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:26:30.391 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:26:30.391 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:26:30.391 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:26:30.391 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:26:30.391 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:26:30.392 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:26:30.392 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:26:30.392 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:26:30.392 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:26:30.392 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:26:30.392 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:26:30.392 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:26:30.392 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:26:30.392 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:26:30.392 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:26:30.392 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:26:30.392 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:26:30.392 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "68", 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:26:30.393 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:26:30.394 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:26:30.394 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:26:30.394 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:26:30.394 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:26:30.394 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:26:30.394 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:26:30.394 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:26:30.394 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:26:30.394 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:26:30.394 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:26:30.394 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:26:30.394 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:26:30.395 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:26:30.395 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:26:30.395 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:26:30.395 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.395 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:26:30.395 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:26:30.395 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:26:30.395 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:26:30.396 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2, 2024-02-05T08:26:30.397 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2, 2024-02-05T08:26:30.397 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 1, 2024-02-05T08:26:30.397 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 2, 2024-02-05T08:26:30.397 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 2, 2024-02-05T08:26:30.397 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 0.87969207763671875, 2024-02-05T08:26:30.397 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T08:26:30.397 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T08:26:30.397 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.397 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.397 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.397 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 4, 2024-02-05T08:26:30.397 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.log", 2024-02-05T08:26:30.397 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:41.851891+0000", 2024-02-05T08:26:30.398 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:26:30.398 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:26:30.398 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:26:30.398 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:26:30.398 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:26:30.398 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:26:30.398 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:26:30.398 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:26:30.398 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:26:30.398 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:26:30.398 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:26:30.398 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:26:30.398 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "70", 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:26:30.399 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:26:30.400 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:26:30.400 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:26:30.400 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:26:30.400 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:26:30.400 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:26:30.400 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:26:30.400 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:26:30.400 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:26:30.400 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:26:30.400 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:26:30.400 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.401 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:26:30.402 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:26:30.402 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:26:30.402 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:26:30.402 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:26:30.402 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:26:30.402 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:26:30.402 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:26:30.402 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:26:30.402 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:26:30.402 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:26:30.402 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:26:30.402 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.25, 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.25, 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 1, 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 2.25, 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 2.25, 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 0.87969207763671875, 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.403 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 5, 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.control", 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:43.793804+0000", 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:26:30.404 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:26:30.405 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:26:30.405 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:26:30.405 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:26:30.405 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:26:30.405 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:26:30.405 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:26:30.405 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:26:30.405 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:26:30.405 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:26:30.405 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:26:30.405 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:26:30.405 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:26:30.405 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:26:30.406 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.406 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "72", 2024-02-05T08:26:30.406 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:26:30.406 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:26:30.406 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:26:30.406 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:26:30.406 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:26:30.406 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:26:30.406 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:26:30.406 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:26:30.406 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:26:30.407 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:26:30.408 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:26:30.408 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:26:30.408 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.408 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:26:30.408 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:26:30.408 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:26:30.408 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:26:30.408 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:26:30.408 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:26:30.408 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:26:30.408 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:26:30.408 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:26:30.408 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:26:30.409 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:26:30.409 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:26:30.409 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:26:30.409 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:26:30.409 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.409 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:26:30.410 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:26:30.410 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.5, 2024-02-05T08:26:30.410 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.5, 2024-02-05T08:26:30.410 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 1, 2024-02-05T08:26:30.410 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.5, 2024-02-05T08:26:30.410 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.5, 2024-02-05T08:26:30.410 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 0.87969207763671875, 2024-02-05T08:26:30.410 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T08:26:30.411 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:30.412 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T08:26:30.412 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.412 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.412 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.412 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 6, 2024-02-05T08:26:30.412 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "default.rgw.meta", 2024-02-05T08:26:30.412 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:04:45.853999+0000", 2024-02-05T08:26:30.412 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:26:30.412 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:26:30.412 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 32, 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 32, 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 32, 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 32, 2024-02-05T08:26:30.413 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 32, 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "74", 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:26:30.414 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:26:30.415 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:26:30.415 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:26:30.415 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:26:30.415 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:26:30.415 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:26:30.415 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:26:30.415 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:26:30.415 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:26:30.415 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:26:30.415 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:26:30.415 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:26:30.415 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:26:30.416 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:26:30.416 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:26:30.416 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:26:30.416 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:26:30.416 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:26:30.416 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:26:30.416 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:26:30.416 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:26:30.416 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.417 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:26:30.417 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:26:30.417 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:26:30.417 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:26:30.417 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:26:30.417 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:26:30.417 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: "options": { 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_bias": 4 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: "rgw": {} 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 1.75, 2024-02-05T08:26:30.418 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 1.75, 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 1, 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 1.75, 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 1.75, 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 0.87969207763671875, 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: "pool": 11, 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_name": "unique_pool_4", 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: "create_time": "2024-02-05T08:23:07.682500+0000", 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: "flags": 1, 2024-02-05T08:26:30.419 INFO:teuthology.orchestra.run.smithi134.stdout: "flags_names": "hashpspool", 2024-02-05T08:26:30.420 INFO:teuthology.orchestra.run.smithi134.stdout: "type": 1, 2024-02-05T08:26:30.420 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 3, 2024-02-05T08:26:30.420 INFO:teuthology.orchestra.run.smithi134.stdout: "min_size": 2, 2024-02-05T08:26:30.420 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_rule": 0, 2024-02-05T08:26:30.420 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-02-05T08:26:30.420 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-02-05T08:26:30.420 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T08:26:30.420 INFO:teuthology.orchestra.run.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T08:26:30.420 INFO:teuthology.orchestra.run.smithi134.stdout: "object_hash": 2, 2024-02-05T08:26:30.420 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_autoscale_mode": "off", 2024-02-05T08:26:30.420 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num": 16, 2024-02-05T08:26:30.420 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num": 16, 2024-02-05T08:26:30.420 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_placement_num_target": 16, 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_target": 16, 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_num_pending": 16, 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: "last_pg_merge_meta": { 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: "source_pgid": "0.0", 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: "ready_epoch": 0, 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_started": 0, 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: "last_epoch_clean": 0, 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: "source_version": "0'0", 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: "target_version": "0'0" 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: "last_change": "396", 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend": "0", 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T08:26:30.421 INFO:teuthology.orchestra.run.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "auid": 0, 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_mode": "selfmanaged", 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_seq": 0, 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "snap_epoch": 0, 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "pool_snaps": [], 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps": "[]", 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_bytes": 0, 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "quota_max_objects": 0, 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "tiers": [], 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "tier_of": -1, 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "read_tier": -1, 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "write_tier": -1, 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_mode": "none", 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_bytes": 0, 2024-02-05T08:26:30.422 INFO:teuthology.orchestra.run.smithi134.stdout: "target_max_objects": 0, 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_flush_age": 0, 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "cache_min_evict_age": 0, 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profile": "", 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_params": { 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "none" 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_period": 0, 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_count": 0, 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "use_gmt_hitset": true, 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-02-05T08:26:30.423 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: "hit_set_search_last_n": 0, 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: "grade_table": [], 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: "stripe_width": 0, 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: "expected_num_objects": 0, 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: "fast_read": false, 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: "options": {}, 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: "application_metadata": { 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: "rados": {} 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: "read_balance": { 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: "score_type": "Fair distribution", 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: "score_acting": 2.5, 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: "score_stable": 2.5, 2024-02-05T08:26:30.424 INFO:teuthology.orchestra.run.smithi134.stdout: "optimal_score": 1, 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_acting": 2.5, 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: "raw_score_stable": 2.5, 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity_weighted": 0.87969207763671875, 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: "osds": [ 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 0, 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "d90a5ea0-9a19-452e-9564-ca7b7a7c45e4", 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:26:30.425 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 8, 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 417, 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6802", 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.426 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6803", 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6804", 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.427 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6805", 2024-02-05T08:26:30.428 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:26:30.428 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.428 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.428 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.428 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:26:30.428 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.428 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.428 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.428 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6808", 2024-02-05T08:26:30.428 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:26:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6809", 2024-02-05T08:26:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:26:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:26:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.429 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6806", 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6807", 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 2540271081 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6803/2540271081", 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6805/2540271081", 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6809/2540271081", 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6807/2540271081", 2024-02-05T08:26:30.430 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 1, 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4", 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 13, 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 323, 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 366, 2024-02-05T08:26:30.431 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 416, 2024-02-05T08:26:30.432 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 324, 2024-02-05T08:26:30.432 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:26:30.432 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:26:30.432 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.432 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.432 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6810", 2024-02-05T08:26:30.432 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:26:30.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.432 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.432 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6811", 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6812", 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.433 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6813", 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6816", 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.434 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6817", 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6814", 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6815", 2024-02-05T08:26:30.435 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1259179009 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6811/1259179009", 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6813/1259179009", 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6817/1259179009", 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6815/1259179009", 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.436 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 2, 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "078e578b-855b-49ff-83bd-0548a9c5a708", 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 0.4826202392578125, 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 18, 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 416, 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.438 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.438 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6818", 2024-02-05T08:26:30.438 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:26:30.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.438 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.438 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6819", 2024-02-05T08:26:30.438 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:26:30.438 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.438 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.439 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:26:30.439 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.439 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.439 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6820", 2024-02-05T08:26:30.439 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:26:30.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.439 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.439 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6821", 2024-02-05T08:26:30.439 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:26:30.439 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6824", 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6825", 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.440 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6822", 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6823", 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 153827987 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.441 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6819/153827987", 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6821/153827987", 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6825/153827987", 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6823/153827987", 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 3, 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "6774e330-3202-4671-894e-172cc2f6189d", 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:26:30.442 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 27, 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 421, 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6826", 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6827", 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6828", 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6829", 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6832", 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6833", 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6830", 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.446 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.447 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.134:6831", 2024-02-05T08:26:30.447 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1527507056 2024-02-05T08:26:30.447 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.447 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.447 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.134:6827/1527507056", 2024-02-05T08:26:30.447 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.134:6829/1527507056", 2024-02-05T08:26:30.447 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.134:6833/1527507056", 2024-02-05T08:26:30.449 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.134:6831/1527507056", 2024-02-05T08:26:30.449 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:26:30.449 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 4, 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "52f70cfa-ab0e-4751-a5f8-1045c61c1693", 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 32, 2024-02-05T08:26:30.450 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 416, 2024-02-05T08:26:30.451 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:26:30.451 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:26:30.451 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:26:30.451 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.451 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.451 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6800", 2024-02-05T08:26:30.452 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:26:30.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.452 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.452 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.452 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6801", 2024-02-05T08:26:30.452 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:26:30.453 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.453 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.453 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.453 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:26:30.453 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.453 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.454 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.454 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6802", 2024-02-05T08:26:30.454 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:26:30.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.454 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.454 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.454 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6803", 2024-02-05T08:26:30.454 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:26:30.454 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.454 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.455 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:26:30.455 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.455 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.455 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6806", 2024-02-05T08:26:30.455 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:26:30.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.456 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.456 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6807", 2024-02-05T08:26:30.456 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:26:30.456 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.456 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.456 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:26:30.456 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.456 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.457 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6804", 2024-02-05T08:26:30.457 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:26:30.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.457 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.457 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6805", 2024-02-05T08:26:30.457 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3005156808 2024-02-05T08:26:30.457 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.457 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6801/3005156808", 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6803/3005156808", 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6807/3005156808", 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6805/3005156808", 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 5, 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "b4b21769-d40f-43fd-b7e9-2477b74d3955", 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:26:30.458 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 0.5549163818359375, 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 42, 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 409, 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6808", 2024-02-05T08:26:30.459 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:26:30.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.460 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.460 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6809", 2024-02-05T08:26:30.460 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:26:30.460 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.460 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.460 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:26:30.460 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.460 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.460 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6810", 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6811", 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6814", 2024-02-05T08:26:30.461 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6815", 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6812", 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:26:30.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6813", 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 1603224792 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6809/1603224792", 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6811/1603224792", 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6815/1603224792", 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6813/1603224792", 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:26:30.463 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:26:30.464 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.464 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 6, 2024-02-05T08:26:30.464 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "af1cced4-0015-47f5-b105-27641bfe4f4a", 2024-02-05T08:26:30.464 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:26:30.464 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:26:30.464 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:26:30.464 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:26:30.464 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:26:30.464 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 51, 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 420, 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6816", 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.465 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6817", 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6818", 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6819", 2024-02-05T08:26:30.466 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:26:30.467 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.467 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.467 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:26:30.467 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.467 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.467 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.467 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6822", 2024-02-05T08:26:30.467 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:26:30.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.467 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.467 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.468 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6823", 2024-02-05T08:26:30.468 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:26:30.468 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.468 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.468 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.468 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:26:30.468 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.468 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.468 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.468 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6820", 2024-02-05T08:26:30.468 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:26:30.468 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.468 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6821", 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 3727413143 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6817/3727413143", 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6819/3727413143", 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6823/3727413143", 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6821/3727413143", 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:26:30.469 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 7, 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: "uuid": "c54beeae-9f7c-4723-a369-13bb531b4057", 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: "up": 1, 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: "in": 1, 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: "weight": 1, 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_affinity": 1, 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_begin": 0, 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: "last_clean_end": 0, 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: "up_from": 60, 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: "up_thru": 416, 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: "down_at": 0, 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: "lost_at": 0, 2024-02-05T08:26:30.470 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addrs": { 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6824", 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6825", 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addrs": { 2024-02-05T08:26:30.471 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.472 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.472 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6826", 2024-02-05T08:26:30.472 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:26:30.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.472 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.472 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6827", 2024-02-05T08:26:30.472 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:26:30.472 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.472 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.472 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addrs": { 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6830", 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6831", 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addrs": { 2024-02-05T08:26:30.473 INFO:teuthology.orchestra.run.smithi134.stdout: "addrvec": [ 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v2", 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6828", 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: "type": "v1", 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: "addr": "172.21.15.196:6829", 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: "nonce": 4206571010 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: "public_addr": "172.21.15.196:6825/4206571010", 2024-02-05T08:26:30.474 INFO:teuthology.orchestra.run.smithi134.stdout: "cluster_addr": "172.21.15.196:6827/4206571010", 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_back_addr": "172.21.15.196:6831/4206571010", 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: "heartbeat_front_addr": "172.21.15.196:6829/4206571010", 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: "state": [ 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: "exists", 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: "up" 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: "osd_xinfo": [ 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 0, 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:26:30.475 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:26:30.476 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:26:30.476 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:26:30.476 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:25:59.840728+0000", 2024-02-05T08:26:30.476 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:26:30.476 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.476 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.476 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 1, 2024-02-05T08:26:30.476 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:26:30.476 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:26:30.476 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:26:30.476 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:26:30.476 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:26:30.476 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:25:39.306744+0000", 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 323 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 2, 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:26:05.494116+0000", 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.477 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 3, 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:25:57.888022+0000", 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 4, 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:26:30.478 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:25:56.383775+0000", 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 5, 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:25:38.896269+0000", 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.479 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 6, 2024-02-05T08:26:30.480 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:26:30.480 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:26:30.480 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:26:30.480 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:26:30.480 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:26:30.480 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:25:57.864927+0000", 2024-02-05T08:26:30.480 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:26:30.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.480 INFO:teuthology.orchestra.run.smithi134.stdout: "osd": 7, 2024-02-05T08:26:30.480 INFO:teuthology.orchestra.run.smithi134.stdout: "down_stamp": "0.000000", 2024-02-05T08:26:30.480 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_probability": 0, 2024-02-05T08:26:30.480 INFO:teuthology.orchestra.run.smithi134.stdout: "laggy_interval": 0, 2024-02-05T08:26:30.481 INFO:teuthology.orchestra.run.smithi134.stdout: "features": 4540701272860131327, 2024-02-05T08:26:30.481 INFO:teuthology.orchestra.run.smithi134.stdout: "old_weight": 0, 2024-02-05T08:26:30.481 INFO:teuthology.orchestra.run.smithi134.stdout: "last_purged_snaps_scrub": "2024-02-05T08:25:33.765646+0000", 2024-02-05T08:26:30.481 INFO:teuthology.orchestra.run.smithi134.stdout: "dead_epoch": 0 2024-02-05T08:26:30.481 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.481 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:26:30.481 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap": [ 2024-02-05T08:26:30.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.481 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "3.0", 2024-02-05T08:26:30.481 INFO:teuthology.orchestra.run.smithi134.stdout: "osds": [ 2024-02-05T08:26:30.481 INFO:teuthology.orchestra.run.smithi134.stdout: 3, 2024-02-05T08:26:30.481 INFO:teuthology.orchestra.run.smithi134.stdout: 1, 2024-02-05T08:26:30.481 INFO:teuthology.orchestra.run.smithi134.stdout: 5 2024-02-05T08:26:30.482 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.482 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "6.6", 2024-02-05T08:26:30.482 INFO:teuthology.orchestra.run.smithi134.stdout: "osds": [ 2024-02-05T08:26:30.482 INFO:teuthology.orchestra.run.smithi134.stdout: 4, 2024-02-05T08:26:30.482 INFO:teuthology.orchestra.run.smithi134.stdout: 1, 2024-02-05T08:26:30.482 INFO:teuthology.orchestra.run.smithi134.stdout: 3 2024-02-05T08:26:30.482 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.482 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.482 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:26:30.482 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap_items": [ 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "3.6", 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 4, 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.6", 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:26:30.483 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 5 2024-02-05T08:26:30.484 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.484 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.484 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.7", 2024-02-05T08:26:30.484 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:26:30.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.484 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:26:30.484 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:26:30.484 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.484 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.485 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.a", 2024-02-05T08:26:30.485 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:26:30.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.485 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:26:30.485 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.12", 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 5 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: "pgid": "4.1f", 2024-02-05T08:26:30.486 INFO:teuthology.orchestra.run.smithi134.stdout: "mappings": [ 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: "from": 1, 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: "to": 2 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: ], 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_upmap_primaries": [], 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: "pg_temp": [], 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: "primary_temp": [], 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: "blocklist": { 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/788812699": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/1046500326": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:26:30.487 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/464208089": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:26:30.488 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/301941854": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:26:30.488 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/3607168099": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:26:30.488 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/673876301": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:26:30.488 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/4225724311": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:26:30.488 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6800/1525062017": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:26:30.488 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/3607168099": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:26:30.488 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/673876301": "2024-02-06T07:59:36.275039+0000", 2024-02-05T08:26:30.488 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3200884805": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:26:30.488 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/2952444615": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:26:30.488 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3325835814": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:26:30.488 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/3161126612": "2024-02-06T08:00:02.709280+0000", 2024-02-05T08:26:30.488 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/69097091": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:6801/1525062017": "2024-02-06T08:05:21.465844+0000", 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: "172.21.15.134:0/1645475987": "2024-02-06T08:05:21.465844+0000" 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: "range_blocklist": {}, 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: "erasure_code_profiles": { 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: "default": { 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: "crush-failure-domain": "osd", 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: "k": "2", 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: "m": "1", 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: "plugin": "jerasure", 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: "technique": "reed_sol_van" 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:26:30.489 INFO:teuthology.orchestra.run.smithi134.stdout: "removed_snaps_queue": [], 2024-02-05T08:26:30.490 INFO:teuthology.orchestra.run.smithi134.stdout: "new_removed_snaps": [], 2024-02-05T08:26:30.490 INFO:teuthology.orchestra.run.smithi134.stdout: "new_purged_snaps": [], 2024-02-05T08:26:30.490 INFO:teuthology.orchestra.run.smithi134.stdout: "crush_node_flags": {}, 2024-02-05T08:26:30.490 INFO:teuthology.orchestra.run.smithi134.stdout: "device_class_flags": {}, 2024-02-05T08:26:30.490 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode": { 2024-02-05T08:26:30.490 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode_enabled": false, 2024-02-05T08:26:30.490 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_bucket_count": 0, 2024-02-05T08:26:30.490 INFO:teuthology.orchestra.run.smithi134.stdout: "degraded_stretch_mode": 0, 2024-02-05T08:26:30.490 INFO:teuthology.orchestra.run.smithi134.stdout: "recovering_stretch_mode": 0, 2024-02-05T08:26:30.490 INFO:teuthology.orchestra.run.smithi134.stdout: "stretch_mode_bucket": 0 2024-02-05T08:26:30.490 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:26:30.490 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:26:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:30.491 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:30.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:30.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:30 smithi196 ceph-mon[33018]: 4.1c scrub starts 2024-02-05T08:26:30.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:30 smithi196 ceph-mon[33018]: 4.1c scrub ok 2024-02-05T08:26:30.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:30 smithi196 ceph-mon[33018]: 4.8 scrub starts 2024-02-05T08:26:30.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:30 smithi196 ceph-mon[33018]: 4.8 scrub ok 2024-02-05T08:26:30.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:30 smithi196 ceph-mon[33018]: 11.8 scrub starts 2024-02-05T08:26:30.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:30 smithi196 ceph-mon[33018]: 11.6 scrub starts 2024-02-05T08:26:30.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:30 smithi196 ceph-mon[33018]: 11.8 scrub ok 2024-02-05T08:26:30.593 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:30.629 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:30.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:30.731 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:30.740 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T08:26:30.770 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:30.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:30.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:30.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:30.921 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:30.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:31.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:31.064 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:31.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:31.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:31.123 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:26:31.166 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:31.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:31.204 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:31.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:31.306 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:31.349 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:31.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:31.452 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:31.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[30086]: 6.1 scrub starts 2024-02-05T08:26:31.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[30086]: 6.1 scrub ok 2024-02-05T08:26:31.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[30086]: 4.1f scrub starts 2024-02-05T08:26:31.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[30086]: 4.1f scrub ok 2024-02-05T08:26:31.457 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[30086]: 11.6 scrub ok 2024-02-05T08:26:31.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[30086]: pgmap v977: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 5.7 MiB/s, 91 objects/s recovering 2024-02-05T08:26:31.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[30086]: 5.1e scrub starts 2024-02-05T08:26:31.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[30086]: 5.1e scrub ok 2024-02-05T08:26:31.458 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4282611691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:26:31.458 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[39278]: 6.1 scrub starts 2024-02-05T08:26:31.459 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[39278]: 6.1 scrub ok 2024-02-05T08:26:31.459 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[39278]: 4.1f scrub starts 2024-02-05T08:26:31.459 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[39278]: 4.1f scrub ok 2024-02-05T08:26:31.459 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[39278]: 11.6 scrub ok 2024-02-05T08:26:31.459 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[39278]: pgmap v977: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 5.7 MiB/s, 91 objects/s recovering 2024-02-05T08:26:31.459 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[39278]: 5.1e scrub starts 2024-02-05T08:26:31.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[39278]: 5.1e scrub ok 2024-02-05T08:26:31.460 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:31 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4282611691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:26:31.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:31.490 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:31.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:31.508 INFO:tasks.thrashosds.thrasher:j is {'epoch': 422, 'fsid': '43e13aa4-c3fc-11ee-95b6-87774f69a715', 'created': '2024-02-05T07:59:09.321262+0000', 'modified': '2024-02-05T08:26:27.944534+0000', 'last_up_change': '2024-02-05T08:23:06.892657+0000', 'last_in_change': '2024-02-05T08:25:48.086167+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T08:02:21.034012+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '303', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '301', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.8796920776367188, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T08:04:39.400746+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 70, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8796920776367188, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T08:04:40.005769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8796920776367188, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T08:04:41.851891+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.8796920776367188, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T08:04:43.793804+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8796920776367188, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T08:04:45.853999+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8796920776367188, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-02-05T08:23:07.682500+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '396', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.8796920776367188, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'd90a5ea0-9a19-452e-9564-ca7b7a7c45e4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6802', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6803', 'nonce': 2540271081}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6804', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6805', 'nonce': 2540271081}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6808', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6809', 'nonce': 2540271081}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6806', 'nonce': 2540271081}, {'type': 'v1', 'addr': '172.21.15.134:6807', 'nonce': 2540271081}]}, 'public_addr': '172.21.15.134:6803/2540271081', 'cluster_addr': '172.21.15.134:6805/2540271081', 'heartbeat_back_addr': '172.21.15.134:6809/2540271081', 'heartbeat_front_addr': '172.21.15.134:6807/2540271081', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 323, 'up_from': 366, 'up_thru': 416, 'down_at': 324, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6810', 'nonce': 1259179009}, {'type': 'v1', 'addr': '172.21.15.134:6811', 'nonce': 1259179009}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6812', 'nonce': 1259179009}, {'type': 'v1', 'addr': '172.21.15.134:6813', 'nonce': 1259179009}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6816', 'nonce': 1259179009}, {'type': 'v1', 'addr': '172.21.15.134:6817', 'nonce': 1259179009}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6814', 'nonce': 1259179009}, {'type': 'v1', 'addr': '172.21.15.134:6815', 'nonce': 1259179009}]}, 'public_addr': '172.21.15.134:6811/1259179009', 'cluster_addr': '172.21.15.134:6813/1259179009', 'heartbeat_back_addr': '172.21.15.134:6817/1259179009', 'heartbeat_front_addr': '172.21.15.134:6815/1259179009', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '078e578b-855b-49ff-83bd-0548a9c5a708', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.4826202392578125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 416, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6818', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6819', 'nonce': 153827987}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6820', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6821', 'nonce': 153827987}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6824', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6825', 'nonce': 153827987}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6822', 'nonce': 153827987}, {'type': 'v1', 'addr': '172.21.15.134:6823', 'nonce': 153827987}]}, 'public_addr': '172.21.15.134:6819/153827987', 'cluster_addr': '172.21.15.134:6821/153827987', 'heartbeat_back_addr': '172.21.15.134:6825/153827987', 'heartbeat_front_addr': '172.21.15.134:6823/153827987', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '6774e330-3202-4671-894e-172cc2f6189d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 421, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6826', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6827', 'nonce': 1527507056}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6828', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6829', 'nonce': 1527507056}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6832', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6833', 'nonce': 1527507056}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.134:6830', 'nonce': 1527507056}, {'type': 'v1', 'addr': '172.21.15.134:6831', 'nonce': 1527507056}]}, 'public_addr': '172.21.15.134:6827/1527507056', 'cluster_addr': '172.21.15.134:6829/1527507056', 'heartbeat_back_addr': '172.21.15.134:6833/1527507056', 'heartbeat_front_addr': '172.21.15.134:6831/1527507056', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '52f70cfa-ab0e-4751-a5f8-1045c61c1693', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 416, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6800', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6801', 'nonce': 3005156808}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6802', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6803', 'nonce': 3005156808}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6806', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6807', 'nonce': 3005156808}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6804', 'nonce': 3005156808}, {'type': 'v1', 'addr': '172.21.15.196:6805', 'nonce': 3005156808}]}, 'public_addr': '172.21.15.196:6801/3005156808', 'cluster_addr': '172.21.15.196:6803/3005156808', 'heartbeat_back_addr': '172.21.15.196:6807/3005156808', 'heartbeat_front_addr': '172.21.15.196:6805/3005156808', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b4b21769-d40f-43fd-b7e9-2477b74d3955', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.5549163818359375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 409, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6808', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6809', 'nonce': 1603224792}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6810', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6811', 'nonce': 1603224792}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6814', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6815', 'nonce': 1603224792}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6812', 'nonce': 1603224792}, {'type': 'v1', 'addr': '172.21.15.196:6813', 'nonce': 1603224792}]}, 'public_addr': '172.21.15.196:6809/1603224792', 'cluster_addr': '172.21.15.196:6811/1603224792', 'heartbeat_back_addr': '172.21.15.196:6815/1603224792', 'heartbeat_front_addr': '172.21.15.196:6813/1603224792', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'af1cced4-0015-47f5-b105-27641bfe4f4a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6816', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6817', 'nonce': 3727413143}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6818', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6819', 'nonce': 3727413143}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6822', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6823', 'nonce': 3727413143}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6820', 'nonce': 3727413143}, {'type': 'v1', 'addr': '172.21.15.196:6821', 'nonce': 3727413143}]}, 'public_addr': '172.21.15.196:6817/3727413143', 'cluster_addr': '172.21.15.196:6819/3727413143', 'heartbeat_back_addr': '172.21.15.196:6823/3727413143', 'heartbeat_front_addr': '172.21.15.196:6821/3727413143', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'c54beeae-9f7c-4723-a369-13bb531b4057', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 416, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6824', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6825', 'nonce': 4206571010}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6826', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6827', 'nonce': 4206571010}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6830', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6831', 'nonce': 4206571010}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6828', 'nonce': 4206571010}, {'type': 'v1', 'addr': '172.21.15.196:6829', 'nonce': 4206571010}]}, 'public_addr': '172.21.15.196:6825/4206571010', 'cluster_addr': '172.21.15.196:6827/4206571010', 'heartbeat_back_addr': '172.21.15.196:6831/4206571010', 'heartbeat_front_addr': '172.21.15.196:6829/4206571010', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:25:59.840728+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:25:39.306744+0000', 'dead_epoch': 323}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:26:05.494116+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:25:57.888022+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:25:56.383775+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:25:38.896269+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:25:57.864927+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T08:25:33.765646+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.0', 'osds': [3, 1, 5]}, {'pgid': '6.6', 'osds': [4, 1, 3]}], 'pg_upmap_items': [{'pgid': '3.6', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.7', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.134:0/788812699': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/1046500326': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/464208089': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/301941854': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:6800/3607168099': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6800/673876301': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/4225724311': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6800/1525062017': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:6801/3607168099': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:6801/673876301': '2024-02-06T07:59:36.275039+0000', '172.21.15.134:0/3200884805': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/2952444615': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:0/3325835814': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/3161126612': '2024-02-06T08:00:02.709280+0000', '172.21.15.134:0/69097091': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:6801/1525062017': '2024-02-06T08:05:21.465844+0000', '172.21.15.134:0/1645475987': '2024-02-06T08:05:21.465844+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T08:26:31.509 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:26:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:31 smithi196 ceph-mon[33018]: 6.1 scrub starts 2024-02-05T08:26:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:31 smithi196 ceph-mon[33018]: 6.1 scrub ok 2024-02-05T08:26:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:31 smithi196 ceph-mon[33018]: 4.1f scrub starts 2024-02-05T08:26:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:31 smithi196 ceph-mon[33018]: 4.1f scrub ok 2024-02-05T08:26:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:31 smithi196 ceph-mon[33018]: 11.6 scrub ok 2024-02-05T08:26:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:31 smithi196 ceph-mon[33018]: pgmap v977: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.4 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 5.7 MiB/s, 91 objects/s recovering 2024-02-05T08:26:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:31 smithi196 ceph-mon[33018]: 5.1e scrub starts 2024-02-05T08:26:31.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:31 smithi196 ceph-mon[33018]: 5.1e scrub ok 2024-02-05T08:26:31.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:31 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4282611691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T08:26:31.591 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:31.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:31.640 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:31.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:31.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:31.785 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:31.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:31.886 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:31.929 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:31.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:32.031 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:32.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:32.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:32.167 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:32.209 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:32.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:32.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:32 smithi196 ceph-mon[33018]: 4.4 scrub starts 2024-02-05T08:26:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:32 smithi196 ceph-mon[33018]: 4.4 scrub ok 2024-02-05T08:26:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:32 smithi196 ceph-mon[33018]: 5.7 scrub starts 2024-02-05T08:26:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:32 smithi196 ceph-mon[33018]: 1.6 scrub starts 2024-02-05T08:26:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:32 smithi196 ceph-mon[33018]: 1.6 scrub ok 2024-02-05T08:26:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:32 smithi196 ceph-mon[33018]: 5.7 scrub ok 2024-02-05T08:26:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:32 smithi196 ceph-mon[33018]: 4.1b scrub starts 2024-02-05T08:26:32.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:32 smithi196 ceph-mon[33018]: 4.1b scrub ok 2024-02-05T08:26:32.344 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:32.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:32.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:32.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[30086]: 4.4 scrub starts 2024-02-05T08:26:32.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[30086]: 4.4 scrub ok 2024-02-05T08:26:32.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[30086]: 5.7 scrub starts 2024-02-05T08:26:32.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[30086]: 1.6 scrub starts 2024-02-05T08:26:32.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[30086]: 1.6 scrub ok 2024-02-05T08:26:32.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[30086]: 5.7 scrub ok 2024-02-05T08:26:32.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[30086]: 4.1b scrub starts 2024-02-05T08:26:32.350 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[30086]: 4.1b scrub ok 2024-02-05T08:26:32.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[39278]: 4.4 scrub starts 2024-02-05T08:26:32.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[39278]: 4.4 scrub ok 2024-02-05T08:26:32.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[39278]: 5.7 scrub starts 2024-02-05T08:26:32.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[39278]: 1.6 scrub starts 2024-02-05T08:26:32.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[39278]: 1.6 scrub ok 2024-02-05T08:26:32.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[39278]: 5.7 scrub ok 2024-02-05T08:26:32.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[39278]: 4.1b scrub starts 2024-02-05T08:26:32.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:32 smithi134 ceph-mon[39278]: 4.1b scrub ok 2024-02-05T08:26:32.446 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:32.484 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:32.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:32.586 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:32.614 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:32.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:32.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:32.716 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:32.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:32.765 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:32.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:32.866 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:32.909 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:32.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:32.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:33.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:33.045 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:33.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:33.147 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:33.189 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:33.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:33.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:33.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:33 smithi134 ceph-mon[30086]: 1.7 scrub starts 2024-02-05T08:26:33.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:33 smithi134 ceph-mon[30086]: 1.7 scrub ok 2024-02-05T08:26:33.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:33 smithi134 ceph-mon[30086]: pgmap v978: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 5.6 MiB/s, 90 objects/s recovering 2024-02-05T08:26:33.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:33 smithi134 ceph-mon[30086]: 3.13 scrub starts 2024-02-05T08:26:33.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:33 smithi134 ceph-mon[30086]: 3.13 scrub ok 2024-02-05T08:26:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:33 smithi134 ceph-mon[39278]: 1.7 scrub starts 2024-02-05T08:26:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:33 smithi134 ceph-mon[39278]: 1.7 scrub ok 2024-02-05T08:26:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:33 smithi134 ceph-mon[39278]: pgmap v978: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 5.6 MiB/s, 90 objects/s recovering 2024-02-05T08:26:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:33 smithi134 ceph-mon[39278]: 3.13 scrub starts 2024-02-05T08:26:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:33 smithi134 ceph-mon[39278]: 3.13 scrub ok 2024-02-05T08:26:33.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:33.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:33.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:33.384 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:33 smithi196 ceph-mon[33018]: 1.7 scrub starts 2024-02-05T08:26:33.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:33 smithi196 ceph-mon[33018]: 1.7 scrub ok 2024-02-05T08:26:33.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:33 smithi196 ceph-mon[33018]: pgmap v978: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 682 op/s; 5.6 MiB/s, 90 objects/s recovering 2024-02-05T08:26:33.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:33 smithi196 ceph-mon[33018]: 3.13 scrub starts 2024-02-05T08:26:33.385 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:33 smithi196 ceph-mon[33018]: 3.13 scrub ok 2024-02-05T08:26:33.422 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:33.468 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:33.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:33.569 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:33.578 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:26:33.578 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [], 2024-02-05T08:26:33.578 INFO:teuthology.orchestra.run.smithi196.stdout: "complaint_time": 30, 2024-02-05T08:26:33.578 INFO:teuthology.orchestra.run.smithi196.stdout: "num_blocked_ops": 0 2024-02-05T08:26:33.578 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:26:33.619 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:33.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:33.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:33.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:33.736 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:33.778 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:33.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:33.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:33.929 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:33.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:33.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:34.031 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:34.073 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:34.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:34.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:34.175 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:34.196 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:34 smithi196 ceph-mon[33018]: 3.a scrub starts 2024-02-05T08:26:34.196 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:34 smithi196 ceph-mon[33018]: 3.a scrub ok 2024-02-05T08:26:34.196 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:34 smithi196 ceph-mon[33018]: 6.18 scrub starts 2024-02-05T08:26:34.196 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:34 smithi196 ceph-mon[33018]: 6.18 scrub ok 2024-02-05T08:26:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:34.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:34.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:34 smithi134 ceph-mon[30086]: 3.a scrub starts 2024-02-05T08:26:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:34 smithi134 ceph-mon[30086]: 3.a scrub ok 2024-02-05T08:26:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:34 smithi134 ceph-mon[30086]: 6.18 scrub starts 2024-02-05T08:26:34.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:34 smithi134 ceph-mon[30086]: 6.18 scrub ok 2024-02-05T08:26:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:34 smithi134 ceph-mon[39278]: 3.a scrub starts 2024-02-05T08:26:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:34 smithi134 ceph-mon[39278]: 3.a scrub ok 2024-02-05T08:26:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:34 smithi134 ceph-mon[39278]: 6.18 scrub starts 2024-02-05T08:26:34.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:34 smithi134 ceph-mon[39278]: 6.18 scrub ok 2024-02-05T08:26:34.290 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T08:26:34.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:34.365 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:34.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:34.466 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:34.504 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:34.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:34.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:34.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:26:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:26:34.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:34.637 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:34.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:34.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:34.664 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:26:34.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:34.781 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:34.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:34.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:34.883 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:34.927 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:34.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:34.963 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:26:35.006 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:26:35.028 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:35.044 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:26:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:35.076 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:35.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:35.089 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:26:35.134 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:26:35.177 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:26:35.177 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:35.180 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:26:35.229 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:26:35.231 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:35.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:35.267 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:26:35.268 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[30086]: 5.12 scrub starts 2024-02-05T08:26:35.268 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[30086]: 5.12 scrub ok 2024-02-05T08:26:35.268 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[30086]: 5.1b scrub starts 2024-02-05T08:26:35.268 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[30086]: 5.1b scrub ok 2024-02-05T08:26:35.268 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[30086]: 5.11 scrub starts 2024-02-05T08:26:35.268 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[30086]: 5.11 scrub ok 2024-02-05T08:26:35.268 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[30086]: pgmap v979: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 4.2 MiB/s, 68 objects/s recovering 2024-02-05T08:26:35.268 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/465780704' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:26:35.269 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:26:35.269 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:35.269 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[30086]: 5.8 scrub starts 2024-02-05T08:26:35.269 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[30086]: 5.8 scrub ok 2024-02-05T08:26:35.269 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3170942334' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:35.269 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[39278]: 5.12 scrub starts 2024-02-05T08:26:35.269 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[39278]: 5.12 scrub ok 2024-02-05T08:26:35.269 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[39278]: 5.1b scrub starts 2024-02-05T08:26:35.269 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[39278]: 5.1b scrub ok 2024-02-05T08:26:35.269 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[39278]: 5.11 scrub starts 2024-02-05T08:26:35.269 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[39278]: 5.11 scrub ok 2024-02-05T08:26:35.269 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[39278]: pgmap v979: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 4.2 MiB/s, 68 objects/s recovering 2024-02-05T08:26:35.270 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/465780704' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:26:35.270 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:26:35.270 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:35.270 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[39278]: 5.8 scrub starts 2024-02-05T08:26:35.270 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[39278]: 5.8 scrub ok 2024-02-05T08:26:35.270 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:35 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3170942334' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:35.279 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:26:35.279 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:26:35.279 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:26:35.279 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:26:35.279 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:26:35.279 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:26:35.280 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:26:35.280 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:26:35.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:35.366 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:35.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:35.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:35.498 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:35.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:35.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:35 smithi196 ceph-mon[33018]: 5.12 scrub starts 2024-02-05T08:26:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:35 smithi196 ceph-mon[33018]: 5.12 scrub ok 2024-02-05T08:26:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:35 smithi196 ceph-mon[33018]: 5.1b scrub starts 2024-02-05T08:26:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:35 smithi196 ceph-mon[33018]: 5.1b scrub ok 2024-02-05T08:26:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:35 smithi196 ceph-mon[33018]: 5.11 scrub starts 2024-02-05T08:26:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:35 smithi196 ceph-mon[33018]: 5.11 scrub ok 2024-02-05T08:26:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:35 smithi196 ceph-mon[33018]: pgmap v979: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 4.2 MiB/s, 68 objects/s recovering 2024-02-05T08:26:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:35 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/465780704' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:26:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:35 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:26:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:35 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:35 smithi196 ceph-mon[33018]: 5.8 scrub starts 2024-02-05T08:26:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:35 smithi196 ceph-mon[33018]: 5.8 scrub ok 2024-02-05T08:26:35.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:35 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3170942334' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:35.600 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:35.641 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:35.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:35.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:35.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:35.778 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:35.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:35.879 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:35.915 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:35.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:36.017 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:36.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:36.061 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:36.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:36.162 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:36.202 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:36.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:36.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:36.303 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:36.341 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:36.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:36.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:36.441 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:36 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:26:36.441 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:36 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:26:36.441 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:36 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:26:36.441 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:36 smithi134 ceph-mon[30086]: osdmap e423: 8 total, 8 up, 8 in 2024-02-05T08:26:36.441 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:36 smithi134 ceph-mon[30086]: 4.18 scrub starts 2024-02-05T08:26:36.441 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:36 smithi134 ceph-mon[30086]: 4.18 scrub ok 2024-02-05T08:26:36.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:36 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:26:36.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:36 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:26:36.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:36 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:26:36.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:36 smithi134 ceph-mon[39278]: osdmap e423: 8 total, 8 up, 8 in 2024-02-05T08:26:36.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:36 smithi134 ceph-mon[39278]: 4.18 scrub starts 2024-02-05T08:26:36.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:36 smithi134 ceph-mon[39278]: 4.18 scrub ok 2024-02-05T08:26:36.442 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:36.447 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:36 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:26:36.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:36 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:26:36.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:36 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:26:36.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:36 smithi196 ceph-mon[33018]: osdmap e423: 8 total, 8 up, 8 in 2024-02-05T08:26:36.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:36 smithi196 ceph-mon[33018]: 4.18 scrub starts 2024-02-05T08:26:36.448 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:36 smithi196 ceph-mon[33018]: 4.18 scrub ok 2024-02-05T08:26:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:36.481 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:36.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:36.582 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:36.620 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:36.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:36.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:36.722 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:36.763 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:36.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:36.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:36.796 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:26:36.797 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:26:36.865 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:36.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:36.896 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:36.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:36.997 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:37.028 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:37.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:37.130 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:37.174 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:37.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:37.276 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:37.317 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:37.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:37.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:37.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:37.457 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:37.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:37.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[30086]: pgmap v981: 158 pgs: 158 active+clean; 1.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 4.2 MiB/s, 68 objects/s recovering 2024-02-05T08:26:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[30086]: 6.11 scrub starts 2024-02-05T08:26:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[30086]: 6.11 scrub ok 2024-02-05T08:26:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[30086]: 4.e scrub starts 2024-02-05T08:26:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[30086]: 4.e scrub ok 2024-02-05T08:26:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:26:37.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:26:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[39278]: pgmap v981: 158 pgs: 158 active+clean; 1.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 4.2 MiB/s, 68 objects/s recovering 2024-02-05T08:26:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[39278]: 6.11 scrub starts 2024-02-05T08:26:37.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[39278]: 6.11 scrub ok 2024-02-05T08:26:37.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[39278]: 4.e scrub starts 2024-02-05T08:26:37.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[39278]: 4.e scrub ok 2024-02-05T08:26:37.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:26:37.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:26:37.560 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:37 smithi196 ceph-mon[33018]: pgmap v981: 158 pgs: 158 active+clean; 1.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 4.2 MiB/s, 68 objects/s recovering 2024-02-05T08:26:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:37 smithi196 ceph-mon[33018]: 6.11 scrub starts 2024-02-05T08:26:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:37 smithi196 ceph-mon[33018]: 6.11 scrub ok 2024-02-05T08:26:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:37 smithi196 ceph-mon[33018]: 4.e scrub starts 2024-02-05T08:26:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:37 smithi196 ceph-mon[33018]: 4.e scrub ok 2024-02-05T08:26:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:37 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:26:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:37 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:26:37.601 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:37.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:37.703 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:37.747 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:37.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:37.848 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:37.889 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:37.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:37.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:38.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:26:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:26:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:26:38.027 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:38.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:38.129 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:38.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:38.165 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:38.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:38.236 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:38 smithi196 ceph-mon[33018]: osdmap e424: 8 total, 8 up, 8 in 2024-02-05T08:26:38.236 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:38 smithi196 ceph-mon[33018]: 6.9 scrub starts 2024-02-05T08:26:38.236 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:38 smithi196 ceph-mon[33018]: 4.9 scrub starts 2024-02-05T08:26:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:38 smithi196 ceph-mon[33018]: 4.9 scrub ok 2024-02-05T08:26:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:38 smithi196 ceph-mon[33018]: 6.9 scrub ok 2024-02-05T08:26:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:38 smithi196 ceph-mon[33018]: 2.1 scrub starts 2024-02-05T08:26:38.237 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:38 smithi196 ceph-mon[33018]: 2.1 scrub ok 2024-02-05T08:26:38.266 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:38.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T08:26:38.295 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:38.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:38.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:26:38.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:38.436 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:38.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:38.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:38.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[30086]: osdmap e424: 8 total, 8 up, 8 in 2024-02-05T08:26:38.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[30086]: 6.9 scrub starts 2024-02-05T08:26:38.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[30086]: 4.9 scrub starts 2024-02-05T08:26:38.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[30086]: 4.9 scrub ok 2024-02-05T08:26:38.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[30086]: 6.9 scrub ok 2024-02-05T08:26:38.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[30086]: 2.1 scrub starts 2024-02-05T08:26:38.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[30086]: 2.1 scrub ok 2024-02-05T08:26:38.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[39278]: osdmap e424: 8 total, 8 up, 8 in 2024-02-05T08:26:38.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[39278]: 6.9 scrub starts 2024-02-05T08:26:38.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[39278]: 4.9 scrub starts 2024-02-05T08:26:38.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[39278]: 4.9 scrub ok 2024-02-05T08:26:38.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[39278]: 6.9 scrub ok 2024-02-05T08:26:38.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[39278]: 2.1 scrub starts 2024-02-05T08:26:38.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:38 smithi134 ceph-mon[39278]: 2.1 scrub ok 2024-02-05T08:26:38.538 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:38.573 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:38.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:38.623 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:26:38.675 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:38.731 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:38.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:38.832 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:38.874 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:38.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:38.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:38.975 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:39.007 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:39.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:39.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:39.109 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:39.132 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:39.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:39.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:39.139 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T08:26:39.233 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:39.260 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:39.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:39.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:39.389 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:39.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:39.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:39.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:39.491 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:39 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:26:39.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:39 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:26:39.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:39 smithi196 ceph-mon[33018]: pgmap v983: 158 pgs: 158 active+clean; 1018 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 782 op/s 2024-02-05T08:26:39.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:39 smithi196 ceph-mon[33018]: 4.1e scrub starts 2024-02-05T08:26:39.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:39 smithi196 ceph-mon[33018]: 4.1e scrub ok 2024-02-05T08:26:39.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:39 smithi196 ceph-mon[33018]: 6.14 deep-scrub starts 2024-02-05T08:26:39.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:39 smithi196 ceph-mon[33018]: 6.14 deep-scrub ok 2024-02-05T08:26:39.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:26:39.492 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:26:39.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:26:39.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:26:39.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[30086]: pgmap v983: 158 pgs: 158 active+clean; 1018 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 782 op/s 2024-02-05T08:26:39.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[30086]: 4.1e scrub starts 2024-02-05T08:26:39.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[30086]: 4.1e scrub ok 2024-02-05T08:26:39.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[30086]: 6.14 deep-scrub starts 2024-02-05T08:26:39.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[30086]: 6.14 deep-scrub ok 2024-02-05T08:26:39.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:26:39.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:26:39.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:26:39.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:26:39.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[39278]: pgmap v983: 158 pgs: 158 active+clean; 1018 MiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 782 op/s 2024-02-05T08:26:39.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[39278]: 4.1e scrub starts 2024-02-05T08:26:39.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[39278]: 4.1e scrub ok 2024-02-05T08:26:39.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[39278]: 6.14 deep-scrub starts 2024-02-05T08:26:39.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[39278]: 6.14 deep-scrub ok 2024-02-05T08:26:39.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:26:39.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:26:39.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:39.531 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:39.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:39.632 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:39.667 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:39.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:39.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:39.769 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:39.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:39.799 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:39.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:39.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:39.936 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:39.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:40.038 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:40.064 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:40.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:40.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:40.151 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:40.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:40.194 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:40.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:40.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:40.267 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:40 smithi134 ceph-mon[30086]: 4.17 scrub starts 2024-02-05T08:26:40.267 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:40 smithi134 ceph-mon[30086]: 4.17 scrub ok 2024-02-05T08:26:40.267 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:40 smithi134 ceph-mon[30086]: 5.9 scrub starts 2024-02-05T08:26:40.267 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:40 smithi134 ceph-mon[30086]: 5.9 scrub ok 2024-02-05T08:26:40.267 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:40 smithi134 ceph-mon[39278]: 4.17 scrub starts 2024-02-05T08:26:40.267 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:40 smithi134 ceph-mon[39278]: 4.17 scrub ok 2024-02-05T08:26:40.267 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:40 smithi134 ceph-mon[39278]: 5.9 scrub starts 2024-02-05T08:26:40.267 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:40 smithi134 ceph-mon[39278]: 5.9 scrub ok 2024-02-05T08:26:40.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:40.330 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:40.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:40.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:40.432 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:40.465 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:40.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:40 smithi196 ceph-mon[33018]: 4.17 scrub starts 2024-02-05T08:26:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:40 smithi196 ceph-mon[33018]: 4.17 scrub ok 2024-02-05T08:26:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:40 smithi196 ceph-mon[33018]: 5.9 scrub starts 2024-02-05T08:26:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:40 smithi196 ceph-mon[33018]: 5.9 scrub ok 2024-02-05T08:26:40.567 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:40.597 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:40.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:40.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:40.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:40.731 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:40.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:40.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:40.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:40.849 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:40.874 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:40.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:40.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:40.976 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:41.006 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:41.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:41.108 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:41.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:41.152 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:41.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:41.255 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:41.282 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:41.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:41.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:41.347 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:26:41.351 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:26:41.384 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:41.412 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:41.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:41.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:41.493 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:41 smithi196 ceph-mon[33018]: pgmap v984: 158 pgs: 158 active+clean; 700 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-02-05T08:26:41.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:41 smithi196 ceph-mon[33018]: 3.19 scrub starts 2024-02-05T08:26:41.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:41 smithi196 ceph-mon[33018]: 3.19 scrub ok 2024-02-05T08:26:41.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:41 smithi196 ceph-mon[33018]: 5.4 scrub starts 2024-02-05T08:26:41.494 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:41 smithi196 ceph-mon[33018]: 5.4 scrub ok 2024-02-05T08:26:41.501 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:41 smithi134 ceph-mon[30086]: pgmap v984: 158 pgs: 158 active+clean; 700 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-02-05T08:26:41.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:41 smithi134 ceph-mon[30086]: 3.19 scrub starts 2024-02-05T08:26:41.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:41 smithi134 ceph-mon[30086]: 3.19 scrub ok 2024-02-05T08:26:41.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:41 smithi134 ceph-mon[30086]: 5.4 scrub starts 2024-02-05T08:26:41.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:41 smithi134 ceph-mon[30086]: 5.4 scrub ok 2024-02-05T08:26:41.502 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:41 smithi134 ceph-mon[39278]: pgmap v984: 158 pgs: 158 active+clean; 700 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-02-05T08:26:41.502 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:41 smithi134 ceph-mon[39278]: 3.19 scrub starts 2024-02-05T08:26:41.502 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:41 smithi134 ceph-mon[39278]: 3.19 scrub ok 2024-02-05T08:26:41.502 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:41 smithi134 ceph-mon[39278]: 5.4 scrub starts 2024-02-05T08:26:41.502 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:41 smithi134 ceph-mon[39278]: 5.4 scrub ok 2024-02-05T08:26:41.515 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:41.542 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:26:41.543 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:26:41.549 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:41.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:41.651 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:41.697 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:41.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:41.798 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:41.836 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:41.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:41.938 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:41.974 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:41.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:42.076 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:42.119 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:42.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:42.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:42.256 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:42.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:42.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:42.358 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:42.394 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:42.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:42.496 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:42.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:42 smithi134 ceph-mon[30086]: from='client.25642 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:26:42.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:42 smithi134 ceph-mon[39278]: from='client.25642 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:26:42.528 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:42.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:42.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:42 smithi196 ceph-mon[33018]: from='client.25642 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:26:42.630 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:42.661 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:42.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:42.762 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:42.796 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:42.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:42.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:42.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:42.934 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:42.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:43.035 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:43.080 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:43.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:43.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:43.182 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:43.216 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:43.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:43.275 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T08:26:43.318 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:43.358 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:43.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:43.459 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:43.498 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:43.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[30086]: 11.7 scrub starts 2024-02-05T08:26:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[30086]: pgmap v985: 158 pgs: 158 active+clean; 657 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 860 op/s 2024-02-05T08:26:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[30086]: 11.7 scrub ok 2024-02-05T08:26:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[30086]: 3.1d scrub starts 2024-02-05T08:26:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[30086]: 3.1d scrub ok 2024-02-05T08:26:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[30086]: 1.9 scrub starts 2024-02-05T08:26:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[30086]: 1.9 scrub ok 2024-02-05T08:26:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[30086]: 5.1d scrub starts 2024-02-05T08:26:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[30086]: 5.1d scrub ok 2024-02-05T08:26:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:26:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:26:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[39278]: 11.7 scrub starts 2024-02-05T08:26:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[39278]: pgmap v985: 158 pgs: 158 active+clean; 657 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 860 op/s 2024-02-05T08:26:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[39278]: 11.7 scrub ok 2024-02-05T08:26:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[39278]: 3.1d scrub starts 2024-02-05T08:26:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[39278]: 3.1d scrub ok 2024-02-05T08:26:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[39278]: 1.9 scrub starts 2024-02-05T08:26:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[39278]: 1.9 scrub ok 2024-02-05T08:26:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[39278]: 5.1d scrub starts 2024-02-05T08:26:43.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[39278]: 5.1d scrub ok 2024-02-05T08:26:43.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:26:43.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:43 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:26:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:43 smithi196 ceph-mon[33018]: 11.7 scrub starts 2024-02-05T08:26:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:43 smithi196 ceph-mon[33018]: pgmap v985: 158 pgs: 158 active+clean; 657 MiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 860 op/s 2024-02-05T08:26:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:43 smithi196 ceph-mon[33018]: 11.7 scrub ok 2024-02-05T08:26:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:43 smithi196 ceph-mon[33018]: 3.1d scrub starts 2024-02-05T08:26:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:43 smithi196 ceph-mon[33018]: 3.1d scrub ok 2024-02-05T08:26:43.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:43 smithi196 ceph-mon[33018]: 1.9 scrub starts 2024-02-05T08:26:43.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:43 smithi196 ceph-mon[33018]: 1.9 scrub ok 2024-02-05T08:26:43.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:43 smithi196 ceph-mon[33018]: 5.1d scrub starts 2024-02-05T08:26:43.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:43 smithi196 ceph-mon[33018]: 5.1d scrub ok 2024-02-05T08:26:43.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:43 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:26:43.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:43 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:26:43.599 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:43.628 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:26:43.647 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:43.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:43.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:43.749 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:43.790 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:43.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:43.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:43.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:43.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:43.996 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:43.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:44.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:44.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:44.135 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:44.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:44.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:44.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:44.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:44.371 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:44.431 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:44.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:44.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:44 smithi134 ceph-mon[30086]: 4.1d deep-scrub starts 2024-02-05T08:26:44.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:44 smithi134 ceph-mon[30086]: osdmap e425: 8 total, 8 up, 8 in 2024-02-05T08:26:44.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:44 smithi134 ceph-mon[30086]: 4.1d deep-scrub ok 2024-02-05T08:26:44.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:44 smithi134 ceph-mon[39278]: 4.1d deep-scrub starts 2024-02-05T08:26:44.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:44 smithi134 ceph-mon[39278]: osdmap e425: 8 total, 8 up, 8 in 2024-02-05T08:26:44.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:44 smithi134 ceph-mon[39278]: 4.1d deep-scrub ok 2024-02-05T08:26:44.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:44.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:44 smithi196 ceph-mon[33018]: 4.1d deep-scrub starts 2024-02-05T08:26:44.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:44 smithi196 ceph-mon[33018]: osdmap e425: 8 total, 8 up, 8 in 2024-02-05T08:26:44.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:44 smithi196 ceph-mon[33018]: 4.1d deep-scrub ok 2024-02-05T08:26:44.565 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:26:44 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:26:44.570 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:44.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:44.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:44.672 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:44.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:44.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:44.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:44.858 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:44.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:44.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:44.960 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:45.009 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:45.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:45.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:45.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:45.149 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:45.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:45.192 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":984,"stamp":"2024-02-05T08:26:39.970856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":734337396,"num_objects":11395,"num_object_clones":0,"num_object_copies":34185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11395,"num_whiteouts":0,"num_read":2846,"num_read_kb":2598,"num_write":260011,"num_write_kb":8646382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12002,"num_bytes_recovered":759377647,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81574,"ondisk_log_size":81574,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":477,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8063628,"kb_used_data":2531644,"kb_used_omap":15,"kb_used_meta":5531632,"kb_avail":741897588,"statfs":{"total":767960285184,"available":759703130112,"internally_reserved":0,"allocated":2592403456,"data_stored":2589816048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15596,"internal_metadata":5664391956},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-709492736,"num_objects":-10826,"num_object_clones":0,"num_object_copies":-32478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10826,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10826,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002232"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":793,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.161628+0000","last_change":"2024-02-05T08:24:36.101074+0000","last_active":"2024-02-05T08:26:37.161628+0000","last_peered":"2024-02-05T08:26:37.161628+0000","last_clean":"2024-02-05T08:26:37.161628+0000","last_became_active":"2024-02-05T08:23:07.931642+0000","last_became_peered":"2024-02-05T08:23:07.931642+0000","last_unstale":"2024-02-05T08:26:37.161628+0000","last_undegraded":"2024-02-05T08:26:37.161628+0000","last_fullsized":"2024-02-05T08:26:37.161628+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:36.100961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:36.100961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":659,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.163891+0000","last_change":"2024-02-05T08:26:25.495777+0000","last_active":"2024-02-05T08:26:35.163891+0000","last_peered":"2024-02-05T08:26:35.163891+0000","last_clean":"2024-02-05T08:26:35.163891+0000","last_became_active":"2024-02-05T08:23:07.938772+0000","last_became_peered":"2024-02-05T08:23:07.938772+0000","last_unstale":"2024-02-05T08:26:35.163891+0000","last_undegraded":"2024-02-05T08:26:35.163891+0000","last_fullsized":"2024-02-05T08:26:35.163891+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:25.494948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:26:25.494948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:34.726850+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":45,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175851+0000","last_change":"2024-02-05T08:25:49.142878+0000","last_active":"2024-02-05T08:26:37.175851+0000","last_peered":"2024-02-05T08:26:37.175851+0000","last_clean":"2024-02-05T08:26:37.175851+0000","last_became_active":"2024-02-05T08:25:49.142192+0000","last_became_peered":"2024-02-05T08:25:49.142192+0000","last_unstale":"2024-02-05T08:26:37.175851+0000","last_undegraded":"2024-02-05T08:26:37.175851+0000","last_fullsized":"2024-02-05T08:26:37.175851+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:26.865662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:25:26.865662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:53.268325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"397'25","reported_seq":872,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.161391+0000","last_change":"2024-02-05T08:26:35.411562+0000","last_active":"2024-02-05T08:26:37.161391+0000","last_peered":"2024-02-05T08:26:37.161391+0000","last_clean":"2024-02-05T08:26:37.161391+0000","last_became_active":"2024-02-05T08:23:07.937041+0000","last_became_peered":"2024-02-05T08:23:07.937041+0000","last_unstale":"2024-02-05T08:26:37.161391+0000","last_undegraded":"2024-02-05T08:26:37.161391+0000","last_fullsized":"2024-02-05T08:26:37.161391+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-02-05T08:26:35.411412+0000","last_deep_scrub":"90'13","last_deep_scrub_stamp":"2024-02-05T08:05:58.103749+0000","last_clean_scrub_stamp":"2024-02-05T08:26:35.411412+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:53.484129+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":296,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.211250+0000","last_change":"2024-02-05T08:23:24.076782+0000","last_active":"2024-02-05T08:26:37.211250+0000","last_peered":"2024-02-05T08:26:37.211250+0000","last_clean":"2024-02-05T08:26:37.211250+0000","last_became_active":"2024-02-05T08:18:34.633083+0000","last_became_peered":"2024-02-05T08:18:34.633083+0000","last_unstale":"2024-02-05T08:26:37.211250+0000","last_undegraded":"2024-02-05T08:26:37.211250+0000","last_fullsized":"2024-02-05T08:26:37.211250+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:24.076590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:23:24.076590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":810,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.161571+0000","last_change":"2024-02-05T08:26:27.430624+0000","last_active":"2024-02-05T08:26:37.161571+0000","last_peered":"2024-02-05T08:26:37.161571+0000","last_clean":"2024-02-05T08:26:37.161571+0000","last_became_active":"2024-02-05T08:23:07.937180+0000","last_became_peered":"2024-02-05T08:23:07.937180+0000","last_unstale":"2024-02-05T08:26:37.161571+0000","last_undegraded":"2024-02-05T08:26:37.161571+0000","last_fullsized":"2024-02-05T08:26:37.161571+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:27.430499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:41.460468+0000","last_clean_scrub_stamp":"2024-02-05T08:26:27.430499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:33.795037+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":306,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.177458+0000","last_change":"2024-02-05T08:24:20.030502+0000","last_active":"2024-02-05T08:26:37.177458+0000","last_peered":"2024-02-05T08:26:37.177458+0000","last_clean":"2024-02-05T08:26:37.177458+0000","last_became_active":"2024-02-05T08:21:41.880038+0000","last_became_peered":"2024-02-05T08:21:41.880038+0000","last_unstale":"2024-02-05T08:26:37.177458+0000","last_undegraded":"2024-02-05T08:26:37.177458+0000","last_fullsized":"2024-02-05T08:26:37.177458+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:20.030382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:29.985333+0000","last_clean_scrub_stamp":"2024-02-05T08:24:20.030382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"397'46","reported_seq":382,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.178141+0000","last_change":"2024-02-05T08:25:49.182238+0000","last_active":"2024-02-05T08:26:37.178141+0000","last_peered":"2024-02-05T08:26:37.178141+0000","last_clean":"2024-02-05T08:26:37.178141+0000","last_became_active":"2024-02-05T08:25:49.140432+0000","last_became_peered":"2024-02-05T08:25:49.140432+0000","last_unstale":"2024-02-05T08:26:37.178141+0000","last_undegraded":"2024-02-05T08:26:37.178141+0000","last_fullsized":"2024-02-05T08:26:37.178141+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'34","last_scrub_stamp":"2024-02-05T08:24:30.076161+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:24:30.076161+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176022+0000","last_change":"2024-02-05T08:26:33.338485+0000","last_active":"2024-02-05T08:26:37.176022+0000","last_peered":"2024-02-05T08:26:37.176022+0000","last_clean":"2024-02-05T08:26:37.176022+0000","last_became_active":"2024-02-05T08:25:49.139935+0000","last_became_peered":"2024-02-05T08:25:49.139935+0000","last_unstale":"2024-02-05T08:26:37.176022+0000","last_undegraded":"2024-02-05T08:26:37.176022+0000","last_fullsized":"2024-02-05T08:26:37.176022+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:33.338370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:26:33.338370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:00.251731+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":824,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.165438+0000","last_change":"2024-02-05T08:26:33.693805+0000","last_active":"2024-02-05T08:26:37.165438+0000","last_peered":"2024-02-05T08:26:37.165438+0000","last_clean":"2024-02-05T08:26:37.165438+0000","last_became_active":"2024-02-05T08:25:49.116678+0000","last_became_peered":"2024-02-05T08:25:49.116678+0000","last_unstale":"2024-02-05T08:26:37.165438+0000","last_undegraded":"2024-02-05T08:26:37.165438+0000","last_fullsized":"2024-02-05T08:26:37.165438+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:33.693680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:02.192889+0000","last_clean_scrub_stamp":"2024-02-05T08:26:33.693680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:56.506419+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":316,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162816+0000","last_change":"2024-02-05T08:26:17.046984+0000","last_active":"2024-02-05T08:26:37.162816+0000","last_peered":"2024-02-05T08:26:37.162816+0000","last_clean":"2024-02-05T08:26:37.162816+0000","last_became_active":"2024-02-05T08:26:17.046521+0000","last_became_peered":"2024-02-05T08:26:17.046521+0000","last_unstale":"2024-02-05T08:26:37.162816+0000","last_undegraded":"2024-02-05T08:26:37.162816+0000","last_fullsized":"2024-02-05T08:26:37.162816+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:40.813940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_clean_scrub_stamp":"2024-02-05T08:23:40.813940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"397'25","reported_seq":908,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.161901+0000","last_change":"2024-02-05T08:25:49.173163+0000","last_active":"2024-02-05T08:26:37.161901+0000","last_peered":"2024-02-05T08:26:37.161901+0000","last_clean":"2024-02-05T08:26:37.161901+0000","last_became_active":"2024-02-05T08:25:49.126860+0000","last_became_peered":"2024-02-05T08:25:49.126860+0000","last_unstale":"2024-02-05T08:26:37.161901+0000","last_undegraded":"2024-02-05T08:26:37.161901+0000","last_fullsized":"2024-02-05T08:26:37.161901+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-02-05T08:25:22.167194+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:25:22.167194+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:46.393830+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":806,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.165688+0000","last_change":"2024-02-05T08:25:49.013590+0000","last_active":"2024-02-05T08:26:37.165688+0000","last_peered":"2024-02-05T08:26:37.165688+0000","last_clean":"2024-02-05T08:26:37.165688+0000","last_became_active":"2024-02-05T08:23:07.932548+0000","last_became_peered":"2024-02-05T08:23:07.932548+0000","last_unstale":"2024-02-05T08:26:37.165688+0000","last_undegraded":"2024-02-05T08:26:37.165688+0000","last_fullsized":"2024-02-05T08:26:37.165688+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:49.013492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:25:49.013492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:06.032519+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":806,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.173602+0000","last_change":"2024-02-05T08:25:32.874890+0000","last_active":"2024-02-05T08:26:37.173602+0000","last_peered":"2024-02-05T08:26:37.173602+0000","last_clean":"2024-02-05T08:26:37.173602+0000","last_became_active":"2024-02-05T08:23:07.939630+0000","last_became_peered":"2024-02-05T08:23:07.939630+0000","last_unstale":"2024-02-05T08:26:37.173602+0000","last_undegraded":"2024-02-05T08:26:37.173602+0000","last_fullsized":"2024-02-05T08:26:37.173602+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:32.874673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:25:32.874673+0000","last_clean_scrub_stamp":"2024-02-05T08:25:32.874673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":699,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162369+0000","last_change":"2024-02-05T08:26:17.044456+0000","last_active":"2024-02-05T08:26:37.162369+0000","last_peered":"2024-02-05T08:26:37.162369+0000","last_clean":"2024-02-05T08:26:37.162369+0000","last_became_active":"2024-02-05T08:26:17.044091+0000","last_became_peered":"2024-02-05T08:26:17.044091+0000","last_unstale":"2024-02-05T08:26:37.162369+0000","last_undegraded":"2024-02-05T08:26:37.162369+0000","last_fullsized":"2024-02-05T08:26:37.162369+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:32.886199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:46.458244+0000","last_clean_scrub_stamp":"2024-02-05T08:25:32.886199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"397'9","reported_seq":841,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162480+0000","last_change":"2024-02-05T08:26:31.419976+0000","last_active":"2024-02-05T08:26:37.162480+0000","last_peered":"2024-02-05T08:26:37.162480+0000","last_clean":"2024-02-05T08:26:37.162480+0000","last_became_active":"2024-02-05T08:23:07.938087+0000","last_became_peered":"2024-02-05T08:23:07.938087+0000","last_unstale":"2024-02-05T08:26:37.162480+0000","last_undegraded":"2024-02-05T08:26:37.162480+0000","last_fullsized":"2024-02-05T08:26:37.162480+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'9","last_scrub_stamp":"2024-02-05T08:26:31.419856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:26:31.419856+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:53.630507+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":806,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162170+0000","last_change":"2024-02-05T08:24:41.278446+0000","last_active":"2024-02-05T08:26:37.162170+0000","last_peered":"2024-02-05T08:26:37.162170+0000","last_clean":"2024-02-05T08:26:37.162170+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:26:37.162170+0000","last_undegraded":"2024-02-05T08:26:37.162170+0000","last_fullsized":"2024-02-05T08:26:37.162170+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:41.278322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:39.852038+0000","last_clean_scrub_stamp":"2024-02-05T08:24:41.278322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":706,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.164920+0000","last_change":"2024-02-05T08:25:49.136510+0000","last_active":"2024-02-05T08:26:35.164920+0000","last_peered":"2024-02-05T08:26:35.164920+0000","last_clean":"2024-02-05T08:26:35.164920+0000","last_became_active":"2024-02-05T08:25:49.135987+0000","last_became_peered":"2024-02-05T08:25:49.135987+0000","last_unstale":"2024-02-05T08:26:35.164920+0000","last_undegraded":"2024-02-05T08:26:35.164920+0000","last_fullsized":"2024-02-05T08:26:35.164920+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:45.270592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_clean_scrub_stamp":"2024-02-05T08:24:45.270592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":45,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176470+0000","last_change":"2024-02-05T08:25:49.126010+0000","last_active":"2024-02-05T08:26:37.176470+0000","last_peered":"2024-02-05T08:26:37.176470+0000","last_clean":"2024-02-05T08:26:37.176470+0000","last_became_active":"2024-02-05T08:25:49.124061+0000","last_became_peered":"2024-02-05T08:25:49.124061+0000","last_unstale":"2024-02-05T08:26:37.176470+0000","last_undegraded":"2024-02-05T08:26:37.176470+0000","last_fullsized":"2024-02-05T08:26:37.176470+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:24.117759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:04.290125+0000","last_clean_scrub_stamp":"2024-02-05T08:25:24.117759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:45.892471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"397'46","reported_seq":937,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.172536+0000","last_change":"2024-02-05T08:26:28.428771+0000","last_active":"2024-02-05T08:26:35.172536+0000","last_peered":"2024-02-05T08:26:35.172536+0000","last_clean":"2024-02-05T08:26:35.172536+0000","last_became_active":"2024-02-05T08:25:04.978627+0000","last_became_peered":"2024-02-05T08:25:04.978627+0000","last_unstale":"2024-02-05T08:26:35.172536+0000","last_undegraded":"2024-02-05T08:26:35.172536+0000","last_fullsized":"2024-02-05T08:26:35.172536+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-02-05T08:26:28.428633+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:26:28.428633+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:47.387096+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":296,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.178139+0000","last_change":"2024-02-05T08:24:26.342293+0000","last_active":"2024-02-05T08:26:37.178139+0000","last_peered":"2024-02-05T08:26:37.178139+0000","last_clean":"2024-02-05T08:26:37.178139+0000","last_became_active":"2024-02-05T08:18:34.642494+0000","last_became_peered":"2024-02-05T08:18:34.642494+0000","last_unstale":"2024-02-05T08:26:37.178139+0000","last_undegraded":"2024-02-05T08:26:37.178139+0000","last_fullsized":"2024-02-05T08:26:37.178139+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:26.342171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_clean_scrub_stamp":"2024-02-05T08:24:26.342171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":816,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162610+0000","last_change":"2024-02-05T08:26:22.412607+0000","last_active":"2024-02-05T08:26:37.162610+0000","last_peered":"2024-02-05T08:26:37.162610+0000","last_clean":"2024-02-05T08:26:37.162610+0000","last_became_active":"2024-02-05T08:18:34.638652+0000","last_became_peered":"2024-02-05T08:18:34.638652+0000","last_unstale":"2024-02-05T08:26:37.162610+0000","last_undegraded":"2024-02-05T08:26:37.162610+0000","last_fullsized":"2024-02-05T08:26:37.162610+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:22.412488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:26:22.412488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:44.026610+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":797,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162191+0000","last_change":"2024-02-05T08:24:22.154106+0000","last_active":"2024-02-05T08:26:37.162191+0000","last_peered":"2024-02-05T08:26:37.162191+0000","last_clean":"2024-02-05T08:26:37.162191+0000","last_became_active":"2024-02-05T08:23:07.937958+0000","last_became_peered":"2024-02-05T08:23:07.937958+0000","last_unstale":"2024-02-05T08:26:37.162191+0000","last_undegraded":"2024-02-05T08:26:37.162191+0000","last_fullsized":"2024-02-05T08:26:37.162191+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:22.153964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:24:22.153964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"397'29","reported_seq":310,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176990+0000","last_change":"2024-02-05T08:23:25.937497+0000","last_active":"2024-02-05T08:26:37.176990+0000","last_peered":"2024-02-05T08:26:37.176990+0000","last_clean":"2024-02-05T08:26:37.176990+0000","last_became_active":"2024-02-05T08:23:07.925083+0000","last_became_peered":"2024-02-05T08:23:07.925083+0000","last_unstale":"2024-02-05T08:26:37.176990+0000","last_undegraded":"2024-02-05T08:26:37.176990+0000","last_fullsized":"2024-02-05T08:26:37.176990+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'23","last_scrub_stamp":"2024-02-05T08:23:25.935815+0000","last_deep_scrub":"90'17","last_deep_scrub_stamp":"2024-02-05T08:05:58.285304+0000","last_clean_scrub_stamp":"2024-02-05T08:23:25.935815+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":811,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.174295+0000","last_change":"2024-02-05T08:26:27.597333+0000","last_active":"2024-02-05T08:26:37.174295+0000","last_peered":"2024-02-05T08:26:37.174295+0000","last_clean":"2024-02-05T08:26:37.174295+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:26:37.174295+0000","last_undegraded":"2024-02-05T08:26:37.174295+0000","last_fullsized":"2024-02-05T08:26:37.174295+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:26:27.597213+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:26:27.597213+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:31.538784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":733,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176307+0000","last_change":"2024-02-05T08:25:44.460996+0000","last_active":"2024-02-05T08:26:37.176307+0000","last_peered":"2024-02-05T08:26:37.176307+0000","last_clean":"2024-02-05T08:26:37.176307+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:26:37.176307+0000","last_undegraded":"2024-02-05T08:26:37.176307+0000","last_fullsized":"2024-02-05T08:26:37.176307+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:25:44.460873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:25:44.460873+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:06.290315+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":729,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.164061+0000","last_change":"2024-02-05T08:24:24.293664+0000","last_active":"2024-02-05T08:26:35.164061+0000","last_peered":"2024-02-05T08:26:35.164061+0000","last_clean":"2024-02-05T08:26:35.164061+0000","last_became_active":"2024-02-05T08:23:07.940135+0000","last_became_peered":"2024-02-05T08:23:07.940135+0000","last_unstale":"2024-02-05T08:26:35.164061+0000","last_undegraded":"2024-02-05T08:26:35.164061+0000","last_fullsized":"2024-02-05T08:26:35.164061+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:24:24.293517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:24.293517+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"397'30","reported_seq":65,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.311218+0000","last_change":"2024-02-05T08:26:38.311275+0000","last_active":"2024-02-05T08:26:38.311218+0000","last_peered":"2024-02-05T08:26:38.311218+0000","last_clean":"2024-02-05T08:26:38.311218+0000","last_became_active":"2024-02-05T08:25:49.130322+0000","last_became_peered":"2024-02-05T08:25:49.130322+0000","last_unstale":"2024-02-05T08:26:38.311218+0000","last_undegraded":"2024-02-05T08:26:38.311218+0000","last_fullsized":"2024-02-05T08:26:38.311218+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-02-05T08:26:38.311129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:26:38.311129+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:04.640458+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":663,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.165038+0000","last_change":"2024-02-05T08:25:43.425669+0000","last_active":"2024-02-05T08:26:35.165038+0000","last_peered":"2024-02-05T08:26:35.165038+0000","last_clean":"2024-02-05T08:26:35.165038+0000","last_became_active":"2024-02-05T08:23:07.941095+0000","last_became_peered":"2024-02-05T08:23:07.941095+0000","last_unstale":"2024-02-05T08:26:35.165038+0000","last_undegraded":"2024-02-05T08:26:35.165038+0000","last_fullsized":"2024-02-05T08:26:35.165038+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:43.425538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:25:43.425538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:52.752683+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":53,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.178895+0000","last_change":"2024-02-05T08:26:30.280317+0000","last_active":"2024-02-05T08:26:37.178895+0000","last_peered":"2024-02-05T08:26:37.178895+0000","last_clean":"2024-02-05T08:26:37.178895+0000","last_became_active":"2024-02-05T08:25:49.126916+0000","last_became_peered":"2024-02-05T08:25:49.126916+0000","last_unstale":"2024-02-05T08:26:37.178895+0000","last_undegraded":"2024-02-05T08:26:37.178895+0000","last_fullsized":"2024-02-05T08:26:37.178895+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:30.280164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:19.107555+0000","last_clean_scrub_stamp":"2024-02-05T08:26:30.280164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:50.384024+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":315,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176047+0000","last_change":"2024-02-05T08:25:49.151621+0000","last_active":"2024-02-05T08:26:37.176047+0000","last_peered":"2024-02-05T08:26:37.176047+0000","last_clean":"2024-02-05T08:26:37.176047+0000","last_became_active":"2024-02-05T08:25:49.151068+0000","last_became_peered":"2024-02-05T08:25:49.151068+0000","last_unstale":"2024-02-05T08:26:37.176047+0000","last_undegraded":"2024-02-05T08:26:37.176047+0000","last_fullsized":"2024-02-05T08:26:37.176047+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:29.157028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:25:29.157028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:48.889105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"397'27","reported_seq":587,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176153+0000","last_change":"2024-02-05T08:26:29.787501+0000","last_active":"2024-02-05T08:26:37.176153+0000","last_peered":"2024-02-05T08:26:37.176153+0000","last_clean":"2024-02-05T08:26:37.176153+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:26:37.176153+0000","last_undegraded":"2024-02-05T08:26:37.176153+0000","last_fullsized":"2024-02-05T08:26:37.176153+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'27","last_scrub_stamp":"2024-02-05T08:26:29.787375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:26:29.787375+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:43.796155+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":786,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.173736+0000","last_change":"2024-02-05T08:24:16.984051+0000","last_active":"2024-02-05T08:26:37.173736+0000","last_peered":"2024-02-05T08:26:37.173736+0000","last_clean":"2024-02-05T08:26:37.173736+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:26:37.173736+0000","last_undegraded":"2024-02-05T08:26:37.173736+0000","last_fullsized":"2024-02-05T08:26:37.173736+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:16.983946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:14.159628+0000","last_clean_scrub_stamp":"2024-02-05T08:24:16.983946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":554,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175757+0000","last_change":"2024-02-05T08:26:33.830902+0000","last_active":"2024-02-05T08:26:37.175757+0000","last_peered":"2024-02-05T08:26:37.175757+0000","last_clean":"2024-02-05T08:26:37.175757+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:26:37.175757+0000","last_undegraded":"2024-02-05T08:26:37.175757+0000","last_fullsized":"2024-02-05T08:26:37.175757+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:33.830797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:26:33.830797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:57.130362+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":45,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175995+0000","last_change":"2024-02-05T08:25:49.143129+0000","last_active":"2024-02-05T08:26:37.175995+0000","last_peered":"2024-02-05T08:26:37.175995+0000","last_clean":"2024-02-05T08:26:37.175995+0000","last_became_active":"2024-02-05T08:25:49.142594+0000","last_became_peered":"2024-02-05T08:25:49.142594+0000","last_unstale":"2024-02-05T08:26:37.175995+0000","last_undegraded":"2024-02-05T08:26:37.175995+0000","last_fullsized":"2024-02-05T08:26:37.175995+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:31.862841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:31.862841+0000","last_clean_scrub_stamp":"2024-02-05T08:24:31.862841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"397'12","reported_seq":309,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.179395+0000","last_change":"2024-02-05T08:23:31.939827+0000","last_active":"2024-02-05T08:26:37.179395+0000","last_peered":"2024-02-05T08:26:37.179395+0000","last_clean":"2024-02-05T08:26:37.179395+0000","last_became_active":"2024-02-05T08:23:07.937911+0000","last_became_peered":"2024-02-05T08:23:07.937911+0000","last_unstale":"2024-02-05T08:26:37.179395+0000","last_undegraded":"2024-02-05T08:26:37.179395+0000","last_fullsized":"2024-02-05T08:26:37.179395+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'8","last_scrub_stamp":"2024-02-05T08:23:31.939578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:23:31.939578+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":316,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.211153+0000","last_change":"2024-02-05T08:25:49.143961+0000","last_active":"2024-02-05T08:26:37.211153+0000","last_peered":"2024-02-05T08:26:37.211153+0000","last_clean":"2024-02-05T08:26:37.211153+0000","last_became_active":"2024-02-05T08:25:49.139211+0000","last_became_peered":"2024-02-05T08:25:49.139211+0000","last_unstale":"2024-02-05T08:26:37.211153+0000","last_undegraded":"2024-02-05T08:26:37.211153+0000","last_fullsized":"2024-02-05T08:26:37.211153+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:38.306795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:25:38.306795+0000","last_clean_scrub_stamp":"2024-02-05T08:25:38.306795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:50.598475+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":818,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.174240+0000","last_change":"2024-02-05T08:25:17.893419+0000","last_active":"2024-02-05T08:26:37.174240+0000","last_peered":"2024-02-05T08:26:37.174240+0000","last_clean":"2024-02-05T08:26:37.174240+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:26:37.174240+0000","last_undegraded":"2024-02-05T08:26:37.174240+0000","last_fullsized":"2024-02-05T08:26:37.174240+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:17.893349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:54.156080+0000","last_clean_scrub_stamp":"2024-02-05T08:25:17.893349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":814,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.165816+0000","last_change":"2024-02-05T08:25:34.929143+0000","last_active":"2024-02-05T08:26:37.165816+0000","last_peered":"2024-02-05T08:26:37.165816+0000","last_clean":"2024-02-05T08:26:37.165816+0000","last_became_active":"2024-02-05T08:23:07.932374+0000","last_became_peered":"2024-02-05T08:23:07.932374+0000","last_unstale":"2024-02-05T08:26:37.165816+0000","last_undegraded":"2024-02-05T08:26:37.165816+0000","last_fullsized":"2024-02-05T08:26:37.165816+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:34.929019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:25:34.929019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:56.051390+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"397'32","reported_seq":320,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.178926+0000","last_change":"2024-02-05T08:24:28.130743+0000","last_active":"2024-02-05T08:26:37.178926+0000","last_peered":"2024-02-05T08:26:37.178926+0000","last_clean":"2024-02-05T08:26:37.178926+0000","last_became_active":"2024-02-05T08:18:34.631912+0000","last_became_peered":"2024-02-05T08:18:34.631912+0000","last_unstale":"2024-02-05T08:26:37.178926+0000","last_undegraded":"2024-02-05T08:26:37.178926+0000","last_fullsized":"2024-02-05T08:26:37.178926+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:24:28.130616+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:24:28.130616+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175702+0000","last_change":"2024-02-05T08:25:49.142355+0000","last_active":"2024-02-05T08:26:37.175702+0000","last_peered":"2024-02-05T08:26:37.175702+0000","last_clean":"2024-02-05T08:26:37.175702+0000","last_became_active":"2024-02-05T08:25:49.142057+0000","last_became_peered":"2024-02-05T08:25:49.142057+0000","last_unstale":"2024-02-05T08:26:37.175702+0000","last_undegraded":"2024-02-05T08:26:37.175702+0000","last_fullsized":"2024-02-05T08:26:37.175702+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:37.823577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:24:37.823577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":315,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.211072+0000","last_change":"2024-02-05T08:25:49.139465+0000","last_active":"2024-02-05T08:26:37.211072+0000","last_peered":"2024-02-05T08:26:37.211072+0000","last_clean":"2024-02-05T08:26:37.211072+0000","last_became_active":"2024-02-05T08:25:49.138377+0000","last_became_peered":"2024-02-05T08:25:49.138377+0000","last_unstale":"2024-02-05T08:26:37.211072+0000","last_undegraded":"2024-02-05T08:26:37.211072+0000","last_fullsized":"2024-02-05T08:26:37.211072+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:33.208642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:25:33.208642+0000","last_clean_scrub_stamp":"2024-02-05T08:25:33.208642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":818,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.174430+0000","last_change":"2024-02-05T08:24:33.916510+0000","last_active":"2024-02-05T08:26:37.174430+0000","last_peered":"2024-02-05T08:26:37.174430+0000","last_clean":"2024-02-05T08:26:37.174430+0000","last_became_active":"2024-02-05T08:18:34.642387+0000","last_became_peered":"2024-02-05T08:18:34.642387+0000","last_unstale":"2024-02-05T08:26:37.174430+0000","last_undegraded":"2024-02-05T08:26:37.174430+0000","last_fullsized":"2024-02-05T08:26:37.174430+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:33.916402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:33.916402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"397'25","reported_seq":490,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.165320+0000","last_change":"2024-02-05T08:26:24.882943+0000","last_active":"2024-02-05T08:26:37.165320+0000","last_peered":"2024-02-05T08:26:37.165320+0000","last_clean":"2024-02-05T08:26:37.165320+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:26:37.165320+0000","last_undegraded":"2024-02-05T08:26:37.165320+0000","last_fullsized":"2024-02-05T08:26:37.165320+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-02-05T08:26:24.882810+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_clean_scrub_stamp":"2024-02-05T08:26:24.882810+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:50.937692+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":324,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.178794+0000","last_change":"2024-02-05T08:26:36.398095+0000","last_active":"2024-02-05T08:26:37.178794+0000","last_peered":"2024-02-05T08:26:37.178794+0000","last_clean":"2024-02-05T08:26:37.178794+0000","last_became_active":"2024-02-05T08:25:49.140007+0000","last_became_peered":"2024-02-05T08:25:49.140007+0000","last_unstale":"2024-02-05T08:26:37.178794+0000","last_undegraded":"2024-02-05T08:26:37.178794+0000","last_fullsized":"2024-02-05T08:26:37.178794+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:36.397870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:26:36.397870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:50.649340+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":714,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.164269+0000","last_change":"2024-02-05T08:26:33.367071+0000","last_active":"2024-02-05T08:26:35.164269+0000","last_peered":"2024-02-05T08:26:35.164269+0000","last_clean":"2024-02-05T08:26:35.164269+0000","last_became_active":"2024-02-05T08:23:07.936401+0000","last_became_peered":"2024-02-05T08:23:07.936401+0000","last_unstale":"2024-02-05T08:26:35.164269+0000","last_undegraded":"2024-02-05T08:26:35.164269+0000","last_fullsized":"2024-02-05T08:26:35.164269+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:33.366951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:26:33.366951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:00.766888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":825,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162748+0000","last_change":"2024-02-05T08:25:21.174248+0000","last_active":"2024-02-05T08:26:37.162748+0000","last_peered":"2024-02-05T08:26:37.162748+0000","last_clean":"2024-02-05T08:26:37.162748+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:26:37.162748+0000","last_undegraded":"2024-02-05T08:26:37.162748+0000","last_fullsized":"2024-02-05T08:26:37.162748+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:21.173883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:25:21.173883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:51.161556+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"397'32","reported_seq":886,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162618+0000","last_change":"2024-02-05T08:24:34.133027+0000","last_active":"2024-02-05T08:26:37.162618+0000","last_peered":"2024-02-05T08:26:37.162618+0000","last_clean":"2024-02-05T08:26:37.162618+0000","last_became_active":"2024-02-05T08:23:07.938523+0000","last_became_peered":"2024-02-05T08:23:07.938523+0000","last_unstale":"2024-02-05T08:26:37.162618+0000","last_undegraded":"2024-02-05T08:26:37.162618+0000","last_fullsized":"2024-02-05T08:26:37.162618+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'24","last_scrub_stamp":"2024-02-05T08:24:34.132899+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:24:34.132899+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":45,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.178968+0000","last_change":"2024-02-05T08:25:49.131566+0000","last_active":"2024-02-05T08:26:37.178968+0000","last_peered":"2024-02-05T08:26:37.178968+0000","last_clean":"2024-02-05T08:26:37.178968+0000","last_became_active":"2024-02-05T08:25:49.130779+0000","last_became_peered":"2024-02-05T08:25:49.130779+0000","last_unstale":"2024-02-05T08:26:37.178968+0000","last_undegraded":"2024-02-05T08:26:37.178968+0000","last_fullsized":"2024-02-05T08:26:37.178968+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:42.905809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:33.061104+0000","last_clean_scrub_stamp":"2024-02-05T08:25:42.905809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:03.870761+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"407'15","reported_seq":1110,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.166093+0000","last_change":"2024-02-05T08:26:27.818959+0000","last_active":"2024-02-05T08:26:37.166093+0000","last_peered":"2024-02-05T08:26:37.166093+0000","last_clean":"2024-02-05T08:26:37.166093+0000","last_became_active":"2024-02-05T08:25:49.135603+0000","last_became_peered":"2024-02-05T08:25:49.135603+0000","last_unstale":"2024-02-05T08:26:37.166093+0000","last_undegraded":"2024-02-05T08:26:37.166093+0000","last_fullsized":"2024-02-05T08:26:37.166093+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'15","last_scrub_stamp":"2024-02-05T08:26:27.818832+0000","last_deep_scrub":"391'14","last_deep_scrub_stamp":"2024-02-05T08:24:33.873828+0000","last_clean_scrub_stamp":"2024-02-05T08:26:27.818832+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:57.169673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":814,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.174096+0000","last_change":"2024-02-05T08:26:32.523132+0000","last_active":"2024-02-05T08:26:37.174096+0000","last_peered":"2024-02-05T08:26:37.174096+0000","last_clean":"2024-02-05T08:26:37.174096+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:26:37.174096+0000","last_undegraded":"2024-02-05T08:26:37.174096+0000","last_fullsized":"2024-02-05T08:26:37.174096+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:32.523033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:26:32.523033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:41.042523+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"397'35","reported_seq":324,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176409+0000","last_change":"2024-02-05T08:24:19.039429+0000","last_active":"2024-02-05T08:26:37.176409+0000","last_peered":"2024-02-05T08:26:37.176409+0000","last_clean":"2024-02-05T08:26:37.176409+0000","last_became_active":"2024-02-05T08:23:07.928966+0000","last_became_peered":"2024-02-05T08:23:07.928966+0000","last_unstale":"2024-02-05T08:26:37.176409+0000","last_undegraded":"2024-02-05T08:26:37.176409+0000","last_fullsized":"2024-02-05T08:26:37.176409+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-02-05T08:24:19.039293+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:24:19.039293+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":788,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.164043+0000","last_change":"2024-02-05T08:24:37.328751+0000","last_active":"2024-02-05T08:26:37.164043+0000","last_peered":"2024-02-05T08:26:37.164043+0000","last_clean":"2024-02-05T08:26:37.164043+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:26:37.164043+0000","last_undegraded":"2024-02-05T08:26:37.164043+0000","last_fullsized":"2024-02-05T08:26:37.164043+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:37.328629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:28.519149+0000","last_clean_scrub_stamp":"2024-02-05T08:24:37.328629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"295'10","reported_seq":398,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176676+0000","last_change":"2024-02-05T08:25:27.159721+0000","last_active":"2024-02-05T08:26:37.176676+0000","last_peered":"2024-02-05T08:26:37.176676+0000","last_clean":"2024-02-05T08:26:37.176676+0000","last_became_active":"2024-02-05T08:18:34.644517+0000","last_became_peered":"2024-02-05T08:18:34.644517+0000","last_unstale":"2024-02-05T08:26:37.176676+0000","last_undegraded":"2024-02-05T08:26:37.176676+0000","last_fullsized":"2024-02-05T08:26:37.176676+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'10","last_scrub_stamp":"2024-02-05T08:25:27.159518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:25:27.159518+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":45,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.178018+0000","last_change":"2024-02-05T08:25:49.131558+0000","last_active":"2024-02-05T08:26:37.178018+0000","last_peered":"2024-02-05T08:26:37.178018+0000","last_clean":"2024-02-05T08:26:37.178018+0000","last_became_active":"2024-02-05T08:25:49.126790+0000","last_became_peered":"2024-02-05T08:25:49.126790+0000","last_unstale":"2024-02-05T08:26:37.178018+0000","last_undegraded":"2024-02-05T08:26:37.178018+0000","last_fullsized":"2024-02-05T08:26:37.178018+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:44.889942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:25:44.889942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:48.456182+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"397'21","reported_seq":844,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.174916+0000","last_change":"2024-02-05T08:26:17.049726+0000","last_active":"2024-02-05T08:26:37.174916+0000","last_peered":"2024-02-05T08:26:37.174916+0000","last_clean":"2024-02-05T08:26:37.174916+0000","last_became_active":"2024-02-05T08:26:17.049423+0000","last_became_peered":"2024-02-05T08:26:17.049423+0000","last_unstale":"2024-02-05T08:26:37.174916+0000","last_undegraded":"2024-02-05T08:26:37.174916+0000","last_fullsized":"2024-02-05T08:26:37.174916+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-02-05T08:25:30.881441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:25:30.881441+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:56.736589+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":833,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.590348+0000","last_change":"2024-02-05T08:26:38.590395+0000","last_active":"2024-02-05T08:26:38.590348+0000","last_peered":"2024-02-05T08:26:38.590348+0000","last_clean":"2024-02-05T08:26:38.590348+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:26:38.590348+0000","last_undegraded":"2024-02-05T08:26:38.590348+0000","last_fullsized":"2024-02-05T08:26:38.590348+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:38.587082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:26:38.587082+0000","last_clean_scrub_stamp":"2024-02-05T08:26:38.587082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:45.617185+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":297,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.178061+0000","last_change":"2024-02-05T08:23:42.025667+0000","last_active":"2024-02-05T08:26:37.178061+0000","last_peered":"2024-02-05T08:26:37.178061+0000","last_clean":"2024-02-05T08:26:37.178061+0000","last_became_active":"2024-02-05T08:23:07.928375+0000","last_became_peered":"2024-02-05T08:23:07.928375+0000","last_unstale":"2024-02-05T08:26:37.178061+0000","last_undegraded":"2024-02-05T08:26:37.178061+0000","last_fullsized":"2024-02-05T08:26:37.178061+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:42.025471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:42.025471+0000","last_clean_scrub_stamp":"2024-02-05T08:23:42.025471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":818,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.173615+0000","last_change":"2024-02-05T08:25:26.855295+0000","last_active":"2024-02-05T08:26:37.173615+0000","last_peered":"2024-02-05T08:26:37.173615+0000","last_clean":"2024-02-05T08:26:37.173615+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:26:37.173615+0000","last_undegraded":"2024-02-05T08:26:37.173615+0000","last_fullsized":"2024-02-05T08:26:37.173615+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:26.855155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:25:26.855155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"397'25","reported_seq":55,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.178814+0000","last_change":"2024-02-05T08:25:49.164977+0000","last_active":"2024-02-05T08:26:37.178814+0000","last_peered":"2024-02-05T08:26:37.178814+0000","last_clean":"2024-02-05T08:26:37.178814+0000","last_became_active":"2024-02-05T08:25:49.129856+0000","last_became_peered":"2024-02-05T08:25:49.129856+0000","last_unstale":"2024-02-05T08:26:37.178814+0000","last_undegraded":"2024-02-05T08:26:37.178814+0000","last_fullsized":"2024-02-05T08:26:37.178814+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"278'19","last_scrub_stamp":"2024-02-05T08:24:45.102570+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:24:45.102570+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":806,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.174338+0000","last_change":"2024-02-05T08:24:45.975121+0000","last_active":"2024-02-05T08:26:37.174338+0000","last_peered":"2024-02-05T08:26:37.174338+0000","last_clean":"2024-02-05T08:26:37.174338+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:26:37.174338+0000","last_undegraded":"2024-02-05T08:26:37.174338+0000","last_fullsized":"2024-02-05T08:26:37.174338+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:45.974690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:45.974690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":818,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.165961+0000","last_change":"2024-02-05T08:26:21.853247+0000","last_active":"2024-02-05T08:26:37.165961+0000","last_peered":"2024-02-05T08:26:37.165961+0000","last_clean":"2024-02-05T08:26:37.165961+0000","last_became_active":"2024-02-05T08:23:07.932741+0000","last_became_peered":"2024-02-05T08:23:07.932741+0000","last_unstale":"2024-02-05T08:26:37.165961+0000","last_undegraded":"2024-02-05T08:26:37.165961+0000","last_fullsized":"2024-02-05T08:26:37.165961+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:21.853135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:26:21.853135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:30.512560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":563,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176225+0000","last_change":"2024-02-05T08:25:49.126994+0000","last_active":"2024-02-05T08:26:37.176225+0000","last_peered":"2024-02-05T08:26:37.176225+0000","last_clean":"2024-02-05T08:26:37.176225+0000","last_became_active":"2024-02-05T08:25:49.126639+0000","last_became_peered":"2024-02-05T08:25:49.126639+0000","last_unstale":"2024-02-05T08:26:37.176225+0000","last_undegraded":"2024-02-05T08:26:37.176225+0000","last_fullsized":"2024-02-05T08:26:37.176225+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:43.433498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:25:43.433498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:58.700594+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"397'21","reported_seq":306,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.177927+0000","last_change":"2024-02-05T08:23:26.922363+0000","last_active":"2024-02-05T08:26:37.177927+0000","last_peered":"2024-02-05T08:26:37.177927+0000","last_clean":"2024-02-05T08:26:37.177927+0000","last_became_active":"2024-02-05T08:23:07.927922+0000","last_became_peered":"2024-02-05T08:23:07.927922+0000","last_unstale":"2024-02-05T08:26:37.177927+0000","last_undegraded":"2024-02-05T08:26:37.177927+0000","last_fullsized":"2024-02-05T08:26:37.177927+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-02-05T08:23:26.922227+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:23:26.922227+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"424'16261","reported_seq":16449,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.648799+0000","last_change":"2024-02-05T08:25:29.435158+0000","last_active":"2024-02-05T08:26:38.648799+0000","last_peered":"2024-02-05T08:26:38.648799+0000","last_clean":"2024-02-05T08:26:38.648799+0000","last_became_active":"2024-02-05T08:23:08.913227+0000","last_became_peered":"2024-02-05T08:23:08.913227+0000","last_unstale":"2024-02-05T08:26:38.648799+0000","last_undegraded":"2024-02-05T08:26:38.648799+0000","last_fullsized":"2024-02-05T08:26:38.648799+0000","mapping_epoch":367,"log_start":"402'11000","ondisk_log_start":"402'11000","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'11786","last_scrub_stamp":"2024-02-05T08:25:29.434992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:25:29.434992+0000","objects_scrubbed":5182,"log_size":5261,"log_dups_size":0,"ondisk_log_size":5261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:40.000540+0000","scrub_duration":1537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42926080,"num_objects":655,"num_object_clones":0,"num_object_copies":1965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16261,"num_write_kb":541312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":816,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175526+0000","last_change":"2024-02-05T08:26:25.822142+0000","last_active":"2024-02-05T08:26:37.175526+0000","last_peered":"2024-02-05T08:26:37.175526+0000","last_clean":"2024-02-05T08:26:37.175526+0000","last_became_active":"2024-02-05T08:26:17.060002+0000","last_became_peered":"2024-02-05T08:26:17.060002+0000","last_unstale":"2024-02-05T08:26:37.175526+0000","last_undegraded":"2024-02-05T08:26:37.175526+0000","last_fullsized":"2024-02-05T08:26:37.175526+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:25.822020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:26:25.822020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:26.639024+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":854,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.174759+0000","last_change":"2024-02-05T08:25:49.162506+0000","last_active":"2024-02-05T08:26:37.174759+0000","last_peered":"2024-02-05T08:26:37.174759+0000","last_clean":"2024-02-05T08:26:37.174759+0000","last_became_active":"2024-02-05T08:25:49.136088+0000","last_became_peered":"2024-02-05T08:25:49.136088+0000","last_unstale":"2024-02-05T08:26:37.174759+0000","last_undegraded":"2024-02-05T08:26:37.174759+0000","last_fullsized":"2024-02-05T08:26:37.174759+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:25:47.931759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:25:47.931759+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:55.409917+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"397'46","reported_seq":922,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.165194+0000","last_change":"2024-02-05T08:26:28.771081+0000","last_active":"2024-02-05T08:26:37.165194+0000","last_peered":"2024-02-05T08:26:37.165194+0000","last_clean":"2024-02-05T08:26:37.165194+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:26:37.165194+0000","last_undegraded":"2024-02-05T08:26:37.165194+0000","last_fullsized":"2024-02-05T08:26:37.165194+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-02-05T08:26:28.770992+0000","last_deep_scrub":"262'34","last_deep_scrub_stamp":"2024-02-05T08:19:34.364738+0000","last_clean_scrub_stamp":"2024-02-05T08:26:28.770992+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:48.412446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":1092,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.173931+0000","last_change":"2024-02-05T08:24:26.044610+0000","last_active":"2024-02-05T08:26:37.173931+0000","last_peered":"2024-02-05T08:26:37.173931+0000","last_clean":"2024-02-05T08:26:37.173931+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:26:37.173931+0000","last_undegraded":"2024-02-05T08:26:37.173931+0000","last_fullsized":"2024-02-05T08:26:37.173931+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:24:26.044511+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_clean_scrub_stamp":"2024-02-05T08:24:26.044511+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"424'16092","reported_seq":21618,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:39.443854+0000","last_change":"2024-02-05T08:26:29.952270+0000","last_active":"2024-02-05T08:26:39.443854+0000","last_peered":"2024-02-05T08:26:39.443854+0000","last_clean":"2024-02-05T08:26:39.443854+0000","last_became_active":"2024-02-05T08:26:18.075027+0000","last_became_peered":"2024-02-05T08:26:18.075027+0000","last_unstale":"2024-02-05T08:26:39.443854+0000","last_undegraded":"2024-02-05T08:26:39.443854+0000","last_fullsized":"2024-02-05T08:26:39.443854+0000","mapping_epoch":417,"log_start":"401'10444","ondisk_log_start":"401'10444","created":367,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'15418","last_scrub_stamp":"2024-02-05T08:26:29.951938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:26:29.951938+0000","objects_scrubbed":1328,"log_size":5648,"log_dups_size":0,"ondisk_log_size":5648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:36.465086+0000","scrub_duration":656,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40894464,"num_objects":624,"num_object_clones":0,"num_object_copies":1872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16092,"num_write_kb":534912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2639,"num_bytes_recovered":172883968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":289,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.211328+0000","last_change":"2024-02-05T08:23:29.086552+0000","last_active":"2024-02-05T08:26:37.211328+0000","last_peered":"2024-02-05T08:26:37.211328+0000","last_clean":"2024-02-05T08:26:37.211328+0000","last_became_active":"2024-02-05T08:23:07.929942+0000","last_became_peered":"2024-02-05T08:23:07.929942+0000","last_unstale":"2024-02-05T08:26:37.211328+0000","last_undegraded":"2024-02-05T08:26:37.211328+0000","last_fullsized":"2024-02-05T08:26:37.211328+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:29.086293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.086293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":815,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.174547+0000","last_change":"2024-02-05T08:26:26.567831+0000","last_active":"2024-02-05T08:26:37.174547+0000","last_peered":"2024-02-05T08:26:37.174547+0000","last_clean":"2024-02-05T08:26:37.174547+0000","last_became_active":"2024-02-05T08:23:07.937863+0000","last_became_peered":"2024-02-05T08:23:07.937863+0000","last_unstale":"2024-02-05T08:26:37.174547+0000","last_undegraded":"2024-02-05T08:26:37.174547+0000","last_fullsized":"2024-02-05T08:26:37.174547+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:26.567754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:26:26.567754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:39.059132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"397'24","reported_seq":880,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.462812+0000","last_change":"2024-02-05T08:26:37.462860+0000","last_active":"2024-02-05T08:26:37.462812+0000","last_peered":"2024-02-05T08:26:37.462812+0000","last_clean":"2024-02-05T08:26:37.462812+0000","last_became_active":"2024-02-05T08:23:07.929900+0000","last_became_peered":"2024-02-05T08:23:07.929900+0000","last_unstale":"2024-02-05T08:26:37.462812+0000","last_undegraded":"2024-02-05T08:26:37.462812+0000","last_fullsized":"2024-02-05T08:26:37.462812+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'24","last_scrub_stamp":"2024-02-05T08:26:37.462723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:26:37.462723+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:48.764660+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":53,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175872+0000","last_change":"2024-02-05T08:26:34.233326+0000","last_active":"2024-02-05T08:26:37.175872+0000","last_peered":"2024-02-05T08:26:37.175872+0000","last_clean":"2024-02-05T08:26:37.175872+0000","last_became_active":"2024-02-05T08:25:49.139202+0000","last_became_peered":"2024-02-05T08:25:49.139202+0000","last_unstale":"2024-02-05T08:26:37.175872+0000","last_undegraded":"2024-02-05T08:26:37.175872+0000","last_fullsized":"2024-02-05T08:26:37.175872+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:34.233208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:26:34.233208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:49.831252+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"424'16231","reported_seq":16415,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.497187+0000","last_change":"2024-02-05T08:25:30.559856+0000","last_active":"2024-02-05T08:26:38.497187+0000","last_peered":"2024-02-05T08:26:38.497187+0000","last_clean":"2024-02-05T08:26:38.497187+0000","last_became_active":"2024-02-05T08:23:08.911591+0000","last_became_peered":"2024-02-05T08:23:08.911591+0000","last_unstale":"2024-02-05T08:26:38.497187+0000","last_undegraded":"2024-02-05T08:26:38.497187+0000","last_fullsized":"2024-02-05T08:26:38.497187+0000","mapping_epoch":367,"log_start":"402'11300","ondisk_log_start":"402'11300","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'11984","last_scrub_stamp":"2024-02-05T08:25:30.559622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:25:30.559622+0000","objects_scrubbed":4998,"log_size":4931,"log_dups_size":0,"ondisk_log_size":4931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:49.518440+0000","scrub_duration":1717,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44761088,"num_objects":683,"num_object_clones":0,"num_object_copies":2049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16231,"num_write_kb":541248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":807,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.173333+0000","last_change":"2024-02-05T08:24:27.030998+0000","last_active":"2024-02-05T08:26:37.173333+0000","last_peered":"2024-02-05T08:26:37.173333+0000","last_clean":"2024-02-05T08:26:37.173333+0000","last_became_active":"2024-02-05T08:18:34.642239+0000","last_became_peered":"2024-02-05T08:18:34.642239+0000","last_unstale":"2024-02-05T08:26:37.173333+0000","last_undegraded":"2024-02-05T08:26:37.173333+0000","last_fullsized":"2024-02-05T08:26:37.173333+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:27.030762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:27.030762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":833,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.168172+0000","last_change":"2024-02-05T08:25:49.119283+0000","last_active":"2024-02-05T08:26:37.168172+0000","last_peered":"2024-02-05T08:26:37.168172+0000","last_clean":"2024-02-05T08:26:37.168172+0000","last_became_active":"2024-02-05T08:25:49.118945+0000","last_became_peered":"2024-02-05T08:25:49.118945+0000","last_unstale":"2024-02-05T08:26:37.168172+0000","last_undegraded":"2024-02-05T08:26:37.168172+0000","last_fullsized":"2024-02-05T08:26:37.168172+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:38.903640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:25:38.903640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:46.718213+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":819,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.174835+0000","last_change":"2024-02-05T08:25:21.849368+0000","last_active":"2024-02-05T08:26:37.174835+0000","last_peered":"2024-02-05T08:26:37.174835+0000","last_clean":"2024-02-05T08:26:37.174835+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:26:37.174835+0000","last_undegraded":"2024-02-05T08:26:37.174835+0000","last_fullsized":"2024-02-05T08:26:37.174835+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:21.849247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_clean_scrub_stamp":"2024-02-05T08:25:21.849247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"397'51","reported_seq":627,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.929081+0000","last_change":"2024-02-05T08:26:37.929132+0000","last_active":"2024-02-05T08:26:37.929081+0000","last_peered":"2024-02-05T08:26:37.929081+0000","last_clean":"2024-02-05T08:26:37.929081+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:26:37.929081+0000","last_undegraded":"2024-02-05T08:26:37.929081+0000","last_fullsized":"2024-02-05T08:26:37.929081+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'51","last_scrub_stamp":"2024-02-05T08:26:37.929009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:26:37.929009+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:45.741249+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"424'16338","reported_seq":16521,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.494752+0000","last_change":"2024-02-05T08:25:36.494141+0000","last_active":"2024-02-05T08:26:38.494752+0000","last_peered":"2024-02-05T08:26:38.494752+0000","last_clean":"2024-02-05T08:26:38.494752+0000","last_became_active":"2024-02-05T08:23:08.911987+0000","last_became_peered":"2024-02-05T08:23:08.911987+0000","last_unstale":"2024-02-05T08:26:38.494752+0000","last_undegraded":"2024-02-05T08:26:38.494752+0000","last_fullsized":"2024-02-05T08:26:38.494752+0000","mapping_epoch":367,"log_start":"402'11400","ondisk_log_start":"402'11400","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'12413","last_scrub_stamp":"2024-02-05T08:25:36.493883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:25:36.493883+0000","objects_scrubbed":4707,"log_size":4938,"log_dups_size":0,"ondisk_log_size":4938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:01.755171+0000","scrub_duration":1607,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46530560,"num_objects":710,"num_object_clones":0,"num_object_copies":2130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16338,"num_write_kb":545536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":53,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.500603+0000","last_change":"2024-02-05T08:26:37.500670+0000","last_active":"2024-02-05T08:26:37.500603+0000","last_peered":"2024-02-05T08:26:37.500603+0000","last_clean":"2024-02-05T08:26:37.500603+0000","last_became_active":"2024-02-05T08:25:49.125995+0000","last_became_peered":"2024-02-05T08:25:49.125995+0000","last_unstale":"2024-02-05T08:26:37.500603+0000","last_undegraded":"2024-02-05T08:26:37.500603+0000","last_fullsized":"2024-02-05T08:26:37.500603+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:37.500508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_clean_scrub_stamp":"2024-02-05T08:26:37.500508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:49.351384+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":581,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162132+0000","last_change":"2024-02-05T08:25:04.987081+0000","last_active":"2024-02-05T08:26:37.162132+0000","last_peered":"2024-02-05T08:26:37.162132+0000","last_clean":"2024-02-05T08:26:37.162132+0000","last_became_active":"2024-02-05T08:25:04.986710+0000","last_became_peered":"2024-02-05T08:25:04.986710+0000","last_unstale":"2024-02-05T08:26:37.162132+0000","last_undegraded":"2024-02-05T08:26:37.162132+0000","last_fullsized":"2024-02-05T08:26:37.162132+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:33.808037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:24:33.808037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":561,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176218+0000","last_change":"2024-02-05T08:26:17.049126+0000","last_active":"2024-02-05T08:26:37.176218+0000","last_peered":"2024-02-05T08:26:37.176218+0000","last_clean":"2024-02-05T08:26:37.176218+0000","last_became_active":"2024-02-05T08:26:17.048142+0000","last_became_peered":"2024-02-05T08:26:17.048142+0000","last_unstale":"2024-02-05T08:26:37.176218+0000","last_undegraded":"2024-02-05T08:26:37.176218+0000","last_fullsized":"2024-02-05T08:26:37.176218+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:24.920931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:01.230038+0000","last_clean_scrub_stamp":"2024-02-05T08:25:24.920931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:42.206759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"397'25","reported_seq":55,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176379+0000","last_change":"2024-02-05T08:25:49.162214+0000","last_active":"2024-02-05T08:26:37.176379+0000","last_peered":"2024-02-05T08:26:37.176379+0000","last_clean":"2024-02-05T08:26:37.176379+0000","last_became_active":"2024-02-05T08:25:49.134815+0000","last_became_peered":"2024-02-05T08:25:49.134815+0000","last_unstale":"2024-02-05T08:26:37.176379+0000","last_undegraded":"2024-02-05T08:26:37.176379+0000","last_fullsized":"2024-02-05T08:26:37.176379+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'19","last_scrub_stamp":"2024-02-05T08:24:36.258646+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:24:36.258646+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"424'16005","reported_seq":21570,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.608413+0000","last_change":"2024-02-05T08:26:17.169604+0000","last_active":"2024-02-05T08:26:38.608413+0000","last_peered":"2024-02-05T08:26:38.608413+0000","last_clean":"2024-02-05T08:26:38.608413+0000","last_became_active":"2024-02-05T08:26:17.071398+0000","last_became_peered":"2024-02-05T08:26:17.071398+0000","last_unstale":"2024-02-05T08:26:38.608413+0000","last_undegraded":"2024-02-05T08:26:38.608413+0000","last_fullsized":"2024-02-05T08:26:38.608413+0000","mapping_epoch":416,"log_start":"402'11600","ondisk_log_start":"402'11600","created":367,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'13043","last_scrub_stamp":"2024-02-05T08:25:47.414442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:25:47.414442+0000","objects_scrubbed":3661,"log_size":4405,"log_dups_size":0,"ondisk_log_size":4405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:15.019947+0000","scrub_duration":1170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43450368,"num_objects":663,"num_object_clones":0,"num_object_copies":1989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16005,"num_write_kb":533376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2689,"num_bytes_recovered":176160768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"82'130","reported_seq":587,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.207337+0000","last_change":"2024-02-05T08:26:28.266498+0000","last_active":"2024-02-05T08:26:37.207337+0000","last_peered":"2024-02-05T08:26:37.207337+0000","last_clean":"2024-02-05T08:26:37.207337+0000","last_became_active":"2024-02-05T08:26:17.060725+0000","last_became_peered":"2024-02-05T08:26:17.060725+0000","last_unstale":"2024-02-05T08:26:37.207337+0000","last_undegraded":"2024-02-05T08:26:37.207337+0000","last_fullsized":"2024-02-05T08:26:37.207337+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:26:28.266132+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:26:28.266132+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:48.034268+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":525,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.161892+0000","last_change":"2024-02-05T08:26:26.435657+0000","last_active":"2024-02-05T08:26:37.161892+0000","last_peered":"2024-02-05T08:26:37.161892+0000","last_clean":"2024-02-05T08:26:37.161892+0000","last_became_active":"2024-02-05T08:23:07.936660+0000","last_became_peered":"2024-02-05T08:23:07.936660+0000","last_unstale":"2024-02-05T08:26:37.161892+0000","last_undegraded":"2024-02-05T08:26:37.161892+0000","last_fullsized":"2024-02-05T08:26:37.161892+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:26.435559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:26:26.435559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:44.579619+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":88,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.164517+0000","last_change":"2024-02-05T08:26:17.046544+0000","last_active":"2024-02-05T08:26:35.164517+0000","last_peered":"2024-02-05T08:26:35.164517+0000","last_clean":"2024-02-05T08:26:35.164517+0000","last_became_active":"2024-02-05T08:26:17.046055+0000","last_became_peered":"2024-02-05T08:26:17.046055+0000","last_unstale":"2024-02-05T08:26:35.164517+0000","last_undegraded":"2024-02-05T08:26:35.164517+0000","last_fullsized":"2024-02-05T08:26:35.164517+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:25:41.877014+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:05:52.503651+0000","last_clean_scrub_stamp":"2024-02-05T08:25:41.877014+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:11.434279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":834,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162264+0000","last_change":"2024-02-05T08:25:49.114245+0000","last_active":"2024-02-05T08:26:37.162264+0000","last_peered":"2024-02-05T08:26:37.162264+0000","last_clean":"2024-02-05T08:26:37.162264+0000","last_became_active":"2024-02-05T08:25:49.113840+0000","last_became_peered":"2024-02-05T08:25:49.113840+0000","last_unstale":"2024-02-05T08:26:37.162264+0000","last_undegraded":"2024-02-05T08:26:37.162264+0000","last_fullsized":"2024-02-05T08:26:37.162264+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:40.135374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:40.135374+0000","last_clean_scrub_stamp":"2024-02-05T08:24:40.135374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"397'66","reported_seq":1071,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.172050+0000","last_change":"2024-02-05T08:26:30.406638+0000","last_active":"2024-02-05T08:26:35.172050+0000","last_peered":"2024-02-05T08:26:35.172050+0000","last_clean":"2024-02-05T08:26:35.172050+0000","last_became_active":"2024-02-05T08:23:07.932513+0000","last_became_peered":"2024-02-05T08:23:07.932513+0000","last_unstale":"2024-02-05T08:26:35.172050+0000","last_undegraded":"2024-02-05T08:26:35.172050+0000","last_fullsized":"2024-02-05T08:26:35.172050+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'66","last_scrub_stamp":"2024-02-05T08:26:30.406383+0000","last_deep_scrub":"85'38","last_deep_scrub_stamp":"2024-02-05T08:05:43.858841+0000","last_clean_scrub_stamp":"2024-02-05T08:26:30.406383+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:48.591233+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":45,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175575+0000","last_change":"2024-02-05T08:25:49.135204+0000","last_active":"2024-02-05T08:26:37.175575+0000","last_peered":"2024-02-05T08:26:37.175575+0000","last_clean":"2024-02-05T08:26:37.175575+0000","last_became_active":"2024-02-05T08:25:49.134871+0000","last_became_peered":"2024-02-05T08:25:49.134871+0000","last_unstale":"2024-02-05T08:26:37.175575+0000","last_undegraded":"2024-02-05T08:26:37.175575+0000","last_fullsized":"2024-02-05T08:26:37.175575+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:32.208069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:52.171116+0000","last_clean_scrub_stamp":"2024-02-05T08:24:32.208069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"424'16235","reported_seq":16382,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.615101+0000","last_change":"2024-02-05T08:26:29.782088+0000","last_active":"2024-02-05T08:26:38.615101+0000","last_peered":"2024-02-05T08:26:38.615101+0000","last_clean":"2024-02-05T08:26:38.615101+0000","last_became_active":"2024-02-05T08:23:08.915520+0000","last_became_peered":"2024-02-05T08:23:08.915520+0000","last_unstale":"2024-02-05T08:26:38.615101+0000","last_undegraded":"2024-02-05T08:26:38.615101+0000","last_fullsized":"2024-02-05T08:26:38.615101+0000","mapping_epoch":367,"log_start":"402'11807","ondisk_log_start":"402'11807","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'15641","last_scrub_stamp":"2024-02-05T08:26:29.781838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:26:29.781838+0000","objects_scrubbed":1267,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:32.171173+0000","scrub_duration":522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43188224,"num_objects":659,"num_object_clones":0,"num_object_copies":1977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16235,"num_write_kb":540608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"82'130","reported_seq":1130,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.173611+0000","last_change":"2024-02-05T08:25:52.156966+0000","last_active":"2024-02-05T08:26:37.173611+0000","last_peered":"2024-02-05T08:26:37.173611+0000","last_clean":"2024-02-05T08:26:37.173611+0000","last_became_active":"2024-02-05T08:25:52.156544+0000","last_became_peered":"2024-02-05T08:25:52.156544+0000","last_unstale":"2024-02-05T08:26:37.173611+0000","last_undegraded":"2024-02-05T08:26:37.173611+0000","last_fullsized":"2024-02-05T08:26:37.173611+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:25:30.843058+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:25:30.843058+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:44.803385+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":819,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175004+0000","last_change":"2024-02-05T08:25:22.982720+0000","last_active":"2024-02-05T08:26:37.175004+0000","last_peered":"2024-02-05T08:26:37.175004+0000","last_clean":"2024-02-05T08:26:37.175004+0000","last_became_active":"2024-02-05T08:18:34.643109+0000","last_became_peered":"2024-02-05T08:18:34.643109+0000","last_unstale":"2024-02-05T08:26:37.175004+0000","last_undegraded":"2024-02-05T08:26:37.175004+0000","last_fullsized":"2024-02-05T08:26:37.175004+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:22.982281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:25:22.982281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:48.901253+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":91,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.179292+0000","last_change":"2024-02-05T08:24:08.030313+0000","last_active":"2024-02-05T08:26:37.179292+0000","last_peered":"2024-02-05T08:26:37.179292+0000","last_clean":"2024-02-05T08:26:37.179292+0000","last_became_active":"2024-02-05T08:24:08.005056+0000","last_became_peered":"2024-02-05T08:24:08.005056+0000","last_unstale":"2024-02-05T08:26:37.179292+0000","last_undegraded":"2024-02-05T08:26:37.179292+0000","last_fullsized":"2024-02-05T08:26:37.179292+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:23:23.016855+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:23:23.016855+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":875,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.565193+0000","last_change":"2024-02-05T08:26:37.565790+0000","last_active":"2024-02-05T08:26:37.565193+0000","last_peered":"2024-02-05T08:26:37.565193+0000","last_clean":"2024-02-05T08:26:37.565193+0000","last_became_active":"2024-02-05T08:25:49.127248+0000","last_became_peered":"2024-02-05T08:25:49.127248+0000","last_unstale":"2024-02-05T08:26:37.565193+0000","last_undegraded":"2024-02-05T08:26:37.565193+0000","last_fullsized":"2024-02-05T08:26:37.565193+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:37.565107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_clean_scrub_stamp":"2024-02-05T08:26:37.565107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:40.880171+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"397'26","reported_seq":882,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.168268+0000","last_change":"2024-02-05T08:26:28.660283+0000","last_active":"2024-02-05T08:26:37.168268+0000","last_peered":"2024-02-05T08:26:37.168268+0000","last_clean":"2024-02-05T08:26:37.168268+0000","last_became_active":"2024-02-05T08:25:49.115716+0000","last_became_peered":"2024-02-05T08:25:49.115716+0000","last_unstale":"2024-02-05T08:26:37.168268+0000","last_undegraded":"2024-02-05T08:26:37.168268+0000","last_fullsized":"2024-02-05T08:26:37.168268+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'26","last_scrub_stamp":"2024-02-05T08:26:28.660156+0000","last_deep_scrub":"363'22","last_deep_scrub_stamp":"2024-02-05T08:22:56.032522+0000","last_clean_scrub_stamp":"2024-02-05T08:26:28.660156+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:30.430186+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":830,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.172652+0000","last_change":"2024-02-05T08:25:46.935787+0000","last_active":"2024-02-05T08:26:37.172652+0000","last_peered":"2024-02-05T08:26:37.172652+0000","last_clean":"2024-02-05T08:26:37.172652+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:26:37.172652+0000","last_undegraded":"2024-02-05T08:26:37.172652+0000","last_fullsized":"2024-02-05T08:26:37.172652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:46.935681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:25:46.935681+0000","last_clean_scrub_stamp":"2024-02-05T08:25:46.935681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:49.495182+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"424'16419","reported_seq":16594,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.615701+0000","last_change":"2024-02-05T08:25:46.500473+0000","last_active":"2024-02-05T08:26:38.615701+0000","last_peered":"2024-02-05T08:26:38.615701+0000","last_clean":"2024-02-05T08:26:38.615701+0000","last_became_active":"2024-02-05T08:23:08.914611+0000","last_became_peered":"2024-02-05T08:23:08.914611+0000","last_unstale":"2024-02-05T08:26:38.615701+0000","last_undegraded":"2024-02-05T08:26:38.615701+0000","last_fullsized":"2024-02-05T08:26:38.615701+0000","mapping_epoch":367,"log_start":"402'12000","ondisk_log_start":"402'12000","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'13216","last_scrub_stamp":"2024-02-05T08:25:46.500320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:25:46.500320+0000","objects_scrubbed":3933,"log_size":4419,"log_dups_size":0,"ondisk_log_size":4419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:10.868093+0000","scrub_duration":1263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45416448,"num_objects":693,"num_object_clones":0,"num_object_copies":2079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16419,"num_write_kb":547584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":814,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.165815+0000","last_change":"2024-02-05T08:26:24.070698+0000","last_active":"2024-02-05T08:26:37.165815+0000","last_peered":"2024-02-05T08:26:37.165815+0000","last_clean":"2024-02-05T08:26:37.165815+0000","last_became_active":"2024-02-05T08:18:34.633450+0000","last_became_peered":"2024-02-05T08:18:34.633450+0000","last_unstale":"2024-02-05T08:26:37.165815+0000","last_undegraded":"2024-02-05T08:26:37.165815+0000","last_fullsized":"2024-02-05T08:26:37.165815+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:24.070580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:26:24.070580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:31.375501+0000","scrub_duration":170,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":1128,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.173478+0000","last_change":"2024-02-05T08:26:22.636852+0000","last_active":"2024-02-05T08:26:37.173478+0000","last_peered":"2024-02-05T08:26:37.173478+0000","last_clean":"2024-02-05T08:26:37.173478+0000","last_became_active":"2024-02-05T08:25:52.159754+0000","last_became_peered":"2024-02-05T08:25:52.159754+0000","last_unstale":"2024-02-05T08:26:37.173478+0000","last_undegraded":"2024-02-05T08:26:37.173478+0000","last_fullsized":"2024-02-05T08:26:37.173478+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:26:22.636584+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_clean_scrub_stamp":"2024-02-05T08:26:22.636584+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:30.123995+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":304,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176867+0000","last_change":"2024-02-05T08:25:30.182895+0000","last_active":"2024-02-05T08:26:37.176867+0000","last_peered":"2024-02-05T08:26:37.176867+0000","last_clean":"2024-02-05T08:26:37.176867+0000","last_became_active":"2024-02-05T08:18:34.631974+0000","last_became_peered":"2024-02-05T08:18:34.631974+0000","last_unstale":"2024-02-05T08:26:37.176867+0000","last_undegraded":"2024-02-05T08:26:37.176867+0000","last_fullsized":"2024-02-05T08:26:37.176867+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:30.182774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:25:30.182774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:55.002934+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"397'42","reported_seq":767,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176650+0000","last_change":"2024-02-05T08:25:32.258788+0000","last_active":"2024-02-05T08:26:37.176650+0000","last_peered":"2024-02-05T08:26:37.176650+0000","last_clean":"2024-02-05T08:26:37.176650+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:26:37.176650+0000","last_undegraded":"2024-02-05T08:26:37.176650+0000","last_fullsized":"2024-02-05T08:26:37.176650+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'42","last_scrub_stamp":"2024-02-05T08:25:32.258665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:25:32.258665+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:48.950208+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":799,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.174863+0000","last_change":"2024-02-05T08:24:27.990755+0000","last_active":"2024-02-05T08:26:37.174863+0000","last_peered":"2024-02-05T08:26:37.174863+0000","last_clean":"2024-02-05T08:26:37.174863+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:26:37.174863+0000","last_undegraded":"2024-02-05T08:26:37.174863+0000","last_fullsized":"2024-02-05T08:26:37.174863+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:27.990653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_clean_scrub_stamp":"2024-02-05T08:24:27.990653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"424'16164","reported_seq":19977,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.622271+0000","last_change":"2024-02-05T08:26:27.979396+0000","last_active":"2024-02-05T08:26:38.622271+0000","last_peered":"2024-02-05T08:26:38.622271+0000","last_clean":"2024-02-05T08:26:38.622271+0000","last_became_active":"2024-02-05T08:26:27.955864+0000","last_became_peered":"2024-02-05T08:26:27.955864+0000","last_unstale":"2024-02-05T08:26:38.622271+0000","last_undegraded":"2024-02-05T08:26:38.622271+0000","last_fullsized":"2024-02-05T08:26:38.622271+0000","mapping_epoch":421,"log_start":"402'11724","ondisk_log_start":"402'11724","created":367,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'12783","last_scrub_stamp":"2024-02-05T08:25:42.914649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:25:42.914649+0000","objects_scrubbed":4115,"log_size":4440,"log_dups_size":0,"ondisk_log_size":4440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:05.095075+0000","scrub_duration":1629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44171264,"num_objects":674,"num_object_clones":0,"num_object_copies":2022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16164,"num_write_kb":538816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1812,"num_bytes_recovered":118751232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"82'130","reported_seq":466,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.168886+0000","last_change":"2024-02-05T08:25:27.246456+0000","last_active":"2024-02-05T08:26:35.168886+0000","last_peered":"2024-02-05T08:26:35.168886+0000","last_clean":"2024-02-05T08:26:35.168886+0000","last_became_active":"2024-02-05T08:23:07.941142+0000","last_became_peered":"2024-02-05T08:23:07.941142+0000","last_unstale":"2024-02-05T08:26:35.168886+0000","last_undegraded":"2024-02-05T08:26:35.168886+0000","last_fullsized":"2024-02-05T08:26:35.168886+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:25:27.246047+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:25:27.246047+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:38.282823+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":708,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.172234+0000","last_change":"2024-02-05T08:26:27.425554+0000","last_active":"2024-02-05T08:26:35.172234+0000","last_peered":"2024-02-05T08:26:35.172234+0000","last_clean":"2024-02-05T08:26:35.172234+0000","last_became_active":"2024-02-05T08:23:07.935640+0000","last_became_peered":"2024-02-05T08:23:07.935640+0000","last_unstale":"2024-02-05T08:26:35.172234+0000","last_undegraded":"2024-02-05T08:26:35.172234+0000","last_fullsized":"2024-02-05T08:26:35.172234+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:27.425482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:26:27.425482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:49.824659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":45,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.207569+0000","last_change":"2024-02-05T08:25:49.135301+0000","last_active":"2024-02-05T08:26:37.207569+0000","last_peered":"2024-02-05T08:26:37.207569+0000","last_clean":"2024-02-05T08:26:37.207569+0000","last_became_active":"2024-02-05T08:25:49.134615+0000","last_became_peered":"2024-02-05T08:25:49.134615+0000","last_unstale":"2024-02-05T08:26:37.207569+0000","last_undegraded":"2024-02-05T08:26:37.207569+0000","last_fullsized":"2024-02-05T08:26:37.207569+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:36.214392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:25:36.214392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:53.322572+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":2151,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.209708+0000","last_change":"2024-02-05T08:25:49.158242+0000","last_active":"2024-02-05T08:26:37.209708+0000","last_peered":"2024-02-05T08:26:37.209708+0000","last_clean":"2024-02-05T08:26:37.209708+0000","last_became_active":"2024-02-05T08:25:49.148530+0000","last_became_peered":"2024-02-05T08:25:49.148530+0000","last_unstale":"2024-02-05T08:26:37.209708+0000","last_undegraded":"2024-02-05T08:26:37.209708+0000","last_fullsized":"2024-02-05T08:26:37.209708+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:25:38.835807+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:25:38.835807+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:42.081765+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1291,"num_read_kb":1286,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"397'36","reported_seq":59,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.207479+0000","last_change":"2024-02-05T08:25:49.185774+0000","last_active":"2024-02-05T08:26:37.207479+0000","last_peered":"2024-02-05T08:26:37.207479+0000","last_clean":"2024-02-05T08:26:37.207479+0000","last_became_active":"2024-02-05T08:25:49.144423+0000","last_became_peered":"2024-02-05T08:25:49.144423+0000","last_unstale":"2024-02-05T08:26:37.207479+0000","last_undegraded":"2024-02-05T08:26:37.207479+0000","last_fullsized":"2024-02-05T08:26:37.207479+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-02-05T08:24:19.820518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:24:19.820518+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":825,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.166307+0000","last_change":"2024-02-05T08:26:30.828019+0000","last_active":"2024-02-05T08:26:37.166307+0000","last_peered":"2024-02-05T08:26:37.166307+0000","last_clean":"2024-02-05T08:26:37.166307+0000","last_became_active":"2024-02-05T08:25:49.116332+0000","last_became_peered":"2024-02-05T08:25:49.116332+0000","last_unstale":"2024-02-05T08:26:37.166307+0000","last_undegraded":"2024-02-05T08:26:37.166307+0000","last_fullsized":"2024-02-05T08:26:37.166307+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:30.827900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:26:30.827900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:47.704310+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"423'15885","reported_seq":16075,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:36.277087+0000","last_change":"2024-02-05T08:25:23.078226+0000","last_active":"2024-02-05T08:26:36.277087+0000","last_peered":"2024-02-05T08:26:36.277087+0000","last_clean":"2024-02-05T08:26:36.277087+0000","last_became_active":"2024-02-05T08:23:08.920532+0000","last_became_peered":"2024-02-05T08:23:08.920532+0000","last_unstale":"2024-02-05T08:26:36.277087+0000","last_undegraded":"2024-02-05T08:26:36.277087+0000","last_fullsized":"2024-02-05T08:26:36.277087+0000","mapping_epoch":367,"log_start":"402'11100","ondisk_log_start":"402'11100","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'11027","last_scrub_stamp":"2024-02-05T08:25:23.077997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:25:23.077997+0000","objects_scrubbed":5794,"log_size":4785,"log_dups_size":0,"ondisk_log_size":4785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56557568,"num_objects":863,"num_object_clones":0,"num_object_copies":2589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15885,"num_write_kb":535936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"82'130","reported_seq":875,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175759+0000","last_change":"2024-02-05T08:26:30.806482+0000","last_active":"2024-02-05T08:26:37.175759+0000","last_peered":"2024-02-05T08:26:37.175759+0000","last_clean":"2024-02-05T08:26:37.175759+0000","last_became_active":"2024-02-05T08:25:52.168497+0000","last_became_peered":"2024-02-05T08:25:52.168497+0000","last_unstale":"2024-02-05T08:26:37.175759+0000","last_undegraded":"2024-02-05T08:26:37.175759+0000","last_fullsized":"2024-02-05T08:26:37.175759+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:26:30.806359+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:26:30.806359+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:36.729695+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":717,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.163979+0000","last_change":"2024-02-05T08:26:29.450323+0000","last_active":"2024-02-05T08:26:35.163979+0000","last_peered":"2024-02-05T08:26:35.163979+0000","last_clean":"2024-02-05T08:26:35.163979+0000","last_became_active":"2024-02-05T08:23:07.928514+0000","last_became_peered":"2024-02-05T08:23:07.928514+0000","last_unstale":"2024-02-05T08:26:35.163979+0000","last_undegraded":"2024-02-05T08:26:35.163979+0000","last_fullsized":"2024-02-05T08:26:35.163979+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:29.450016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:26:29.450016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:59.167387+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":672,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.172450+0000","last_change":"2024-02-05T08:25:33.241808+0000","last_active":"2024-02-05T08:26:35.172450+0000","last_peered":"2024-02-05T08:26:35.172450+0000","last_clean":"2024-02-05T08:26:35.172450+0000","last_became_active":"2024-02-05T08:23:07.935692+0000","last_became_peered":"2024-02-05T08:23:07.935692+0000","last_unstale":"2024-02-05T08:26:35.172450+0000","last_undegraded":"2024-02-05T08:26:35.172450+0000","last_fullsized":"2024-02-05T08:26:35.172450+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:33.240907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:25:33.240907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:48.477170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"397'47","reported_seq":71,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176537+0000","last_change":"2024-02-05T08:26:27.267651+0000","last_active":"2024-02-05T08:26:37.176537+0000","last_peered":"2024-02-05T08:26:37.176537+0000","last_clean":"2024-02-05T08:26:37.176537+0000","last_became_active":"2024-02-05T08:25:49.143415+0000","last_became_peered":"2024-02-05T08:25:49.143415+0000","last_unstale":"2024-02-05T08:26:37.176537+0000","last_undegraded":"2024-02-05T08:26:37.176537+0000","last_fullsized":"2024-02-05T08:26:37.176537+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'47","last_scrub_stamp":"2024-02-05T08:26:27.267562+0000","last_deep_scrub":"381'37","last_deep_scrub_stamp":"2024-02-05T08:23:36.866312+0000","last_clean_scrub_stamp":"2024-02-05T08:26:27.267562+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:49.512666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":563,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175832+0000","last_change":"2024-02-05T08:26:26.719783+0000","last_active":"2024-02-05T08:26:37.175832+0000","last_peered":"2024-02-05T08:26:37.175832+0000","last_clean":"2024-02-05T08:26:37.175832+0000","last_became_active":"2024-02-05T08:25:49.126901+0000","last_became_peered":"2024-02-05T08:25:49.126901+0000","last_unstale":"2024-02-05T08:26:37.175832+0000","last_undegraded":"2024-02-05T08:26:37.175832+0000","last_fullsized":"2024-02-05T08:26:37.175832+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:26.719705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:26:26.719705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:53.251671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"424'16142","reported_seq":16398,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.700612+0000","last_change":"2024-02-05T08:25:44.102546+0000","last_active":"2024-02-05T08:26:38.700612+0000","last_peered":"2024-02-05T08:26:38.700612+0000","last_clean":"2024-02-05T08:26:38.700612+0000","last_became_active":"2024-02-05T08:23:08.911093+0000","last_became_peered":"2024-02-05T08:23:08.911093+0000","last_unstale":"2024-02-05T08:26:38.700612+0000","last_undegraded":"2024-02-05T08:26:38.700612+0000","last_fullsized":"2024-02-05T08:26:38.700612+0000","mapping_epoch":367,"log_start":"401'10100","ondisk_log_start":"401'10100","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'12844","last_scrub_stamp":"2024-02-05T08:25:44.102458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:25:44.102458+0000","objects_scrubbed":3999,"log_size":6042,"log_dups_size":0,"ondisk_log_size":6042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:50.944820+0000","scrub_duration":1267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43188248,"num_objects":660,"num_object_clones":0,"num_object_copies":1980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":660,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16142,"num_write_kb":537601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"82'130","reported_seq":857,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175997+0000","last_change":"2024-02-05T08:26:31.832482+0000","last_active":"2024-02-05T08:26:37.175997+0000","last_peered":"2024-02-05T08:26:37.175997+0000","last_clean":"2024-02-05T08:26:37.175997+0000","last_became_active":"2024-02-05T08:19:01.815446+0000","last_became_peered":"2024-02-05T08:19:01.815446+0000","last_unstale":"2024-02-05T08:26:37.175997+0000","last_undegraded":"2024-02-05T08:26:37.175997+0000","last_fullsized":"2024-02-05T08:26:37.175997+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:26:31.832363+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:35.974462+0000","last_clean_scrub_stamp":"2024-02-05T08:26:31.832363+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:35.789911+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":53,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175790+0000","last_change":"2024-02-05T08:26:26.273476+0000","last_active":"2024-02-05T08:26:37.175790+0000","last_peered":"2024-02-05T08:26:37.175790+0000","last_clean":"2024-02-05T08:26:37.175790+0000","last_became_active":"2024-02-05T08:25:49.133655+0000","last_became_peered":"2024-02-05T08:25:49.133655+0000","last_unstale":"2024-02-05T08:26:37.175790+0000","last_undegraded":"2024-02-05T08:26:37.175790+0000","last_fullsized":"2024-02-05T08:26:37.175790+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:26.273355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:26:26.273355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:48.028512+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":826,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.166109+0000","last_change":"2024-02-05T08:25:44.934508+0000","last_active":"2024-02-05T08:26:37.166109+0000","last_peered":"2024-02-05T08:26:37.166109+0000","last_clean":"2024-02-05T08:26:37.166109+0000","last_became_active":"2024-02-05T08:18:34.633263+0000","last_became_peered":"2024-02-05T08:18:34.633263+0000","last_unstale":"2024-02-05T08:26:37.166109+0000","last_undegraded":"2024-02-05T08:26:37.166109+0000","last_fullsized":"2024-02-05T08:26:37.166109+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:44.934390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:25:44.934390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:46.163100+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"397'30","reported_seq":743,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.164345+0000","last_change":"2024-02-05T08:26:26.444720+0000","last_active":"2024-02-05T08:26:35.164345+0000","last_peered":"2024-02-05T08:26:35.164345+0000","last_clean":"2024-02-05T08:26:35.164345+0000","last_became_active":"2024-02-05T08:23:07.941561+0000","last_became_peered":"2024-02-05T08:23:07.941561+0000","last_unstale":"2024-02-05T08:26:35.164345+0000","last_undegraded":"2024-02-05T08:26:35.164345+0000","last_fullsized":"2024-02-05T08:26:35.164345+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-02-05T08:26:26.444633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:26:26.444633+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:38.013158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"407'13","reported_seq":60,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176559+0000","last_change":"2024-02-05T08:25:49.167615+0000","last_active":"2024-02-05T08:26:37.176559+0000","last_peered":"2024-02-05T08:26:37.176559+0000","last_clean":"2024-02-05T08:26:37.176559+0000","last_became_active":"2024-02-05T08:25:49.142430+0000","last_became_peered":"2024-02-05T08:25:49.142430+0000","last_unstale":"2024-02-05T08:26:37.176559+0000","last_undegraded":"2024-02-05T08:26:37.176559+0000","last_fullsized":"2024-02-05T08:26:37.176559+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'12","last_scrub_stamp":"2024-02-05T08:24:38.385240+0000","last_deep_scrub":"355'12","last_deep_scrub_stamp":"2024-02-05T08:24:38.385240+0000","last_clean_scrub_stamp":"2024-02-05T08:24:38.385240+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.e","version":"424'16082","reported_seq":16231,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.964209+0000","last_change":"2024-02-05T08:26:25.968980+0000","last_active":"2024-02-05T08:26:37.964209+0000","last_peered":"2024-02-05T08:26:37.964209+0000","last_clean":"2024-02-05T08:26:37.964209+0000","last_became_active":"2024-02-05T08:23:08.910360+0000","last_became_peered":"2024-02-05T08:23:08.910360+0000","last_unstale":"2024-02-05T08:26:37.964209+0000","last_undegraded":"2024-02-05T08:26:37.964209+0000","last_fullsized":"2024-02-05T08:26:37.964209+0000","mapping_epoch":367,"log_start":"402'11500","ondisk_log_start":"402'11500","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"419'15341","last_scrub_stamp":"2024-02-05T08:26:25.968735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:26:25.968735+0000","objects_scrubbed":1454,"log_size":4582,"log_dups_size":0,"ondisk_log_size":4582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:54.478724+0000","scrub_duration":557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45481984,"num_objects":694,"num_object_clones":0,"num_object_copies":2082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16082,"num_write_kb":536832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"82'130","reported_seq":580,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176315+0000","last_change":"2024-02-05T08:25:39.247161+0000","last_active":"2024-02-05T08:26:37.176315+0000","last_peered":"2024-02-05T08:26:37.176315+0000","last_clean":"2024-02-05T08:26:37.176315+0000","last_became_active":"2024-02-05T08:19:01.819856+0000","last_became_peered":"2024-02-05T08:19:01.819856+0000","last_unstale":"2024-02-05T08:26:37.176315+0000","last_undegraded":"2024-02-05T08:26:37.176315+0000","last_fullsized":"2024-02-05T08:26:37.176315+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:25:39.247028+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:25:39.247028+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:56.225862+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":807,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.174420+0000","last_change":"2024-02-05T08:26:23.775980+0000","last_active":"2024-02-05T08:26:37.174420+0000","last_peered":"2024-02-05T08:26:37.174420+0000","last_clean":"2024-02-05T08:26:37.174420+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:26:37.174420+0000","last_undegraded":"2024-02-05T08:26:37.174420+0000","last_fullsized":"2024-02-05T08:26:37.174420+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:23.775876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:26:23.775876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:49.493502+0000","scrub_duration":191,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":49,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.207638+0000","last_change":"2024-02-05T08:25:49.176972+0000","last_active":"2024-02-05T08:26:37.207638+0000","last_peered":"2024-02-05T08:26:37.207638+0000","last_clean":"2024-02-05T08:26:37.207638+0000","last_became_active":"2024-02-05T08:25:49.135389+0000","last_became_peered":"2024-02-05T08:25:49.135389+0000","last_unstale":"2024-02-05T08:26:37.207638+0000","last_undegraded":"2024-02-05T08:26:37.207638+0000","last_fullsized":"2024-02-05T08:26:37.207638+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:24:30.256524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:30.256524+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"397'42","reported_seq":901,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.161849+0000","last_change":"2024-02-05T08:24:16.241737+0000","last_active":"2024-02-05T08:26:37.161849+0000","last_peered":"2024-02-05T08:26:37.161849+0000","last_clean":"2024-02-05T08:26:37.161849+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:26:37.161849+0000","last_undegraded":"2024-02-05T08:26:37.161849+0000","last_fullsized":"2024-02-05T08:26:37.161849+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'28","last_scrub_stamp":"2024-02-05T08:24:16.241606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:24:16.241606+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"370'13","reported_seq":402,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.179456+0000","last_change":"2024-02-05T08:23:33.927361+0000","last_active":"2024-02-05T08:26:37.179456+0000","last_peered":"2024-02-05T08:26:37.179456+0000","last_clean":"2024-02-05T08:26:37.179456+0000","last_became_active":"2024-02-05T08:23:07.936347+0000","last_became_peered":"2024-02-05T08:23:07.936347+0000","last_unstale":"2024-02-05T08:26:37.179456+0000","last_undegraded":"2024-02-05T08:26:37.179456+0000","last_fullsized":"2024-02-05T08:26:37.179456+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'13","last_scrub_stamp":"2024-02-05T08:23:33.927148+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:23:33.927148+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"424'15988","reported_seq":16173,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.552706+0000","last_change":"2024-02-05T08:25:31.482450+0000","last_active":"2024-02-05T08:26:38.552706+0000","last_peered":"2024-02-05T08:26:38.552706+0000","last_clean":"2024-02-05T08:26:38.552706+0000","last_became_active":"2024-02-05T08:23:08.921121+0000","last_became_peered":"2024-02-05T08:23:08.921121+0000","last_unstale":"2024-02-05T08:26:38.552706+0000","last_undegraded":"2024-02-05T08:26:38.552706+0000","last_fullsized":"2024-02-05T08:26:38.552706+0000","mapping_epoch":367,"log_start":"402'11100","ondisk_log_start":"402'11100","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'11668","last_scrub_stamp":"2024-02-05T08:25:31.482232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:25:31.482232+0000","objects_scrubbed":5077,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:44.653839+0000","scrub_duration":1655,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45481984,"num_objects":694,"num_object_clones":0,"num_object_copies":2082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15988,"num_write_kb":533824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"82'130","reported_seq":580,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162564+0000","last_change":"2024-02-05T08:26:23.365586+0000","last_active":"2024-02-05T08:26:37.162564+0000","last_peered":"2024-02-05T08:26:37.162564+0000","last_clean":"2024-02-05T08:26:37.162564+0000","last_became_active":"2024-02-05T08:19:01.816923+0000","last_became_peered":"2024-02-05T08:19:01.816923+0000","last_unstale":"2024-02-05T08:26:37.162564+0000","last_undegraded":"2024-02-05T08:26:37.162564+0000","last_fullsized":"2024-02-05T08:26:37.162564+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:26:23.365483+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:26:23.365483+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:34.060227+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":814,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.161712+0000","last_change":"2024-02-05T08:25:31.210523+0000","last_active":"2024-02-05T08:26:37.161712+0000","last_peered":"2024-02-05T08:26:37.161712+0000","last_clean":"2024-02-05T08:26:37.161712+0000","last_became_active":"2024-02-05T08:18:34.635908+0000","last_became_peered":"2024-02-05T08:18:34.635908+0000","last_unstale":"2024-02-05T08:26:37.161712+0000","last_undegraded":"2024-02-05T08:26:37.161712+0000","last_fullsized":"2024-02-05T08:26:37.161712+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:31.210408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:08.117925+0000","last_clean_scrub_stamp":"2024-02-05T08:25:31.210408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:54.374514+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":328,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.177843+0000","last_change":"2024-02-05T08:25:49.155417+0000","last_active":"2024-02-05T08:26:37.177843+0000","last_peered":"2024-02-05T08:26:37.177843+0000","last_clean":"2024-02-05T08:26:37.177843+0000","last_became_active":"2024-02-05T08:25:49.127834+0000","last_became_peered":"2024-02-05T08:25:49.127834+0000","last_unstale":"2024-02-05T08:26:37.177843+0000","last_undegraded":"2024-02-05T08:26:37.177843+0000","last_fullsized":"2024-02-05T08:26:37.177843+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:25:31.175430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:25:31.175430+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:37.711238+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"397'50","reported_seq":393,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176188+0000","last_change":"2024-02-05T08:25:49.182225+0000","last_active":"2024-02-05T08:26:37.176188+0000","last_peered":"2024-02-05T08:26:37.176188+0000","last_clean":"2024-02-05T08:26:37.176188+0000","last_became_active":"2024-02-05T08:25:49.145063+0000","last_became_peered":"2024-02-05T08:25:49.145063+0000","last_unstale":"2024-02-05T08:26:37.176188+0000","last_undegraded":"2024-02-05T08:26:37.176188+0000","last_fullsized":"2024-02-05T08:26:37.176188+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'50","last_scrub_stamp":"2024-02-05T08:25:35.210592+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:25:35.210592+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:42.322432+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":810,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.161624+0000","last_change":"2024-02-05T08:25:28.129670+0000","last_active":"2024-02-05T08:26:37.161624+0000","last_peered":"2024-02-05T08:26:37.161624+0000","last_clean":"2024-02-05T08:26:37.161624+0000","last_became_active":"2024-02-05T08:18:34.636667+0000","last_became_peered":"2024-02-05T08:18:34.636667+0000","last_unstale":"2024-02-05T08:26:37.161624+0000","last_undegraded":"2024-02-05T08:26:37.161624+0000","last_fullsized":"2024-02-05T08:26:37.161624+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:28.129546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:31.133206+0000","last_clean_scrub_stamp":"2024-02-05T08:25:28.129546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"423'16050","reported_seq":16243,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:36.243024+0000","last_change":"2024-02-05T08:25:20.245674+0000","last_active":"2024-02-05T08:26:36.243024+0000","last_peered":"2024-02-05T08:26:36.243024+0000","last_clean":"2024-02-05T08:26:36.243024+0000","last_became_active":"2024-02-05T08:23:08.916901+0000","last_became_peered":"2024-02-05T08:23:08.916901+0000","last_unstale":"2024-02-05T08:26:36.243024+0000","last_undegraded":"2024-02-05T08:26:36.243024+0000","last_fullsized":"2024-02-05T08:26:36.243024+0000","mapping_epoch":367,"log_start":"402'11200","ondisk_log_start":"402'11200","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"402'10893","last_scrub_stamp":"2024-02-05T08:25:20.245455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:25:20.245455+0000","objects_scrubbed":6069,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:43.636796+0000","scrub_duration":2064,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53870592,"num_objects":822,"num_object_clones":0,"num_object_copies":2466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":822,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16050,"num_write_kb":539904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"82'130","reported_seq":332,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.168245+0000","last_change":"2024-02-05T08:26:23.678828+0000","last_active":"2024-02-05T08:26:37.168245+0000","last_peered":"2024-02-05T08:26:37.168245+0000","last_clean":"2024-02-05T08:26:37.168245+0000","last_became_active":"2024-02-05T08:26:17.055436+0000","last_became_peered":"2024-02-05T08:26:17.055436+0000","last_unstale":"2024-02-05T08:26:37.168245+0000","last_undegraded":"2024-02-05T08:26:37.168245+0000","last_fullsized":"2024-02-05T08:26:37.168245+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:26:23.678550+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:26:23.678550+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:45.055193+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":793,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.164795+0000","last_change":"2024-02-05T08:26:17.058537+0000","last_active":"2024-02-05T08:26:35.164795+0000","last_peered":"2024-02-05T08:26:35.164795+0000","last_clean":"2024-02-05T08:26:35.164795+0000","last_became_active":"2024-02-05T08:26:17.058234+0000","last_became_peered":"2024-02-05T08:26:17.058234+0000","last_unstale":"2024-02-05T08:26:35.164795+0000","last_undegraded":"2024-02-05T08:26:35.164795+0000","last_fullsized":"2024-02-05T08:26:35.164795+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:46.039077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:46.039077+0000","last_clean_scrub_stamp":"2024-02-05T08:24:46.039077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"407'10","reported_seq":1090,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162252+0000","last_change":"2024-02-05T08:25:49.128624+0000","last_active":"2024-02-05T08:26:37.162252+0000","last_peered":"2024-02-05T08:26:37.162252+0000","last_clean":"2024-02-05T08:26:37.162252+0000","last_became_active":"2024-02-05T08:25:49.113977+0000","last_became_peered":"2024-02-05T08:25:49.113977+0000","last_unstale":"2024-02-05T08:26:37.162252+0000","last_undegraded":"2024-02-05T08:26:37.162252+0000","last_fullsized":"2024-02-05T08:26:37.162252+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'9","last_scrub_stamp":"2024-02-05T08:25:38.271999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:25:38.271999+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:05.853647+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":297,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.179238+0000","last_change":"2024-02-05T08:23:29.957871+0000","last_active":"2024-02-05T08:26:37.179238+0000","last_peered":"2024-02-05T08:26:37.179238+0000","last_clean":"2024-02-05T08:26:37.179238+0000","last_became_active":"2024-02-05T08:23:07.937468+0000","last_became_peered":"2024-02-05T08:23:07.937468+0000","last_unstale":"2024-02-05T08:26:37.179238+0000","last_undegraded":"2024-02-05T08:26:37.179238+0000","last_fullsized":"2024-02-05T08:26:37.179238+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:23:29.957655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:23:29.957655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"397'39","reported_seq":777,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.165453+0000","last_change":"2024-02-05T08:25:41.369756+0000","last_active":"2024-02-05T08:26:35.165453+0000","last_peered":"2024-02-05T08:26:35.165453+0000","last_clean":"2024-02-05T08:26:35.165453+0000","last_became_active":"2024-02-05T08:23:07.939747+0000","last_became_peered":"2024-02-05T08:23:07.939747+0000","last_unstale":"2024-02-05T08:26:35.165453+0000","last_undegraded":"2024-02-05T08:26:35.165453+0000","last_fullsized":"2024-02-05T08:26:35.165453+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'39","last_scrub_stamp":"2024-02-05T08:25:41.369380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:25:41.369380+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:03.231748+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"424'16171","reported_seq":20209,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.375269+0000","last_change":"2024-02-05T08:26:27.872513+0000","last_active":"2024-02-05T08:26:38.375269+0000","last_peered":"2024-02-05T08:26:38.375269+0000","last_clean":"2024-02-05T08:26:38.375269+0000","last_became_active":"2024-02-05T08:26:27.009995+0000","last_became_peered":"2024-02-05T08:26:27.009995+0000","last_unstale":"2024-02-05T08:26:38.375269+0000","last_undegraded":"2024-02-05T08:26:38.375269+0000","last_fullsized":"2024-02-05T08:26:38.375269+0000","mapping_epoch":420,"log_start":"402'10700","ondisk_log_start":"402'10700","created":367,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'15401","last_scrub_stamp":"2024-02-05T08:26:27.872314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:26:27.872314+0000","objects_scrubbed":1483,"log_size":5471,"log_dups_size":0,"ondisk_log_size":5471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:45.575199+0000","scrub_duration":148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46727168,"num_objects":713,"num_object_clones":0,"num_object_copies":2139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16171,"num_write_kb":540288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1898,"num_bytes_recovered":124256256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":304,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.176500+0000","last_change":"2024-02-05T08:25:43.291784+0000","last_active":"2024-02-05T08:26:37.176500+0000","last_peered":"2024-02-05T08:26:37.176500+0000","last_clean":"2024-02-05T08:26:37.176500+0000","last_became_active":"2024-02-05T08:18:34.627261+0000","last_became_peered":"2024-02-05T08:18:34.627261+0000","last_unstale":"2024-02-05T08:26:37.176500+0000","last_undegraded":"2024-02-05T08:26:37.176500+0000","last_fullsized":"2024-02-05T08:26:37.176500+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:43.291041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:25:43.291041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:01.601353+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":308,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.164030+0000","last_change":"2024-02-05T08:26:17.044613+0000","last_active":"2024-02-05T08:26:37.164030+0000","last_peered":"2024-02-05T08:26:37.164030+0000","last_clean":"2024-02-05T08:26:37.164030+0000","last_became_active":"2024-02-05T08:26:17.044241+0000","last_became_peered":"2024-02-05T08:26:17.044241+0000","last_unstale":"2024-02-05T08:26:37.164030+0000","last_undegraded":"2024-02-05T08:26:37.164030+0000","last_fullsized":"2024-02-05T08:26:37.164030+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:30.867869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:24:30.867869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":806,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.161709+0000","last_change":"2024-02-05T08:24:38.141660+0000","last_active":"2024-02-05T08:26:37.161709+0000","last_peered":"2024-02-05T08:26:37.161709+0000","last_clean":"2024-02-05T08:26:37.161709+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:26:37.161709+0000","last_undegraded":"2024-02-05T08:26:37.161709+0000","last_fullsized":"2024-02-05T08:26:37.161709+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:38.141536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:38.141536+0000","last_clean_scrub_stamp":"2024-02-05T08:24:38.141536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"397'27","reported_seq":881,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162088+0000","last_change":"2024-02-05T08:26:36.435186+0000","last_active":"2024-02-05T08:26:37.162088+0000","last_peered":"2024-02-05T08:26:37.162088+0000","last_clean":"2024-02-05T08:26:37.162088+0000","last_became_active":"2024-02-05T08:23:07.937017+0000","last_became_peered":"2024-02-05T08:23:07.937017+0000","last_unstale":"2024-02-05T08:26:37.162088+0000","last_undegraded":"2024-02-05T08:26:37.162088+0000","last_fullsized":"2024-02-05T08:26:37.162088+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'27","last_scrub_stamp":"2024-02-05T08:26:36.435053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:26:36.435053+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:49.859248+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"424'16521","reported_seq":21852,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.623797+0000","last_change":"2024-02-05T08:26:18.101295+0000","last_active":"2024-02-05T08:26:38.623797+0000","last_peered":"2024-02-05T08:26:38.623797+0000","last_clean":"2024-02-05T08:26:38.623797+0000","last_became_active":"2024-02-05T08:26:18.075280+0000","last_became_peered":"2024-02-05T08:26:18.075280+0000","last_unstale":"2024-02-05T08:26:38.623797+0000","last_undegraded":"2024-02-05T08:26:38.623797+0000","last_fullsized":"2024-02-05T08:26:38.623797+0000","mapping_epoch":417,"log_start":"402'12166","ondisk_log_start":"402'12166","created":367,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'6207","last_scrub_stamp":"2024-02-05T08:24:17.908422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:24:17.908422+0000","objects_scrubbed":6107,"log_size":4355,"log_dups_size":0,"ondisk_log_size":4355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2250,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45809664,"num_objects":699,"num_object_clones":0,"num_object_copies":2097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16521,"num_write_kb":551040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2555,"num_bytes_recovered":167313408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"82'130","reported_seq":332,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175814+0000","last_change":"2024-02-05T08:26:28.280453+0000","last_active":"2024-02-05T08:26:37.175814+0000","last_peered":"2024-02-05T08:26:37.175814+0000","last_clean":"2024-02-05T08:26:37.175814+0000","last_became_active":"2024-02-05T08:26:17.056472+0000","last_became_peered":"2024-02-05T08:26:17.056472+0000","last_unstale":"2024-02-05T08:26:37.175814+0000","last_undegraded":"2024-02-05T08:26:37.175814+0000","last_fullsized":"2024-02-05T08:26:37.175814+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:26:28.280333+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:26:28.280333+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:38.719061+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":582,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.177657+0000","last_change":"2024-02-05T08:25:04.981918+0000","last_active":"2024-02-05T08:26:37.177657+0000","last_peered":"2024-02-05T08:26:37.177657+0000","last_clean":"2024-02-05T08:26:37.177657+0000","last_became_active":"2024-02-05T08:25:04.981516+0000","last_became_peered":"2024-02-05T08:25:04.981516+0000","last_unstale":"2024-02-05T08:26:37.177657+0000","last_undegraded":"2024-02-05T08:26:37.177657+0000","last_fullsized":"2024-02-05T08:26:37.177657+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:24:45.091300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:24:45.091300+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":655,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T08:26:35.172149+0000","last_change":"2024-02-05T08:25:49.136293+0000","last_active":"2024-02-05T08:26:35.172149+0000","last_peered":"2024-02-05T08:26:35.172149+0000","last_clean":"2024-02-05T08:26:35.172149+0000","last_became_active":"2024-02-05T08:25:49.135936+0000","last_became_peered":"2024-02-05T08:25:49.135936+0000","last_unstale":"2024-02-05T08:26:35.172149+0000","last_undegraded":"2024-02-05T08:26:35.172149+0000","last_fullsized":"2024-02-05T08:26:35.172149+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:25:20.244668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:25:20.244668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:40.705000+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":552,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.175994+0000","last_change":"2024-02-05T08:26:32.867980+0000","last_active":"2024-02-05T08:26:37.175994+0000","last_peered":"2024-02-05T08:26:37.175994+0000","last_clean":"2024-02-05T08:26:37.175994+0000","last_became_active":"2024-02-05T08:23:07.929749+0000","last_became_peered":"2024-02-05T08:23:07.929749+0000","last_unstale":"2024-02-05T08:26:37.175994+0000","last_undegraded":"2024-02-05T08:26:37.175994+0000","last_fullsized":"2024-02-05T08:26:37.175994+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:26:32.867761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:26:32.867761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:35.311004+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"397'45","reported_seq":864,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162836+0000","last_change":"2024-02-05T08:24:26.158100+0000","last_active":"2024-02-05T08:26:37.162836+0000","last_peered":"2024-02-05T08:26:37.162836+0000","last_clean":"2024-02-05T08:26:37.162836+0000","last_became_active":"2024-02-05T08:23:07.930130+0000","last_became_peered":"2024-02-05T08:23:07.930130+0000","last_unstale":"2024-02-05T08:26:37.162836+0000","last_undegraded":"2024-02-05T08:26:37.162836+0000","last_fullsized":"2024-02-05T08:26:37.162836+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'31","last_scrub_stamp":"2024-02-05T08:24:26.157978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:24:26.157978+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"424'16419","reported_seq":16680,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:38.387488+0000","last_change":"2024-02-05T08:25:46.620560+0000","last_active":"2024-02-05T08:26:38.387488+0000","last_peered":"2024-02-05T08:26:38.387488+0000","last_clean":"2024-02-05T08:26:38.387488+0000","last_became_active":"2024-02-05T08:23:08.920062+0000","last_became_peered":"2024-02-05T08:23:08.920062+0000","last_unstale":"2024-02-05T08:26:38.387488+0000","last_undegraded":"2024-02-05T08:26:38.387488+0000","last_fullsized":"2024-02-05T08:26:38.387488+0000","mapping_epoch":367,"log_start":"402'10900","ondisk_log_start":"402'10900","created":367,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'13374","last_scrub_stamp":"2024-02-05T08:25:46.620443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:07.894546+0000","last_clean_scrub_stamp":"2024-02-05T08:25:46.620443+0000","objects_scrubbed":3777,"log_size":5519,"log_dups_size":0,"ondisk_log_size":5519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:26:53.537767+0000","scrub_duration":1189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45285376,"num_objects":691,"num_object_clones":0,"num_object_copies":2073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16419,"num_write_kb":547520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"82'130","reported_seq":1104,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.173398+0000","last_change":"2024-02-05T08:24:22.941847+0000","last_active":"2024-02-05T08:26:37.173398+0000","last_peered":"2024-02-05T08:26:37.173398+0000","last_clean":"2024-02-05T08:26:37.173398+0000","last_became_active":"2024-02-05T08:19:01.816744+0000","last_became_peered":"2024-02-05T08:19:01.816744+0000","last_unstale":"2024-02-05T08:26:37.173398+0000","last_undegraded":"2024-02-05T08:26:37.173398+0000","last_fullsized":"2024-02-05T08:26:37.173398+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:24:22.941728+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:24:22.941728+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":786,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162345+0000","last_change":"2024-02-05T08:24:19.346422+0000","last_active":"2024-02-05T08:26:37.162345+0000","last_peered":"2024-02-05T08:26:37.162345+0000","last_clean":"2024-02-05T08:26:37.162345+0000","last_became_active":"2024-02-05T08:23:07.937641+0000","last_became_peered":"2024-02-05T08:23:07.937641+0000","last_unstale":"2024-02-05T08:26:37.162345+0000","last_undegraded":"2024-02-05T08:26:37.162345+0000","last_fullsized":"2024-02-05T08:26:37.162345+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:19.346083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:24:19.346083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"401'10","reported_seq":289,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.174629+0000","last_change":"2024-02-05T08:25:43.892092+0000","last_active":"2024-02-05T08:26:37.174629+0000","last_peered":"2024-02-05T08:26:37.174629+0000","last_clean":"2024-02-05T08:26:37.174629+0000","last_became_active":"2024-02-05T08:25:04.978603+0000","last_became_peered":"2024-02-05T08:25:04.978603+0000","last_unstale":"2024-02-05T08:26:37.174629+0000","last_undegraded":"2024-02-05T08:26:37.174629+0000","last_fullsized":"2024-02-05T08:26:37.174629+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'10","last_scrub_stamp":"2024-02-05T08:25:43.891673+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:25:43.891673+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:13.829179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":320,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.211406+0000","last_change":"2024-02-05T08:25:49.141396+0000","last_active":"2024-02-05T08:26:37.211406+0000","last_peered":"2024-02-05T08:26:37.211406+0000","last_clean":"2024-02-05T08:26:37.211406+0000","last_became_active":"2024-02-05T08:25:49.138786+0000","last_became_peered":"2024-02-05T08:25:49.138786+0000","last_unstale":"2024-02-05T08:26:37.211406+0000","last_undegraded":"2024-02-05T08:26:37.211406+0000","last_fullsized":"2024-02-05T08:26:37.211406+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:24:46.140359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:24:46.140359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"397'35","reported_seq":931,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T08:26:37.162523+0000","last_change":"2024-02-05T08:25:43.299893+0000","last_active":"2024-02-05T08:26:37.162523+0000","last_peered":"2024-02-05T08:26:37.162523+0000","last_clean":"2024-02-05T08:26:37.162523+0000","last_became_active":"2024-02-05T08:18:34.637413+0000","last_became_peered":"2024-02-05T08:18:34.637413+0000","last_unstale":"2024-02-05T08:26:37.162523+0000","last_undegraded":"2024-02-05T08:26:37.162523+0000","last_fullsized":"2024-02-05T08:26:37.162523+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'35","last_scrub_stamp":"2024-02-05T08:25:43.299755+0000","last_deep_scrub":"397'35","last_deep_scrub_stamp":"2024-02-05T08:25:43.299755+0000","last_clean_scrub_stamp":"2024-02-05T08:25:43.299755+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:27:07.881623+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":733741080,"num_objects":11197,"num_object_clones":0,"num_object_copies":33591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11197,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":259003,"num_write_kb":8646337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11593,"num_bytes_recovered":759365632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2473406464,"data_stored":2473394248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78962,"ondisk_log_size":78962,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1291,"num_read_kb":1286,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1537,"num_read_kb":1294,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":7882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1071,"ondisk_log_size":1071,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038028,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1031892,"kb_used_data":264844,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92713260,"statfs":{"total":95995035648,"available":94938378240,"internally_reserved":0,"allocated":271200256,"data_stored":270881987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":785381834},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:26:06 2024","interfaces":[{"interface":"back","average":{"1min":1.6519999999999999,"5min":1.3049999999999999,"15min":1.484},"min":{"1min":0.26200000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.194,"5min":20.344999999999999,"15min":52.488999999999997},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.153,"15min":1.5149999999999999},"min":{"1min":0.27100000000000002,"5min":0.24099999999999999,"15min":0.23200000000000001},"max":{"1min":3.3079999999999998,"5min":21.347000000000001,"15min":52.335000000000001},"last":0.58799999999999997}]},{"osd":1,"last update":"Mon Feb 5 08:26:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.54,"15min":1.302},"min":{"1min":0.28599999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":11.835000000000001,"5min":31.495999999999999,"15min":31.495999999999999},"last":2.0990000000000002},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.5740000000000001,"15min":1.3700000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":3.835,"5min":31.547000000000001,"15min":31.547000000000001},"last":2.109}]},{"osd":2,"last update":"Mon Feb 5 08:26:06 2024","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.806,"15min":1.2909999999999999},"min":{"1min":0.27700000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.9660000000000002,"5min":31.585999999999999,"15min":31.585999999999999},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.696,"5min":1.827,"15min":1.3759999999999999},"min":{"1min":0.31,"5min":0.248,"15min":0.248},"max":{"1min":11.757999999999999,"5min":31.614999999999998,"15min":31.614999999999998},"last":0.46600000000000003}]},{"osd":3,"last update":"Mon Feb 5 08:26:06 2024","interfaces":[{"interface":"back","average":{"1min":3.6099999999999999,"5min":1.9450000000000001,"15min":1.349},"min":{"1min":0.29599999999999999,"5min":0.26200000000000001,"15min":0.249},"max":{"1min":34.792999999999999,"5min":34.792999999999999,"15min":34.792999999999999},"last":1.5289999999999999},{"interface":"front","average":{"1min":2.7629999999999999,"5min":1.738,"15min":1.29},"min":{"1min":0.30599999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":35.082000000000001,"5min":35.082000000000001,"15min":35.082000000000001},"last":1.3100000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:26:06 2024","interfaces":[{"interface":"back","average":{"1min":1.663,"5min":1.28,"15min":1.2829999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":5.125,"5min":6.5289999999999999,"15min":49.601999999999997},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.804,"5min":1.208,"15min":1.329},"min":{"1min":0.314,"5min":0.249,"15min":0.249},"max":{"1min":11.890000000000001,"5min":11.890000000000001,"15min":49.795999999999999},"last":0.498}]},{"osd":5,"last update":"Mon Feb 5 08:26:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.1699999999999999,"15min":1.3029999999999999},"min":{"1min":0.33900000000000002,"5min":0.27600000000000002,"15min":0.251},"max":{"1min":7.173,"5min":9.6980000000000004,"15min":49.914000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.611,"5min":1.2170000000000001,"15min":1.3340000000000001},"min":{"1min":0.32800000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.0590000000000002,"5min":8.7240000000000002,"15min":52.256},"last":0.56000000000000005}]},{"osd":6,"last update":"Mon Feb 5 08:26:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.6060000000000001,"15min":1.6160000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.28599999999999998},"max":{"1min":3.6629999999999998,"5min":23.800000000000001,"15min":84.254999999999995},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.736,"5min":1.8080000000000001,"15min":1.661},"min":{"1min":0.39100000000000001,"5min":0.34999999999999998,"15min":0.27300000000000002},"max":{"1min":11.866,"5min":29.742999999999999,"15min":84.667000000000002},"last":0.88900000000000001}]}]},{"osd":6,"up_from":51,"seq":219043332370,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1066244,"kb_used_data":377016,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":92678908,"statfs":{"total":95995035648,"available":94903201792,"internally_reserved":0,"allocated":386064384,"data_stored":385764694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":705755590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:25:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.278,"15min":1.8400000000000001},"min":{"1min":0.28399999999999997,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.5129999999999999,"5min":14.836,"15min":102.819},"last":1.123},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.79400000000000004,"15min":1.653},"min":{"1min":0.29099999999999998,"5min":0.24099999999999999,"15min":0.214},"max":{"1min":2.7240000000000002,"5min":3.7850000000000001,"15min":102.75},"last":1.792}]},{"osd":1,"last update":"Mon Feb 5 08:26:10 2024","interfaces":[{"interface":"back","average":{"1min":3.4089999999999998,"5min":2.1899999999999999,"15min":2.052},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":35.718000000000004,"5min":35.718000000000004,"15min":35.718000000000004},"last":3.4289999999999998},{"interface":"front","average":{"1min":3.589,"5min":2.548,"15min":2.4849999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":35.780000000000001,"5min":35.780000000000001,"15min":35.780000000000001},"last":0.32000000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:25:47 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.278,"15min":1.218},"min":{"1min":0.26600000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.6500000000000004,"5min":25.751000000000001,"15min":25.751000000000001},"last":1.885},{"interface":"front","average":{"1min":1.302,"5min":1.278,"15min":1.1779999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.23400000000000001},"max":{"1min":2.7530000000000001,"5min":26.210999999999999,"15min":26.210999999999999},"last":1.2410000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:25:47 2024","interfaces":[{"interface":"back","average":{"1min":4.4180000000000001,"5min":2.2690000000000001,"15min":1.6659999999999999},"min":{"1min":0.32000000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":50.380000000000003,"5min":50.380000000000003,"15min":102.438},"last":3.573},{"interface":"front","average":{"1min":4.609,"5min":1.998,"15min":1.643},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":50.136000000000003,"5min":50.136000000000003,"15min":102.288},"last":2.835}]},{"osd":4,"last update":"Mon Feb 5 08:25:47 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.9359999999999999,"15min":1.244},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":5.5880000000000001,"5min":46.125,"15min":46.125},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.417,"5min":2.2170000000000001,"15min":1.385},"min":{"1min":0.29899999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.8620000000000001,"5min":48.817999999999998,"15min":48.817999999999998},"last":0.56699999999999995}]},{"osd":5,"last update":"Mon Feb 5 08:25:47 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.115,"15min":1.3029999999999999},"min":{"1min":0.34599999999999997,"5min":0.26000000000000001,"15min":0.24199999999999999},"max":{"1min":6.3209999999999997,"5min":7.6760000000000002,"15min":43.643000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.3620000000000001,"15min":1.359},"min":{"1min":0.27100000000000002,"5min":0.246,"15min":0.246},"max":{"1min":6.415,"5min":9.2279999999999998,"15min":43.893000000000001},"last":0.53600000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:26:20 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.99299999999999999,"15min":1.139},"min":{"1min":0.36899999999999999,"5min":0.307,"15min":0.307},"max":{"1min":2.0920000000000001,"5min":4.7530000000000001,"15min":24.728999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.252,"5min":1.4910000000000001,"15min":1.3680000000000001},"min":{"1min":0.43099999999999999,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":4.5640000000000001,"5min":14.988,"15min":21.91},"last":0.59799999999999998}]}]},{"osd":1,"up_from":366,"seq":1571958030379,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":778976,"kb_used_data":548052,"kb_used_omap":4,"kb_used_meta":230907,"kb_avail":92966176,"statfs":{"total":95995035648,"available":95197364224,"internally_reserved":0,"allocated":561205248,"data_stored":560870077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":236449446},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:26:15 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":0.91500000000000004,"15min":0.71699999999999997},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":6.0019999999999998,"5min":6.5999999999999996,"15min":6.5999999999999996},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.7569999999999999,"5min":1.2050000000000001,"15min":1.135},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":10.927,"5min":10.927,"15min":10.927},"last":0.318}]},{"osd":2,"last update":"Mon Feb 5 08:26:15 2024","interfaces":[{"interface":"back","average":{"1min":1.8049999999999999,"5min":1.6819999999999999,"15min":1.615},"min":{"1min":0.29999999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":4.8540000000000001,"5min":7.3289999999999997,"15min":7.3289999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.526,"15min":1.3540000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":6.7949999999999999,"5min":6.7949999999999999,"15min":6.7949999999999999},"last":0.371}]},{"osd":3,"last update":"Mon Feb 5 08:26:15 2024","interfaces":[{"interface":"back","average":{"1min":2.0640000000000001,"5min":1.4590000000000001,"15min":1.329},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":6.6879999999999997,"5min":7.3719999999999999,"15min":7.3719999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":3.1160000000000001,"5min":1.679,"15min":1.3580000000000001},"min":{"1min":0.36499999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":7.3479999999999999,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":0.55600000000000005}]},{"osd":4,"last update":"Mon Feb 5 08:26:15 2024","interfaces":[{"interface":"back","average":{"1min":1.637,"5min":1.458,"15min":1.327},"min":{"1min":0.30199999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.6909999999999998,"5min":6.3920000000000003,"15min":6.3920000000000003},"last":5.3330000000000002},{"interface":"front","average":{"1min":1.516,"5min":1.2789999999999999,"15min":1.2230000000000001},"min":{"1min":0.40500000000000003,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":4.952,"5min":7.1890000000000001,"15min":7.1890000000000001},"last":0.60599999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:26:15 2024","interfaces":[{"interface":"back","average":{"1min":2.2309999999999999,"5min":1.3819999999999999,"15min":1.1930000000000001},"min":{"1min":0.39900000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":7.056,"5min":7.056,"15min":7.056},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.3009999999999999,"15min":1.149},"min":{"1min":0.34200000000000003,"5min":0.311,"15min":0.311},"max":{"1min":4.7930000000000001,"5min":6.4240000000000004,"15min":6.4240000000000004},"last":4.7750000000000004}]},{"osd":6,"last update":"Mon Feb 5 08:26:15 2024","interfaces":[{"interface":"back","average":{"1min":3.8959999999999999,"5min":1.7989999999999999,"15min":1.3999999999999999},"min":{"1min":0.435,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":42.783000000000001,"5min":42.783000000000001,"15min":42.783000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":3.9359999999999999,"5min":1.8420000000000001,"15min":1.4299999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":42.898000000000003,"5min":42.898000000000003,"15min":42.898000000000003},"last":0.52700000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:26:15 2024","interfaces":[{"interface":"back","average":{"1min":4.6970000000000001,"5min":1.96,"15min":1.45},"min":{"1min":0.499,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":64.049999999999997,"5min":64.049999999999997,"15min":64.049999999999997},"last":4.8170000000000002},{"interface":"front","average":{"1min":5.2270000000000003,"5min":2.21,"15min":1.661},"min":{"1min":0.45100000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":64.144000000000005,"5min":64.144000000000005,"15min":64.144000000000005},"last":1.5720000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738672,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1060056,"kb_used_data":204240,"kb_used_omap":1,"kb_used_meta":855806,"kb_avail":92685096,"statfs":{"total":95995035648,"available":94909538304,"internally_reserved":0,"allocated":209141760,"data_stored":208819138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":876345797},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:26:12 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":0.90400000000000003,"15min":0.79600000000000004},"min":{"1min":0.24299999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":5.1970000000000001,"5min":5.1970000000000001,"15min":5.1970000000000001},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.506,"5min":1.014,"15min":0.90500000000000003},"min":{"1min":0.29299999999999998,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.6480000000000001,"5min":3.6480000000000001,"15min":3.6480000000000001},"last":0.68700000000000006}]},{"osd":2,"last update":"Mon Feb 5 08:26:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.83599999999999997,"15min":1.4910000000000001},"min":{"1min":0.24099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":2.1230000000000002,"5min":9.1050000000000004,"15min":105.96599999999999},"last":0.377},{"interface":"front","average":{"1min":1.006,"5min":0.98199999999999998,"15min":1.518},"min":{"1min":0.255,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":5.2830000000000004,"5min":7.3170000000000002,"15min":106.137},"last":0.63200000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:26:38 2024","interfaces":[{"interface":"back","average":{"1min":2.048,"5min":1.0469999999999999,"15min":1.2989999999999999},"min":{"1min":0.27500000000000002,"5min":0.217,"15min":0.20899999999999999},"max":{"1min":6.6849999999999996,"5min":6.6849999999999996,"15min":13.013},"last":0.32600000000000001},{"interface":"front","average":{"1min":2.1040000000000001,"5min":1.1419999999999999,"15min":1.353},"min":{"1min":0.27700000000000002,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":5.7370000000000001,"5min":7.4050000000000002,"15min":16.289999999999999},"last":0.79100000000000004}]},{"osd":4,"last update":"Mon Feb 5 08:26:06 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":0.92000000000000004,"15min":1.3220000000000001},"min":{"1min":0.316,"5min":0.27900000000000003,"15min":0.26300000000000001},"max":{"1min":4.5679999999999996,"5min":4.5679999999999996,"15min":30.914000000000001},"last":0.436},{"interface":"front","average":{"1min":0.88500000000000001,"5min":1.0389999999999999,"15min":1.3320000000000001},"min":{"1min":0.34200000000000003,"5min":0.254,"15min":0.254},"max":{"1min":2.7959999999999998,"5min":8.1899999999999995,"15min":30.93},"last":0.52600000000000002}]},{"osd":5,"last update":"Mon Feb 5 08:26:25 2024","interfaces":[{"interface":"back","average":{"1min":1.6879999999999999,"5min":1.131,"15min":1.589},"min":{"1min":0.314,"5min":0.30199999999999999,"15min":0.28299999999999997},"max":{"1min":5.7800000000000002,"5min":10.888,"15min":38.979999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":1.304,"5min":0.97099999999999997,"15min":1.5269999999999999},"min":{"1min":0.442,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.9910000000000001,"5min":8.2720000000000002,"15min":38.960999999999999},"last":0.35299999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:25:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.2909999999999999,"15min":1.24},"min":{"1min":0.34000000000000002,"5min":0.32700000000000001,"15min":0.30199999999999999},"max":{"1min":3.46,"5min":8.2370000000000001,"15min":8.2370000000000001},"last":4.3929999999999998},{"interface":"front","average":{"1min":1.113,"5min":1.2250000000000001,"15min":1.264},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.27100000000000002},"max":{"1min":3.476,"5min":7.3730000000000002,"15min":15.721},"last":4.4660000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:26:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.091,"15min":1.718},"min":{"1min":0.38,"5min":0.314,"15min":0.29299999999999998},"max":{"1min":4.6029999999999998,"5min":14.507999999999999,"15min":68.241},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.94999999999999996,"5min":1.286,"15min":1.782},"min":{"1min":0.44500000000000001,"5min":0.36799999999999999,"15min":0.252},"max":{"1min":3.0990000000000002,"5min":14.712999999999999,"15min":68.980000000000004},"last":2.5569999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411622,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1055648,"kb_used_data":266324,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":92689504,"statfs":{"total":95995035648,"available":94914052096,"internally_reserved":0,"allocated":272715776,"data_stored":272388681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":808122822},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:26:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":0.76400000000000001,"15min":1.75},"min":{"1min":0.20699999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":5.0869999999999997,"5min":5.2610000000000001,"15min":107.155},"last":1.7589999999999999},{"interface":"front","average":{"1min":1.282,"5min":0.96799999999999997,"15min":2.0019999999999998},"min":{"1min":0.23200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":7.8410000000000002,"5min":7.8410000000000002,"15min":107.221},"last":1.9850000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:26:10 2024","interfaces":[{"interface":"back","average":{"1min":1.8120000000000001,"5min":1.1890000000000001,"15min":1.0349999999999999},"min":{"1min":0.49099999999999999,"5min":0.19,"15min":0.19},"max":{"1min":6.891,"5min":6.891,"15min":6.891},"last":4.5460000000000003},{"interface":"front","average":{"1min":1.373,"5min":1.3560000000000001,"15min":1.3},"min":{"1min":0.23899999999999999,"5min":0.222,"15min":0.222},"max":{"1min":6.9539999999999997,"5min":7.0709999999999997,"15min":7.0709999999999997},"last":0.29799999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:26:27 2024","interfaces":[{"interface":"back","average":{"1min":2.9689999999999999,"5min":1.3109999999999999,"15min":1.2689999999999999},"min":{"1min":0.376,"5min":0.20999999999999999,"15min":0.20499999999999999},"max":{"1min":11.122,"5min":11.122,"15min":11.122},"last":2.0590000000000002},{"interface":"front","average":{"1min":2.3540000000000001,"5min":1.2190000000000001,"15min":1.0980000000000001},"min":{"1min":0.313,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.5590000000000002,"5min":7.8289999999999997,"15min":7.8289999999999997},"last":4.2309999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:25:57 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.3109999999999999,"15min":1.4490000000000001},"min":{"1min":0.30099999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.7490000000000001,"5min":7.2549999999999999,"15min":31.596},"last":2.0369999999999999},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.169,"15min":1.3460000000000001},"min":{"1min":0.34799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":3.8860000000000001,"5min":4.1349999999999998,"15min":31.57},"last":1.8240000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:26:16 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.96599999999999997,"15min":1.5740000000000001},"min":{"1min":0.38800000000000001,"5min":0.308,"15min":0.246},"max":{"1min":1.909,"5min":4.2999999999999998,"15min":58.799999999999997},"last":1.792},{"interface":"front","average":{"1min":1.0409999999999999,"5min":1.2290000000000001,"15min":1.6180000000000001},"min":{"1min":0.36799999999999999,"5min":0.27300000000000002,"15min":0.26900000000000002},"max":{"1min":2.4649999999999999,"5min":6.0759999999999996,"15min":58.615000000000002},"last":0.47399999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:25:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.4790000000000001,"15min":1.341},"min":{"1min":0.34000000000000002,"5min":0.307,"15min":0.29199999999999998},"max":{"1min":2.73,"5min":21.885000000000002,"15min":21.885000000000002},"last":2.0800000000000001},{"interface":"front","average":{"1min":1.143,"5min":1.4119999999999999,"15min":1.2609999999999999},"min":{"1min":0.41899999999999998,"5min":0.35199999999999998,"15min":0.317},"max":{"1min":2.6280000000000001,"5min":22.158000000000001,"15min":22.158000000000001},"last":1.847}]},{"osd":7,"last update":"Mon Feb 5 08:26:05 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":2.0590000000000002,"15min":1.534},"min":{"1min":0.34100000000000003,"5min":0.33900000000000002,"15min":0.28899999999999998},"max":{"1min":4.0090000000000003,"5min":72.334999999999994,"15min":72.334999999999994},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.7390000000000001,"5min":2.3290000000000002,"15min":1.7929999999999999},"min":{"1min":0.40200000000000002,"5min":0.35699999999999998,"15min":0.32700000000000001},"max":{"1min":4.0350000000000001,"5min":72.222999999999999,"15min":72.222999999999999},"last":2.0129999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117281,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1019760,"kb_used_data":316648,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92725392,"statfs":{"total":95995035648,"available":94950801408,"internally_reserved":0,"allocated":324247552,"data_stored":323901406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":719976905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:26:35 2024","interfaces":[{"interface":"back","average":{"1min":1.7569999999999999,"5min":1.0029999999999999,"15min":1.4399999999999999},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.19700000000000001},"max":{"1min":8.4700000000000006,"5min":8.4700000000000006,"15min":52.831000000000003},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.6850000000000001,"5min":0.90100000000000002,"15min":1.548},"min":{"1min":0.23899999999999999,"5min":0.20699999999999999,"15min":0.17699999999999999},"max":{"1min":10.613,"5min":10.613,"15min":53.859999999999999},"last":0.42599999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:26:17 2024","interfaces":[{"interface":"back","average":{"1min":2.2069999999999999,"5min":1.593,"15min":1.4790000000000001},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":11.339,"5min":11.339,"15min":11.339},"last":0.502},{"interface":"front","average":{"1min":2.452,"5min":1.7350000000000001,"15min":1.605},"min":{"1min":0.29299999999999998,"5min":0.253,"15min":0.253},"max":{"1min":10.744999999999999,"5min":10.744999999999999,"15min":10.744999999999999},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:26:35 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.415,"15min":1.367},"min":{"1min":0.27900000000000003,"5min":0.23799999999999999,"15min":0.22},"max":{"1min":5.2850000000000001,"5min":11.936999999999999,"15min":35.277000000000001},"last":0.253},{"interface":"front","average":{"1min":2.1589999999999998,"5min":1.458,"15min":1.246},"min":{"1min":0.34899999999999998,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":8.4429999999999996,"5min":8.4429999999999996,"15min":35.442999999999998},"last":0.40000000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:25:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.7609999999999999,"15min":1.26},"min":{"1min":0.29499999999999998,"5min":0.28999999999999998,"15min":0.27600000000000002},"max":{"1min":6.9409999999999998,"5min":54.502000000000002,"15min":54.502000000000002},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.641,"15min":1.2290000000000001},"min":{"1min":0.40699999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.0949999999999998,"5min":54.540999999999997,"15min":54.540999999999997},"last":2.52}]},{"osd":5,"last update":"Mon Feb 5 08:26:17 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.3180000000000001,"15min":1.8320000000000001},"min":{"1min":0.308,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":8.2609999999999992,"5min":8.2609999999999992,"15min":117.432},"last":2.6560000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.002,"15min":1.706},"min":{"1min":0.36699999999999999,"5min":0.30599999999999999,"15min":0.27600000000000002},"max":{"1min":4.0460000000000003,"5min":4.0460000000000003,"15min":114.828},"last":0.55900000000000005}]},{"osd":6,"last update":"Mon Feb 5 08:26:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8799999999999999,"5min":1.9810000000000001,"15min":1.4219999999999999},"min":{"1min":0.39900000000000002,"5min":0.36299999999999999,"15min":0.29899999999999999},"max":{"1min":10.534000000000001,"5min":54.219999999999999,"15min":54.219999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.9019999999999999,"15min":1.389},"min":{"1min":0.41499999999999998,"5min":0.31900000000000001,"15min":0.29099999999999998},"max":{"1min":3.972,"5min":54.530999999999999,"15min":54.530999999999999},"last":0.318}]},{"osd":7,"last update":"Mon Feb 5 08:25:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.3080000000000001,"15min":1.752},"min":{"1min":0.41599999999999998,"5min":0.34699999999999998,"15min":0.33500000000000002},"max":{"1min":6.1109999999999998,"5min":6.4800000000000004,"15min":114.84999999999999},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.3240000000000001,"15min":1.716},"min":{"1min":0.42099999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":3.0939999999999999,"5min":5.891,"15min":114.871},"last":5.7119999999999997}]}]},{"osd":4,"up_from":32,"seq":137438953756,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1095200,"kb_used_data":364244,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92649952,"statfs":{"total":95995035648,"available":94873550848,"internally_reserved":0,"allocated":372985856,"data_stored":372658066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:25:41 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.0549999999999999,"15min":1.327},"min":{"1min":0.28799999999999998,"5min":0.23300000000000001,"15min":0.192},"max":{"1min":4.54,"5min":4.8529999999999998,"15min":39.552},"last":1.8440000000000001},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.2769999999999999,"15min":1.4339999999999999},"min":{"1min":0.32600000000000001,"5min":0.27600000000000002,"15min":0.214},"max":{"1min":4.4470000000000001,"5min":5.5229999999999997,"15min":39.350999999999999},"last":0.75700000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.98399999999999999,"15min":1.0029999999999999},"min":{"1min":0.26300000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.1520000000000001,"5min":3.915,"15min":3.915},"last":2.0659999999999998},{"interface":"front","average":{"1min":2.444,"5min":1.25,"15min":1.0589999999999999},"min":{"1min":0.32600000000000001,"5min":0.254,"15min":0.254},"max":{"1min":9.8569999999999993,"5min":9.8569999999999993,"15min":9.8569999999999993},"last":2.3180000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:25:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3109999999999999,"15min":1.121},"min":{"1min":0.41999999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":4.4260000000000002,"5min":6.1749999999999998,"15min":6.1749999999999998},"last":2.3519999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.3919999999999999,"15min":1.173},"min":{"1min":0.372,"5min":0.28299999999999997,"15min":0.26300000000000001},"max":{"1min":4.3550000000000004,"5min":8.8490000000000002,"15min":8.8490000000000002},"last":0.68300000000000005}]},{"osd":3,"last update":"Mon Feb 5 08:25:41 2024","interfaces":[{"interface":"back","average":{"1min":3.0009999999999999,"5min":1.802,"15min":1.659},"min":{"1min":0.41599999999999998,"5min":0.312,"15min":0.28599999999999998},"max":{"1min":29.905000000000001,"5min":29.905000000000001,"15min":57.204000000000001},"last":1.911},{"interface":"front","average":{"1min":2.8399999999999999,"5min":1.5509999999999999,"15min":1.573},"min":{"1min":0.30399999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":30.094999999999999,"5min":30.094999999999999,"15min":57.606999999999999},"last":1.7909999999999999}]},{"osd":5,"last update":"Mon Feb 5 08:26:23 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":1.127,"15min":1.2509999999999999},"min":{"1min":0.27200000000000002,"5min":0.24199999999999999,"15min":0.224},"max":{"1min":5.0899999999999999,"5min":5.468,"15min":34.167000000000002},"last":3.8839999999999999},{"interface":"front","average":{"1min":0.85199999999999998,"5min":1.036,"15min":1.2609999999999999},"min":{"1min":0.32100000000000001,"5min":0.28599999999999998,"15min":0.24199999999999999},"max":{"1min":4.4189999999999996,"5min":6.3360000000000003,"15min":36.051000000000002},"last":0.78900000000000003}]},{"osd":6,"last update":"Mon Feb 5 08:25:41 2024","interfaces":[{"interface":"back","average":{"1min":1.6439999999999999,"5min":1.5940000000000001,"15min":1.1759999999999999},"min":{"1min":0.33000000000000002,"5min":0.32600000000000001,"15min":0.26900000000000002},"max":{"1min":4.3879999999999999,"5min":6.375,"15min":6.375},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.3120000000000001,"15min":1.077},"min":{"1min":0.29099999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":3.3029999999999999,"5min":6.9690000000000003,"15min":6.9690000000000003},"last":5.1950000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:25:56 2024","interfaces":[{"interface":"back","average":{"1min":1.8220000000000001,"5min":1.2829999999999999,"15min":1.1830000000000001},"min":{"1min":0.34999999999999998,"5min":0.32100000000000001,"15min":0.307},"max":{"1min":5.8689999999999998,"5min":6.1280000000000001,"15min":12.683999999999999},"last":1.8779999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.2250000000000001,"15min":1.1160000000000001},"min":{"1min":0.373,"5min":0.35099999999999998,"15min":0.27900000000000003},"max":{"1min":3.2730000000000001,"5min":7.6820000000000004,"15min":7.6820000000000004},"last":1.758}]}]},{"osd":5,"up_from":42,"seq":180388626710,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":955852,"kb_used_data":190276,"kb_used_omap":1,"kb_used_meta":765566,"kb_avail":92789300,"statfs":{"total":95995035648,"available":95016243200,"internally_reserved":0,"allocated":194842624,"data_stored":194531999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":783940037},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.93799999999999994,"15min":1.498},"min":{"1min":0.217,"5min":0.217,"15min":0.20799999999999999},"max":{"1min":2.96,"5min":6.4660000000000002,"15min":46.962000000000003},"last":1.996},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.996,"15min":1.462},"min":{"1min":0.27800000000000002,"5min":0.26600000000000001,"15min":0.24299999999999999},"max":{"1min":1.792,"5min":5.0819999999999999,"15min":47.067},"last":1.788}]},{"osd":1,"last update":"Mon Feb 5 08:26:12 2024","interfaces":[{"interface":"back","average":{"1min":3.141,"5min":2.4569999999999999,"15min":2.5920000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":26.98,"5min":26.98,"15min":26.98},"last":1.8360000000000001},{"interface":"front","average":{"1min":3.3799999999999999,"5min":2.423,"15min":2.5139999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":27.474,"5min":27.474,"15min":27.474},"last":1.9650000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":1.038,"15min":1.1659999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":3.5910000000000002,"5min":7.6600000000000001,"15min":17.09},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.123,"15min":1.3029999999999999},"min":{"1min":0.311,"5min":0.26100000000000001,"15min":0.25600000000000001},"max":{"1min":3.9359999999999999,"5min":6.6440000000000001,"15min":17.515999999999998},"last":2.0619999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:26:04 2024","interfaces":[{"interface":"back","average":{"1min":4.7249999999999996,"5min":1.9139999999999999,"15min":1.5600000000000001},"min":{"1min":0.307,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":27.725000000000001,"5min":27.725000000000001,"15min":46.569000000000003},"last":0.68700000000000006},{"interface":"front","average":{"1min":4.3120000000000003,"5min":1.7330000000000001,"15min":1.6200000000000001},"min":{"1min":0.307,"5min":0.28699999999999998,"15min":0.28100000000000003},"max":{"1min":27.241,"5min":27.241,"15min":47.820999999999998},"last":2.0179999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":1.008,"15min":1.0289999999999999},"min":{"1min":0.26300000000000001,"5min":0.25900000000000001,"15min":0.23699999999999999},"max":{"1min":2.7770000000000001,"5min":7.3609999999999998,"15min":7.7389999999999999},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.056,"5min":1.24,"15min":1.087},"min":{"1min":0.36299999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.9020000000000001,"5min":7.609,"15min":7.609},"last":3.2509999999999999}]},{"osd":6,"last update":"Mon Feb 5 08:25:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.5209999999999999,"15min":1.228},"min":{"1min":0.35299999999999998,"5min":0.29399999999999998,"15min":0.26600000000000001},"max":{"1min":5.1989999999999998,"5min":30.581,"15min":30.581},"last":0.624},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.524,"15min":1.2150000000000001},"min":{"1min":0.48599999999999999,"5min":0.32900000000000001,"15min":0.316},"max":{"1min":3.6280000000000001,"5min":30.620999999999999,"15min":30.620999999999999},"last":3.3660000000000001}]},{"osd":7,"last update":"Mon Feb 5 08:26:04 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.4179999999999999,"15min":1.3420000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.28799999999999998},"max":{"1min":4.7960000000000003,"5min":34.695,"15min":34.695},"last":2.0409999999999999},{"interface":"front","average":{"1min":0.748,"5min":1.3959999999999999,"15min":1.2669999999999999},"min":{"1min":0.34000000000000002,"5min":0.32300000000000001,"15min":0.308},"max":{"1min":2.4039999999999999,"5min":34.514000000000003,"15min":34.514000000000003},"last":1.7589999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":203882496,"data_stored":203882496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":521207808,"data_stored":521207808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":267325440,"data_stored":267321368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":311099392,"data_stored":311099392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":361828352,"data_stored":361824280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":175308800,"data_stored":175308800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":364974080,"data_stored":364970008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":267780096,"data_stored":267780096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:26:45.200 INFO:tasks.thrashosds.thrasher:Setting 1.1 to [3, 1, 0, 6, 7, 4] 2024-02-05T08:26:45.200 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.1', '3', '1', '0', '6', '7', '4'] 2024-02-05T08:26:45.201 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pg-upmap-items 1.1 3 1 0 6 7 4 2024-02-05T08:26:45.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:45.284 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:45.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:45.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:45.439 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:45.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:45.515 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:45 smithi196 ceph-mon[33018]: pgmap v987: 158 pgs: 158 active+clean; 381 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T08:26:45.515 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:45 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:45.515 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:45 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:26:45.515 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:45 smithi196 ceph-mon[33018]: 3.6 scrub starts 2024-02-05T08:26:45.515 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:45 smithi196 ceph-mon[33018]: 3.6 scrub ok 2024-02-05T08:26:45.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:45 smithi134 ceph-mon[30086]: pgmap v987: 158 pgs: 158 active+clean; 381 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T08:26:45.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:45 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:45.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:45 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:26:45.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:45 smithi134 ceph-mon[30086]: 3.6 scrub starts 2024-02-05T08:26:45.529 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:45 smithi134 ceph-mon[30086]: 3.6 scrub ok 2024-02-05T08:26:45.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:45 smithi134 ceph-mon[39278]: pgmap v987: 158 pgs: 158 active+clean; 381 MiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T08:26:45.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:45 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:45.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:45 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:26:45.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:45 smithi134 ceph-mon[39278]: 3.6 scrub starts 2024-02-05T08:26:45.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:45 smithi134 ceph-mon[39278]: 3.6 scrub ok 2024-02-05T08:26:45.541 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:45.590 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:45.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:45.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:45.692 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:45.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:45.744 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:45.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:45.845 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:45.890 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:45.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:45.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:46.032 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:46.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:46.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:46.134 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:46.166 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:46.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:46.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:46.268 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:46.305 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:46.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:46.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:46.348 INFO:teuthology.orchestra.run.smithi196.stdout:ERROR: (22) Invalid argument 2024-02-05T08:26:46.349 INFO:teuthology.orchestra.run.smithi196.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:26:46.407 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:46.442 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:46.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:46.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[30086]: 3.8 scrub starts 2024-02-05T08:26:46.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[30086]: 3.8 scrub ok 2024-02-05T08:26:46.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[30086]: 3.1a scrub starts 2024-02-05T08:26:46.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[30086]: 3.1a scrub ok 2024-02-05T08:26:46.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[30086]: 6.8 scrub starts 2024-02-05T08:26:46.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[30086]: 6.8 scrub ok 2024-02-05T08:26:46.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[30086]: 4.6 deep-scrub starts 2024-02-05T08:26:46.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[30086]: 4.6 deep-scrub ok 2024-02-05T08:26:46.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:26:46.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:26:46.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:26:46.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[39278]: 3.8 scrub starts 2024-02-05T08:26:46.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[39278]: 3.8 scrub ok 2024-02-05T08:26:46.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[39278]: 3.1a scrub starts 2024-02-05T08:26:46.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[39278]: 3.1a scrub ok 2024-02-05T08:26:46.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[39278]: 6.8 scrub starts 2024-02-05T08:26:46.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[39278]: 6.8 scrub ok 2024-02-05T08:26:46.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[39278]: 4.6 deep-scrub starts 2024-02-05T08:26:46.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[39278]: 4.6 deep-scrub ok 2024-02-05T08:26:46.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:26:46.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:26:46.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:46 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:26:46.543 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:46 smithi196 ceph-mon[33018]: 3.8 scrub starts 2024-02-05T08:26:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:46 smithi196 ceph-mon[33018]: 3.8 scrub ok 2024-02-05T08:26:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:46 smithi196 ceph-mon[33018]: 3.1a scrub starts 2024-02-05T08:26:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:46 smithi196 ceph-mon[33018]: 3.1a scrub ok 2024-02-05T08:26:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:46 smithi196 ceph-mon[33018]: 6.8 scrub starts 2024-02-05T08:26:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:46 smithi196 ceph-mon[33018]: 6.8 scrub ok 2024-02-05T08:26:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:46 smithi196 ceph-mon[33018]: 4.6 deep-scrub starts 2024-02-05T08:26:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:46 smithi196 ceph-mon[33018]: 4.6 deep-scrub ok 2024-02-05T08:26:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:46 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:26:46.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:46 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:26:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:46 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:26:46.569 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:46.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:46.671 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:46.702 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:26:46.719 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:46.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:46.741 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:26:46.785 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:26:46.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:46.829 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:26:46.864 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:26:46.865 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:46.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:46.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:46.907 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:26:46.952 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:26:46.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:46.998 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:26:47.001 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:47.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:47.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:26:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:26:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:26:47.002 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:26:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:26:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:26:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:26:47.003 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:26:47.103 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:47.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:47.136 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:47.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:47.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T08:26:47.238 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:47.269 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:47.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:47.370 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:47.405 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:47.409 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:47.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:47.511 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:47.567 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:47.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:47.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:47.578 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T08:26:47.669 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:47.710 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:47.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:47.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:47.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:47 smithi134 ceph-mon[30086]: pgmap v988: 158 pgs: 158 active+clean; 360 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-02-05T08:26:47.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:47 smithi134 ceph-mon[30086]: 6.d scrub starts 2024-02-05T08:26:47.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:47 smithi134 ceph-mon[30086]: 6.d scrub ok 2024-02-05T08:26:47.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:47 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/306010030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:47.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:47 smithi134 ceph-mon[39278]: pgmap v988: 158 pgs: 158 active+clean; 360 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-02-05T08:26:47.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:47 smithi134 ceph-mon[39278]: 6.d scrub starts 2024-02-05T08:26:47.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:47 smithi134 ceph-mon[39278]: 6.d scrub ok 2024-02-05T08:26:47.741 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:47 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/306010030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:47.812 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:47.847 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:47.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:47.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:47.995 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:47.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:48.004 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:26:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:26:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:26:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:47 smithi196 ceph-mon[33018]: pgmap v988: 158 pgs: 158 active+clean; 360 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-02-05T08:26:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:47 smithi196 ceph-mon[33018]: 6.d scrub starts 2024-02-05T08:26:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:47 smithi196 ceph-mon[33018]: 6.d scrub ok 2024-02-05T08:26:48.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:47 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/306010030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:48.098 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:48.145 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:48.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:48.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:48.292 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:48.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:48.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:48.394 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:48.430 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Removed 135099 objects 2024-02-05T08:26:48.431 INFO:tasks.radosbench.radosbench.0.smithi134.stdout:Clean up completed and total clean up time :118.678 2024-02-05T08:26:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:48.432 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:48.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:48.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:48.556 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:48.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:48.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:48.578 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is set 2024-02-05T08:26:48.657 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:48.682 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:48.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:48.783 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:48.808 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:48.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:48.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:48 smithi196 ceph-mon[33018]: 6.12 scrub starts 2024-02-05T08:26:48.893 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:48 smithi196 ceph-mon[33018]: 6.12 scrub ok 2024-02-05T08:26:48.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:48 smithi196 ceph-mon[33018]: 11.c scrub starts 2024-02-05T08:26:48.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:48 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1129124036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:26:48.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:48 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:26:48.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:48 smithi196 ceph-mon[33018]: 3.5 scrub starts 2024-02-05T08:26:48.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:48 smithi196 ceph-mon[33018]: 3.5 scrub ok 2024-02-05T08:26:48.894 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:48 smithi196 ceph-mon[33018]: 11.c scrub ok 2024-02-05T08:26:48.908 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:48.932 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:48.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:48.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:48.936 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[30086]: 6.12 scrub starts 2024-02-05T08:26:48.937 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[30086]: 6.12 scrub ok 2024-02-05T08:26:48.937 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[30086]: 11.c scrub starts 2024-02-05T08:26:48.937 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1129124036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:26:48.937 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:26:48.937 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[30086]: 3.5 scrub starts 2024-02-05T08:26:48.937 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[30086]: 3.5 scrub ok 2024-02-05T08:26:48.937 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[30086]: 11.c scrub ok 2024-02-05T08:26:48.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[39278]: 6.12 scrub starts 2024-02-05T08:26:48.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[39278]: 6.12 scrub ok 2024-02-05T08:26:48.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[39278]: 11.c scrub starts 2024-02-05T08:26:48.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1129124036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:26:48.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T08:26:48.938 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[39278]: 3.5 scrub starts 2024-02-05T08:26:48.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[39278]: 3.5 scrub ok 2024-02-05T08:26:48.939 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:48 smithi134 ceph-mon[39278]: 11.c scrub ok 2024-02-05T08:26:49.034 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:49.059 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:49.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:49.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:49.184 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:49.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:49.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:49.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:49.311 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:49.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:49.413 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:49.438 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:49.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:49.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:49.540 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:49.567 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:49.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:49.669 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:49.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:49.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:49.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:49 smithi196 ceph-mon[33018]: pgmap v989: 158 pgs: 158 active+clean; 317 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T08:26:49.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:49 smithi196 ceph-mon[33018]: 4.1 scrub starts 2024-02-05T08:26:49.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:49 smithi196 ceph-mon[33018]: 4.1 scrub ok 2024-02-05T08:26:49.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:49 smithi196 ceph-mon[33018]: 1.3 scrub starts 2024-02-05T08:26:49.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:49 smithi196 ceph-mon[33018]: 1.3 scrub ok 2024-02-05T08:26:49.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:49 smithi196 ceph-mon[33018]: 5.b scrub starts 2024-02-05T08:26:49.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:49 smithi196 ceph-mon[33018]: 5.b scrub ok 2024-02-05T08:26:49.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:49 smithi196 ceph-mon[33018]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:26:49.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:49 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:26:49.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:49 smithi196 ceph-mon[33018]: osdmap e426: 8 total, 8 up, 8 in 2024-02-05T08:26:49.795 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:49.820 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:49.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:49.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:49.924 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T08:26:49.953 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:49.954 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[30086]: pgmap v989: 158 pgs: 158 active+clean; 317 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T08:26:49.954 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[30086]: 4.1 scrub starts 2024-02-05T08:26:49.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[30086]: 4.1 scrub ok 2024-02-05T08:26:49.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[30086]: 1.3 scrub starts 2024-02-05T08:26:49.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[30086]: 1.3 scrub ok 2024-02-05T08:26:49.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[30086]: 5.b scrub starts 2024-02-05T08:26:49.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[30086]: 5.b scrub ok 2024-02-05T08:26:49.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[30086]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:26:49.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:26:49.955 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[30086]: osdmap e426: 8 total, 8 up, 8 in 2024-02-05T08:26:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:49.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:49.956 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[39278]: pgmap v989: 158 pgs: 158 active+clean; 317 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T08:26:49.956 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[39278]: 4.1 scrub starts 2024-02-05T08:26:49.956 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[39278]: 4.1 scrub ok 2024-02-05T08:26:49.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[39278]: 1.3 scrub starts 2024-02-05T08:26:49.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[39278]: 1.3 scrub ok 2024-02-05T08:26:49.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[39278]: 5.b scrub starts 2024-02-05T08:26:49.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[39278]: 5.b scrub ok 2024-02-05T08:26:49.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[39278]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:26:49.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T08:26:49.957 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:49 smithi134 ceph-mon[39278]: osdmap e426: 8 total, 8 up, 8 in 2024-02-05T08:26:50.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:50.072 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-02-05T08:26:50.072 INFO:teuthology.orchestra.run.smithi196.stdout: "size": 20, 2024-02-05T08:26:50.072 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 600, 2024-02-05T08:26:50.072 INFO:teuthology.orchestra.run.smithi196.stdout: "ops": [ 2024-02-05T08:26:50.072 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.072 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.15714.0:225861 10.c 10:3bf43d57:::benchmark_data_smithi134_174861_object94399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-02-05T08:26:50.073 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:22:08.077029+0000", 2024-02-05T08:26:50.073 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 281.99285945100002, 2024-02-05T08:26:50.073 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.163278091, 2024-02-05T08:26:50.073 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.073 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "reached pg", 2024-02-05T08:26:50.073 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.073 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.15714", 2024-02-05T08:26:50.073 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/290375505", 2024-02-05T08:26:50.074 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 225861 2024-02-05T08:26:50.074 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.074 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.074 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.074 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.074 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077029+0000", 2024-02-05T08:26:50.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.075 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.075 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.075 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077029+0000", 2024-02-05T08:26:50.075 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.075 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077031+0000", 2024-02-05T08:26:50.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6080000000000001e-06 2024-02-05T08:26:50.076 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.076 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.076 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.076 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077038+0000", 2024-02-05T08:26:50.076 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1720000000000001e-06 2024-02-05T08:26:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077057+0000", 2024-02-05T08:26:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8802000000000001e-05 2024-02-05T08:26:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.077 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077066+0000", 2024-02-05T08:26:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.4660000000000005e-06 2024-02-05T08:26:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077142+0000", 2024-02-05T08:26:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5902000000000006e-05 2024-02-05T08:26:50.078 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077281+0000", 2024-02-05T08:26:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000138897 2024-02-05T08:26:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 0,4", 2024-02-05T08:26:50.079 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.077415+0000", 2024-02-05T08:26:50.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013382800000000001 2024-02-05T08:26:50.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:50.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.083926+0000", 2024-02-05T08:26:50.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0065109119999999998 2024-02-05T08:26:50.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.080 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.080 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.098019+0000", 2024-02-05T08:26:50.080 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.014093022 2024-02-05T08:26:50.080 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.080 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:26:50.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.098035+0000", 2024-02-05T08:26:50.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.5625e-05 2024-02-05T08:26:50.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.081 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.093969+0000", 2024-02-05T08:26:50.081 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.99593420600000004 2024-02-05T08:26:50.081 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.081 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.081 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.093985+0000", 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6659000000000001e-05 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.240212+0000", 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.146226985 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.240307+0000", 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5006999999999993e-05 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.082 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.083 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.083 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.083 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.083 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24388.0:1033 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e351)", 2024-02-05T08:26:50.083 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:22:08.178524+0000", 2024-02-05T08:26:50.083 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 281.89136442400002, 2024-02-05T08:26:50.083 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.926063401, 2024-02-05T08:26:50.083 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.083 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T08:26:50.083 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.083 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24388", 2024-02-05T08:26:50.083 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/2822725514", 2024-02-05T08:26:50.084 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1033 2024-02-05T08:26:50.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.084 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.084 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178524+0000", 2024-02-05T08:26:50.084 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.084 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.084 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.084 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178524+0000", 2024-02-05T08:26:50.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178526+0000", 2024-02-05T08:26:50.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.978e-06 2024-02-05T08:26:50.085 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.085 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.085 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.085 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178538+0000", 2024-02-05T08:26:50.085 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2140000000000001e-05 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178555+0000", 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.6260999999999999e-05 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178564+0000", 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2550000000000003e-06 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178645+0000", 2024-02-05T08:26:50.086 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1417999999999999e-05 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:08.178654+0000", 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9819999999999997e-06 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.094408+0000", 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.91575361200000005 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:09.094417+0000", 2024-02-05T08:26:50.087 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.2059999999999996e-06 2024-02-05T08:26:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.104208+0000", 2024-02-05T08:26:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.009791187 2024-02-05T08:26:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.104367+0000", 2024-02-05T08:26:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000158846 2024-02-05T08:26:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.088 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.089 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:22:10.104588+0000", 2024-02-05T08:26:50.089 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000220516 2024-02-05T08:26:50.089 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.089 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.089 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.089 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.089 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:183180 11.f 11:fc5d1cd8:::benchmark_data_smithi134_205380_object48078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-02-05T08:26:50.089 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:25:26.069392+0000", 2024-02-05T08:26:50.089 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 84.000496502000004, 2024-02-05T08:26:50.090 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.061094247999999997, 2024-02-05T08:26:50.090 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.090 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.090 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.090 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:50.090 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:50.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:50.093 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 183180 2024-02-05T08:26:50.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.093 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.093 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.093 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.069392+0000", 2024-02-05T08:26:50.093 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.093 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.094 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.069392+0000", 2024-02-05T08:26:50.094 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.094 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.094 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.094 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.069393+0000", 2024-02-05T08:26:50.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9999999999999997e-07 2024-02-05T08:26:50.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.095 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.095 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.069396+0000", 2024-02-05T08:26:50.095 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5530000000000001e-06 2024-02-05T08:26:50.095 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.095 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.069405+0000", 2024-02-05T08:26:50.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9139999999999997e-06 2024-02-05T08:26:50.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.069411+0000", 2024-02-05T08:26:50.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1299999999999998e-06 2024-02-05T08:26:50.096 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.096 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.096 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.096 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.073022+0000", 2024-02-05T08:26:50.096 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0036109789999999998 2024-02-05T08:26:50.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.073136+0000", 2024-02-05T08:26:50.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011333 2024-02-05T08:26:50.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.097 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:26:50.097 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.073266+0000", 2024-02-05T08:26:50.097 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000130343 2024-02-05T08:26:50.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.097 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.075356+0000", 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020895409999999999 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.077383+0000", 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020274189999999999 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.130389+0000", 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.053005442999999999 2024-02-05T08:26:50.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.130447+0000", 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8378000000000002e-05 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.130487+0000", 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9617999999999999e-05 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.099 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:183187 11.f 11:f5d90ec4:::benchmark_data_smithi134_205380_object48085:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-02-05T08:26:50.100 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:25:26.074657+0000", 2024-02-05T08:26:50.100 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 83.995231609000001, 2024-02-05T08:26:50.100 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.055958967999999998, 2024-02-05T08:26:50.100 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.100 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.100 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.100 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:50.100 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:50.100 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 183187 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.074657+0000", 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.074657+0000", 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.101 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.074658+0000", 2024-02-05T08:26:50.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.127e-06 2024-02-05T08:26:50.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.074663+0000", 2024-02-05T08:26:50.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0010000000000001e-06 2024-02-05T08:26:50.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.102 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.102 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.102 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.074682+0000", 2024-02-05T08:26:50.102 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9157999999999999e-05 2024-02-05T08:26:50.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.074689+0000", 2024-02-05T08:26:50.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6120000000000004e-06 2024-02-05T08:26:50.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.103 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.103 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.103 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.075384+0000", 2024-02-05T08:26:50.103 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00069538900000000003 2024-02-05T08:26:50.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.075467+0000", 2024-02-05T08:26:50.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2694000000000005e-05 2024-02-05T08:26:50.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:26:50.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.075585+0000", 2024-02-05T08:26:50.104 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011801199999999999 2024-02-05T08:26:50.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.104 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.104 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.104 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.077415+0000", 2024-02-05T08:26:50.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0018298489999999999 2024-02-05T08:26:50.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:50.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.077491+0000", 2024-02-05T08:26:50.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.551e-05 2024-02-05T08:26:50.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.105 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.105 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.130542+0000", 2024-02-05T08:26:50.105 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.053051955999999997 2024-02-05T08:26:50.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.105 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.130586+0000", 2024-02-05T08:26:50.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3084000000000003e-05 2024-02-05T08:26:50.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.106 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.106 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.106 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:26.130616+0000", 2024-02-05T08:26:50.106 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.0576e-05 2024-02-05T08:26:50.106 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.24388.0:1249 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e405)", 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:25:48.264489+0000", 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 61.805399604000002, 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88477393900000001, 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.24388", 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.196:0/2822725514", 2024-02-05T08:26:50.107 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 1249 2024-02-05T08:26:50.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.108 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.264489+0000", 2024-02-05T08:26:50.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.108 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.108 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.264489+0000", 2024-02-05T08:26:50.108 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.108 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.264491+0000", 2024-02-05T08:26:50.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9069999999999999e-06 2024-02-05T08:26:50.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.264505+0000", 2024-02-05T08:26:50.109 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4008e-05 2024-02-05T08:26:50.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.109 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.109 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.109 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.264527+0000", 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2234000000000001e-05 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.264539+0000", 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1732e-05 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.264651+0000", 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011202200000000001 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.110 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for peered", 2024-02-05T08:26:50.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:48.264664+0000", 2024-02-05T08:26:50.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.2921999999999999e-05 2024-02-05T08:26:50.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.111 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.111 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:49.148963+0000", 2024-02-05T08:26:50.111 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.88429884700000005 2024-02-05T08:26:50.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.111 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:49.149115+0000", 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00015192 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:49.149263+0000", 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014834700000000001 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.112 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:212144 11.f 11:f9634c48:::benchmark_data_smithi134_205380_object77042:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:25:50.256220+0000", 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 59.813668515000003, 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.056568267999999998, 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 212144 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.113 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.256220+0000", 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.256220+0000", 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.256221+0000", 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.9299999999999997e-07 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.256224+0000", 2024-02-05T08:26:50.114 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4450000000000001e-06 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.256235+0000", 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.022e-05 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.256242+0000", 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.695e-06 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.115 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.256300+0000", 2024-02-05T08:26:50.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7738000000000002e-05 2024-02-05T08:26:50.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.256414+0000", 2024-02-05T08:26:50.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000113538 2024-02-05T08:26:50.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.116 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.116 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:26:50.116 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.256477+0000", 2024-02-05T08:26:50.116 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3226999999999996e-05 2024-02-05T08:26:50.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.259188+0000", 2024-02-05T08:26:50.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002711607 2024-02-05T08:26:50.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.117 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:50.117 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.259299+0000", 2024-02-05T08:26:50.117 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000110055 2024-02-05T08:26:50.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.117 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.312728+0000", 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.053429941000000002 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.312764+0000", 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.5246000000000001e-05 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:50.312789+0000", 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.4862999999999999e-05 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.118 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:215270 11.f 11:f05a386a:::benchmark_data_smithi134_205380_object80168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:25:53.147640+0000", 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 56.922248799999998, 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.055224669999999997, 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 215270 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.119 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.147640+0000", 2024-02-05T08:26:50.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.147640+0000", 2024-02-05T08:26:50.120 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.120 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.120 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.120 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.120 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.147641+0000", 2024-02-05T08:26:50.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.3399999999999998e-07 2024-02-05T08:26:50.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.147645+0000", 2024-02-05T08:26:50.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8139999999999998e-06 2024-02-05T08:26:50.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.121 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.121 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.147658+0000", 2024-02-05T08:26:50.121 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3535e-05 2024-02-05T08:26:50.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.121 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.147665+0000", 2024-02-05T08:26:50.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.2169999999999997e-06 2024-02-05T08:26:50.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.122 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.122 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.149419+0000", 2024-02-05T08:26:50.122 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0017538950000000001 2024-02-05T08:26:50.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.122 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.123 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.149520+0000", 2024-02-05T08:26:50.123 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000101042 2024-02-05T08:26:50.123 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.123 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.123 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:26:50.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.149617+0000", 2024-02-05T08:26:50.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.6977000000000003e-05 2024-02-05T08:26:50.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:50.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.154385+0000", 2024-02-05T08:26:50.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0047674809999999996 2024-02-05T08:26:50.124 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.124 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.124 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.124 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.162068+0000", 2024-02-05T08:26:50.124 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0076827909999999996 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.202767+0000", 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.040698956000000001 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.202823+0000", 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6322e-05 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.125 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.202865+0000", 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1805999999999999e-05 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:215271 11.f 11:f3e80c95:::benchmark_data_smithi134_205380_object80169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:25:53.148236+0000", 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 56.921652883999997, 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.059590434999999997, 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.126 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:50.127 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:50.127 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 215271 2024-02-05T08:26:50.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.127 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.127 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.127 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.148236+0000", 2024-02-05T08:26:50.127 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.127 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.148236+0000", 2024-02-05T08:26:50.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.148237+0000", 2024-02-05T08:26:50.128 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.16e-07 2024-02-05T08:26:50.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.128 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.128 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.128 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.148241+0000", 2024-02-05T08:26:50.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.2259999999999999e-06 2024-02-05T08:26:50.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.148256+0000", 2024-02-05T08:26:50.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.4868000000000001e-05 2024-02-05T08:26:50.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.129 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.129 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.148263+0000", 2024-02-05T08:26:50.129 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5320000000000001e-06 2024-02-05T08:26:50.129 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.129 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.152074+0000", 2024-02-05T08:26:50.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.003810992 2024-02-05T08:26:50.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.152166+0000", 2024-02-05T08:26:50.130 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.1802e-05 2024-02-05T08:26:50.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.130 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.130 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:26:50.130 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.152234+0000", 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7770999999999999e-05 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.154517+0000", 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002283541 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.162118+0000", 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0076002659999999996 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.131 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.207732+0000", 2024-02-05T08:26:50.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.045614413999999999 2024-02-05T08:26:50.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.132 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.132 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.207788+0000", 2024-02-05T08:26:50.132 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6057999999999998e-05 2024-02-05T08:26:50.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.132 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.207826+0000", 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.8148999999999997e-05 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:215281 11.f 11:f619bf1f:::benchmark_data_smithi134_205380_object80179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:25:53.159454+0000", 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 56.910434967, 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.055754597000000003, 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.133 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 215281 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.159454+0000", 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.159454+0000", 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.134 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.159455+0000", 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.466e-06 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.159461+0000", 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.2259999999999996e-06 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.159471+0000", 2024-02-05T08:26:50.135 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0613e-05 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.159478+0000", 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3479999999999998e-06 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.159742+0000", 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00026330700000000001 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.136 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.159887+0000", 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014558099999999999 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.159999+0000", 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011175000000000001 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.162168+0000", 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0021687249999999998 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.137 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.167617+0000", 2024-02-05T08:26:50.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0054495519999999999 2024-02-05T08:26:50.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.138 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.138 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.215109+0000", 2024-02-05T08:26:50.138 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.047492000999999999 2024-02-05T08:26:50.138 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.138 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.215169+0000", 2024-02-05T08:26:50.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.9098999999999998e-05 2024-02-05T08:26:50.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.139 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.139 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:25:53.215208+0000", 2024-02-05T08:26:50.139 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9928999999999997e-05 2024-02-05T08:26:50.139 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.139 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.139 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.139 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.139 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:226349 11.f 11:fd8fdd86:::benchmark_data_smithi134_205380_object91247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:03.767549+0000", 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 46.302339443999998, 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.078096328000000007, 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 226349 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.140 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.767549+0000", 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.767549+0000", 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.767551+0000", 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1850000000000001e-06 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.141 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.767555+0000", 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9899999999999999e-06 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.767568+0000", 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3122000000000001e-05 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.767574+0000", 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1109999999999996e-06 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.142 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.767633+0000", 2024-02-05T08:26:50.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8971999999999997e-05 2024-02-05T08:26:50.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.143 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.143 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.767773+0000", 2024-02-05T08:26:50.143 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013994300000000001 2024-02-05T08:26:50.143 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.143 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:26:50.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.767900+0000", 2024-02-05T08:26:50.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00012742400000000001 2024-02-05T08:26:50.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.144 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:50.144 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.770629+0000", 2024-02-05T08:26:50.144 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0027291149999999998 2024-02-05T08:26:50.144 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.144 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.771735+0000", 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0011062540000000001 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.845574+0000", 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.073839025000000003 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.845614+0000", 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9453000000000003e-05 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.145 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:03.845646+0000", 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.1733999999999998e-05 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25820.0:244382 11.3 e419/367 11:c28b9872:::benchmark_data_smithi134_205380_object109280:head v 419'15489, mlcod=419'15487)", 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:23.594934+0000", 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 26.474954404000002, 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17369327200000001, 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.146 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.594934+0000", 2024-02-05T08:26:50.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.594934+0000", 2024-02-05T08:26:50.147 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.147 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.147 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.147 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.147 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.594935+0000", 2024-02-05T08:26:50.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1299999999999999e-07 2024-02-05T08:26:50.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.594942+0000", 2024-02-05T08:26:50.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.832e-06 2024-02-05T08:26:50.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.148 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.148 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.148 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.594956+0000", 2024-02-05T08:26:50.148 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3709e-05 2024-02-05T08:26:50.148 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.149 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.149 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.594963+0000", 2024-02-05T08:26:50.149 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.1169999999999998e-06 2024-02-05T08:26:50.149 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.149 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.596798+0000", 2024-02-05T08:26:50.150 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0018355369999999999 2024-02-05T08:26:50.150 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.150 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.150 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.150 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.596821+0000", 2024-02-05T08:26:50.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2986e-05 2024-02-05T08:26:50.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.768590+0000", 2024-02-05T08:26:50.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.17176866099999999 2024-02-05T08:26:50.151 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.151 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.151 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.151 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.768628+0000", 2024-02-05T08:26:50.151 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7716999999999998e-05 2024-02-05T08:26:50.151 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.152 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.152 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.152 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.152 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.152 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25820.0:244401 11.3 e419/367 11:ca457a02:::benchmark_data_smithi134_205380_object109299:head v 419'15490, mlcod=419'15487)", 2024-02-05T08:26:50.153 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:23.614131+0000", 2024-02-05T08:26:50.153 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 26.455757982000002, 2024-02-05T08:26:50.153 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.154528523, 2024-02-05T08:26:50.153 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.153 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.153 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.154 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.614131+0000", 2024-02-05T08:26:50.154 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.154 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.154 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.154 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.614131+0000", 2024-02-05T08:26:50.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.614131+0000", 2024-02-05T08:26:50.155 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9800000000000004e-07 2024-02-05T08:26:50.155 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.155 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.155 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.155 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.614136+0000", 2024-02-05T08:26:50.156 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7600000000000002e-06 2024-02-05T08:26:50.156 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.156 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.156 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.156 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.614144+0000", 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5089999999999996e-06 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.614149+0000", 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7980000000000002e-06 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.614283+0000", 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00013385099999999999 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.157 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.614305+0000", 2024-02-05T08:26:50.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.2089000000000001e-05 2024-02-05T08:26:50.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.768631+0000", 2024-02-05T08:26:50.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.15432549000000001 2024-02-05T08:26:50.158 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.158 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.158 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.158 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.768659+0000", 2024-02-05T08:26:50.158 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.8527999999999999e-05 2024-02-05T08:26:50.158 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25820.0:244421 11.3 e419/367 11:c494d292:::benchmark_data_smithi134_205380_object109319:head v 419'15491, mlcod=419'15487)", 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:23.630498+0000", 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 26.439391020999999, 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13818719099999999, 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.630498+0000", 2024-02-05T08:26:50.159 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.630498+0000", 2024-02-05T08:26:50.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.160 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.630498+0000", 2024-02-05T08:26:50.160 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.0399999999999995e-07 2024-02-05T08:26:50.160 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.160 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.160 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.630504+0000", 2024-02-05T08:26:50.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6659999999999996e-06 2024-02-05T08:26:50.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.161 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.161 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.161 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.630513+0000", 2024-02-05T08:26:50.161 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.9439999999999994e-06 2024-02-05T08:26:50.161 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.630519+0000", 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.2990000000000002e-06 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.630778+0000", 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00025881100000000001 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.630812+0000", 2024-02-05T08:26:50.162 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.3662999999999999e-05 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.768661+0000", 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.13784960299999999 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.768685+0000", 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.3501000000000001e-05 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.163 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.164 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25820.0:244438 11.3 e419/367 11:c3cc062e:::benchmark_data_smithi134_205380_object109336:head v 419'15492, mlcod=419'15487)", 2024-02-05T08:26:50.164 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:23.650957+0000", 2024-02-05T08:26:50.164 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 26.418932164000001, 2024-02-05T08:26:50.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.117752234, 2024-02-05T08:26:50.164 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.164 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.164 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.164 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.164 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.650957+0000", 2024-02-05T08:26:50.164 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.164 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.164 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.650957+0000", 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.650957+0000", 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3000000000000001e-07 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.650963+0000", 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8749999999999997e-06 2024-02-05T08:26:50.165 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.650972+0000", 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.6829999999999997e-06 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.650978+0000", 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.1879999999999999e-06 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.651169+0000", 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00019155600000000001 2024-02-05T08:26:50.166 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.651189+0000", 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.9483999999999998e-05 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.768687+0000", 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.11749807 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.768709+0000", 2024-02-05T08:26:50.167 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.1848000000000002e-05 2024-02-05T08:26:50.168 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.168 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.168 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.168 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.168 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.168 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "rep_scrubmap(6.3 e187 from shard 6)", 2024-02-05T08:26:50.168 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:23.714042+0000", 2024-02-05T08:26:50.168 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 26.355846313000001, 2024-02-05T08:26:50.168 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.055019272000000001, 2024-02-05T08:26:50.168 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "started", 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.714042+0000", 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.714042+0000", 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.169 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.714043+0000", 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7999999999999995e-07 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.714047+0000", 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.1409999999999998e-06 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.714057+0000", 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 9.5959999999999993e-06 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.170 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.714063+0000", 2024-02-05T08:26:50.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.8860000000000001e-06 2024-02-05T08:26:50.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.769037+0000", 2024-02-05T08:26:50.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.054974124999999999 2024-02-05T08:26:50.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.171 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.171 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.171 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.769047+0000", 2024-02-05T08:26:50.171 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.0034000000000001e-05 2024-02-05T08:26:50.171 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.172 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.172 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.172 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.769062+0000", 2024-02-05T08:26:50.172 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:50.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:50.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.491e-05 2024-02-05T08:26:50.176 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.176 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.176 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.176 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.176 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.176 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_repop(client.25820.0:244514 11.3 e419/367 11:c73b0421:::benchmark_data_smithi134_205380_object109412:head v 419'15493, mlcod=419'15487)", 2024-02-05T08:26:50.176 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:23.721384+0000", 2024-02-05T08:26:50.176 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 26.348505251999999, 2024-02-05T08:26:50.176 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.057012710000000001, 2024-02-05T08:26:50.176 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.176 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.721384+0000", 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.721384+0000", 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.721384+0000", 2024-02-05T08:26:50.177 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.0100000000000005e-07 2024-02-05T08:26:50.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.721391+0000", 2024-02-05T08:26:50.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.3829999999999997e-06 2024-02-05T08:26:50.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.178 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.721399+0000", 2024-02-05T08:26:50.178 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.2509999999999994e-06 2024-02-05T08:26:50.178 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.178 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.178 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.721404+0000", 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6949999999999997e-06 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.768732+0000", 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.047327064000000002 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.768750+0000", 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.8822999999999999e-05 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.179 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.778360+0000", 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0096092250000000008 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:23.778396+0000", 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.6668000000000002e-05 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:244985 11.4 11:26b95593:::benchmark_data_smithi134_205380_object109883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-02-05T08:26:50.180 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:24.129977+0000", 2024-02-05T08:26:50.181 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.939912021000001, 2024-02-05T08:26:50.181 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.059011347999999998, 2024-02-05T08:26:50.181 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.181 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.181 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.181 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:50.181 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:50.181 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 244985 2024-02-05T08:26:50.181 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.181 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.181 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.129977+0000", 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.129977+0000", 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.129978+0000", 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.034e-06 2024-02-05T08:26:50.182 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.129982+0000", 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.9770000000000002e-06 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.129989+0000", 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.3490000000000003e-06 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.129994+0000", 2024-02-05T08:26:50.183 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.0010000000000001e-06 2024-02-05T08:26:50.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.130038+0000", 2024-02-05T08:26:50.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.3381999999999999e-05 2024-02-05T08:26:50.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.184 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.184 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.130123+0000", 2024-02-05T08:26:50.184 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.5433999999999996e-05 2024-02-05T08:26:50.184 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.184 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,6", 2024-02-05T08:26:50.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.130229+0000", 2024-02-05T08:26:50.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000106209 2024-02-05T08:26:50.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:50.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.132285+0000", 2024-02-05T08:26:50.185 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0020561949999999998 2024-02-05T08:26:50.185 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.185 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.185 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.185 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.134226+0000", 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.001940944 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.188871+0000", 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.054645165000000002 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.188938+0000", 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6885999999999996e-05 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.186 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.188988+0000", 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.9771999999999997e-05 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:245434 11.f 11:fe729b31:::benchmark_data_smithi134_205380_object110332:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:24.627243+0000", 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 25.442645528, 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.052288011000000002, 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:50.187 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:50.188 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 245434 2024-02-05T08:26:50.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.188 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.627243+0000", 2024-02-05T08:26:50.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.188 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.188 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.188 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.188 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.627243+0000", 2024-02-05T08:26:50.188 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.627244+0000", 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.7000000000000004e-07 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.627247+0000", 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.4800000000000001e-06 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.627258+0000", 2024-02-05T08:26:50.189 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1011e-05 2024-02-05T08:26:50.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.627266+0000", 2024-02-05T08:26:50.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.943e-06 2024-02-05T08:26:50.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.190 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.190 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.190 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.628299+0000", 2024-02-05T08:26:50.190 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0010324380000000001 2024-02-05T08:26:50.190 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.628416+0000", 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000116884 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.628530+0000", 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011448200000000001 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.631193+0000", 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.002663187 2024-02-05T08:26:50.191 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.676185+0000", 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.044991759999999999 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.679431+0000", 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00324617 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.679485+0000", 2024-02-05T08:26:50.192 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.3677999999999997e-05 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:24.679531+0000", 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6307999999999998e-05 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:246469 11.f 11:f4ab6f5d:::benchmark_data_smithi134_205380_object111367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:25.643029+0000", 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.426860090000002, 2024-02-05T08:26:50.193 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.052664115999999997, 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 246469 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.643029+0000", 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.194 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.643029+0000", 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.643030+0000", 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.268e-06 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.643035+0000", 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.6809999999999997e-06 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.195 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.643046+0000", 2024-02-05T08:26:50.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.1005000000000001e-05 2024-02-05T08:26:50.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.643053+0000", 2024-02-05T08:26:50.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.7109999999999995e-06 2024-02-05T08:26:50.196 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.196 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.196 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.196 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.643091+0000", 2024-02-05T08:26:50.196 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7588e-05 2024-02-05T08:26:50.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.643166+0000", 2024-02-05T08:26:50.197 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 7.5119999999999999e-05 2024-02-05T08:26:50.197 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.197 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.197 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 1,2", 2024-02-05T08:26:50.197 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.643233+0000", 2024-02-05T08:26:50.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 6.6762999999999998e-05 2024-02-05T08:26:50.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.694934+0000", 2024-02-05T08:26:50.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.051701398000000003 2024-02-05T08:26:50.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.198 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.198 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:50.198 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.695149+0000", 2024-02-05T08:26:50.198 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000215243 2024-02-05T08:26:50.198 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.695598+0000", 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00044866300000000002 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.695656+0000", 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.7661999999999998e-05 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.695693+0000", 2024-02-05T08:26:50.199 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.7014000000000002e-05 2024-02-05T08:26:50.200 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.200 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.200 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.200 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.200 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.200 INFO:teuthology.orchestra.run.smithi196.stdout: "description": "osd_op(client.25820.0:246475 11.4 11:29bc7d48:::benchmark_data_smithi134_205380_object111373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-02-05T08:26:50.200 INFO:teuthology.orchestra.run.smithi196.stdout: "initiated_at": "2024-02-05T08:26:25.648167+0000", 2024-02-05T08:26:50.200 INFO:teuthology.orchestra.run.smithi196.stdout: "age": 24.421721351999999, 2024-02-05T08:26:50.200 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0523169, 2024-02-05T08:26:50.200 INFO:teuthology.orchestra.run.smithi196.stdout: "type_data": { 2024-02-05T08:26:50.200 INFO:teuthology.orchestra.run.smithi196.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:26:50.200 INFO:teuthology.orchestra.run.smithi196.stdout: "client_info": { 2024-02-05T08:26:50.200 INFO:teuthology.orchestra.run.smithi196.stdout: "client": "client.25820", 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: "tid": 246475 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: "events": [ 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "initiated", 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.648167+0000", 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "header_read", 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.648167+0000", 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.201 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "throttled", 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.648169+0000", 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.147e-06 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "all_read", 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.648173+0000", 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 4.7809999999999996e-06 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "dispatched", 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.692134+0000", 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.043960183 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.202 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "queued_for_pg", 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.692147+0000", 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 1.3298e-05 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "reached_pg", 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.692296+0000", 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00014894800000000001 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "started", 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.692431+0000", 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.000135518 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.203 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "waiting for subops from 2,6", 2024-02-05T08:26:50.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.692547+0000", 2024-02-05T08:26:50.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.00011576100000000001 2024-02-05T08:26:50.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.700358+0000", 2024-02-05T08:26:50.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 0.0078107530000000001 2024-02-05T08:26:50.204 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.204 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.204 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:26:50.204 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.700414+0000", 2024-02-05T08:26:50.204 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 5.6394999999999998e-05 2024-02-05T08:26:50.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "op_commit", 2024-02-05T08:26:50.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.700422+0000", 2024-02-05T08:26:50.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 8.1119999999999996e-06 2024-02-05T08:26:50.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.205 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "commit_sent", 2024-02-05T08:26:50.205 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.700455+0000", 2024-02-05T08:26:50.205 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 3.2780999999999997e-05 2024-02-05T08:26:50.205 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-02-05T08:26:50.205 INFO:teuthology.orchestra.run.smithi196.stdout: { 2024-02-05T08:26:50.206 INFO:teuthology.orchestra.run.smithi196.stdout: "event": "done", 2024-02-05T08:26:50.206 INFO:teuthology.orchestra.run.smithi196.stdout: "time": "2024-02-05T08:26:25.700484+0000", 2024-02-05T08:26:50.206 INFO:teuthology.orchestra.run.smithi196.stdout: "duration": 2.9223000000000001e-05 2024-02-05T08:26:50.206 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.206 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.206 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.206 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-02-05T08:26:50.206 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-02-05T08:26:50.206 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-02-05T08:26:50.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:50.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:50.298 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:50.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:50.399 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:50.423 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:50.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:50.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:50.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:50.551 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:50.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:50.652 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:50.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:50.677 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:50.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:50.778 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:50.802 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:50.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:50.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:50.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:50.913 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T08:26:50.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:50.928 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:50.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:50.984 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:51.029 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:51.055 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:51.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:51.156 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:51.180 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:51.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:51.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:51.227 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:50 smithi134 ceph-mon[30086]: pgmap v991: 158 pgs: 158 active+clean; 22 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T08:26:51.228 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:50 smithi134 ceph-mon[39278]: pgmap v991: 158 pgs: 158 active+clean; 22 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T08:26:51.282 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:51.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T08:26:51.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:51.307 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:51.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:50 smithi196 ceph-mon[33018]: pgmap v991: 158 pgs: 158 active+clean; 22 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T08:26:51.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:51.433 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:51.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:51.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:51.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:51.560 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:51.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:51.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:51.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:51.688 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:51.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:51.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:51.747 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:51.790 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:51.813 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:51.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:51.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:51.939 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:51.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:51.971 INFO:teuthology.orchestra.run.smithi134.stderr:set 1.1 pg_upmap_items mapping to [3->1,0->6,7->4] 2024-02-05T08:26:52.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:52.065 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:52.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:52.141 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:51 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1457867318' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1", "id": [3, 1, 0, 6, 7, 4]}]: dispatch 2024-02-05T08:26:52.141 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:51 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1457867318' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1", "id": [3, 1, 0, 6, 7, 4]}]: dispatch 2024-02-05T08:26:52.167 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:52.190 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:52.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:52.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:52.291 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:51 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1457867318' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.1", "id": [3, 1, 0, 6, 7, 4]}]: dispatch 2024-02-05T08:26:52.316 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:52.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:52.418 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:52.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:52.444 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:52.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:52.545 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:52.568 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:52.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:52.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:52.670 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:52.694 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:52.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:52.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:52.796 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:52.819 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:52.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:52.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:52.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:52.946 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:52.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:52.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:53.025 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:52 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1457867318' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1", "id": [3, 1, 0, 6, 7, 4]}]': finished 2024-02-05T08:26:53.025 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:52 smithi134 ceph-mon[30086]: osdmap e427: 8 total, 8 up, 8 in 2024-02-05T08:26:53.025 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:52 smithi134 ceph-mon[30086]: pgmap v993: 158 pgs: 158 active+clean; 3.1 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 756 op/s 2024-02-05T08:26:53.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:52 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1457867318' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1", "id": [3, 1, 0, 6, 7, 4]}]': finished 2024-02-05T08:26:53.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:52 smithi134 ceph-mon[39278]: osdmap e427: 8 total, 8 up, 8 in 2024-02-05T08:26:53.026 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:52 smithi134 ceph-mon[39278]: pgmap v993: 158 pgs: 158 active+clean; 3.1 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 756 op/s 2024-02-05T08:26:53.029 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:53.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:53.072 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:53.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:53.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:53.114 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-02-05T08:26:53.114 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-02-05T08:26:53.174 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:53.198 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:53.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:53.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:53.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:52 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1457867318' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.1", "id": [3, 1, 0, 6, 7, 4]}]': finished 2024-02-05T08:26:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:52 smithi196 ceph-mon[33018]: osdmap e427: 8 total, 8 up, 8 in 2024-02-05T08:26:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:52 smithi196 ceph-mon[33018]: pgmap v993: 158 pgs: 158 active+clean; 3.1 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 756 op/s 2024-02-05T08:26:53.325 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:53.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:53.426 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:53.450 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:53.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:53.553 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:53.576 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:53.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:53.678 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:53.703 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:53.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:53.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:53.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:53.828 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:53.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:53.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:53.930 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:53.954 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:53.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:54.056 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:54.081 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:54.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:54.182 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:54.206 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:54.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:54.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:54.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:53 smithi134 ceph-mon[30086]: osdmap e428: 8 total, 8 up, 8 in 2024-02-05T08:26:54.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:26:54.235 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:53 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:26:54.235 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:53 smithi134 ceph-mon[39278]: osdmap e428: 8 total, 8 up, 8 in 2024-02-05T08:26:54.235 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:26:54.235 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:53 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:26:54.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:53 smithi196 ceph-mon[33018]: osdmap e428: 8 total, 8 up, 8 in 2024-02-05T08:26:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:26:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:53 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:26:54.315 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:26:54 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:26:54.332 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:54.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:54.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:54.435 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:54.460 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:54.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:54.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:54.562 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:54.586 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:54.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:54.688 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:54.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:54.715 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:54.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:54.816 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:54.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:54.842 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:54.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:54.943 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:54.969 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:54.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:55.056 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:54 smithi134 ceph-mon[30086]: pgmap v996: 158 pgs: 158 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 947 B/s rd, 0 B/s wr, 544 op/s 2024-02-05T08:26:55.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:54 smithi134 ceph-mon[30086]: osdmap e429: 8 total, 8 up, 8 in 2024-02-05T08:26:55.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:54 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:26:55.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:54 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:26:55.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:54 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:55.057 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:54 smithi134 ceph-mon[30086]: osdmap e430: 8 total, 8 up, 8 in 2024-02-05T08:26:55.057 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:54 smithi134 ceph-mon[39278]: pgmap v996: 158 pgs: 158 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 947 B/s rd, 0 B/s wr, 544 op/s 2024-02-05T08:26:55.057 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:54 smithi134 ceph-mon[39278]: osdmap e429: 8 total, 8 up, 8 in 2024-02-05T08:26:55.057 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:54 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:26:55.057 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:54 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:26:55.058 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:54 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:55.058 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:54 smithi134 ceph-mon[39278]: osdmap e430: 8 total, 8 up, 8 in 2024-02-05T08:26:55.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:55.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:55.096 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:55.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:55.197 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:55.221 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:55.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:55.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:54 smithi196 ceph-mon[33018]: pgmap v996: 158 pgs: 158 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 947 B/s rd, 0 B/s wr, 544 op/s 2024-02-05T08:26:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:54 smithi196 ceph-mon[33018]: osdmap e429: 8 total, 8 up, 8 in 2024-02-05T08:26:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:54 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:26:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:54 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:26:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:54 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:26:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:54 smithi196 ceph-mon[33018]: osdmap e430: 8 total, 8 up, 8 in 2024-02-05T08:26:55.323 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:55.347 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:55.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:55.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:55.449 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:55.474 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:55.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:55.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:55.576 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:55.600 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:55.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:55.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:55.702 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:55.725 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:55.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:55.827 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:55.851 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:55.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:55.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:55.953 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:26:55.979 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:55.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:55.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:26:55.999 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:26:55.999 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [], 2024-02-05T08:26:55.999 INFO:teuthology.orchestra.run.smithi134.stdout: "num_ops": 0 2024-02-05T08:26:55.999 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:26:56.082 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:56.108 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:56.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:56.210 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:56.234 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:56.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:56.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:56.359 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:56.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:56.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:56.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:56.488 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:56.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:56.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:56.590 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:56.617 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:56.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:56.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:56.620 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:26:56.620 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:26:56.621 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:26:56.621 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:26:56.621 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:26:56.621 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:26:56.621 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:26:56.621 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:26:56.621 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:26:56.621 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:26:56.621 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:26:56.621 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:26:56.621 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:26:56.622 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:26:56.622 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:26:56.622 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:26:56.719 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:56.744 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:56.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:56.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:56.869 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:56.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:56.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:56.999 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:57.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:57.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:57.101 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:57.128 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:57.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:57.230 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:57.254 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:57.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:57.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:56 smithi134 ceph-mon[30086]: osdmap e431: 8 total, 8 up, 8 in 2024-02-05T08:26:57.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:56 smithi134 ceph-mon[30086]: pgmap v999: 158 pgs: 1 active+remapped, 157 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-05T08:26:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:56 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3674287059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:56 smithi134 ceph-mon[39278]: osdmap e431: 8 total, 8 up, 8 in 2024-02-05T08:26:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:56 smithi134 ceph-mon[39278]: pgmap v999: 158 pgs: 1 active+remapped, 157 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-05T08:26:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:56 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3674287059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:56 smithi196 ceph-mon[33018]: osdmap e431: 8 total, 8 up, 8 in 2024-02-05T08:26:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:56 smithi196 ceph-mon[33018]: pgmap v999: 158 pgs: 1 active+remapped, 157 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 11 op/s 2024-02-05T08:26:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:56 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3674287059' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:26:57.359 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:57.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:57.384 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:57.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:57.485 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:57.513 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:57.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:57.615 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:57.645 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:57.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:57.746 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:57.774 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:57.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:57.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:57.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T08:26:57.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:57.903 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:57.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:57.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:58.005 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:26:58.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:26:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:26:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:26:58.033 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:58.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:26:58.060 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 7, 6, 3, 0] out_osds: [] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-02-05T08:26:58.060 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T08:26:58.060 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-05T08:26:58.060 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.216479 2024-02-05T08:26:58.061 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 1 0.21647929937570753 2024-02-05T08:26:58.135 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:58.162 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:58.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:58.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:58.264 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:58.288 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:58.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:58.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:58.390 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:58.414 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:58.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:58.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:58.515 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:58.539 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:58.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:58.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:58.641 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:26:58.665 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:58.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:58.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:26:58.767 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:58.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:58.793 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:58.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:58.894 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:58.923 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:58.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:58.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:59.025 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:59.034 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:59.043 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:59.050 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:59.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:59.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:59.060 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:26:59.153 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:26:59.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:58 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:26:59.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:58 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:26:59.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:58 smithi134 ceph-mon[30086]: osdmap e432: 8 total, 8 up, 8 in 2024-02-05T08:26:59.162 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:58 smithi134 ceph-mon[30086]: pgmap v1001: 158 pgs: 1 active+remapped, 157 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:26:59.162 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:58 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:26:59.162 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:58 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:26:59.162 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:58 smithi134 ceph-mon[39278]: osdmap e432: 8 total, 8 up, 8 in 2024-02-05T08:26:59.163 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:58 smithi134 ceph-mon[39278]: pgmap v1001: 158 pgs: 1 active+remapped, 157 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:26:59.177 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:59.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:59.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:26:59.278 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:59.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:59.305 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:59.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:58 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:26:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:58 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:26:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:58 smithi196 ceph-mon[33018]: osdmap e432: 8 total, 8 up, 8 in 2024-02-05T08:26:59.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:58 smithi196 ceph-mon[33018]: pgmap v1001: 158 pgs: 1 active+remapped, 157 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:26:59.406 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:26:59.429 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:59.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:59.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:26:59.531 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:59.556 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:59.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:59.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:59.657 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:26:59.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:59.682 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:26:59.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:26:59.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:26:59.811 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:59.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:26:59.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:26:59.944 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:26:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:26:59.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:26:59.990 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is set 2024-02-05T08:27:00.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:59 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/452442433' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:27:00.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:59 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:27:00.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:59 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:27:00.027 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:26:59 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:27:00.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:59 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/452442433' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:27:00.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:59 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:27:00.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:59 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:27:00.028 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:26:59 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:27:00.046 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:27:00.071 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:00.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:27:00.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:27:00.198 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:00.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:27:00.300 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:59 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/452442433' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:27:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:59 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:27:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:59 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:27:00.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:26:59 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:27:00.326 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:00.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:00.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:00.428 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:27:00.453 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:00.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:27:00.555 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:27:00.580 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:00.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:00.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:27:00.618 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T08:27:00.682 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:27:00.706 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:00.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:00.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:27:00.808 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:00.832 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:00.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:00.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:00.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:27:00.958 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:00.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:00.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:27:01.060 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:27:01.084 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:01.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:01.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:27:01.186 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:27:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:01.210 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:01.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:27:01.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:00 smithi134 ceph-mon[30086]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:27:01.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:00 smithi134 ceph-mon[30086]: pgmap v1002: 158 pgs: 158 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:01.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:00 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:27:01.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:00 smithi134 ceph-mon[30086]: osdmap e433: 8 total, 8 up, 8 in 2024-02-05T08:27:01.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:00 smithi134 ceph-mon[39278]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:27:01.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:00 smithi134 ceph-mon[39278]: pgmap v1002: 158 pgs: 158 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:01.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:00 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:27:01.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:00 smithi134 ceph-mon[39278]: osdmap e433: 8 total, 8 up, 8 in 2024-02-05T08:27:01.311 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:27:01.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:00 smithi196 ceph-mon[33018]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:27:01.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:00 smithi196 ceph-mon[33018]: pgmap v1002: 158 pgs: 158 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:01.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:00 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T08:27:01.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:00 smithi196 ceph-mon[33018]: osdmap e433: 8 total, 8 up, 8 in 2024-02-05T08:27:01.335 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:01.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:27:01.437 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:01.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:01.462 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:01.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:01.563 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:27:01.591 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:01.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:27:01.693 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:01.717 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:01.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:01.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:01.819 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:27:01.845 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:01.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:27:01.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:27:01.975 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:01.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:01.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:27:02.077 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:27:02.101 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:02.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:02.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:27:02.203 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:27:02.231 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:02.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:27:02.333 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:27:02.359 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:02.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:02.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:27:02.461 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:27:02.485 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:02.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:02.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:27:02.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:02.611 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:02.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:02.713 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:27:02.736 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:02.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:02.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:27:02.764 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:02.838 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:27:02.862 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:02.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:27:02.909 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:27:02.964 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:27:02.992 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:02.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:27:03.094 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:27:03.121 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:03.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:03.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:27:03.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:27:03.239 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:02 smithi134 ceph-mon[30086]: pgmap v1004: 158 pgs: 158 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-02-05T08:27:03.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:02 smithi134 ceph-mon[39278]: pgmap v1004: 158 pgs: 158 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-02-05T08:27:03.251 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:03.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:03.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:27:03.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:02 smithi196 ceph-mon[33018]: pgmap v1004: 158 pgs: 158 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-02-05T08:27:03.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:27:03.380 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:03.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:27:03.482 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:27:03.510 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:03.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:27:03.612 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:27:03.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:03.640 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:03.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:27:03.741 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:27:03.766 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:03.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:27:03.868 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:27:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:03.895 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:03.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:27:03.996 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:27:04.002 INFO:teuthology.orchestra.run.smithi134.stderr:set osd.1 primary-affinity to 0.216479 (8141872) 2024-02-05T08:27:04.024 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:04.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:04.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:27:04.126 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:27:04.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:04.150 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:04.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:27:04.251 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:04.275 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:04.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:04.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:03 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4249163659' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.21647929937570753}]: dispatch 2024-02-05T08:27:04.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:03 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.21647929937570753}]: dispatch 2024-02-05T08:27:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:03 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4249163659' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.21647929937570753}]: dispatch 2024-02-05T08:27:04.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:03 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.21647929937570753}]: dispatch 2024-02-05T08:27:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:03 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4249163659' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.21647929937570753}]: dispatch 2024-02-05T08:27:04.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:03 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.21647929937570753}]: dispatch 2024-02-05T08:27:04.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:27:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:27:04.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:04.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:04.526 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:04.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:04.627 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:27:04.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:04.655 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:04.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:27:04.756 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:27:04.784 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:04.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:27:04.886 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:27:04.914 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:04.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:27:05.016 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:05.041 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:05.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:05.143 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:27:05.170 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:05.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:27:05.272 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:27:05.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:04 smithi134 ceph-mon[30086]: pgmap v1005: 158 pgs: 158 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:27:05.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:04 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.21647929937570753}]': finished 2024-02-05T08:27:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:04 smithi134 ceph-mon[30086]: osdmap e434: 8 total, 8 up, 8 in 2024-02-05T08:27:05.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:04 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:04 smithi134 ceph-mon[39278]: pgmap v1005: 158 pgs: 158 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:27:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:04 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.21647929937570753}]': finished 2024-02-05T08:27:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:04 smithi134 ceph-mon[39278]: osdmap e434: 8 total, 8 up, 8 in 2024-02-05T08:27:05.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:04 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:05.297 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:05.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:05.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:27:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:04 smithi196 ceph-mon[33018]: pgmap v1005: 158 pgs: 158 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:27:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:04 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.21647929937570753}]': finished 2024-02-05T08:27:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:04 smithi196 ceph-mon[33018]: osdmap e434: 8 total, 8 up, 8 in 2024-02-05T08:27:05.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:04 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:05.400 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:27:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:05.425 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:05.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:27:05.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:27:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:05.551 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:05.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:27:05.652 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:27:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:05.677 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:05.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:27:05.778 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:27:05.806 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:05.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:05.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:27:05.908 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:05.932 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:05.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:05.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:05.945 INFO:teuthology.orchestra.run.smithi134.stdout:ERROR: (22) Invalid argument 2024-02-05T08:27:05.945 INFO:teuthology.orchestra.run.smithi134.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T08:27:06.034 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:27:06.062 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:06.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:27:06.164 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:27:06.192 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:06.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:06.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:27:06.238 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:06 smithi134 ceph-mon[30086]: osdmap e435: 8 total, 8 up, 8 in 2024-02-05T08:27:06.239 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:06 smithi134 ceph-mon[39278]: osdmap e435: 8 total, 8 up, 8 in 2024-02-05T08:27:06.294 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4 2024-02-05T08:27:06.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:06 smithi196 ceph-mon[33018]: osdmap e435: 8 total, 8 up, 8 in 2024-02-05T08:27:06.320 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:06.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.4' 2024-02-05T08:27:06.422 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:27:06.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:06.450 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:06.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:27:06.551 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:27:06.577 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:06.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:27:06.600 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T08:27:06.679 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:27:06.709 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:06.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:27:06.811 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:06.836 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:06.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:06.937 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:27:06.962 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:06.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:06.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:27:07.064 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:27:07.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:07.090 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:07.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:27:07.191 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3 2024-02-05T08:27:07.216 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:07.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:07.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.3' 2024-02-05T08:27:07.262 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:07 smithi134 ceph-mon[30086]: pgmap v1008: 158 pgs: 1 peering, 157 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:07.262 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:07 smithi134 ceph-mon[30086]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-05T08:27:07.262 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:07 smithi134 ceph-mon[39278]: pgmap v1008: 158 pgs: 1 peering, 157 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:07.262 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:07 smithi134 ceph-mon[39278]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-05T08:27:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:07 smithi196 ceph-mon[33018]: pgmap v1008: 158 pgs: 1 peering, 157 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:07.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:07 smithi196 ceph-mon[33018]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-02-05T08:27:07.319 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:07.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:07.343 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:07.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:07.444 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7 2024-02-05T08:27:07.471 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:07.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:07.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.7' 2024-02-05T08:27:07.573 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:07.600 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:07.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:07.701 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:27:07.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:07.726 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:07.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:27:07.827 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0 2024-02-05T08:27:07.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:07.851 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:07.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.0' 2024-02-05T08:27:07.952 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:07.976 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:07.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:08.004 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:27:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:27:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:27:08.031 INFO:teuthology.orchestra.run.smithi134.stderr:pool 'unique_pool_4' removed 2024-02-05T08:27:08.077 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5 2024-02-05T08:27:08.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:08.115 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:08.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.5' 2024-02-05T08:27:08.216 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:27:08.241 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:08.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:08.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:27:08.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:08 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1896545190' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:27:08.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:08 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:27:08.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:08 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1896545190' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:27:08.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:08 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:27:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:08 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1896545190' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:27:08.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:08 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T08:27:08.343 DEBUG:teuthology.orchestra.run.smithi196:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6 2024-02-05T08:27:08.367 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: docker: command not found 2024-02-05T08:27:08.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:08.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi196 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.6' 2024-02-05T08:27:08.469 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2 2024-02-05T08:27:08.493 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:08.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:08.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.2' 2024-02-05T08:27:08.595 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:08.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:08.619 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:08.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:08.720 DEBUG:teuthology.orchestra.run.smithi134:> sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1 2024-02-05T08:27:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:27:08.745 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: docker: command not found 2024-02-05T08:27:08.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi134 with status 1: 'sudo docker kill -s 1 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd.1' 2024-02-05T08:27:08.834 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-02-05T08:27:08.844 INFO:tasks.thrashosds:joining thrashosds 2024-02-05T08:27:09.102 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:09 smithi134 ceph-mon[30086]: pgmap v1009: 158 pgs: 1 peering, 157 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 640 B/s rd, 0 op/s 2024-02-05T08:27:09.102 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:09 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:27:09.102 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:09 smithi134 ceph-mon[30086]: osdmap e436: 8 total, 8 up, 8 in 2024-02-05T08:27:09.102 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:09 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:27:09.102 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:09 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:27:09.102 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:09 smithi134 ceph-mon[39278]: pgmap v1009: 158 pgs: 1 peering, 157 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 640 B/s rd, 0 op/s 2024-02-05T08:27:09.102 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:09 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:27:09.103 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:09 smithi134 ceph-mon[39278]: osdmap e436: 8 total, 8 up, 8 in 2024-02-05T08:27:09.103 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:09 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:27:09.103 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:09 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:27:09.108 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:09.143 INFO:teuthology.orchestra.run.smithi134.stdout:osd.0: {} 2024-02-05T08:27:09.151 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:09.168 INFO:teuthology.orchestra.run.smithi134.stdout:osd.1: {} 2024-02-05T08:27:09.189 INFO:teuthology.orchestra.run.smithi134.stdout:osd.2: {} 2024-02-05T08:27:09.210 INFO:teuthology.orchestra.run.smithi134.stdout:osd.3: {} 2024-02-05T08:27:09.231 INFO:teuthology.orchestra.run.smithi134.stdout:osd.4: {} 2024-02-05T08:27:09.252 INFO:teuthology.orchestra.run.smithi134.stdout:osd.5: {} 2024-02-05T08:27:09.271 INFO:teuthology.orchestra.run.smithi134.stdout:osd.6: {} 2024-02-05T08:27:09.292 INFO:teuthology.orchestra.run.smithi134.stdout:osd.7: {} 2024-02-05T08:27:09.293 INFO:teuthology.orchestra.run.smithi134.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T08:27:09.293 INFO:teuthology.orchestra.run.smithi134.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T08:27:09.294 INFO:teuthology.orchestra.run.smithi134.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T08:27:09.294 INFO:teuthology.orchestra.run.smithi134.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T08:27:09.294 INFO:teuthology.orchestra.run.smithi134.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T08:27:09.294 INFO:teuthology.orchestra.run.smithi134.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T08:27:09.294 INFO:teuthology.orchestra.run.smithi134.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T08:27:09.294 INFO:teuthology.orchestra.run.smithi134.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T08:27:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:09 smithi196 ceph-mon[33018]: pgmap v1009: 158 pgs: 1 peering, 157 active+clean; 582 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 640 B/s rd, 0 op/s 2024-02-05T08:27:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:09 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T08:27:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:09 smithi196 ceph-mon[33018]: osdmap e436: 8 total, 8 up, 8 in 2024-02-05T08:27:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:09 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:27:09.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:09 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:27:10.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:10 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2825996687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:27:10.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:10 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2825996687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:27:10.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:10 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2825996687' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T08:27:11.046 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:11 smithi134 ceph-mon[30086]: pgmap v1011: 142 pgs: 142 active+clean; 635 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 5.0 KiB/s rd, 3 op/s 2024-02-05T08:27:11.046 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:11 smithi134 ceph-mon[30086]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-05T08:27:11.046 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:11 smithi134 ceph-mon[39278]: pgmap v1011: 142 pgs: 142 active+clean; 635 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 5.0 KiB/s rd, 3 op/s 2024-02-05T08:27:11.046 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:11 smithi134 ceph-mon[39278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-05T08:27:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:11 smithi196 ceph-mon[33018]: pgmap v1011: 142 pgs: 142 active+clean; 635 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 5.0 KiB/s rd, 3 op/s 2024-02-05T08:27:11.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:11 smithi196 ceph-mon[33018]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-02-05T08:27:13.072 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-02-05T08:27:13.073 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-02-05T08:27:13.099 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:13 smithi134 ceph-mon[30086]: pgmap v1012: 142 pgs: 142 active+clean; 635 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 3.8 KiB/s rd, 2 op/s 2024-02-05T08:27:13.099 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:13 smithi134 ceph-mon[39278]: pgmap v1012: 142 pgs: 142 active+clean; 635 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 3.8 KiB/s rd, 2 op/s 2024-02-05T08:27:13.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:13 smithi196 ceph-mon[33018]: pgmap v1012: 142 pgs: 142 active+clean; 635 KiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 3.8 KiB/s rd, 2 op/s 2024-02-05T08:27:13.415 INFO:teuthology.orchestra.run.smithi134.stdout:{ 2024-02-05T08:27:13.415 INFO:teuthology.orchestra.run.smithi134.stdout: "size": 20, 2024-02-05T08:27:13.415 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 600, 2024-02-05T08:27:13.415 INFO:teuthology.orchestra.run.smithi134.stdout: "ops": [ 2024-02-05T08:27:13.415 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.415 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:133556 11.0 11:01f803a0:::benchmark_data_smithi134_205380_object133555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-05T08:27:13.415 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:24:48.468686+0000", 2024-02-05T08:27:13.415 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 144.94426794399999, 2024-02-05T08:27:13.415 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12119440099999999, 2024-02-05T08:27:13.415 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 133556 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.468686+0000", 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.416 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.468686+0000", 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.468687+0000", 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.063e-06 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.468706+0000", 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9378e-05 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.417 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.468718+0000", 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1038e-05 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.468780+0000", 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2049000000000002e-05 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.474468+0000", 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0056886289999999997 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.418 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.536299+0000", 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.061830417999999998 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.536490+0000", 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000191573 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.540626+0000", 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0041357870000000001 2024-02-05T08:27:13.419 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.420 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.420 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.420 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.540794+0000", 2024-02-05T08:27:13.420 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016840599999999999 2024-02-05T08:27:13.420 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.420 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.420 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.420 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.589762+0000", 2024-02-05T08:27:13.420 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.048967837 2024-02-05T08:27:13.420 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.420 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.420 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.420 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.589837+0000", 2024-02-05T08:27:13.421 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4793e-05 2024-02-05T08:27:13.421 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.421 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.421 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.421 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.589880+0000", 2024-02-05T08:27:13.421 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3430000000000003e-05 2024-02-05T08:27:13.421 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.421 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.421 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.421 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.421 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.421 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:133558 11.0 11:08a767e4:::benchmark_data_smithi134_205380_object133557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-05T08:27:13.422 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:24:48.468967+0000", 2024-02-05T08:27:13.422 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 144.94398733, 2024-02-05T08:27:13.422 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.12683085799999999, 2024-02-05T08:27:13.422 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.422 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.422 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.422 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.422 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.422 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 133558 2024-02-05T08:27:13.422 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.468967+0000", 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.468967+0000", 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.423 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.468967+0000", 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.1699999999999998e-07 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.468987+0000", 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9656e-05 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.468996+0000", 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.8999999999999995e-06 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.424 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.469002+0000", 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2269999999999998e-06 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.538503+0000", 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.069501321000000005 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.538645+0000", 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014120800000000001 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:27:13.425 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.538765+0000", 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000120412 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.540631+0000", 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0018664269999999999 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.542482+0000", 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0018510550000000001 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.426 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.595281+0000", 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.052798955000000002 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.595747+0000", 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00046548599999999998 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.595797+0000", 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0593999999999997e-05 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.427 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.428 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.428 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:133562 11.0 11:041c14c7:::benchmark_data_smithi134_205380_object133561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-05T08:27:13.428 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:24:48.473597+0000", 2024-02-05T08:27:13.428 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 144.93935683800001, 2024-02-05T08:27:13.428 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.122498185, 2024-02-05T08:27:13.428 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.428 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.428 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.428 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.428 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.429 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 133562 2024-02-05T08:27:13.429 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.429 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.429 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.429 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.429 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.473597+0000", 2024-02-05T08:27:13.429 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.429 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.429 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.473597+0000", 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.473599+0000", 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.3850000000000001e-06 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.473619+0000", 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.0078999999999999e-05 2024-02-05T08:27:13.430 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.431 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.431 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.431 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.473630+0000", 2024-02-05T08:27:13.431 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.131e-05 2024-02-05T08:27:13.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.432 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.432 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.473638+0000", 2024-02-05T08:27:13.432 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8159999999999994e-06 2024-02-05T08:27:13.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.432 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.432 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.432 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.540836+0000", 2024-02-05T08:27:13.432 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.067198038000000002 2024-02-05T08:27:13.432 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.433 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.433 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.433 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.540934+0000", 2024-02-05T08:27:13.433 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8204999999999999e-05 2024-02-05T08:27:13.433 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.433 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.435 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:27:13.435 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.541028+0000", 2024-02-05T08:27:13.435 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.4396999999999998e-05 2024-02-05T08:27:13.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.435 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.435 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.544942+0000", 2024-02-05T08:27:13.435 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0039139079999999998 2024-02-05T08:27:13.435 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.435 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.544982+0000", 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9279000000000001e-05 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.595854+0000", 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.050872033999999997 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.596051+0000", 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00019741600000000001 2024-02-05T08:27:13.436 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.437 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.437 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:48.596095+0000", 2024-02-05T08:27:13.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4317999999999998e-05 2024-02-05T08:27:13.437 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.437 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.437 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.437 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.437 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.437 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:135911 11.0 11:0b03f375:::benchmark_data_smithi134_205380_object809:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-05T08:27:13.437 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:24:50.385800+0000", 2024-02-05T08:27:13.437 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 143.02715385100001, 2024-02-05T08:27:13.437 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.105128649, 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 135911 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.385800+0000", 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.438 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.385800+0000", 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.385801+0000", 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6799999999999999e-07 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.385804+0000", 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1820000000000002e-06 2024-02-05T08:27:13.439 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.385813+0000", 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9639999999999992e-06 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.385819+0000", 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5910000000000003e-06 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.385858+0000", 2024-02-05T08:27:13.440 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8897999999999999e-05 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.385938+0000", 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0346999999999995e-05 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.386010+0000", 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2251999999999999e-05 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.441 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.388162+0000", 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002151942 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.388260+0000", 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8210999999999999e-05 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.490835+0000", 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.102574284 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.442 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.490888+0000", 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3705000000000002e-05 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.490929+0000", 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0404999999999997e-05 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:135924 11.0 11:02f7dffd:::benchmark_data_smithi134_205380_object822:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-02-05T08:27:13.443 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:24:50.396089+0000", 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 143.016864828, 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.094970748999999993, 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 135924 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.396089+0000", 2024-02-05T08:27:13.444 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.396089+0000", 2024-02-05T08:27:13.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.445 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.445 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.396090+0000", 2024-02-05T08:27:13.445 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1650000000000001e-06 2024-02-05T08:27:13.445 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.445 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.447 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.447 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.396094+0000", 2024-02-05T08:27:13.447 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.6430000000000001e-06 2024-02-05T08:27:13.447 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.447 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.447 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.447 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.396104+0000", 2024-02-05T08:27:13.448 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0076e-05 2024-02-05T08:27:13.448 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.448 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.448 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.448 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.396110+0000", 2024-02-05T08:27:13.448 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.6049999999999997e-06 2024-02-05T08:27:13.448 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.448 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.448 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.448 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.396150+0000", 2024-02-05T08:27:13.448 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.9916000000000002e-05 2024-02-05T08:27:13.448 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.471776+0000", 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.075626815 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.471904+0000", 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012773799999999999 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.473789+0000", 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.001884782 2024-02-05T08:27:13.449 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.474079+0000", 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000290117 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.490966+0000", 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.016886768 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.491030+0000", 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4257999999999994e-05 2024-02-05T08:27:13.450 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:24:50.491060+0000", 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.9865999999999998e-05 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:164326 11.6 11:65cd5721:::benchmark_data_smithi134_205380_object29224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:12.058483+0000", 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 121.354470834, 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.094133254999999999, 2024-02-05T08:27:13.451 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.452 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.452 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.452 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.452 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.452 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 164326 2024-02-05T08:27:13.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.452 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.452 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.452 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.452 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.058483+0000", 2024-02-05T08:27:13.452 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.452 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.452 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.453 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.453 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.058483+0000", 2024-02-05T08:27:13.453 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.453 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.453 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.453 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.453 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.058484+0000", 2024-02-05T08:27:13.453 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.4900000000000005e-07 2024-02-05T08:27:13.453 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.453 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.453 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.453 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.058487+0000", 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4149999999999999e-06 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.058496+0000", 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.1360000000000003e-06 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.058503+0000", 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3890000000000002e-06 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.454 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.061328+0000", 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0028251119999999998 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.061694+0000", 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00036590800000000002 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.061804+0000", 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000110522 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.142944+0000", 2024-02-05T08:27:13.455 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.081139469000000006 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.145551+0000", 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0026067170000000001 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.152530+0000", 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0069790119999999997 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.456 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.152579+0000", 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.9387e-05 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.152616+0000", 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7438999999999998e-05 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_repop(client.25820.0:164330 11.e e401/367 11:749cd764:::benchmark_data_smithi134_205380_object29228:head v 401'10189, mlcod=401'10187)", 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:12.061839+0000", 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 121.351114478, 2024-02-05T08:27:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.090813009, 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.061839+0000", 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.061839+0000", 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.458 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.061840+0000", 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7400000000000003e-07 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.061847+0000", 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.3940000000000001e-06 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.139350+0000", 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.077503308000000007 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.459 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.139376+0000", 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.5680000000000001e-05 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.142976+0000", 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0036008149999999998 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.142996+0000", 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.9933e-05 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.460 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.152620+0000", 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0096233640000000006 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:12.152653+0000", 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.2840999999999999e-05 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.461 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:173362 11.0 11:0f75c1a6:::benchmark_data_smithi134_205380_object38260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:18.919553+0000", 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 114.49340057400001, 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.093325278999999997, 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 173362 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.919553+0000", 2024-02-05T08:27:13.462 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.919553+0000", 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.919554+0000", 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9299999999999996e-07 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.463 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.919559+0000", 2024-02-05T08:27:13.464 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6179999999999997e-06 2024-02-05T08:27:13.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.464 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.464 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.919570+0000", 2024-02-05T08:27:13.464 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1517e-05 2024-02-05T08:27:13.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.464 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.464 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.919579+0000", 2024-02-05T08:27:13.464 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.1820000000000006e-06 2024-02-05T08:27:13.464 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.464 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.465 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.465 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.956169+0000", 2024-02-05T08:27:13.465 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.036590737999999998 2024-02-05T08:27:13.465 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.465 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.465 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.465 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.956279+0000", 2024-02-05T08:27:13.465 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010939500000000001 2024-02-05T08:27:13.465 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.465 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.465 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:27:13.465 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.956384+0000", 2024-02-05T08:27:13.465 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010538 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.960260+0000", 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0038754390000000001 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.961948+0000", 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0016881450000000001 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.466 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:19.012551+0000", 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.050603634000000002 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:19.012823+0000", 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00027207700000000002 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:19.012879+0000", 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5260999999999998e-05 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.467 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.468 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.468 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.468 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:173367 11.0 11:025698cd:::benchmark_data_smithi134_205380_object38265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-02-05T08:27:13.468 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:18.921824+0000", 2024-02-05T08:27:13.468 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 114.491129743, 2024-02-05T08:27:13.468 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.091290600999999999, 2024-02-05T08:27:13.468 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.468 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.468 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.468 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.468 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.468 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 173367 2024-02-05T08:27:13.468 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.921824+0000", 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.921824+0000", 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.469 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.921825+0000", 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.2999999999999999e-07 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.921829+0000", 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4400000000000001e-06 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.921837+0000", 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3250000000000008e-06 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.470 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.921843+0000", 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5230000000000001e-06 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.958629+0000", 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.036785855999999999 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.958705+0000", 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.5994000000000005e-05 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.471 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.958777+0000", 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1377000000000005e-05 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.963031+0000", 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0042546420000000003 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:18.986846+0000", 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.023815092 2024-02-05T08:27:13.472 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:19.012922+0000", 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.026076052999999998 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:19.013068+0000", 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00014511099999999999 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:19.013115+0000", 2024-02-05T08:27:13.473 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.7258000000000001e-05 2024-02-05T08:27:13.474 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.474 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.474 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.474 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.474 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.474 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212067 11.6 11:6d0089da:::benchmark_data_smithi134_205380_object76965:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e407)", 2024-02-05T08:27:13.474 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:49.114499+0000", 2024-02-05T08:27:13.474 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 84.298454618999997, 2024-02-05T08:27:13.474 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.120803711, 2024-02-05T08:27:13.474 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.474 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "reached pg", 2024-02-05T08:27:13.474 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.474 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212067 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114499+0000", 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114499+0000", 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.475 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114502+0000", 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1909999999999999e-06 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114508+0000", 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6470000000000001e-06 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114519+0000", 2024-02-05T08:27:13.476 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1308000000000001e-05 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114531+0000", 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.164e-05 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114706+0000", 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00017522999999999999 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114716+0000", 2024-02-05T08:27:13.477 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.6099999999999995e-06 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.235216+0000", 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.120500474 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.235303+0000", 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.6611000000000001e-05 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.478 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212086 11.6 11:69fe03b7:::benchmark_data_smithi134_205380_object76984:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e407)", 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:49.114553+0000", 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 84.298400983999997, 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.131956631, 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212086 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.479 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114553+0000", 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114553+0000", 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114553+0000", 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2500000000000001e-07 2024-02-05T08:27:13.480 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114556+0000", 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.6410000000000002e-06 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114563+0000", 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.4999999999999996e-06 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114568+0000", 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4589999999999997e-06 2024-02-05T08:27:13.481 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114737+0000", 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00016897099999999999 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114743+0000", 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7660000000000004e-06 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.235569+0000", 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.120826399 2024-02-05T08:27:13.482 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.235690+0000", 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012132499999999999 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,2,4", 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.235814+0000", 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000123898 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.240870+0000", 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0050554220000000004 2024-02-05T08:27:13.483 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.240915+0000", 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4845000000000002e-05 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.246021+0000", 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0051063109999999997 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.246297+0000", 2024-02-05T08:27:13.484 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00027585499999999998 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.246396+0000", 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.9605999999999995e-05 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.246510+0000", 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000113208 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.485 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212094 11.6 11:63e3d6d9:::benchmark_data_smithi134_205380_object76992:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e407)", 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:49.114574+0000", 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 84.298379819999994, 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1415594680000001, 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212094 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114574+0000", 2024-02-05T08:27:13.486 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114574+0000", 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114575+0000", 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7e-07 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.487 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114577+0000", 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1600000000000001e-06 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114582+0000", 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.395e-06 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114587+0000", 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.4349999999999999e-06 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.488 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114761+0000", 2024-02-05T08:27:13.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000174842 2024-02-05T08:27:13.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:27:13.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114767+0000", 2024-02-05T08:27:13.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.604e-06 2024-02-05T08:27:13.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.489 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.489 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.489 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.238195+0000", 2024-02-05T08:27:13.489 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1234282040000001 2024-02-05T08:27:13.489 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.238332+0000", 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000136441 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,2,4", 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.238470+0000", 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013812199999999999 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.240928+0000", 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0024585919999999999 2024-02-05T08:27:13.490 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.246244+0000", 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0053156810000000001 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.246605+0000", 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00036089299999999999 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.255962+0000", 2024-02-05T08:27:13.491 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0093573500000000004 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256025+0000", 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.2651999999999996e-05 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256134+0000", 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000108727 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.492 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212096 11.6 11:6c9e3c86:::benchmark_data_smithi134_205380_object76994:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e407)", 2024-02-05T08:27:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:49.114591+0000", 2024-02-05T08:27:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 84.298362564000001, 2024-02-05T08:27:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1418646349999999, 2024-02-05T08:27:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212096 2024-02-05T08:27:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.493 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114591+0000", 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114591+0000", 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114592+0000", 2024-02-05T08:27:13.494 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4299999999999999e-07 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114594+0000", 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.238e-06 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114600+0000", 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.7150000000000003e-06 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114604+0000", 2024-02-05T08:27:13.495 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.3039999999999998e-06 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114806+0000", 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020233499999999999 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114812+0000", 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8830000000000003e-06 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.496 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.240956+0000", 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1261435 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.241073+0000", 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000117071 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,2,4", 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.241206+0000", 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000133106 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.497 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.246040+0000", 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0048345740000000003 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.246561+0000", 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00052088000000000004 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.247290+0000", 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00072899800000000002 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.498 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256292+0000", 2024-02-05T08:27:13.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0090013000000000003 2024-02-05T08:27:13.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256350+0000", 2024-02-05T08:27:13.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.8539000000000003e-05 2024-02-05T08:27:13.499 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.499 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.499 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.499 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256456+0000", 2024-02-05T08:27:13.499 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000105849 2024-02-05T08:27:13.499 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212099 11.6 11:68795a7e:::benchmark_data_smithi134_205380_object76997:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e407)", 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:49.114609+0000", 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 84.298345209000004, 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1420990579999999, 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.500 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212099 2024-02-05T08:27:13.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.501 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114609+0000", 2024-02-05T08:27:13.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.501 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.501 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114609+0000", 2024-02-05T08:27:13.501 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.501 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.501 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114609+0000", 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.1600000000000002e-07 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114611+0000", 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.1909999999999999e-06 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114617+0000", 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5620000000000003e-06 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.502 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114621+0000", 2024-02-05T08:27:13.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.2139999999999998e-06 2024-02-05T08:27:13.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114830+0000", 2024-02-05T08:27:13.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00020933900000000001 2024-02-05T08:27:13.503 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.503 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.503 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:27:13.503 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114836+0000", 2024-02-05T08:27:13.503 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.3480000000000003e-06 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.243477+0000", 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1286409630000001 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.243612+0000", 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013550699999999999 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,2,4", 2024-02-05T08:27:13.504 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.243747+0000", 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00013452599999999999 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.249982+0000", 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0062353900000000004 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256197+0000", 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0062144590000000003 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.505 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256251+0000", 2024-02-05T08:27:13.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.4546999999999998e-05 2024-02-05T08:27:13.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256581+0000", 2024-02-05T08:27:13.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00032992600000000003 2024-02-05T08:27:13.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.506 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.506 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.506 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256618+0000", 2024-02-05T08:27:13.506 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.7317999999999997e-05 2024-02-05T08:27:13.506 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256708+0000", 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9451999999999997e-05 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212106 11.6 11:6b408111:::benchmark_data_smithi134_205380_object77004:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e407)", 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:49.114626+0000", 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 84.298328323000007, 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.1428813039999999, 2024-02-05T08:27:13.507 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.508 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.508 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.508 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.508 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.508 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212106 2024-02-05T08:27:13.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.508 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.508 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.508 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114626+0000", 2024-02-05T08:27:13.508 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.508 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.508 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114626+0000", 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114626+0000", 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.7300000000000002e-07 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114628+0000", 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8679999999999999e-06 2024-02-05T08:27:13.509 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114633+0000", 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.1279999999999999e-06 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114637+0000", 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.138e-06 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114855+0000", 2024-02-05T08:27:13.510 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021765 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for peered", 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:49.114860+0000", 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5110000000000003e-06 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.247329+0000", 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.132469019 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.511 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.247453+0000", 2024-02-05T08:27:13.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000123915 2024-02-05T08:27:13.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.512 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 0,2,4", 2024-02-05T08:27:13.512 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.247583+0000", 2024-02-05T08:27:13.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00012964699999999999 2024-02-05T08:27:13.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.512 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.512 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256214+0000", 2024-02-05T08:27:13.512 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.008631629 2024-02-05T08:27:13.512 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.512 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256501+0000", 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00028627600000000002 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256546+0000", 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5336000000000001e-05 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256759+0000", 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00021270600000000001 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.513 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.257409+0000", 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000650126 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.257507+0000", 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 9.8082e-05 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.514 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:212148 11.0 11:00b6897a:::benchmark_data_smithi134_205380_object77046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-02-05T08:27:13.515 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:50.256769+0000", 2024-02-05T08:27:13.515 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 83.156185038000004, 2024-02-05T08:27:13.515 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.092110440000000002, 2024-02-05T08:27:13.515 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.515 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.515 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.515 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.515 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.515 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 212148 2024-02-05T08:27:13.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.516 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256769+0000", 2024-02-05T08:27:13.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.516 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.516 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.516 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256769+0000", 2024-02-05T08:27:13.516 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.516 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256770+0000", 2024-02-05T08:27:13.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.6199999999999997e-07 2024-02-05T08:27:13.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256773+0000", 2024-02-05T08:27:13.517 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8789999999999996e-06 2024-02-05T08:27:13.517 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.517 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.517 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.517 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256782+0000", 2024-02-05T08:27:13.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9330000000000007e-06 2024-02-05T08:27:13.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.518 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.518 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256788+0000", 2024-02-05T08:27:13.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.2569999999999998e-06 2024-02-05T08:27:13.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.518 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.518 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256833+0000", 2024-02-05T08:27:13.518 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5092999999999997e-05 2024-02-05T08:27:13.518 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.518 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256915+0000", 2024-02-05T08:27:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.2473999999999997e-05 2024-02-05T08:27:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:27:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.256983+0000", 2024-02-05T08:27:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.7336999999999996e-05 2024-02-05T08:27:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.307415+0000", 2024-02-05T08:27:13.519 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.050431933999999998 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.307437+0000", 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.2603999999999999e-05 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.348769+0000", 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.041331457000000002 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.520 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.348834+0000", 2024-02-05T08:27:13.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.5574000000000006e-05 2024-02-05T08:27:13.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.521 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.521 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:50.348879+0000", 2024-02-05T08:27:13.521 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5235999999999999e-05 2024-02-05T08:27:13.521 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.521 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.521 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.521 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.521 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.521 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:220084 11.0 11:0eb5dd33:::benchmark_data_smithi134_205380_object84982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-02-05T08:27:13.521 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:25:57.777646+0000", 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 75.635308210000005, 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.098443455999999999, 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 220084 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.777646+0000", 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.522 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.777646+0000", 2024-02-05T08:27:13.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.777647+0000", 2024-02-05T08:27:13.523 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.637e-06 2024-02-05T08:27:13.523 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.523 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.523 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.523 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.777652+0000", 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5390000000000001e-06 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.777662+0000", 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0182e-05 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.777677+0000", 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.5038000000000001e-05 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.524 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.841401+0000", 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.063723641999999997 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.841522+0000", 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.000120968 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.841629+0000", 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00010686100000000001 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.525 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.844662+0000", 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0030329599999999999 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.844849+0000", 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018700699999999999 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.875995+0000", 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.031146552000000001 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.526 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.876050+0000", 2024-02-05T08:27:13.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.507e-05 2024-02-05T08:27:13.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.527 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.527 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:25:57.876089+0000", 2024-02-05T08:27:13.527 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.8999999999999999e-05 2024-02-05T08:27:13.527 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.527 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.527 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.527 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.527 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:244266 11.0 11:088943c4:::benchmark_data_smithi134_205380_object109164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:26:23.411170+0000", 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 50.001783875999998, 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.093332773999999993, 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 244266 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.528 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.411170+0000", 2024-02-05T08:27:13.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.411170+0000", 2024-02-05T08:27:13.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.529 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.529 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.529 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.411171+0000", 2024-02-05T08:27:13.529 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0920000000000001e-06 2024-02-05T08:27:13.529 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.411176+0000", 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.8520000000000003e-06 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.411194+0000", 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.8088999999999999e-05 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.411202+0000", 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9500000000000001e-06 2024-02-05T08:27:13.530 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.531 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.411248+0000", 2024-02-05T08:27:13.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.5506000000000001e-05 2024-02-05T08:27:13.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.531 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.411337+0000", 2024-02-05T08:27:13.531 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.9748999999999997e-05 2024-02-05T08:27:13.531 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.531 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.531 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:27:13.531 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.411409+0000", 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.1227000000000001e-05 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.413562+0000", 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0021539369999999999 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.414105+0000", 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00054288999999999997 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.532 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.504396+0000", 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.090290192000000005 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.504456+0000", 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.0911e-05 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.504503+0000", 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.6378999999999998e-05 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.533 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:244269 11.0 11:0e8835e2:::benchmark_data_smithi134_205380_object109167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:26:23.414511+0000", 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 49.998442636, 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.090393052000000002, 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 244269 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.534 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.414511+0000", 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.414511+0000", 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.414512+0000", 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.8999999999999995e-07 2024-02-05T08:27:13.535 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.414516+0000", 2024-02-05T08:27:13.536 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.4290000000000001e-06 2024-02-05T08:27:13.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.414524+0000", 2024-02-05T08:27:13.536 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.9529999999999999e-06 2024-02-05T08:27:13.536 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.536 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.536 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.536 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.414529+0000", 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.0549999999999999e-06 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.414564+0000", 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 3.5683999999999999e-05 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.414637+0000", 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2861000000000006e-05 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 3,7", 2024-02-05T08:27:13.537 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.414718+0000", 2024-02-05T08:27:13.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.0715000000000002e-05 2024-02-05T08:27:13.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.416964+0000", 2024-02-05T08:27:13.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0022460829999999998 2024-02-05T08:27:13.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.538 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.538 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.417037+0000", 2024-02-05T08:27:13.538 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.3276999999999997e-05 2024-02-05T08:27:13.538 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.538 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.504679+0000", 2024-02-05T08:27:13.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.087642271999999993 2024-02-05T08:27:13.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.504863+0000", 2024-02-05T08:27:13.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00018338999999999999 2024-02-05T08:27:13.539 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.539 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.539 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.539 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:23.504904+0000", 2024-02-05T08:27:13.539 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.1542999999999998e-05 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: "description": "osd_op(client.25820.0:246441 11.c 11:38a19f6f:::benchmark_data_smithi134_205380_object111339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: "initiated_at": "2024-02-05T08:26:25.618292+0000", 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: "age": 47.794662146999997, 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.102795189, 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: "type_data": { 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: "client_info": { 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: "client": "client.25820", 2024-02-05T08:27:13.540 INFO:teuthology.orchestra.run.smithi134.stdout: "client_addr": "172.21.15.134:0/3532812543", 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: "tid": 246441 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: "events": [ 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "initiated", 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.618292+0000", 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "header_read", 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.618292+0000", 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.541 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "throttled", 2024-02-05T08:27:13.542 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.618293+0000", 2024-02-05T08:27:13.542 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.285e-06 2024-02-05T08:27:13.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.542 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "all_read", 2024-02-05T08:27:13.542 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.618299+0000", 2024-02-05T08:27:13.542 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 5.5219999999999998e-06 2024-02-05T08:27:13.542 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.542 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.542 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "dispatched", 2024-02-05T08:27:13.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.618309+0000", 2024-02-05T08:27:13.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 1.0798999999999999e-05 2024-02-05T08:27:13.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.543 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "queued_for_pg", 2024-02-05T08:27:13.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.618318+0000", 2024-02-05T08:27:13.543 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 8.3399999999999998e-06 2024-02-05T08:27:13.543 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.543 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.543 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "reached_pg", 2024-02-05T08:27:13.543 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.624819+0000", 2024-02-05T08:27:13.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.0065014900000000004 2024-02-05T08:27:13.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.544 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "started", 2024-02-05T08:27:13.544 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.624892+0000", 2024-02-05T08:27:13.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 7.2789999999999999e-05 2024-02-05T08:27:13.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.544 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "waiting for subops from 2,4", 2024-02-05T08:27:13.544 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.624962+0000", 2024-02-05T08:27:13.544 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 6.9826999999999994e-05 2024-02-05T08:27:13.544 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.544 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "op_commit", 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.627538+0000", 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.002576559 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.628520+0000", 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.00098179800000000009 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "sub_op_commit_rec", 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.721022+0000", 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 0.092501748999999994 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.545 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "commit_sent", 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.721062+0000", 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 4.0198999999999999e-05 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout: }, 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout: { 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout: "event": "done", 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout: "time": "2024-02-05T08:26:25.721087+0000", 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout: "duration": 2.4831e-05 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout: } 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout: ] 2024-02-05T08:27:13.546 INFO:teuthology.orchestra.run.smithi134.stdout:} 2024-02-05T08:27:14.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:27:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:27:15.012 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:15.243 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:15 smithi134 ceph-mon[30086]: pgmap v1013: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.9 KiB/s rd, 3 op/s 2024-02-05T08:27:15.243 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:15 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:15.244 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:15 smithi134 ceph-mon[39278]: pgmap v1013: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.9 KiB/s rd, 3 op/s 2024-02-05T08:27:15.244 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:15 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:15 smithi196 ceph-mon[33018]: pgmap v1013: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.9 KiB/s rd, 3 op/s 2024-02-05T08:27:15.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:15 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:16.037 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:27:16.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:16 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2687716569' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:27:16.064 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:16 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:27:16.164 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:16 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2687716569' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:27:16.164 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:16 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:27:16.164 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:16 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2687716569' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:27:16.164 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:16 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:27:17.093 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:17 smithi134 ceph-mon[30086]: pgmap v1014: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.5 KiB/s rd, 2 op/s 2024-02-05T08:27:17.093 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:17 smithi134 ceph-mon[30086]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:27:17.093 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:17 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:27:17.094 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:17 smithi134 ceph-mon[30086]: osdmap e437: 8 total, 8 up, 8 in 2024-02-05T08:27:17.094 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:17 smithi134 ceph-mon[30086]: 3.4 scrub starts 2024-02-05T08:27:17.094 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:17 smithi134 ceph-mon[30086]: 3.4 scrub ok 2024-02-05T08:27:17.094 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:17 smithi134 ceph-mon[39278]: pgmap v1014: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.5 KiB/s rd, 2 op/s 2024-02-05T08:27:17.094 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:17 smithi134 ceph-mon[39278]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:27:17.094 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:17 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:27:17.094 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:17 smithi134 ceph-mon[39278]: osdmap e437: 8 total, 8 up, 8 in 2024-02-05T08:27:17.094 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:17 smithi134 ceph-mon[39278]: 3.4 scrub starts 2024-02-05T08:27:17.094 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:17 smithi134 ceph-mon[39278]: 3.4 scrub ok 2024-02-05T08:27:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:17 smithi196 ceph-mon[33018]: pgmap v1014: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.5 KiB/s rd, 2 op/s 2024-02-05T08:27:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:17 smithi196 ceph-mon[33018]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T08:27:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:17 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:27:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:17 smithi196 ceph-mon[33018]: osdmap e437: 8 total, 8 up, 8 in 2024-02-05T08:27:17.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:17 smithi196 ceph-mon[33018]: 3.4 scrub starts 2024-02-05T08:27:17.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:17 smithi196 ceph-mon[33018]: 3.4 scrub ok 2024-02-05T08:27:18.023 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:27:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:27:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:27:18.038 INFO:teuthology.orchestra.run.smithi134.stderr:set pool 1 pgp_num to 11 2024-02-05T08:27:18.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:18 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:27:18.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:18 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:27:18.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:18 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4288464655' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-02-05T08:27:18.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:18 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:27:18.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:18 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:27:18.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:18 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4288464655' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-02-05T08:27:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:18 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:27:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:18 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:27:18.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:18 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4288464655' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-02-05T08:27:18.763 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-02-05T08:27:18.876 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T08:27:19.011 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:19 smithi196 ceph-mon[33018]: 6.1d scrub starts 2024-02-05T08:27:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:19 smithi196 ceph-mon[33018]: 6.1d scrub ok 2024-02-05T08:27:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:19 smithi196 ceph-mon[33018]: pgmap v1016: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.6 KiB/s rd, 22 KiB/s wr, 4 op/s; 21 KiB/s, 0 objects/s recovering 2024-02-05T08:27:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:19 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4288464655' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-02-05T08:27:19.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:19 smithi196 ceph-mon[33018]: osdmap e438: 8 total, 8 up, 8 in 2024-02-05T08:27:19.438 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:19 smithi134 ceph-mon[30086]: 6.1d scrub starts 2024-02-05T08:27:19.438 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:19 smithi134 ceph-mon[30086]: 6.1d scrub ok 2024-02-05T08:27:19.438 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:19 smithi134 ceph-mon[30086]: pgmap v1016: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.6 KiB/s rd, 22 KiB/s wr, 4 op/s; 21 KiB/s, 0 objects/s recovering 2024-02-05T08:27:19.438 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:19 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4288464655' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-02-05T08:27:19.438 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:19 smithi134 ceph-mon[30086]: osdmap e438: 8 total, 8 up, 8 in 2024-02-05T08:27:19.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:19 smithi134 ceph-mon[39278]: 6.1d scrub starts 2024-02-05T08:27:19.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:19 smithi134 ceph-mon[39278]: 6.1d scrub ok 2024-02-05T08:27:19.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:19 smithi134 ceph-mon[39278]: pgmap v1016: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 3.6 KiB/s rd, 22 KiB/s wr, 4 op/s; 21 KiB/s, 0 objects/s recovering 2024-02-05T08:27:19.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:19 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4288464655' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-02-05T08:27:19.439 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:19 smithi134 ceph-mon[39278]: osdmap e438: 8 total, 8 up, 8 in 2024-02-05T08:27:20.268 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:20 smithi134 ceph-mon[30086]: 3.16 scrub starts 2024-02-05T08:27:20.268 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:20 smithi134 ceph-mon[30086]: 3.16 scrub ok 2024-02-05T08:27:20.268 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:20 smithi134 ceph-mon[39278]: 3.16 scrub starts 2024-02-05T08:27:20.269 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:20 smithi134 ceph-mon[39278]: 3.16 scrub ok 2024-02-05T08:27:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:20 smithi196 ceph-mon[33018]: 3.16 scrub starts 2024-02-05T08:27:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:20 smithi196 ceph-mon[33018]: 3.16 scrub ok 2024-02-05T08:27:20.802 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:20.820 INFO:teuthology.orchestra.run.smithi134.stdout:mon.a: {} 2024-02-05T08:27:20.832 INFO:teuthology.orchestra.run.smithi134.stdout:mon.b: {} 2024-02-05T08:27:20.842 INFO:teuthology.orchestra.run.smithi134.stdout:mon.c: {} 2024-02-05T08:27:20.843 INFO:teuthology.orchestra.run.smithi134.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T08:27:20.843 INFO:teuthology.orchestra.run.smithi134.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T08:27:20.843 INFO:teuthology.orchestra.run.smithi134.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T08:27:21.107 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[30086]: 6.19 scrub starts 2024-02-05T08:27:21.107 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[30086]: 6.19 scrub ok 2024-02-05T08:27:21.107 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[30086]: pgmap v1018: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:27:21.107 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[30086]: 1.3 scrub starts 2024-02-05T08:27:21.108 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[30086]: 1.3 scrub ok 2024-02-05T08:27:21.108 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[30086]: 6.15 scrub starts 2024-02-05T08:27:21.108 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:27:21.108 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:27:21.108 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[30086]: 6.15 scrub ok 2024-02-05T08:27:21.108 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/492519446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:27:21.109 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[39278]: 6.19 scrub starts 2024-02-05T08:27:21.109 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[39278]: 6.19 scrub ok 2024-02-05T08:27:21.109 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[39278]: pgmap v1018: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:27:21.109 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[39278]: 1.3 scrub starts 2024-02-05T08:27:21.109 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[39278]: 1.3 scrub ok 2024-02-05T08:27:21.109 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[39278]: 6.15 scrub starts 2024-02-05T08:27:21.109 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:27:21.109 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:27:21.109 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[39278]: 6.15 scrub ok 2024-02-05T08:27:21.109 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:21 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/492519446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:27:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:21 smithi196 ceph-mon[33018]: 6.19 scrub starts 2024-02-05T08:27:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:21 smithi196 ceph-mon[33018]: 6.19 scrub ok 2024-02-05T08:27:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:21 smithi196 ceph-mon[33018]: pgmap v1018: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:27:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:21 smithi196 ceph-mon[33018]: 1.3 scrub starts 2024-02-05T08:27:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:21 smithi196 ceph-mon[33018]: 1.3 scrub ok 2024-02-05T08:27:21.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:21 smithi196 ceph-mon[33018]: 6.15 scrub starts 2024-02-05T08:27:21.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:21 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:27:21.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:21 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:27:21.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:21 smithi196 ceph-mon[33018]: 6.15 scrub ok 2024-02-05T08:27:21.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:21 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/492519446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T08:27:21.834 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 0 1 2024-02-05T08:27:22.072 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[30086]: osdmap e439: 8 total, 8 up, 8 in 2024-02-05T08:27:22.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[30086]: 3.0 scrub starts 2024-02-05T08:27:22.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[30086]: 3.0 scrub ok 2024-02-05T08:27:22.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[30086]: 4.5 scrub starts 2024-02-05T08:27:22.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[30086]: 4.5 scrub ok 2024-02-05T08:27:22.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:27:22.073 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:27:22.073 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[39278]: osdmap e439: 8 total, 8 up, 8 in 2024-02-05T08:27:22.073 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[39278]: 3.0 scrub starts 2024-02-05T08:27:22.073 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[39278]: 3.0 scrub ok 2024-02-05T08:27:22.073 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[39278]: 4.5 scrub starts 2024-02-05T08:27:22.073 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[39278]: 4.5 scrub ok 2024-02-05T08:27:22.074 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:27:22.074 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:22 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:27:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:22 smithi196 ceph-mon[33018]: osdmap e439: 8 total, 8 up, 8 in 2024-02-05T08:27:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:22 smithi196 ceph-mon[33018]: 3.0 scrub starts 2024-02-05T08:27:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:22 smithi196 ceph-mon[33018]: 3.0 scrub ok 2024-02-05T08:27:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:22 smithi196 ceph-mon[33018]: 4.5 scrub starts 2024-02-05T08:27:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:22 smithi196 ceph-mon[33018]: 4.5 scrub ok 2024-02-05T08:27:22.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:22 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:27:22.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:22 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:27:22.842 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:23 smithi196 ceph-mon[33018]: 4.f scrub starts 2024-02-05T08:27:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:23 smithi196 ceph-mon[33018]: 4.f scrub ok 2024-02-05T08:27:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:23 smithi196 ceph-mon[33018]: pgmap v1020: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:23 smithi196 ceph-mon[33018]: osdmap e440: 8 total, 8 up, 8 in 2024-02-05T08:27:23.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:23 smithi196 ceph-mon[33018]: 4.10 scrub starts 2024-02-05T08:27:23.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:23 smithi196 ceph-mon[33018]: 4.10 scrub ok 2024-02-05T08:27:23.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:23 smithi196 ceph-mon[33018]: 3.17 scrub starts 2024-02-05T08:27:23.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:23 smithi196 ceph-mon[33018]: 3.17 scrub ok 2024-02-05T08:27:23.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:23 smithi196 ceph-mon[33018]: 3.15 scrub starts 2024-02-05T08:27:23.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:23 smithi196 ceph-mon[33018]: 3.15 scrub ok 2024-02-05T08:27:23.399 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[30086]: 4.f scrub starts 2024-02-05T08:27:23.399 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[30086]: 4.f scrub ok 2024-02-05T08:27:23.399 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[30086]: pgmap v1020: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:23.399 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[30086]: osdmap e440: 8 total, 8 up, 8 in 2024-02-05T08:27:23.399 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[30086]: 4.10 scrub starts 2024-02-05T08:27:23.399 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[30086]: 4.10 scrub ok 2024-02-05T08:27:23.399 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[30086]: 3.17 scrub starts 2024-02-05T08:27:23.399 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[30086]: 3.17 scrub ok 2024-02-05T08:27:23.399 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[30086]: 3.15 scrub starts 2024-02-05T08:27:23.399 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[30086]: 3.15 scrub ok 2024-02-05T08:27:23.400 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[39278]: 4.f scrub starts 2024-02-05T08:27:23.400 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[39278]: 4.f scrub ok 2024-02-05T08:27:23.400 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[39278]: pgmap v1020: 142 pgs: 142 active+clean; 635 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:23.400 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[39278]: osdmap e440: 8 total, 8 up, 8 in 2024-02-05T08:27:23.400 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[39278]: 4.10 scrub starts 2024-02-05T08:27:23.400 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[39278]: 4.10 scrub ok 2024-02-05T08:27:23.400 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[39278]: 3.17 scrub starts 2024-02-05T08:27:23.400 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[39278]: 3.17 scrub ok 2024-02-05T08:27:23.400 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[39278]: 3.15 scrub starts 2024-02-05T08:27:23.400 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:23 smithi134 ceph-mon[39278]: 3.15 scrub ok 2024-02-05T08:27:24.060 INFO:teuthology.orchestra.run.smithi134.stderr:noscrub is unset 2024-02-05T08:27:24.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:27:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:27:24.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:24 smithi196 ceph-mon[33018]: 4.d scrub starts 2024-02-05T08:27:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:24 smithi196 ceph-mon[33018]: 4.d scrub ok 2024-02-05T08:27:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:24 smithi196 ceph-mon[33018]: 6.1b scrub starts 2024-02-05T08:27:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:24 smithi196 ceph-mon[33018]: 4.b scrub starts 2024-02-05T08:27:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:24 smithi196 ceph-mon[33018]: 4.b scrub ok 2024-02-05T08:27:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:24 smithi196 ceph-mon[33018]: 6.1b scrub ok 2024-02-05T08:27:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:24 smithi196 ceph-mon[33018]: 5.10 scrub starts 2024-02-05T08:27:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:24 smithi196 ceph-mon[33018]: 5.10 scrub ok 2024-02-05T08:27:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:24 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1224393464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:27:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:24 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:27:24.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:24 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:27:24.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[30086]: 4.d scrub starts 2024-02-05T08:27:24.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[30086]: 4.d scrub ok 2024-02-05T08:27:24.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[30086]: 6.1b scrub starts 2024-02-05T08:27:24.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[30086]: 4.b scrub starts 2024-02-05T08:27:24.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[30086]: 4.b scrub ok 2024-02-05T08:27:24.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[30086]: 6.1b scrub ok 2024-02-05T08:27:24.440 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[30086]: 5.10 scrub starts 2024-02-05T08:27:24.441 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[30086]: 5.10 scrub ok 2024-02-05T08:27:24.441 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1224393464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:27:24.441 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:27:24.441 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:27:24.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[39278]: 4.d scrub starts 2024-02-05T08:27:24.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[39278]: 4.d scrub ok 2024-02-05T08:27:24.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[39278]: 6.1b scrub starts 2024-02-05T08:27:24.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[39278]: 4.b scrub starts 2024-02-05T08:27:24.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[39278]: 4.b scrub ok 2024-02-05T08:27:24.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[39278]: 6.1b scrub ok 2024-02-05T08:27:24.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[39278]: 5.10 scrub starts 2024-02-05T08:27:24.441 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[39278]: 5.10 scrub ok 2024-02-05T08:27:24.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1224393464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T08:27:24.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:27:24.442 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:24 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:27:24.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T08:27:25.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:25 smithi134 ceph-mon[30086]: 2.2 scrub starts 2024-02-05T08:27:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:25 smithi134 ceph-mon[30086]: 2.2 scrub ok 2024-02-05T08:27:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:25 smithi134 ceph-mon[30086]: pgmap v1022: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:25 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1224393464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:27:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:25 smithi134 ceph-mon[30086]: osdmap e441: 8 total, 8 up, 8 in 2024-02-05T08:27:25.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:25 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:25 smithi134 ceph-mon[39278]: 2.2 scrub starts 2024-02-05T08:27:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:25 smithi134 ceph-mon[39278]: 2.2 scrub ok 2024-02-05T08:27:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:25 smithi134 ceph-mon[39278]: pgmap v1022: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:25 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1224393464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:27:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:25 smithi134 ceph-mon[39278]: osdmap e441: 8 total, 8 up, 8 in 2024-02-05T08:27:25.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:25 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:25 smithi196 ceph-mon[33018]: 2.2 scrub starts 2024-02-05T08:27:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:25 smithi196 ceph-mon[33018]: 2.2 scrub ok 2024-02-05T08:27:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:25 smithi196 ceph-mon[33018]: pgmap v1022: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:25 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1224393464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T08:27:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:25 smithi196 ceph-mon[33018]: osdmap e441: 8 total, 8 up, 8 in 2024-02-05T08:27:25.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:25 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:25.469 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:26.251 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:26 smithi134 ceph-mon[30086]: 5.0 scrub starts 2024-02-05T08:27:26.251 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:26 smithi134 ceph-mon[30086]: 5.0 scrub ok 2024-02-05T08:27:26.251 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:26 smithi134 ceph-mon[30086]: 5.5 scrub starts 2024-02-05T08:27:26.251 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:26 smithi134 ceph-mon[30086]: 5.5 scrub ok 2024-02-05T08:27:26.251 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:26 smithi134 ceph-mon[30086]: 3.d scrub starts 2024-02-05T08:27:26.251 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:26 smithi134 ceph-mon[30086]: 3.d scrub ok 2024-02-05T08:27:26.252 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:26 smithi134 ceph-mon[39278]: 5.0 scrub starts 2024-02-05T08:27:26.252 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:26 smithi134 ceph-mon[39278]: 5.0 scrub ok 2024-02-05T08:27:26.252 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:26 smithi134 ceph-mon[39278]: 5.5 scrub starts 2024-02-05T08:27:26.252 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:26 smithi134 ceph-mon[39278]: 5.5 scrub ok 2024-02-05T08:27:26.252 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:26 smithi134 ceph-mon[39278]: 3.d scrub starts 2024-02-05T08:27:26.252 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:26 smithi134 ceph-mon[39278]: 3.d scrub ok 2024-02-05T08:27:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:26 smithi196 ceph-mon[33018]: 5.0 scrub starts 2024-02-05T08:27:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:26 smithi196 ceph-mon[33018]: 5.0 scrub ok 2024-02-05T08:27:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:26 smithi196 ceph-mon[33018]: 5.5 scrub starts 2024-02-05T08:27:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:26 smithi196 ceph-mon[33018]: 5.5 scrub ok 2024-02-05T08:27:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:26 smithi196 ceph-mon[33018]: 3.d scrub starts 2024-02-05T08:27:26.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:26 smithi196 ceph-mon[33018]: 3.d scrub ok 2024-02-05T08:27:27.069 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.0 to 1 (10000) 2024-02-05T08:27:27.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:27 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:27:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:27 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:27:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:27 smithi134 ceph-mon[30086]: pgmap v1024: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:27 smithi134 ceph-mon[30086]: 5.3 scrub starts 2024-02-05T08:27:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:27 smithi134 ceph-mon[30086]: 5.3 scrub ok 2024-02-05T08:27:27.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:27 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2498920637' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:27:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:27 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:27:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:27 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:27:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:27 smithi134 ceph-mon[39278]: pgmap v1024: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:27 smithi134 ceph-mon[39278]: 5.3 scrub starts 2024-02-05T08:27:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:27 smithi134 ceph-mon[39278]: 5.3 scrub ok 2024-02-05T08:27:27.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:27 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2498920637' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:27:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:27 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:27:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:27 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:27:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:27 smithi196 ceph-mon[33018]: pgmap v1024: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:27 smithi196 ceph-mon[33018]: 5.3 scrub starts 2024-02-05T08:27:27.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:27 smithi196 ceph-mon[33018]: 5.3 scrub ok 2024-02-05T08:27:27.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:27 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2498920637' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:27:28.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:27:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:27:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:27:28.089 INFO:teuthology.orchestra.run.smithi134.stderr:nodeep-scrub is unset 2024-02-05T08:27:28.218 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-05T08:27:28.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:28 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2498920637' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:27:28.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:28 smithi134 ceph-mon[30086]: osdmap e442: 8 total, 8 up, 8 in 2024-02-05T08:27:28.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:28 smithi134 ceph-mon[30086]: 6.10 scrub starts 2024-02-05T08:27:28.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:28 smithi134 ceph-mon[30086]: 6.10 scrub ok 2024-02-05T08:27:28.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:28 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2056911443' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:27:28.519 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:28 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:27:28.519 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:28 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2498920637' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:27:28.519 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:28 smithi134 ceph-mon[39278]: osdmap e442: 8 total, 8 up, 8 in 2024-02-05T08:27:28.520 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:28 smithi134 ceph-mon[39278]: 6.10 scrub starts 2024-02-05T08:27:28.520 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:28 smithi134 ceph-mon[39278]: 6.10 scrub ok 2024-02-05T08:27:28.520 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:28 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2056911443' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:27:28.520 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:28 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:27:28.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:28 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2498920637' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:27:28.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:28 smithi196 ceph-mon[33018]: osdmap e442: 8 total, 8 up, 8 in 2024-02-05T08:27:28.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:28 smithi196 ceph-mon[33018]: 6.10 scrub starts 2024-02-05T08:27:28.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:28 smithi196 ceph-mon[33018]: 6.10 scrub ok 2024-02-05T08:27:28.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:28 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2056911443' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:27:28.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:28 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T08:27:28.926 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:29.339 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[30086]: pgmap v1026: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:29.339 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[30086]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:27:29.339 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:27:29.340 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:27:29.340 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[30086]: osdmap e443: 8 total, 8 up, 8 in 2024-02-05T08:27:29.340 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[30086]: 3.11 scrub starts 2024-02-05T08:27:29.340 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[30086]: 3.11 scrub ok 2024-02-05T08:27:29.340 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[39278]: pgmap v1026: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:29.340 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[39278]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:27:29.340 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:27:29.340 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:27:29.340 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[39278]: osdmap e443: 8 total, 8 up, 8 in 2024-02-05T08:27:29.340 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[39278]: 3.11 scrub starts 2024-02-05T08:27:29.341 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:29 smithi134 ceph-mon[39278]: 3.11 scrub ok 2024-02-05T08:27:29.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:29 smithi196 ceph-mon[33018]: pgmap v1026: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:29.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:29 smithi196 ceph-mon[33018]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T08:27:29.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:29 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:27:29.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:29 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T08:27:29.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:29 smithi196 ceph-mon[33018]: osdmap e443: 8 total, 8 up, 8 in 2024-02-05T08:27:29.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:29 smithi196 ceph-mon[33018]: 3.11 scrub starts 2024-02-05T08:27:29.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:29 smithi196 ceph-mon[33018]: 3.11 scrub ok 2024-02-05T08:27:30.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:30 smithi134 ceph-mon[30086]: 4.16 scrub starts 2024-02-05T08:27:30.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:30 smithi134 ceph-mon[30086]: 4.16 scrub ok 2024-02-05T08:27:30.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:30 smithi134 ceph-mon[30086]: 5.1a deep-scrub starts 2024-02-05T08:27:30.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:30 smithi134 ceph-mon[30086]: 5.1a deep-scrub ok 2024-02-05T08:27:30.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:30 smithi134 ceph-mon[30086]: 3.10 scrub starts 2024-02-05T08:27:30.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:30 smithi134 ceph-mon[30086]: 3.10 scrub ok 2024-02-05T08:27:30.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:30 smithi134 ceph-mon[39278]: 4.16 scrub starts 2024-02-05T08:27:30.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:30 smithi134 ceph-mon[39278]: 4.16 scrub ok 2024-02-05T08:27:30.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:30 smithi134 ceph-mon[39278]: 5.1a deep-scrub starts 2024-02-05T08:27:30.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:30 smithi134 ceph-mon[39278]: 5.1a deep-scrub ok 2024-02-05T08:27:30.353 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:30 smithi134 ceph-mon[39278]: 3.10 scrub starts 2024-02-05T08:27:30.354 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:30 smithi134 ceph-mon[39278]: 3.10 scrub ok 2024-02-05T08:27:30.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:30 smithi196 ceph-mon[33018]: 4.16 scrub starts 2024-02-05T08:27:30.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:30 smithi196 ceph-mon[33018]: 4.16 scrub ok 2024-02-05T08:27:30.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:30 smithi196 ceph-mon[33018]: 5.1a deep-scrub starts 2024-02-05T08:27:30.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:30 smithi196 ceph-mon[33018]: 5.1a deep-scrub ok 2024-02-05T08:27:30.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:30 smithi196 ceph-mon[33018]: 3.10 scrub starts 2024-02-05T08:27:30.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:30 smithi196 ceph-mon[33018]: 3.10 scrub ok 2024-02-05T08:27:31.090 INFO:teuthology.orchestra.run.smithi134.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-05T08:27:31.417 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:31 smithi134 ceph-mon[30086]: 4.2 scrub starts 2024-02-05T08:27:31.417 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:31 smithi134 ceph-mon[30086]: 4.2 scrub ok 2024-02-05T08:27:31.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:31 smithi134 ceph-mon[30086]: pgmap v1028: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:31.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:31 smithi134 ceph-mon[30086]: 4.7 scrub starts 2024-02-05T08:27:31.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:31 smithi134 ceph-mon[30086]: 4.7 scrub ok 2024-02-05T08:27:31.418 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:31 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/753861440' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:27:31.418 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:31 smithi134 ceph-mon[39278]: 4.2 scrub starts 2024-02-05T08:27:31.418 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:31 smithi134 ceph-mon[39278]: 4.2 scrub ok 2024-02-05T08:27:31.418 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:31 smithi134 ceph-mon[39278]: pgmap v1028: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:31.418 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:31 smithi134 ceph-mon[39278]: 4.7 scrub starts 2024-02-05T08:27:31.418 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:31 smithi134 ceph-mon[39278]: 4.7 scrub ok 2024-02-05T08:27:31.418 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:31 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/753861440' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:27:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:31 smithi196 ceph-mon[33018]: 4.2 scrub starts 2024-02-05T08:27:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:31 smithi196 ceph-mon[33018]: 4.2 scrub ok 2024-02-05T08:27:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:31 smithi196 ceph-mon[33018]: pgmap v1028: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:31 smithi196 ceph-mon[33018]: 4.7 scrub starts 2024-02-05T08:27:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:31 smithi196 ceph-mon[33018]: 4.7 scrub ok 2024-02-05T08:27:31.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:31 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/753861440' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T08:27:31.857 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 2 1 2024-02-05T08:27:32.104 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:32.360 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: 5.f scrub starts 2024-02-05T08:27:32.360 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: 5.f scrub ok 2024-02-05T08:27:32.360 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: 5.16 scrub starts 2024-02-05T08:27:32.360 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: 5.16 scrub ok 2024-02-05T08:27:32.360 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:27:32.360 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:27:32.360 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/753861440' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:27:32.360 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: osdmap e444: 8 total, 8 up, 8 in 2024-02-05T08:27:32.360 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: 3.1b deep-scrub starts 2024-02-05T08:27:32.361 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: 3.1b deep-scrub ok 2024-02-05T08:27:32.361 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: 2.0 scrub starts 2024-02-05T08:27:32.361 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: 2.0 scrub ok 2024-02-05T08:27:32.361 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:27:32.361 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:27:32.361 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: 5.f scrub starts 2024-02-05T08:27:32.362 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: 5.f scrub ok 2024-02-05T08:27:32.362 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: 5.16 scrub starts 2024-02-05T08:27:32.362 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: 5.16 scrub ok 2024-02-05T08:27:32.362 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:27:32.362 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:27:32.362 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/753861440' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:27:32.362 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: osdmap e444: 8 total, 8 up, 8 in 2024-02-05T08:27:32.362 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: 3.1b deep-scrub starts 2024-02-05T08:27:32.363 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: 3.1b deep-scrub ok 2024-02-05T08:27:32.363 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: 2.0 scrub starts 2024-02-05T08:27:32.363 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: 2.0 scrub ok 2024-02-05T08:27:32.363 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:27:32.363 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:32 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:27:32.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: 5.f scrub starts 2024-02-05T08:27:32.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: 5.f scrub ok 2024-02-05T08:27:32.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: 5.16 scrub starts 2024-02-05T08:27:32.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: 5.16 scrub ok 2024-02-05T08:27:32.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:27:32.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:27:32.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/753861440' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-05T08:27:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: osdmap e444: 8 total, 8 up, 8 in 2024-02-05T08:27:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: 3.1b deep-scrub starts 2024-02-05T08:27:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: 3.1b deep-scrub ok 2024-02-05T08:27:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: 2.0 scrub starts 2024-02-05T08:27:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: 2.0 scrub ok 2024-02-05T08:27:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:27:32.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:32 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:27:33.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:33 smithi134 ceph-mon[30086]: pgmap v1030: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:33.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:33 smithi134 ceph-mon[30086]: 6.b scrub starts 2024-02-05T08:27:33.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:33 smithi134 ceph-mon[30086]: 6.b scrub ok 2024-02-05T08:27:33.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:33 smithi134 ceph-mon[30086]: 6.a scrub starts 2024-02-05T08:27:33.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:33 smithi134 ceph-mon[30086]: 6.a scrub ok 2024-02-05T08:27:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:33 smithi134 ceph-mon[39278]: pgmap v1030: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:33 smithi134 ceph-mon[39278]: 6.b scrub starts 2024-02-05T08:27:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:33 smithi134 ceph-mon[39278]: 6.b scrub ok 2024-02-05T08:27:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:33 smithi134 ceph-mon[39278]: 6.a scrub starts 2024-02-05T08:27:33.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:33 smithi134 ceph-mon[39278]: 6.a scrub ok 2024-02-05T08:27:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:33 smithi196 ceph-mon[33018]: pgmap v1030: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:27:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:33 smithi196 ceph-mon[33018]: 6.b scrub starts 2024-02-05T08:27:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:33 smithi196 ceph-mon[33018]: 6.b scrub ok 2024-02-05T08:27:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:33 smithi196 ceph-mon[33018]: 6.a scrub starts 2024-02-05T08:27:33.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:33 smithi196 ceph-mon[33018]: 6.a scrub ok 2024-02-05T08:27:34.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:34 smithi134 ceph-mon[39278]: 6.e scrub starts 2024-02-05T08:27:34.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:34 smithi134 ceph-mon[39278]: 6.e scrub ok 2024-02-05T08:27:34.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:34 smithi134 ceph-mon[39278]: 3.3 scrub starts 2024-02-05T08:27:34.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:34 smithi134 ceph-mon[39278]: 3.3 scrub ok 2024-02-05T08:27:34.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:34 smithi134 ceph-mon[39278]: 1.a scrub starts 2024-02-05T08:27:34.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:34 smithi134 ceph-mon[39278]: 1.a scrub ok 2024-02-05T08:27:34.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:34 smithi134 ceph-mon[30086]: 6.e scrub starts 2024-02-05T08:27:34.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:34 smithi134 ceph-mon[30086]: 6.e scrub ok 2024-02-05T08:27:34.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:34 smithi134 ceph-mon[30086]: 3.3 scrub starts 2024-02-05T08:27:34.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:34 smithi134 ceph-mon[30086]: 3.3 scrub ok 2024-02-05T08:27:34.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:34 smithi134 ceph-mon[30086]: 1.a scrub starts 2024-02-05T08:27:34.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:34 smithi134 ceph-mon[30086]: 1.a scrub ok 2024-02-05T08:27:34.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:27:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:27:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:34 smithi196 ceph-mon[33018]: 6.e scrub starts 2024-02-05T08:27:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:34 smithi196 ceph-mon[33018]: 6.e scrub ok 2024-02-05T08:27:34.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:34 smithi196 ceph-mon[33018]: 3.3 scrub starts 2024-02-05T08:27:34.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:34 smithi196 ceph-mon[33018]: 3.3 scrub ok 2024-02-05T08:27:34.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:34 smithi196 ceph-mon[33018]: 1.a scrub starts 2024-02-05T08:27:34.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:34 smithi196 ceph-mon[33018]: 1.a scrub ok 2024-02-05T08:27:35.122 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.2 to 1 (10000) 2024-02-05T08:27:35.434 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:35 smithi134 ceph-mon[30086]: pgmap v1031: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T08:27:35.434 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:35 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1895438826' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:27:35.435 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:35 smithi134 ceph-mon[30086]: 6.1a scrub starts 2024-02-05T08:27:35.435 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:35 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:35.435 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:35 smithi134 ceph-mon[30086]: 6.1a scrub ok 2024-02-05T08:27:35.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:35 smithi134 ceph-mon[39278]: pgmap v1031: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T08:27:35.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:35 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1895438826' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:27:35.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:35 smithi134 ceph-mon[39278]: 6.1a scrub starts 2024-02-05T08:27:35.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:35 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:35.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:35 smithi134 ceph-mon[39278]: 6.1a scrub ok 2024-02-05T08:27:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:35 smithi196 ceph-mon[33018]: pgmap v1031: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T08:27:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:35 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1895438826' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:27:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:35 smithi196 ceph-mon[33018]: 6.1a scrub starts 2024-02-05T08:27:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:35 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:35.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:35 smithi196 ceph-mon[33018]: 6.1a scrub ok 2024-02-05T08:27:35.816 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 2 1 2024-02-05T08:27:36.061 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:36.430 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:36 smithi134 ceph-mon[30086]: 6.7 scrub starts 2024-02-05T08:27:36.431 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:36 smithi134 ceph-mon[30086]: 6.7 scrub ok 2024-02-05T08:27:36.431 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:36 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1895438826' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:27:36.431 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:36 smithi134 ceph-mon[30086]: osdmap e445: 8 total, 8 up, 8 in 2024-02-05T08:27:36.431 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:36 smithi134 ceph-mon[30086]: 3.12 scrub starts 2024-02-05T08:27:36.431 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:36 smithi134 ceph-mon[30086]: 3.12 scrub ok 2024-02-05T08:27:36.431 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:36 smithi134 ceph-mon[39278]: 6.7 scrub starts 2024-02-05T08:27:36.431 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:36 smithi134 ceph-mon[39278]: 6.7 scrub ok 2024-02-05T08:27:36.431 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:36 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1895438826' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:27:36.431 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:36 smithi134 ceph-mon[39278]: osdmap e445: 8 total, 8 up, 8 in 2024-02-05T08:27:36.431 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:36 smithi134 ceph-mon[39278]: 3.12 scrub starts 2024-02-05T08:27:36.432 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:36 smithi134 ceph-mon[39278]: 3.12 scrub ok 2024-02-05T08:27:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:36 smithi196 ceph-mon[33018]: 6.7 scrub starts 2024-02-05T08:27:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:36 smithi196 ceph-mon[33018]: 6.7 scrub ok 2024-02-05T08:27:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:36 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1895438826' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:27:36.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:36 smithi196 ceph-mon[33018]: osdmap e445: 8 total, 8 up, 8 in 2024-02-05T08:27:36.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:36 smithi196 ceph-mon[33018]: 3.12 scrub starts 2024-02-05T08:27:36.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:36 smithi196 ceph-mon[33018]: 3.12 scrub ok 2024-02-05T08:27:37.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:37 smithi134 ceph-mon[30086]: pgmap v1033: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:27:37.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:37 smithi134 ceph-mon[30086]: 3.9 scrub starts 2024-02-05T08:27:37.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:37 smithi134 ceph-mon[30086]: 3.9 scrub ok 2024-02-05T08:27:37.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:37 smithi134 ceph-mon[30086]: 3.a scrub starts 2024-02-05T08:27:37.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:37 smithi134 ceph-mon[30086]: 3.a scrub ok 2024-02-05T08:27:37.210 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:37 smithi134 ceph-mon[39278]: pgmap v1033: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:27:37.211 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:37 smithi134 ceph-mon[39278]: 3.9 scrub starts 2024-02-05T08:27:37.211 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:37 smithi134 ceph-mon[39278]: 3.9 scrub ok 2024-02-05T08:27:37.211 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:37 smithi134 ceph-mon[39278]: 3.a scrub starts 2024-02-05T08:27:37.211 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:37 smithi134 ceph-mon[39278]: 3.a scrub ok 2024-02-05T08:27:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:37 smithi196 ceph-mon[33018]: pgmap v1033: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:27:37.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:37 smithi196 ceph-mon[33018]: 3.9 scrub starts 2024-02-05T08:27:37.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:37 smithi196 ceph-mon[33018]: 3.9 scrub ok 2024-02-05T08:27:37.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:37 smithi196 ceph-mon[33018]: 3.a scrub starts 2024-02-05T08:27:37.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:37 smithi196 ceph-mon[33018]: 3.a scrub ok 2024-02-05T08:27:37.776 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:27:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:27:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:27:38.136 INFO:teuthology.orchestra.run.smithi134.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-02-05T08:27:38.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:38 smithi134 ceph-mon[30086]: 3.1f scrub starts 2024-02-05T08:27:38.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:38 smithi134 ceph-mon[30086]: 3.1f scrub ok 2024-02-05T08:27:38.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:38 smithi134 ceph-mon[30086]: 5.17 scrub starts 2024-02-05T08:27:38.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:38 smithi134 ceph-mon[30086]: 5.17 scrub ok 2024-02-05T08:27:38.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:38 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1389848621' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:27:38.510 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:38 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:27:38.510 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:38 smithi134 ceph-mon[39278]: 3.1f scrub starts 2024-02-05T08:27:38.511 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:38 smithi134 ceph-mon[39278]: 3.1f scrub ok 2024-02-05T08:27:38.511 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:38 smithi134 ceph-mon[39278]: 5.17 scrub starts 2024-02-05T08:27:38.511 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:38 smithi134 ceph-mon[39278]: 5.17 scrub ok 2024-02-05T08:27:38.511 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:38 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1389848621' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:27:38.511 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:38 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:27:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:38 smithi196 ceph-mon[33018]: 3.1f scrub starts 2024-02-05T08:27:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:38 smithi196 ceph-mon[33018]: 3.1f scrub ok 2024-02-05T08:27:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:38 smithi196 ceph-mon[33018]: 5.17 scrub starts 2024-02-05T08:27:38.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:38 smithi196 ceph-mon[33018]: 5.17 scrub ok 2024-02-05T08:27:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:38 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1389848621' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:27:38.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:38 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T08:27:38.813 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 3 1 2024-02-05T08:27:39.061 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:39.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[30086]: pgmap v1034: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:27:39.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:27:39.447 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[30086]: osdmap e446: 8 total, 8 up, 8 in 2024-02-05T08:27:39.447 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[30086]: 4.13 scrub starts 2024-02-05T08:27:39.447 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[30086]: 4.13 scrub ok 2024-02-05T08:27:39.447 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:27:39.447 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:27:39.447 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[30086]: osdmap e447: 8 total, 8 up, 8 in 2024-02-05T08:27:39.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[39278]: pgmap v1034: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:27:39.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:27:39.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[39278]: osdmap e446: 8 total, 8 up, 8 in 2024-02-05T08:27:39.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[39278]: 4.13 scrub starts 2024-02-05T08:27:39.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[39278]: 4.13 scrub ok 2024-02-05T08:27:39.448 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:27:39.448 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:27:39.448 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:39 smithi134 ceph-mon[39278]: osdmap e447: 8 total, 8 up, 8 in 2024-02-05T08:27:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:39 smithi196 ceph-mon[33018]: pgmap v1034: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:27:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:39 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-05T08:27:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:39 smithi196 ceph-mon[33018]: osdmap e446: 8 total, 8 up, 8 in 2024-02-05T08:27:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:39 smithi196 ceph-mon[33018]: 4.13 scrub starts 2024-02-05T08:27:39.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:39 smithi196 ceph-mon[33018]: 4.13 scrub ok 2024-02-05T08:27:39.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:27:39.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:27:39.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:39 smithi196 ceph-mon[33018]: osdmap e447: 8 total, 8 up, 8 in 2024-02-05T08:27:40.209 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[30086]: 6.17 deep-scrub starts 2024-02-05T08:27:40.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[30086]: 3.1 scrub starts 2024-02-05T08:27:40.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[30086]: 3.1 scrub ok 2024-02-05T08:27:40.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[30086]: 3.1e deep-scrub starts 2024-02-05T08:27:40.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[30086]: 3.1e deep-scrub ok 2024-02-05T08:27:40.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[30086]: 6.17 deep-scrub ok 2024-02-05T08:27:40.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[30086]: 1.8 scrub starts 2024-02-05T08:27:40.210 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[30086]: 1.8 scrub ok 2024-02-05T08:27:40.210 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[39278]: 6.17 deep-scrub starts 2024-02-05T08:27:40.210 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[39278]: 3.1 scrub starts 2024-02-05T08:27:40.210 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[39278]: 3.1 scrub ok 2024-02-05T08:27:40.210 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[39278]: 3.1e deep-scrub starts 2024-02-05T08:27:40.210 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[39278]: 3.1e deep-scrub ok 2024-02-05T08:27:40.211 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[39278]: 6.17 deep-scrub ok 2024-02-05T08:27:40.211 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[39278]: 1.8 scrub starts 2024-02-05T08:27:40.211 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:40 smithi134 ceph-mon[39278]: 1.8 scrub ok 2024-02-05T08:27:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:40 smithi196 ceph-mon[33018]: 6.17 deep-scrub starts 2024-02-05T08:27:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:40 smithi196 ceph-mon[33018]: 3.1 scrub starts 2024-02-05T08:27:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:40 smithi196 ceph-mon[33018]: 3.1 scrub ok 2024-02-05T08:27:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:40 smithi196 ceph-mon[33018]: 3.1e deep-scrub starts 2024-02-05T08:27:40.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:40 smithi196 ceph-mon[33018]: 3.1e deep-scrub ok 2024-02-05T08:27:40.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:40 smithi196 ceph-mon[33018]: 6.17 deep-scrub ok 2024-02-05T08:27:40.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:40 smithi196 ceph-mon[33018]: 1.8 scrub starts 2024-02-05T08:27:40.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:40 smithi196 ceph-mon[33018]: 1.8 scrub ok 2024-02-05T08:27:41.139 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.3 to 1 (10000) 2024-02-05T08:27:41.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:41 smithi134 ceph-mon[39278]: pgmap v1037: 142 pgs: 6 peering, 136 active+clean; 378 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.5 KiB/s rd, 36 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-02-05T08:27:41.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:41 smithi134 ceph-mon[39278]: 4.14 scrub starts 2024-02-05T08:27:41.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:41 smithi134 ceph-mon[39278]: 4.14 scrub ok 2024-02-05T08:27:41.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:41 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2386310320' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T08:27:41.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:41 smithi134 ceph-mon[39278]: Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-02-05T08:27:41.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:41 smithi134 ceph-mon[39278]: 6.1e deep-scrub starts 2024-02-05T08:27:41.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:41 smithi134 ceph-mon[30086]: pgmap v1037: 142 pgs: 6 peering, 136 active+clean; 378 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.5 KiB/s rd, 36 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-02-05T08:27:41.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:41 smithi134 ceph-mon[30086]: 4.14 scrub starts 2024-02-05T08:27:41.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:41 smithi134 ceph-mon[30086]: 4.14 scrub ok 2024-02-05T08:27:41.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:41 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2386310320' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T08:27:41.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:41 smithi134 ceph-mon[30086]: Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-02-05T08:27:41.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:41 smithi134 ceph-mon[30086]: 6.1e deep-scrub starts 2024-02-05T08:27:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:41 smithi196 ceph-mon[33018]: pgmap v1037: 142 pgs: 6 peering, 136 active+clean; 378 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.5 KiB/s rd, 36 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-02-05T08:27:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:41 smithi196 ceph-mon[33018]: 4.14 scrub starts 2024-02-05T08:27:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:41 smithi196 ceph-mon[33018]: 4.14 scrub ok 2024-02-05T08:27:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:41 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2386310320' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T08:27:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:41 smithi196 ceph-mon[33018]: Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-02-05T08:27:41.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:41 smithi196 ceph-mon[33018]: 6.1e deep-scrub starts 2024-02-05T08:27:42.078 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 3 1 2024-02-05T08:27:42.324 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:42.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[30086]: 5.12 scrub starts 2024-02-05T08:27:42.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[30086]: 5.12 scrub ok 2024-02-05T08:27:42.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2386310320' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T08:27:42.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[30086]: osdmap e448: 8 total, 8 up, 8 in 2024-02-05T08:27:42.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[30086]: 6.1e deep-scrub ok 2024-02-05T08:27:42.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[30086]: 6.16 deep-scrub starts 2024-02-05T08:27:42.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[30086]: 6.1f scrub starts 2024-02-05T08:27:42.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[30086]: 6.16 deep-scrub ok 2024-02-05T08:27:42.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[30086]: 6.1f scrub ok 2024-02-05T08:27:42.446 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:27:42.447 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:27:42.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[39278]: 5.12 scrub starts 2024-02-05T08:27:42.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[39278]: 5.12 scrub ok 2024-02-05T08:27:42.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2386310320' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T08:27:42.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[39278]: osdmap e448: 8 total, 8 up, 8 in 2024-02-05T08:27:42.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[39278]: 6.1e deep-scrub ok 2024-02-05T08:27:42.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[39278]: 6.16 deep-scrub starts 2024-02-05T08:27:42.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[39278]: 6.1f scrub starts 2024-02-05T08:27:42.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[39278]: 6.16 deep-scrub ok 2024-02-05T08:27:42.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[39278]: 6.1f scrub ok 2024-02-05T08:27:42.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:27:42.447 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:42 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:27:42.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:42 smithi196 ceph-mon[33018]: 5.12 scrub starts 2024-02-05T08:27:42.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:42 smithi196 ceph-mon[33018]: 5.12 scrub ok 2024-02-05T08:27:42.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:42 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2386310320' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T08:27:42.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:42 smithi196 ceph-mon[33018]: osdmap e448: 8 total, 8 up, 8 in 2024-02-05T08:27:42.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:42 smithi196 ceph-mon[33018]: 6.1e deep-scrub ok 2024-02-05T08:27:42.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:42 smithi196 ceph-mon[33018]: 6.16 deep-scrub starts 2024-02-05T08:27:42.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:42 smithi196 ceph-mon[33018]: 6.1f scrub starts 2024-02-05T08:27:42.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:42 smithi196 ceph-mon[33018]: 6.16 deep-scrub ok 2024-02-05T08:27:42.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:42 smithi196 ceph-mon[33018]: 6.1f scrub ok 2024-02-05T08:27:42.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:42 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:27:42.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:42 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:27:43.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:43 smithi134 ceph-mon[39278]: pgmap v1039: 142 pgs: 6 peering, 136 active+clean; 378 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.5 KiB/s rd, 36 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-02-05T08:27:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:43 smithi134 ceph-mon[39278]: 5.d scrub starts 2024-02-05T08:27:43.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:43 smithi134 ceph-mon[39278]: 5.d scrub ok 2024-02-05T08:27:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:43 smithi134 ceph-mon[30086]: pgmap v1039: 142 pgs: 6 peering, 136 active+clean; 378 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.5 KiB/s rd, 36 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-02-05T08:27:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:43 smithi134 ceph-mon[30086]: 5.d scrub starts 2024-02-05T08:27:43.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:43 smithi134 ceph-mon[30086]: 5.d scrub ok 2024-02-05T08:27:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:43 smithi196 ceph-mon[33018]: pgmap v1039: 142 pgs: 6 peering, 136 active+clean; 378 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 5.5 KiB/s rd, 36 KiB/s wr, 6 op/s; 26 KiB/s, 0 objects/s recovering 2024-02-05T08:27:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:43 smithi196 ceph-mon[33018]: 5.d scrub starts 2024-02-05T08:27:43.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:43 smithi196 ceph-mon[33018]: 5.d scrub ok 2024-02-05T08:27:44.276 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:44 smithi134 ceph-mon[39278]: 3.b scrub starts 2024-02-05T08:27:44.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:44 smithi134 ceph-mon[39278]: 3.b scrub ok 2024-02-05T08:27:44.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:44 smithi134 ceph-mon[30086]: 3.b scrub starts 2024-02-05T08:27:44.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:44 smithi134 ceph-mon[30086]: 3.b scrub ok 2024-02-05T08:27:44.465 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:44 smithi196 ceph-mon[33018]: 3.b scrub starts 2024-02-05T08:27:44.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:44 smithi196 ceph-mon[33018]: 3.b scrub ok 2024-02-05T08:27:44.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:27:44 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:27:45.167 INFO:teuthology.orchestra.run.smithi134.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-02-05T08:27:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[39278]: pgmap v1040: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-02-05T08:27:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[39278]: 6.5 scrub starts 2024-02-05T08:27:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[39278]: 6.5 scrub ok 2024-02-05T08:27:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[39278]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-02-05T08:27:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[39278]: Cluster is now healthy 2024-02-05T08:27:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[39278]: 4.11 scrub starts 2024-02-05T08:27:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[39278]: 4.11 scrub ok 2024-02-05T08:27:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/177761868' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T08:27:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[39278]: 4.1f scrub starts 2024-02-05T08:27:45.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[39278]: 4.1f scrub ok 2024-02-05T08:27:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[30086]: pgmap v1040: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-02-05T08:27:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[30086]: 6.5 scrub starts 2024-02-05T08:27:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[30086]: 6.5 scrub ok 2024-02-05T08:27:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[30086]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-02-05T08:27:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[30086]: Cluster is now healthy 2024-02-05T08:27:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[30086]: 4.11 scrub starts 2024-02-05T08:27:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[30086]: 4.11 scrub ok 2024-02-05T08:27:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/177761868' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T08:27:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[30086]: 4.1f scrub starts 2024-02-05T08:27:45.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:45 smithi134 ceph-mon[30086]: 4.1f scrub ok 2024-02-05T08:27:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:45 smithi196 ceph-mon[33018]: pgmap v1040: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s; 2 B/s, 0 objects/s recovering 2024-02-05T08:27:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:45 smithi196 ceph-mon[33018]: 6.5 scrub starts 2024-02-05T08:27:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:45 smithi196 ceph-mon[33018]: 6.5 scrub ok 2024-02-05T08:27:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:45 smithi196 ceph-mon[33018]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 6 pgs peering) 2024-02-05T08:27:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:45 smithi196 ceph-mon[33018]: Cluster is now healthy 2024-02-05T08:27:45.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:45 smithi196 ceph-mon[33018]: 4.11 scrub starts 2024-02-05T08:27:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:45 smithi196 ceph-mon[33018]: 4.11 scrub ok 2024-02-05T08:27:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:45 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:45 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/177761868' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T08:27:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:45 smithi196 ceph-mon[33018]: 4.1f scrub starts 2024-02-05T08:27:45.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:45 smithi196 ceph-mon[33018]: 4.1f scrub ok 2024-02-05T08:27:45.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 4 1 2024-02-05T08:27:46.178 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[30086]: 1.2 scrub starts 2024-02-05T08:27:46.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[30086]: 1.2 scrub ok 2024-02-05T08:27:46.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/177761868' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-05T08:27:46.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[30086]: osdmap e449: 8 total, 8 up, 8 in 2024-02-05T08:27:46.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[30086]: 4.19 scrub starts 2024-02-05T08:27:46.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[30086]: 4.19 scrub ok 2024-02-05T08:27:46.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:27:46.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[30086]: 5.1 deep-scrub starts 2024-02-05T08:27:46.179 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[30086]: 5.1 deep-scrub ok 2024-02-05T08:27:46.179 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[39278]: 1.2 scrub starts 2024-02-05T08:27:46.180 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[39278]: 1.2 scrub ok 2024-02-05T08:27:46.180 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/177761868' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-05T08:27:46.180 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[39278]: osdmap e449: 8 total, 8 up, 8 in 2024-02-05T08:27:46.180 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[39278]: 4.19 scrub starts 2024-02-05T08:27:46.180 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[39278]: 4.19 scrub ok 2024-02-05T08:27:46.180 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:27:46.180 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[39278]: 5.1 deep-scrub starts 2024-02-05T08:27:46.180 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:46 smithi134 ceph-mon[39278]: 5.1 deep-scrub ok 2024-02-05T08:27:46.207 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:46 smithi196 ceph-mon[33018]: 1.2 scrub starts 2024-02-05T08:27:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:46 smithi196 ceph-mon[33018]: 1.2 scrub ok 2024-02-05T08:27:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:46 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/177761868' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-05T08:27:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:46 smithi196 ceph-mon[33018]: osdmap e449: 8 total, 8 up, 8 in 2024-02-05T08:27:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:46 smithi196 ceph-mon[33018]: 4.19 scrub starts 2024-02-05T08:27:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:46 smithi196 ceph-mon[33018]: 4.19 scrub ok 2024-02-05T08:27:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:46 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:27:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:46 smithi196 ceph-mon[33018]: 5.1 deep-scrub starts 2024-02-05T08:27:46.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:46 smithi196 ceph-mon[33018]: 5.1 deep-scrub ok 2024-02-05T08:27:47.477 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[30086]: pgmap v1042: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:47.477 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[30086]: 3.f scrub starts 2024-02-05T08:27:47.478 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[30086]: 3.f scrub ok 2024-02-05T08:27:47.478 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[30086]: 5.13 scrub starts 2024-02-05T08:27:47.478 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[30086]: 5.13 scrub ok 2024-02-05T08:27:47.478 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:27:47.478 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:27:47.478 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:27:47.478 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[39278]: pgmap v1042: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:47.478 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[39278]: 3.f scrub starts 2024-02-05T08:27:47.478 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[39278]: 3.f scrub ok 2024-02-05T08:27:47.478 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[39278]: 5.13 scrub starts 2024-02-05T08:27:47.478 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[39278]: 5.13 scrub ok 2024-02-05T08:27:47.479 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:27:47.479 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:27:47.479 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:27:47.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:47 smithi196 ceph-mon[33018]: pgmap v1042: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:47.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:47 smithi196 ceph-mon[33018]: 3.f scrub starts 2024-02-05T08:27:47.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:47 smithi196 ceph-mon[33018]: 3.f scrub ok 2024-02-05T08:27:47.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:47 smithi196 ceph-mon[33018]: 5.13 scrub starts 2024-02-05T08:27:47.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:47 smithi196 ceph-mon[33018]: 5.13 scrub ok 2024-02-05T08:27:47.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:47 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:27:47.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:47 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:27:47.567 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:47 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:27:48.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:27:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:27:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:27:48.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:48 smithi134 ceph-mon[30086]: 5.8 scrub starts 2024-02-05T08:27:48.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:48 smithi134 ceph-mon[30086]: 5.8 scrub ok 2024-02-05T08:27:48.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:48 smithi134 ceph-mon[39278]: 5.8 scrub starts 2024-02-05T08:27:48.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:48 smithi134 ceph-mon[39278]: 5.8 scrub ok 2024-02-05T08:27:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:48 smithi196 ceph-mon[33018]: 5.8 scrub starts 2024-02-05T08:27:48.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:48 smithi196 ceph-mon[33018]: 5.8 scrub ok 2024-02-05T08:27:49.198 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.4 to 1 (10000) 2024-02-05T08:27:49.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:49 smithi134 ceph-mon[30086]: pgmap v1043: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:49.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:49 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/723014623' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:27:49.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:49 smithi134 ceph-mon[30086]: 5.14 scrub starts 2024-02-05T08:27:49.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:49 smithi134 ceph-mon[30086]: 5.14 scrub ok 2024-02-05T08:27:49.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:49 smithi134 ceph-mon[39278]: pgmap v1043: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:49.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:49 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/723014623' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:27:49.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:49 smithi134 ceph-mon[39278]: 5.14 scrub starts 2024-02-05T08:27:49.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:49 smithi134 ceph-mon[39278]: 5.14 scrub ok 2024-02-05T08:27:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:49 smithi196 ceph-mon[33018]: pgmap v1043: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:49 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/723014623' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:27:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:49 smithi196 ceph-mon[33018]: 5.14 scrub starts 2024-02-05T08:27:49.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:49 smithi196 ceph-mon[33018]: 5.14 scrub ok 2024-02-05T08:27:49.929 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 4 1 2024-02-05T08:27:50.188 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:50.454 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:50 smithi134 ceph-mon[30086]: 5.7 scrub starts 2024-02-05T08:27:50.454 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:50 smithi134 ceph-mon[30086]: 5.7 scrub ok 2024-02-05T08:27:50.454 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:50 smithi134 ceph-mon[30086]: 4.15 scrub starts 2024-02-05T08:27:50.454 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:50 smithi134 ceph-mon[30086]: 4.15 scrub ok 2024-02-05T08:27:50.454 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:50 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/723014623' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:27:50.454 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:50 smithi134 ceph-mon[30086]: osdmap e450: 8 total, 8 up, 8 in 2024-02-05T08:27:50.454 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:50 smithi134 ceph-mon[39278]: 5.7 scrub starts 2024-02-05T08:27:50.454 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:50 smithi134 ceph-mon[39278]: 5.7 scrub ok 2024-02-05T08:27:50.454 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:50 smithi134 ceph-mon[39278]: 4.15 scrub starts 2024-02-05T08:27:50.454 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:50 smithi134 ceph-mon[39278]: 4.15 scrub ok 2024-02-05T08:27:50.454 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:50 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/723014623' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:27:50.455 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:50 smithi134 ceph-mon[39278]: osdmap e450: 8 total, 8 up, 8 in 2024-02-05T08:27:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:50 smithi196 ceph-mon[33018]: 5.7 scrub starts 2024-02-05T08:27:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:50 smithi196 ceph-mon[33018]: 5.7 scrub ok 2024-02-05T08:27:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:50 smithi196 ceph-mon[33018]: 4.15 scrub starts 2024-02-05T08:27:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:50 smithi196 ceph-mon[33018]: 4.15 scrub ok 2024-02-05T08:27:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:50 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/723014623' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:27:50.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:50 smithi196 ceph-mon[33018]: osdmap e450: 8 total, 8 up, 8 in 2024-02-05T08:27:51.470 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[30086]: pgmap v1045: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:51.470 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[30086]: 5.19 scrub starts 2024-02-05T08:27:51.470 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[30086]: 5.19 scrub ok 2024-02-05T08:27:51.470 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[30086]: 6.4 scrub starts 2024-02-05T08:27:51.470 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[30086]: 6.4 scrub ok 2024-02-05T08:27:51.471 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[30086]: 3.1c deep-scrub starts 2024-02-05T08:27:51.471 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[30086]: 3.1c deep-scrub ok 2024-02-05T08:27:51.471 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[39278]: pgmap v1045: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:51.471 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[39278]: 5.19 scrub starts 2024-02-05T08:27:51.471 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[39278]: 5.19 scrub ok 2024-02-05T08:27:51.471 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[39278]: 6.4 scrub starts 2024-02-05T08:27:51.471 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[39278]: 6.4 scrub ok 2024-02-05T08:27:51.471 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[39278]: 3.1c deep-scrub starts 2024-02-05T08:27:51.471 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:51 smithi134 ceph-mon[39278]: 3.1c deep-scrub ok 2024-02-05T08:27:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:51 smithi196 ceph-mon[33018]: pgmap v1045: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail 2024-02-05T08:27:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:51 smithi196 ceph-mon[33018]: 5.19 scrub starts 2024-02-05T08:27:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:51 smithi196 ceph-mon[33018]: 5.19 scrub ok 2024-02-05T08:27:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:51 smithi196 ceph-mon[33018]: 6.4 scrub starts 2024-02-05T08:27:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:51 smithi196 ceph-mon[33018]: 6.4 scrub ok 2024-02-05T08:27:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:51 smithi196 ceph-mon[33018]: 3.1c deep-scrub starts 2024-02-05T08:27:51.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:51 smithi196 ceph-mon[33018]: 3.1c deep-scrub ok 2024-02-05T08:27:52.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:52 smithi134 ceph-mon[30086]: 3.5 scrub starts 2024-02-05T08:27:52.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:52 smithi134 ceph-mon[30086]: 3.5 scrub ok 2024-02-05T08:27:52.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:52 smithi134 ceph-mon[30086]: 6.1c scrub starts 2024-02-05T08:27:52.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:52 smithi134 ceph-mon[30086]: 6.1c scrub ok 2024-02-05T08:27:52.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:52 smithi134 ceph-mon[30086]: 6.0 scrub starts 2024-02-05T08:27:52.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:52 smithi134 ceph-mon[30086]: 6.0 scrub ok 2024-02-05T08:27:52.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:52 smithi134 ceph-mon[39278]: 3.5 scrub starts 2024-02-05T08:27:52.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:52 smithi134 ceph-mon[39278]: 3.5 scrub ok 2024-02-05T08:27:52.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:52 smithi134 ceph-mon[39278]: 6.1c scrub starts 2024-02-05T08:27:52.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:52 smithi134 ceph-mon[39278]: 6.1c scrub ok 2024-02-05T08:27:52.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:52 smithi134 ceph-mon[39278]: 6.0 scrub starts 2024-02-05T08:27:52.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:52 smithi134 ceph-mon[39278]: 6.0 scrub ok 2024-02-05T08:27:52.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:52 smithi196 ceph-mon[33018]: 3.5 scrub starts 2024-02-05T08:27:52.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:52 smithi196 ceph-mon[33018]: 3.5 scrub ok 2024-02-05T08:27:52.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:52 smithi196 ceph-mon[33018]: 6.1c scrub starts 2024-02-05T08:27:52.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:52 smithi196 ceph-mon[33018]: 6.1c scrub ok 2024-02-05T08:27:52.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:52 smithi196 ceph-mon[33018]: 6.0 scrub starts 2024-02-05T08:27:52.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:52 smithi196 ceph-mon[33018]: 6.0 scrub ok 2024-02-05T08:27:53.217 INFO:teuthology.orchestra.run.smithi134.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-02-05T08:27:53.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:53 smithi134 ceph-mon[30086]: pgmap v1046: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:27:53.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:53 smithi134 ceph-mon[30086]: 3.e scrub starts 2024-02-05T08:27:53.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:53 smithi134 ceph-mon[30086]: 3.e scrub ok 2024-02-05T08:27:53.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:53 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2945633166' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:27:53.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:53 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:27:53.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:53 smithi134 ceph-mon[39278]: pgmap v1046: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:27:53.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:53 smithi134 ceph-mon[39278]: 3.e scrub starts 2024-02-05T08:27:53.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:53 smithi134 ceph-mon[39278]: 3.e scrub ok 2024-02-05T08:27:53.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:53 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2945633166' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:27:53.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:53 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:27:53.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:53 smithi196 ceph-mon[33018]: pgmap v1046: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:27:53.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:53 smithi196 ceph-mon[33018]: 3.e scrub starts 2024-02-05T08:27:53.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:53 smithi196 ceph-mon[33018]: 3.e scrub ok 2024-02-05T08:27:53.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:53 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2945633166' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:27:53.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:53 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T08:27:53.888 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 5 1 2024-02-05T08:27:54.130 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:54.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:54 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:27:54.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:54 smithi134 ceph-mon[30086]: osdmap e451: 8 total, 8 up, 8 in 2024-02-05T08:27:54.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:54 smithi134 ceph-mon[30086]: 6.9 scrub starts 2024-02-05T08:27:54.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:54 smithi134 ceph-mon[30086]: 6.9 scrub ok 2024-02-05T08:27:54.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:27:54.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:54 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:27:54.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:54 smithi134 ceph-mon[39278]: osdmap e451: 8 total, 8 up, 8 in 2024-02-05T08:27:54.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:54 smithi134 ceph-mon[39278]: 6.9 scrub starts 2024-02-05T08:27:54.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:54 smithi134 ceph-mon[39278]: 6.9 scrub ok 2024-02-05T08:27:54.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:27:54.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:27:54 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:27:54.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:54 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-05T08:27:54.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:54 smithi196 ceph-mon[33018]: osdmap e451: 8 total, 8 up, 8 in 2024-02-05T08:27:54.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:54 smithi196 ceph-mon[33018]: 6.9 scrub starts 2024-02-05T08:27:54.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:54 smithi196 ceph-mon[33018]: 6.9 scrub ok 2024-02-05T08:27:54.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:27:55.461 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:55 smithi134 ceph-mon[30086]: pgmap v1048: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:27:55.461 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:55 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:55.462 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:55 smithi134 ceph-mon[30086]: 3.19 scrub starts 2024-02-05T08:27:55.462 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:55 smithi134 ceph-mon[30086]: 3.19 scrub ok 2024-02-05T08:27:55.462 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:55 smithi134 ceph-mon[30086]: 2.1 scrub starts 2024-02-05T08:27:55.462 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:55 smithi134 ceph-mon[30086]: 2.1 scrub ok 2024-02-05T08:27:55.462 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:55 smithi134 ceph-mon[39278]: pgmap v1048: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:27:55.462 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:55 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:55.462 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:55 smithi134 ceph-mon[39278]: 3.19 scrub starts 2024-02-05T08:27:55.462 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:55 smithi134 ceph-mon[39278]: 3.19 scrub ok 2024-02-05T08:27:55.462 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:55 smithi134 ceph-mon[39278]: 2.1 scrub starts 2024-02-05T08:27:55.462 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:55 smithi134 ceph-mon[39278]: 2.1 scrub ok 2024-02-05T08:27:55.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:55 smithi196 ceph-mon[33018]: pgmap v1048: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:27:55.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:55 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:27:55.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:55 smithi196 ceph-mon[33018]: 3.19 scrub starts 2024-02-05T08:27:55.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:55 smithi196 ceph-mon[33018]: 3.19 scrub ok 2024-02-05T08:27:55.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:55 smithi196 ceph-mon[33018]: 2.1 scrub starts 2024-02-05T08:27:55.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:55 smithi196 ceph-mon[33018]: 2.1 scrub ok 2024-02-05T08:27:56.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:56 smithi134 ceph-mon[30086]: 4.8 deep-scrub starts 2024-02-05T08:27:56.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:56 smithi134 ceph-mon[30086]: 4.8 deep-scrub ok 2024-02-05T08:27:56.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:56 smithi134 ceph-mon[30086]: 6.14 scrub starts 2024-02-05T08:27:56.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:56 smithi134 ceph-mon[30086]: 6.14 scrub ok 2024-02-05T08:27:56.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:56 smithi134 ceph-mon[30086]: 5.c scrub starts 2024-02-05T08:27:56.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:56 smithi134 ceph-mon[30086]: 5.c scrub ok 2024-02-05T08:27:56.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:56 smithi134 ceph-mon[39278]: 4.8 deep-scrub starts 2024-02-05T08:27:56.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:56 smithi134 ceph-mon[39278]: 4.8 deep-scrub ok 2024-02-05T08:27:56.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:56 smithi134 ceph-mon[39278]: 6.14 scrub starts 2024-02-05T08:27:56.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:56 smithi134 ceph-mon[39278]: 6.14 scrub ok 2024-02-05T08:27:56.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:56 smithi134 ceph-mon[39278]: 5.c scrub starts 2024-02-05T08:27:56.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:56 smithi134 ceph-mon[39278]: 5.c scrub ok 2024-02-05T08:27:56.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:56 smithi196 ceph-mon[33018]: 4.8 deep-scrub starts 2024-02-05T08:27:56.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:56 smithi196 ceph-mon[33018]: 4.8 deep-scrub ok 2024-02-05T08:27:56.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:56 smithi196 ceph-mon[33018]: 6.14 scrub starts 2024-02-05T08:27:56.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:56 smithi196 ceph-mon[33018]: 6.14 scrub ok 2024-02-05T08:27:56.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:56 smithi196 ceph-mon[33018]: 5.c scrub starts 2024-02-05T08:27:56.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:56 smithi196 ceph-mon[33018]: 5.c scrub ok 2024-02-05T08:27:57.231 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.5 to 1 (10000) 2024-02-05T08:27:57.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:57 smithi134 ceph-mon[30086]: pgmap v1049: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:27:57.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:57 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/921267820' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:27:57.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:57 smithi134 ceph-mon[30086]: 4.3 deep-scrub starts 2024-02-05T08:27:57.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:57 smithi134 ceph-mon[30086]: 4.3 deep-scrub ok 2024-02-05T08:27:57.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:57 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:27:57.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:57 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:27:57.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:57 smithi134 ceph-mon[39278]: pgmap v1049: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:27:57.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:57 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/921267820' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:27:57.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:57 smithi134 ceph-mon[39278]: 4.3 deep-scrub starts 2024-02-05T08:27:57.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:57 smithi134 ceph-mon[39278]: 4.3 deep-scrub ok 2024-02-05T08:27:57.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:57 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:27:57.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:57 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:27:57.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:57 smithi196 ceph-mon[33018]: pgmap v1049: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:27:57.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:57 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/921267820' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:27:57.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:57 smithi196 ceph-mon[33018]: 4.3 deep-scrub starts 2024-02-05T08:27:57.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:57 smithi196 ceph-mon[33018]: 4.3 deep-scrub ok 2024-02-05T08:27:57.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:57 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:27:57.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:57 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:27:58.012 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:27:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:27:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:27:58.048 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 5 1 2024-02-05T08:27:58.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:58 smithi134 ceph-mon[30086]: 3.7 scrub starts 2024-02-05T08:27:58.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:58 smithi134 ceph-mon[30086]: 3.7 scrub ok 2024-02-05T08:27:58.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:58 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/921267820' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:27:58.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:58 smithi134 ceph-mon[30086]: osdmap e452: 8 total, 8 up, 8 in 2024-02-05T08:27:58.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:58 smithi134 ceph-mon[39278]: 3.7 scrub starts 2024-02-05T08:27:58.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:58 smithi134 ceph-mon[39278]: 3.7 scrub ok 2024-02-05T08:27:58.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:58 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/921267820' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:27:58.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:58 smithi134 ceph-mon[39278]: osdmap e452: 8 total, 8 up, 8 in 2024-02-05T08:27:58.296 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:27:58.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:58 smithi196 ceph-mon[33018]: 3.7 scrub starts 2024-02-05T08:27:58.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:58 smithi196 ceph-mon[33018]: 3.7 scrub ok 2024-02-05T08:27:58.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:58 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/921267820' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:27:58.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:58 smithi196 ceph-mon[33018]: osdmap e452: 8 total, 8 up, 8 in 2024-02-05T08:27:59.496 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:59 smithi134 ceph-mon[30086]: 4.12 scrub starts 2024-02-05T08:27:59.496 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:59 smithi134 ceph-mon[30086]: 4.12 scrub ok 2024-02-05T08:27:59.496 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:59 smithi134 ceph-mon[30086]: pgmap v1051: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:27:59.496 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:59 smithi134 ceph-mon[30086]: 3.c scrub starts 2024-02-05T08:27:59.496 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:27:59 smithi134 ceph-mon[30086]: 3.c scrub ok 2024-02-05T08:27:59.497 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:59 smithi134 ceph-mon[39278]: 4.12 scrub starts 2024-02-05T08:27:59.497 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:59 smithi134 ceph-mon[39278]: 4.12 scrub ok 2024-02-05T08:27:59.497 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:59 smithi134 ceph-mon[39278]: pgmap v1051: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:27:59.497 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:59 smithi134 ceph-mon[39278]: 3.c scrub starts 2024-02-05T08:27:59.497 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:27:59 smithi134 ceph-mon[39278]: 3.c scrub ok 2024-02-05T08:27:59.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:59 smithi196 ceph-mon[33018]: 4.12 scrub starts 2024-02-05T08:27:59.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:59 smithi196 ceph-mon[33018]: 4.12 scrub ok 2024-02-05T08:27:59.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:59 smithi196 ceph-mon[33018]: pgmap v1051: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:27:59.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:59 smithi196 ceph-mon[33018]: 3.c scrub starts 2024-02-05T08:27:59.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:27:59 smithi196 ceph-mon[33018]: 3.c scrub ok 2024-02-05T08:28:00.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:00 smithi134 ceph-mon[39278]: 5.1e scrub starts 2024-02-05T08:28:00.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:00 smithi134 ceph-mon[39278]: 5.1e scrub ok 2024-02-05T08:28:00.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:00 smithi134 ceph-mon[39278]: 4.1a scrub starts 2024-02-05T08:28:00.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:00 smithi134 ceph-mon[39278]: 4.1a scrub ok 2024-02-05T08:28:00.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:00 smithi134 ceph-mon[30086]: 5.1e scrub starts 2024-02-05T08:28:00.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:00 smithi134 ceph-mon[30086]: 5.1e scrub ok 2024-02-05T08:28:00.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:00 smithi134 ceph-mon[30086]: 4.1a scrub starts 2024-02-05T08:28:00.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:00 smithi134 ceph-mon[30086]: 4.1a scrub ok 2024-02-05T08:28:00.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:00 smithi196 ceph-mon[33018]: 5.1e scrub starts 2024-02-05T08:28:00.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:00 smithi196 ceph-mon[33018]: 5.1e scrub ok 2024-02-05T08:28:00.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:00 smithi196 ceph-mon[33018]: 4.1a scrub starts 2024-02-05T08:28:00.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:00 smithi196 ceph-mon[33018]: 4.1a scrub ok 2024-02-05T08:28:01.250 INFO:teuthology.orchestra.run.smithi134.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-02-05T08:28:01.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[30086]: pgmap v1052: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:01.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[30086]: 3.13 scrub starts 2024-02-05T08:28:01.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[30086]: 3.13 scrub ok 2024-02-05T08:28:01.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3687048331' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:28:01.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:28:01.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[30086]: 6.d scrub starts 2024-02-05T08:28:01.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[30086]: 6.d scrub ok 2024-02-05T08:28:01.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[30086]: 3.14 scrub starts 2024-02-05T08:28:01.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[30086]: 3.14 scrub ok 2024-02-05T08:28:01.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[39278]: pgmap v1052: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:01.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[39278]: 3.13 scrub starts 2024-02-05T08:28:01.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[39278]: 3.13 scrub ok 2024-02-05T08:28:01.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3687048331' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:28:01.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:28:01.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[39278]: 6.d scrub starts 2024-02-05T08:28:01.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[39278]: 6.d scrub ok 2024-02-05T08:28:01.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[39278]: 3.14 scrub starts 2024-02-05T08:28:01.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:01 smithi134 ceph-mon[39278]: 3.14 scrub ok 2024-02-05T08:28:01.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:01 smithi196 ceph-mon[33018]: pgmap v1052: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:01.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:01 smithi196 ceph-mon[33018]: 3.13 scrub starts 2024-02-05T08:28:01.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:01 smithi196 ceph-mon[33018]: 3.13 scrub ok 2024-02-05T08:28:01.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:01 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3687048331' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:28:01.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:01 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T08:28:01.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:01 smithi196 ceph-mon[33018]: 6.d scrub starts 2024-02-05T08:28:01.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:01 smithi196 ceph-mon[33018]: 6.d scrub ok 2024-02-05T08:28:01.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:01 smithi196 ceph-mon[33018]: 3.14 scrub starts 2024-02-05T08:28:01.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:01 smithi196 ceph-mon[33018]: 3.14 scrub ok 2024-02-05T08:28:02.067 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 6 1 2024-02-05T08:28:02.313 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:02.435 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[39278]: 5.1b scrub starts 2024-02-05T08:28:02.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[39278]: 5.1b scrub ok 2024-02-05T08:28:02.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:28:02.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[39278]: osdmap e453: 8 total, 8 up, 8 in 2024-02-05T08:28:02.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[39278]: 6.1 scrub starts 2024-02-05T08:28:02.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[39278]: 6.1 scrub ok 2024-02-05T08:28:02.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[39278]: 6.18 scrub starts 2024-02-05T08:28:02.436 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[39278]: 6.18 scrub ok 2024-02-05T08:28:02.436 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[30086]: 5.1b scrub starts 2024-02-05T08:28:02.436 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[30086]: 5.1b scrub ok 2024-02-05T08:28:02.436 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:28:02.436 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[30086]: osdmap e453: 8 total, 8 up, 8 in 2024-02-05T08:28:02.437 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[30086]: 6.1 scrub starts 2024-02-05T08:28:02.437 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[30086]: 6.1 scrub ok 2024-02-05T08:28:02.437 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[30086]: 6.18 scrub starts 2024-02-05T08:28:02.437 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:02 smithi134 ceph-mon[30086]: 6.18 scrub ok 2024-02-05T08:28:02.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:02 smithi196 ceph-mon[33018]: 5.1b scrub starts 2024-02-05T08:28:02.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:02 smithi196 ceph-mon[33018]: 5.1b scrub ok 2024-02-05T08:28:02.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:02 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-05T08:28:02.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:02 smithi196 ceph-mon[33018]: osdmap e453: 8 total, 8 up, 8 in 2024-02-05T08:28:02.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:02 smithi196 ceph-mon[33018]: 6.1 scrub starts 2024-02-05T08:28:02.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:02 smithi196 ceph-mon[33018]: 6.1 scrub ok 2024-02-05T08:28:02.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:02 smithi196 ceph-mon[33018]: 6.18 scrub starts 2024-02-05T08:28:02.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:02 smithi196 ceph-mon[33018]: 6.18 scrub ok 2024-02-05T08:28:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:03 smithi134 ceph-mon[30086]: 5.15 scrub starts 2024-02-05T08:28:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:03 smithi134 ceph-mon[30086]: 5.15 scrub ok 2024-02-05T08:28:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:03 smithi134 ceph-mon[30086]: pgmap v1054: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:28:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:03 smithi134 ceph-mon[30086]: osdmap e454: 8 total, 8 up, 8 in 2024-02-05T08:28:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:03 smithi134 ceph-mon[30086]: 5.2 scrub starts 2024-02-05T08:28:03.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:03 smithi134 ceph-mon[30086]: 5.2 scrub ok 2024-02-05T08:28:03.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:03 smithi134 ceph-mon[39278]: 5.15 scrub starts 2024-02-05T08:28:03.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:03 smithi134 ceph-mon[39278]: 5.15 scrub ok 2024-02-05T08:28:03.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:03 smithi134 ceph-mon[39278]: pgmap v1054: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:28:03.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:03 smithi134 ceph-mon[39278]: osdmap e454: 8 total, 8 up, 8 in 2024-02-05T08:28:03.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:03 smithi134 ceph-mon[39278]: 5.2 scrub starts 2024-02-05T08:28:03.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:03 smithi134 ceph-mon[39278]: 5.2 scrub ok 2024-02-05T08:28:03.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:03 smithi196 ceph-mon[33018]: 5.15 scrub starts 2024-02-05T08:28:03.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:03 smithi196 ceph-mon[33018]: 5.15 scrub ok 2024-02-05T08:28:03.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:03 smithi196 ceph-mon[33018]: pgmap v1054: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:28:03.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:03 smithi196 ceph-mon[33018]: osdmap e454: 8 total, 8 up, 8 in 2024-02-05T08:28:03.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:03 smithi196 ceph-mon[33018]: 5.2 scrub starts 2024-02-05T08:28:03.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:03 smithi196 ceph-mon[33018]: 5.2 scrub ok 2024-02-05T08:28:04.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[30086]: 1.1 scrub starts 2024-02-05T08:28:04.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[30086]: 1.1 scrub ok 2024-02-05T08:28:04.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[30086]: 1.0 scrub starts 2024-02-05T08:28:04.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[30086]: 1.0 scrub ok 2024-02-05T08:28:04.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[30086]: 4.0 scrub starts 2024-02-05T08:28:04.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[30086]: 4.0 scrub ok 2024-02-05T08:28:04.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:28:04.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:28:04.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[39278]: 1.1 scrub starts 2024-02-05T08:28:04.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[39278]: 1.1 scrub ok 2024-02-05T08:28:04.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[39278]: 1.0 scrub starts 2024-02-05T08:28:04.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[39278]: 1.0 scrub ok 2024-02-05T08:28:04.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[39278]: 4.0 scrub starts 2024-02-05T08:28:04.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[39278]: 4.0 scrub ok 2024-02-05T08:28:04.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:28:04.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:04 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:28:04.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:28:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:28:04.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:04 smithi196 ceph-mon[33018]: 1.1 scrub starts 2024-02-05T08:28:04.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:04 smithi196 ceph-mon[33018]: 1.1 scrub ok 2024-02-05T08:28:04.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:04 smithi196 ceph-mon[33018]: 1.0 scrub starts 2024-02-05T08:28:04.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:04 smithi196 ceph-mon[33018]: 1.0 scrub ok 2024-02-05T08:28:04.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:04 smithi196 ceph-mon[33018]: 4.0 scrub starts 2024-02-05T08:28:04.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:04 smithi196 ceph-mon[33018]: 4.0 scrub ok 2024-02-05T08:28:04.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:04 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:28:04.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:04 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:28:05.266 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.6 to 1 (10000) 2024-02-05T08:28:05.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[30086]: pgmap v1056: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 5 B/s, 0 objects/s recovering 2024-02-05T08:28:05.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[30086]: 6.3 scrub starts 2024-02-05T08:28:05.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[30086]: osdmap e455: 8 total, 8 up, 8 in 2024-02-05T08:28:05.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:05.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:28:05.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:28:05.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[30086]: 6.3 scrub ok 2024-02-05T08:28:05.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2267407207' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:28:05.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[30086]: 4.4 deep-scrub starts 2024-02-05T08:28:05.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[30086]: 4.4 deep-scrub ok 2024-02-05T08:28:05.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[39278]: pgmap v1056: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 5 B/s, 0 objects/s recovering 2024-02-05T08:28:05.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[39278]: 6.3 scrub starts 2024-02-05T08:28:05.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[39278]: osdmap e455: 8 total, 8 up, 8 in 2024-02-05T08:28:05.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:05.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:28:05.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:28:05.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[39278]: 6.3 scrub ok 2024-02-05T08:28:05.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2267407207' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:28:05.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[39278]: 4.4 deep-scrub starts 2024-02-05T08:28:05.529 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:05 smithi134 ceph-mon[39278]: 4.4 deep-scrub ok 2024-02-05T08:28:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:05 smithi196 ceph-mon[33018]: pgmap v1056: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 5 B/s, 0 objects/s recovering 2024-02-05T08:28:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:05 smithi196 ceph-mon[33018]: 6.3 scrub starts 2024-02-05T08:28:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:05 smithi196 ceph-mon[33018]: osdmap e455: 8 total, 8 up, 8 in 2024-02-05T08:28:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:05 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:05 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:28:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:05 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:28:05.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:05 smithi196 ceph-mon[33018]: 6.3 scrub ok 2024-02-05T08:28:05.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:05 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2267407207' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:28:05.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:05 smithi196 ceph-mon[33018]: 4.4 deep-scrub starts 2024-02-05T08:28:05.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:05 smithi196 ceph-mon[33018]: 4.4 deep-scrub ok 2024-02-05T08:28:06.012 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 6 1 2024-02-05T08:28:06.262 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:06.525 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:06 smithi134 ceph-mon[30086]: 6.8 scrub starts 2024-02-05T08:28:06.525 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:06 smithi134 ceph-mon[30086]: 6.8 scrub ok 2024-02-05T08:28:06.525 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:06 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2267407207' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T08:28:06.525 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:06 smithi134 ceph-mon[30086]: osdmap e456: 8 total, 8 up, 8 in 2024-02-05T08:28:06.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:06 smithi134 ceph-mon[39278]: 6.8 scrub starts 2024-02-05T08:28:06.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:06 smithi134 ceph-mon[39278]: 6.8 scrub ok 2024-02-05T08:28:06.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:06 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2267407207' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T08:28:06.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:06 smithi134 ceph-mon[39278]: osdmap e456: 8 total, 8 up, 8 in 2024-02-05T08:28:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:06 smithi196 ceph-mon[33018]: 6.8 scrub starts 2024-02-05T08:28:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:06 smithi196 ceph-mon[33018]: 6.8 scrub ok 2024-02-05T08:28:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:06 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2267407207' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T08:28:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:06 smithi196 ceph-mon[33018]: osdmap e456: 8 total, 8 up, 8 in 2024-02-05T08:28:07.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[39278]: pgmap v1059: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 7 B/s, 0 objects/s recovering 2024-02-05T08:28:07.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[39278]: 1.7 scrub starts 2024-02-05T08:28:07.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[39278]: 1.7 scrub ok 2024-02-05T08:28:07.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[39278]: 3.6 scrub starts 2024-02-05T08:28:07.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[39278]: 3.6 scrub ok 2024-02-05T08:28:07.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[39278]: 4.1c scrub starts 2024-02-05T08:28:07.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[39278]: 4.1c scrub ok 2024-02-05T08:28:07.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[30086]: pgmap v1059: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 7 B/s, 0 objects/s recovering 2024-02-05T08:28:07.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[30086]: 1.7 scrub starts 2024-02-05T08:28:07.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[30086]: 1.7 scrub ok 2024-02-05T08:28:07.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[30086]: 3.6 scrub starts 2024-02-05T08:28:07.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[30086]: 3.6 scrub ok 2024-02-05T08:28:07.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[30086]: 4.1c scrub starts 2024-02-05T08:28:07.528 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-mon[30086]: 4.1c scrub ok 2024-02-05T08:28:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:07 smithi196 ceph-mon[33018]: pgmap v1059: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 7 B/s, 0 objects/s recovering 2024-02-05T08:28:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:07 smithi196 ceph-mon[33018]: 1.7 scrub starts 2024-02-05T08:28:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:07 smithi196 ceph-mon[33018]: 1.7 scrub ok 2024-02-05T08:28:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:07 smithi196 ceph-mon[33018]: 3.6 scrub starts 2024-02-05T08:28:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:07 smithi196 ceph-mon[33018]: 3.6 scrub ok 2024-02-05T08:28:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:07 smithi196 ceph-mon[33018]: 4.1c scrub starts 2024-02-05T08:28:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:07 smithi196 ceph-mon[33018]: 4.1c scrub ok 2024-02-05T08:28:08.014 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:28:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:28:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:28:08.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:08 smithi134 ceph-mon[30086]: 1.9 deep-scrub starts 2024-02-05T08:28:08.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:08 smithi134 ceph-mon[30086]: 1.9 deep-scrub ok 2024-02-05T08:28:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:08 smithi134 ceph-mon[39278]: 1.9 deep-scrub starts 2024-02-05T08:28:08.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:08 smithi134 ceph-mon[39278]: 1.9 deep-scrub ok 2024-02-05T08:28:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:08 smithi196 ceph-mon[33018]: 1.9 deep-scrub starts 2024-02-05T08:28:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:08 smithi196 ceph-mon[33018]: 1.9 deep-scrub ok 2024-02-05T08:28:09.297 INFO:teuthology.orchestra.run.smithi134.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-02-05T08:28:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:09 smithi196 ceph-mon[33018]: pgmap v1060: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 6 B/s, 0 objects/s recovering 2024-02-05T08:28:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:09 smithi196 ceph-mon[33018]: 5.9 scrub starts 2024-02-05T08:28:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:09 smithi196 ceph-mon[33018]: 5.9 scrub ok 2024-02-05T08:28:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:09 smithi196 ceph-mon[33018]: 3.18 scrub starts 2024-02-05T08:28:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:09 smithi196 ceph-mon[33018]: 3.18 scrub ok 2024-02-05T08:28:09.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:09 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1887878734' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:28:09.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:09 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:28:09.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:09 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:28:09.610 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[30086]: pgmap v1060: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 6 B/s, 0 objects/s recovering 2024-02-05T08:28:09.610 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[30086]: 5.9 scrub starts 2024-02-05T08:28:09.610 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[30086]: 5.9 scrub ok 2024-02-05T08:28:09.610 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[30086]: 3.18 scrub starts 2024-02-05T08:28:09.610 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[30086]: 3.18 scrub ok 2024-02-05T08:28:09.610 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1887878734' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:28:09.611 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:28:09.611 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:28:09.611 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[39278]: pgmap v1060: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 6 B/s, 0 objects/s recovering 2024-02-05T08:28:09.611 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[39278]: 5.9 scrub starts 2024-02-05T08:28:09.611 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[39278]: 5.9 scrub ok 2024-02-05T08:28:09.611 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[39278]: 3.18 scrub starts 2024-02-05T08:28:09.611 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[39278]: 3.18 scrub ok 2024-02-05T08:28:09.611 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1887878734' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:28:09.611 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T08:28:09.611 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:09 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:28:10.007 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 7 1 2024-02-05T08:28:10.251 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:10 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-05T08:28:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:10 smithi196 ceph-mon[33018]: osdmap e457: 8 total, 8 up, 8 in 2024-02-05T08:28:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:10 smithi196 ceph-mon[33018]: 1.6 scrub starts 2024-02-05T08:28:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:10 smithi196 ceph-mon[33018]: 1.6 scrub ok 2024-02-05T08:28:10.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:10 smithi196 ceph-mon[33018]: 4.1e scrub starts 2024-02-05T08:28:10.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:10 smithi196 ceph-mon[33018]: 4.1e scrub ok 2024-02-05T08:28:10.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:10 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:28:10.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:10 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:28:10.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-05T08:28:10.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[30086]: osdmap e457: 8 total, 8 up, 8 in 2024-02-05T08:28:10.655 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[30086]: 1.6 scrub starts 2024-02-05T08:28:10.656 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[30086]: 1.6 scrub ok 2024-02-05T08:28:10.656 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[30086]: 4.1e scrub starts 2024-02-05T08:28:10.656 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[30086]: 4.1e scrub ok 2024-02-05T08:28:10.656 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:28:10.656 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:28:10.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-05T08:28:10.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[39278]: osdmap e457: 8 total, 8 up, 8 in 2024-02-05T08:28:10.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[39278]: 1.6 scrub starts 2024-02-05T08:28:10.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[39278]: 1.6 scrub ok 2024-02-05T08:28:10.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[39278]: 4.1e scrub starts 2024-02-05T08:28:10.656 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[39278]: 4.1e scrub ok 2024-02-05T08:28:10.657 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:28:10.657 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:10 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:28:11.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[30086]: pgmap v1062: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:11.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[30086]: 5.4 scrub starts 2024-02-05T08:28:11.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[30086]: 5.4 scrub ok 2024-02-05T08:28:11.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[30086]: 5.11 scrub starts 2024-02-05T08:28:11.493 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[30086]: 5.11 scrub ok 2024-02-05T08:28:11.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[30086]: osdmap e458: 8 total, 8 up, 8 in 2024-02-05T08:28:11.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[30086]: 6.2 scrub starts 2024-02-05T08:28:11.494 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[30086]: 6.2 scrub ok 2024-02-05T08:28:11.494 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[39278]: pgmap v1062: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:11.494 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[39278]: 5.4 scrub starts 2024-02-05T08:28:11.494 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[39278]: 5.4 scrub ok 2024-02-05T08:28:11.494 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[39278]: 5.11 scrub starts 2024-02-05T08:28:11.494 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[39278]: 5.11 scrub ok 2024-02-05T08:28:11.494 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[39278]: osdmap e458: 8 total, 8 up, 8 in 2024-02-05T08:28:11.494 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[39278]: 6.2 scrub starts 2024-02-05T08:28:11.494 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:11 smithi134 ceph-mon[39278]: 6.2 scrub ok 2024-02-05T08:28:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:11 smithi196 ceph-mon[33018]: pgmap v1062: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:11 smithi196 ceph-mon[33018]: 5.4 scrub starts 2024-02-05T08:28:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:11 smithi196 ceph-mon[33018]: 5.4 scrub ok 2024-02-05T08:28:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:11 smithi196 ceph-mon[33018]: 5.11 scrub starts 2024-02-05T08:28:11.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:11 smithi196 ceph-mon[33018]: 5.11 scrub ok 2024-02-05T08:28:11.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:11 smithi196 ceph-mon[33018]: osdmap e458: 8 total, 8 up, 8 in 2024-02-05T08:28:11.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:11 smithi196 ceph-mon[33018]: 6.2 scrub starts 2024-02-05T08:28:11.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:11 smithi196 ceph-mon[33018]: 6.2 scrub ok 2024-02-05T08:28:12.314 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.7 to 1 (10000) 2024-02-05T08:28:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:12 smithi196 ceph-mon[33018]: 5.e scrub starts 2024-02-05T08:28:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:12 smithi196 ceph-mon[33018]: 5.e scrub ok 2024-02-05T08:28:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:12 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1542288981' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:28:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:12 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:28:12.666 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:12 smithi134 ceph-mon[30086]: 5.e scrub starts 2024-02-05T08:28:12.666 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:12 smithi134 ceph-mon[30086]: 5.e scrub ok 2024-02-05T08:28:12.667 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:12 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1542288981' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:28:12.667 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:12 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:28:12.667 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:12 smithi134 ceph-mon[39278]: 5.e scrub starts 2024-02-05T08:28:12.667 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:12 smithi134 ceph-mon[39278]: 5.e scrub ok 2024-02-05T08:28:12.667 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:12 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1542288981' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:28:12.667 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:12 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:28:12.970 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-05T08:28:13.213 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:13.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:13 smithi196 ceph-mon[33018]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-02-05T08:28:13.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:13 smithi196 ceph-mon[33018]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:28:13.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:13 smithi196 ceph-mon[33018]: osdmap e459: 8 total, 8 up, 8 in 2024-02-05T08:28:13.604 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:13 smithi134 ceph-mon[30086]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-02-05T08:28:13.604 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:13 smithi134 ceph-mon[30086]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:28:13.604 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:13 smithi134 ceph-mon[30086]: osdmap e459: 8 total, 8 up, 8 in 2024-02-05T08:28:13.604 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:13 smithi134 ceph-mon[39278]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-02-05T08:28:13.604 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:13 smithi134 ceph-mon[39278]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:28:13.604 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:13 smithi134 ceph-mon[39278]: osdmap e459: 8 total, 8 up, 8 in 2024-02-05T08:28:14.409 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:14 smithi134 ceph-mon[30086]: 5.b scrub starts 2024-02-05T08:28:14.409 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:14 smithi134 ceph-mon[30086]: 5.b scrub ok 2024-02-05T08:28:14.409 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:14 smithi134 ceph-mon[39278]: 5.b scrub starts 2024-02-05T08:28:14.409 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:14 smithi134 ceph-mon[39278]: 5.b scrub ok 2024-02-05T08:28:14.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:14 smithi196 ceph-mon[33018]: 5.b scrub starts 2024-02-05T08:28:14.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:14 smithi196 ceph-mon[33018]: 5.b scrub ok 2024-02-05T08:28:14.565 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:28:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:28:15.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[30086]: 3.1d scrub starts 2024-02-05T08:28:15.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[30086]: 3.1d scrub ok 2024-02-05T08:28:15.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[30086]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T08:28:15.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[30086]: 6.13 deep-scrub starts 2024-02-05T08:28:15.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[30086]: 6.13 deep-scrub ok 2024-02-05T08:28:15.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:15.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[30086]: 4.c deep-scrub starts 2024-02-05T08:28:15.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[30086]: 4.c deep-scrub ok 2024-02-05T08:28:15.778 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/659827153' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:28:15.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[39278]: 3.1d scrub starts 2024-02-05T08:28:15.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[39278]: 3.1d scrub ok 2024-02-05T08:28:15.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[39278]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T08:28:15.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[39278]: 6.13 deep-scrub starts 2024-02-05T08:28:15.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[39278]: 6.13 deep-scrub ok 2024-02-05T08:28:15.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:15.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[39278]: 4.c deep-scrub starts 2024-02-05T08:28:15.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[39278]: 4.c deep-scrub ok 2024-02-05T08:28:15.779 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:15 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/659827153' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:28:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:15 smithi196 ceph-mon[33018]: 3.1d scrub starts 2024-02-05T08:28:15.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:15 smithi196 ceph-mon[33018]: 3.1d scrub ok 2024-02-05T08:28:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:15 smithi196 ceph-mon[33018]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T08:28:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:15 smithi196 ceph-mon[33018]: 6.13 deep-scrub starts 2024-02-05T08:28:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:15 smithi196 ceph-mon[33018]: 6.13 deep-scrub ok 2024-02-05T08:28:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:15 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:15 smithi196 ceph-mon[33018]: 4.c deep-scrub starts 2024-02-05T08:28:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:15 smithi196 ceph-mon[33018]: 4.c deep-scrub ok 2024-02-05T08:28:15.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:15 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/659827153' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T08:28:15.974 INFO:teuthology.orchestra.run.smithi134.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-05T08:28:16.624 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:16 smithi134 ceph-mon[30086]: 6.12 scrub starts 2024-02-05T08:28:16.624 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:16 smithi134 ceph-mon[30086]: 6.12 scrub ok 2024-02-05T08:28:16.624 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:16 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/659827153' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:28:16.624 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:16 smithi134 ceph-mon[30086]: osdmap e460: 8 total, 8 up, 8 in 2024-02-05T08:28:16.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:16 smithi134 ceph-mon[39278]: 6.12 scrub starts 2024-02-05T08:28:16.624 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:16 smithi134 ceph-mon[39278]: 6.12 scrub ok 2024-02-05T08:28:16.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:16 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/659827153' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:28:16.625 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:16 smithi134 ceph-mon[39278]: osdmap e460: 8 total, 8 up, 8 in 2024-02-05T08:28:16.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd reweight 1 1 2024-02-05T08:28:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:16 smithi196 ceph-mon[33018]: 6.12 scrub starts 2024-02-05T08:28:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:16 smithi196 ceph-mon[33018]: 6.12 scrub ok 2024-02-05T08:28:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:16 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/659827153' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-05T08:28:16.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:16 smithi196 ceph-mon[33018]: osdmap e460: 8 total, 8 up, 8 in 2024-02-05T08:28:16.897 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:17.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:17 smithi134 ceph-mon[39278]: pgmap v1068: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:17.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:17 smithi134 ceph-mon[30086]: pgmap v1068: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:17.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:17 smithi196 ceph-mon[33018]: pgmap v1068: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:18.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:28:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:28:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:28:18.383 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:18 smithi134 ceph-mon[30086]: 4.6 deep-scrub starts 2024-02-05T08:28:18.384 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:18 smithi134 ceph-mon[30086]: 4.6 deep-scrub ok 2024-02-05T08:28:18.384 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:18 smithi134 ceph-mon[39278]: 4.6 deep-scrub starts 2024-02-05T08:28:18.384 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:18 smithi134 ceph-mon[39278]: 4.6 deep-scrub ok 2024-02-05T08:28:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:18 smithi196 ceph-mon[33018]: 4.6 deep-scrub starts 2024-02-05T08:28:18.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:18 smithi196 ceph-mon[33018]: 4.6 deep-scrub ok 2024-02-05T08:28:19.335 INFO:teuthology.orchestra.run.smithi134.stderr:reweighted osd.1 to 1 (10000) 2024-02-05T08:28:19.696 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:19 smithi134 ceph-mon[30086]: pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 665 B/s rd, 0 op/s 2024-02-05T08:28:19.696 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:19 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:28:19.697 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:19 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:28:19.697 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:19 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2533176650' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:28:19.697 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:19 smithi134 ceph-mon[39278]: pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 665 B/s rd, 0 op/s 2024-02-05T08:28:19.697 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:19 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:28:19.697 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:19 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:28:19.697 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:19 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2533176650' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:28:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:19 smithi196 ceph-mon[33018]: pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 665 B/s rd, 0 op/s 2024-02-05T08:28:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:19 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:28:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:19 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:28:19.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:19 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2533176650' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:28:20.100 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 1 1 2024-02-05T08:28:20.344 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:20.588 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:20 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2533176650' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:28:20.588 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:20 smithi134 ceph-mon[30086]: osdmap e461: 8 total, 8 up, 8 in 2024-02-05T08:28:20.588 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:20 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2533176650' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:28:20.588 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:20 smithi134 ceph-mon[39278]: osdmap e461: 8 total, 8 up, 8 in 2024-02-05T08:28:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:20 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2533176650' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:28:20.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:20 smithi196 ceph-mon[33018]: osdmap e461: 8 total, 8 up, 8 in 2024-02-05T08:28:21.480 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:21 smithi134 ceph-mon[30086]: pgmap v1071: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:28:21.480 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:21 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:28:21.480 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:21 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:28:21.480 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:21 smithi134 ceph-mon[30086]: 3.4 deep-scrub starts 2024-02-05T08:28:21.480 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:21 smithi134 ceph-mon[30086]: 3.4 deep-scrub ok 2024-02-05T08:28:21.480 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:21 smithi134 ceph-mon[30086]: osdmap e462: 8 total, 8 up, 8 in 2024-02-05T08:28:21.480 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:21 smithi134 ceph-mon[39278]: pgmap v1071: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:28:21.480 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:21 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:28:21.480 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:21 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:28:21.481 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:21 smithi134 ceph-mon[39278]: 3.4 deep-scrub starts 2024-02-05T08:28:21.481 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:21 smithi134 ceph-mon[39278]: 3.4 deep-scrub ok 2024-02-05T08:28:21.481 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:21 smithi134 ceph-mon[39278]: osdmap e462: 8 total, 8 up, 8 in 2024-02-05T08:28:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:21 smithi196 ceph-mon[33018]: pgmap v1071: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:28:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:21 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:28:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:21 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:28:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:21 smithi196 ceph-mon[33018]: 3.4 deep-scrub starts 2024-02-05T08:28:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:21 smithi196 ceph-mon[33018]: 3.4 deep-scrub ok 2024-02-05T08:28:21.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:21 smithi196 ceph-mon[33018]: osdmap e462: 8 total, 8 up, 8 in 2024-02-05T08:28:22.345 INFO:teuthology.orchestra.run.smithi134.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-02-05T08:28:22.712 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:22 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2002314587' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:28:22.712 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:22 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2002314587' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:28:22.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:22 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2002314587' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T08:28:23.175 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-02-05T08:28:23.175 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-02-05T08:28:23.175 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-02-05T08:28:23.175 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-02-05T08:28:23.175 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-05T08:28:23.175 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T08:28:23.425 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[30086]: pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 849 B/s rd, 0 op/s 2024-02-05T08:28:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[30086]: 3.2 scrub starts 2024-02-05T08:28:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[30086]: 3.2 scrub ok 2024-02-05T08:28:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2002314587' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:28:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[30086]: osdmap e463: 8 total, 8 up, 8 in 2024-02-05T08:28:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[30086]: 5.18 scrub starts 2024-02-05T08:28:23.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[30086]: 5.18 scrub ok 2024-02-05T08:28:23.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[39278]: pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 849 B/s rd, 0 op/s 2024-02-05T08:28:23.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[39278]: 3.2 scrub starts 2024-02-05T08:28:23.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[39278]: 3.2 scrub ok 2024-02-05T08:28:23.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2002314587' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:28:23.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[39278]: osdmap e463: 8 total, 8 up, 8 in 2024-02-05T08:28:23.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[39278]: 5.18 scrub starts 2024-02-05T08:28:23.528 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:23 smithi134 ceph-mon[39278]: 5.18 scrub ok 2024-02-05T08:28:23.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:23 smithi196 ceph-mon[33018]: pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 849 B/s rd, 0 op/s 2024-02-05T08:28:23.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:23 smithi196 ceph-mon[33018]: 3.2 scrub starts 2024-02-05T08:28:23.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:23 smithi196 ceph-mon[33018]: 3.2 scrub ok 2024-02-05T08:28:23.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:23 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2002314587' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-05T08:28:23.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:23 smithi196 ceph-mon[33018]: osdmap e463: 8 total, 8 up, 8 in 2024-02-05T08:28:23.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:23 smithi196 ceph-mon[33018]: 5.18 scrub starts 2024-02-05T08:28:23.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:23 smithi196 ceph-mon[33018]: 5.18 scrub ok 2024-02-05T08:28:24.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:28:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:28:24.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:24 smithi196 ceph-mon[33018]: 3.16 scrub starts 2024-02-05T08:28:24.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:24 smithi196 ceph-mon[33018]: 3.16 scrub ok 2024-02-05T08:28:24.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:24 smithi196 ceph-mon[33018]: 6.c scrub starts 2024-02-05T08:28:24.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:24 smithi196 ceph-mon[33018]: 6.c scrub ok 2024-02-05T08:28:24.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:24 smithi196 ceph-mon[33018]: osdmap e464: 8 total, 8 up, 8 in 2024-02-05T08:28:24.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:24 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:28:24.704 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:24 smithi134 ceph-mon[30086]: 3.16 scrub starts 2024-02-05T08:28:24.704 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:24 smithi134 ceph-mon[30086]: 3.16 scrub ok 2024-02-05T08:28:24.704 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:24 smithi134 ceph-mon[30086]: 6.c scrub starts 2024-02-05T08:28:24.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:24 smithi134 ceph-mon[30086]: 6.c scrub ok 2024-02-05T08:28:24.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:24 smithi134 ceph-mon[30086]: osdmap e464: 8 total, 8 up, 8 in 2024-02-05T08:28:24.705 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:24 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:28:24.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:24 smithi134 ceph-mon[39278]: 3.16 scrub starts 2024-02-05T08:28:24.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:24 smithi134 ceph-mon[39278]: 3.16 scrub ok 2024-02-05T08:28:24.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:24 smithi134 ceph-mon[39278]: 6.c scrub starts 2024-02-05T08:28:24.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:24 smithi134 ceph-mon[39278]: 6.c scrub ok 2024-02-05T08:28:24.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:24 smithi134 ceph-mon[39278]: osdmap e464: 8 total, 8 up, 8 in 2024-02-05T08:28:24.705 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:24 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:28:25.419 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:28:25.420 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":464,"fsid":"43e13aa4-c3fc-11ee-95b6-87774f69a715","created":"2024-02-05T07:59:09.321262+0000","modified":"2024-02-05T08:28:23.338868+0000","last_up_change":"2024-02-05T08:23:06.892657+0000","last_in_change":"2024-02-05T08:25:48.086167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T08:02:21.034012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"438","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"301","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-05T08:04:39.400746+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-05T08:04:40.005769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-05T08:04:41.851891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-05T08:04:43.793804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-05T08:04:45.853999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d90a5ea0-9a19-452e-9564-ca7b7a7c45e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6803","nonce":2540271081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6805","nonce":2540271081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6809","nonce":2540271081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6807","nonce":2540271081}]},"public_addr":"172.21.15.134:6803/2540271081","cluster_addr":"172.21.15.134:6805/2540271081","heartbeat_back_addr":"172.21.15.134:6809/2540271081","heartbeat_front_addr":"172.21.15.134:6807/2540271081","state":["exists","up"]},{"osd":1,"uuid":"4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":323,"up_from":366,"up_thru":463,"down_at":324,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1259179009},{"type":"v1","addr":"172.21.15.134:6811","nonce":1259179009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1259179009},{"type":"v1","addr":"172.21.15.134:6813","nonce":1259179009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":1259179009},{"type":"v1","addr":"172.21.15.134:6817","nonce":1259179009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1259179009},{"type":"v1","addr":"172.21.15.134:6815","nonce":1259179009}]},"public_addr":"172.21.15.134:6811/1259179009","cluster_addr":"172.21.15.134:6813/1259179009","heartbeat_back_addr":"172.21.15.134:6817/1259179009","heartbeat_front_addr":"172.21.15.134:6815/1259179009","state":["exists","up"]},{"osd":2,"uuid":"078e578b-855b-49ff-83bd-0548a9c5a708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":446,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6819","nonce":153827987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6821","nonce":153827987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6825","nonce":153827987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6823","nonce":153827987}]},"public_addr":"172.21.15.134:6819/153827987","cluster_addr":"172.21.15.134:6821/153827987","heartbeat_back_addr":"172.21.15.134:6825/153827987","heartbeat_front_addr":"172.21.15.134:6823/153827987","state":["exists","up"]},{"osd":3,"uuid":"6774e330-3202-4671-894e-172cc2f6189d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6827","nonce":1527507056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6829","nonce":1527507056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6833","nonce":1527507056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6831","nonce":1527507056}]},"public_addr":"172.21.15.134:6827/1527507056","cluster_addr":"172.21.15.134:6829/1527507056","heartbeat_back_addr":"172.21.15.134:6833/1527507056","heartbeat_front_addr":"172.21.15.134:6831/1527507056","state":["exists","up"]},{"osd":4,"uuid":"52f70cfa-ab0e-4751-a5f8-1045c61c1693","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6801","nonce":3005156808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6803","nonce":3005156808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6807","nonce":3005156808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6805","nonce":3005156808}]},"public_addr":"172.21.15.196:6801/3005156808","cluster_addr":"172.21.15.196:6803/3005156808","heartbeat_back_addr":"172.21.15.196:6807/3005156808","heartbeat_front_addr":"172.21.15.196:6805/3005156808","state":["exists","up"]},{"osd":5,"uuid":"b4b21769-d40f-43fd-b7e9-2477b74d3955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":453,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6809","nonce":1603224792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6811","nonce":1603224792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6815","nonce":1603224792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6813","nonce":1603224792}]},"public_addr":"172.21.15.196:6809/1603224792","cluster_addr":"172.21.15.196:6811/1603224792","heartbeat_back_addr":"172.21.15.196:6815/1603224792","heartbeat_front_addr":"172.21.15.196:6813/1603224792","state":["exists","up"]},{"osd":6,"uuid":"af1cced4-0015-47f5-b105-27641bfe4f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6817","nonce":3727413143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6819","nonce":3727413143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6823","nonce":3727413143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6821","nonce":3727413143}]},"public_addr":"172.21.15.196:6817/3727413143","cluster_addr":"172.21.15.196:6819/3727413143","heartbeat_back_addr":"172.21.15.196:6823/3727413143","heartbeat_front_addr":"172.21.15.196:6821/3727413143","state":["exists","up"]},{"osd":7,"uuid":"c54beeae-9f7c-4723-a369-13bb531b4057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6825","nonce":4206571010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6827","nonce":4206571010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6831","nonce":4206571010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6829","nonce":4206571010}]},"public_addr":"172.21.15.196:6825/4206571010","cluster_addr":"172.21.15.196:6827/4206571010","heartbeat_back_addr":"172.21.15.196:6831/4206571010","heartbeat_front_addr":"172.21.15.196:6829/4206571010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:27:21.664694+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:28:09.525480+0000","dead_epoch":323},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:27:31.083637+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:28:18.623790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:28:04.280110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:28:20.270890+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:27:20.293294+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:28:03.525909+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.0","osds":[3,1,5]},{"pgid":"6.6","osds":[4,1,3]}],"pg_upmap_items":[{"pgid":"1.1","mappings":[{"from":0,"to":6}]},{"pgid":"3.6","mappings":[{"from":4,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/788812699":"2024-02-06T08:05:21.465844+0000","172.21.15.134:0/1046500326":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/464208089":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/301941854":"2024-02-06T07:59:36.275039+0000","172.21.15.134:6800/3607168099":"2024-02-06T08:00:02.709280+0000","172.21.15.134:6800/673876301":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/4225724311":"2024-02-06T08:00:02.709280+0000","172.21.15.134:6800/1525062017":"2024-02-06T08:05:21.465844+0000","172.21.15.134:6801/3607168099":"2024-02-06T08:00:02.709280+0000","172.21.15.134:6801/673876301":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/3200884805":"2024-02-06T08:05:21.465844+0000","172.21.15.134:0/2952444615":"2024-02-06T08:00:02.709280+0000","172.21.15.134:0/3325835814":"2024-02-06T08:05:21.465844+0000","172.21.15.134:0/3161126612":"2024-02-06T08:00:02.709280+0000","172.21.15.134:0/69097091":"2024-02-06T08:05:21.465844+0000","172.21.15.134:6801/1525062017":"2024-02-06T08:05:21.465844+0000","172.21.15.134:0/1645475987":"2024-02-06T08:05:21.465844+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T08:28:25.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:25 smithi134 ceph-mon[30086]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:28:25.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:25 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:25.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:25 smithi134 ceph-mon[30086]: 4.b scrub starts 2024-02-05T08:28:25.728 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:25 smithi134 ceph-mon[30086]: 4.b scrub ok 2024-02-05T08:28:25.728 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:25 smithi134 ceph-mon[39278]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:28:25.728 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:25 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:25.728 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:25 smithi134 ceph-mon[39278]: 4.b scrub starts 2024-02-05T08:28:25.728 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:25 smithi134 ceph-mon[39278]: 4.b scrub ok 2024-02-05T08:28:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:25 smithi196 ceph-mon[33018]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T08:28:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:25 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:25 smithi196 ceph-mon[33018]: 4.b scrub starts 2024-02-05T08:28:25.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:25 smithi196 ceph-mon[33018]: 4.b scrub ok 2024-02-05T08:28:26.039 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-05T08:28:26.039 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T08:28:26.283 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:26.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:26 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/271080626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:28:26.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:26 smithi134 ceph-mon[30086]: 5.1c scrub starts 2024-02-05T08:28:26.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:26 smithi134 ceph-mon[30086]: 5.1c scrub ok 2024-02-05T08:28:26.731 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:26 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/271080626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:28:26.731 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:26 smithi134 ceph-mon[39278]: 5.1c scrub starts 2024-02-05T08:28:26.731 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:26 smithi134 ceph-mon[39278]: 5.1c scrub ok 2024-02-05T08:28:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:26 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/271080626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:28:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:26 smithi196 ceph-mon[33018]: 5.1c scrub starts 2024-02-05T08:28:26.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:26 smithi196 ceph-mon[33018]: 5.1c scrub ok 2024-02-05T08:28:27.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:27 smithi134 ceph-mon[30086]: pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:27.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:27 smithi134 ceph-mon[39278]: pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:27.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:27 smithi196 ceph-mon[33018]: pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:27.937 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:28:27 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:28:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:28:28.399 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:28:28.400 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":465,"fsid":"43e13aa4-c3fc-11ee-95b6-87774f69a715","created":"2024-02-05T07:59:09.321262+0000","modified":"2024-02-05T08:28:27.361226+0000","last_up_change":"2024-02-05T08:23:06.892657+0000","last_in_change":"2024-02-05T08:25:48.086167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T08:02:21.034012+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"438","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"301","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-05T08:04:39.400746+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":70,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-05T08:04:40.005769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-05T08:04:41.851891+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-05T08:04:43.793804+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-05T08:04:45.853999+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d90a5ea0-9a19-452e-9564-ca7b7a7c45e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6803","nonce":2540271081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6805","nonce":2540271081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6809","nonce":2540271081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":2540271081},{"type":"v1","addr":"172.21.15.134:6807","nonce":2540271081}]},"public_addr":"172.21.15.134:6803/2540271081","cluster_addr":"172.21.15.134:6805/2540271081","heartbeat_back_addr":"172.21.15.134:6809/2540271081","heartbeat_front_addr":"172.21.15.134:6807/2540271081","state":["exists","up"]},{"osd":1,"uuid":"4911ac1a-4d83-4e79-b96d-8be0d9cc5ad4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":323,"up_from":366,"up_thru":463,"down_at":324,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6810","nonce":1259179009},{"type":"v1","addr":"172.21.15.134:6811","nonce":1259179009}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6812","nonce":1259179009},{"type":"v1","addr":"172.21.15.134:6813","nonce":1259179009}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6816","nonce":1259179009},{"type":"v1","addr":"172.21.15.134:6817","nonce":1259179009}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6814","nonce":1259179009},{"type":"v1","addr":"172.21.15.134:6815","nonce":1259179009}]},"public_addr":"172.21.15.134:6811/1259179009","cluster_addr":"172.21.15.134:6813/1259179009","heartbeat_back_addr":"172.21.15.134:6817/1259179009","heartbeat_front_addr":"172.21.15.134:6815/1259179009","state":["exists","up"]},{"osd":2,"uuid":"078e578b-855b-49ff-83bd-0548a9c5a708","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":446,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6818","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6819","nonce":153827987}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6820","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6821","nonce":153827987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6824","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6825","nonce":153827987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6822","nonce":153827987},{"type":"v1","addr":"172.21.15.134:6823","nonce":153827987}]},"public_addr":"172.21.15.134:6819/153827987","cluster_addr":"172.21.15.134:6821/153827987","heartbeat_back_addr":"172.21.15.134:6825/153827987","heartbeat_front_addr":"172.21.15.134:6823/153827987","state":["exists","up"]},{"osd":3,"uuid":"6774e330-3202-4671-894e-172cc2f6189d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6826","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6827","nonce":1527507056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6828","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6829","nonce":1527507056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6832","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6833","nonce":1527507056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6830","nonce":1527507056},{"type":"v1","addr":"172.21.15.134:6831","nonce":1527507056}]},"public_addr":"172.21.15.134:6827/1527507056","cluster_addr":"172.21.15.134:6829/1527507056","heartbeat_back_addr":"172.21.15.134:6833/1527507056","heartbeat_front_addr":"172.21.15.134:6831/1527507056","state":["exists","up"]},{"osd":4,"uuid":"52f70cfa-ab0e-4751-a5f8-1045c61c1693","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6801","nonce":3005156808}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6803","nonce":3005156808}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6807","nonce":3005156808}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3005156808},{"type":"v1","addr":"172.21.15.196:6805","nonce":3005156808}]},"public_addr":"172.21.15.196:6801/3005156808","cluster_addr":"172.21.15.196:6803/3005156808","heartbeat_back_addr":"172.21.15.196:6807/3005156808","heartbeat_front_addr":"172.21.15.196:6805/3005156808","state":["exists","up"]},{"osd":5,"uuid":"b4b21769-d40f-43fd-b7e9-2477b74d3955","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":453,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6809","nonce":1603224792}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6811","nonce":1603224792}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6815","nonce":1603224792}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":1603224792},{"type":"v1","addr":"172.21.15.196:6813","nonce":1603224792}]},"public_addr":"172.21.15.196:6809/1603224792","cluster_addr":"172.21.15.196:6811/1603224792","heartbeat_back_addr":"172.21.15.196:6815/1603224792","heartbeat_front_addr":"172.21.15.196:6813/1603224792","state":["exists","up"]},{"osd":6,"uuid":"af1cced4-0015-47f5-b105-27641bfe4f4a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6817","nonce":3727413143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6819","nonce":3727413143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6823","nonce":3727413143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":3727413143},{"type":"v1","addr":"172.21.15.196:6821","nonce":3727413143}]},"public_addr":"172.21.15.196:6817/3727413143","cluster_addr":"172.21.15.196:6819/3727413143","heartbeat_back_addr":"172.21.15.196:6823/3727413143","heartbeat_front_addr":"172.21.15.196:6821/3727413143","state":["exists","up"]},{"osd":7,"uuid":"c54beeae-9f7c-4723-a369-13bb531b4057","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6825","nonce":4206571010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6827","nonce":4206571010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6831","nonce":4206571010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":4206571010},{"type":"v1","addr":"172.21.15.196:6829","nonce":4206571010}]},"public_addr":"172.21.15.196:6825/4206571010","cluster_addr":"172.21.15.196:6827/4206571010","heartbeat_back_addr":"172.21.15.196:6831/4206571010","heartbeat_front_addr":"172.21.15.196:6829/4206571010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:27:21.664694+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:28:09.525480+0000","dead_epoch":323},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:27:31.083637+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:28:18.623790+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:28:04.280110+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:28:20.270890+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:28:27.229182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T08:28:03.525909+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.0","osds":[3,1,5]},{"pgid":"6.6","osds":[4,1,3]}],"pg_upmap_items":[{"pgid":"1.1","mappings":[{"from":0,"to":6}]},{"pgid":"3.6","mappings":[{"from":4,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.7","mappings":[{"from":1,"to":2}]},{"pgid":"4.a","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.134:0/788812699":"2024-02-06T08:05:21.465844+0000","172.21.15.134:0/1046500326":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/464208089":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/301941854":"2024-02-06T07:59:36.275039+0000","172.21.15.134:6800/3607168099":"2024-02-06T08:00:02.709280+0000","172.21.15.134:6800/673876301":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/4225724311":"2024-02-06T08:00:02.709280+0000","172.21.15.134:6800/1525062017":"2024-02-06T08:05:21.465844+0000","172.21.15.134:6801/3607168099":"2024-02-06T08:00:02.709280+0000","172.21.15.134:6801/673876301":"2024-02-06T07:59:36.275039+0000","172.21.15.134:0/3200884805":"2024-02-06T08:05:21.465844+0000","172.21.15.134:0/2952444615":"2024-02-06T08:00:02.709280+0000","172.21.15.134:0/3325835814":"2024-02-06T08:05:21.465844+0000","172.21.15.134:0/3161126612":"2024-02-06T08:00:02.709280+0000","172.21.15.134:0/69097091":"2024-02-06T08:05:21.465844+0000","172.21.15.134:6801/1525062017":"2024-02-06T08:05:21.465844+0000","172.21.15.134:0/1645475987":"2024-02-06T08:05:21.465844+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T08:28:28.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:28 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:28:28.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:28 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:28:28.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:28 smithi134 ceph-mon[30086]: osdmap e465: 8 total, 8 up, 8 in 2024-02-05T08:28:28.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:28 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:28:28.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:28 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:28:28.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:28 smithi134 ceph-mon[39278]: osdmap e465: 8 total, 8 up, 8 in 2024-02-05T08:28:28.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:28 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:28:28.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:28 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:28:28.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:28 smithi196 ceph-mon[33018]: osdmap e465: 8 total, 8 up, 8 in 2024-02-05T08:28:29.141 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-05T08:28:29.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-05T08:28:29.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-05T08:28:29.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-05T08:28:29.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-05T08:28:29.143 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-05T08:28:29.144 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-05T08:28:29.144 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-05T08:28:29.377 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:29 smithi134 ceph-mon[30086]: pgmap v1079: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:29.377 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:29 smithi134 ceph-mon[30086]: 1.4 scrub starts 2024-02-05T08:28:29.377 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:29 smithi134 ceph-mon[30086]: 1.4 scrub ok 2024-02-05T08:28:29.377 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:29 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/1382966086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:28:29.677 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:29.678 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:29.678 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:29.679 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:29.679 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:29.684 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:29.688 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:29.688 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:29.776 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:29 smithi134 ceph-mon[39278]: pgmap v1079: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:29.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:29 smithi134 ceph-mon[39278]: 1.4 scrub starts 2024-02-05T08:28:29.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:29 smithi134 ceph-mon[39278]: 1.4 scrub ok 2024-02-05T08:28:29.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:29 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/1382966086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:28:29.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:29 smithi196 ceph-mon[33018]: pgmap v1079: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:29.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:29 smithi196 ceph-mon[33018]: 1.4 scrub starts 2024-02-05T08:28:29.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:29 smithi196 ceph-mon[33018]: 1.4 scrub ok 2024-02-05T08:28:29.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:29 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/1382966086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T08:28:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:30 smithi134 ceph-mon[30086]: 6.19 scrub starts 2024-02-05T08:28:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:30 smithi134 ceph-mon[30086]: 6.19 scrub ok 2024-02-05T08:28:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:30 smithi134 ceph-mon[30086]: 6.11 scrub starts 2024-02-05T08:28:30.671 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:30 smithi134 ceph-mon[30086]: 6.11 scrub ok 2024-02-05T08:28:30.671 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:30 smithi134 ceph-mon[39278]: 6.19 scrub starts 2024-02-05T08:28:30.671 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:30 smithi134 ceph-mon[39278]: 6.19 scrub ok 2024-02-05T08:28:30.672 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:30 smithi134 ceph-mon[39278]: 6.11 scrub starts 2024-02-05T08:28:30.672 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:30 smithi134 ceph-mon[39278]: 6.11 scrub ok 2024-02-05T08:28:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:30 smithi196 ceph-mon[33018]: 6.19 scrub starts 2024-02-05T08:28:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:30 smithi196 ceph-mon[33018]: 6.19 scrub ok 2024-02-05T08:28:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:30 smithi196 ceph-mon[33018]: 6.11 scrub starts 2024-02-05T08:28:30.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:30 smithi196 ceph-mon[33018]: 6.11 scrub ok 2024-02-05T08:28:31.619 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:31 smithi134 ceph-mon[30086]: pgmap v1080: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-05T08:28:31.619 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:31 smithi134 ceph-mon[39278]: pgmap v1080: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-05T08:28:31.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:31 smithi196 ceph-mon[33018]: pgmap v1080: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-05T08:28:33.634 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:33 smithi134 ceph-mon[30086]: pgmap v1081: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T08:28:33.634 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:33 smithi134 ceph-mon[30086]: 6.6 scrub starts 2024-02-05T08:28:33.634 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:33 smithi134 ceph-mon[30086]: 6.6 scrub ok 2024-02-05T08:28:33.634 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:33 smithi134 ceph-mon[39278]: pgmap v1081: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T08:28:33.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:33 smithi134 ceph-mon[39278]: 6.6 scrub starts 2024-02-05T08:28:33.635 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:33 smithi134 ceph-mon[39278]: 6.6 scrub ok 2024-02-05T08:28:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:33 smithi196 ceph-mon[33018]: pgmap v1081: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T08:28:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:33 smithi196 ceph-mon[33018]: 6.6 scrub starts 2024-02-05T08:28:33.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:33 smithi196 ceph-mon[33018]: 6.6 scrub ok 2024-02-05T08:28:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:34 smithi196 ceph-mon[33018]: 5.3 scrub starts 2024-02-05T08:28:34.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:34 smithi196 ceph-mon[33018]: 5.3 scrub ok 2024-02-05T08:28:34.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:28:34 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:28:34.731 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:34 smithi134 ceph-mon[30086]: 5.3 scrub starts 2024-02-05T08:28:34.732 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:34 smithi134 ceph-mon[30086]: 5.3 scrub ok 2024-02-05T08:28:34.732 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:34 smithi134 ceph-mon[39278]: 5.3 scrub starts 2024-02-05T08:28:34.732 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:34 smithi134 ceph-mon[39278]: 5.3 scrub ok 2024-02-05T08:28:35.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:35 smithi134 ceph-mon[30086]: pgmap v1082: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:35.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:35 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:35.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:35 smithi134 ceph-mon[30086]: 4.16 scrub starts 2024-02-05T08:28:35.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:35 smithi134 ceph-mon[30086]: 4.16 scrub ok 2024-02-05T08:28:35.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:35 smithi134 ceph-mon[39278]: pgmap v1082: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:35.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:35 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:35.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:35 smithi134 ceph-mon[39278]: 4.16 scrub starts 2024-02-05T08:28:35.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:35 smithi134 ceph-mon[39278]: 4.16 scrub ok 2024-02-05T08:28:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:35 smithi196 ceph-mon[33018]: pgmap v1082: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:35 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:35 smithi196 ceph-mon[33018]: 4.16 scrub starts 2024-02-05T08:28:35.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:35 smithi196 ceph-mon[33018]: 4.16 scrub ok 2024-02-05T08:28:36.570 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:36 smithi134 ceph-mon[30086]: 6.a scrub starts 2024-02-05T08:28:36.570 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:36 smithi134 ceph-mon[30086]: 6.a scrub ok 2024-02-05T08:28:36.570 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:36 smithi134 ceph-mon[30086]: 4.9 deep-scrub starts 2024-02-05T08:28:36.570 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:36 smithi134 ceph-mon[30086]: 4.9 deep-scrub ok 2024-02-05T08:28:36.570 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:36 smithi134 ceph-mon[39278]: 6.a scrub starts 2024-02-05T08:28:36.570 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:36 smithi134 ceph-mon[39278]: 6.a scrub ok 2024-02-05T08:28:36.570 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:36 smithi134 ceph-mon[39278]: 4.9 deep-scrub starts 2024-02-05T08:28:36.570 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:36 smithi134 ceph-mon[39278]: 4.9 deep-scrub ok 2024-02-05T08:28:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:36 smithi196 ceph-mon[33018]: 6.a scrub starts 2024-02-05T08:28:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:36 smithi196 ceph-mon[33018]: 6.a scrub ok 2024-02-05T08:28:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:36 smithi196 ceph-mon[33018]: 4.9 deep-scrub starts 2024-02-05T08:28:36.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:36 smithi196 ceph-mon[33018]: 4.9 deep-scrub ok 2024-02-05T08:28:37.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[30086]: pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:37.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[30086]: 3.10 scrub starts 2024-02-05T08:28:37.738 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[30086]: 3.10 scrub ok 2024-02-05T08:28:37.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[30086]: 4.e deep-scrub starts 2024-02-05T08:28:37.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[30086]: 4.e deep-scrub ok 2024-02-05T08:28:37.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[30086]: 5.1d scrub starts 2024-02-05T08:28:37.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[30086]: 5.1d scrub ok 2024-02-05T08:28:37.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[30086]: 4.7 deep-scrub starts 2024-02-05T08:28:37.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[30086]: 4.7 deep-scrub ok 2024-02-05T08:28:37.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[39278]: pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:37.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[39278]: 3.10 scrub starts 2024-02-05T08:28:37.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[39278]: 3.10 scrub ok 2024-02-05T08:28:37.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[39278]: 4.e deep-scrub starts 2024-02-05T08:28:37.739 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[39278]: 4.e deep-scrub ok 2024-02-05T08:28:37.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[39278]: 5.1d scrub starts 2024-02-05T08:28:37.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[39278]: 5.1d scrub ok 2024-02-05T08:28:37.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[39278]: 4.7 deep-scrub starts 2024-02-05T08:28:37.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-mon[39278]: 4.7 deep-scrub ok 2024-02-05T08:28:37.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:37 smithi196 ceph-mon[33018]: pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:37.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:37 smithi196 ceph-mon[33018]: 3.10 scrub starts 2024-02-05T08:28:37.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:37 smithi196 ceph-mon[33018]: 3.10 scrub ok 2024-02-05T08:28:37.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:37 smithi196 ceph-mon[33018]: 4.e deep-scrub starts 2024-02-05T08:28:37.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:37 smithi196 ceph-mon[33018]: 4.e deep-scrub ok 2024-02-05T08:28:37.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:37 smithi196 ceph-mon[33018]: 5.1d scrub starts 2024-02-05T08:28:37.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:37 smithi196 ceph-mon[33018]: 5.1d scrub ok 2024-02-05T08:28:37.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:37 smithi196 ceph-mon[33018]: 4.7 deep-scrub starts 2024-02-05T08:28:37.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:37 smithi196 ceph-mon[33018]: 4.7 deep-scrub ok 2024-02-05T08:28:37.990 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:28:37 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:28:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:28:38.421 INFO:teuthology.orchestra.run.smithi134.stdout:115964117305 2024-02-05T08:28:38.421 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-05T08:28:39.682 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:39 smithi134 ceph-mon[30086]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:28:39.683 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:39 smithi134 ceph-mon[30086]: 1.5 scrub starts 2024-02-05T08:28:39.683 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:39 smithi134 ceph-mon[30086]: 1.5 scrub ok 2024-02-05T08:28:39.683 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:39 smithi134 ceph-mon[30086]: 6.1d deep-scrub starts 2024-02-05T08:28:39.683 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:39 smithi134 ceph-mon[30086]: 6.1d deep-scrub ok 2024-02-05T08:28:39.683 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:39 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:28:39.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:39 smithi134 ceph-mon[39278]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:28:39.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:39 smithi134 ceph-mon[39278]: 1.5 scrub starts 2024-02-05T08:28:39.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:39 smithi134 ceph-mon[39278]: 1.5 scrub ok 2024-02-05T08:28:39.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:39 smithi134 ceph-mon[39278]: 6.1d deep-scrub starts 2024-02-05T08:28:39.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:39 smithi134 ceph-mon[39278]: 6.1d deep-scrub ok 2024-02-05T08:28:39.683 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:39 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:28:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:39 smithi196 ceph-mon[33018]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T08:28:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:39 smithi196 ceph-mon[33018]: 1.5 scrub starts 2024-02-05T08:28:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:39 smithi196 ceph-mon[33018]: 1.5 scrub ok 2024-02-05T08:28:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:39 smithi196 ceph-mon[33018]: 6.1d deep-scrub starts 2024-02-05T08:28:39.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:39 smithi196 ceph-mon[33018]: 6.1d deep-scrub ok 2024-02-05T08:28:39.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:39 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:28:40.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:40 smithi134 ceph-mon[30086]: 5.10 scrub starts 2024-02-05T08:28:40.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:40 smithi134 ceph-mon[30086]: 5.10 scrub ok 2024-02-05T08:28:40.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:40 smithi134 ceph-mon[39278]: 5.10 scrub starts 2024-02-05T08:28:40.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:40 smithi134 ceph-mon[39278]: 5.10 scrub ok 2024-02-05T08:28:40.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:40 smithi196 ceph-mon[33018]: 5.10 scrub starts 2024-02-05T08:28:40.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:40 smithi196 ceph-mon[33018]: 5.10 scrub ok 2024-02-05T08:28:41.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:41 smithi134 ceph-mon[30086]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:41.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:41 smithi134 ceph-mon[30086]: 5.5 scrub starts 2024-02-05T08:28:41.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:41 smithi134 ceph-mon[30086]: 5.5 scrub ok 2024-02-05T08:28:41.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:41 smithi134 ceph-mon[39278]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:41.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:41 smithi134 ceph-mon[39278]: 5.5 scrub starts 2024-02-05T08:28:41.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:41 smithi134 ceph-mon[39278]: 5.5 scrub ok 2024-02-05T08:28:41.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:41 smithi196 ceph-mon[33018]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:41.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:41 smithi196 ceph-mon[33018]: 5.5 scrub starts 2024-02-05T08:28:41.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:41 smithi196 ceph-mon[33018]: 5.5 scrub ok 2024-02-05T08:28:41.905 INFO:teuthology.orchestra.run.smithi134.stdout:257698038052 2024-02-05T08:28:41.905 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-05T08:28:42.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:42 smithi134 ceph-mon[30086]: 5.f scrub starts 2024-02-05T08:28:42.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:42 smithi134 ceph-mon[30086]: 5.f scrub ok 2024-02-05T08:28:42.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:42 smithi134 ceph-mon[30086]: 4.5 scrub starts 2024-02-05T08:28:42.527 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:42 smithi134 ceph-mon[30086]: 4.5 scrub ok 2024-02-05T08:28:42.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:42 smithi134 ceph-mon[39278]: 5.f scrub starts 2024-02-05T08:28:42.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:42 smithi134 ceph-mon[39278]: 5.f scrub ok 2024-02-05T08:28:42.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:42 smithi134 ceph-mon[39278]: 4.5 scrub starts 2024-02-05T08:28:42.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:42 smithi134 ceph-mon[39278]: 4.5 scrub ok 2024-02-05T08:28:42.810 INFO:teuthology.orchestra.run.smithi134.stdout:219043332395 2024-02-05T08:28:42.810 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-05T08:28:42.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:42 smithi196 ceph-mon[33018]: 5.f scrub starts 2024-02-05T08:28:42.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:42 smithi196 ceph-mon[33018]: 5.f scrub ok 2024-02-05T08:28:42.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:42 smithi196 ceph-mon[33018]: 4.5 scrub starts 2024-02-05T08:28:42.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:42 smithi196 ceph-mon[33018]: 4.5 scrub ok 2024-02-05T08:28:43.673 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:43 smithi134 ceph-mon[30086]: pgmap v1086: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:43.673 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:43 smithi134 ceph-mon[39278]: pgmap v1086: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:43.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:43 smithi196 ceph-mon[33018]: pgmap v1086: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:44.331 INFO:teuthology.orchestra.run.smithi134.stdout:1571958030405 2024-02-05T08:28:44.331 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-05T08:28:44.605 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:44 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:28:44.605 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:44 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:28:44.605 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:44 smithi134 ceph-mon[30086]: osdmap e466: 8 total, 8 up, 8 in 2024-02-05T08:28:44.605 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:44 smithi134 ceph-mon[30086]: 5.a scrub starts 2024-02-05T08:28:44.605 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:44 smithi134 ceph-mon[30086]: 5.a scrub ok 2024-02-05T08:28:44.606 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:44 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:28:44.606 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:44 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:28:44.606 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:44 smithi134 ceph-mon[39278]: osdmap e466: 8 total, 8 up, 8 in 2024-02-05T08:28:44.606 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:44 smithi134 ceph-mon[39278]: 5.a scrub starts 2024-02-05T08:28:44.606 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:44 smithi134 ceph-mon[39278]: 5.a scrub ok 2024-02-05T08:28:44.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:28:44 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:28:44.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:44 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:28:44.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:44 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:28:44.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:44 smithi196 ceph-mon[33018]: osdmap e466: 8 total, 8 up, 8 in 2024-02-05T08:28:44.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:44 smithi196 ceph-mon[33018]: 5.a scrub starts 2024-02-05T08:28:44.716 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:44 smithi196 ceph-mon[33018]: 5.a scrub ok 2024-02-05T08:28:45.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[30086]: pgmap v1088: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:28:45.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:45.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[30086]: 3.17 scrub starts 2024-02-05T08:28:45.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[30086]: 3.17 scrub ok 2024-02-05T08:28:45.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[30086]: 5.6 scrub starts 2024-02-05T08:28:45.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[30086]: 5.6 scrub ok 2024-02-05T08:28:45.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:28:45.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:28:45.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[39278]: pgmap v1088: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:28:45.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:45.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[39278]: 3.17 scrub starts 2024-02-05T08:28:45.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[39278]: 3.17 scrub ok 2024-02-05T08:28:45.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[39278]: 5.6 scrub starts 2024-02-05T08:28:45.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[39278]: 5.6 scrub ok 2024-02-05T08:28:45.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:28:45.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:45 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:28:45.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:45 smithi196 ceph-mon[33018]: pgmap v1088: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:28:45.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:45 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:45.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:45 smithi196 ceph-mon[33018]: 3.17 scrub starts 2024-02-05T08:28:45.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:45 smithi196 ceph-mon[33018]: 3.17 scrub ok 2024-02-05T08:28:45.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:45 smithi196 ceph-mon[33018]: 5.6 scrub starts 2024-02-05T08:28:45.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:45 smithi196 ceph-mon[33018]: 5.6 scrub ok 2024-02-05T08:28:45.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:45 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:28:45.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:45 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:28:45.933 INFO:teuthology.orchestra.run.smithi134.stdout:137438953782 2024-02-05T08:28:45.933 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-05T08:28:46.485 INFO:teuthology.orchestra.run.smithi134.stdout:34359738697 2024-02-05T08:28:46.485 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-05T08:28:46.703 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:46.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[30086]: 4.18 deep-scrub starts 2024-02-05T08:28:46.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[30086]: 4.18 deep-scrub ok 2024-02-05T08:28:46.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[30086]: 3.1b scrub starts 2024-02-05T08:28:46.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[30086]: 3.1b scrub ok 2024-02-05T08:28:46.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[30086]: osdmap e467: 8 total, 8 up, 8 in 2024-02-05T08:28:46.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[30086]: 4.f scrub starts 2024-02-05T08:28:46.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[30086]: 4.f scrub ok 2024-02-05T08:28:46.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[39278]: 4.18 deep-scrub starts 2024-02-05T08:28:46.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[39278]: 4.18 deep-scrub ok 2024-02-05T08:28:46.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[39278]: 3.1b scrub starts 2024-02-05T08:28:46.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[39278]: 3.1b scrub ok 2024-02-05T08:28:46.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[39278]: osdmap e467: 8 total, 8 up, 8 in 2024-02-05T08:28:46.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[39278]: 4.f scrub starts 2024-02-05T08:28:46.778 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:46 smithi134 ceph-mon[39278]: 4.f scrub ok 2024-02-05T08:28:46.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:46 smithi196 ceph-mon[33018]: 4.18 deep-scrub starts 2024-02-05T08:28:46.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:46 smithi196 ceph-mon[33018]: 4.18 deep-scrub ok 2024-02-05T08:28:46.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:46 smithi196 ceph-mon[33018]: 3.1b scrub starts 2024-02-05T08:28:46.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:46 smithi196 ceph-mon[33018]: 3.1b scrub ok 2024-02-05T08:28:46.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:46 smithi196 ceph-mon[33018]: osdmap e467: 8 total, 8 up, 8 in 2024-02-05T08:28:46.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:46 smithi196 ceph-mon[33018]: 4.f scrub starts 2024-02-05T08:28:46.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:46 smithi196 ceph-mon[33018]: 4.f scrub ok 2024-02-05T08:28:47.150 INFO:teuthology.orchestra.run.smithi134.stdout:77309411648 2024-02-05T08:28:47.150 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-05T08:28:47.654 INFO:teuthology.orchestra.run.smithi134.stdout:180388626736 2024-02-05T08:28:47.654 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-05T08:28:47.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[30086]: pgmap v1090: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-05T08:28:47.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[30086]: 2.2 deep-scrub starts 2024-02-05T08:28:47.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[30086]: 2.2 deep-scrub ok 2024-02-05T08:28:47.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:28:47.739 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:28:47.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:28:47.740 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:28:47.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[39278]: pgmap v1090: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-05T08:28:47.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[39278]: 2.2 deep-scrub starts 2024-02-05T08:28:47.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[39278]: 2.2 deep-scrub ok 2024-02-05T08:28:47.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:28:47.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:28:47.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:28:47.740 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:28:47.814 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:47.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:47 smithi196 ceph-mon[33018]: pgmap v1090: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-02-05T08:28:47.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:47 smithi196 ceph-mon[33018]: 2.2 deep-scrub starts 2024-02-05T08:28:47.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:47 smithi196 ceph-mon[33018]: 2.2 deep-scrub ok 2024-02-05T08:28:47.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:47 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:28:47.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:47 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:28:47.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:47 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:28:47.815 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:47 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:28:47.818 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:47.951 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:48.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:28:47 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:28:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:28:48.556 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:48 smithi134 ceph-mon[30086]: 6.15 scrub starts 2024-02-05T08:28:48.556 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:48 smithi134 ceph-mon[30086]: 6.15 scrub ok 2024-02-05T08:28:48.556 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:48 smithi134 ceph-mon[39278]: 6.15 scrub starts 2024-02-05T08:28:48.556 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:48 smithi134 ceph-mon[39278]: 6.15 scrub ok 2024-02-05T08:28:48.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:48 smithi196 ceph-mon[33018]: 6.15 scrub starts 2024-02-05T08:28:48.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:48 smithi196 ceph-mon[33018]: 6.15 scrub ok 2024-02-05T08:28:49.776 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:49 smithi134 ceph-mon[30086]: pgmap v1091: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:28:49.777 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:49 smithi134 ceph-mon[39278]: pgmap v1091: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:28:49.814 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:49 smithi196 ceph-mon[33018]: pgmap v1091: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T08:28:50.632 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:50.667 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:51.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:50 smithi134 ceph-mon[30086]: pgmap v1092: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:51.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:50 smithi134 ceph-mon[30086]: 1.a scrub starts 2024-02-05T08:28:51.027 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:50 smithi134 ceph-mon[30086]: 1.a scrub ok 2024-02-05T08:28:51.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:50 smithi134 ceph-mon[39278]: pgmap v1092: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:51.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:50 smithi134 ceph-mon[39278]: 1.a scrub starts 2024-02-05T08:28:51.028 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:50 smithi134 ceph-mon[39278]: 1.a scrub ok 2024-02-05T08:28:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:50 smithi196 ceph-mon[33018]: pgmap v1092: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:50 smithi196 ceph-mon[33018]: 1.a scrub starts 2024-02-05T08:28:51.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:50 smithi196 ceph-mon[33018]: 1.a scrub ok 2024-02-05T08:28:51.524 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:52.120 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:51 smithi134 ceph-mon[39278]: 5.1f scrub starts 2024-02-05T08:28:52.120 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:51 smithi134 ceph-mon[39278]: 5.1f scrub ok 2024-02-05T08:28:52.121 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:51 smithi134 ceph-mon[39278]: 3.1f scrub starts 2024-02-05T08:28:52.121 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:51 smithi134 ceph-mon[39278]: 3.1f scrub ok 2024-02-05T08:28:52.121 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:51 smithi134 ceph-mon[30086]: 5.1f scrub starts 2024-02-05T08:28:52.121 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:51 smithi134 ceph-mon[30086]: 5.1f scrub ok 2024-02-05T08:28:52.121 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:51 smithi134 ceph-mon[30086]: 3.1f scrub starts 2024-02-05T08:28:52.121 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:51 smithi134 ceph-mon[30086]: 3.1f scrub ok 2024-02-05T08:28:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:51 smithi196 ceph-mon[33018]: 5.1f scrub starts 2024-02-05T08:28:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:51 smithi196 ceph-mon[33018]: 5.1f scrub ok 2024-02-05T08:28:52.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:51 smithi196 ceph-mon[33018]: 3.1f scrub starts 2024-02-05T08:28:52.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:51 smithi196 ceph-mon[33018]: 3.1f scrub ok 2024-02-05T08:28:53.226 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[30086]: 4.1b scrub starts 2024-02-05T08:28:53.226 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[30086]: 4.1b scrub ok 2024-02-05T08:28:53.226 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[30086]: 1.3 deep-scrub starts 2024-02-05T08:28:53.226 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[30086]: 1.3 deep-scrub ok 2024-02-05T08:28:53.226 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[30086]: 3.15 scrub starts 2024-02-05T08:28:53.226 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[30086]: 3.15 scrub ok 2024-02-05T08:28:53.226 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[30086]: pgmap v1093: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:53.226 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[30086]: 4.17 scrub starts 2024-02-05T08:28:53.227 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[30086]: 4.17 scrub ok 2024-02-05T08:28:53.227 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[39278]: 4.1b scrub starts 2024-02-05T08:28:53.227 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[39278]: 4.1b scrub ok 2024-02-05T08:28:53.227 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[39278]: 1.3 deep-scrub starts 2024-02-05T08:28:53.227 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[39278]: 1.3 deep-scrub ok 2024-02-05T08:28:53.227 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[39278]: 3.15 scrub starts 2024-02-05T08:28:53.227 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[39278]: 3.15 scrub ok 2024-02-05T08:28:53.227 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[39278]: pgmap v1093: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:53.227 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[39278]: 4.17 scrub starts 2024-02-05T08:28:53.227 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:52 smithi134 ceph-mon[39278]: 4.17 scrub ok 2024-02-05T08:28:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:52 smithi196 ceph-mon[33018]: 4.1b scrub starts 2024-02-05T08:28:53.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:52 smithi196 ceph-mon[33018]: 4.1b scrub ok 2024-02-05T08:28:53.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:52 smithi196 ceph-mon[33018]: 1.3 deep-scrub starts 2024-02-05T08:28:53.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:52 smithi196 ceph-mon[33018]: 1.3 deep-scrub ok 2024-02-05T08:28:53.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:52 smithi196 ceph-mon[33018]: 3.15 scrub starts 2024-02-05T08:28:53.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:52 smithi196 ceph-mon[33018]: 3.15 scrub ok 2024-02-05T08:28:53.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:52 smithi196 ceph-mon[33018]: pgmap v1093: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:28:53.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:52 smithi196 ceph-mon[33018]: 4.17 scrub starts 2024-02-05T08:28:53.315 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:52 smithi196 ceph-mon[33018]: 4.17 scrub ok 2024-02-05T08:28:53.907 INFO:teuthology.orchestra.run.smithi134.stdout:257698038055 2024-02-05T08:28:54.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[30086]: 3.1a scrub starts 2024-02-05T08:28:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[30086]: 3.1a scrub ok 2024-02-05T08:28:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[30086]: 4.1d scrub starts 2024-02-05T08:28:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[30086]: 4.1d scrub ok 2024-02-05T08:28:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[30086]: 6.10 scrub starts 2024-02-05T08:28:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[30086]: 6.10 scrub ok 2024-02-05T08:28:54.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3118928240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T08:28:54.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[39278]: 3.1a scrub starts 2024-02-05T08:28:54.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[39278]: 3.1a scrub ok 2024-02-05T08:28:54.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[39278]: 4.1d scrub starts 2024-02-05T08:28:54.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[39278]: 4.1d scrub ok 2024-02-05T08:28:54.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[39278]: 6.10 scrub starts 2024-02-05T08:28:54.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[39278]: 6.10 scrub ok 2024-02-05T08:28:54.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:53 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3118928240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T08:28:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:53 smithi196 ceph-mon[33018]: 3.1a scrub starts 2024-02-05T08:28:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:53 smithi196 ceph-mon[33018]: 3.1a scrub ok 2024-02-05T08:28:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:53 smithi196 ceph-mon[33018]: 4.1d scrub starts 2024-02-05T08:28:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:53 smithi196 ceph-mon[33018]: 4.1d scrub ok 2024-02-05T08:28:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:53 smithi196 ceph-mon[33018]: 6.10 scrub starts 2024-02-05T08:28:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:53 smithi196 ceph-mon[33018]: 6.10 scrub ok 2024-02-05T08:28:54.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:53 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3118928240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T08:28:54.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:28:54 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:28:55.043 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698038052 got 257698038055 for osd.7 2024-02-05T08:28:55.044 DEBUG:teuthology.parallel:result is None 2024-02-05T08:28:55.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[30086]: pgmap v1094: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:28:55.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:28:55.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[30086]: 3.8 scrub starts 2024-02-05T08:28:55.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[30086]: 3.8 scrub ok 2024-02-05T08:28:55.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[30086]: 1.8 scrub starts 2024-02-05T08:28:55.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[30086]: 1.8 scrub ok 2024-02-05T08:28:55.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[39278]: pgmap v1094: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:28:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:28:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[39278]: 3.8 scrub starts 2024-02-05T08:28:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[39278]: 3.8 scrub ok 2024-02-05T08:28:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[39278]: 1.8 scrub starts 2024-02-05T08:28:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[39278]: 1.8 scrub ok 2024-02-05T08:28:55.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:54 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:54 smithi196 ceph-mon[33018]: pgmap v1094: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:28:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:54 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:28:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:54 smithi196 ceph-mon[33018]: 3.8 scrub starts 2024-02-05T08:28:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:54 smithi196 ceph-mon[33018]: 3.8 scrub ok 2024-02-05T08:28:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:54 smithi196 ceph-mon[33018]: 1.8 scrub starts 2024-02-05T08:28:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:54 smithi196 ceph-mon[33018]: 1.8 scrub ok 2024-02-05T08:28:55.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:54 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:28:56.327 INFO:teuthology.orchestra.run.smithi134.stdout:1571958030407 2024-02-05T08:28:57.044 INFO:teuthology.orchestra.run.smithi134.stdout:137438953784 2024-02-05T08:28:57.261 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:28:57.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[30086]: pgmap v1095: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-02-05T08:28:57.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[30086]: 4.1 scrub starts 2024-02-05T08:28:57.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[30086]: 4.1 scrub ok 2024-02-05T08:28:57.276 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[30086]: 3.11 scrub starts 2024-02-05T08:28:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[30086]: 3.11 scrub ok 2024-02-05T08:28:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/2283575300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T08:28:57.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4063905438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T08:28:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[39278]: pgmap v1095: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-02-05T08:28:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[39278]: 4.1 scrub starts 2024-02-05T08:28:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[39278]: 4.1 scrub ok 2024-02-05T08:28:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[39278]: 3.11 scrub starts 2024-02-05T08:28:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[39278]: 3.11 scrub ok 2024-02-05T08:28:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/2283575300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T08:28:57.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4063905438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T08:28:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:57 smithi196 ceph-mon[33018]: pgmap v1095: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 967 B/s rd, 0 op/s 2024-02-05T08:28:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:57 smithi196 ceph-mon[33018]: 4.1 scrub starts 2024-02-05T08:28:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:57 smithi196 ceph-mon[33018]: 4.1 scrub ok 2024-02-05T08:28:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:57 smithi196 ceph-mon[33018]: 3.11 scrub starts 2024-02-05T08:28:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:57 smithi196 ceph-mon[33018]: 3.11 scrub ok 2024-02-05T08:28:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:57 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/2283575300' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T08:28:57.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:57 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4063905438' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T08:28:58.027 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:28:57 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:28:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:28:58.293 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953782 got 137438953784 for osd.4 2024-02-05T08:28:58.294 DEBUG:teuthology.parallel:result is None 2024-02-05T08:28:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:58 smithi196 ceph-mon[33018]: 2.0 scrub starts 2024-02-05T08:28:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:58 smithi196 ceph-mon[33018]: 2.0 scrub ok 2024-02-05T08:28:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:58 smithi196 ceph-mon[33018]: 4.d scrub starts 2024-02-05T08:28:58.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:58 smithi196 ceph-mon[33018]: 4.d scrub ok 2024-02-05T08:28:58.388 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:58 smithi134 ceph-mon[30086]: 2.0 scrub starts 2024-02-05T08:28:58.388 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:58 smithi134 ceph-mon[30086]: 2.0 scrub ok 2024-02-05T08:28:58.388 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:58 smithi134 ceph-mon[30086]: 4.d scrub starts 2024-02-05T08:28:58.388 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:58 smithi134 ceph-mon[30086]: 4.d scrub ok 2024-02-05T08:28:58.389 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:58 smithi134 ceph-mon[39278]: 2.0 scrub starts 2024-02-05T08:28:58.389 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:58 smithi134 ceph-mon[39278]: 2.0 scrub ok 2024-02-05T08:28:58.389 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:58 smithi134 ceph-mon[39278]: 4.d scrub starts 2024-02-05T08:28:58.389 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:58 smithi134 ceph-mon[39278]: 4.d scrub ok 2024-02-05T08:28:59.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:59 smithi134 ceph-mon[30086]: pgmap v1096: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:59.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:59 smithi134 ceph-mon[30086]: 4.10 scrub starts 2024-02-05T08:28:59.396 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:59 smithi134 ceph-mon[30086]: 4.10 scrub ok 2024-02-05T08:28:59.396 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:59 smithi134 ceph-mon[30086]: 5.8 scrub starts 2024-02-05T08:28:59.396 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:28:59 smithi134 ceph-mon[30086]: 5.8 scrub ok 2024-02-05T08:28:59.396 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:59 smithi134 ceph-mon[39278]: pgmap v1096: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:59.396 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:59 smithi134 ceph-mon[39278]: 4.10 scrub starts 2024-02-05T08:28:59.396 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:59 smithi134 ceph-mon[39278]: 4.10 scrub ok 2024-02-05T08:28:59.396 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:59 smithi134 ceph-mon[39278]: 5.8 scrub starts 2024-02-05T08:28:59.396 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:28:59 smithi134 ceph-mon[39278]: 5.8 scrub ok 2024-02-05T08:28:59.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:59 smithi196 ceph-mon[33018]: pgmap v1096: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:28:59.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:59 smithi196 ceph-mon[33018]: 4.10 scrub starts 2024-02-05T08:28:59.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:59 smithi196 ceph-mon[33018]: 4.10 scrub ok 2024-02-05T08:28:59.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:59 smithi196 ceph-mon[33018]: 5.8 scrub starts 2024-02-05T08:28:59.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:28:59 smithi196 ceph-mon[33018]: 5.8 scrub ok 2024-02-05T08:28:59.580 INFO:teuthology.orchestra.run.smithi134.stdout:77309411650 2024-02-05T08:29:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[30086]: 4.2 deep-scrub starts 2024-02-05T08:29:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[30086]: 4.2 deep-scrub ok 2024-02-05T08:29:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[30086]: 6.e scrub starts 2024-02-05T08:29:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[30086]: 3.12 scrub starts 2024-02-05T08:29:00.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[30086]: 3.12 scrub ok 2024-02-05T08:29:00.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3540852282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T08:29:00.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3000980485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T08:29:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[39278]: 4.2 deep-scrub starts 2024-02-05T08:29:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[39278]: 4.2 deep-scrub ok 2024-02-05T08:29:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[39278]: 6.e scrub starts 2024-02-05T08:29:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[39278]: 3.12 scrub starts 2024-02-05T08:29:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[39278]: 3.12 scrub ok 2024-02-05T08:29:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3540852282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T08:29:00.278 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:00 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3000980485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T08:29:00.414 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411648 got 77309411650 for osd.2 2024-02-05T08:29:00.415 DEBUG:teuthology.parallel:result is None 2024-02-05T08:29:00.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:00 smithi196 ceph-mon[33018]: 4.2 deep-scrub starts 2024-02-05T08:29:00.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:00 smithi196 ceph-mon[33018]: 4.2 deep-scrub ok 2024-02-05T08:29:00.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:00 smithi196 ceph-mon[33018]: 6.e scrub starts 2024-02-05T08:29:00.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:00 smithi196 ceph-mon[33018]: 3.12 scrub starts 2024-02-05T08:29:00.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:00 smithi196 ceph-mon[33018]: 3.12 scrub ok 2024-02-05T08:29:00.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:00 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3540852282' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T08:29:00.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:00 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3000980485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T08:29:00.620 INFO:teuthology.orchestra.run.smithi134.stdout:219043332398 2024-02-05T08:29:01.137 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:01 smithi134 ceph-mon[39278]: 6.e scrub ok 2024-02-05T08:29:01.137 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:01 smithi134 ceph-mon[39278]: pgmap v1097: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:29:01.137 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:01 smithi134 ceph-mon[39278]: 4.13 scrub starts 2024-02-05T08:29:01.137 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:01 smithi134 ceph-mon[39278]: 4.13 scrub ok 2024-02-05T08:29:01.138 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:01 smithi134 ceph-mon[30086]: 6.e scrub ok 2024-02-05T08:29:01.138 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:01 smithi134 ceph-mon[30086]: pgmap v1097: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:29:01.138 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:01 smithi134 ceph-mon[30086]: 4.13 scrub starts 2024-02-05T08:29:01.138 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:01 smithi134 ceph-mon[30086]: 4.13 scrub ok 2024-02-05T08:29:01.139 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332395 got 219043332398 for osd.6 2024-02-05T08:29:01.139 DEBUG:teuthology.parallel:result is None 2024-02-05T08:29:01.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:01 smithi196 ceph-mon[33018]: 6.e scrub ok 2024-02-05T08:29:01.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:01 smithi196 ceph-mon[33018]: pgmap v1097: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:29:01.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:01 smithi196 ceph-mon[33018]: 4.13 scrub starts 2024-02-05T08:29:01.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:01 smithi196 ceph-mon[33018]: 4.13 scrub ok 2024-02-05T08:29:02.365 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:02 smithi134 ceph-mon[30086]: 6.1a scrub starts 2024-02-05T08:29:02.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:02 smithi134 ceph-mon[30086]: 3.0 scrub starts 2024-02-05T08:29:02.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:02 smithi134 ceph-mon[30086]: 3.0 scrub ok 2024-02-05T08:29:02.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:02 smithi134 ceph-mon[30086]: 6.7 scrub starts 2024-02-05T08:29:02.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:02 smithi134 ceph-mon[30086]: 6.7 scrub ok 2024-02-05T08:29:02.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:02 smithi134 ceph-mon[39278]: 6.1a scrub starts 2024-02-05T08:29:02.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:02 smithi134 ceph-mon[39278]: 3.0 scrub starts 2024-02-05T08:29:02.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:02 smithi134 ceph-mon[39278]: 3.0 scrub ok 2024-02-05T08:29:02.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:02 smithi134 ceph-mon[39278]: 6.7 scrub starts 2024-02-05T08:29:02.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:02 smithi134 ceph-mon[39278]: 6.7 scrub ok 2024-02-05T08:29:02.480 INFO:teuthology.orchestra.run.smithi134.stdout:115964117310 2024-02-05T08:29:02.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:02 smithi196 ceph-mon[33018]: 6.1a scrub starts 2024-02-05T08:29:02.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:02 smithi196 ceph-mon[33018]: 3.0 scrub starts 2024-02-05T08:29:02.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:02 smithi196 ceph-mon[33018]: 3.0 scrub ok 2024-02-05T08:29:02.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:02 smithi196 ceph-mon[33018]: 6.7 scrub starts 2024-02-05T08:29:02.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:02 smithi196 ceph-mon[33018]: 6.7 scrub ok 2024-02-05T08:29:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:03 smithi134 ceph-mon[30086]: 6.1a scrub ok 2024-02-05T08:29:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:03 smithi134 ceph-mon[30086]: pgmap v1098: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:29:03.277 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:03 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/3696430526' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T08:29:03.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:03 smithi134 ceph-mon[39278]: 6.1a scrub ok 2024-02-05T08:29:03.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:03 smithi134 ceph-mon[39278]: pgmap v1098: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:29:03.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:03 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/3696430526' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T08:29:03.308 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1571958030405 got 1571958030407 for osd.1 2024-02-05T08:29:03.308 DEBUG:teuthology.parallel:result is None 2024-02-05T08:29:03.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:03 smithi196 ceph-mon[33018]: 6.1a scrub ok 2024-02-05T08:29:03.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:03 smithi196 ceph-mon[33018]: pgmap v1098: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:29:03.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:03 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/3696430526' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T08:29:04.011 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117305 got 115964117310 for osd.3 2024-02-05T08:29:04.011 DEBUG:teuthology.parallel:result is None 2024-02-05T08:29:04.080 INFO:teuthology.orchestra.run.smithi134.stdout:34359738701 2024-02-05T08:29:04.521 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:04 smithi134 ceph-mon[30086]: 5.12 scrub starts 2024-02-05T08:29:04.521 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:04 smithi134 ceph-mon[30086]: 5.12 scrub ok 2024-02-05T08:29:04.521 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:04 smithi134 ceph-mon[30086]: 5.1a scrub starts 2024-02-05T08:29:04.521 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:04 smithi134 ceph-mon[30086]: 5.1a scrub ok 2024-02-05T08:29:04.521 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:04 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/175979429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T08:29:04.522 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:04 smithi134 ceph-mon[39278]: 5.12 scrub starts 2024-02-05T08:29:04.522 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:04 smithi134 ceph-mon[39278]: 5.12 scrub ok 2024-02-05T08:29:04.522 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:04 smithi134 ceph-mon[39278]: 5.1a scrub starts 2024-02-05T08:29:04.522 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:04 smithi134 ceph-mon[39278]: 5.1a scrub ok 2024-02-05T08:29:04.522 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:04 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/175979429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T08:29:04.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:29:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:29:04.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:04 smithi196 ceph-mon[33018]: 5.12 scrub starts 2024-02-05T08:29:04.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:04 smithi196 ceph-mon[33018]: 5.12 scrub ok 2024-02-05T08:29:04.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:04 smithi196 ceph-mon[33018]: 5.1a scrub starts 2024-02-05T08:29:04.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:04 smithi196 ceph-mon[33018]: 5.1a scrub ok 2024-02-05T08:29:04.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:04 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/175979429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T08:29:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:05 smithi134 ceph-mon[30086]: 6.4 scrub starts 2024-02-05T08:29:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:05 smithi134 ceph-mon[30086]: 6.4 scrub ok 2024-02-05T08:29:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:05 smithi134 ceph-mon[30086]: pgmap v1099: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:29:05.428 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:05 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:29:05.428 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:05 smithi134 ceph-mon[39278]: 6.4 scrub starts 2024-02-05T08:29:05.428 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:05 smithi134 ceph-mon[39278]: 6.4 scrub ok 2024-02-05T08:29:05.428 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:05 smithi134 ceph-mon[39278]: pgmap v1099: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:29:05.428 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:05 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:29:05.456 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738697 got 34359738701 for osd.0 2024-02-05T08:29:05.456 DEBUG:teuthology.parallel:result is None 2024-02-05T08:29:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:05 smithi196 ceph-mon[33018]: 6.4 scrub starts 2024-02-05T08:29:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:05 smithi196 ceph-mon[33018]: 6.4 scrub ok 2024-02-05T08:29:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:05 smithi196 ceph-mon[33018]: pgmap v1099: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:29:05.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:05 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:29:05.958 INFO:teuthology.orchestra.run.smithi134.stdout:180388626740 2024-02-05T08:29:06.526 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:06 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/837062686' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T08:29:06.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:06 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/837062686' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T08:29:06.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:06 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/837062686' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T08:29:06.668 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626736 got 180388626740 for osd.5 2024-02-05T08:29:06.668 DEBUG:teuthology.parallel:result is None 2024-02-05T08:29:06.668 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-05T08:29:06.668 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:29:06.911 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:29:07.320 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[30086]: pgmap v1100: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:29:07.320 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[30086]: 3.3 scrub starts 2024-02-05T08:29:07.321 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[30086]: 3.3 scrub ok 2024-02-05T08:29:07.321 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[30086]: 3.d scrub starts 2024-02-05T08:29:07.321 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[30086]: 3.d scrub ok 2024-02-05T08:29:07.321 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[30086]: 6.b scrub starts 2024-02-05T08:29:07.321 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[30086]: 6.b scrub ok 2024-02-05T08:29:07.321 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[30086]: 5.d scrub starts 2024-02-05T08:29:07.321 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[30086]: 5.d scrub ok 2024-02-05T08:29:07.321 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[30086]: 6.16 scrub starts 2024-02-05T08:29:07.321 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[30086]: 6.16 scrub ok 2024-02-05T08:29:07.321 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[39278]: pgmap v1100: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:29:07.321 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[39278]: 3.3 scrub starts 2024-02-05T08:29:07.322 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[39278]: 3.3 scrub ok 2024-02-05T08:29:07.322 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[39278]: 3.d scrub starts 2024-02-05T08:29:07.322 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[39278]: 3.d scrub ok 2024-02-05T08:29:07.322 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[39278]: 6.b scrub starts 2024-02-05T08:29:07.322 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[39278]: 6.b scrub ok 2024-02-05T08:29:07.322 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[39278]: 5.d scrub starts 2024-02-05T08:29:07.322 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[39278]: 5.d scrub ok 2024-02-05T08:29:07.322 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[39278]: 6.16 scrub starts 2024-02-05T08:29:07.322 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-mon[39278]: 6.16 scrub ok 2024-02-05T08:29:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:07 smithi196 ceph-mon[33018]: pgmap v1100: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:29:07.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:07 smithi196 ceph-mon[33018]: 3.3 scrub starts 2024-02-05T08:29:07.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:07 smithi196 ceph-mon[33018]: 3.3 scrub ok 2024-02-05T08:29:07.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:07 smithi196 ceph-mon[33018]: 3.d scrub starts 2024-02-05T08:29:07.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:07 smithi196 ceph-mon[33018]: 3.d scrub ok 2024-02-05T08:29:07.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:07 smithi196 ceph-mon[33018]: 6.b scrub starts 2024-02-05T08:29:07.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:07 smithi196 ceph-mon[33018]: 6.b scrub ok 2024-02-05T08:29:07.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:07 smithi196 ceph-mon[33018]: 5.d scrub starts 2024-02-05T08:29:07.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:07 smithi196 ceph-mon[33018]: 5.d scrub ok 2024-02-05T08:29:07.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:07 smithi196 ceph-mon[33018]: 6.16 scrub starts 2024-02-05T08:29:07.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:07 smithi196 ceph-mon[33018]: 6.16 scrub ok 2024-02-05T08:29:08.026 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:07 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:29:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:29:08.450 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:08 smithi134 ceph-mon[30086]: 5.16 scrub starts 2024-02-05T08:29:08.450 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:08 smithi134 ceph-mon[30086]: 5.16 scrub ok 2024-02-05T08:29:08.450 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:08 smithi134 ceph-mon[39278]: 5.16 scrub starts 2024-02-05T08:29:08.450 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:08 smithi134 ceph-mon[39278]: 5.16 scrub ok 2024-02-05T08:29:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:08 smithi196 ceph-mon[33018]: 5.16 scrub starts 2024-02-05T08:29:08.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:08 smithi196 ceph-mon[33018]: 5.16 scrub ok 2024-02-05T08:29:09.007 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:29:09.010 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:29:09.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:09 smithi134 ceph-mon[30086]: pgmap v1101: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:29:09.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:09 smithi134 ceph-mon[30086]: 3.f scrub starts 2024-02-05T08:29:09.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:09 smithi134 ceph-mon[30086]: 3.f scrub ok 2024-02-05T08:29:09.366 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:09 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:29:09.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:09 smithi134 ceph-mon[39278]: pgmap v1101: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:29:09.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:09 smithi134 ceph-mon[39278]: 3.f scrub starts 2024-02-05T08:29:09.366 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:09 smithi134 ceph-mon[39278]: 3.f scrub ok 2024-02-05T08:29:09.367 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:09 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:29:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:09 smithi196 ceph-mon[33018]: pgmap v1101: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T08:29:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:09 smithi196 ceph-mon[33018]: 3.f scrub starts 2024-02-05T08:29:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:09 smithi196 ceph-mon[33018]: 3.f scrub ok 2024-02-05T08:29:09.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:09 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T08:29:09.670 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":1101,"stamp":"2024-02-05T08:29:08.011316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2992,"num_read_kb":2744,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":12066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2636,"ondisk_log_size":2636,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5567068,"kb_used_data":7580,"kb_used_omap":15,"kb_used_meta":5559408,"kb_avail":744394148,"statfs":{"total":767960285184,"available":762259607552,"internally_reserved":0,"allocated":7761920,"data_stored":5245764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15596,"internal_metadata":5692834580},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003693"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":896,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.125062+0000","last_change":"2024-02-05T08:29:01.125104+0000","last_active":"2024-02-05T08:29:01.125062+0000","last_peered":"2024-02-05T08:29:01.125062+0000","last_clean":"2024-02-05T08:29:01.125062+0000","last_became_active":"2024-02-05T08:23:07.931642+0000","last_became_peered":"2024-02-05T08:23:07.931642+0000","last_unstale":"2024-02-05T08:29:01.125062+0000","last_undegraded":"2024-02-05T08:29:01.125062+0000","last_fullsized":"2024-02-05T08:29:01.125062+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:01.124974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:29:01.124974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:04.280969+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":696,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.433766+0000","last_change":"2024-02-05T08:28:24.058083+0000","last_active":"2024-02-05T08:28:45.433766+0000","last_peered":"2024-02-05T08:28:45.433766+0000","last_clean":"2024-02-05T08:28:45.433766+0000","last_became_active":"2024-02-05T08:28:24.057847+0000","last_became_peered":"2024-02-05T08:28:24.057847+0000","last_unstale":"2024-02-05T08:28:45.433766+0000","last_undegraded":"2024-02-05T08:28:45.433766+0000","last_fullsized":"2024-02-05T08:28:45.433766+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:50.062807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:27:50.062807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:08.748754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:51.465632+0000","last_change":"2024-02-05T08:28:51.465653+0000","last_active":"2024-02-05T08:28:51.465632+0000","last_peered":"2024-02-05T08:28:51.465632+0000","last_clean":"2024-02-05T08:28:51.465632+0000","last_became_active":"2024-02-05T08:25:49.142192+0000","last_became_peered":"2024-02-05T08:25:49.142192+0000","last_unstale":"2024-02-05T08:28:51.465632+0000","last_undegraded":"2024-02-05T08:28:51.465632+0000","last_fullsized":"2024-02-05T08:28:51.465632+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:51.465566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:28:51.465566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:04.377005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"397'25","reported_seq":967,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735128+0000","last_change":"2024-02-05T08:28:45.204298+0000","last_active":"2024-02-05T08:28:45.735128+0000","last_peered":"2024-02-05T08:28:45.735128+0000","last_clean":"2024-02-05T08:28:45.735128+0000","last_became_active":"2024-02-05T08:23:07.937041+0000","last_became_peered":"2024-02-05T08:23:07.937041+0000","last_unstale":"2024-02-05T08:28:45.735128+0000","last_undegraded":"2024-02-05T08:28:45.735128+0000","last_fullsized":"2024-02-05T08:28:45.735128+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-02-05T08:28:45.204152+0000","last_deep_scrub":"397'25","last_deep_scrub_stamp":"2024-02-05T08:28:45.204152+0000","last_clean_scrub_stamp":"2024-02-05T08:28:45.204152+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:51.623766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":391,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733280+0000","last_change":"2024-02-05T08:27:23.238054+0000","last_active":"2024-02-05T08:28:45.733280+0000","last_peered":"2024-02-05T08:28:45.733280+0000","last_clean":"2024-02-05T08:28:45.733280+0000","last_became_active":"2024-02-05T08:18:34.633083+0000","last_became_peered":"2024-02-05T08:18:34.633083+0000","last_unstale":"2024-02-05T08:28:45.733280+0000","last_undegraded":"2024-02-05T08:28:45.733280+0000","last_fullsized":"2024-02-05T08:28:45.733280+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:23.237950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:27:23.237950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":905,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735296+0000","last_change":"2024-02-05T08:28:23.102978+0000","last_active":"2024-02-05T08:28:45.735296+0000","last_peered":"2024-02-05T08:28:45.735296+0000","last_clean":"2024-02-05T08:28:45.735296+0000","last_became_active":"2024-02-05T08:23:07.937180+0000","last_became_peered":"2024-02-05T08:23:07.937180+0000","last_unstale":"2024-02-05T08:28:45.735296+0000","last_undegraded":"2024-02-05T08:28:45.735296+0000","last_fullsized":"2024-02-05T08:28:45.735296+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:23.102838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:41.460468+0000","last_clean_scrub_stamp":"2024-02-05T08:28:23.102838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:42.009179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":401,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733658+0000","last_change":"2024-02-05T08:27:39.196102+0000","last_active":"2024-02-05T08:28:45.733658+0000","last_peered":"2024-02-05T08:28:45.733658+0000","last_clean":"2024-02-05T08:28:45.733658+0000","last_became_active":"2024-02-05T08:21:41.880038+0000","last_became_peered":"2024-02-05T08:21:41.880038+0000","last_unstale":"2024-02-05T08:28:45.733658+0000","last_undegraded":"2024-02-05T08:28:45.733658+0000","last_fullsized":"2024-02-05T08:28:45.733658+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:39.196031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:39.196031+0000","last_clean_scrub_stamp":"2024-02-05T08:27:39.196031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:02.870660+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"397'46","reported_seq":477,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.732973+0000","last_change":"2024-02-05T08:27:45.248656+0000","last_active":"2024-02-05T08:28:45.732973+0000","last_peered":"2024-02-05T08:28:45.732973+0000","last_clean":"2024-02-05T08:28:45.732973+0000","last_became_active":"2024-02-05T08:25:49.140432+0000","last_became_peered":"2024-02-05T08:25:49.140432+0000","last_unstale":"2024-02-05T08:28:45.732973+0000","last_undegraded":"2024-02-05T08:28:45.732973+0000","last_fullsized":"2024-02-05T08:28:45.732973+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-02-05T08:27:45.248551+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:27:45.248551+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:09.415834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434897+0000","last_change":"2024-02-05T08:28:01.438689+0000","last_active":"2024-02-05T08:28:45.434897+0000","last_peered":"2024-02-05T08:28:45.434897+0000","last_clean":"2024-02-05T08:28:45.434897+0000","last_became_active":"2024-02-05T08:25:49.139935+0000","last_became_peered":"2024-02-05T08:25:49.139935+0000","last_unstale":"2024-02-05T08:28:45.434897+0000","last_undegraded":"2024-02-05T08:28:45.434897+0000","last_fullsized":"2024-02-05T08:28:45.434897+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:01.438584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:01.438584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:28.393732+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":919,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429724+0000","last_change":"2024-02-05T08:28:00.828354+0000","last_active":"2024-02-05T08:28:45.429724+0000","last_peered":"2024-02-05T08:28:45.429724+0000","last_clean":"2024-02-05T08:28:45.429724+0000","last_became_active":"2024-02-05T08:25:49.116678+0000","last_became_peered":"2024-02-05T08:25:49.116678+0000","last_unstale":"2024-02-05T08:28:45.429724+0000","last_undegraded":"2024-02-05T08:28:45.429724+0000","last_fullsized":"2024-02-05T08:28:45.429724+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:00.828248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:02.192889+0000","last_clean_scrub_stamp":"2024-02-05T08:28:00.828248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:23.104062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":851,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.430025+0000","last_change":"2024-02-05T08:28:13.604842+0000","last_active":"2024-02-05T08:28:45.430025+0000","last_peered":"2024-02-05T08:28:45.430025+0000","last_clean":"2024-02-05T08:28:45.430025+0000","last_became_active":"2024-02-05T08:28:02.684565+0000","last_became_peered":"2024-02-05T08:28:02.684565+0000","last_unstale":"2024-02-05T08:28:45.430025+0000","last_undegraded":"2024-02-05T08:28:45.430025+0000","last_fullsized":"2024-02-05T08:28:45.430025+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:13.604741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_clean_scrub_stamp":"2024-02-05T08:28:13.604741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:15.015055+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"397'25","reported_seq":1003,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735541+0000","last_change":"2024-02-05T08:27:59.963350+0000","last_active":"2024-02-05T08:28:45.735541+0000","last_peered":"2024-02-05T08:28:45.735541+0000","last_clean":"2024-02-05T08:28:45.735541+0000","last_became_active":"2024-02-05T08:25:49.126860+0000","last_became_peered":"2024-02-05T08:25:49.126860+0000","last_unstale":"2024-02-05T08:28:45.735541+0000","last_undegraded":"2024-02-05T08:28:45.735541+0000","last_fullsized":"2024-02-05T08:28:45.735541+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-02-05T08:27:59.963221+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:27:59.963221+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:10.426288+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":909,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429202+0000","last_change":"2024-02-05T08:28:28.770061+0000","last_active":"2024-02-05T08:28:45.429202+0000","last_peered":"2024-02-05T08:28:45.429202+0000","last_clean":"2024-02-05T08:28:45.429202+0000","last_became_active":"2024-02-05T08:23:07.932548+0000","last_became_peered":"2024-02-05T08:23:07.932548+0000","last_unstale":"2024-02-05T08:28:45.429202+0000","last_undegraded":"2024-02-05T08:28:45.429202+0000","last_fullsized":"2024-02-05T08:28:45.429202+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:28.769944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:28.769944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:49.467708+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":909,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:03.064061+0000","last_change":"2024-02-05T08:29:03.064085+0000","last_active":"2024-02-05T08:29:03.064061+0000","last_peered":"2024-02-05T08:29:03.064061+0000","last_clean":"2024-02-05T08:29:03.064061+0000","last_became_active":"2024-02-05T08:23:07.939630+0000","last_became_peered":"2024-02-05T08:23:07.939630+0000","last_unstale":"2024-02-05T08:29:03.064061+0000","last_undegraded":"2024-02-05T08:29:03.064061+0000","last_fullsized":"2024-02-05T08:29:03.064061+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:03.063995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:29.345593+0000","last_clean_scrub_stamp":"2024-02-05T08:29:03.063995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:10.318219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":832,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429921+0000","last_change":"2024-02-05T08:28:02.686352+0000","last_active":"2024-02-05T08:28:45.429921+0000","last_peered":"2024-02-05T08:28:45.429921+0000","last_clean":"2024-02-05T08:28:45.429921+0000","last_became_active":"2024-02-05T08:28:02.685758+0000","last_became_peered":"2024-02-05T08:28:02.685758+0000","last_unstale":"2024-02-05T08:28:45.429921+0000","last_undegraded":"2024-02-05T08:28:45.429921+0000","last_fullsized":"2024-02-05T08:28:45.429921+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:51.008413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:51.008413+0000","last_clean_scrub_stamp":"2024-02-05T08:27:51.008413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:03.625376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"397'9","reported_seq":936,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:51.122485+0000","last_change":"2024-02-05T08:28:51.122531+0000","last_active":"2024-02-05T08:28:51.122485+0000","last_peered":"2024-02-05T08:28:51.122485+0000","last_clean":"2024-02-05T08:28:51.122485+0000","last_became_active":"2024-02-05T08:23:07.938087+0000","last_became_peered":"2024-02-05T08:23:07.938087+0000","last_unstale":"2024-02-05T08:28:51.122485+0000","last_undegraded":"2024-02-05T08:28:51.122485+0000","last_fullsized":"2024-02-05T08:28:51.122485+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'9","last_scrub_stamp":"2024-02-05T08:28:51.122405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:51.122405+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:05.292307+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":902,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.736116+0000","last_change":"2024-02-05T08:27:41.138216+0000","last_active":"2024-02-05T08:28:45.736116+0000","last_peered":"2024-02-05T08:28:45.736116+0000","last_clean":"2024-02-05T08:28:45.736116+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:28:45.736116+0000","last_undegraded":"2024-02-05T08:28:45.736116+0000","last_fullsized":"2024-02-05T08:28:45.736116+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:41.138106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:41.138106+0000","last_clean_scrub_stamp":"2024-02-05T08:27:41.138106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:01.617131+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":759,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434552+0000","last_change":"2024-02-05T08:28:36.515091+0000","last_active":"2024-02-05T08:28:45.434552+0000","last_peered":"2024-02-05T08:28:45.434552+0000","last_clean":"2024-02-05T08:28:45.434552+0000","last_became_active":"2024-02-05T08:28:23.479495+0000","last_became_peered":"2024-02-05T08:28:23.479495+0000","last_unstale":"2024-02-05T08:28:45.434552+0000","last_undegraded":"2024-02-05T08:28:45.434552+0000","last_fullsized":"2024-02-05T08:28:45.434552+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:36.515007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_clean_scrub_stamp":"2024-02-05T08:28:36.515007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:05.800487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435368+0000","last_change":"2024-02-05T08:28:45.430239+0000","last_active":"2024-02-05T08:28:45.435368+0000","last_peered":"2024-02-05T08:28:45.435368+0000","last_clean":"2024-02-05T08:28:45.435368+0000","last_became_active":"2024-02-05T08:25:49.124061+0000","last_became_peered":"2024-02-05T08:25:49.124061+0000","last_unstale":"2024-02-05T08:28:45.435368+0000","last_undegraded":"2024-02-05T08:28:45.435368+0000","last_fullsized":"2024-02-05T08:28:45.435368+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:45.430141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:31.189113+0000","last_clean_scrub_stamp":"2024-02-05T08:28:45.430141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:50.163064+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"397'46","reported_seq":938,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005735+0000","last_change":"2024-02-05T08:28:06.665664+0000","last_active":"2024-02-05T08:28:46.005735+0000","last_peered":"2024-02-05T08:28:46.005735+0000","last_clean":"2024-02-05T08:28:46.005735+0000","last_became_active":"2024-02-05T08:27:39.135175+0000","last_became_peered":"2024-02-05T08:27:39.135175+0000","last_unstale":"2024-02-05T08:28:46.005735+0000","last_undegraded":"2024-02-05T08:28:46.005735+0000","last_fullsized":"2024-02-05T08:28:46.005735+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-02-05T08:28:06.665571+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:28:06.665571+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:26.499745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":392,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733848+0000","last_change":"2024-02-05T08:27:41.377827+0000","last_active":"2024-02-05T08:28:45.733848+0000","last_peered":"2024-02-05T08:28:45.733848+0000","last_clean":"2024-02-05T08:28:45.733848+0000","last_became_active":"2024-02-05T08:18:34.642494+0000","last_became_peered":"2024-02-05T08:18:34.642494+0000","last_unstale":"2024-02-05T08:28:45.733848+0000","last_undegraded":"2024-02-05T08:28:45.733848+0000","last_fullsized":"2024-02-05T08:28:45.733848+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:41.377728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_clean_scrub_stamp":"2024-02-05T08:27:41.377728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":911,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735804+0000","last_change":"2024-02-05T08:28:26.075634+0000","last_active":"2024-02-05T08:28:45.735804+0000","last_peered":"2024-02-05T08:28:45.735804+0000","last_clean":"2024-02-05T08:28:45.735804+0000","last_became_active":"2024-02-05T08:18:34.638652+0000","last_became_peered":"2024-02-05T08:18:34.638652+0000","last_unstale":"2024-02-05T08:28:45.735804+0000","last_undegraded":"2024-02-05T08:28:45.735804+0000","last_fullsized":"2024-02-05T08:28:45.735804+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:26.074066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:28:26.074066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:34.926112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":900,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:52.139686+0000","last_change":"2024-02-05T08:28:52.139721+0000","last_active":"2024-02-05T08:28:52.139686+0000","last_peered":"2024-02-05T08:28:52.139686+0000","last_clean":"2024-02-05T08:28:52.139686+0000","last_became_active":"2024-02-05T08:23:07.937958+0000","last_became_peered":"2024-02-05T08:23:07.937958+0000","last_unstale":"2024-02-05T08:28:52.139686+0000","last_undegraded":"2024-02-05T08:28:52.139686+0000","last_fullsized":"2024-02-05T08:28:52.139686+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:52.139603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:28:52.139603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:12.183850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"397'29","reported_seq":413,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:53.143327+0000","last_change":"2024-02-05T08:28:53.143359+0000","last_active":"2024-02-05T08:28:53.143327+0000","last_peered":"2024-02-05T08:28:53.143327+0000","last_clean":"2024-02-05T08:28:53.143327+0000","last_became_active":"2024-02-05T08:23:07.925083+0000","last_became_peered":"2024-02-05T08:23:07.925083+0000","last_unstale":"2024-02-05T08:28:53.143327+0000","last_undegraded":"2024-02-05T08:28:53.143327+0000","last_fullsized":"2024-02-05T08:28:53.143327+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'29","last_scrub_stamp":"2024-02-05T08:28:53.143267+0000","last_deep_scrub":"397'29","last_deep_scrub_stamp":"2024-02-05T08:26:43.239403+0000","last_clean_scrub_stamp":"2024-02-05T08:28:53.143267+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:10.454239+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":906,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006870+0000","last_change":"2024-02-05T08:27:51.110610+0000","last_active":"2024-02-05T08:28:46.006870+0000","last_peered":"2024-02-05T08:28:46.006870+0000","last_clean":"2024-02-05T08:28:46.006870+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:28:46.006870+0000","last_undegraded":"2024-02-05T08:28:46.006870+0000","last_fullsized":"2024-02-05T08:28:46.006870+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:27:51.110518+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:27:51.110518+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:18.147023+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":865,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.869936+0000","last_change":"2024-02-05T08:28:51.003498+0000","last_active":"2024-02-05T08:28:56.869936+0000","last_peered":"2024-02-05T08:28:56.869936+0000","last_clean":"2024-02-05T08:28:56.869936+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:28:56.869936+0000","last_undegraded":"2024-02-05T08:28:56.869936+0000","last_fullsized":"2024-02-05T08:28:56.869936+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:28:51.003393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:28:51.003393+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:07.112138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":834,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.433899+0000","last_change":"2024-02-05T08:27:54.409095+0000","last_active":"2024-02-05T08:28:45.433899+0000","last_peered":"2024-02-05T08:28:45.433899+0000","last_clean":"2024-02-05T08:28:45.433899+0000","last_became_active":"2024-02-05T08:23:07.940135+0000","last_became_peered":"2024-02-05T08:23:07.940135+0000","last_unstale":"2024-02-05T08:28:45.433899+0000","last_undegraded":"2024-02-05T08:28:45.433899+0000","last_fullsized":"2024-02-05T08:28:45.433899+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:27:54.408999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:27:54.408999+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:01.161930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"397'30","reported_seq":160,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435135+0000","last_change":"2024-02-05T08:28:09.466883+0000","last_active":"2024-02-05T08:28:45.435135+0000","last_peered":"2024-02-05T08:28:45.435135+0000","last_clean":"2024-02-05T08:28:45.435135+0000","last_became_active":"2024-02-05T08:25:49.130322+0000","last_became_peered":"2024-02-05T08:25:49.130322+0000","last_unstale":"2024-02-05T08:28:45.435135+0000","last_undegraded":"2024-02-05T08:28:45.435135+0000","last_fullsized":"2024-02-05T08:28:45.435135+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-02-05T08:28:09.466779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:09.466779+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:35.992366+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":708,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434649+0000","last_change":"2024-02-05T08:28:38.625974+0000","last_active":"2024-02-05T08:28:45.434649+0000","last_peered":"2024-02-05T08:28:45.434649+0000","last_clean":"2024-02-05T08:28:45.434649+0000","last_became_active":"2024-02-05T08:28:23.351251+0000","last_became_peered":"2024-02-05T08:28:23.351251+0000","last_unstale":"2024-02-05T08:28:45.434649+0000","last_undegraded":"2024-02-05T08:28:45.434649+0000","last_fullsized":"2024-02-05T08:28:45.434649+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:38.625850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:28:38.625850+0000","last_clean_scrub_stamp":"2024-02-05T08:28:38.625850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:41.166723+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435031+0000","last_change":"2024-02-05T08:27:59.396478+0000","last_active":"2024-02-05T08:28:45.435031+0000","last_peered":"2024-02-05T08:28:45.435031+0000","last_clean":"2024-02-05T08:28:45.435031+0000","last_became_active":"2024-02-05T08:25:49.126916+0000","last_became_peered":"2024-02-05T08:25:49.126916+0000","last_unstale":"2024-02-05T08:28:45.435031+0000","last_undegraded":"2024-02-05T08:28:45.435031+0000","last_fullsized":"2024-02-05T08:28:45.435031+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:59.396397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:19.107555+0000","last_clean_scrub_stamp":"2024-02-05T08:27:59.396397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:17.099060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":410,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.732787+0000","last_change":"2024-02-05T08:28:08.179703+0000","last_active":"2024-02-05T08:28:45.732787+0000","last_peered":"2024-02-05T08:28:45.732787+0000","last_clean":"2024-02-05T08:28:45.732787+0000","last_became_active":"2024-02-05T08:25:49.151068+0000","last_became_peered":"2024-02-05T08:25:49.151068+0000","last_unstale":"2024-02-05T08:28:45.732787+0000","last_undegraded":"2024-02-05T08:28:45.732787+0000","last_fullsized":"2024-02-05T08:28:45.732787+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:08.179625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:28:08.179625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:17.126883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"397'27","reported_seq":682,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005903+0000","last_change":"2024-02-05T08:27:44.386311+0000","last_active":"2024-02-05T08:28:46.005903+0000","last_peered":"2024-02-05T08:28:46.005903+0000","last_clean":"2024-02-05T08:28:46.005903+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:28:46.005903+0000","last_undegraded":"2024-02-05T08:28:46.005903+0000","last_fullsized":"2024-02-05T08:28:46.005903+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'27","last_scrub_stamp":"2024-02-05T08:27:44.386207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:27:44.386207+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:11.102207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":889,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006594+0000","last_change":"2024-02-05T08:28:15.152931+0000","last_active":"2024-02-05T08:28:46.006594+0000","last_peered":"2024-02-05T08:28:46.006594+0000","last_clean":"2024-02-05T08:28:46.006594+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:28:46.006594+0000","last_undegraded":"2024-02-05T08:28:46.006594+0000","last_fullsized":"2024-02-05T08:28:46.006594+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:15.152778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:14.159628+0000","last_clean_scrub_stamp":"2024-02-05T08:28:15.152778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:43.384102+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":649,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005364+0000","last_change":"2024-02-05T08:28:10.297093+0000","last_active":"2024-02-05T08:28:46.005364+0000","last_peered":"2024-02-05T08:28:46.005364+0000","last_clean":"2024-02-05T08:28:46.005364+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:28:46.005364+0000","last_undegraded":"2024-02-05T08:28:46.005364+0000","last_fullsized":"2024-02-05T08:28:46.005364+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:10.296986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:28:10.296986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:32.257799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435361+0000","last_change":"2024-02-05T08:28:44.425313+0000","last_active":"2024-02-05T08:28:45.435361+0000","last_peered":"2024-02-05T08:28:45.435361+0000","last_clean":"2024-02-05T08:28:45.435361+0000","last_became_active":"2024-02-05T08:25:49.142594+0000","last_became_peered":"2024-02-05T08:25:49.142594+0000","last_unstale":"2024-02-05T08:28:45.435361+0000","last_undegraded":"2024-02-05T08:28:45.435361+0000","last_fullsized":"2024-02-05T08:28:45.435361+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:44.425209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:31.862841+0000","last_clean_scrub_stamp":"2024-02-05T08:28:44.425209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:10.469263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"397'12","reported_seq":412,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:58.111711+0000","last_change":"2024-02-05T08:28:58.111737+0000","last_active":"2024-02-05T08:28:58.111711+0000","last_peered":"2024-02-05T08:28:58.111711+0000","last_clean":"2024-02-05T08:28:58.111711+0000","last_became_active":"2024-02-05T08:23:07.937911+0000","last_became_peered":"2024-02-05T08:23:07.937911+0000","last_unstale":"2024-02-05T08:28:58.111711+0000","last_undegraded":"2024-02-05T08:28:58.111711+0000","last_fullsized":"2024-02-05T08:28:58.111711+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'12","last_scrub_stamp":"2024-02-05T08:28:58.111640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:58.111640+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:19.420470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":411,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733470+0000","last_change":"2024-02-05T08:28:14.182876+0000","last_active":"2024-02-05T08:28:45.733470+0000","last_peered":"2024-02-05T08:28:45.733470+0000","last_clean":"2024-02-05T08:28:45.733470+0000","last_became_active":"2024-02-05T08:25:49.139211+0000","last_became_peered":"2024-02-05T08:25:49.139211+0000","last_unstale":"2024-02-05T08:28:45.733470+0000","last_undegraded":"2024-02-05T08:28:45.733470+0000","last_fullsized":"2024-02-05T08:28:45.733470+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:14.182776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:28:14.182776+0000","last_clean_scrub_stamp":"2024-02-05T08:28:14.182776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:43.226323+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":921,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005364+0000","last_change":"2024-02-05T08:28:38.966213+0000","last_active":"2024-02-05T08:28:46.005364+0000","last_peered":"2024-02-05T08:28:46.005364+0000","last_clean":"2024-02-05T08:28:46.005364+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:28:46.005364+0000","last_undegraded":"2024-02-05T08:28:46.005364+0000","last_fullsized":"2024-02-05T08:28:46.005364+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:38.966118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:54.156080+0000","last_clean_scrub_stamp":"2024-02-05T08:28:38.966118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:01.440679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":917,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.430191+0000","last_change":"2024-02-05T08:28:22.738090+0000","last_active":"2024-02-05T08:28:45.430191+0000","last_peered":"2024-02-05T08:28:45.430191+0000","last_clean":"2024-02-05T08:28:45.430191+0000","last_became_active":"2024-02-05T08:23:07.932374+0000","last_became_peered":"2024-02-05T08:23:07.932374+0000","last_unstale":"2024-02-05T08:28:45.430191+0000","last_undegraded":"2024-02-05T08:28:45.430191+0000","last_fullsized":"2024-02-05T08:28:45.430191+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:22.737997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:28:22.737997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:37.249324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"397'32","reported_seq":415,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733195+0000","last_change":"2024-02-05T08:27:44.217952+0000","last_active":"2024-02-05T08:28:45.733195+0000","last_peered":"2024-02-05T08:28:45.733195+0000","last_clean":"2024-02-05T08:28:45.733195+0000","last_became_active":"2024-02-05T08:18:34.631912+0000","last_became_peered":"2024-02-05T08:18:34.631912+0000","last_unstale":"2024-02-05T08:28:45.733195+0000","last_undegraded":"2024-02-05T08:28:45.733195+0000","last_fullsized":"2024-02-05T08:28:45.733195+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'32","last_scrub_stamp":"2024-02-05T08:27:44.217867+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:27:44.217867+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:09.556276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:53.525126+0000","last_change":"2024-02-05T08:28:53.525153+0000","last_active":"2024-02-05T08:28:53.525126+0000","last_peered":"2024-02-05T08:28:53.525126+0000","last_clean":"2024-02-05T08:28:53.525126+0000","last_became_active":"2024-02-05T08:25:49.142057+0000","last_became_peered":"2024-02-05T08:25:49.142057+0000","last_unstale":"2024-02-05T08:28:53.525126+0000","last_undegraded":"2024-02-05T08:28:53.525126+0000","last_fullsized":"2024-02-05T08:28:53.525126+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:53.525042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:28:53.525042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:21.470993+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":410,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733021+0000","last_change":"2024-02-05T08:27:46.252636+0000","last_active":"2024-02-05T08:28:45.733021+0000","last_peered":"2024-02-05T08:28:45.733021+0000","last_clean":"2024-02-05T08:28:45.733021+0000","last_became_active":"2024-02-05T08:25:49.138377+0000","last_became_peered":"2024-02-05T08:25:49.138377+0000","last_unstale":"2024-02-05T08:28:45.733021+0000","last_undegraded":"2024-02-05T08:28:45.733021+0000","last_fullsized":"2024-02-05T08:28:45.733021+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:46.252510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:25:33.208642+0000","last_clean_scrub_stamp":"2024-02-05T08:27:46.252510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:55.912234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":921,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:51.998633+0000","last_change":"2024-02-05T08:28:51.998633+0000","last_active":"2024-02-05T08:28:51.998633+0000","last_peered":"2024-02-05T08:28:51.998633+0000","last_clean":"2024-02-05T08:28:51.998633+0000","last_became_active":"2024-02-05T08:18:34.642387+0000","last_became_peered":"2024-02-05T08:18:34.642387+0000","last_unstale":"2024-02-05T08:28:51.998633+0000","last_undegraded":"2024-02-05T08:28:51.998633+0000","last_fullsized":"2024-02-05T08:28:51.998633+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:51.998537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:28:51.998537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:13.838050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"397'25","reported_seq":585,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429105+0000","last_change":"2024-02-05T08:27:57.844837+0000","last_active":"2024-02-05T08:28:45.429105+0000","last_peered":"2024-02-05T08:28:45.429105+0000","last_clean":"2024-02-05T08:28:45.429105+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:28:45.429105+0000","last_undegraded":"2024-02-05T08:28:45.429105+0000","last_fullsized":"2024-02-05T08:28:45.429105+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-02-05T08:27:57.844718+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_clean_scrub_stamp":"2024-02-05T08:27:57.844718+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:17.818872+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":419,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733086+0000","last_change":"2024-02-05T08:28:29.252131+0000","last_active":"2024-02-05T08:28:45.733086+0000","last_peered":"2024-02-05T08:28:45.733086+0000","last_clean":"2024-02-05T08:28:45.733086+0000","last_became_active":"2024-02-05T08:25:49.140007+0000","last_became_peered":"2024-02-05T08:25:49.140007+0000","last_unstale":"2024-02-05T08:28:45.733086+0000","last_undegraded":"2024-02-05T08:28:45.733086+0000","last_fullsized":"2024-02-05T08:28:45.733086+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:29.251979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:28:29.251979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:32.782087+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":759,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:02.787053+0000","last_change":"2024-02-05T08:29:02.787088+0000","last_active":"2024-02-05T08:29:02.787053+0000","last_peered":"2024-02-05T08:29:02.787053+0000","last_clean":"2024-02-05T08:29:02.787053+0000","last_became_active":"2024-02-05T08:28:23.352654+0000","last_became_peered":"2024-02-05T08:28:23.352654+0000","last_unstale":"2024-02-05T08:29:02.787053+0000","last_undegraded":"2024-02-05T08:29:02.787053+0000","last_fullsized":"2024-02-05T08:29:02.787053+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:02.786983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:29:02.786983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:07.029496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":920,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735561+0000","last_change":"2024-02-05T08:28:00.983432+0000","last_active":"2024-02-05T08:28:45.735561+0000","last_peered":"2024-02-05T08:28:45.735561+0000","last_clean":"2024-02-05T08:28:45.735561+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:28:45.735561+0000","last_undegraded":"2024-02-05T08:28:45.735561+0000","last_fullsized":"2024-02-05T08:28:45.735561+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:00.983322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:28:00.983322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:21.294561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"397'32","reported_seq":989,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:00.078241+0000","last_change":"2024-02-05T08:29:00.078273+0000","last_active":"2024-02-05T08:29:00.078241+0000","last_peered":"2024-02-05T08:29:00.078241+0000","last_clean":"2024-02-05T08:29:00.078241+0000","last_became_active":"2024-02-05T08:23:07.938523+0000","last_became_peered":"2024-02-05T08:23:07.938523+0000","last_unstale":"2024-02-05T08:29:00.078241+0000","last_undegraded":"2024-02-05T08:29:00.078241+0000","last_fullsized":"2024-02-05T08:29:00.078241+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'32","last_scrub_stamp":"2024-02-05T08:29:00.078172+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:29:00.078172+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:29.883907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":140,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435066+0000","last_change":"2024-02-05T08:27:41.228135+0000","last_active":"2024-02-05T08:28:45.435066+0000","last_peered":"2024-02-05T08:28:45.435066+0000","last_clean":"2024-02-05T08:28:45.435066+0000","last_became_active":"2024-02-05T08:25:49.130779+0000","last_became_peered":"2024-02-05T08:25:49.130779+0000","last_unstale":"2024-02-05T08:28:45.435066+0000","last_undegraded":"2024-02-05T08:28:45.435066+0000","last_fullsized":"2024-02-05T08:28:45.435066+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:41.228018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:41.228018+0000","last_clean_scrub_stamp":"2024-02-05T08:27:41.228018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:03.384055+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"407'15","reported_seq":1234,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.869900+0000","last_change":"2024-02-05T08:28:01.813418+0000","last_active":"2024-02-05T08:28:56.869900+0000","last_peered":"2024-02-05T08:28:56.869900+0000","last_clean":"2024-02-05T08:28:56.869900+0000","last_became_active":"2024-02-05T08:25:49.135603+0000","last_became_peered":"2024-02-05T08:25:49.135603+0000","last_unstale":"2024-02-05T08:28:56.869900+0000","last_undegraded":"2024-02-05T08:28:56.869900+0000","last_fullsized":"2024-02-05T08:28:56.869900+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'15","last_scrub_stamp":"2024-02-05T08:28:01.813308+0000","last_deep_scrub":"391'14","last_deep_scrub_stamp":"2024-02-05T08:24:33.873828+0000","last_clean_scrub_stamp":"2024-02-05T08:28:01.813308+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:07.086906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":909,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006427+0000","last_change":"2024-02-05T08:28:00.198702+0000","last_active":"2024-02-05T08:28:46.006427+0000","last_peered":"2024-02-05T08:28:46.006427+0000","last_clean":"2024-02-05T08:28:46.006427+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:28:46.006427+0000","last_undegraded":"2024-02-05T08:28:46.006427+0000","last_fullsized":"2024-02-05T08:28:46.006427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:00.198610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:28:00.198610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:06.654293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"397'35","reported_seq":419,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733317+0000","last_change":"2024-02-05T08:27:40.203035+0000","last_active":"2024-02-05T08:28:45.733317+0000","last_peered":"2024-02-05T08:28:45.733317+0000","last_clean":"2024-02-05T08:28:45.733317+0000","last_became_active":"2024-02-05T08:23:07.928966+0000","last_became_peered":"2024-02-05T08:23:07.928966+0000","last_unstale":"2024-02-05T08:28:45.733317+0000","last_undegraded":"2024-02-05T08:28:45.733317+0000","last_fullsized":"2024-02-05T08:28:45.733317+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'35","last_scrub_stamp":"2024-02-05T08:27:40.202921+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:27:40.202921+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:54.216541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":883,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.736175+0000","last_change":"2024-02-05T08:27:39.290278+0000","last_active":"2024-02-05T08:28:45.736175+0000","last_peered":"2024-02-05T08:28:45.736175+0000","last_clean":"2024-02-05T08:28:45.736175+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:28:45.736175+0000","last_undegraded":"2024-02-05T08:28:45.736175+0000","last_fullsized":"2024-02-05T08:28:45.736175+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:39.290177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:39.290177+0000","last_clean_scrub_stamp":"2024-02-05T08:27:39.290177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"295'10","reported_seq":522,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.870079+0000","last_change":"2024-02-05T08:27:48.313032+0000","last_active":"2024-02-05T08:28:56.870079+0000","last_peered":"2024-02-05T08:28:56.870079+0000","last_clean":"2024-02-05T08:28:56.870079+0000","last_became_active":"2024-02-05T08:18:34.644517+0000","last_became_peered":"2024-02-05T08:18:34.644517+0000","last_unstale":"2024-02-05T08:28:56.870079+0000","last_undegraded":"2024-02-05T08:28:56.870079+0000","last_fullsized":"2024-02-05T08:28:56.870079+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'10","last_scrub_stamp":"2024-02-05T08:27:48.312951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:27:48.312951+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:59.486284+0000","last_change":"2024-02-05T08:28:59.486303+0000","last_active":"2024-02-05T08:28:59.486284+0000","last_peered":"2024-02-05T08:28:59.486284+0000","last_clean":"2024-02-05T08:28:59.486284+0000","last_became_active":"2024-02-05T08:25:49.126790+0000","last_became_peered":"2024-02-05T08:25:49.126790+0000","last_unstale":"2024-02-05T08:28:59.486284+0000","last_undegraded":"2024-02-05T08:28:59.486284+0000","last_fullsized":"2024-02-05T08:28:59.486284+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:59.486230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:28:59.486230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:17.131261+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"397'21","reported_seq":901,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.430339+0000","last_change":"2024-02-05T08:28:02.257090+0000","last_active":"2024-02-05T08:28:45.430339+0000","last_peered":"2024-02-05T08:28:45.430339+0000","last_clean":"2024-02-05T08:28:45.430339+0000","last_became_active":"2024-02-05T08:28:02.256642+0000","last_became_peered":"2024-02-05T08:28:02.256642+0000","last_unstale":"2024-02-05T08:28:45.430339+0000","last_undegraded":"2024-02-05T08:28:45.430339+0000","last_fullsized":"2024-02-05T08:28:45.430339+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-02-05T08:27:49.038465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:27:49.038465+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:07.417809+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":928,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005954+0000","last_change":"2024-02-05T08:27:55.654783+0000","last_active":"2024-02-05T08:28:46.005954+0000","last_peered":"2024-02-05T08:28:46.005954+0000","last_clean":"2024-02-05T08:28:46.005954+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:28:46.005954+0000","last_undegraded":"2024-02-05T08:28:46.005954+0000","last_fullsized":"2024-02-05T08:28:46.005954+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:55.654704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:26:38.587082+0000","last_clean_scrub_stamp":"2024-02-05T08:27:55.654704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:21.827619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":392,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733757+0000","last_change":"2024-02-05T08:27:37.243710+0000","last_active":"2024-02-05T08:28:45.733757+0000","last_peered":"2024-02-05T08:28:45.733757+0000","last_clean":"2024-02-05T08:28:45.733757+0000","last_became_active":"2024-02-05T08:23:07.928375+0000","last_became_peered":"2024-02-05T08:23:07.928375+0000","last_unstale":"2024-02-05T08:28:45.733757+0000","last_undegraded":"2024-02-05T08:28:45.733757+0000","last_fullsized":"2024-02-05T08:28:45.733757+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:37.243631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:42.025471+0000","last_clean_scrub_stamp":"2024-02-05T08:27:37.243631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:54.394523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":921,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.067915+0000","last_change":"2024-02-05T08:28:56.067938+0000","last_active":"2024-02-05T08:28:56.067915+0000","last_peered":"2024-02-05T08:28:56.067915+0000","last_clean":"2024-02-05T08:28:56.067915+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:28:56.067915+0000","last_undegraded":"2024-02-05T08:28:56.067915+0000","last_fullsized":"2024-02-05T08:28:56.067915+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:56.067851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:28:56.067851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:59.989176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"397'25","reported_seq":158,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434970+0000","last_change":"2024-02-05T08:28:34.558605+0000","last_active":"2024-02-05T08:28:45.434970+0000","last_peered":"2024-02-05T08:28:45.434970+0000","last_clean":"2024-02-05T08:28:45.434970+0000","last_became_active":"2024-02-05T08:25:49.129856+0000","last_became_peered":"2024-02-05T08:25:49.129856+0000","last_unstale":"2024-02-05T08:28:45.434970+0000","last_undegraded":"2024-02-05T08:28:45.434970+0000","last_fullsized":"2024-02-05T08:28:45.434970+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-02-05T08:28:34.558501+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:28:34.558501+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:45.806187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":909,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:47.132987+0000","last_change":"2024-02-05T08:28:47.132987+0000","last_active":"2024-02-05T08:28:47.132987+0000","last_peered":"2024-02-05T08:28:47.132987+0000","last_clean":"2024-02-05T08:28:47.132987+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:28:47.132987+0000","last_undegraded":"2024-02-05T08:28:47.132987+0000","last_fullsized":"2024-02-05T08:28:47.132987+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:47.132890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:47.132890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:04.414482+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":913,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.430510+0000","last_change":"2024-02-05T08:27:30.611622+0000","last_active":"2024-02-05T08:28:45.430510+0000","last_peered":"2024-02-05T08:28:45.430510+0000","last_clean":"2024-02-05T08:28:45.430510+0000","last_became_active":"2024-02-05T08:23:07.932741+0000","last_became_peered":"2024-02-05T08:23:07.932741+0000","last_unstale":"2024-02-05T08:28:45.430510+0000","last_undegraded":"2024-02-05T08:28:45.430510+0000","last_fullsized":"2024-02-05T08:28:45.430510+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:30.611531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:27:30.611531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:54.940912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":666,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006516+0000","last_change":"2024-02-05T08:28:36.195968+0000","last_active":"2024-02-05T08:28:46.006516+0000","last_peered":"2024-02-05T08:28:46.006516+0000","last_clean":"2024-02-05T08:28:46.006516+0000","last_became_active":"2024-02-05T08:25:49.126639+0000","last_became_peered":"2024-02-05T08:25:49.126639+0000","last_unstale":"2024-02-05T08:28:46.006516+0000","last_undegraded":"2024-02-05T08:28:46.006516+0000","last_fullsized":"2024-02-05T08:28:46.006516+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:36.195873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:28:36.195873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:39.178323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"397'21","reported_seq":409,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:52.146658+0000","last_change":"2024-02-05T08:28:52.146683+0000","last_active":"2024-02-05T08:28:52.146658+0000","last_peered":"2024-02-05T08:28:52.146658+0000","last_clean":"2024-02-05T08:28:52.146658+0000","last_became_active":"2024-02-05T08:23:07.927922+0000","last_became_peered":"2024-02-05T08:23:07.927922+0000","last_unstale":"2024-02-05T08:28:52.146658+0000","last_undegraded":"2024-02-05T08:28:52.146658+0000","last_fullsized":"2024-02-05T08:28:52.146658+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-02-05T08:28:52.146601+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:28:52.146601+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:06.641926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":852,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429763+0000","last_change":"2024-02-05T08:28:34.656447+0000","last_active":"2024-02-05T08:28:45.429763+0000","last_peered":"2024-02-05T08:28:45.429763+0000","last_clean":"2024-02-05T08:28:45.429763+0000","last_became_active":"2024-02-05T08:28:02.279886+0000","last_became_peered":"2024-02-05T08:28:02.279886+0000","last_unstale":"2024-02-05T08:28:45.429763+0000","last_undegraded":"2024-02-05T08:28:45.429763+0000","last_fullsized":"2024-02-05T08:28:45.429763+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:34.656353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:34.656353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:41.007049+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":949,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.007615+0000","last_change":"2024-02-05T08:27:46.028882+0000","last_active":"2024-02-05T08:28:46.007615+0000","last_peered":"2024-02-05T08:28:46.007615+0000","last_clean":"2024-02-05T08:28:46.007615+0000","last_became_active":"2024-02-05T08:25:49.136088+0000","last_became_peered":"2024-02-05T08:25:49.136088+0000","last_unstale":"2024-02-05T08:28:46.007615+0000","last_undegraded":"2024-02-05T08:28:46.007615+0000","last_fullsized":"2024-02-05T08:28:46.007615+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:27:46.028778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:27:46.028778+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:54.255991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"397'46","reported_seq":1017,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.428928+0000","last_change":"2024-02-05T08:27:54.783339+0000","last_active":"2024-02-05T08:28:45.428928+0000","last_peered":"2024-02-05T08:28:45.428928+0000","last_clean":"2024-02-05T08:28:45.428928+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:28:45.428928+0000","last_undegraded":"2024-02-05T08:28:45.428928+0000","last_fullsized":"2024-02-05T08:28:45.428928+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-02-05T08:27:54.783199+0000","last_deep_scrub":"397'46","last_deep_scrub_stamp":"2024-02-05T08:27:54.783199+0000","last_clean_scrub_stamp":"2024-02-05T08:27:54.783199+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:19.558353+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":1224,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.869931+0000","last_change":"2024-02-05T08:28:08.122653+0000","last_active":"2024-02-05T08:28:56.869931+0000","last_peered":"2024-02-05T08:28:56.869931+0000","last_clean":"2024-02-05T08:28:56.869931+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:28:56.869931+0000","last_undegraded":"2024-02-05T08:28:56.869931+0000","last_fullsized":"2024-02-05T08:28:56.869931+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:28:08.122530+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_clean_scrub_stamp":"2024-02-05T08:28:08.122530+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:19.942679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":384,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733463+0000","last_change":"2024-02-05T08:27:32.246441+0000","last_active":"2024-02-05T08:28:45.733463+0000","last_peered":"2024-02-05T08:28:45.733463+0000","last_clean":"2024-02-05T08:28:45.733463+0000","last_became_active":"2024-02-05T08:23:07.929942+0000","last_became_peered":"2024-02-05T08:23:07.929942+0000","last_unstale":"2024-02-05T08:28:45.733463+0000","last_undegraded":"2024-02-05T08:28:45.733463+0000","last_fullsized":"2024-02-05T08:28:45.733463+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:32.246345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:27:32.246345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":910,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.007212+0000","last_change":"2024-02-05T08:27:52.126046+0000","last_active":"2024-02-05T08:28:46.007212+0000","last_peered":"2024-02-05T08:28:46.007212+0000","last_clean":"2024-02-05T08:28:46.007212+0000","last_became_active":"2024-02-05T08:23:07.937863+0000","last_became_peered":"2024-02-05T08:23:07.937863+0000","last_unstale":"2024-02-05T08:28:46.007212+0000","last_undegraded":"2024-02-05T08:28:46.007212+0000","last_fullsized":"2024-02-05T08:28:46.007212+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:52.125951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:27:52.125951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:09.057377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"465'29","reported_seq":980,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735472+0000","last_change":"2024-02-05T08:28:35.249411+0000","last_active":"2024-02-05T08:28:45.735472+0000","last_peered":"2024-02-05T08:28:45.735472+0000","last_clean":"2024-02-05T08:28:45.735472+0000","last_became_active":"2024-02-05T08:23:07.929900+0000","last_became_peered":"2024-02-05T08:23:07.929900+0000","last_unstale":"2024-02-05T08:28:45.735472+0000","last_undegraded":"2024-02-05T08:28:45.735472+0000","last_fullsized":"2024-02-05T08:28:45.735472+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'29","last_scrub_stamp":"2024-02-05T08:28:35.249301+0000","last_deep_scrub":"465'29","last_deep_scrub_stamp":"2024-02-05T08:28:35.249301+0000","last_clean_scrub_stamp":"2024-02-05T08:28:35.249301+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:04.389919+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":835,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:58.371889+0000","last_change":"2024-02-05T08:28:58.371909+0000","last_active":"2024-02-05T08:28:58.371889+0000","last_peered":"2024-02-05T08:28:58.371889+0000","last_clean":"2024-02-05T08:28:58.371889+0000","last_became_active":"2024-02-05T08:27:39.132668+0000","last_became_peered":"2024-02-05T08:27:39.132668+0000","last_unstale":"2024-02-05T08:28:58.371889+0000","last_undegraded":"2024-02-05T08:28:58.371889+0000","last_fullsized":"2024-02-05T08:28:58.371889+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:58.371831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:28:58.371831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:13.373419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":910,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005939+0000","last_change":"2024-02-05T08:28:05.146381+0000","last_active":"2024-02-05T08:28:46.005939+0000","last_peered":"2024-02-05T08:28:46.005939+0000","last_clean":"2024-02-05T08:28:46.005939+0000","last_became_active":"2024-02-05T08:18:34.642239+0000","last_became_peered":"2024-02-05T08:18:34.642239+0000","last_unstale":"2024-02-05T08:28:46.005939+0000","last_undegraded":"2024-02-05T08:28:46.005939+0000","last_fullsized":"2024-02-05T08:28:46.005939+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:05.146259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:05.146259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:13.388312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":936,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005717+0000","last_change":"2024-02-05T08:28:13.671524+0000","last_active":"2024-02-05T08:28:46.005717+0000","last_peered":"2024-02-05T08:28:46.005717+0000","last_clean":"2024-02-05T08:28:46.005717+0000","last_became_active":"2024-02-05T08:25:49.118945+0000","last_became_peered":"2024-02-05T08:25:49.118945+0000","last_unstale":"2024-02-05T08:28:46.005717+0000","last_undegraded":"2024-02-05T08:28:46.005717+0000","last_fullsized":"2024-02-05T08:28:46.005717+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:13.671444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:28:13.671444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:26.437635+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":914,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005592+0000","last_change":"2024-02-05T08:27:25.268382+0000","last_active":"2024-02-05T08:28:46.005592+0000","last_peered":"2024-02-05T08:28:46.005592+0000","last_clean":"2024-02-05T08:28:46.005592+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:28:46.005592+0000","last_undegraded":"2024-02-05T08:28:46.005592+0000","last_fullsized":"2024-02-05T08:28:46.005592+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:25.268245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_clean_scrub_stamp":"2024-02-05T08:27:25.268245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:52.818850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"397'51","reported_seq":722,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.007428+0000","last_change":"2024-02-05T08:27:56.400181+0000","last_active":"2024-02-05T08:28:46.007428+0000","last_peered":"2024-02-05T08:28:46.007428+0000","last_clean":"2024-02-05T08:28:46.007428+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:28:46.007428+0000","last_undegraded":"2024-02-05T08:28:46.007428+0000","last_fullsized":"2024-02-05T08:28:46.007428+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'51","last_scrub_stamp":"2024-02-05T08:27:56.400058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:27:56.400058+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:06.515590+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435107+0000","last_change":"2024-02-05T08:27:53.451662+0000","last_active":"2024-02-05T08:28:45.435107+0000","last_peered":"2024-02-05T08:28:45.435107+0000","last_clean":"2024-02-05T08:28:45.435107+0000","last_became_active":"2024-02-05T08:25:49.125995+0000","last_became_peered":"2024-02-05T08:25:49.125995+0000","last_unstale":"2024-02-05T08:28:45.435107+0000","last_undegraded":"2024-02-05T08:28:45.435107+0000","last_fullsized":"2024-02-05T08:28:45.435107+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:53.451556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_clean_scrub_stamp":"2024-02-05T08:27:53.451556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:22.102024+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":842,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005517+0000","last_change":"2024-02-05T08:28:43.527959+0000","last_active":"2024-02-05T08:28:46.005517+0000","last_peered":"2024-02-05T08:28:46.005517+0000","last_clean":"2024-02-05T08:28:46.005517+0000","last_became_active":"2024-02-05T08:27:39.133621+0000","last_became_peered":"2024-02-05T08:27:39.133621+0000","last_unstale":"2024-02-05T08:28:46.005517+0000","last_undegraded":"2024-02-05T08:28:46.005517+0000","last_fullsized":"2024-02-05T08:28:46.005517+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:43.527843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:28:43.527843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:01.982544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":856,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429986+0000","last_change":"2024-02-05T08:28:02.253974+0000","last_active":"2024-02-05T08:28:45.429986+0000","last_peered":"2024-02-05T08:28:45.429986+0000","last_clean":"2024-02-05T08:28:45.429986+0000","last_became_active":"2024-02-05T08:28:02.253234+0000","last_became_peered":"2024-02-05T08:28:02.253234+0000","last_unstale":"2024-02-05T08:28:45.429986+0000","last_undegraded":"2024-02-05T08:28:45.429986+0000","last_fullsized":"2024-02-05T08:28:45.429986+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:58.255975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:01.230038+0000","last_clean_scrub_stamp":"2024-02-05T08:27:58.255975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:22.588813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"397'25","reported_seq":158,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435320+0000","last_change":"2024-02-05T08:28:24.442805+0000","last_active":"2024-02-05T08:28:45.435320+0000","last_peered":"2024-02-05T08:28:45.435320+0000","last_clean":"2024-02-05T08:28:45.435320+0000","last_became_active":"2024-02-05T08:25:49.134815+0000","last_became_peered":"2024-02-05T08:25:49.134815+0000","last_unstale":"2024-02-05T08:28:45.435320+0000","last_undegraded":"2024-02-05T08:28:45.435320+0000","last_fullsized":"2024-02-05T08:28:45.435320+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-02-05T08:28:24.442707+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:28:24.442707+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:26.061102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"82'130","reported_seq":625,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.428993+0000","last_change":"2024-02-05T08:28:02.779779+0000","last_active":"2024-02-05T08:28:45.428993+0000","last_peered":"2024-02-05T08:28:45.428993+0000","last_clean":"2024-02-05T08:28:45.428993+0000","last_became_active":"2024-02-05T08:28:02.257244+0000","last_became_peered":"2024-02-05T08:28:02.257244+0000","last_unstale":"2024-02-05T08:28:45.428993+0000","last_undegraded":"2024-02-05T08:28:45.428993+0000","last_fullsized":"2024-02-05T08:28:45.428993+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:02.779675+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:28:02.779675+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:12.155730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":620,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735864+0000","last_change":"2024-02-05T08:28:32.278888+0000","last_active":"2024-02-05T08:28:45.735864+0000","last_peered":"2024-02-05T08:28:45.735864+0000","last_clean":"2024-02-05T08:28:45.735864+0000","last_became_active":"2024-02-05T08:23:07.936660+0000","last_became_peered":"2024-02-05T08:23:07.936660+0000","last_unstale":"2024-02-05T08:28:45.735864+0000","last_undegraded":"2024-02-05T08:28:45.735864+0000","last_fullsized":"2024-02-05T08:28:45.735864+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:32.278771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:32.278771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:47.953613+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":874,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.420173+0000","last_change":"2024-02-05T08:28:46.420202+0000","last_active":"2024-02-05T08:28:46.420173+0000","last_peered":"2024-02-05T08:28:46.420173+0000","last_clean":"2024-02-05T08:28:46.420173+0000","last_became_active":"2024-02-05T08:28:02.254335+0000","last_became_peered":"2024-02-05T08:28:02.254335+0000","last_unstale":"2024-02-05T08:28:46.420173+0000","last_undegraded":"2024-02-05T08:28:46.420173+0000","last_fullsized":"2024-02-05T08:28:46.420173+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:28:46.420096+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:28:46.420096+0000","last_clean_scrub_stamp":"2024-02-05T08:28:46.420096+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:46.897187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":937,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:06.037240+0000","last_change":"2024-02-05T08:29:06.037281+0000","last_active":"2024-02-05T08:29:06.037240+0000","last_peered":"2024-02-05T08:29:06.037240+0000","last_clean":"2024-02-05T08:29:06.037240+0000","last_became_active":"2024-02-05T08:25:49.113840+0000","last_became_peered":"2024-02-05T08:25:49.113840+0000","last_unstale":"2024-02-05T08:29:06.037240+0000","last_undegraded":"2024-02-05T08:29:06.037240+0000","last_fullsized":"2024-02-05T08:29:06.037240+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:06.037148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:40.135374+0000","last_clean_scrub_stamp":"2024-02-05T08:29:06.037148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:21.097231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"464'68","reported_seq":1122,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.869859+0000","last_change":"2024-02-05T08:28:23.352523+0000","last_active":"2024-02-05T08:28:56.869859+0000","last_peered":"2024-02-05T08:28:56.869859+0000","last_clean":"2024-02-05T08:28:56.869859+0000","last_became_active":"2024-02-05T08:28:23.352147+0000","last_became_peered":"2024-02-05T08:28:23.352147+0000","last_unstale":"2024-02-05T08:28:56.869859+0000","last_undegraded":"2024-02-05T08:28:56.869859+0000","last_fullsized":"2024-02-05T08:28:56.869859+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'67","last_scrub_stamp":"2024-02-05T08:28:04.672873+0000","last_deep_scrub":"435'67","last_deep_scrub_stamp":"2024-02-05T08:28:04.672873+0000","last_clean_scrub_stamp":"2024-02-05T08:28:04.672873+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:04.877722+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434746+0000","last_change":"2024-02-05T08:28:40.512383+0000","last_active":"2024-02-05T08:28:45.434746+0000","last_peered":"2024-02-05T08:28:45.434746+0000","last_clean":"2024-02-05T08:28:45.434746+0000","last_became_active":"2024-02-05T08:25:49.134871+0000","last_became_peered":"2024-02-05T08:25:49.134871+0000","last_unstale":"2024-02-05T08:28:45.434746+0000","last_undegraded":"2024-02-05T08:28:45.434746+0000","last_fullsized":"2024-02-05T08:28:45.434746+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:40.512307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:52.171116+0000","last_clean_scrub_stamp":"2024-02-05T08:28:40.512307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:55.874569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"82'130","reported_seq":1226,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006476+0000","last_change":"2024-02-05T08:27:44.978552+0000","last_active":"2024-02-05T08:28:46.006476+0000","last_peered":"2024-02-05T08:28:46.006476+0000","last_clean":"2024-02-05T08:28:46.006476+0000","last_became_active":"2024-02-05T08:25:52.156544+0000","last_became_peered":"2024-02-05T08:25:52.156544+0000","last_unstale":"2024-02-05T08:28:46.006476+0000","last_undegraded":"2024-02-05T08:28:46.006476+0000","last_fullsized":"2024-02-05T08:28:46.006476+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:27:44.978433+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:27:44.978433+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:11.489411+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":914,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.007149+0000","last_change":"2024-02-05T08:27:44.094804+0000","last_active":"2024-02-05T08:28:46.007149+0000","last_peered":"2024-02-05T08:28:46.007149+0000","last_clean":"2024-02-05T08:28:46.007149+0000","last_became_active":"2024-02-05T08:18:34.643109+0000","last_became_peered":"2024-02-05T08:18:34.643109+0000","last_unstale":"2024-02-05T08:28:46.007149+0000","last_undegraded":"2024-02-05T08:28:46.007149+0000","last_fullsized":"2024-02-05T08:28:46.007149+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:44.094699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:27:44.094699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:13.833035+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":194,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.185962+0000","last_change":"2024-02-05T08:29:01.186002+0000","last_active":"2024-02-05T08:29:01.185962+0000","last_peered":"2024-02-05T08:29:01.185962+0000","last_clean":"2024-02-05T08:29:01.185962+0000","last_became_active":"2024-02-05T08:24:08.005056+0000","last_became_peered":"2024-02-05T08:24:08.005056+0000","last_unstale":"2024-02-05T08:29:01.185962+0000","last_undegraded":"2024-02-05T08:29:01.185962+0000","last_fullsized":"2024-02-05T08:29:01.185962+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:29:01.185872+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:29:01.185872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:23.034864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":970,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005443+0000","last_change":"2024-02-05T08:27:54.691753+0000","last_active":"2024-02-05T08:28:46.005443+0000","last_peered":"2024-02-05T08:28:46.005443+0000","last_clean":"2024-02-05T08:28:46.005443+0000","last_became_active":"2024-02-05T08:25:49.127248+0000","last_became_peered":"2024-02-05T08:25:49.127248+0000","last_unstale":"2024-02-05T08:28:46.005443+0000","last_undegraded":"2024-02-05T08:28:46.005443+0000","last_fullsized":"2024-02-05T08:28:46.005443+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:54.691677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_clean_scrub_stamp":"2024-02-05T08:27:54.691677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:06.570786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"397'26","reported_seq":986,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006140+0000","last_change":"2024-02-05T08:28:36.577381+0000","last_active":"2024-02-05T08:28:46.006140+0000","last_peered":"2024-02-05T08:28:46.006140+0000","last_clean":"2024-02-05T08:28:46.006140+0000","last_became_active":"2024-02-05T08:25:49.115716+0000","last_became_peered":"2024-02-05T08:25:49.115716+0000","last_unstale":"2024-02-05T08:28:46.006140+0000","last_undegraded":"2024-02-05T08:28:46.006140+0000","last_fullsized":"2024-02-05T08:28:46.006140+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'26","last_scrub_stamp":"2024-02-05T08:28:36.577260+0000","last_deep_scrub":"397'26","last_deep_scrub_stamp":"2024-02-05T08:28:36.577260+0000","last_clean_scrub_stamp":"2024-02-05T08:28:36.577260+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:01.816591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":933,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005117+0000","last_change":"2024-02-05T08:28:44.516566+0000","last_active":"2024-02-05T08:28:46.005117+0000","last_peered":"2024-02-05T08:28:46.005117+0000","last_clean":"2024-02-05T08:28:46.005117+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:28:46.005117+0000","last_undegraded":"2024-02-05T08:28:46.005117+0000","last_fullsized":"2024-02-05T08:28:46.005117+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:44.516454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:25:46.935681+0000","last_clean_scrub_stamp":"2024-02-05T08:28:44.516454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:07.807695+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":917,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.388848+0000","last_change":"2024-02-05T08:29:01.388873+0000","last_active":"2024-02-05T08:29:01.388848+0000","last_peered":"2024-02-05T08:29:01.388848+0000","last_clean":"2024-02-05T08:29:01.388848+0000","last_became_active":"2024-02-05T08:18:34.633450+0000","last_became_peered":"2024-02-05T08:18:34.633450+0000","last_unstale":"2024-02-05T08:29:01.388848+0000","last_undegraded":"2024-02-05T08:29:01.388848+0000","last_fullsized":"2024-02-05T08:29:01.388848+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:01.388774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:29:01.388774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:22.767933+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":1224,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006115+0000","last_change":"2024-02-05T08:28:03.210289+0000","last_active":"2024-02-05T08:28:46.006115+0000","last_peered":"2024-02-05T08:28:46.006115+0000","last_clean":"2024-02-05T08:28:46.006115+0000","last_became_active":"2024-02-05T08:25:52.159754+0000","last_became_peered":"2024-02-05T08:25:52.159754+0000","last_unstale":"2024-02-05T08:28:46.006115+0000","last_undegraded":"2024-02-05T08:28:46.006115+0000","last_fullsized":"2024-02-05T08:28:46.006115+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:03.210194+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_clean_scrub_stamp":"2024-02-05T08:28:03.210194+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:17.363251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":400,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733095+0000","last_change":"2024-02-05T08:28:22.093244+0000","last_active":"2024-02-05T08:28:45.733095+0000","last_peered":"2024-02-05T08:28:45.733095+0000","last_clean":"2024-02-05T08:28:45.733095+0000","last_became_active":"2024-02-05T08:18:34.631974+0000","last_became_peered":"2024-02-05T08:18:34.631974+0000","last_unstale":"2024-02-05T08:28:45.733095+0000","last_undegraded":"2024-02-05T08:28:45.733095+0000","last_fullsized":"2024-02-05T08:28:45.733095+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:22.093154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:22.093154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:26.195903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"397'42","reported_seq":899,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.870083+0000","last_change":"2024-02-05T08:28:41.099100+0000","last_active":"2024-02-05T08:28:56.870083+0000","last_peered":"2024-02-05T08:28:56.870083+0000","last_clean":"2024-02-05T08:28:56.870083+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:28:56.870083+0000","last_undegraded":"2024-02-05T08:28:56.870083+0000","last_fullsized":"2024-02-05T08:28:56.870083+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'42","last_scrub_stamp":"2024-02-05T08:28:41.098988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:41.098988+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:58.509615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":902,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005456+0000","last_change":"2024-02-05T08:28:10.128826+0000","last_active":"2024-02-05T08:28:46.005456+0000","last_peered":"2024-02-05T08:28:46.005456+0000","last_clean":"2024-02-05T08:28:46.005456+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:28:46.005456+0000","last_undegraded":"2024-02-05T08:28:46.005456+0000","last_fullsized":"2024-02-05T08:28:46.005456+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:10.128728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_clean_scrub_stamp":"2024-02-05T08:28:10.128728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:30.132839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"82'130","reported_seq":519,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:51.808807+0000","last_change":"2024-02-05T08:28:51.808850+0000","last_active":"2024-02-05T08:28:51.808807+0000","last_peered":"2024-02-05T08:28:51.808807+0000","last_clean":"2024-02-05T08:28:51.808807+0000","last_became_active":"2024-02-05T08:28:23.348448+0000","last_became_peered":"2024-02-05T08:28:23.348448+0000","last_unstale":"2024-02-05T08:28:51.808807+0000","last_undegraded":"2024-02-05T08:28:51.808807+0000","last_fullsized":"2024-02-05T08:28:51.808807+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:51.808727+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:28:51.808727+0000","last_clean_scrub_stamp":"2024-02-05T08:28:51.808727+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:17.006732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":813,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:03.819141+0000","last_change":"2024-02-05T08:29:03.819161+0000","last_active":"2024-02-05T08:29:03.819141+0000","last_peered":"2024-02-05T08:29:03.819141+0000","last_clean":"2024-02-05T08:29:03.819141+0000","last_became_active":"2024-02-05T08:23:07.935640+0000","last_became_peered":"2024-02-05T08:23:07.935640+0000","last_unstale":"2024-02-05T08:29:03.819141+0000","last_undegraded":"2024-02-05T08:29:03.819141+0000","last_fullsized":"2024-02-05T08:29:03.819141+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:03.819081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:29:03.819081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:26.593845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":140,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435066+0000","last_change":"2024-02-05T08:27:39.173280+0000","last_active":"2024-02-05T08:28:45.435066+0000","last_peered":"2024-02-05T08:28:45.435066+0000","last_clean":"2024-02-05T08:28:45.435066+0000","last_became_active":"2024-02-05T08:25:49.134615+0000","last_became_peered":"2024-02-05T08:25:49.134615+0000","last_unstale":"2024-02-05T08:28:45.435066+0000","last_undegraded":"2024-02-05T08:28:45.435066+0000","last_fullsized":"2024-02-05T08:28:45.435066+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:39.173205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:27:39.173205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:03.645543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":2400,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:02.432861+0000","last_change":"2024-02-05T08:28:57.106025+0000","last_active":"2024-02-05T08:29:02.432861+0000","last_peered":"2024-02-05T08:29:02.432861+0000","last_clean":"2024-02-05T08:29:02.432861+0000","last_became_active":"2024-02-05T08:25:49.148530+0000","last_became_peered":"2024-02-05T08:25:49.148530+0000","last_unstale":"2024-02-05T08:29:02.432861+0000","last_undegraded":"2024-02-05T08:29:02.432861+0000","last_fullsized":"2024-02-05T08:29:02.432861+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:28:57.105911+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:28:57.105911+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:16.124700+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1436,"num_read_kb":1431,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"425'42","reported_seq":168,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435136+0000","last_change":"2024-02-05T08:28:17.495594+0000","last_active":"2024-02-05T08:28:45.435136+0000","last_peered":"2024-02-05T08:28:45.435136+0000","last_clean":"2024-02-05T08:28:45.435136+0000","last_became_active":"2024-02-05T08:25:49.144423+0000","last_became_peered":"2024-02-05T08:25:49.144423+0000","last_unstale":"2024-02-05T08:28:45.435136+0000","last_undegraded":"2024-02-05T08:28:45.435136+0000","last_fullsized":"2024-02-05T08:28:45.435136+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'42","last_scrub_stamp":"2024-02-05T08:28:17.495486+0000","last_deep_scrub":"425'42","last_deep_scrub_stamp":"2024-02-05T08:28:17.495486+0000","last_clean_scrub_stamp":"2024-02-05T08:28:17.495486+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:37.194565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":920,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.431032+0000","last_change":"2024-02-05T08:27:48.658294+0000","last_active":"2024-02-05T08:28:45.431032+0000","last_peered":"2024-02-05T08:28:45.431032+0000","last_clean":"2024-02-05T08:28:45.431032+0000","last_became_active":"2024-02-05T08:25:49.116332+0000","last_became_peered":"2024-02-05T08:25:49.116332+0000","last_unstale":"2024-02-05T08:28:45.431032+0000","last_undegraded":"2024-02-05T08:28:45.431032+0000","last_fullsized":"2024-02-05T08:28:45.431032+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:48.658184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:27:48.658184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:17.571240+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"82'130","reported_seq":971,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.007594+0000","last_change":"2024-02-05T08:28:09.308052+0000","last_active":"2024-02-05T08:28:46.007594+0000","last_peered":"2024-02-05T08:28:46.007594+0000","last_clean":"2024-02-05T08:28:46.007594+0000","last_became_active":"2024-02-05T08:25:52.168497+0000","last_became_peered":"2024-02-05T08:25:52.168497+0000","last_unstale":"2024-02-05T08:28:46.007594+0000","last_undegraded":"2024-02-05T08:28:46.007594+0000","last_fullsized":"2024-02-05T08:28:46.007594+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:09.307951+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:28:09.307951+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:37.753750+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":754,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434142+0000","last_change":"2024-02-05T08:28:23.347264+0000","last_active":"2024-02-05T08:28:45.434142+0000","last_peered":"2024-02-05T08:28:45.434142+0000","last_clean":"2024-02-05T08:28:45.434142+0000","last_became_active":"2024-02-05T08:28:23.346839+0000","last_became_peered":"2024-02-05T08:28:23.346839+0000","last_unstale":"2024-02-05T08:28:45.434142+0000","last_undegraded":"2024-02-05T08:28:45.434142+0000","last_fullsized":"2024-02-05T08:28:45.434142+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:01.368364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:01.368364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:06.399915+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":709,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434555+0000","last_change":"2024-02-05T08:28:23.353158+0000","last_active":"2024-02-05T08:28:45.434555+0000","last_peered":"2024-02-05T08:28:45.434555+0000","last_clean":"2024-02-05T08:28:45.434555+0000","last_became_active":"2024-02-05T08:28:23.352870+0000","last_became_peered":"2024-02-05T08:28:23.352870+0000","last_unstale":"2024-02-05T08:28:45.434555+0000","last_undegraded":"2024-02-05T08:28:45.434555+0000","last_fullsized":"2024-02-05T08:28:45.434555+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:20.607154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:28:20.607154+0000","last_clean_scrub_stamp":"2024-02-05T08:28:20.607154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:32.095919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"397'47","reported_seq":166,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435389+0000","last_change":"2024-02-05T08:27:56.383915+0000","last_active":"2024-02-05T08:28:45.435389+0000","last_peered":"2024-02-05T08:28:45.435389+0000","last_clean":"2024-02-05T08:28:45.435389+0000","last_became_active":"2024-02-05T08:25:49.143415+0000","last_became_peered":"2024-02-05T08:25:49.143415+0000","last_unstale":"2024-02-05T08:28:45.435389+0000","last_undegraded":"2024-02-05T08:28:45.435389+0000","last_fullsized":"2024-02-05T08:28:45.435389+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'47","last_scrub_stamp":"2024-02-05T08:27:56.383805+0000","last_deep_scrub":"397'47","last_deep_scrub_stamp":"2024-02-05T08:27:56.383805+0000","last_clean_scrub_stamp":"2024-02-05T08:27:56.383805+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:10.453342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":658,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.007364+0000","last_change":"2024-02-05T08:28:02.281875+0000","last_active":"2024-02-05T08:28:46.007364+0000","last_peered":"2024-02-05T08:28:46.007364+0000","last_clean":"2024-02-05T08:28:46.007364+0000","last_became_active":"2024-02-05T08:25:49.126901+0000","last_became_peered":"2024-02-05T08:25:49.126901+0000","last_unstale":"2024-02-05T08:28:46.007364+0000","last_undegraded":"2024-02-05T08:28:46.007364+0000","last_fullsized":"2024-02-05T08:28:46.007364+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:02.281787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:28:02.281787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:15.080859+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"82'130","reported_seq":953,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006031+0000","last_change":"2024-02-05T08:28:06.245990+0000","last_active":"2024-02-05T08:28:46.006031+0000","last_peered":"2024-02-05T08:28:46.006031+0000","last_clean":"2024-02-05T08:28:46.006031+0000","last_became_active":"2024-02-05T08:19:01.815446+0000","last_became_peered":"2024-02-05T08:19:01.815446+0000","last_unstale":"2024-02-05T08:28:46.006031+0000","last_undegraded":"2024-02-05T08:28:46.006031+0000","last_fullsized":"2024-02-05T08:28:46.006031+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:06.245891+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:35.974462+0000","last_clean_scrub_stamp":"2024-02-05T08:28:06.245891+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:09.347090+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434784+0000","last_change":"2024-02-05T08:27:51.319090+0000","last_active":"2024-02-05T08:28:45.434784+0000","last_peered":"2024-02-05T08:28:45.434784+0000","last_clean":"2024-02-05T08:28:45.434784+0000","last_became_active":"2024-02-05T08:25:49.133655+0000","last_became_peered":"2024-02-05T08:25:49.133655+0000","last_unstale":"2024-02-05T08:28:45.434784+0000","last_undegraded":"2024-02-05T08:28:45.434784+0000","last_fullsized":"2024-02-05T08:28:45.434784+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:51.318965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:27:51.318965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:12.413995+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":929,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.430177+0000","last_change":"2024-02-05T08:27:50.709457+0000","last_active":"2024-02-05T08:28:45.430177+0000","last_peered":"2024-02-05T08:28:45.430177+0000","last_clean":"2024-02-05T08:28:45.430177+0000","last_became_active":"2024-02-05T08:18:34.633263+0000","last_became_peered":"2024-02-05T08:18:34.633263+0000","last_unstale":"2024-02-05T08:28:45.430177+0000","last_undegraded":"2024-02-05T08:28:45.430177+0000","last_fullsized":"2024-02-05T08:28:45.430177+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:50.709314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:27:50.709314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:10.342352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"467'35","reported_seq":793,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:58.749144+0000","last_change":"2024-02-05T08:28:58.749186+0000","last_active":"2024-02-05T08:28:58.749144+0000","last_peered":"2024-02-05T08:28:58.749144+0000","last_clean":"2024-02-05T08:28:58.749144+0000","last_became_active":"2024-02-05T08:28:23.349323+0000","last_became_peered":"2024-02-05T08:28:23.349323+0000","last_unstale":"2024-02-05T08:28:58.749144+0000","last_undegraded":"2024-02-05T08:28:58.749144+0000","last_fullsized":"2024-02-05T08:28:58.749144+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'35","last_scrub_stamp":"2024-02-05T08:28:58.749084+0000","last_deep_scrub":"467'35","last_deep_scrub_stamp":"2024-02-05T08:28:58.749084+0000","last_clean_scrub_stamp":"2024-02-05T08:28:58.749084+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:06.007233+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"407'13","reported_seq":192,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.869850+0000","last_change":"2024-02-05T08:28:33.558264+0000","last_active":"2024-02-05T08:28:56.869850+0000","last_peered":"2024-02-05T08:28:56.869850+0000","last_clean":"2024-02-05T08:28:56.869850+0000","last_became_active":"2024-02-05T08:25:49.142430+0000","last_became_peered":"2024-02-05T08:25:49.142430+0000","last_unstale":"2024-02-05T08:28:56.869850+0000","last_undegraded":"2024-02-05T08:28:56.869850+0000","last_fullsized":"2024-02-05T08:28:56.869850+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'13","last_scrub_stamp":"2024-02-05T08:28:33.558130+0000","last_deep_scrub":"355'12","last_deep_scrub_stamp":"2024-02-05T08:24:38.385240+0000","last_clean_scrub_stamp":"2024-02-05T08:28:33.558130+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:48.865948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"82'130","reported_seq":676,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733287+0000","last_change":"2024-02-05T08:28:28.184075+0000","last_active":"2024-02-05T08:28:45.733287+0000","last_peered":"2024-02-05T08:28:45.733287+0000","last_clean":"2024-02-05T08:28:45.733287+0000","last_became_active":"2024-02-05T08:19:01.819856+0000","last_became_peered":"2024-02-05T08:19:01.819856+0000","last_unstale":"2024-02-05T08:28:45.733287+0000","last_undegraded":"2024-02-05T08:28:45.733287+0000","last_fullsized":"2024-02-05T08:28:45.733287+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:28.183945+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:28:28.183945+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:42.118675+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":902,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006820+0000","last_change":"2024-02-05T08:28:04.294410+0000","last_active":"2024-02-05T08:28:46.006820+0000","last_peered":"2024-02-05T08:28:46.006820+0000","last_clean":"2024-02-05T08:28:46.006820+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:28:46.006820+0000","last_undegraded":"2024-02-05T08:28:46.006820+0000","last_fullsized":"2024-02-05T08:28:46.006820+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:04.294307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:04.294307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:04.296318+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":152,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435253+0000","last_change":"2024-02-05T08:28:06.413485+0000","last_active":"2024-02-05T08:28:45.435253+0000","last_peered":"2024-02-05T08:28:45.435253+0000","last_clean":"2024-02-05T08:28:45.435253+0000","last_became_active":"2024-02-05T08:25:49.135389+0000","last_became_peered":"2024-02-05T08:25:49.135389+0000","last_unstale":"2024-02-05T08:28:45.435253+0000","last_undegraded":"2024-02-05T08:28:45.435253+0000","last_fullsized":"2024-02-05T08:28:45.435253+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:28:06.413405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:28:06.413405+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:10.481847+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"397'42","reported_seq":1004,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.065996+0000","last_change":"2024-02-05T08:28:56.066061+0000","last_active":"2024-02-05T08:28:56.065996+0000","last_peered":"2024-02-05T08:28:56.065996+0000","last_clean":"2024-02-05T08:28:56.065996+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:28:56.065996+0000","last_undegraded":"2024-02-05T08:28:56.065996+0000","last_fullsized":"2024-02-05T08:28:56.065996+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'42","last_scrub_stamp":"2024-02-05T08:28:56.065880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:56.065880+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:58.747276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"370'13","reported_seq":526,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.869832+0000","last_change":"2024-02-05T08:27:25.061394+0000","last_active":"2024-02-05T08:28:56.869832+0000","last_peered":"2024-02-05T08:28:56.869832+0000","last_clean":"2024-02-05T08:28:56.869832+0000","last_became_active":"2024-02-05T08:23:07.936347+0000","last_became_peered":"2024-02-05T08:23:07.936347+0000","last_unstale":"2024-02-05T08:28:56.869832+0000","last_undegraded":"2024-02-05T08:28:56.869832+0000","last_fullsized":"2024-02-05T08:28:56.869832+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'13","last_scrub_stamp":"2024-02-05T08:27:25.061300+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:27:25.061300+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"82'130","reported_seq":676,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735868+0000","last_change":"2024-02-05T08:28:38.235566+0000","last_active":"2024-02-05T08:28:45.735868+0000","last_peered":"2024-02-05T08:28:45.735868+0000","last_clean":"2024-02-05T08:28:45.735868+0000","last_became_active":"2024-02-05T08:19:01.816923+0000","last_became_peered":"2024-02-05T08:19:01.816923+0000","last_unstale":"2024-02-05T08:28:45.735868+0000","last_undegraded":"2024-02-05T08:28:45.735868+0000","last_fullsized":"2024-02-05T08:28:45.735868+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:38.235465+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:28:38.235465+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:02.399391+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":909,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735372+0000","last_change":"2024-02-05T08:28:11.009285+0000","last_active":"2024-02-05T08:28:45.735372+0000","last_peered":"2024-02-05T08:28:45.735372+0000","last_clean":"2024-02-05T08:28:45.735372+0000","last_became_active":"2024-02-05T08:18:34.635908+0000","last_became_peered":"2024-02-05T08:18:34.635908+0000","last_unstale":"2024-02-05T08:28:45.735372+0000","last_undegraded":"2024-02-05T08:28:45.735372+0000","last_fullsized":"2024-02-05T08:28:45.735372+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:11.009184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:08.117925+0000","last_clean_scrub_stamp":"2024-02-05T08:28:11.009184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:23.144509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":423,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733630+0000","last_change":"2024-02-05T08:27:57.218264+0000","last_active":"2024-02-05T08:28:45.733630+0000","last_peered":"2024-02-05T08:28:45.733630+0000","last_clean":"2024-02-05T08:28:45.733630+0000","last_became_active":"2024-02-05T08:25:49.127834+0000","last_became_peered":"2024-02-05T08:25:49.127834+0000","last_unstale":"2024-02-05T08:28:45.733630+0000","last_undegraded":"2024-02-05T08:28:45.733630+0000","last_fullsized":"2024-02-05T08:28:45.733630+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:27:57.218152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:27:57.218152+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:27.115332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"397'50","reported_seq":488,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.732802+0000","last_change":"2024-02-05T08:28:03.234464+0000","last_active":"2024-02-05T08:28:45.732802+0000","last_peered":"2024-02-05T08:28:45.732802+0000","last_clean":"2024-02-05T08:28:45.732802+0000","last_became_active":"2024-02-05T08:25:49.145063+0000","last_became_peered":"2024-02-05T08:25:49.145063+0000","last_unstale":"2024-02-05T08:28:45.732802+0000","last_undegraded":"2024-02-05T08:28:45.732802+0000","last_fullsized":"2024-02-05T08:28:45.732802+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'50","last_scrub_stamp":"2024-02-05T08:28:03.234373+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:28:03.234373+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:31.440155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":905,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735217+0000","last_change":"2024-02-05T08:27:46.017317+0000","last_active":"2024-02-05T08:28:45.735217+0000","last_peered":"2024-02-05T08:28:45.735217+0000","last_clean":"2024-02-05T08:28:45.735217+0000","last_became_active":"2024-02-05T08:18:34.636667+0000","last_became_peered":"2024-02-05T08:18:34.636667+0000","last_unstale":"2024-02-05T08:28:45.735217+0000","last_undegraded":"2024-02-05T08:28:45.735217+0000","last_fullsized":"2024-02-05T08:28:45.735217+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:46.017240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:46.017240+0000","last_clean_scrub_stamp":"2024-02-05T08:27:46.017240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:07.609848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'130","reported_seq":607,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:50.283456+0000","last_change":"2024-02-05T08:28:50.283486+0000","last_active":"2024-02-05T08:28:50.283456+0000","last_peered":"2024-02-05T08:28:50.283456+0000","last_clean":"2024-02-05T08:28:50.283456+0000","last_became_active":"2024-02-05T08:28:02.255460+0000","last_became_peered":"2024-02-05T08:28:02.255460+0000","last_unstale":"2024-02-05T08:28:50.283456+0000","last_undegraded":"2024-02-05T08:28:50.283456+0000","last_fullsized":"2024-02-05T08:28:50.283456+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:50.283378+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:28:50.283378+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:03.377431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":831,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429990+0000","last_change":"2024-02-05T08:28:02.280306+0000","last_active":"2024-02-05T08:28:45.429990+0000","last_peered":"2024-02-05T08:28:45.429990+0000","last_clean":"2024-02-05T08:28:45.429990+0000","last_became_active":"2024-02-05T08:28:02.279886+0000","last_became_peered":"2024-02-05T08:28:02.279886+0000","last_unstale":"2024-02-05T08:28:45.429990+0000","last_undegraded":"2024-02-05T08:28:45.429990+0000","last_fullsized":"2024-02-05T08:28:45.429990+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:00.590484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:46.039077+0000","last_clean_scrub_stamp":"2024-02-05T08:28:00.590484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:23.841998+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"407'10","reported_seq":1215,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.870614+0000","last_change":"2024-02-05T08:28:12.006316+0000","last_active":"2024-02-05T08:29:01.870614+0000","last_peered":"2024-02-05T08:29:01.870614+0000","last_clean":"2024-02-05T08:29:01.870614+0000","last_became_active":"2024-02-05T08:25:49.113977+0000","last_became_peered":"2024-02-05T08:25:49.113977+0000","last_unstale":"2024-02-05T08:29:01.870614+0000","last_undegraded":"2024-02-05T08:29:01.870614+0000","last_fullsized":"2024-02-05T08:29:01.870614+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'10","last_scrub_stamp":"2024-02-05T08:28:12.006214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:28:12.006214+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:33.082745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":400,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:54.107191+0000","last_change":"2024-02-05T08:28:54.107212+0000","last_active":"2024-02-05T08:28:54.107191+0000","last_peered":"2024-02-05T08:28:54.107191+0000","last_clean":"2024-02-05T08:28:54.107191+0000","last_became_active":"2024-02-05T08:23:07.937468+0000","last_became_peered":"2024-02-05T08:23:07.937468+0000","last_unstale":"2024-02-05T08:28:54.107191+0000","last_undegraded":"2024-02-05T08:28:54.107191+0000","last_fullsized":"2024-02-05T08:28:54.107191+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:54.107124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:28:54.107124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:16.372407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"397'39","reported_seq":882,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.738796+0000","last_change":"2024-02-05T08:28:45.738826+0000","last_active":"2024-02-05T08:28:45.738796+0000","last_peered":"2024-02-05T08:28:45.738796+0000","last_clean":"2024-02-05T08:28:45.738796+0000","last_became_active":"2024-02-05T08:23:07.939747+0000","last_became_peered":"2024-02-05T08:23:07.939747+0000","last_unstale":"2024-02-05T08:28:45.738796+0000","last_undegraded":"2024-02-05T08:28:45.738796+0000","last_fullsized":"2024-02-05T08:28:45.738796+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'39","last_scrub_stamp":"2024-02-05T08:28:45.738720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:45.738720+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:14.543320+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":399,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733383+0000","last_change":"2024-02-05T08:28:23.104290+0000","last_active":"2024-02-05T08:28:45.733383+0000","last_peered":"2024-02-05T08:28:45.733383+0000","last_clean":"2024-02-05T08:28:45.733383+0000","last_became_active":"2024-02-05T08:18:34.627261+0000","last_became_peered":"2024-02-05T08:18:34.627261+0000","last_unstale":"2024-02-05T08:28:45.733383+0000","last_undegraded":"2024-02-05T08:28:45.733383+0000","last_fullsized":"2024-02-05T08:28:45.733383+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:23.104206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:28:23.104206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:47.811650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":844,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.430962+0000","last_change":"2024-02-05T08:28:40.603832+0000","last_active":"2024-02-05T08:28:45.430962+0000","last_peered":"2024-02-05T08:28:45.430962+0000","last_clean":"2024-02-05T08:28:45.430962+0000","last_became_active":"2024-02-05T08:28:02.685842+0000","last_became_peered":"2024-02-05T08:28:02.685842+0000","last_unstale":"2024-02-05T08:28:45.430962+0000","last_undegraded":"2024-02-05T08:28:45.430962+0000","last_fullsized":"2024-02-05T08:28:45.430962+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:40.603745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:28:40.603745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:47.829322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":901,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735242+0000","last_change":"2024-02-05T08:27:36.142024+0000","last_active":"2024-02-05T08:28:45.735242+0000","last_peered":"2024-02-05T08:28:45.735242+0000","last_clean":"2024-02-05T08:28:45.735242+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:28:45.735242+0000","last_undegraded":"2024-02-05T08:28:45.735242+0000","last_fullsized":"2024-02-05T08:28:45.735242+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:36.141928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:38.141536+0000","last_clean_scrub_stamp":"2024-02-05T08:27:36.141928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:03.496668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"465'32","reported_seq":981,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735990+0000","last_change":"2024-02-05T08:28:36.264397+0000","last_active":"2024-02-05T08:28:45.735990+0000","last_peered":"2024-02-05T08:28:45.735990+0000","last_clean":"2024-02-05T08:28:45.735990+0000","last_became_active":"2024-02-05T08:23:07.937017+0000","last_became_peered":"2024-02-05T08:23:07.937017+0000","last_unstale":"2024-02-05T08:28:45.735990+0000","last_undegraded":"2024-02-05T08:28:45.735990+0000","last_fullsized":"2024-02-05T08:28:45.735990+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'32","last_scrub_stamp":"2024-02-05T08:28:36.264233+0000","last_deep_scrub":"465'32","last_deep_scrub_stamp":"2024-02-05T08:28:36.264233+0000","last_clean_scrub_stamp":"2024-02-05T08:28:36.264233+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:41.909540+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"82'130","reported_seq":607,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:54.259939+0000","last_change":"2024-02-05T08:28:54.259989+0000","last_active":"2024-02-05T08:28:54.259939+0000","last_peered":"2024-02-05T08:28:54.259939+0000","last_clean":"2024-02-05T08:28:54.259939+0000","last_became_active":"2024-02-05T08:28:02.253635+0000","last_became_peered":"2024-02-05T08:28:02.253635+0000","last_unstale":"2024-02-05T08:28:54.259939+0000","last_undegraded":"2024-02-05T08:28:54.259939+0000","last_fullsized":"2024-02-05T08:28:54.259939+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:54.259849+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:28:54.259849+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:02.635631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":846,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006186+0000","last_change":"2024-02-05T08:27:41.707412+0000","last_active":"2024-02-05T08:28:46.006186+0000","last_peered":"2024-02-05T08:28:46.006186+0000","last_clean":"2024-02-05T08:28:46.006186+0000","last_became_active":"2024-02-05T08:27:39.134979+0000","last_became_peered":"2024-02-05T08:27:39.134979+0000","last_unstale":"2024-02-05T08:28:46.006186+0000","last_undegraded":"2024-02-05T08:28:46.006186+0000","last_fullsized":"2024-02-05T08:28:46.006186+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:27:41.707282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:27:41.707282+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:06.682434+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":692,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434374+0000","last_change":"2024-02-05T08:28:23.479856+0000","last_active":"2024-02-05T08:28:45.434374+0000","last_peered":"2024-02-05T08:28:45.434374+0000","last_clean":"2024-02-05T08:28:45.434374+0000","last_became_active":"2024-02-05T08:28:23.479539+0000","last_became_peered":"2024-02-05T08:28:23.479539+0000","last_unstale":"2024-02-05T08:28:45.434374+0000","last_undegraded":"2024-02-05T08:28:45.434374+0000","last_fullsized":"2024-02-05T08:28:45.434374+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:55.984303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:27:55.984303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:07.901302+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":647,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.007469+0000","last_change":"2024-02-05T08:27:36.429731+0000","last_active":"2024-02-05T08:28:46.007469+0000","last_peered":"2024-02-05T08:28:46.007469+0000","last_clean":"2024-02-05T08:28:46.007469+0000","last_became_active":"2024-02-05T08:23:07.929749+0000","last_became_peered":"2024-02-05T08:23:07.929749+0000","last_unstale":"2024-02-05T08:28:46.007469+0000","last_undegraded":"2024-02-05T08:28:46.007469+0000","last_fullsized":"2024-02-05T08:28:46.007469+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:36.429639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:27:36.429639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:06.285395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"397'45","reported_seq":967,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:57.112277+0000","last_change":"2024-02-05T08:28:57.112311+0000","last_active":"2024-02-05T08:28:57.112277+0000","last_peered":"2024-02-05T08:28:57.112277+0000","last_clean":"2024-02-05T08:28:57.112277+0000","last_became_active":"2024-02-05T08:23:07.930130+0000","last_became_peered":"2024-02-05T08:23:07.930130+0000","last_unstale":"2024-02-05T08:28:57.112277+0000","last_undegraded":"2024-02-05T08:28:57.112277+0000","last_fullsized":"2024-02-05T08:28:57.112277+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'45","last_scrub_stamp":"2024-02-05T08:28:57.112205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:57.112205+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:24.661110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"82'130","reported_seq":1208,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005641+0000","last_change":"2024-02-05T08:28:07.143373+0000","last_active":"2024-02-05T08:28:46.005641+0000","last_peered":"2024-02-05T08:28:46.005641+0000","last_clean":"2024-02-05T08:28:46.005641+0000","last_became_active":"2024-02-05T08:19:01.816744+0000","last_became_peered":"2024-02-05T08:19:01.816744+0000","last_unstale":"2024-02-05T08:28:46.005641+0000","last_undegraded":"2024-02-05T08:28:46.005641+0000","last_fullsized":"2024-02-05T08:28:46.005641+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:07.143254+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:28:07.143254+0000","last_clean_scrub_stamp":"2024-02-05T08:28:07.143254+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:37.096677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":889,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:59.149691+0000","last_change":"2024-02-05T08:28:59.149718+0000","last_active":"2024-02-05T08:28:59.149691+0000","last_peered":"2024-02-05T08:28:59.149691+0000","last_clean":"2024-02-05T08:28:59.149691+0000","last_became_active":"2024-02-05T08:23:07.937641+0000","last_became_peered":"2024-02-05T08:23:07.937641+0000","last_unstale":"2024-02-05T08:28:59.149691+0000","last_undegraded":"2024-02-05T08:28:59.149691+0000","last_fullsized":"2024-02-05T08:28:59.149691+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:59.149580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:28:59.149580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:15.915198+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"448'11","reported_seq":1093,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.869840+0000","last_change":"2024-02-05T08:27:42.456886+0000","last_active":"2024-02-05T08:28:56.869840+0000","last_peered":"2024-02-05T08:28:56.869840+0000","last_clean":"2024-02-05T08:28:56.869840+0000","last_became_active":"2024-02-05T08:27:39.136144+0000","last_became_peered":"2024-02-05T08:27:39.136144+0000","last_unstale":"2024-02-05T08:28:56.869840+0000","last_undegraded":"2024-02-05T08:28:56.869840+0000","last_fullsized":"2024-02-05T08:28:56.869840+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'11","last_scrub_stamp":"2024-02-05T08:27:42.456809+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:27:42.456809+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:03.970199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":415,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733570+0000","last_change":"2024-02-05T08:27:43.175780+0000","last_active":"2024-02-05T08:28:45.733570+0000","last_peered":"2024-02-05T08:28:45.733570+0000","last_clean":"2024-02-05T08:28:45.733570+0000","last_became_active":"2024-02-05T08:25:49.138786+0000","last_became_peered":"2024-02-05T08:25:49.138786+0000","last_unstale":"2024-02-05T08:28:45.733570+0000","last_undegraded":"2024-02-05T08:28:45.733570+0000","last_fullsized":"2024-02-05T08:28:45.733570+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:43.175670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:27:43.175670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:51.266108+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"397'35","reported_seq":1026,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735729+0000","last_change":"2024-02-05T08:28:15.002778+0000","last_active":"2024-02-05T08:28:45.735729+0000","last_peered":"2024-02-05T08:28:45.735729+0000","last_clean":"2024-02-05T08:28:45.735729+0000","last_became_active":"2024-02-05T08:18:34.637413+0000","last_became_peered":"2024-02-05T08:18:34.637413+0000","last_unstale":"2024-02-05T08:28:45.735729+0000","last_undegraded":"2024-02-05T08:28:45.735729+0000","last_fullsized":"2024-02-05T08:28:45.735729+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'35","last_scrub_stamp":"2024-02-05T08:28:15.002667+0000","last_deep_scrub":"397'35","last_deep_scrub_stamp":"2024-02-05T08:28:15.002667+0000","last_clean_scrub_stamp":"2024-02-05T08:28:15.002667+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:28.843134+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1436,"num_read_kb":1431,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1539,"num_read_kb":1296,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":7882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1094,"ondisk_log_size":1094,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038058,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":786080,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92959072,"statfs":{"total":95995035648,"available":95190089728,"internally_reserved":0,"allocated":1073152,"data_stored":762760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":803862986},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.99299999999999999,"15min":1.347},"min":{"1min":0.38300000000000001,"5min":0.26100000000000001,"15min":0.20399999999999999},"max":{"1min":0.74099999999999999,"5min":6.5650000000000004,"15min":52.488999999999997},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.876,"15min":1.3939999999999999},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23200000000000001},"max":{"1min":0.77200000000000002,"5min":4.633,"15min":52.335000000000001},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.3779999999999999,"15min":1.248},"min":{"1min":0.39800000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.80700000000000005,"5min":31.495999999999999,"15min":31.495999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.387,"15min":1.3069999999999999},"min":{"1min":0.35999999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.71699999999999997,"5min":31.547000000000001,"15min":31.547000000000001},"last":0.621}]},{"osd":2,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":1.5549999999999999,"15min":1.288},"min":{"1min":0.34000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.749,"5min":31.585999999999999,"15min":31.585999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":1.546,"15min":1.3100000000000001},"min":{"1min":0.318,"5min":0.255,"15min":0.248},"max":{"1min":0.77000000000000002,"5min":31.614999999999998,"15min":31.614999999999998},"last":0.59899999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.9419999999999999,"15min":1.323},"min":{"1min":0.45600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.748,"5min":34.792999999999999,"15min":34.792999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":1.6910000000000001,"15min":1.246},"min":{"1min":0.372,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.78500000000000003,"5min":35.082000000000001,"15min":35.082000000000001},"last":0.56000000000000005}]},{"osd":4,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":1.1080000000000001,"15min":1.2310000000000001},"min":{"1min":0.33600000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.72099999999999997,"5min":6.5289999999999999,"15min":49.601999999999997},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.0349999999999999,"15min":1.2649999999999999},"min":{"1min":0.34899999999999998,"5min":0.30199999999999999,"15min":0.249},"max":{"1min":0.81899999999999995,"5min":11.890000000000001,"15min":49.795999999999999},"last":0.65600000000000003}]},{"osd":5,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.995,"15min":1.3109999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.25800000000000001},"max":{"1min":0.79600000000000004,"5min":9.6980000000000004,"15min":49.914000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.022,"15min":1.3420000000000001},"min":{"1min":0.39300000000000002,"5min":0.31900000000000001,"15min":0.28999999999999998},"max":{"1min":0.81399999999999995,"5min":6.5410000000000004,"15min":52.256},"last":0.68500000000000005}]},{"osd":6,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":1.0629999999999999,"15min":1.637},"min":{"1min":0.33400000000000002,"5min":0.30299999999999999,"15min":0.28599999999999998},"max":{"1min":0.83899999999999997,"5min":4.4610000000000003,"15min":84.254999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.622,"5min":1.0940000000000001,"15min":1.6699999999999999},"min":{"1min":0.45600000000000002,"5min":0.34999999999999998,"15min":0.32500000000000001},"max":{"1min":0.86399999999999999,"5min":11.866,"15min":84.667000000000002},"last":0.69999999999999996}]}]},{"osd":6,"up_from":51,"seq":219043332400,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681608,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063544,"statfs":{"total":95995035648,"available":95297069056,"internally_reserved":0,"allocated":720896,"data_stored":433155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":697235910},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.92500000000000004,"15min":1.8440000000000001},"min":{"1min":0.33900000000000002,"5min":0.23100000000000001,"15min":0.19600000000000001},"max":{"1min":0.84099999999999997,"5min":4.8040000000000003,"15min":102.819},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.73599999999999999,"15min":1.5840000000000001},"min":{"1min":0.35599999999999998,"5min":0.246,"15min":0.214},"max":{"1min":0.79100000000000004,"5min":5.6369999999999996,"15min":102.75},"last":0.54500000000000004}]},{"osd":1,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.7609999999999999,"15min":1.909},"min":{"1min":0.33500000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.92000000000000004,"5min":35.718000000000004,"15min":35.718000000000004},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":1.827,"15min":2.2450000000000001},"min":{"1min":0.38400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.89300000000000002,"5min":35.780000000000001,"15min":35.780000000000001},"last":0.46000000000000002}]},{"osd":2,"last update":"Mon Feb 5 08:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":1.002,"15min":1.1579999999999999},"min":{"1min":0.35199999999999998,"5min":0.26600000000000001,"15min":0.26000000000000001},"max":{"1min":0.752,"5min":6.6550000000000002,"15min":25.751000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.92400000000000004,"15min":1.1040000000000001},"min":{"1min":0.42499999999999999,"5min":0.26000000000000001,"15min":0.23400000000000001},"max":{"1min":0.86099999999999999,"5min":6.3949999999999996,"15min":26.210999999999999},"last":0.49199999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":1.6140000000000001,"15min":1.6499999999999999},"min":{"1min":0.39600000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73799999999999999,"5min":50.380000000000003,"15min":102.438},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":1.742,"15min":1.6970000000000001},"min":{"1min":0.40500000000000003,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80600000000000005,"5min":50.136000000000003,"15min":102.288},"last":0.51500000000000001}]},{"osd":4,"last update":"Mon Feb 5 08:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.80000000000000004,"15min":1.25},"min":{"1min":0.36699999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.78700000000000003,"5min":5.5880000000000001,"15min":46.125},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.89300000000000002,"15min":1.399},"min":{"1min":0.36299999999999999,"5min":0.29899999999999999,"15min":0.25600000000000001},"max":{"1min":0.85999999999999999,"5min":5.5039999999999996,"15min":48.817999999999998},"last":0.69599999999999995}]},{"osd":5,"last update":"Mon Feb 5 08:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.81000000000000005,"15min":1.1599999999999999},"min":{"1min":0.41899999999999998,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":0.82799999999999996,"5min":6.3209999999999997,"15min":24.675999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.89100000000000001,"15min":1.208},"min":{"1min":0.33800000000000002,"5min":0.27100000000000002,"15min":0.246},"max":{"1min":0.88400000000000001,"5min":6.415,"15min":19.132000000000001},"last":0.73999999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.76800000000000002,"15min":1.137},"min":{"1min":0.42199999999999999,"5min":0.35999999999999999,"15min":0.307},"max":{"1min":0.753,"5min":3.3959999999999999,"15min":24.728999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.0129999999999999,"15min":1.381},"min":{"1min":0.375,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":0.88600000000000001,"5min":4.6420000000000003,"15min":21.91},"last":0.76500000000000001}]}]},{"osd":1,"up_from":366,"seq":1571958030410,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":227508,"kb_used_data":748,"kb_used_omap":4,"kb_used_meta":226747,"kb_avail":93517644,"statfs":{"total":95995035648,"available":95762067456,"internally_reserved":0,"allocated":765952,"data_stored":434678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":232189606},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.89100000000000001,"15min":0.70899999999999996},"min":{"1min":0.23400000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.443,"5min":6.5999999999999996,"15min":6.5999999999999996},"last":0.308},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.98099999999999998,"15min":1.0600000000000001},"min":{"1min":0.22900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.62,"5min":10.927,"15min":10.927},"last":0.42799999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.3009999999999999,"15min":1.488},"min":{"1min":0.28100000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69799999999999995,"5min":7.3289999999999997,"15min":7.3289999999999997},"last":0.38},{"interface":"front","average":{"1min":0.36099999999999999,"5min":1.3129999999999999,"15min":1.2829999999999999},"min":{"1min":0.27800000000000002,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.46000000000000002,"5min":6.7949999999999999,"15min":6.7949999999999999},"last":0.36099999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":1.1990000000000001,"15min":1.2430000000000001},"min":{"1min":0.30199999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":7.3719999999999999,"15min":7.3719999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":1.5209999999999999,"15min":1.3049999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55300000000000005,"5min":8.6150000000000002,"15min":8.6150000000000002},"last":0.45900000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":1.335,"15min":1.286},"min":{"1min":0.40500000000000003,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77300000000000002,"5min":6.3920000000000003,"15min":6.3920000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":1.085,"15min":1.1579999999999999},"min":{"1min":0.41499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84199999999999997,"5min":7.1890000000000001,"15min":7.1890000000000001},"last":0.52900000000000003}]},{"osd":5,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.2889999999999999,"15min":1.1619999999999999},"min":{"1min":0.433,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83099999999999996,"5min":8.6999999999999993,"15min":8.6999999999999993},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.298,"15min":1.1479999999999999},"min":{"1min":0.41499999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.80800000000000005,"5min":6.4240000000000004,"15min":6.4240000000000004},"last":0.44500000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":1.698,"15min":1.3660000000000001},"min":{"1min":0.44800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.89500000000000002,"5min":42.783000000000001,"15min":42.783000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":1.7050000000000001,"15min":1.3839999999999999},"min":{"1min":0.438,"5min":0.372,"15min":0.372},"max":{"1min":0.82899999999999996,"5min":42.898000000000003,"15min":42.898000000000003},"last":0.498}]},{"osd":7,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.8160000000000001,"15min":1.4019999999999999},"min":{"1min":0.40600000000000003,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84999999999999998,"5min":64.049999999999997,"15min":64.049999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":2.04,"15min":1.6040000000000001},"min":{"1min":0.41199999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85199999999999998,"5min":64.144000000000005,"15min":64.144000000000005},"last":0.54900000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738702,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852388,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892764,"statfs":{"total":95995035648,"available":95122190336,"internally_reserved":0,"allocated":749568,"data_stored":434823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:28:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.86399999999999999,"15min":0.78200000000000003},"min":{"1min":0.21299999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.47799999999999998,"5min":5.1970000000000001,"15min":5.1970000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.999,"15min":0.90000000000000002},"min":{"1min":0.218,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.55600000000000005,"5min":6.718,"15min":6.718},"last":0.40899999999999997}]},{"osd":2,"last update":"Mon Feb 5 08:28:09 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.82799999999999996,"15min":1.4039999999999999},"min":{"1min":0.23599999999999999,"5min":0.22500000000000001,"15min":0.161},"max":{"1min":0.57599999999999996,"5min":6.9889999999999999,"15min":105.96599999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.98399999999999999,"15min":1.377},"min":{"1min":0.28299999999999997,"5min":0.252,"15min":0.16600000000000001},"max":{"1min":0.60899999999999999,"5min":6.8849999999999998,"15min":106.137},"last":0.56799999999999995}]},{"osd":3,"last update":"Mon Feb 5 08:28:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.98599999999999999,"15min":0.97899999999999998},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.20899999999999999},"max":{"1min":0.53500000000000003,"5min":6.6849999999999996,"15min":7.3929999999999998},"last":0.435},{"interface":"front","average":{"1min":0.43099999999999999,"5min":1.04,"15min":0.95799999999999996},"min":{"1min":0.29299999999999998,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.58099999999999996,"5min":6.056,"15min":8.0239999999999991},"last":0.499}]},{"osd":4,"last update":"Mon Feb 5 08:28:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.4510000000000001,"15min":1.4550000000000001},"min":{"1min":0.40500000000000003,"5min":0.30099999999999999,"15min":0.27700000000000002},"max":{"1min":0.78400000000000003,"5min":59.350999999999999,"15min":59.350999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.627,"5min":1.4339999999999999,"15min":1.476},"min":{"1min":0.44,"5min":0.30299999999999999,"15min":0.254},"max":{"1min":0.80900000000000005,"5min":59.299999999999997,"15min":59.299999999999997},"last":0.63}]},{"osd":5,"last update":"Mon Feb 5 08:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.0800000000000001,"15min":1.5229999999999999},"min":{"1min":0.41099999999999998,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.78600000000000003,"5min":15.914,"15min":38.979999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.97899999999999998,"15min":1.4810000000000001},"min":{"1min":0.33700000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.83899999999999997,"5min":15.887,"15min":38.960999999999999},"last":0.59499999999999997}]},{"osd":6,"last update":"Mon Feb 5 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.163,"15min":1.21},"min":{"1min":0.48299999999999998,"5min":0.34000000000000002,"15min":0.30199999999999999},"max":{"1min":0.80600000000000005,"5min":16.725999999999999,"15min":16.725999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.64400000000000002,"5min":1.294,"15min":1.206},"min":{"1min":0.496,"5min":0.32900000000000001,"15min":0.27100000000000002},"max":{"1min":0.82799999999999996,"5min":16.803999999999998,"15min":16.803999999999998},"last":0.64600000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:28:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.151,"15min":1.6779999999999999},"min":{"1min":0.46100000000000002,"5min":0.32900000000000001,"15min":0.29299999999999998},"max":{"1min":0.84499999999999997,"5min":16.007000000000001,"15min":68.241},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.629,"5min":1.232,"15min":1.7490000000000001},"min":{"1min":0.47099999999999997,"5min":0.36799999999999999,"15min":0.252},"max":{"1min":0.82399999999999995,"5min":15.952999999999999,"15min":68.980000000000004},"last":0.53300000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411652,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790180,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":92954972,"statfs":{"total":95995035648,"available":95185891328,"internally_reserved":0,"allocated":1011712,"data_stored":696566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":808122822},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.94499999999999995,"15min":1.4419999999999999},"min":{"1min":0.20599999999999999,"5min":0.20100000000000001,"15min":0.17999999999999999},"max":{"1min":0.46700000000000003,"5min":7.4900000000000002,"15min":107.155},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.79900000000000004,"15min":1.571},"min":{"1min":0.246,"5min":0.20399999999999999,"15min":0.20000000000000001},"max":{"1min":0.71099999999999997,"5min":7.8410000000000002,"15min":107.221},"last":0.30199999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":1.3169999999999999,"15min":1.0780000000000001},"min":{"1min":0.29399999999999998,"5min":0.19,"15min":0.19},"max":{"1min":0.54600000000000004,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":1.121,"15min":1.2210000000000001},"min":{"1min":0.24399999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.45400000000000001,"5min":7.0709999999999997,"15min":7.0709999999999997},"last":0.38100000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":1.274,"15min":1.069},"min":{"1min":0.27200000000000002,"5min":0.21199999999999999,"15min":0.20999999999999999},"max":{"1min":0.56899999999999995,"5min":11.122,"15min":11.122},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":1.4179999999999999,"15min":0.98299999999999998},"min":{"1min":0.313,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":0.63300000000000001,"5min":7.8289999999999997,"15min":7.8289999999999997},"last":0.36699999999999999}]},{"osd":4,"last update":"Mon Feb 5 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":1.038,"15min":1.2190000000000001},"min":{"1min":0.47599999999999998,"5min":0.27600000000000002,"15min":0.26400000000000001},"max":{"1min":0.77000000000000002,"5min":3.7490000000000001,"15min":8.2620000000000005},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":1.1699999999999999,"15min":1.236},"min":{"1min":0.44900000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.80200000000000005,"5min":10.307,"15min":10.307},"last":0.57499999999999996}]},{"osd":5,"last update":"Mon Feb 5 08:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":1.0529999999999999,"15min":1.3420000000000001},"min":{"1min":0.48699999999999999,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":0.72799999999999998,"5min":7.3899999999999997,"15min":26.373000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":1.0900000000000001,"15min":1.2789999999999999},"min":{"1min":0.45400000000000001,"5min":0.32900000000000001,"15min":0.26900000000000002},"max":{"1min":0.84099999999999997,"5min":11.308999999999999,"15min":27.968},"last":0.47499999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.351,"15min":1.3100000000000001},"min":{"1min":0.42499999999999999,"5min":0.32700000000000001,"15min":0.29199999999999998},"max":{"1min":0.85199999999999998,"5min":21.885000000000002,"15min":21.885000000000002},"last":0.499},{"interface":"front","average":{"1min":0.61599999999999999,"5min":1.3839999999999999,"15min":1.27},"min":{"1min":0.432,"5min":0.40600000000000003,"15min":0.317},"max":{"1min":0.872,"5min":22.158000000000001,"15min":22.158000000000001},"last":0.53100000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":2.0369999999999999,"15min":1.45},"min":{"1min":0.45300000000000001,"5min":0.33900000000000002,"15min":0.28899999999999998},"max":{"1min":0.76200000000000001,"5min":72.334999999999994,"15min":72.334999999999994},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":2.105,"15min":1.6950000000000001},"min":{"1min":0.41899999999999998,"5min":0.40200000000000002,"15min":0.32700000000000001},"max":{"1min":0.89000000000000001,"5min":72.222999999999999,"15min":72.222999999999999},"last":0.60399999999999998}]}]},{"osd":3,"up_from":27,"seq":115964117311,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704444,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93040708,"statfs":{"total":95995035648,"available":95273684992,"internally_reserved":0,"allocated":1363968,"data_stored":1025699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":719976905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.83899999999999997,"15min":1.081},"min":{"1min":0.27200000000000002,"5min":0.20300000000000001,"15min":0.19700000000000001},"max":{"1min":0.46899999999999997,"5min":8.4700000000000006,"15min":52.831000000000003},"last":0.312},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.92400000000000004,"15min":1.115},"min":{"1min":0.22600000000000001,"5min":0.218,"15min":0.17699999999999999},"max":{"1min":0.55500000000000005,"5min":10.613,"15min":53.859999999999999},"last":0.36799999999999999}]},{"osd":1,"last update":"Mon Feb 5 08:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":1.3580000000000001,"15min":1.3999999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.63500000000000001,"5min":11.339,"15min":11.339},"last":0.252},{"interface":"front","average":{"1min":0.39000000000000001,"5min":1.4870000000000001,"15min":1.522},"min":{"1min":0.309,"5min":0.25,"15min":0.25},"max":{"1min":0.51100000000000001,"5min":10.744999999999999,"15min":10.744999999999999},"last":0.32100000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":1.1359999999999999,"15min":1.1100000000000001},"min":{"1min":0.246,"5min":0.218,"15min":0.218},"max":{"1min":0.441,"5min":6.5999999999999996,"15min":35.277000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":1.3300000000000001,"15min":1.141},"min":{"1min":0.33600000000000002,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":0.53400000000000003,"5min":8.6039999999999992,"15min":35.442999999999998},"last":0.34399999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.216,"15min":1.2569999999999999},"min":{"1min":0.47199999999999998,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":0.81399999999999995,"5min":8.2140000000000004,"15min":54.502000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":1.167,"15min":1.161},"min":{"1min":0.44900000000000001,"5min":0.33500000000000002,"15min":0.29199999999999998},"max":{"1min":0.89600000000000002,"5min":10.912000000000001,"15min":54.540999999999997},"last":0.498}]},{"osd":5,"last update":"Mon Feb 5 08:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":1.216,"15min":1.744},"min":{"1min":0.40799999999999997,"5min":0.308,"15min":0.28899999999999998},"max":{"1min":0.78400000000000003,"5min":8.2609999999999992,"15min":117.432},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.0109999999999999,"15min":1.569},"min":{"1min":0.39900000000000002,"5min":0.36699999999999999,"15min":0.27600000000000002},"max":{"1min":0.73599999999999999,"5min":4.0460000000000003,"15min":114.828},"last":0.442}]},{"osd":6,"last update":"Mon Feb 5 08:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.363,"15min":1.2749999999999999},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.29899999999999999},"max":{"1min":0.85499999999999998,"5min":10.534000000000001,"15min":54.219999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.153,"15min":1.266},"min":{"1min":0.434,"5min":0.318,"15min":0.29099999999999998},"max":{"1min":0.76900000000000002,"5min":6.6820000000000004,"15min":54.530999999999999},"last":0.52500000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":1.2290000000000001,"15min":1.6930000000000001},"min":{"1min":0.45700000000000002,"5min":0.36899999999999999,"15min":0.33500000000000002},"max":{"1min":0.877,"5min":8.4060000000000006,"15min":114.84999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.63,"5min":1.3700000000000001,"15min":1.647},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.36399999999999999},"max":{"1min":0.78100000000000003,"5min":10.58,"15min":114.871},"last":0.41599999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953787,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762148,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92983004,"statfs":{"total":95995035648,"available":95214596096,"internally_reserved":0,"allocated":749568,"data_stored":433727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.79000000000000004,"15min":1.1819999999999999},"min":{"1min":0.34100000000000003,"5min":0.23599999999999999,"15min":0.192},"max":{"1min":0.745,"5min":4.54,"15min":39.552},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.998,"15min":1.3620000000000001},"min":{"1min":0.41499999999999998,"5min":0.27700000000000002,"15min":0.23499999999999999},"max":{"1min":0.73299999999999998,"5min":5.593,"15min":39.350999999999999},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.89300000000000002,"15min":0.97299999999999998},"min":{"1min":0.35999999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.76800000000000002,"5min":4.9930000000000003,"15min":4.9930000000000003},"last":0.751},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.141,"15min":1.0229999999999999},"min":{"1min":0.35399999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.89300000000000002,"5min":9.8569999999999993,"15min":9.8569999999999993},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Feb 5 08:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":1.022,"15min":1.0800000000000001},"min":{"1min":0.39800000000000002,"5min":0.28299999999999997,"15min":0.23200000000000001},"max":{"1min":0.68000000000000005,"5min":6.1749999999999998,"15min":6.1749999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.59099999999999997,"5min":1.0880000000000001,"15min":1.1240000000000001},"min":{"1min":0.45800000000000002,"5min":0.27700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69899999999999995,"5min":8.8490000000000002,"15min":8.8490000000000002},"last":0.73499999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.53,"15min":1.4850000000000001},"min":{"1min":0.39700000000000002,"5min":0.32200000000000001,"15min":0.312},"max":{"1min":0.74399999999999999,"5min":29.905000000000001,"15min":38.985999999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":1.3440000000000001,"15min":1.3360000000000001},"min":{"1min":0.48399999999999999,"5min":0.30399999999999999,"15min":0.27100000000000002},"max":{"1min":0.80700000000000005,"5min":30.094999999999999,"15min":39.290999999999997},"last":0.79200000000000004}]},{"osd":5,"last update":"Mon Feb 5 08:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.85899999999999999,"15min":1.2490000000000001},"min":{"1min":0.32700000000000001,"5min":0.26100000000000001,"15min":0.24199999999999999},"max":{"1min":0.86199999999999999,"5min":5.468,"15min":34.167000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.76300000000000001,"15min":1.2629999999999999},"min":{"1min":0.34999999999999998,"5min":0.28299999999999997,"15min":0.24199999999999999},"max":{"1min":0.97999999999999998,"5min":4.4189999999999996,"15min":36.051000000000002},"last":0.434}]},{"osd":6,"last update":"Mon Feb 5 08:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.97599999999999998,"15min":1.1870000000000001},"min":{"1min":0.40000000000000002,"5min":0.32200000000000001,"15min":0.26900000000000002},"max":{"1min":0.72599999999999998,"5min":4.3879999999999999,"15min":6.375},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.96999999999999997,"15min":1.103},"min":{"1min":0.45300000000000001,"5min":0.29099999999999998,"15min":0.26100000000000001},"max":{"1min":0.81999999999999995,"5min":6.0179999999999998,"15min":6.9690000000000003},"last":0.60399999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:29:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.93700000000000006,"15min":1.169},"min":{"1min":0.41299999999999998,"5min":0.32100000000000001,"15min":0.307},"max":{"1min":0.76500000000000001,"5min":5.8689999999999998,"15min":12.683999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.82799999999999996,"15min":1.127},"min":{"1min":0.48899999999999999,"5min":0.31,"15min":0.27900000000000003},"max":{"1min":0.80900000000000005,"5min":3.2730000000000001,"15min":7.6820000000000004},"last":0.49299999999999999}]}]},{"osd":5,"up_from":42,"seq":180388626740,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762712,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92982440,"statfs":{"total":95995035648,"available":95214018560,"internally_reserved":0,"allocated":1327104,"data_stored":1024356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.76000000000000001,"15min":1.2869999999999999},"min":{"1min":0.34100000000000003,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.69999999999999996,"5min":4.7290000000000001,"15min":30.065999999999999},"last":0.5},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.84499999999999997,"15min":1.246},"min":{"1min":0.41699999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.68100000000000005,"5min":5.4169999999999998,"15min":30.173999999999999},"last":0.59799999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.863,"15min":2.3940000000000001},"min":{"1min":0.36299999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71499999999999997,"5min":26.98,"15min":26.98},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":1.847,"15min":2.3220000000000001},"min":{"1min":0.46500000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71599999999999997,"5min":27.474,"15min":27.474},"last":0.42899999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.78800000000000003,"15min":1.1080000000000001},"min":{"1min":0.34399999999999997,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.73299999999999998,"5min":3.5910000000000002,"15min":17.09},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.0109999999999999,"15min":1.248},"min":{"1min":0.38700000000000001,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":0.73799999999999999,"5min":3.9359999999999999,"15min":17.515999999999998},"last":0.55800000000000005}]},{"osd":3,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":1.841,"15min":1.393},"min":{"1min":0.41399999999999998,"5min":0.27300000000000002,"15min":0.26900000000000002},"max":{"1min":0.81399999999999995,"5min":27.725000000000001,"15min":27.725000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.708,"15min":1.446},"min":{"1min":0.376,"5min":0.307,"15min":0.28100000000000003},"max":{"1min":0.74299999999999999,"5min":27.241,"15min":27.241},"last":0.626}]},{"osd":4,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.72299999999999998,"15min":1.0529999999999999},"min":{"1min":0.29299999999999998,"5min":0.253,"15min":0.23699999999999999},"max":{"1min":0.64900000000000002,"5min":5.375,"15min":7.7389999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.84499999999999997,"15min":1.107},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.23300000000000001},"max":{"1min":0.746,"5min":6.0979999999999999,"15min":7.609},"last":0.57899999999999996}]},{"osd":6,"last update":"Mon Feb 5 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.81599999999999995,"15min":1.232},"min":{"1min":0.38200000000000001,"5min":0.28599999999999998,"15min":0.26600000000000001},"max":{"1min":0.73799999999999999,"5min":5.1989999999999998,"15min":30.581},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.84899999999999998,"15min":1.2250000000000001},"min":{"1min":0.505,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":0.83699999999999997,"5min":5.1909999999999998,"15min":30.620999999999999},"last":0.78500000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.82299999999999995,"15min":1.3620000000000001},"min":{"1min":0.35999999999999999,"5min":0.30499999999999999,"15min":0.28799999999999998},"max":{"1min":0.874,"5min":4.7960000000000003,"15min":34.695},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.84199999999999997,"15min":1.2949999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.308},"max":{"1min":0.82399999999999995,"5min":5.444,"15min":34.514000000000003},"last":0.46100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:29:09.677 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T08:29:09.919 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:29:10.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[30086]: from='client.16476 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:29:10.353 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[30086]: 3.a scrub starts 2024-02-05T08:29:10.354 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[30086]: 3.a scrub ok 2024-02-05T08:29:10.354 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[30086]: 6.3 scrub starts 2024-02-05T08:29:10.354 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[30086]: 3.1c scrub starts 2024-02-05T08:29:10.354 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[30086]: 3.1c scrub ok 2024-02-05T08:29:10.354 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[30086]: 6.3 scrub ok 2024-02-05T08:29:10.354 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[39278]: from='client.16476 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:29:10.354 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[39278]: 3.a scrub starts 2024-02-05T08:29:10.354 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[39278]: 3.a scrub ok 2024-02-05T08:29:10.354 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[39278]: 6.3 scrub starts 2024-02-05T08:29:10.354 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[39278]: 3.1c scrub starts 2024-02-05T08:29:10.354 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[39278]: 3.1c scrub ok 2024-02-05T08:29:10.354 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:10 smithi134 ceph-mon[39278]: 6.3 scrub ok 2024-02-05T08:29:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:10 smithi196 ceph-mon[33018]: from='client.16476 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:29:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:10 smithi196 ceph-mon[33018]: 3.a scrub starts 2024-02-05T08:29:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:10 smithi196 ceph-mon[33018]: 3.a scrub ok 2024-02-05T08:29:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:10 smithi196 ceph-mon[33018]: 6.3 scrub starts 2024-02-05T08:29:10.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:10 smithi196 ceph-mon[33018]: 3.1c scrub starts 2024-02-05T08:29:10.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:10 smithi196 ceph-mon[33018]: 3.1c scrub ok 2024-02-05T08:29:10.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:10 smithi196 ceph-mon[33018]: 6.3 scrub ok 2024-02-05T08:29:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[30086]: 3.19 scrub starts 2024-02-05T08:29:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[30086]: 3.19 scrub ok 2024-02-05T08:29:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[30086]: pgmap v1102: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:29:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[30086]: 4.a scrub starts 2024-02-05T08:29:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[30086]: 4.a scrub ok 2024-02-05T08:29:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[30086]: 6.1b scrub starts 2024-02-05T08:29:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[30086]: 6.1b scrub ok 2024-02-05T08:29:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:29:11.423 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:29:11.424 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[39278]: 3.19 scrub starts 2024-02-05T08:29:11.424 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[39278]: 3.19 scrub ok 2024-02-05T08:29:11.424 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[39278]: pgmap v1102: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:29:11.424 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[39278]: 4.a scrub starts 2024-02-05T08:29:11.424 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[39278]: 4.a scrub ok 2024-02-05T08:29:11.424 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[39278]: 6.1b scrub starts 2024-02-05T08:29:11.424 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[39278]: 6.1b scrub ok 2024-02-05T08:29:11.424 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:29:11.424 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:11 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:29:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:11 smithi196 ceph-mon[33018]: 3.19 scrub starts 2024-02-05T08:29:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:11 smithi196 ceph-mon[33018]: 3.19 scrub ok 2024-02-05T08:29:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:11 smithi196 ceph-mon[33018]: pgmap v1102: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T08:29:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:11 smithi196 ceph-mon[33018]: 4.a scrub starts 2024-02-05T08:29:11.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:11 smithi196 ceph-mon[33018]: 4.a scrub ok 2024-02-05T08:29:11.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:11 smithi196 ceph-mon[33018]: 6.1b scrub starts 2024-02-05T08:29:11.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:11 smithi196 ceph-mon[33018]: 6.1b scrub ok 2024-02-05T08:29:11.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:11 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:29:11.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:11 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:29:11.894 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-02-05T08:29:11.896 INFO:teuthology.orchestra.run.smithi134.stderr:dumped all 2024-02-05T08:29:12.264 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:12 smithi134 ceph-mon[30086]: 5.0 scrub starts 2024-02-05T08:29:12.264 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:12 smithi134 ceph-mon[30086]: 5.0 scrub ok 2024-02-05T08:29:12.265 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:12 smithi134 ceph-mon[30086]: osdmap e468: 8 total, 8 up, 8 in 2024-02-05T08:29:12.265 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:12 smithi134 ceph-mon[39278]: 5.0 scrub starts 2024-02-05T08:29:12.265 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:12 smithi134 ceph-mon[39278]: 5.0 scrub ok 2024-02-05T08:29:12.265 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:12 smithi134 ceph-mon[39278]: osdmap e468: 8 total, 8 up, 8 in 2024-02-05T08:29:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:12 smithi196 ceph-mon[33018]: 5.0 scrub starts 2024-02-05T08:29:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:12 smithi196 ceph-mon[33018]: 5.0 scrub ok 2024-02-05T08:29:12.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:12 smithi196 ceph-mon[33018]: osdmap e468: 8 total, 8 up, 8 in 2024-02-05T08:29:12.693 INFO:teuthology.orchestra.run.smithi134.stdout:{"pg_ready":true,"pg_map":{"version":1103,"stamp":"2024-02-05T08:29:11.118406+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2997,"num_read_kb":2749,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":412,"num_bytes_recovered":12066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2636,"ondisk_log_size":2636,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5567068,"kb_used_data":7580,"kb_used_omap":15,"kb_used_meta":5559408,"kb_avail":744394148,"statfs":{"total":767960285184,"available":762259607552,"internally_reserved":0,"allocated":7761920,"data_stored":5245764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15596,"internal_metadata":5692834580},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.109771"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":896,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.125062+0000","last_change":"2024-02-05T08:29:01.125104+0000","last_active":"2024-02-05T08:29:01.125062+0000","last_peered":"2024-02-05T08:29:01.125062+0000","last_clean":"2024-02-05T08:29:01.125062+0000","last_became_active":"2024-02-05T08:23:07.931642+0000","last_became_peered":"2024-02-05T08:23:07.931642+0000","last_unstale":"2024-02-05T08:29:01.125062+0000","last_undegraded":"2024-02-05T08:29:01.125062+0000","last_fullsized":"2024-02-05T08:29:01.125062+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:01.124974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:29:01.124974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:04.280969+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":696,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.433766+0000","last_change":"2024-02-05T08:28:24.058083+0000","last_active":"2024-02-05T08:28:45.433766+0000","last_peered":"2024-02-05T08:28:45.433766+0000","last_clean":"2024-02-05T08:28:45.433766+0000","last_became_active":"2024-02-05T08:28:24.057847+0000","last_became_peered":"2024-02-05T08:28:24.057847+0000","last_unstale":"2024-02-05T08:28:45.433766+0000","last_undegraded":"2024-02-05T08:28:45.433766+0000","last_fullsized":"2024-02-05T08:28:45.433766+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:50.062807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:27:50.062807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:08.748754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:51.465632+0000","last_change":"2024-02-05T08:28:51.465653+0000","last_active":"2024-02-05T08:28:51.465632+0000","last_peered":"2024-02-05T08:28:51.465632+0000","last_clean":"2024-02-05T08:28:51.465632+0000","last_became_active":"2024-02-05T08:25:49.142192+0000","last_became_peered":"2024-02-05T08:25:49.142192+0000","last_unstale":"2024-02-05T08:28:51.465632+0000","last_undegraded":"2024-02-05T08:28:51.465632+0000","last_fullsized":"2024-02-05T08:28:51.465632+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:51.465566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:28:51.465566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:04.377005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"397'25","reported_seq":967,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735128+0000","last_change":"2024-02-05T08:28:45.204298+0000","last_active":"2024-02-05T08:28:45.735128+0000","last_peered":"2024-02-05T08:28:45.735128+0000","last_clean":"2024-02-05T08:28:45.735128+0000","last_became_active":"2024-02-05T08:23:07.937041+0000","last_became_peered":"2024-02-05T08:23:07.937041+0000","last_unstale":"2024-02-05T08:28:45.735128+0000","last_undegraded":"2024-02-05T08:28:45.735128+0000","last_fullsized":"2024-02-05T08:28:45.735128+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-02-05T08:28:45.204152+0000","last_deep_scrub":"397'25","last_deep_scrub_stamp":"2024-02-05T08:28:45.204152+0000","last_clean_scrub_stamp":"2024-02-05T08:28:45.204152+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:51.623766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":391,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733280+0000","last_change":"2024-02-05T08:27:23.238054+0000","last_active":"2024-02-05T08:28:45.733280+0000","last_peered":"2024-02-05T08:28:45.733280+0000","last_clean":"2024-02-05T08:28:45.733280+0000","last_became_active":"2024-02-05T08:18:34.633083+0000","last_became_peered":"2024-02-05T08:18:34.633083+0000","last_unstale":"2024-02-05T08:28:45.733280+0000","last_undegraded":"2024-02-05T08:28:45.733280+0000","last_fullsized":"2024-02-05T08:28:45.733280+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:23.237950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:22.548159+0000","last_clean_scrub_stamp":"2024-02-05T08:27:23.237950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":905,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735296+0000","last_change":"2024-02-05T08:28:23.102978+0000","last_active":"2024-02-05T08:28:45.735296+0000","last_peered":"2024-02-05T08:28:45.735296+0000","last_clean":"2024-02-05T08:28:45.735296+0000","last_became_active":"2024-02-05T08:23:07.937180+0000","last_became_peered":"2024-02-05T08:23:07.937180+0000","last_unstale":"2024-02-05T08:28:45.735296+0000","last_undegraded":"2024-02-05T08:28:45.735296+0000","last_fullsized":"2024-02-05T08:28:45.735296+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:23.102838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:41.460468+0000","last_clean_scrub_stamp":"2024-02-05T08:28:23.102838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:42.009179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":401,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733658+0000","last_change":"2024-02-05T08:27:39.196102+0000","last_active":"2024-02-05T08:28:45.733658+0000","last_peered":"2024-02-05T08:28:45.733658+0000","last_clean":"2024-02-05T08:28:45.733658+0000","last_became_active":"2024-02-05T08:21:41.880038+0000","last_became_peered":"2024-02-05T08:21:41.880038+0000","last_unstale":"2024-02-05T08:28:45.733658+0000","last_undegraded":"2024-02-05T08:28:45.733658+0000","last_fullsized":"2024-02-05T08:28:45.733658+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:39.196031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:39.196031+0000","last_clean_scrub_stamp":"2024-02-05T08:27:39.196031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:02.870660+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"397'46","reported_seq":477,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.732973+0000","last_change":"2024-02-05T08:27:45.248656+0000","last_active":"2024-02-05T08:28:45.732973+0000","last_peered":"2024-02-05T08:28:45.732973+0000","last_clean":"2024-02-05T08:28:45.732973+0000","last_became_active":"2024-02-05T08:25:49.140432+0000","last_became_peered":"2024-02-05T08:25:49.140432+0000","last_unstale":"2024-02-05T08:28:45.732973+0000","last_undegraded":"2024-02-05T08:28:45.732973+0000","last_fullsized":"2024-02-05T08:28:45.732973+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-02-05T08:27:45.248551+0000","last_deep_scrub":"118'22","last_deep_scrub_stamp":"2024-02-05T08:07:45.819004+0000","last_clean_scrub_stamp":"2024-02-05T08:27:45.248551+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:09.415834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434897+0000","last_change":"2024-02-05T08:28:01.438689+0000","last_active":"2024-02-05T08:28:45.434897+0000","last_peered":"2024-02-05T08:28:45.434897+0000","last_clean":"2024-02-05T08:28:45.434897+0000","last_became_active":"2024-02-05T08:25:49.139935+0000","last_became_peered":"2024-02-05T08:25:49.139935+0000","last_unstale":"2024-02-05T08:28:45.434897+0000","last_undegraded":"2024-02-05T08:28:45.434897+0000","last_fullsized":"2024-02-05T08:28:45.434897+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:01.438584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:01.438584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:28.393732+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":919,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429724+0000","last_change":"2024-02-05T08:28:00.828354+0000","last_active":"2024-02-05T08:28:45.429724+0000","last_peered":"2024-02-05T08:28:45.429724+0000","last_clean":"2024-02-05T08:28:45.429724+0000","last_became_active":"2024-02-05T08:25:49.116678+0000","last_became_peered":"2024-02-05T08:25:49.116678+0000","last_unstale":"2024-02-05T08:28:45.429724+0000","last_undegraded":"2024-02-05T08:28:45.429724+0000","last_fullsized":"2024-02-05T08:28:45.429724+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:00.828248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:02.192889+0000","last_clean_scrub_stamp":"2024-02-05T08:28:00.828248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:23.104062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":851,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.430025+0000","last_change":"2024-02-05T08:28:13.604842+0000","last_active":"2024-02-05T08:28:45.430025+0000","last_peered":"2024-02-05T08:28:45.430025+0000","last_clean":"2024-02-05T08:28:45.430025+0000","last_became_active":"2024-02-05T08:28:02.684565+0000","last_became_peered":"2024-02-05T08:28:02.684565+0000","last_unstale":"2024-02-05T08:28:45.430025+0000","last_undegraded":"2024-02-05T08:28:45.430025+0000","last_fullsized":"2024-02-05T08:28:45.430025+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:13.604741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:32.421388+0000","last_clean_scrub_stamp":"2024-02-05T08:28:13.604741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:15.015055+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"397'25","reported_seq":1003,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735541+0000","last_change":"2024-02-05T08:27:59.963350+0000","last_active":"2024-02-05T08:28:45.735541+0000","last_peered":"2024-02-05T08:28:45.735541+0000","last_clean":"2024-02-05T08:28:45.735541+0000","last_became_active":"2024-02-05T08:25:49.126860+0000","last_became_peered":"2024-02-05T08:25:49.126860+0000","last_unstale":"2024-02-05T08:28:45.735541+0000","last_undegraded":"2024-02-05T08:28:45.735541+0000","last_fullsized":"2024-02-05T08:28:45.735541+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-02-05T08:27:59.963221+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:16:07.826018+0000","last_clean_scrub_stamp":"2024-02-05T08:27:59.963221+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:10.426288+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":909,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429202+0000","last_change":"2024-02-05T08:28:28.770061+0000","last_active":"2024-02-05T08:28:45.429202+0000","last_peered":"2024-02-05T08:28:45.429202+0000","last_clean":"2024-02-05T08:28:45.429202+0000","last_became_active":"2024-02-05T08:23:07.932548+0000","last_became_peered":"2024-02-05T08:23:07.932548+0000","last_unstale":"2024-02-05T08:28:45.429202+0000","last_undegraded":"2024-02-05T08:28:45.429202+0000","last_fullsized":"2024-02-05T08:28:45.429202+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:28.769944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:28.769944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:49.467708+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":909,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:03.064061+0000","last_change":"2024-02-05T08:29:03.064085+0000","last_active":"2024-02-05T08:29:03.064061+0000","last_peered":"2024-02-05T08:29:03.064061+0000","last_clean":"2024-02-05T08:29:03.064061+0000","last_became_active":"2024-02-05T08:23:07.939630+0000","last_became_peered":"2024-02-05T08:23:07.939630+0000","last_unstale":"2024-02-05T08:29:03.064061+0000","last_undegraded":"2024-02-05T08:29:03.064061+0000","last_fullsized":"2024-02-05T08:29:03.064061+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:03.063995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:29.345593+0000","last_clean_scrub_stamp":"2024-02-05T08:29:03.063995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:10.318219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":832,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429921+0000","last_change":"2024-02-05T08:28:02.686352+0000","last_active":"2024-02-05T08:28:45.429921+0000","last_peered":"2024-02-05T08:28:45.429921+0000","last_clean":"2024-02-05T08:28:45.429921+0000","last_became_active":"2024-02-05T08:28:02.685758+0000","last_became_peered":"2024-02-05T08:28:02.685758+0000","last_unstale":"2024-02-05T08:28:45.429921+0000","last_undegraded":"2024-02-05T08:28:45.429921+0000","last_fullsized":"2024-02-05T08:28:45.429921+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:51.008413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:51.008413+0000","last_clean_scrub_stamp":"2024-02-05T08:27:51.008413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:03.625376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"397'9","reported_seq":936,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:51.122485+0000","last_change":"2024-02-05T08:28:51.122531+0000","last_active":"2024-02-05T08:28:51.122485+0000","last_peered":"2024-02-05T08:28:51.122485+0000","last_clean":"2024-02-05T08:28:51.122485+0000","last_became_active":"2024-02-05T08:23:07.938087+0000","last_became_peered":"2024-02-05T08:23:07.938087+0000","last_unstale":"2024-02-05T08:28:51.122485+0000","last_undegraded":"2024-02-05T08:28:51.122485+0000","last_fullsized":"2024-02-05T08:28:51.122485+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'9","last_scrub_stamp":"2024-02-05T08:28:51.122405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:51.122405+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:05.292307+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":902,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.736116+0000","last_change":"2024-02-05T08:27:41.138216+0000","last_active":"2024-02-05T08:28:45.736116+0000","last_peered":"2024-02-05T08:28:45.736116+0000","last_clean":"2024-02-05T08:28:45.736116+0000","last_became_active":"2024-02-05T08:12:29.674894+0000","last_became_peered":"2024-02-05T08:12:29.674894+0000","last_unstale":"2024-02-05T08:28:45.736116+0000","last_undegraded":"2024-02-05T08:28:45.736116+0000","last_fullsized":"2024-02-05T08:28:45.736116+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:41.138106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:41.138106+0000","last_clean_scrub_stamp":"2024-02-05T08:27:41.138106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:01.617131+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":759,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434552+0000","last_change":"2024-02-05T08:28:36.515091+0000","last_active":"2024-02-05T08:28:45.434552+0000","last_peered":"2024-02-05T08:28:45.434552+0000","last_clean":"2024-02-05T08:28:45.434552+0000","last_became_active":"2024-02-05T08:28:23.479495+0000","last_became_peered":"2024-02-05T08:28:23.479495+0000","last_unstale":"2024-02-05T08:28:45.434552+0000","last_undegraded":"2024-02-05T08:28:45.434552+0000","last_fullsized":"2024-02-05T08:28:45.434552+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:36.515007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:36.737342+0000","last_clean_scrub_stamp":"2024-02-05T08:28:36.515007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:05.800487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435368+0000","last_change":"2024-02-05T08:28:45.430239+0000","last_active":"2024-02-05T08:28:45.435368+0000","last_peered":"2024-02-05T08:28:45.435368+0000","last_clean":"2024-02-05T08:28:45.435368+0000","last_became_active":"2024-02-05T08:25:49.124061+0000","last_became_peered":"2024-02-05T08:25:49.124061+0000","last_unstale":"2024-02-05T08:28:45.435368+0000","last_undegraded":"2024-02-05T08:28:45.435368+0000","last_fullsized":"2024-02-05T08:28:45.435368+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:45.430141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:31.189113+0000","last_clean_scrub_stamp":"2024-02-05T08:28:45.430141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:50.163064+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"397'46","reported_seq":938,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005735+0000","last_change":"2024-02-05T08:28:06.665664+0000","last_active":"2024-02-05T08:28:46.005735+0000","last_peered":"2024-02-05T08:28:46.005735+0000","last_clean":"2024-02-05T08:28:46.005735+0000","last_became_active":"2024-02-05T08:27:39.135175+0000","last_became_peered":"2024-02-05T08:27:39.135175+0000","last_unstale":"2024-02-05T08:28:46.005735+0000","last_undegraded":"2024-02-05T08:28:46.005735+0000","last_fullsized":"2024-02-05T08:28:46.005735+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-02-05T08:28:06.665571+0000","last_deep_scrub":"230'22","last_deep_scrub_stamp":"2024-02-05T08:14:31.867008+0000","last_clean_scrub_stamp":"2024-02-05T08:28:06.665571+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:26.499745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":392,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733848+0000","last_change":"2024-02-05T08:27:41.377827+0000","last_active":"2024-02-05T08:28:45.733848+0000","last_peered":"2024-02-05T08:28:45.733848+0000","last_clean":"2024-02-05T08:28:45.733848+0000","last_became_active":"2024-02-05T08:18:34.642494+0000","last_became_peered":"2024-02-05T08:18:34.642494+0000","last_unstale":"2024-02-05T08:28:45.733848+0000","last_undegraded":"2024-02-05T08:28:45.733848+0000","last_fullsized":"2024-02-05T08:28:45.733848+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:41.377728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:04.435264+0000","last_clean_scrub_stamp":"2024-02-05T08:27:41.377728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":911,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735804+0000","last_change":"2024-02-05T08:28:26.075634+0000","last_active":"2024-02-05T08:28:45.735804+0000","last_peered":"2024-02-05T08:28:45.735804+0000","last_clean":"2024-02-05T08:28:45.735804+0000","last_became_active":"2024-02-05T08:18:34.638652+0000","last_became_peered":"2024-02-05T08:18:34.638652+0000","last_unstale":"2024-02-05T08:28:45.735804+0000","last_undegraded":"2024-02-05T08:28:45.735804+0000","last_fullsized":"2024-02-05T08:28:45.735804+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:26.074066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:57.052030+0000","last_clean_scrub_stamp":"2024-02-05T08:28:26.074066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:34.926112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":900,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:52.139686+0000","last_change":"2024-02-05T08:28:52.139721+0000","last_active":"2024-02-05T08:28:52.139686+0000","last_peered":"2024-02-05T08:28:52.139686+0000","last_clean":"2024-02-05T08:28:52.139686+0000","last_became_active":"2024-02-05T08:23:07.937958+0000","last_became_peered":"2024-02-05T08:23:07.937958+0000","last_unstale":"2024-02-05T08:28:52.139686+0000","last_undegraded":"2024-02-05T08:28:52.139686+0000","last_fullsized":"2024-02-05T08:28:52.139686+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:52.139603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:07.717888+0000","last_clean_scrub_stamp":"2024-02-05T08:28:52.139603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:12.183850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"397'29","reported_seq":413,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:53.143327+0000","last_change":"2024-02-05T08:28:53.143359+0000","last_active":"2024-02-05T08:28:53.143327+0000","last_peered":"2024-02-05T08:28:53.143327+0000","last_clean":"2024-02-05T08:28:53.143327+0000","last_became_active":"2024-02-05T08:23:07.925083+0000","last_became_peered":"2024-02-05T08:23:07.925083+0000","last_unstale":"2024-02-05T08:28:53.143327+0000","last_undegraded":"2024-02-05T08:28:53.143327+0000","last_fullsized":"2024-02-05T08:28:53.143327+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'29","last_scrub_stamp":"2024-02-05T08:28:53.143267+0000","last_deep_scrub":"397'29","last_deep_scrub_stamp":"2024-02-05T08:26:43.239403+0000","last_clean_scrub_stamp":"2024-02-05T08:28:53.143267+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:10.454239+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"131'2","reported_seq":906,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006870+0000","last_change":"2024-02-05T08:27:51.110610+0000","last_active":"2024-02-05T08:28:46.006870+0000","last_peered":"2024-02-05T08:28:46.006870+0000","last_clean":"2024-02-05T08:28:46.006870+0000","last_became_active":"2024-02-05T08:04:47.579931+0000","last_became_peered":"2024-02-05T08:04:47.579931+0000","last_unstale":"2024-02-05T08:28:46.006870+0000","last_undegraded":"2024-02-05T08:28:46.006870+0000","last_fullsized":"2024-02-05T08:28:46.006870+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'2","last_scrub_stamp":"2024-02-05T08:27:51.110518+0000","last_deep_scrub":"131'2","last_deep_scrub_stamp":"2024-02-05T08:08:48.447317+0000","last_clean_scrub_stamp":"2024-02-05T08:27:51.110518+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:18.147023+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"190'10","reported_seq":866,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.870553+0000","last_change":"2024-02-05T08:28:51.003498+0000","last_active":"2024-02-05T08:29:01.870553+0000","last_peered":"2024-02-05T08:29:01.870553+0000","last_clean":"2024-02-05T08:29:01.870553+0000","last_became_active":"2024-02-05T08:12:29.677205+0000","last_became_peered":"2024-02-05T08:12:29.677205+0000","last_unstale":"2024-02-05T08:29:01.870553+0000","last_undegraded":"2024-02-05T08:29:01.870553+0000","last_fullsized":"2024-02-05T08:29:01.870553+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'10","last_scrub_stamp":"2024-02-05T08:28:51.003393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:28:51.003393+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:07.112138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":834,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.433899+0000","last_change":"2024-02-05T08:27:54.409095+0000","last_active":"2024-02-05T08:28:45.433899+0000","last_peered":"2024-02-05T08:28:45.433899+0000","last_clean":"2024-02-05T08:28:45.433899+0000","last_became_active":"2024-02-05T08:23:07.940135+0000","last_became_peered":"2024-02-05T08:23:07.940135+0000","last_unstale":"2024-02-05T08:28:45.433899+0000","last_undegraded":"2024-02-05T08:28:45.433899+0000","last_fullsized":"2024-02-05T08:28:45.433899+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:27:54.408999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:27:54.408999+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:01.161930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"397'30","reported_seq":160,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435135+0000","last_change":"2024-02-05T08:28:09.466883+0000","last_active":"2024-02-05T08:28:45.435135+0000","last_peered":"2024-02-05T08:28:45.435135+0000","last_clean":"2024-02-05T08:28:45.435135+0000","last_became_active":"2024-02-05T08:25:49.130322+0000","last_became_peered":"2024-02-05T08:25:49.130322+0000","last_unstale":"2024-02-05T08:28:45.435135+0000","last_undegraded":"2024-02-05T08:28:45.435135+0000","last_fullsized":"2024-02-05T08:28:45.435135+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'30","last_scrub_stamp":"2024-02-05T08:28:09.466779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:09.466779+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:35.992366+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":708,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434649+0000","last_change":"2024-02-05T08:28:38.625974+0000","last_active":"2024-02-05T08:28:45.434649+0000","last_peered":"2024-02-05T08:28:45.434649+0000","last_clean":"2024-02-05T08:28:45.434649+0000","last_became_active":"2024-02-05T08:28:23.351251+0000","last_became_peered":"2024-02-05T08:28:23.351251+0000","last_unstale":"2024-02-05T08:28:45.434649+0000","last_undegraded":"2024-02-05T08:28:45.434649+0000","last_fullsized":"2024-02-05T08:28:45.434649+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:38.625850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:28:38.625850+0000","last_clean_scrub_stamp":"2024-02-05T08:28:38.625850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:41.166723+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435031+0000","last_change":"2024-02-05T08:27:59.396478+0000","last_active":"2024-02-05T08:28:45.435031+0000","last_peered":"2024-02-05T08:28:45.435031+0000","last_clean":"2024-02-05T08:28:45.435031+0000","last_became_active":"2024-02-05T08:25:49.126916+0000","last_became_peered":"2024-02-05T08:25:49.126916+0000","last_unstale":"2024-02-05T08:28:45.435031+0000","last_undegraded":"2024-02-05T08:28:45.435031+0000","last_fullsized":"2024-02-05T08:28:45.435031+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:59.396397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:19.107555+0000","last_clean_scrub_stamp":"2024-02-05T08:27:59.396397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:17.099060+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":410,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.732787+0000","last_change":"2024-02-05T08:28:08.179703+0000","last_active":"2024-02-05T08:28:45.732787+0000","last_peered":"2024-02-05T08:28:45.732787+0000","last_clean":"2024-02-05T08:28:45.732787+0000","last_became_active":"2024-02-05T08:25:49.151068+0000","last_became_peered":"2024-02-05T08:25:49.151068+0000","last_unstale":"2024-02-05T08:28:45.732787+0000","last_undegraded":"2024-02-05T08:28:45.732787+0000","last_fullsized":"2024-02-05T08:28:45.732787+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:08.179625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:07.137916+0000","last_clean_scrub_stamp":"2024-02-05T08:28:08.179625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:17.126883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"397'27","reported_seq":682,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005903+0000","last_change":"2024-02-05T08:27:44.386311+0000","last_active":"2024-02-05T08:28:46.005903+0000","last_peered":"2024-02-05T08:28:46.005903+0000","last_clean":"2024-02-05T08:28:46.005903+0000","last_became_active":"2024-02-05T08:12:29.683184+0000","last_became_peered":"2024-02-05T08:12:29.683184+0000","last_unstale":"2024-02-05T08:28:46.005903+0000","last_undegraded":"2024-02-05T08:28:46.005903+0000","last_fullsized":"2024-02-05T08:28:46.005903+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'27","last_scrub_stamp":"2024-02-05T08:27:44.386207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:27:44.386207+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:11.102207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":889,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006594+0000","last_change":"2024-02-05T08:28:15.152931+0000","last_active":"2024-02-05T08:28:46.006594+0000","last_peered":"2024-02-05T08:28:46.006594+0000","last_clean":"2024-02-05T08:28:46.006594+0000","last_became_active":"2024-02-05T08:04:47.579145+0000","last_became_peered":"2024-02-05T08:04:47.579145+0000","last_unstale":"2024-02-05T08:28:46.006594+0000","last_undegraded":"2024-02-05T08:28:46.006594+0000","last_fullsized":"2024-02-05T08:28:46.006594+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:15.152778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:14.159628+0000","last_clean_scrub_stamp":"2024-02-05T08:28:15.152778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:43.384102+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":649,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005364+0000","last_change":"2024-02-05T08:28:10.297093+0000","last_active":"2024-02-05T08:28:46.005364+0000","last_peered":"2024-02-05T08:28:46.005364+0000","last_clean":"2024-02-05T08:28:46.005364+0000","last_became_active":"2024-02-05T08:12:29.678048+0000","last_became_peered":"2024-02-05T08:12:29.678048+0000","last_unstale":"2024-02-05T08:28:46.005364+0000","last_undegraded":"2024-02-05T08:28:46.005364+0000","last_fullsized":"2024-02-05T08:28:46.005364+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:10.296986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:05:54.367588+0000","last_clean_scrub_stamp":"2024-02-05T08:28:10.296986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:32.257799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435361+0000","last_change":"2024-02-05T08:28:44.425313+0000","last_active":"2024-02-05T08:28:45.435361+0000","last_peered":"2024-02-05T08:28:45.435361+0000","last_clean":"2024-02-05T08:28:45.435361+0000","last_became_active":"2024-02-05T08:25:49.142594+0000","last_became_peered":"2024-02-05T08:25:49.142594+0000","last_unstale":"2024-02-05T08:28:45.435361+0000","last_undegraded":"2024-02-05T08:28:45.435361+0000","last_fullsized":"2024-02-05T08:28:45.435361+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:44.425209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:31.862841+0000","last_clean_scrub_stamp":"2024-02-05T08:28:44.425209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:10.469263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"397'12","reported_seq":412,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:58.111711+0000","last_change":"2024-02-05T08:28:58.111737+0000","last_active":"2024-02-05T08:28:58.111711+0000","last_peered":"2024-02-05T08:28:58.111711+0000","last_clean":"2024-02-05T08:28:58.111711+0000","last_became_active":"2024-02-05T08:23:07.937911+0000","last_became_peered":"2024-02-05T08:23:07.937911+0000","last_unstale":"2024-02-05T08:28:58.111711+0000","last_undegraded":"2024-02-05T08:28:58.111711+0000","last_fullsized":"2024-02-05T08:28:58.111711+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'12","last_scrub_stamp":"2024-02-05T08:28:58.111640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:58.111640+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:19.420470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":411,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733470+0000","last_change":"2024-02-05T08:28:14.182876+0000","last_active":"2024-02-05T08:28:45.733470+0000","last_peered":"2024-02-05T08:28:45.733470+0000","last_clean":"2024-02-05T08:28:45.733470+0000","last_became_active":"2024-02-05T08:25:49.139211+0000","last_became_peered":"2024-02-05T08:25:49.139211+0000","last_unstale":"2024-02-05T08:28:45.733470+0000","last_undegraded":"2024-02-05T08:28:45.733470+0000","last_fullsized":"2024-02-05T08:28:45.733470+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:14.182776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:28:14.182776+0000","last_clean_scrub_stamp":"2024-02-05T08:28:14.182776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:43.226323+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":921,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005364+0000","last_change":"2024-02-05T08:28:38.966213+0000","last_active":"2024-02-05T08:28:46.005364+0000","last_peered":"2024-02-05T08:28:46.005364+0000","last_clean":"2024-02-05T08:28:46.005364+0000","last_became_active":"2024-02-05T08:12:29.680456+0000","last_became_peered":"2024-02-05T08:12:29.680456+0000","last_unstale":"2024-02-05T08:28:46.005364+0000","last_undegraded":"2024-02-05T08:28:46.005364+0000","last_fullsized":"2024-02-05T08:28:46.005364+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:38.966118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:21:54.156080+0000","last_clean_scrub_stamp":"2024-02-05T08:28:38.966118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:01.440679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":917,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.430191+0000","last_change":"2024-02-05T08:28:22.738090+0000","last_active":"2024-02-05T08:28:45.430191+0000","last_peered":"2024-02-05T08:28:45.430191+0000","last_clean":"2024-02-05T08:28:45.430191+0000","last_became_active":"2024-02-05T08:23:07.932374+0000","last_became_peered":"2024-02-05T08:23:07.932374+0000","last_unstale":"2024-02-05T08:28:45.430191+0000","last_undegraded":"2024-02-05T08:28:45.430191+0000","last_fullsized":"2024-02-05T08:28:45.430191+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:22.737997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:07.831525+0000","last_clean_scrub_stamp":"2024-02-05T08:28:22.737997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:37.249324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"397'32","reported_seq":415,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733195+0000","last_change":"2024-02-05T08:27:44.217952+0000","last_active":"2024-02-05T08:28:45.733195+0000","last_peered":"2024-02-05T08:28:45.733195+0000","last_clean":"2024-02-05T08:28:45.733195+0000","last_became_active":"2024-02-05T08:18:34.631912+0000","last_became_peered":"2024-02-05T08:18:34.631912+0000","last_unstale":"2024-02-05T08:28:45.733195+0000","last_undegraded":"2024-02-05T08:28:45.733195+0000","last_fullsized":"2024-02-05T08:28:45.733195+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'32","last_scrub_stamp":"2024-02-05T08:27:44.217867+0000","last_deep_scrub":"141'16","last_deep_scrub_stamp":"2024-02-05T08:09:39.112936+0000","last_clean_scrub_stamp":"2024-02-05T08:27:44.217867+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:09.556276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:53.525126+0000","last_change":"2024-02-05T08:28:53.525153+0000","last_active":"2024-02-05T08:28:53.525126+0000","last_peered":"2024-02-05T08:28:53.525126+0000","last_clean":"2024-02-05T08:28:53.525126+0000","last_became_active":"2024-02-05T08:25:49.142057+0000","last_became_peered":"2024-02-05T08:25:49.142057+0000","last_unstale":"2024-02-05T08:28:53.525126+0000","last_undegraded":"2024-02-05T08:28:53.525126+0000","last_fullsized":"2024-02-05T08:28:53.525126+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:53.525042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:09.009398+0000","last_clean_scrub_stamp":"2024-02-05T08:28:53.525042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:21.470993+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":410,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733021+0000","last_change":"2024-02-05T08:27:46.252636+0000","last_active":"2024-02-05T08:28:45.733021+0000","last_peered":"2024-02-05T08:28:45.733021+0000","last_clean":"2024-02-05T08:28:45.733021+0000","last_became_active":"2024-02-05T08:25:49.138377+0000","last_became_peered":"2024-02-05T08:25:49.138377+0000","last_unstale":"2024-02-05T08:28:45.733021+0000","last_undegraded":"2024-02-05T08:28:45.733021+0000","last_fullsized":"2024-02-05T08:28:45.733021+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:46.252510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:25:33.208642+0000","last_clean_scrub_stamp":"2024-02-05T08:27:46.252510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:55.912234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":921,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:51.998633+0000","last_change":"2024-02-05T08:28:51.998633+0000","last_active":"2024-02-05T08:28:51.998633+0000","last_peered":"2024-02-05T08:28:51.998633+0000","last_clean":"2024-02-05T08:28:51.998633+0000","last_became_active":"2024-02-05T08:18:34.642387+0000","last_became_peered":"2024-02-05T08:18:34.642387+0000","last_unstale":"2024-02-05T08:28:51.998633+0000","last_undegraded":"2024-02-05T08:28:51.998633+0000","last_fullsized":"2024-02-05T08:28:51.998633+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:51.998537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:28:51.998537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:13.838050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"397'25","reported_seq":585,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429105+0000","last_change":"2024-02-05T08:27:57.844837+0000","last_active":"2024-02-05T08:28:45.429105+0000","last_peered":"2024-02-05T08:28:45.429105+0000","last_clean":"2024-02-05T08:28:45.429105+0000","last_became_active":"2024-02-05T08:14:24.981654+0000","last_became_peered":"2024-02-05T08:14:24.981654+0000","last_unstale":"2024-02-05T08:28:45.429105+0000","last_undegraded":"2024-02-05T08:28:45.429105+0000","last_fullsized":"2024-02-05T08:28:45.429105+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-02-05T08:27:57.844718+0000","last_deep_scrub":"263'19","last_deep_scrub_stamp":"2024-02-05T08:18:42.566537+0000","last_clean_scrub_stamp":"2024-02-05T08:27:57.844718+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:17.818872+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":419,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733086+0000","last_change":"2024-02-05T08:28:29.252131+0000","last_active":"2024-02-05T08:28:45.733086+0000","last_peered":"2024-02-05T08:28:45.733086+0000","last_clean":"2024-02-05T08:28:45.733086+0000","last_became_active":"2024-02-05T08:25:49.140007+0000","last_became_peered":"2024-02-05T08:25:49.140007+0000","last_unstale":"2024-02-05T08:28:45.733086+0000","last_undegraded":"2024-02-05T08:28:45.733086+0000","last_fullsized":"2024-02-05T08:28:45.733086+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:29.251979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:09:29.262582+0000","last_clean_scrub_stamp":"2024-02-05T08:28:29.251979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:32.782087+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":759,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:02.787053+0000","last_change":"2024-02-05T08:29:02.787088+0000","last_active":"2024-02-05T08:29:02.787053+0000","last_peered":"2024-02-05T08:29:02.787053+0000","last_clean":"2024-02-05T08:29:02.787053+0000","last_became_active":"2024-02-05T08:28:23.352654+0000","last_became_peered":"2024-02-05T08:28:23.352654+0000","last_unstale":"2024-02-05T08:29:02.787053+0000","last_undegraded":"2024-02-05T08:29:02.787053+0000","last_fullsized":"2024-02-05T08:29:02.787053+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:02.786983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:29:02.786983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:07.029496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":920,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735561+0000","last_change":"2024-02-05T08:28:00.983432+0000","last_active":"2024-02-05T08:28:45.735561+0000","last_peered":"2024-02-05T08:28:45.735561+0000","last_clean":"2024-02-05T08:28:45.735561+0000","last_became_active":"2024-02-05T08:12:29.677293+0000","last_became_peered":"2024-02-05T08:12:29.677293+0000","last_unstale":"2024-02-05T08:28:45.735561+0000","last_undegraded":"2024-02-05T08:28:45.735561+0000","last_fullsized":"2024-02-05T08:28:45.735561+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:00.983322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:14:30.022077+0000","last_clean_scrub_stamp":"2024-02-05T08:28:00.983322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:21.294561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"397'32","reported_seq":989,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:00.078241+0000","last_change":"2024-02-05T08:29:00.078273+0000","last_active":"2024-02-05T08:29:00.078241+0000","last_peered":"2024-02-05T08:29:00.078241+0000","last_clean":"2024-02-05T08:29:00.078241+0000","last_became_active":"2024-02-05T08:23:07.938523+0000","last_became_peered":"2024-02-05T08:23:07.938523+0000","last_unstale":"2024-02-05T08:29:00.078241+0000","last_undegraded":"2024-02-05T08:29:00.078241+0000","last_fullsized":"2024-02-05T08:29:00.078241+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'32","last_scrub_stamp":"2024-02-05T08:29:00.078172+0000","last_deep_scrub":"249'24","last_deep_scrub_stamp":"2024-02-05T08:15:24.970769+0000","last_clean_scrub_stamp":"2024-02-05T08:29:00.078172+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:29.883907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:06.607233+0000","last_change":"2024-02-05T08:29:06.607260+0000","last_active":"2024-02-05T08:29:06.607233+0000","last_peered":"2024-02-05T08:29:06.607233+0000","last_clean":"2024-02-05T08:29:06.607233+0000","last_became_active":"2024-02-05T08:25:49.130779+0000","last_became_peered":"2024-02-05T08:25:49.130779+0000","last_unstale":"2024-02-05T08:29:06.607233+0000","last_undegraded":"2024-02-05T08:29:06.607233+0000","last_fullsized":"2024-02-05T08:29:06.607233+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:06.607146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:41.228018+0000","last_clean_scrub_stamp":"2024-02-05T08:29:06.607146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:18.260267+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"407'15","reported_seq":1235,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.870572+0000","last_change":"2024-02-05T08:28:01.813418+0000","last_active":"2024-02-05T08:29:01.870572+0000","last_peered":"2024-02-05T08:29:01.870572+0000","last_clean":"2024-02-05T08:29:01.870572+0000","last_became_active":"2024-02-05T08:25:49.135603+0000","last_became_peered":"2024-02-05T08:25:49.135603+0000","last_unstale":"2024-02-05T08:29:01.870572+0000","last_undegraded":"2024-02-05T08:29:01.870572+0000","last_fullsized":"2024-02-05T08:29:01.870572+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'15","last_scrub_stamp":"2024-02-05T08:28:01.813308+0000","last_deep_scrub":"391'14","last_deep_scrub_stamp":"2024-02-05T08:24:33.873828+0000","last_clean_scrub_stamp":"2024-02-05T08:28:01.813308+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:07.086906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":909,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006427+0000","last_change":"2024-02-05T08:28:00.198702+0000","last_active":"2024-02-05T08:28:46.006427+0000","last_peered":"2024-02-05T08:28:46.006427+0000","last_clean":"2024-02-05T08:28:46.006427+0000","last_became_active":"2024-02-05T08:04:41.540503+0000","last_became_peered":"2024-02-05T08:04:41.540503+0000","last_unstale":"2024-02-05T08:28:46.006427+0000","last_undegraded":"2024-02-05T08:28:46.006427+0000","last_fullsized":"2024-02-05T08:28:46.006427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:00.198610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:28:00.198610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:06.654293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"397'35","reported_seq":419,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733317+0000","last_change":"2024-02-05T08:27:40.203035+0000","last_active":"2024-02-05T08:28:45.733317+0000","last_peered":"2024-02-05T08:28:45.733317+0000","last_clean":"2024-02-05T08:28:45.733317+0000","last_became_active":"2024-02-05T08:23:07.928966+0000","last_became_peered":"2024-02-05T08:23:07.928966+0000","last_unstale":"2024-02-05T08:28:45.733317+0000","last_undegraded":"2024-02-05T08:28:45.733317+0000","last_fullsized":"2024-02-05T08:28:45.733317+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'35","last_scrub_stamp":"2024-02-05T08:27:40.202921+0000","last_deep_scrub":"129'15","last_deep_scrub_stamp":"2024-02-05T08:08:37.128817+0000","last_clean_scrub_stamp":"2024-02-05T08:27:40.202921+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:54.216541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":883,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.736175+0000","last_change":"2024-02-05T08:27:39.290278+0000","last_active":"2024-02-05T08:28:45.736175+0000","last_peered":"2024-02-05T08:28:45.736175+0000","last_clean":"2024-02-05T08:28:45.736175+0000","last_became_active":"2024-02-05T08:04:47.578548+0000","last_became_peered":"2024-02-05T08:04:47.578548+0000","last_unstale":"2024-02-05T08:28:45.736175+0000","last_undegraded":"2024-02-05T08:28:45.736175+0000","last_fullsized":"2024-02-05T08:28:45.736175+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:39.290177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:39.290177+0000","last_clean_scrub_stamp":"2024-02-05T08:27:39.290177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"295'10","reported_seq":523,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.870727+0000","last_change":"2024-02-05T08:27:48.313032+0000","last_active":"2024-02-05T08:29:01.870727+0000","last_peered":"2024-02-05T08:29:01.870727+0000","last_clean":"2024-02-05T08:29:01.870727+0000","last_became_active":"2024-02-05T08:18:34.644517+0000","last_became_peered":"2024-02-05T08:18:34.644517+0000","last_unstale":"2024-02-05T08:29:01.870727+0000","last_undegraded":"2024-02-05T08:29:01.870727+0000","last_fullsized":"2024-02-05T08:29:01.870727+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'10","last_scrub_stamp":"2024-02-05T08:27:48.312951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:27:48.312951+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:59.486284+0000","last_change":"2024-02-05T08:28:59.486303+0000","last_active":"2024-02-05T08:28:59.486284+0000","last_peered":"2024-02-05T08:28:59.486284+0000","last_clean":"2024-02-05T08:28:59.486284+0000","last_became_active":"2024-02-05T08:25:49.126790+0000","last_became_peered":"2024-02-05T08:25:49.126790+0000","last_unstale":"2024-02-05T08:28:59.486284+0000","last_undegraded":"2024-02-05T08:28:59.486284+0000","last_fullsized":"2024-02-05T08:28:59.486284+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:59.486230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:24.316208+0000","last_clean_scrub_stamp":"2024-02-05T08:28:59.486230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:17.131261+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"397'21","reported_seq":901,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.430339+0000","last_change":"2024-02-05T08:28:02.257090+0000","last_active":"2024-02-05T08:28:45.430339+0000","last_peered":"2024-02-05T08:28:45.430339+0000","last_clean":"2024-02-05T08:28:45.430339+0000","last_became_active":"2024-02-05T08:28:02.256642+0000","last_became_peered":"2024-02-05T08:28:02.256642+0000","last_unstale":"2024-02-05T08:28:45.430339+0000","last_undegraded":"2024-02-05T08:28:45.430339+0000","last_fullsized":"2024-02-05T08:28:45.430339+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-02-05T08:27:49.038465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:27:49.038465+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:07.417809+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":928,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005954+0000","last_change":"2024-02-05T08:27:55.654783+0000","last_active":"2024-02-05T08:28:46.005954+0000","last_peered":"2024-02-05T08:28:46.005954+0000","last_clean":"2024-02-05T08:28:46.005954+0000","last_became_active":"2024-02-05T08:04:47.578325+0000","last_became_peered":"2024-02-05T08:04:47.578325+0000","last_unstale":"2024-02-05T08:28:46.005954+0000","last_undegraded":"2024-02-05T08:28:46.005954+0000","last_fullsized":"2024-02-05T08:28:46.005954+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:55.654704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:26:38.587082+0000","last_clean_scrub_stamp":"2024-02-05T08:27:55.654704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:21.827619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":392,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733757+0000","last_change":"2024-02-05T08:27:37.243710+0000","last_active":"2024-02-05T08:28:45.733757+0000","last_peered":"2024-02-05T08:28:45.733757+0000","last_clean":"2024-02-05T08:28:45.733757+0000","last_became_active":"2024-02-05T08:23:07.928375+0000","last_became_peered":"2024-02-05T08:23:07.928375+0000","last_unstale":"2024-02-05T08:28:45.733757+0000","last_undegraded":"2024-02-05T08:28:45.733757+0000","last_fullsized":"2024-02-05T08:28:45.733757+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:37.243631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:42.025471+0000","last_clean_scrub_stamp":"2024-02-05T08:27:37.243631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:54.394523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":921,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.067915+0000","last_change":"2024-02-05T08:28:56.067938+0000","last_active":"2024-02-05T08:28:56.067915+0000","last_peered":"2024-02-05T08:28:56.067915+0000","last_clean":"2024-02-05T08:28:56.067915+0000","last_became_active":"2024-02-05T08:12:29.684546+0000","last_became_peered":"2024-02-05T08:12:29.684546+0000","last_unstale":"2024-02-05T08:28:56.067915+0000","last_undegraded":"2024-02-05T08:28:56.067915+0000","last_fullsized":"2024-02-05T08:28:56.067915+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:56.067851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:25.470117+0000","last_clean_scrub_stamp":"2024-02-05T08:28:56.067851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:59.989176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"397'25","reported_seq":158,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434970+0000","last_change":"2024-02-05T08:28:34.558605+0000","last_active":"2024-02-05T08:28:45.434970+0000","last_peered":"2024-02-05T08:28:45.434970+0000","last_clean":"2024-02-05T08:28:45.434970+0000","last_became_active":"2024-02-05T08:25:49.129856+0000","last_became_peered":"2024-02-05T08:25:49.129856+0000","last_unstale":"2024-02-05T08:28:45.434970+0000","last_undegraded":"2024-02-05T08:28:45.434970+0000","last_fullsized":"2024-02-05T08:28:45.434970+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-02-05T08:28:34.558501+0000","last_deep_scrub":"278'19","last_deep_scrub_stamp":"2024-02-05T08:17:11.006333+0000","last_clean_scrub_stamp":"2024-02-05T08:28:34.558501+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:45.806187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":909,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:47.132987+0000","last_change":"2024-02-05T08:28:47.132987+0000","last_active":"2024-02-05T08:28:47.132987+0000","last_peered":"2024-02-05T08:28:47.132987+0000","last_clean":"2024-02-05T08:28:47.132987+0000","last_became_active":"2024-02-05T08:12:29.676979+0000","last_became_peered":"2024-02-05T08:12:29.676979+0000","last_unstale":"2024-02-05T08:28:47.132987+0000","last_undegraded":"2024-02-05T08:28:47.132987+0000","last_fullsized":"2024-02-05T08:28:47.132987+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:47.132890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:47.132890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:04.414482+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":921,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:07.346824+0000","last_change":"2024-02-05T08:29:07.346846+0000","last_active":"2024-02-05T08:29:07.346824+0000","last_peered":"2024-02-05T08:29:07.346824+0000","last_clean":"2024-02-05T08:29:07.346824+0000","last_became_active":"2024-02-05T08:23:07.932741+0000","last_became_peered":"2024-02-05T08:23:07.932741+0000","last_unstale":"2024-02-05T08:29:07.346824+0000","last_undegraded":"2024-02-05T08:29:07.346824+0000","last_fullsized":"2024-02-05T08:29:07.346824+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:07.346719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:10:31.622683+0000","last_clean_scrub_stamp":"2024-02-05T08:29:07.346719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:13.237205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":666,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006516+0000","last_change":"2024-02-05T08:28:36.195968+0000","last_active":"2024-02-05T08:28:46.006516+0000","last_peered":"2024-02-05T08:28:46.006516+0000","last_clean":"2024-02-05T08:28:46.006516+0000","last_became_active":"2024-02-05T08:25:49.126639+0000","last_became_peered":"2024-02-05T08:25:49.126639+0000","last_unstale":"2024-02-05T08:28:46.006516+0000","last_undegraded":"2024-02-05T08:28:46.006516+0000","last_fullsized":"2024-02-05T08:28:46.006516+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:36.195873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:28:36.195873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:39.178323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"397'21","reported_seq":409,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:52.146658+0000","last_change":"2024-02-05T08:28:52.146683+0000","last_active":"2024-02-05T08:28:52.146658+0000","last_peered":"2024-02-05T08:28:52.146658+0000","last_clean":"2024-02-05T08:28:52.146658+0000","last_became_active":"2024-02-05T08:23:07.927922+0000","last_became_peered":"2024-02-05T08:23:07.927922+0000","last_unstale":"2024-02-05T08:28:52.146658+0000","last_undegraded":"2024-02-05T08:28:52.146658+0000","last_fullsized":"2024-02-05T08:28:52.146658+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'21","last_scrub_stamp":"2024-02-05T08:28:52.146601+0000","last_deep_scrub":"116'9","last_deep_scrub_stamp":"2024-02-05T08:07:42.729538+0000","last_clean_scrub_stamp":"2024-02-05T08:28:52.146601+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:06.641926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":852,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429763+0000","last_change":"2024-02-05T08:28:34.656447+0000","last_active":"2024-02-05T08:28:45.429763+0000","last_peered":"2024-02-05T08:28:45.429763+0000","last_clean":"2024-02-05T08:28:45.429763+0000","last_became_active":"2024-02-05T08:28:02.279886+0000","last_became_peered":"2024-02-05T08:28:02.279886+0000","last_unstale":"2024-02-05T08:28:45.429763+0000","last_undegraded":"2024-02-05T08:28:45.429763+0000","last_fullsized":"2024-02-05T08:28:45.429763+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:34.656353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:34.656353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:41.007049+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":957,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:08.214064+0000","last_change":"2024-02-05T08:29:08.214109+0000","last_active":"2024-02-05T08:29:08.214064+0000","last_peered":"2024-02-05T08:29:08.214064+0000","last_clean":"2024-02-05T08:29:08.214064+0000","last_became_active":"2024-02-05T08:25:49.136088+0000","last_became_peered":"2024-02-05T08:25:49.136088+0000","last_unstale":"2024-02-05T08:29:08.214064+0000","last_undegraded":"2024-02-05T08:29:08.214064+0000","last_fullsized":"2024-02-05T08:29:08.214064+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-02-05T08:29:08.213960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:29:08.213960+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:10.289485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"397'46","reported_seq":1017,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.428928+0000","last_change":"2024-02-05T08:27:54.783339+0000","last_active":"2024-02-05T08:28:45.428928+0000","last_peered":"2024-02-05T08:28:45.428928+0000","last_clean":"2024-02-05T08:28:45.428928+0000","last_became_active":"2024-02-05T08:12:29.684058+0000","last_became_peered":"2024-02-05T08:12:29.684058+0000","last_unstale":"2024-02-05T08:28:45.428928+0000","last_undegraded":"2024-02-05T08:28:45.428928+0000","last_fullsized":"2024-02-05T08:28:45.428928+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'46","last_scrub_stamp":"2024-02-05T08:27:54.783199+0000","last_deep_scrub":"397'46","last_deep_scrub_stamp":"2024-02-05T08:27:54.783199+0000","last_clean_scrub_stamp":"2024-02-05T08:27:54.783199+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:19.558353+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"277'11","reported_seq":1225,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.870577+0000","last_change":"2024-02-05T08:28:08.122653+0000","last_active":"2024-02-05T08:29:01.870577+0000","last_peered":"2024-02-05T08:29:01.870577+0000","last_clean":"2024-02-05T08:29:01.870577+0000","last_became_active":"2024-02-05T08:12:29.684551+0000","last_became_peered":"2024-02-05T08:12:29.684551+0000","last_unstale":"2024-02-05T08:29:01.870577+0000","last_undegraded":"2024-02-05T08:29:01.870577+0000","last_fullsized":"2024-02-05T08:29:01.870577+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"277'11","last_scrub_stamp":"2024-02-05T08:28:08.122530+0000","last_deep_scrub":"277'11","last_deep_scrub_stamp":"2024-02-05T08:19:31.117523+0000","last_clean_scrub_stamp":"2024-02-05T08:28:08.122530+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:19.942679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":392,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:06.266034+0000","last_change":"2024-02-05T08:29:06.266059+0000","last_active":"2024-02-05T08:29:06.266034+0000","last_peered":"2024-02-05T08:29:06.266034+0000","last_clean":"2024-02-05T08:29:06.266034+0000","last_became_active":"2024-02-05T08:23:07.929942+0000","last_became_peered":"2024-02-05T08:23:07.929942+0000","last_unstale":"2024-02-05T08:29:06.266034+0000","last_undegraded":"2024-02-05T08:29:06.266034+0000","last_fullsized":"2024-02-05T08:29:06.266034+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:06.265941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:17:14.099323+0000","last_clean_scrub_stamp":"2024-02-05T08:29:06.265941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:22.822377+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":910,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.007212+0000","last_change":"2024-02-05T08:27:52.126046+0000","last_active":"2024-02-05T08:28:46.007212+0000","last_peered":"2024-02-05T08:28:46.007212+0000","last_clean":"2024-02-05T08:28:46.007212+0000","last_became_active":"2024-02-05T08:23:07.937863+0000","last_became_peered":"2024-02-05T08:23:07.937863+0000","last_unstale":"2024-02-05T08:28:46.007212+0000","last_undegraded":"2024-02-05T08:28:46.007212+0000","last_fullsized":"2024-02-05T08:28:46.007212+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:52.125951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:46.581713+0000","last_clean_scrub_stamp":"2024-02-05T08:27:52.125951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:09.057377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"465'29","reported_seq":980,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735472+0000","last_change":"2024-02-05T08:28:35.249411+0000","last_active":"2024-02-05T08:28:45.735472+0000","last_peered":"2024-02-05T08:28:45.735472+0000","last_clean":"2024-02-05T08:28:45.735472+0000","last_became_active":"2024-02-05T08:23:07.929900+0000","last_became_peered":"2024-02-05T08:23:07.929900+0000","last_unstale":"2024-02-05T08:28:45.735472+0000","last_undegraded":"2024-02-05T08:28:45.735472+0000","last_fullsized":"2024-02-05T08:28:45.735472+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'29","last_scrub_stamp":"2024-02-05T08:28:35.249301+0000","last_deep_scrub":"465'29","last_deep_scrub_stamp":"2024-02-05T08:28:35.249301+0000","last_clean_scrub_stamp":"2024-02-05T08:28:35.249301+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:04.389919+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":835,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:58.371889+0000","last_change":"2024-02-05T08:28:58.371909+0000","last_active":"2024-02-05T08:28:58.371889+0000","last_peered":"2024-02-05T08:28:58.371889+0000","last_clean":"2024-02-05T08:28:58.371889+0000","last_became_active":"2024-02-05T08:27:39.132668+0000","last_became_peered":"2024-02-05T08:27:39.132668+0000","last_unstale":"2024-02-05T08:28:58.371889+0000","last_undegraded":"2024-02-05T08:28:58.371889+0000","last_fullsized":"2024-02-05T08:28:58.371889+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:58.371831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:15:19.765714+0000","last_clean_scrub_stamp":"2024-02-05T08:28:58.371831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:13.373419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":910,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005939+0000","last_change":"2024-02-05T08:28:05.146381+0000","last_active":"2024-02-05T08:28:46.005939+0000","last_peered":"2024-02-05T08:28:46.005939+0000","last_clean":"2024-02-05T08:28:46.005939+0000","last_became_active":"2024-02-05T08:18:34.642239+0000","last_became_peered":"2024-02-05T08:18:34.642239+0000","last_unstale":"2024-02-05T08:28:46.005939+0000","last_undegraded":"2024-02-05T08:28:46.005939+0000","last_fullsized":"2024-02-05T08:28:46.005939+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:05.146259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:05.146259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:13.388312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":936,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005717+0000","last_change":"2024-02-05T08:28:13.671524+0000","last_active":"2024-02-05T08:28:46.005717+0000","last_peered":"2024-02-05T08:28:46.005717+0000","last_clean":"2024-02-05T08:28:46.005717+0000","last_became_active":"2024-02-05T08:25:49.118945+0000","last_became_peered":"2024-02-05T08:25:49.118945+0000","last_unstale":"2024-02-05T08:28:46.005717+0000","last_undegraded":"2024-02-05T08:28:46.005717+0000","last_fullsized":"2024-02-05T08:28:46.005717+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:13.671444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:22.231148+0000","last_clean_scrub_stamp":"2024-02-05T08:28:13.671444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:26.437635+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":922,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:06.133364+0000","last_change":"2024-02-05T08:29:06.133402+0000","last_active":"2024-02-05T08:29:06.133364+0000","last_peered":"2024-02-05T08:29:06.133364+0000","last_clean":"2024-02-05T08:29:06.133364+0000","last_became_active":"2024-02-05T08:12:29.678629+0000","last_became_peered":"2024-02-05T08:12:29.678629+0000","last_unstale":"2024-02-05T08:29:06.133364+0000","last_undegraded":"2024-02-05T08:29:06.133364+0000","last_fullsized":"2024-02-05T08:29:06.133364+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:06.133263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:30.082733+0000","last_clean_scrub_stamp":"2024-02-05T08:29:06.133263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:27.279885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"397'51","reported_seq":722,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.007428+0000","last_change":"2024-02-05T08:27:56.400181+0000","last_active":"2024-02-05T08:28:46.007428+0000","last_peered":"2024-02-05T08:28:46.007428+0000","last_clean":"2024-02-05T08:28:46.007428+0000","last_became_active":"2024-02-05T08:12:29.689760+0000","last_became_peered":"2024-02-05T08:12:29.689760+0000","last_unstale":"2024-02-05T08:28:46.007428+0000","last_undegraded":"2024-02-05T08:28:46.007428+0000","last_fullsized":"2024-02-05T08:28:46.007428+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'51","last_scrub_stamp":"2024-02-05T08:27:56.400058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:27:56.400058+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:06.515590+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435107+0000","last_change":"2024-02-05T08:27:53.451662+0000","last_active":"2024-02-05T08:28:45.435107+0000","last_peered":"2024-02-05T08:28:45.435107+0000","last_clean":"2024-02-05T08:28:45.435107+0000","last_became_active":"2024-02-05T08:25:49.125995+0000","last_became_peered":"2024-02-05T08:25:49.125995+0000","last_unstale":"2024-02-05T08:28:45.435107+0000","last_undegraded":"2024-02-05T08:28:45.435107+0000","last_fullsized":"2024-02-05T08:28:45.435107+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:53.451556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:19:24.707985+0000","last_clean_scrub_stamp":"2024-02-05T08:27:53.451556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:22.102024+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":842,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005517+0000","last_change":"2024-02-05T08:28:43.527959+0000","last_active":"2024-02-05T08:28:46.005517+0000","last_peered":"2024-02-05T08:28:46.005517+0000","last_clean":"2024-02-05T08:28:46.005517+0000","last_became_active":"2024-02-05T08:27:39.133621+0000","last_became_peered":"2024-02-05T08:27:39.133621+0000","last_unstale":"2024-02-05T08:28:46.005517+0000","last_undegraded":"2024-02-05T08:28:46.005517+0000","last_fullsized":"2024-02-05T08:28:46.005517+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:43.527843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:28:43.527843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:01.982544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":856,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429986+0000","last_change":"2024-02-05T08:28:02.253974+0000","last_active":"2024-02-05T08:28:45.429986+0000","last_peered":"2024-02-05T08:28:45.429986+0000","last_clean":"2024-02-05T08:28:45.429986+0000","last_became_active":"2024-02-05T08:28:02.253234+0000","last_became_peered":"2024-02-05T08:28:02.253234+0000","last_unstale":"2024-02-05T08:28:45.429986+0000","last_undegraded":"2024-02-05T08:28:45.429986+0000","last_fullsized":"2024-02-05T08:28:45.429986+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:58.255975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:23:01.230038+0000","last_clean_scrub_stamp":"2024-02-05T08:27:58.255975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:22.588813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"397'25","reported_seq":158,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435320+0000","last_change":"2024-02-05T08:28:24.442805+0000","last_active":"2024-02-05T08:28:45.435320+0000","last_peered":"2024-02-05T08:28:45.435320+0000","last_clean":"2024-02-05T08:28:45.435320+0000","last_became_active":"2024-02-05T08:25:49.134815+0000","last_became_peered":"2024-02-05T08:25:49.134815+0000","last_unstale":"2024-02-05T08:28:45.435320+0000","last_undegraded":"2024-02-05T08:28:45.435320+0000","last_fullsized":"2024-02-05T08:28:45.435320+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'25","last_scrub_stamp":"2024-02-05T08:28:24.442707+0000","last_deep_scrub":"262'19","last_deep_scrub_stamp":"2024-02-05T08:16:04.836768+0000","last_clean_scrub_stamp":"2024-02-05T08:28:24.442707+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:26.061102+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"82'130","reported_seq":625,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.428993+0000","last_change":"2024-02-05T08:28:02.779779+0000","last_active":"2024-02-05T08:28:45.428993+0000","last_peered":"2024-02-05T08:28:45.428993+0000","last_clean":"2024-02-05T08:28:45.428993+0000","last_became_active":"2024-02-05T08:28:02.257244+0000","last_became_peered":"2024-02-05T08:28:02.257244+0000","last_unstale":"2024-02-05T08:28:45.428993+0000","last_undegraded":"2024-02-05T08:28:45.428993+0000","last_fullsized":"2024-02-05T08:28:45.428993+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:02.779675+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:28:02.779675+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:12.155730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":620,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735864+0000","last_change":"2024-02-05T08:28:32.278888+0000","last_active":"2024-02-05T08:28:45.735864+0000","last_peered":"2024-02-05T08:28:45.735864+0000","last_clean":"2024-02-05T08:28:45.735864+0000","last_became_active":"2024-02-05T08:23:07.936660+0000","last_became_peered":"2024-02-05T08:23:07.936660+0000","last_unstale":"2024-02-05T08:28:45.735864+0000","last_undegraded":"2024-02-05T08:28:45.735864+0000","last_fullsized":"2024-02-05T08:28:45.735864+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:32.278771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:32.278771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:47.953613+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"87'3","reported_seq":874,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.420173+0000","last_change":"2024-02-05T08:28:46.420202+0000","last_active":"2024-02-05T08:28:46.420173+0000","last_peered":"2024-02-05T08:28:46.420173+0000","last_clean":"2024-02-05T08:28:46.420173+0000","last_became_active":"2024-02-05T08:28:02.254335+0000","last_became_peered":"2024-02-05T08:28:02.254335+0000","last_unstale":"2024-02-05T08:28:46.420173+0000","last_undegraded":"2024-02-05T08:28:46.420173+0000","last_fullsized":"2024-02-05T08:28:46.420173+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'3","last_scrub_stamp":"2024-02-05T08:28:46.420096+0000","last_deep_scrub":"87'3","last_deep_scrub_stamp":"2024-02-05T08:28:46.420096+0000","last_clean_scrub_stamp":"2024-02-05T08:28:46.420096+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:46.897187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054343100000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":937,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:06.037240+0000","last_change":"2024-02-05T08:29:06.037281+0000","last_active":"2024-02-05T08:29:06.037240+0000","last_peered":"2024-02-05T08:29:06.037240+0000","last_clean":"2024-02-05T08:29:06.037240+0000","last_became_active":"2024-02-05T08:25:49.113840+0000","last_became_peered":"2024-02-05T08:25:49.113840+0000","last_unstale":"2024-02-05T08:29:06.037240+0000","last_undegraded":"2024-02-05T08:29:06.037240+0000","last_fullsized":"2024-02-05T08:29:06.037240+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:06.037148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:40.135374+0000","last_clean_scrub_stamp":"2024-02-05T08:29:06.037148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:21.097231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"464'68","reported_seq":1122,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.869859+0000","last_change":"2024-02-05T08:28:23.352523+0000","last_active":"2024-02-05T08:28:56.869859+0000","last_peered":"2024-02-05T08:28:56.869859+0000","last_clean":"2024-02-05T08:28:56.869859+0000","last_became_active":"2024-02-05T08:28:23.352147+0000","last_became_peered":"2024-02-05T08:28:23.352147+0000","last_unstale":"2024-02-05T08:28:56.869859+0000","last_undegraded":"2024-02-05T08:28:56.869859+0000","last_fullsized":"2024-02-05T08:28:56.869859+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"435'67","last_scrub_stamp":"2024-02-05T08:28:04.672873+0000","last_deep_scrub":"435'67","last_deep_scrub_stamp":"2024-02-05T08:28:04.672873+0000","last_clean_scrub_stamp":"2024-02-05T08:28:04.672873+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:04.877722+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434746+0000","last_change":"2024-02-05T08:28:40.512383+0000","last_active":"2024-02-05T08:28:45.434746+0000","last_peered":"2024-02-05T08:28:45.434746+0000","last_clean":"2024-02-05T08:28:45.434746+0000","last_became_active":"2024-02-05T08:25:49.134871+0000","last_became_peered":"2024-02-05T08:25:49.134871+0000","last_unstale":"2024-02-05T08:28:45.434746+0000","last_undegraded":"2024-02-05T08:28:45.434746+0000","last_fullsized":"2024-02-05T08:28:45.434746+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:40.512307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:52.171116+0000","last_clean_scrub_stamp":"2024-02-05T08:28:40.512307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:55.874569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"82'130","reported_seq":1226,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006476+0000","last_change":"2024-02-05T08:27:44.978552+0000","last_active":"2024-02-05T08:28:46.006476+0000","last_peered":"2024-02-05T08:28:46.006476+0000","last_clean":"2024-02-05T08:28:46.006476+0000","last_became_active":"2024-02-05T08:25:52.156544+0000","last_became_peered":"2024-02-05T08:25:52.156544+0000","last_unstale":"2024-02-05T08:28:46.006476+0000","last_undegraded":"2024-02-05T08:28:46.006476+0000","last_fullsized":"2024-02-05T08:28:46.006476+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:27:44.978433+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:27:44.978433+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:11.489411+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":914,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.007149+0000","last_change":"2024-02-05T08:27:44.094804+0000","last_active":"2024-02-05T08:28:46.007149+0000","last_peered":"2024-02-05T08:28:46.007149+0000","last_clean":"2024-02-05T08:28:46.007149+0000","last_became_active":"2024-02-05T08:18:34.643109+0000","last_became_peered":"2024-02-05T08:18:34.643109+0000","last_unstale":"2024-02-05T08:28:46.007149+0000","last_undegraded":"2024-02-05T08:28:46.007149+0000","last_fullsized":"2024-02-05T08:28:46.007149+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:44.094699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:27:44.094699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:13.833035+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"94'2","reported_seq":194,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.185962+0000","last_change":"2024-02-05T08:29:01.186002+0000","last_active":"2024-02-05T08:29:01.185962+0000","last_peered":"2024-02-05T08:29:01.185962+0000","last_clean":"2024-02-05T08:29:01.185962+0000","last_became_active":"2024-02-05T08:24:08.005056+0000","last_became_peered":"2024-02-05T08:24:08.005056+0000","last_unstale":"2024-02-05T08:29:01.185962+0000","last_undegraded":"2024-02-05T08:29:01.185962+0000","last_fullsized":"2024-02-05T08:29:01.185962+0000","mapping_epoch":386,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'2","last_scrub_stamp":"2024-02-05T08:29:01.185872+0000","last_deep_scrub":"94'2","last_deep_scrub_stamp":"2024-02-05T08:06:34.917832+0000","last_clean_scrub_stamp":"2024-02-05T08:29:01.185872+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:23.034864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":970,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005443+0000","last_change":"2024-02-05T08:27:54.691753+0000","last_active":"2024-02-05T08:28:46.005443+0000","last_peered":"2024-02-05T08:28:46.005443+0000","last_clean":"2024-02-05T08:28:46.005443+0000","last_became_active":"2024-02-05T08:25:49.127248+0000","last_became_peered":"2024-02-05T08:25:49.127248+0000","last_unstale":"2024-02-05T08:28:46.005443+0000","last_undegraded":"2024-02-05T08:28:46.005443+0000","last_fullsized":"2024-02-05T08:28:46.005443+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:54.691677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:20:29.922794+0000","last_clean_scrub_stamp":"2024-02-05T08:27:54.691677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:06.570786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016676569999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"397'26","reported_seq":986,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006140+0000","last_change":"2024-02-05T08:28:36.577381+0000","last_active":"2024-02-05T08:28:46.006140+0000","last_peered":"2024-02-05T08:28:46.006140+0000","last_clean":"2024-02-05T08:28:46.006140+0000","last_became_active":"2024-02-05T08:25:49.115716+0000","last_became_peered":"2024-02-05T08:25:49.115716+0000","last_unstale":"2024-02-05T08:28:46.006140+0000","last_undegraded":"2024-02-05T08:28:46.006140+0000","last_fullsized":"2024-02-05T08:28:46.006140+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'26","last_scrub_stamp":"2024-02-05T08:28:36.577260+0000","last_deep_scrub":"397'26","last_deep_scrub_stamp":"2024-02-05T08:28:36.577260+0000","last_clean_scrub_stamp":"2024-02-05T08:28:36.577260+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:01.816591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":30,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":933,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005117+0000","last_change":"2024-02-05T08:28:44.516566+0000","last_active":"2024-02-05T08:28:46.005117+0000","last_peered":"2024-02-05T08:28:46.005117+0000","last_clean":"2024-02-05T08:28:46.005117+0000","last_became_active":"2024-02-05T08:04:45.566615+0000","last_became_peered":"2024-02-05T08:04:45.566615+0000","last_unstale":"2024-02-05T08:28:46.005117+0000","last_undegraded":"2024-02-05T08:28:46.005117+0000","last_fullsized":"2024-02-05T08:28:46.005117+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:44.516454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:25:46.935681+0000","last_clean_scrub_stamp":"2024-02-05T08:28:44.516454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:07.807695+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":917,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.388848+0000","last_change":"2024-02-05T08:29:01.388873+0000","last_active":"2024-02-05T08:29:01.388848+0000","last_peered":"2024-02-05T08:29:01.388848+0000","last_clean":"2024-02-05T08:29:01.388848+0000","last_became_active":"2024-02-05T08:18:34.633450+0000","last_became_peered":"2024-02-05T08:18:34.633450+0000","last_unstale":"2024-02-05T08:29:01.388848+0000","last_undegraded":"2024-02-05T08:29:01.388848+0000","last_fullsized":"2024-02-05T08:29:01.388848+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:01.388774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:29:01.388774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:22.767933+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"82'130","reported_seq":1224,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006115+0000","last_change":"2024-02-05T08:28:03.210289+0000","last_active":"2024-02-05T08:28:46.006115+0000","last_peered":"2024-02-05T08:28:46.006115+0000","last_clean":"2024-02-05T08:28:46.006115+0000","last_became_active":"2024-02-05T08:25:52.159754+0000","last_became_peered":"2024-02-05T08:25:52.159754+0000","last_unstale":"2024-02-05T08:28:46.006115+0000","last_undegraded":"2024-02-05T08:28:46.006115+0000","last_fullsized":"2024-02-05T08:28:46.006115+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:03.210194+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:33.165371+0000","last_clean_scrub_stamp":"2024-02-05T08:28:03.210194+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:17.363251+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":400,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733095+0000","last_change":"2024-02-05T08:28:22.093244+0000","last_active":"2024-02-05T08:28:45.733095+0000","last_peered":"2024-02-05T08:28:45.733095+0000","last_clean":"2024-02-05T08:28:45.733095+0000","last_became_active":"2024-02-05T08:18:34.631974+0000","last_became_peered":"2024-02-05T08:18:34.631974+0000","last_unstale":"2024-02-05T08:28:45.733095+0000","last_undegraded":"2024-02-05T08:28:45.733095+0000","last_fullsized":"2024-02-05T08:28:45.733095+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:22.093154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:08:39.071354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:22.093154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:26.195903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"397'42","reported_seq":900,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.870700+0000","last_change":"2024-02-05T08:28:41.099100+0000","last_active":"2024-02-05T08:29:01.870700+0000","last_peered":"2024-02-05T08:29:01.870700+0000","last_clean":"2024-02-05T08:29:01.870700+0000","last_became_active":"2024-02-05T08:12:29.689317+0000","last_became_peered":"2024-02-05T08:12:29.689317+0000","last_unstale":"2024-02-05T08:29:01.870700+0000","last_undegraded":"2024-02-05T08:29:01.870700+0000","last_fullsized":"2024-02-05T08:29:01.870700+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'42","last_scrub_stamp":"2024-02-05T08:28:41.098988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:41.098988+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:58.509615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":902,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005456+0000","last_change":"2024-02-05T08:28:10.128826+0000","last_active":"2024-02-05T08:28:46.005456+0000","last_peered":"2024-02-05T08:28:46.005456+0000","last_clean":"2024-02-05T08:28:46.005456+0000","last_became_active":"2024-02-05T08:04:45.564858+0000","last_became_peered":"2024-02-05T08:04:45.564858+0000","last_unstale":"2024-02-05T08:28:46.005456+0000","last_undegraded":"2024-02-05T08:28:46.005456+0000","last_fullsized":"2024-02-05T08:28:46.005456+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:10.128728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:18:42.996695+0000","last_clean_scrub_stamp":"2024-02-05T08:28:10.128728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:30.132839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"82'130","reported_seq":519,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:51.808807+0000","last_change":"2024-02-05T08:28:51.808850+0000","last_active":"2024-02-05T08:28:51.808807+0000","last_peered":"2024-02-05T08:28:51.808807+0000","last_clean":"2024-02-05T08:28:51.808807+0000","last_became_active":"2024-02-05T08:28:23.348448+0000","last_became_peered":"2024-02-05T08:28:23.348448+0000","last_unstale":"2024-02-05T08:28:51.808807+0000","last_undegraded":"2024-02-05T08:28:51.808807+0000","last_fullsized":"2024-02-05T08:28:51.808807+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:51.808727+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:28:51.808727+0000","last_clean_scrub_stamp":"2024-02-05T08:28:51.808727+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:17.006732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":813,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:03.819141+0000","last_change":"2024-02-05T08:29:03.819161+0000","last_active":"2024-02-05T08:29:03.819141+0000","last_peered":"2024-02-05T08:29:03.819141+0000","last_clean":"2024-02-05T08:29:03.819141+0000","last_became_active":"2024-02-05T08:23:07.935640+0000","last_became_peered":"2024-02-05T08:23:07.935640+0000","last_unstale":"2024-02-05T08:29:03.819141+0000","last_undegraded":"2024-02-05T08:29:03.819141+0000","last_fullsized":"2024-02-05T08:29:03.819141+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:29:03.819081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:29:03.819081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:26.593845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":140,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435066+0000","last_change":"2024-02-05T08:27:39.173280+0000","last_active":"2024-02-05T08:28:45.435066+0000","last_peered":"2024-02-05T08:28:45.435066+0000","last_clean":"2024-02-05T08:28:45.435066+0000","last_became_active":"2024-02-05T08:25:49.134615+0000","last_became_peered":"2024-02-05T08:25:49.134615+0000","last_unstale":"2024-02-05T08:28:45.435066+0000","last_undegraded":"2024-02-05T08:28:45.435066+0000","last_fullsized":"2024-02-05T08:28:45.435066+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:39.173205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:27:39.173205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:03.645543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"212'6","reported_seq":2405,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:07.441140+0000","last_change":"2024-02-05T08:28:57.106025+0000","last_active":"2024-02-05T08:29:07.441140+0000","last_peered":"2024-02-05T08:29:07.441140+0000","last_clean":"2024-02-05T08:29:07.441140+0000","last_became_active":"2024-02-05T08:25:49.148530+0000","last_became_peered":"2024-02-05T08:25:49.148530+0000","last_unstale":"2024-02-05T08:29:07.441140+0000","last_undegraded":"2024-02-05T08:29:07.441140+0000","last_fullsized":"2024-02-05T08:29:07.441140+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'6","last_scrub_stamp":"2024-02-05T08:28:57.105911+0000","last_deep_scrub":"212'6","last_deep_scrub_stamp":"2024-02-05T08:13:36.138048+0000","last_clean_scrub_stamp":"2024-02-05T08:28:57.105911+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:16.124700+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00059047500000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1441,"num_read_kb":1436,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"425'42","reported_seq":168,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435136+0000","last_change":"2024-02-05T08:28:17.495594+0000","last_active":"2024-02-05T08:28:45.435136+0000","last_peered":"2024-02-05T08:28:45.435136+0000","last_clean":"2024-02-05T08:28:45.435136+0000","last_became_active":"2024-02-05T08:25:49.144423+0000","last_became_peered":"2024-02-05T08:25:49.144423+0000","last_unstale":"2024-02-05T08:28:45.435136+0000","last_undegraded":"2024-02-05T08:28:45.435136+0000","last_fullsized":"2024-02-05T08:28:45.435136+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'42","last_scrub_stamp":"2024-02-05T08:28:17.495486+0000","last_deep_scrub":"425'42","last_deep_scrub_stamp":"2024-02-05T08:28:17.495486+0000","last_clean_scrub_stamp":"2024-02-05T08:28:17.495486+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:37.194565+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":920,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.431032+0000","last_change":"2024-02-05T08:27:48.658294+0000","last_active":"2024-02-05T08:28:45.431032+0000","last_peered":"2024-02-05T08:28:45.431032+0000","last_clean":"2024-02-05T08:28:45.431032+0000","last_became_active":"2024-02-05T08:25:49.116332+0000","last_became_peered":"2024-02-05T08:25:49.116332+0000","last_unstale":"2024-02-05T08:28:45.431032+0000","last_undegraded":"2024-02-05T08:28:45.431032+0000","last_fullsized":"2024-02-05T08:28:45.431032+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:48.658184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:16:02.899084+0000","last_clean_scrub_stamp":"2024-02-05T08:27:48.658184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:17.571240+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"82'130","reported_seq":971,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.007594+0000","last_change":"2024-02-05T08:28:09.308052+0000","last_active":"2024-02-05T08:28:46.007594+0000","last_peered":"2024-02-05T08:28:46.007594+0000","last_clean":"2024-02-05T08:28:46.007594+0000","last_became_active":"2024-02-05T08:25:52.168497+0000","last_became_peered":"2024-02-05T08:25:52.168497+0000","last_unstale":"2024-02-05T08:28:46.007594+0000","last_undegraded":"2024-02-05T08:28:46.007594+0000","last_fullsized":"2024-02-05T08:28:46.007594+0000","mapping_epoch":409,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":410,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:09.307951+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:28:09.307951+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:37.753750+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":754,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434142+0000","last_change":"2024-02-05T08:28:23.347264+0000","last_active":"2024-02-05T08:28:45.434142+0000","last_peered":"2024-02-05T08:28:45.434142+0000","last_clean":"2024-02-05T08:28:45.434142+0000","last_became_active":"2024-02-05T08:28:23.346839+0000","last_became_peered":"2024-02-05T08:28:23.346839+0000","last_unstale":"2024-02-05T08:28:45.434142+0000","last_undegraded":"2024-02-05T08:28:45.434142+0000","last_fullsized":"2024-02-05T08:28:45.434142+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:01.368364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:01.368364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:06.399915+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":709,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434555+0000","last_change":"2024-02-05T08:28:23.353158+0000","last_active":"2024-02-05T08:28:45.434555+0000","last_peered":"2024-02-05T08:28:45.434555+0000","last_clean":"2024-02-05T08:28:45.434555+0000","last_became_active":"2024-02-05T08:28:23.352870+0000","last_became_peered":"2024-02-05T08:28:23.352870+0000","last_unstale":"2024-02-05T08:28:45.434555+0000","last_undegraded":"2024-02-05T08:28:45.434555+0000","last_fullsized":"2024-02-05T08:28:45.434555+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:20.607154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:28:20.607154+0000","last_clean_scrub_stamp":"2024-02-05T08:28:20.607154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:32.095919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"397'47","reported_seq":166,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435389+0000","last_change":"2024-02-05T08:27:56.383915+0000","last_active":"2024-02-05T08:28:45.435389+0000","last_peered":"2024-02-05T08:28:45.435389+0000","last_clean":"2024-02-05T08:28:45.435389+0000","last_became_active":"2024-02-05T08:25:49.143415+0000","last_became_peered":"2024-02-05T08:25:49.143415+0000","last_unstale":"2024-02-05T08:28:45.435389+0000","last_undegraded":"2024-02-05T08:28:45.435389+0000","last_fullsized":"2024-02-05T08:28:45.435389+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'47","last_scrub_stamp":"2024-02-05T08:27:56.383805+0000","last_deep_scrub":"397'47","last_deep_scrub_stamp":"2024-02-05T08:27:56.383805+0000","last_clean_scrub_stamp":"2024-02-05T08:27:56.383805+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:10.453342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":658,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.007364+0000","last_change":"2024-02-05T08:28:02.281875+0000","last_active":"2024-02-05T08:28:46.007364+0000","last_peered":"2024-02-05T08:28:46.007364+0000","last_clean":"2024-02-05T08:28:46.007364+0000","last_became_active":"2024-02-05T08:25:49.126901+0000","last_became_peered":"2024-02-05T08:25:49.126901+0000","last_unstale":"2024-02-05T08:28:46.007364+0000","last_undegraded":"2024-02-05T08:28:46.007364+0000","last_fullsized":"2024-02-05T08:28:46.007364+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:02.281787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:26.619158+0000","last_clean_scrub_stamp":"2024-02-05T08:28:02.281787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:15.080859+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"82'130","reported_seq":953,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006031+0000","last_change":"2024-02-05T08:28:06.245990+0000","last_active":"2024-02-05T08:28:46.006031+0000","last_peered":"2024-02-05T08:28:46.006031+0000","last_clean":"2024-02-05T08:28:46.006031+0000","last_became_active":"2024-02-05T08:19:01.815446+0000","last_became_peered":"2024-02-05T08:19:01.815446+0000","last_unstale":"2024-02-05T08:28:46.006031+0000","last_undegraded":"2024-02-05T08:28:46.006031+0000","last_fullsized":"2024-02-05T08:28:46.006031+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:06.245891+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:20:35.974462+0000","last_clean_scrub_stamp":"2024-02-05T08:28:06.245891+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:09.347090+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":148,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434784+0000","last_change":"2024-02-05T08:27:51.319090+0000","last_active":"2024-02-05T08:28:45.434784+0000","last_peered":"2024-02-05T08:28:45.434784+0000","last_clean":"2024-02-05T08:28:45.434784+0000","last_became_active":"2024-02-05T08:25:49.133655+0000","last_became_peered":"2024-02-05T08:25:49.133655+0000","last_unstale":"2024-02-05T08:28:45.434784+0000","last_undegraded":"2024-02-05T08:28:45.434784+0000","last_fullsized":"2024-02-05T08:28:45.434784+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:51.318965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:27:51.318965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:12.413995+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":929,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.430177+0000","last_change":"2024-02-05T08:27:50.709457+0000","last_active":"2024-02-05T08:28:45.430177+0000","last_peered":"2024-02-05T08:28:45.430177+0000","last_clean":"2024-02-05T08:28:45.430177+0000","last_became_active":"2024-02-05T08:18:34.633263+0000","last_became_peered":"2024-02-05T08:18:34.633263+0000","last_unstale":"2024-02-05T08:28:45.430177+0000","last_undegraded":"2024-02-05T08:28:45.430177+0000","last_fullsized":"2024-02-05T08:28:45.430177+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:50.709314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:27:50.709314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:10.342352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"467'35","reported_seq":793,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:58.749144+0000","last_change":"2024-02-05T08:28:58.749186+0000","last_active":"2024-02-05T08:28:58.749144+0000","last_peered":"2024-02-05T08:28:58.749144+0000","last_clean":"2024-02-05T08:28:58.749144+0000","last_became_active":"2024-02-05T08:28:23.349323+0000","last_became_peered":"2024-02-05T08:28:23.349323+0000","last_unstale":"2024-02-05T08:28:58.749144+0000","last_undegraded":"2024-02-05T08:28:58.749144+0000","last_fullsized":"2024-02-05T08:28:58.749144+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'35","last_scrub_stamp":"2024-02-05T08:28:58.749084+0000","last_deep_scrub":"467'35","last_deep_scrub_stamp":"2024-02-05T08:28:58.749084+0000","last_clean_scrub_stamp":"2024-02-05T08:28:58.749084+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:06.007233+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"407'13","reported_seq":193,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.870654+0000","last_change":"2024-02-05T08:28:33.558264+0000","last_active":"2024-02-05T08:29:01.870654+0000","last_peered":"2024-02-05T08:29:01.870654+0000","last_clean":"2024-02-05T08:29:01.870654+0000","last_became_active":"2024-02-05T08:25:49.142430+0000","last_became_peered":"2024-02-05T08:25:49.142430+0000","last_unstale":"2024-02-05T08:29:01.870654+0000","last_undegraded":"2024-02-05T08:29:01.870654+0000","last_fullsized":"2024-02-05T08:29:01.870654+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'13","last_scrub_stamp":"2024-02-05T08:28:33.558130+0000","last_deep_scrub":"355'12","last_deep_scrub_stamp":"2024-02-05T08:24:38.385240+0000","last_clean_scrub_stamp":"2024-02-05T08:28:33.558130+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:48.865948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"82'130","reported_seq":676,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733287+0000","last_change":"2024-02-05T08:28:28.184075+0000","last_active":"2024-02-05T08:28:45.733287+0000","last_peered":"2024-02-05T08:28:45.733287+0000","last_clean":"2024-02-05T08:28:45.733287+0000","last_became_active":"2024-02-05T08:19:01.819856+0000","last_became_peered":"2024-02-05T08:19:01.819856+0000","last_unstale":"2024-02-05T08:28:45.733287+0000","last_undegraded":"2024-02-05T08:28:45.733287+0000","last_fullsized":"2024-02-05T08:28:45.733287+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:28.183945+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:28:28.183945+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:42.118675+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":902,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006820+0000","last_change":"2024-02-05T08:28:04.294410+0000","last_active":"2024-02-05T08:28:46.006820+0000","last_peered":"2024-02-05T08:28:46.006820+0000","last_clean":"2024-02-05T08:28:46.006820+0000","last_became_active":"2024-02-05T08:12:29.678186+0000","last_became_peered":"2024-02-05T08:12:29.678186+0000","last_unstale":"2024-02-05T08:28:46.006820+0000","last_undegraded":"2024-02-05T08:28:46.006820+0000","last_fullsized":"2024-02-05T08:28:46.006820+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:04.294307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:28:04.294307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:04.296318+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":152,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.435253+0000","last_change":"2024-02-05T08:28:06.413485+0000","last_active":"2024-02-05T08:28:45.435253+0000","last_peered":"2024-02-05T08:28:45.435253+0000","last_clean":"2024-02-05T08:28:45.435253+0000","last_became_active":"2024-02-05T08:25:49.135389+0000","last_became_peered":"2024-02-05T08:25:49.135389+0000","last_unstale":"2024-02-05T08:28:45.435253+0000","last_undegraded":"2024-02-05T08:28:45.435253+0000","last_fullsized":"2024-02-05T08:28:45.435253+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:28:06.413405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:28:06.413405+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:10.481847+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"397'42","reported_seq":1004,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:56.065996+0000","last_change":"2024-02-05T08:28:56.066061+0000","last_active":"2024-02-05T08:28:56.065996+0000","last_peered":"2024-02-05T08:28:56.065996+0000","last_clean":"2024-02-05T08:28:56.065996+0000","last_became_active":"2024-02-05T08:12:29.666618+0000","last_became_peered":"2024-02-05T08:12:29.666618+0000","last_unstale":"2024-02-05T08:28:56.065996+0000","last_undegraded":"2024-02-05T08:28:56.065996+0000","last_fullsized":"2024-02-05T08:28:56.065996+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'42","last_scrub_stamp":"2024-02-05T08:28:56.065880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:56.065880+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:58.747276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"370'13","reported_seq":527,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.870840+0000","last_change":"2024-02-05T08:27:25.061394+0000","last_active":"2024-02-05T08:29:01.870840+0000","last_peered":"2024-02-05T08:29:01.870840+0000","last_clean":"2024-02-05T08:29:01.870840+0000","last_became_active":"2024-02-05T08:23:07.936347+0000","last_became_peered":"2024-02-05T08:23:07.936347+0000","last_unstale":"2024-02-05T08:29:01.870840+0000","last_undegraded":"2024-02-05T08:29:01.870840+0000","last_fullsized":"2024-02-05T08:29:01.870840+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'13","last_scrub_stamp":"2024-02-05T08:27:25.061300+0000","last_deep_scrub":"168'9","last_deep_scrub_stamp":"2024-02-05T08:11:27.751188+0000","last_clean_scrub_stamp":"2024-02-05T08:27:25.061300+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"82'130","reported_seq":676,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735868+0000","last_change":"2024-02-05T08:28:38.235566+0000","last_active":"2024-02-05T08:28:45.735868+0000","last_peered":"2024-02-05T08:28:45.735868+0000","last_clean":"2024-02-05T08:28:45.735868+0000","last_became_active":"2024-02-05T08:19:01.816923+0000","last_became_peered":"2024-02-05T08:19:01.816923+0000","last_unstale":"2024-02-05T08:28:45.735868+0000","last_undegraded":"2024-02-05T08:28:45.735868+0000","last_fullsized":"2024-02-05T08:28:45.735868+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":3,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:38.235465+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:28:38.235465+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:02.399391+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":909,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735372+0000","last_change":"2024-02-05T08:28:11.009285+0000","last_active":"2024-02-05T08:28:45.735372+0000","last_peered":"2024-02-05T08:28:45.735372+0000","last_clean":"2024-02-05T08:28:45.735372+0000","last_became_active":"2024-02-05T08:18:34.635908+0000","last_became_peered":"2024-02-05T08:18:34.635908+0000","last_unstale":"2024-02-05T08:28:45.735372+0000","last_undegraded":"2024-02-05T08:28:45.735372+0000","last_fullsized":"2024-02-05T08:28:45.735372+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:11.009184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:22:08.117925+0000","last_clean_scrub_stamp":"2024-02-05T08:28:11.009184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:23.144509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"67'1","reported_seq":423,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733630+0000","last_change":"2024-02-05T08:27:57.218264+0000","last_active":"2024-02-05T08:28:45.733630+0000","last_peered":"2024-02-05T08:28:45.733630+0000","last_clean":"2024-02-05T08:28:45.733630+0000","last_became_active":"2024-02-05T08:25:49.127834+0000","last_became_peered":"2024-02-05T08:25:49.127834+0000","last_unstale":"2024-02-05T08:28:45.733630+0000","last_undegraded":"2024-02-05T08:28:45.733630+0000","last_fullsized":"2024-02-05T08:28:45.733630+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-02-05T08:27:57.218152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:27:57.218152+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:27.115332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"397'50","reported_seq":488,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.732802+0000","last_change":"2024-02-05T08:28:03.234464+0000","last_active":"2024-02-05T08:28:45.732802+0000","last_peered":"2024-02-05T08:28:45.732802+0000","last_clean":"2024-02-05T08:28:45.732802+0000","last_became_active":"2024-02-05T08:25:49.145063+0000","last_became_peered":"2024-02-05T08:25:49.145063+0000","last_unstale":"2024-02-05T08:28:45.732802+0000","last_undegraded":"2024-02-05T08:28:45.732802+0000","last_fullsized":"2024-02-05T08:28:45.732802+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'50","last_scrub_stamp":"2024-02-05T08:28:03.234373+0000","last_deep_scrub":"236'38","last_deep_scrub_stamp":"2024-02-05T08:17:05.909305+0000","last_clean_scrub_stamp":"2024-02-05T08:28:03.234373+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:31.440155+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":905,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735217+0000","last_change":"2024-02-05T08:27:46.017317+0000","last_active":"2024-02-05T08:28:45.735217+0000","last_peered":"2024-02-05T08:28:45.735217+0000","last_clean":"2024-02-05T08:28:45.735217+0000","last_became_active":"2024-02-05T08:18:34.636667+0000","last_became_peered":"2024-02-05T08:18:34.636667+0000","last_unstale":"2024-02-05T08:28:45.735217+0000","last_undegraded":"2024-02-05T08:28:45.735217+0000","last_fullsized":"2024-02-05T08:28:45.735217+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:46.017240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:27:46.017240+0000","last_clean_scrub_stamp":"2024-02-05T08:27:46.017240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:07.609848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"82'130","reported_seq":607,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:50.283456+0000","last_change":"2024-02-05T08:28:50.283486+0000","last_active":"2024-02-05T08:28:50.283456+0000","last_peered":"2024-02-05T08:28:50.283456+0000","last_clean":"2024-02-05T08:28:50.283456+0000","last_became_active":"2024-02-05T08:28:02.255460+0000","last_became_peered":"2024-02-05T08:28:02.255460+0000","last_unstale":"2024-02-05T08:28:50.283456+0000","last_undegraded":"2024-02-05T08:28:50.283456+0000","last_fullsized":"2024-02-05T08:28:50.283456+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:50.283378+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:28:50.283378+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:03.377431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":831,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.429990+0000","last_change":"2024-02-05T08:28:02.280306+0000","last_active":"2024-02-05T08:28:45.429990+0000","last_peered":"2024-02-05T08:28:45.429990+0000","last_clean":"2024-02-05T08:28:45.429990+0000","last_became_active":"2024-02-05T08:28:02.279886+0000","last_became_peered":"2024-02-05T08:28:02.279886+0000","last_unstale":"2024-02-05T08:28:45.429990+0000","last_undegraded":"2024-02-05T08:28:45.429990+0000","last_fullsized":"2024-02-05T08:28:45.429990+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:00.590484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:46.039077+0000","last_clean_scrub_stamp":"2024-02-05T08:28:00.590484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:23.841998+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"407'10","reported_seq":1215,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:01.870614+0000","last_change":"2024-02-05T08:28:12.006316+0000","last_active":"2024-02-05T08:29:01.870614+0000","last_peered":"2024-02-05T08:29:01.870614+0000","last_clean":"2024-02-05T08:29:01.870614+0000","last_became_active":"2024-02-05T08:25:49.113977+0000","last_became_peered":"2024-02-05T08:25:49.113977+0000","last_unstale":"2024-02-05T08:29:01.870614+0000","last_undegraded":"2024-02-05T08:29:01.870614+0000","last_fullsized":"2024-02-05T08:29:01.870614+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"407'10","last_scrub_stamp":"2024-02-05T08:28:12.006214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:28:12.006214+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:33.082745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":400,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:54.107191+0000","last_change":"2024-02-05T08:28:54.107212+0000","last_active":"2024-02-05T08:28:54.107191+0000","last_peered":"2024-02-05T08:28:54.107191+0000","last_clean":"2024-02-05T08:28:54.107191+0000","last_became_active":"2024-02-05T08:23:07.937468+0000","last_became_peered":"2024-02-05T08:23:07.937468+0000","last_unstale":"2024-02-05T08:28:54.107191+0000","last_undegraded":"2024-02-05T08:28:54.107191+0000","last_fullsized":"2024-02-05T08:28:54.107191+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:54.107124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:28:54.107124+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:16.372407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"397'39","reported_seq":882,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.738796+0000","last_change":"2024-02-05T08:28:45.738826+0000","last_active":"2024-02-05T08:28:45.738796+0000","last_peered":"2024-02-05T08:28:45.738796+0000","last_clean":"2024-02-05T08:28:45.738796+0000","last_became_active":"2024-02-05T08:23:07.939747+0000","last_became_peered":"2024-02-05T08:23:07.939747+0000","last_unstale":"2024-02-05T08:28:45.738796+0000","last_undegraded":"2024-02-05T08:28:45.738796+0000","last_fullsized":"2024-02-05T08:28:45.738796+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'39","last_scrub_stamp":"2024-02-05T08:28:45.738720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:45.738720+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:14.543320+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":399,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733383+0000","last_change":"2024-02-05T08:28:23.104290+0000","last_active":"2024-02-05T08:28:45.733383+0000","last_peered":"2024-02-05T08:28:45.733383+0000","last_clean":"2024-02-05T08:28:45.733383+0000","last_became_active":"2024-02-05T08:18:34.627261+0000","last_became_peered":"2024-02-05T08:18:34.627261+0000","last_unstale":"2024-02-05T08:28:45.733383+0000","last_undegraded":"2024-02-05T08:28:45.733383+0000","last_fullsized":"2024-02-05T08:28:45.733383+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:23.104206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:06:36.942968+0000","last_clean_scrub_stamp":"2024-02-05T08:28:23.104206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:47.811650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":844,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.430962+0000","last_change":"2024-02-05T08:28:40.603832+0000","last_active":"2024-02-05T08:28:45.430962+0000","last_peered":"2024-02-05T08:28:45.430962+0000","last_clean":"2024-02-05T08:28:45.430962+0000","last_became_active":"2024-02-05T08:28:02.685842+0000","last_became_peered":"2024-02-05T08:28:02.685842+0000","last_unstale":"2024-02-05T08:28:45.430962+0000","last_undegraded":"2024-02-05T08:28:45.430962+0000","last_fullsized":"2024-02-05T08:28:45.430962+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:40.603745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:28:40.603745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:47.829322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":901,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735242+0000","last_change":"2024-02-05T08:27:36.142024+0000","last_active":"2024-02-05T08:28:45.735242+0000","last_peered":"2024-02-05T08:28:45.735242+0000","last_clean":"2024-02-05T08:28:45.735242+0000","last_became_active":"2024-02-05T08:04:41.542630+0000","last_became_peered":"2024-02-05T08:04:41.542630+0000","last_unstale":"2024-02-05T08:28:45.735242+0000","last_undegraded":"2024-02-05T08:28:45.735242+0000","last_fullsized":"2024-02-05T08:28:45.735242+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:36.141928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:24:38.141536+0000","last_clean_scrub_stamp":"2024-02-05T08:27:36.141928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:03.496668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"465'32","reported_seq":981,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735990+0000","last_change":"2024-02-05T08:28:36.264397+0000","last_active":"2024-02-05T08:28:45.735990+0000","last_peered":"2024-02-05T08:28:45.735990+0000","last_clean":"2024-02-05T08:28:45.735990+0000","last_became_active":"2024-02-05T08:23:07.937017+0000","last_became_peered":"2024-02-05T08:23:07.937017+0000","last_unstale":"2024-02-05T08:28:45.735990+0000","last_undegraded":"2024-02-05T08:28:45.735990+0000","last_fullsized":"2024-02-05T08:28:45.735990+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'32","last_scrub_stamp":"2024-02-05T08:28:36.264233+0000","last_deep_scrub":"465'32","last_deep_scrub_stamp":"2024-02-05T08:28:36.264233+0000","last_clean_scrub_stamp":"2024-02-05T08:28:36.264233+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:41.909540+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"82'130","reported_seq":607,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:54.259939+0000","last_change":"2024-02-05T08:28:54.259989+0000","last_active":"2024-02-05T08:28:54.259939+0000","last_peered":"2024-02-05T08:28:54.259939+0000","last_clean":"2024-02-05T08:28:54.259939+0000","last_became_active":"2024-02-05T08:28:02.253635+0000","last_became_peered":"2024-02-05T08:28:02.253635+0000","last_unstale":"2024-02-05T08:28:54.259939+0000","last_undegraded":"2024-02-05T08:28:54.259939+0000","last_fullsized":"2024-02-05T08:28:54.259939+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:54.259849+0000","last_deep_scrub":"78'119","last_deep_scrub_stamp":"2024-02-05T08:05:16.657600+0000","last_clean_scrub_stamp":"2024-02-05T08:28:54.259849+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:02.635631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"74'1","reported_seq":846,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.006186+0000","last_change":"2024-02-05T08:27:41.707412+0000","last_active":"2024-02-05T08:28:46.006186+0000","last_peered":"2024-02-05T08:28:46.006186+0000","last_clean":"2024-02-05T08:28:46.006186+0000","last_became_active":"2024-02-05T08:27:39.134979+0000","last_became_peered":"2024-02-05T08:27:39.134979+0000","last_unstale":"2024-02-05T08:28:46.006186+0000","last_undegraded":"2024-02-05T08:28:46.006186+0000","last_fullsized":"2024-02-05T08:28:46.006186+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-02-05T08:27:41.707282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:46.546354+0000","last_clean_scrub_stamp":"2024-02-05T08:27:41.707282+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:06.682434+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":692,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.434374+0000","last_change":"2024-02-05T08:28:23.479856+0000","last_active":"2024-02-05T08:28:45.434374+0000","last_peered":"2024-02-05T08:28:45.434374+0000","last_clean":"2024-02-05T08:28:45.434374+0000","last_became_active":"2024-02-05T08:28:23.479539+0000","last_became_peered":"2024-02-05T08:28:23.479539+0000","last_unstale":"2024-02-05T08:28:45.434374+0000","last_undegraded":"2024-02-05T08:28:45.434374+0000","last_fullsized":"2024-02-05T08:28:45.434374+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:55.984303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:44.536163+0000","last_clean_scrub_stamp":"2024-02-05T08:27:55.984303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:07.901302+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":647,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.007469+0000","last_change":"2024-02-05T08:27:36.429731+0000","last_active":"2024-02-05T08:28:46.007469+0000","last_peered":"2024-02-05T08:28:46.007469+0000","last_clean":"2024-02-05T08:28:46.007469+0000","last_became_active":"2024-02-05T08:23:07.929749+0000","last_became_peered":"2024-02-05T08:23:07.929749+0000","last_unstale":"2024-02-05T08:28:46.007469+0000","last_undegraded":"2024-02-05T08:28:46.007469+0000","last_fullsized":"2024-02-05T08:28:46.007469+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:36.429639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:07:53.915449+0000","last_clean_scrub_stamp":"2024-02-05T08:27:36.429639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:06.285395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"397'45","reported_seq":967,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:57.112277+0000","last_change":"2024-02-05T08:28:57.112311+0000","last_active":"2024-02-05T08:28:57.112277+0000","last_peered":"2024-02-05T08:28:57.112277+0000","last_clean":"2024-02-05T08:28:57.112277+0000","last_became_active":"2024-02-05T08:23:07.930130+0000","last_became_peered":"2024-02-05T08:23:07.930130+0000","last_unstale":"2024-02-05T08:28:57.112277+0000","last_undegraded":"2024-02-05T08:28:57.112277+0000","last_fullsized":"2024-02-05T08:28:57.112277+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'45","last_scrub_stamp":"2024-02-05T08:28:57.112205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:42.523363+0000","last_clean_scrub_stamp":"2024-02-05T08:28:57.112205+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:24.661110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"82'130","reported_seq":1208,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:46.005641+0000","last_change":"2024-02-05T08:28:07.143373+0000","last_active":"2024-02-05T08:28:46.005641+0000","last_peered":"2024-02-05T08:28:46.005641+0000","last_clean":"2024-02-05T08:28:46.005641+0000","last_became_active":"2024-02-05T08:19:01.816744+0000","last_became_peered":"2024-02-05T08:19:01.816744+0000","last_unstale":"2024-02-05T08:28:46.005641+0000","last_undegraded":"2024-02-05T08:28:46.005641+0000","last_fullsized":"2024-02-05T08:28:46.005641+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":301,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":4,"last_scrub":"82'130","last_scrub_stamp":"2024-02-05T08:28:07.143254+0000","last_deep_scrub":"82'130","last_deep_scrub_stamp":"2024-02-05T08:28:07.143254+0000","last_clean_scrub_stamp":"2024-02-05T08:28:07.143254+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:37.096677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":889,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:59.149691+0000","last_change":"2024-02-05T08:28:59.149718+0000","last_active":"2024-02-05T08:28:59.149691+0000","last_peered":"2024-02-05T08:28:59.149691+0000","last_clean":"2024-02-05T08:28:59.149691+0000","last_became_active":"2024-02-05T08:23:07.937641+0000","last_became_peered":"2024-02-05T08:23:07.937641+0000","last_unstale":"2024-02-05T08:28:59.149691+0000","last_undegraded":"2024-02-05T08:28:59.149691+0000","last_fullsized":"2024-02-05T08:28:59.149691+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:28:59.149580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:11:15.360090+0000","last_clean_scrub_stamp":"2024-02-05T08:28:59.149580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:15.915198+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"448'11","reported_seq":1102,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:29:06.300496+0000","last_change":"2024-02-05T08:29:06.300496+0000","last_active":"2024-02-05T08:29:06.300496+0000","last_peered":"2024-02-05T08:29:06.300496+0000","last_clean":"2024-02-05T08:29:06.300496+0000","last_became_active":"2024-02-05T08:27:39.136144+0000","last_became_peered":"2024-02-05T08:27:39.136144+0000","last_unstale":"2024-02-05T08:29:06.300496+0000","last_undegraded":"2024-02-05T08:29:06.300496+0000","last_fullsized":"2024-02-05T08:29:06.300496+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'11","last_scrub_stamp":"2024-02-05T08:29:06.300366+0000","last_deep_scrub":"129'9","last_deep_scrub_stamp":"2024-02-05T08:08:39.741871+0000","last_clean_scrub_stamp":"2024-02-05T08:29:06.300366+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:30:21.092870+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":415,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.733570+0000","last_change":"2024-02-05T08:27:43.175780+0000","last_active":"2024-02-05T08:28:45.733570+0000","last_peered":"2024-02-05T08:28:45.733570+0000","last_clean":"2024-02-05T08:28:45.733570+0000","last_became_active":"2024-02-05T08:25:49.138786+0000","last_became_peered":"2024-02-05T08:25:49.138786+0000","last_unstale":"2024-02-05T08:28:45.733570+0000","last_undegraded":"2024-02-05T08:28:45.733570+0000","last_fullsized":"2024-02-05T08:28:45.733570+0000","mapping_epoch":406,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T08:27:43.175670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T08:04:40.511631+0000","last_clean_scrub_stamp":"2024-02-05T08:27:43.175670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:28:51.266108+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"397'35","reported_seq":1026,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-02-05T08:28:45.735729+0000","last_change":"2024-02-05T08:28:15.002778+0000","last_active":"2024-02-05T08:28:45.735729+0000","last_peered":"2024-02-05T08:28:45.735729+0000","last_clean":"2024-02-05T08:28:45.735729+0000","last_became_active":"2024-02-05T08:18:34.637413+0000","last_became_peered":"2024-02-05T08:18:34.637413+0000","last_unstale":"2024-02-05T08:28:45.735729+0000","last_undegraded":"2024-02-05T08:28:45.735729+0000","last_fullsized":"2024-02-05T08:28:45.735729+0000","mapping_epoch":294,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":295,"parent":"0.0","parent_split_bits":0,"last_scrub":"397'35","last_scrub_stamp":"2024-02-05T08:28:15.002667+0000","last_deep_scrub":"397'35","last_deep_scrub_stamp":"2024-02-05T08:28:15.002667+0000","last_clean_scrub_stamp":"2024-02-05T08:28:15.002667+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T08:29:28.843134+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1441,"num_read_kb":1436,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1539,"num_read_kb":1296,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":7882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1094,"ondisk_log_size":1094,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038059,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":786080,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92959072,"statfs":{"total":95995035648,"available":95190089728,"internally_reserved":0,"allocated":1073152,"data_stored":762760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":803862986},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.99299999999999999,"15min":1.347},"min":{"1min":0.38300000000000001,"5min":0.26100000000000001,"15min":0.20399999999999999},"max":{"1min":0.74099999999999999,"5min":6.5650000000000004,"15min":52.488999999999997},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.876,"15min":1.3939999999999999},"min":{"1min":0.31,"5min":0.23799999999999999,"15min":0.23200000000000001},"max":{"1min":0.77200000000000002,"5min":4.633,"15min":52.335000000000001},"last":0.57799999999999996}]},{"osd":1,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.3779999999999999,"15min":1.248},"min":{"1min":0.39800000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.80700000000000005,"5min":31.495999999999999,"15min":31.495999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.387,"15min":1.3069999999999999},"min":{"1min":0.35999999999999999,"5min":0.249,"15min":0.249},"max":{"1min":0.71699999999999997,"5min":31.547000000000001,"15min":31.547000000000001},"last":0.69499999999999995}]},{"osd":2,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":1.5549999999999999,"15min":1.288},"min":{"1min":0.34000000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.749,"5min":31.585999999999999,"15min":31.585999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":1.546,"15min":1.3100000000000001},"min":{"1min":0.318,"5min":0.255,"15min":0.248},"max":{"1min":0.77000000000000002,"5min":31.614999999999998,"15min":31.614999999999998},"last":0.56299999999999994}]},{"osd":3,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.9419999999999999,"15min":1.323},"min":{"1min":0.45600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.748,"5min":34.792999999999999,"15min":34.792999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":1.6910000000000001,"15min":1.246},"min":{"1min":0.372,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.78500000000000003,"5min":35.082000000000001,"15min":35.082000000000001},"last":0.60399999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":1.1080000000000001,"15min":1.2310000000000001},"min":{"1min":0.33600000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.72099999999999997,"5min":6.5289999999999999,"15min":49.601999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.0349999999999999,"15min":1.2649999999999999},"min":{"1min":0.34899999999999998,"5min":0.30199999999999999,"15min":0.249},"max":{"1min":0.81899999999999995,"5min":11.890000000000001,"15min":49.795999999999999},"last":0.44800000000000001}]},{"osd":5,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.995,"15min":1.3109999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.25800000000000001},"max":{"1min":0.79600000000000004,"5min":9.6980000000000004,"15min":49.914000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.022,"15min":1.3420000000000001},"min":{"1min":0.39300000000000002,"5min":0.31900000000000001,"15min":0.28999999999999998},"max":{"1min":0.81399999999999995,"5min":6.5410000000000004,"15min":52.256},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":1.0629999999999999,"15min":1.637},"min":{"1min":0.33400000000000002,"5min":0.30299999999999999,"15min":0.28599999999999998},"max":{"1min":0.83899999999999997,"5min":4.4610000000000003,"15min":84.254999999999995},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.622,"5min":1.0940000000000001,"15min":1.6699999999999999},"min":{"1min":0.45600000000000002,"5min":0.34999999999999998,"15min":0.32500000000000001},"max":{"1min":0.86399999999999999,"5min":11.866,"15min":84.667000000000002},"last":0.51400000000000001}]}]},{"osd":6,"up_from":51,"seq":219043332401,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":681608,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93063544,"statfs":{"total":95995035648,"available":95297069056,"internally_reserved":0,"allocated":720896,"data_stored":433155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":697235910},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.92500000000000004,"15min":1.8440000000000001},"min":{"1min":0.33900000000000002,"5min":0.23100000000000001,"15min":0.19600000000000001},"max":{"1min":0.84099999999999997,"5min":4.8040000000000003,"15min":102.819},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.73599999999999999,"15min":1.5840000000000001},"min":{"1min":0.35599999999999998,"5min":0.246,"15min":0.214},"max":{"1min":0.79100000000000004,"5min":5.6369999999999996,"15min":102.75},"last":0.51600000000000001}]},{"osd":1,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.7609999999999999,"15min":1.909},"min":{"1min":0.33500000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.92000000000000004,"5min":35.718000000000004,"15min":35.718000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60099999999999998,"5min":1.827,"15min":2.2450000000000001},"min":{"1min":0.38400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.89300000000000002,"5min":35.780000000000001,"15min":35.780000000000001},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Feb 5 08:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":1.002,"15min":1.1579999999999999},"min":{"1min":0.35199999999999998,"5min":0.26600000000000001,"15min":0.26000000000000001},"max":{"1min":0.752,"5min":6.6550000000000002,"15min":25.751000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.92400000000000004,"15min":1.1040000000000001},"min":{"1min":0.42499999999999999,"5min":0.26000000000000001,"15min":0.23400000000000001},"max":{"1min":0.86099999999999999,"5min":6.3949999999999996,"15min":26.210999999999999},"last":0.64900000000000002}]},{"osd":3,"last update":"Mon Feb 5 08:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":1.6140000000000001,"15min":1.6499999999999999},"min":{"1min":0.39600000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73799999999999999,"5min":50.380000000000003,"15min":102.438},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55600000000000005,"5min":1.742,"15min":1.6970000000000001},"min":{"1min":0.40500000000000003,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80600000000000005,"5min":50.136000000000003,"15min":102.288},"last":0.625}]},{"osd":4,"last update":"Mon Feb 5 08:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.80000000000000004,"15min":1.25},"min":{"1min":0.36699999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.78700000000000003,"5min":5.5880000000000001,"15min":46.125},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.89300000000000002,"15min":1.399},"min":{"1min":0.36299999999999999,"5min":0.29899999999999999,"15min":0.25600000000000001},"max":{"1min":0.85999999999999999,"5min":5.5039999999999996,"15min":48.817999999999998},"last":0.56699999999999995}]},{"osd":5,"last update":"Mon Feb 5 08:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.81000000000000005,"15min":1.1599999999999999},"min":{"1min":0.41899999999999998,"5min":0.28499999999999998,"15min":0.26000000000000001},"max":{"1min":0.82799999999999996,"5min":6.3209999999999997,"15min":24.675999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.89100000000000001,"15min":1.208},"min":{"1min":0.33800000000000002,"5min":0.27100000000000002,"15min":0.246},"max":{"1min":0.88400000000000001,"5min":6.415,"15min":19.132000000000001},"last":0.61199999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:28:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.76800000000000002,"15min":1.137},"min":{"1min":0.42199999999999999,"5min":0.35999999999999999,"15min":0.307},"max":{"1min":0.753,"5min":3.3959999999999999,"15min":24.728999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.0129999999999999,"15min":1.381},"min":{"1min":0.375,"5min":0.311,"15min":0.30199999999999999},"max":{"1min":0.88600000000000001,"5min":4.6420000000000003,"15min":21.91},"last":0.66700000000000004}]}]},{"osd":1,"up_from":366,"seq":1571958030410,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":227508,"kb_used_data":748,"kb_used_omap":4,"kb_used_meta":226747,"kb_avail":93517644,"statfs":{"total":95995035648,"available":95762067456,"internally_reserved":0,"allocated":765952,"data_stored":434678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":232189606},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.89100000000000001,"15min":0.70899999999999996},"min":{"1min":0.23400000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.443,"5min":6.5999999999999996,"15min":6.5999999999999996},"last":0.308},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.98099999999999998,"15min":1.0600000000000001},"min":{"1min":0.22900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.62,"5min":10.927,"15min":10.927},"last":0.42799999999999999}]},{"osd":2,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":1.3009999999999999,"15min":1.488},"min":{"1min":0.28100000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69799999999999995,"5min":7.3289999999999997,"15min":7.3289999999999997},"last":0.38},{"interface":"front","average":{"1min":0.36099999999999999,"5min":1.3129999999999999,"15min":1.2829999999999999},"min":{"1min":0.27800000000000002,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.46000000000000002,"5min":6.7949999999999999,"15min":6.7949999999999999},"last":0.36099999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":1.1990000000000001,"15min":1.2430000000000001},"min":{"1min":0.30199999999999999,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":7.3719999999999999,"15min":7.3719999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.35899999999999999,"5min":1.5209999999999999,"15min":1.3049999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.55300000000000005,"5min":8.6150000000000002,"15min":8.6150000000000002},"last":0.45900000000000002}]},{"osd":4,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":1.335,"15min":1.286},"min":{"1min":0.40500000000000003,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77300000000000002,"5min":6.3920000000000003,"15min":6.3920000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":1.085,"15min":1.1579999999999999},"min":{"1min":0.41499999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84199999999999997,"5min":7.1890000000000001,"15min":7.1890000000000001},"last":0.52900000000000003}]},{"osd":5,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":1.2889999999999999,"15min":1.1619999999999999},"min":{"1min":0.433,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.83099999999999996,"5min":8.6999999999999993,"15min":8.6999999999999993},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":1.298,"15min":1.1479999999999999},"min":{"1min":0.41499999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.80800000000000005,"5min":6.4240000000000004,"15min":6.4240000000000004},"last":0.44500000000000001}]},{"osd":6,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":1.698,"15min":1.3660000000000001},"min":{"1min":0.44800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.89500000000000002,"5min":42.783000000000001,"15min":42.783000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":1.7050000000000001,"15min":1.3839999999999999},"min":{"1min":0.438,"5min":0.372,"15min":0.372},"max":{"1min":0.82899999999999996,"5min":42.898000000000003,"15min":42.898000000000003},"last":0.498}]},{"osd":7,"last update":"Mon Feb 5 08:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.8160000000000001,"15min":1.4019999999999999},"min":{"1min":0.40600000000000003,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84999999999999998,"5min":64.049999999999997,"15min":64.049999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":2.04,"15min":1.6040000000000001},"min":{"1min":0.41199999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85199999999999998,"5min":64.144000000000005,"15min":64.144000000000005},"last":0.54900000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738703,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":852388,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":92892764,"statfs":{"total":95995035648,"available":95122190336,"internally_reserved":0,"allocated":749568,"data_stored":434823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 08:28:12 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.86399999999999999,"15min":0.78200000000000003},"min":{"1min":0.21299999999999999,"5min":0.19,"15min":0.19},"max":{"1min":0.47799999999999998,"5min":5.1970000000000001,"15min":5.1970000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.999,"15min":0.90000000000000002},"min":{"1min":0.218,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.55600000000000005,"5min":6.718,"15min":6.718},"last":0.38700000000000001}]},{"osd":2,"last update":"Mon Feb 5 08:28:09 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.82799999999999996,"15min":1.4039999999999999},"min":{"1min":0.23599999999999999,"5min":0.22500000000000001,"15min":0.161},"max":{"1min":0.57599999999999996,"5min":6.9889999999999999,"15min":105.96599999999999},"last":0.374},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.98399999999999999,"15min":1.377},"min":{"1min":0.28299999999999997,"5min":0.252,"15min":0.16600000000000001},"max":{"1min":0.60899999999999999,"5min":6.8849999999999998,"15min":106.137},"last":0.41899999999999998}]},{"osd":3,"last update":"Mon Feb 5 08:28:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.98599999999999999,"15min":0.97899999999999998},"min":{"1min":0.27000000000000002,"5min":0.254,"15min":0.20899999999999999},"max":{"1min":0.53500000000000003,"5min":6.6849999999999996,"15min":7.3929999999999998},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.43099999999999999,"5min":1.04,"15min":0.95799999999999996},"min":{"1min":0.29299999999999998,"5min":0.24199999999999999,"15min":0.219},"max":{"1min":0.58099999999999996,"5min":6.056,"15min":8.0239999999999991},"last":0.35299999999999998}]},{"osd":4,"last update":"Mon Feb 5 08:28:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.4510000000000001,"15min":1.4550000000000001},"min":{"1min":0.40500000000000003,"5min":0.30099999999999999,"15min":0.27700000000000002},"max":{"1min":0.78400000000000003,"5min":59.350999999999999,"15min":59.350999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.627,"5min":1.4339999999999999,"15min":1.476},"min":{"1min":0.44,"5min":0.30299999999999999,"15min":0.254},"max":{"1min":0.80900000000000005,"5min":59.299999999999997,"15min":59.299999999999997},"last":0.72399999999999998}]},{"osd":5,"last update":"Mon Feb 5 08:28:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.0800000000000001,"15min":1.5229999999999999},"min":{"1min":0.41099999999999998,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":0.78600000000000003,"5min":15.914,"15min":38.979999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.97899999999999998,"15min":1.4810000000000001},"min":{"1min":0.33700000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.83899999999999997,"5min":15.887,"15min":38.960999999999999},"last":0.65500000000000003}]},{"osd":6,"last update":"Mon Feb 5 08:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.163,"15min":1.21},"min":{"1min":0.48299999999999998,"5min":0.34000000000000002,"15min":0.30199999999999999},"max":{"1min":0.80600000000000005,"5min":16.725999999999999,"15min":16.725999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":1.294,"15min":1.206},"min":{"1min":0.496,"5min":0.32900000000000001,"15min":0.27100000000000002},"max":{"1min":0.82799999999999996,"5min":16.803999999999998,"15min":16.803999999999998},"last":0.59999999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:28:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.151,"15min":1.6779999999999999},"min":{"1min":0.46100000000000002,"5min":0.32900000000000001,"15min":0.29299999999999998},"max":{"1min":0.84499999999999997,"5min":16.007000000000001,"15min":68.241},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.629,"5min":1.232,"15min":1.7490000000000001},"min":{"1min":0.47099999999999997,"5min":0.36799999999999999,"15min":0.252},"max":{"1min":0.82399999999999995,"5min":15.952999999999999,"15min":68.980000000000004},"last":0.754}]}]},{"osd":2,"up_from":18,"seq":77309411653,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":790180,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":92954972,"statfs":{"total":95995035648,"available":95185891328,"internally_reserved":0,"allocated":1011712,"data_stored":696566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":808122822},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.94499999999999995,"15min":1.4419999999999999},"min":{"1min":0.20599999999999999,"5min":0.20100000000000001,"15min":0.17999999999999999},"max":{"1min":0.46700000000000003,"5min":7.4900000000000002,"15min":107.155},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.79900000000000004,"15min":1.571},"min":{"1min":0.246,"5min":0.20399999999999999,"15min":0.20000000000000001},"max":{"1min":0.71099999999999997,"5min":7.8410000000000002,"15min":107.221},"last":0.47299999999999998}]},{"osd":1,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":1.3169999999999999,"15min":1.0780000000000001},"min":{"1min":0.29399999999999998,"5min":0.19,"15min":0.19},"max":{"1min":0.54600000000000004,"5min":7.6360000000000001,"15min":7.6360000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.36499999999999999,"5min":1.121,"15min":1.2210000000000001},"min":{"1min":0.24399999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.45400000000000001,"5min":7.0709999999999997,"15min":7.0709999999999997},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Feb 5 08:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":1.274,"15min":1.069},"min":{"1min":0.27200000000000002,"5min":0.21199999999999999,"15min":0.20999999999999999},"max":{"1min":0.56899999999999995,"5min":11.122,"15min":11.122},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":1.4179999999999999,"15min":0.98299999999999998},"min":{"1min":0.313,"5min":0.23999999999999999,"15min":0.20399999999999999},"max":{"1min":0.63300000000000001,"5min":7.8289999999999997,"15min":7.8289999999999997},"last":0.54000000000000004}]},{"osd":4,"last update":"Mon Feb 5 08:29:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.98799999999999999,"15min":1.1679999999999999},"min":{"1min":0.51200000000000001,"5min":0.30099999999999999,"15min":0.26400000000000001},"max":{"1min":0.84299999999999997,"5min":3.7490000000000001,"15min":8.2620000000000005},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.65500000000000003,"5min":1.137,"15min":1.1759999999999999},"min":{"1min":0.49199999999999999,"5min":0.307,"15min":0.28799999999999998},"max":{"1min":0.82699999999999996,"5min":10.307,"15min":10.307},"last":0.69699999999999995}]},{"osd":5,"last update":"Mon Feb 5 08:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":1.0529999999999999,"15min":1.3420000000000001},"min":{"1min":0.48699999999999999,"5min":0.33800000000000002,"15min":0.246},"max":{"1min":0.72799999999999998,"5min":7.3899999999999997,"15min":26.373000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":1.0900000000000001,"15min":1.2789999999999999},"min":{"1min":0.45400000000000001,"5min":0.32900000000000001,"15min":0.26900000000000002},"max":{"1min":0.84099999999999997,"5min":11.308999999999999,"15min":27.968},"last":0.66100000000000003}]},{"osd":6,"last update":"Mon Feb 5 08:29:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":1.0980000000000001,"15min":1.2470000000000001},"min":{"1min":0.443,"5min":0.34000000000000002,"15min":0.29199999999999998},"max":{"1min":0.77800000000000002,"5min":11.048,"15min":21.885000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.629,"5min":1.141,"15min":1.2030000000000001},"min":{"1min":0.497,"5min":0.41899999999999998,"15min":0.317},"max":{"1min":0.78500000000000003,"5min":7.4610000000000003,"15min":22.158000000000001},"last":0.65000000000000002}]},{"osd":7,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":2.0369999999999999,"15min":1.45},"min":{"1min":0.45300000000000001,"5min":0.33900000000000002,"15min":0.28899999999999998},"max":{"1min":0.76200000000000001,"5min":72.334999999999994,"15min":72.334999999999994},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":2.105,"15min":1.6950000000000001},"min":{"1min":0.41899999999999998,"5min":0.40200000000000002,"15min":0.32700000000000001},"max":{"1min":0.89000000000000001,"5min":72.222999999999999,"15min":72.222999999999999},"last":0.58199999999999996}]}]},{"osd":3,"up_from":27,"seq":115964117312,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704444,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93040708,"statfs":{"total":95995035648,"available":95273684992,"internally_reserved":0,"allocated":1363968,"data_stored":1025699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1591,"internal_metadata":719976905},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.83899999999999997,"15min":1.081},"min":{"1min":0.27200000000000002,"5min":0.20300000000000001,"15min":0.19700000000000001},"max":{"1min":0.46899999999999997,"5min":8.4700000000000006,"15min":52.831000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.92400000000000004,"15min":1.115},"min":{"1min":0.22600000000000001,"5min":0.218,"15min":0.17699999999999999},"max":{"1min":0.55500000000000005,"5min":10.613,"15min":53.859999999999999},"last":0.39400000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":1.3580000000000001,"15min":1.3999999999999999},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.63500000000000001,"5min":11.339,"15min":11.339},"last":0.378},{"interface":"front","average":{"1min":0.39000000000000001,"5min":1.4870000000000001,"15min":1.522},"min":{"1min":0.309,"5min":0.25,"15min":0.25},"max":{"1min":0.51100000000000001,"5min":10.744999999999999,"15min":10.744999999999999},"last":0.443}]},{"osd":2,"last update":"Mon Feb 5 08:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":1.1359999999999999,"15min":1.1100000000000001},"min":{"1min":0.246,"5min":0.218,"15min":0.218},"max":{"1min":0.441,"5min":6.5999999999999996,"15min":35.277000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":1.3300000000000001,"15min":1.141},"min":{"1min":0.33600000000000002,"5min":0.247,"15min":0.23200000000000001},"max":{"1min":0.53400000000000003,"5min":8.6039999999999992,"15min":35.442999999999998},"last":0.40799999999999997}]},{"osd":4,"last update":"Mon Feb 5 08:28:55 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.216,"15min":1.2569999999999999},"min":{"1min":0.47199999999999998,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":0.81399999999999995,"5min":8.2140000000000004,"15min":54.502000000000002},"last":0.497},{"interface":"front","average":{"1min":0.63700000000000001,"5min":1.167,"15min":1.161},"min":{"1min":0.44900000000000001,"5min":0.33500000000000002,"15min":0.29199999999999998},"max":{"1min":0.89600000000000002,"5min":10.912000000000001,"15min":54.540999999999997},"last":0.59099999999999997}]},{"osd":5,"last update":"Mon Feb 5 08:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":1.216,"15min":1.744},"min":{"1min":0.40799999999999997,"5min":0.308,"15min":0.28899999999999998},"max":{"1min":0.78400000000000003,"5min":8.2609999999999992,"15min":117.432},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.0109999999999999,"15min":1.569},"min":{"1min":0.39900000000000002,"5min":0.36699999999999999,"15min":0.27600000000000002},"max":{"1min":0.73599999999999999,"5min":4.0460000000000003,"15min":114.828},"last":0.57199999999999995}]},{"osd":6,"last update":"Mon Feb 5 08:28:38 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.363,"15min":1.2749999999999999},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.29899999999999999},"max":{"1min":0.85499999999999998,"5min":10.534000000000001,"15min":54.219999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":1.153,"15min":1.266},"min":{"1min":0.434,"5min":0.318,"15min":0.29099999999999998},"max":{"1min":0.76900000000000002,"5min":6.6820000000000004,"15min":54.530999999999999},"last":0.53100000000000003}]},{"osd":7,"last update":"Mon Feb 5 08:29:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":1.2290000000000001,"15min":1.6930000000000001},"min":{"1min":0.45700000000000002,"5min":0.36899999999999999,"15min":0.33500000000000002},"max":{"1min":0.877,"5min":8.4060000000000006,"15min":114.84999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.63,"5min":1.3700000000000001,"15min":1.647},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.36399999999999999},"max":{"1min":0.78100000000000003,"5min":10.58,"15min":114.871},"last":0.65100000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953787,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762148,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92983004,"statfs":{"total":95995035648,"available":95214596096,"internally_reserved":0,"allocated":749568,"data_stored":433727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.79000000000000004,"15min":1.1819999999999999},"min":{"1min":0.34100000000000003,"5min":0.23599999999999999,"15min":0.192},"max":{"1min":0.745,"5min":4.54,"15min":39.552},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.998,"15min":1.3620000000000001},"min":{"1min":0.41499999999999998,"5min":0.27700000000000002,"15min":0.23499999999999999},"max":{"1min":0.73299999999999998,"5min":5.593,"15min":39.350999999999999},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.89300000000000002,"15min":0.97299999999999998},"min":{"1min":0.35999999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.76800000000000002,"5min":4.9930000000000003,"15min":4.9930000000000003},"last":0.751},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.141,"15min":1.0229999999999999},"min":{"1min":0.35399999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.89300000000000002,"5min":9.8569999999999993,"15min":9.8569999999999993},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Feb 5 08:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":1.022,"15min":1.0800000000000001},"min":{"1min":0.39800000000000002,"5min":0.28299999999999997,"15min":0.23200000000000001},"max":{"1min":0.68000000000000005,"5min":6.1749999999999998,"15min":6.1749999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.59099999999999997,"5min":1.0880000000000001,"15min":1.1240000000000001},"min":{"1min":0.45800000000000002,"5min":0.27700000000000002,"15min":0.26700000000000002},"max":{"1min":0.69899999999999995,"5min":8.8490000000000002,"15min":8.8490000000000002},"last":0.73499999999999999}]},{"osd":3,"last update":"Mon Feb 5 08:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":1.53,"15min":1.4850000000000001},"min":{"1min":0.39700000000000002,"5min":0.32200000000000001,"15min":0.312},"max":{"1min":0.74399999999999999,"5min":29.905000000000001,"15min":38.985999999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":1.3440000000000001,"15min":1.3360000000000001},"min":{"1min":0.48399999999999999,"5min":0.30399999999999999,"15min":0.27100000000000002},"max":{"1min":0.80700000000000005,"5min":30.094999999999999,"15min":39.290999999999997},"last":0.79200000000000004}]},{"osd":5,"last update":"Mon Feb 5 08:28:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.85899999999999999,"15min":1.2490000000000001},"min":{"1min":0.32700000000000001,"5min":0.26100000000000001,"15min":0.24199999999999999},"max":{"1min":0.86199999999999999,"5min":5.468,"15min":34.167000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.76300000000000001,"15min":1.2629999999999999},"min":{"1min":0.34999999999999998,"5min":0.28299999999999997,"15min":0.24199999999999999},"max":{"1min":0.97999999999999998,"5min":4.4189999999999996,"15min":36.051000000000002},"last":0.434}]},{"osd":6,"last update":"Mon Feb 5 08:28:53 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.97599999999999998,"15min":1.1870000000000001},"min":{"1min":0.40000000000000002,"5min":0.32200000000000001,"15min":0.26900000000000002},"max":{"1min":0.72599999999999998,"5min":4.3879999999999999,"15min":6.375},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.96999999999999997,"15min":1.103},"min":{"1min":0.45300000000000001,"5min":0.29099999999999998,"15min":0.26100000000000001},"max":{"1min":0.81999999999999995,"5min":6.0179999999999998,"15min":6.9690000000000003},"last":0.60399999999999998}]},{"osd":7,"last update":"Mon Feb 5 08:29:00 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.93700000000000006,"15min":1.169},"min":{"1min":0.41299999999999998,"5min":0.32100000000000001,"15min":0.307},"max":{"1min":0.76500000000000001,"5min":5.8689999999999998,"15min":12.683999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.82799999999999996,"15min":1.127},"min":{"1min":0.48899999999999999,"5min":0.31,"15min":0.27900000000000003},"max":{"1min":0.80900000000000005,"5min":3.2730000000000001,"15min":7.6820000000000004},"last":0.49299999999999999}]}]},{"osd":5,"up_from":42,"seq":180388626741,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":762712,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92982440,"statfs":{"total":95995035648,"available":95214018560,"internally_reserved":0,"allocated":1327104,"data_stored":1024356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.76000000000000001,"15min":1.2869999999999999},"min":{"1min":0.34100000000000003,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.69999999999999996,"5min":4.7290000000000001,"15min":30.065999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.84499999999999997,"15min":1.246},"min":{"1min":0.41699999999999998,"5min":0.222,"15min":0.222},"max":{"1min":0.68100000000000005,"5min":5.4169999999999998,"15min":30.173999999999999},"last":0.64200000000000002}]},{"osd":1,"last update":"Mon Feb 5 08:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.863,"15min":2.3940000000000001},"min":{"1min":0.36299999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.71499999999999997,"5min":26.98,"15min":26.98},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":1.847,"15min":2.3220000000000001},"min":{"1min":0.46500000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71599999999999997,"5min":27.474,"15min":27.474},"last":0.57499999999999996}]},{"osd":2,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.78800000000000003,"15min":1.1080000000000001},"min":{"1min":0.34399999999999997,"5min":0.25800000000000001,"15min":0.245},"max":{"1min":0.73299999999999998,"5min":3.5910000000000002,"15min":17.09},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.53200000000000003,"5min":1.0109999999999999,"15min":1.248},"min":{"1min":0.38700000000000001,"5min":0.25800000000000001,"15min":0.25600000000000001},"max":{"1min":0.73799999999999999,"5min":3.9359999999999999,"15min":17.515999999999998},"last":0.66500000000000004}]},{"osd":3,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":1.841,"15min":1.393},"min":{"1min":0.41399999999999998,"5min":0.27300000000000002,"15min":0.26900000000000002},"max":{"1min":0.81399999999999995,"5min":27.725000000000001,"15min":27.725000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.708,"15min":1.446},"min":{"1min":0.376,"5min":0.307,"15min":0.28100000000000003},"max":{"1min":0.74299999999999999,"5min":27.241,"15min":27.241},"last":0.629}]},{"osd":4,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.72299999999999998,"15min":1.0529999999999999},"min":{"1min":0.29299999999999998,"5min":0.253,"15min":0.23699999999999999},"max":{"1min":0.64900000000000002,"5min":5.375,"15min":7.7389999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.84499999999999997,"15min":1.107},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.23300000000000001},"max":{"1min":0.746,"5min":6.0979999999999999,"15min":7.609},"last":0.39900000000000002}]},{"osd":6,"last update":"Mon Feb 5 08:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.81599999999999995,"15min":1.232},"min":{"1min":0.38200000000000001,"5min":0.28599999999999998,"15min":0.26600000000000001},"max":{"1min":0.73799999999999999,"5min":5.1989999999999998,"15min":30.581},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.84899999999999998,"15min":1.2250000000000001},"min":{"1min":0.505,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":0.83699999999999997,"5min":5.1909999999999998,"15min":30.620999999999999},"last":0.49299999999999999}]},{"osd":7,"last update":"Mon Feb 5 08:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.82299999999999995,"15min":1.3620000000000001},"min":{"1min":0.35999999999999999,"5min":0.30499999999999999,"15min":0.28799999999999998},"max":{"1min":0.874,"5min":4.7960000000000003,"15min":34.695},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.84199999999999997,"15min":1.2949999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.308},"max":{"1min":0.82399999999999995,"5min":5.444,"15min":34.514000000000003},"last":0.47699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T08:29:12.700 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-05T08:29:12.700 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-02-05T08:29:12.940 INFO:teuthology.orchestra.run.smithi134.stderr:Inferring config /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T08:29:13.378 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:13 smithi134 ceph-mon[30086]: from='client.26303 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:29:13.378 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:13 smithi134 ceph-mon[30086]: pgmap v1104: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:29:13.378 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:13 smithi134 ceph-mon[30086]: 1.7 scrub starts 2024-02-05T08:29:13.378 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:13 smithi134 ceph-mon[30086]: 1.7 scrub ok 2024-02-05T08:29:13.378 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:13 smithi134 ceph-mon[30086]: 2.1 scrub starts 2024-02-05T08:29:13.378 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:13 smithi134 ceph-mon[30086]: 2.1 scrub ok 2024-02-05T08:29:13.379 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:13 smithi134 ceph-mon[39278]: from='client.26303 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:29:13.379 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:13 smithi134 ceph-mon[39278]: pgmap v1104: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:29:13.379 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:13 smithi134 ceph-mon[39278]: 1.7 scrub starts 2024-02-05T08:29:13.379 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:13 smithi134 ceph-mon[39278]: 1.7 scrub ok 2024-02-05T08:29:13.379 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:13 smithi134 ceph-mon[39278]: 2.1 scrub starts 2024-02-05T08:29:13.379 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:13 smithi134 ceph-mon[39278]: 2.1 scrub ok 2024-02-05T08:29:13.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:13 smithi196 ceph-mon[33018]: from='client.26303 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T08:29:13.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:13 smithi196 ceph-mon[33018]: pgmap v1104: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:29:13.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:13 smithi196 ceph-mon[33018]: 1.7 scrub starts 2024-02-05T08:29:13.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:13 smithi196 ceph-mon[33018]: 1.7 scrub ok 2024-02-05T08:29:13.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:13 smithi196 ceph-mon[33018]: 2.1 scrub starts 2024-02-05T08:29:13.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:13 smithi196 ceph-mon[33018]: 2.1 scrub ok 2024-02-05T08:29:14.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[30086]: 3.13 deep-scrub starts 2024-02-05T08:29:14.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[30086]: 3.13 deep-scrub ok 2024-02-05T08:29:14.394 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[30086]: 3.b scrub starts 2024-02-05T08:29:14.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[30086]: 3.b scrub ok 2024-02-05T08:29:14.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[30086]: purged_snaps scrub starts 2024-02-05T08:29:14.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[30086]: purged_snaps scrub ok 2024-02-05T08:29:14.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[30086]: 5.15 scrub starts 2024-02-05T08:29:14.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[30086]: 5.15 scrub ok 2024-02-05T08:29:14.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[30086]: 6.f scrub starts 2024-02-05T08:29:14.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[30086]: 6.f scrub ok 2024-02-05T08:29:14.395 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[30086]: 6.17 scrub starts 2024-02-05T08:29:14.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[39278]: 3.13 deep-scrub starts 2024-02-05T08:29:14.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[39278]: 3.13 deep-scrub ok 2024-02-05T08:29:14.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[39278]: 3.b scrub starts 2024-02-05T08:29:14.395 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[39278]: 3.b scrub ok 2024-02-05T08:29:14.396 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[39278]: purged_snaps scrub starts 2024-02-05T08:29:14.396 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[39278]: purged_snaps scrub ok 2024-02-05T08:29:14.396 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[39278]: 5.15 scrub starts 2024-02-05T08:29:14.396 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[39278]: 5.15 scrub ok 2024-02-05T08:29:14.396 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[39278]: 6.f scrub starts 2024-02-05T08:29:14.396 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[39278]: 6.f scrub ok 2024-02-05T08:29:14.396 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-mon[39278]: 6.17 scrub starts 2024-02-05T08:29:14.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:29:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug there is no tcmu-runner data available 2024-02-05T08:29:14.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:14 smithi196 ceph-mon[33018]: 3.13 deep-scrub starts 2024-02-05T08:29:14.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:14 smithi196 ceph-mon[33018]: 3.13 deep-scrub ok 2024-02-05T08:29:14.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:14 smithi196 ceph-mon[33018]: 3.b scrub starts 2024-02-05T08:29:14.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:14 smithi196 ceph-mon[33018]: 3.b scrub ok 2024-02-05T08:29:14.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:14 smithi196 ceph-mon[33018]: purged_snaps scrub starts 2024-02-05T08:29:14.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:14 smithi196 ceph-mon[33018]: purged_snaps scrub ok 2024-02-05T08:29:14.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:14 smithi196 ceph-mon[33018]: 5.15 scrub starts 2024-02-05T08:29:14.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:14 smithi196 ceph-mon[33018]: 5.15 scrub ok 2024-02-05T08:29:14.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:14 smithi196 ceph-mon[33018]: 6.f scrub starts 2024-02-05T08:29:14.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:14 smithi196 ceph-mon[33018]: 6.f scrub ok 2024-02-05T08:29:14.466 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:14 smithi196 ceph-mon[33018]: 6.17 scrub starts 2024-02-05T08:29:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[39278]: pgmap v1105: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:29:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[39278]: osdmap e469: 8 total, 8 up, 8 in 2024-02-05T08:29:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[39278]: 6.17 scrub ok 2024-02-05T08:29:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[39278]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:29:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4144575414' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T08:29:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[39278]: from='client.? 172.21.15.134:0/4144575414' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-05T08:29:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:29:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:29:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:29:15.277 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[39278]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:29:15.278 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:14 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:14] ENGINE Bus STOPPING 2024-02-05T08:29:15.278 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:15] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T08:29:15.278 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:15] ENGINE Bus STOPPED 2024-02-05T08:29:15.278 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:15] ENGINE Bus STARTING 2024-02-05T08:29:15.278 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:15] ENGINE Serving on http://:::9283 2024-02-05T08:29:15.278 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:15] ENGINE Bus STARTED 2024-02-05T08:29:15.278 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[30086]: pgmap v1105: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:29:15.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[30086]: osdmap e469: 8 total, 8 up, 8 in 2024-02-05T08:29:15.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[30086]: 6.17 scrub ok 2024-02-05T08:29:15.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[30086]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:29:15.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4144575414' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T08:29:15.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[30086]: from='client.? 172.21.15.134:0/4144575414' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-05T08:29:15.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:29:15.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:29:15.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:29:15.279 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:15 smithi134 ceph-mon[30086]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:29:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:15 smithi196 ceph-mon[33018]: pgmap v1105: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T08:29:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:15 smithi196 ceph-mon[33018]: osdmap e469: 8 total, 8 up, 8 in 2024-02-05T08:29:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:15 smithi196 ceph-mon[33018]: 6.17 scrub ok 2024-02-05T08:29:15.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:15 smithi196 ceph-mon[33018]: from='client.24385 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T08:29:15.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:15 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4144575414' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T08:29:15.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:15 smithi196 ceph-mon[33018]: from='client.? 172.21.15.134:0/4144575414' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-05T08:29:15.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:15 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T08:29:15.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:15 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T08:29:15.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:15 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T08:29:15.565 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:15 smithi196 ceph-mon[33018]: from='mgr.24461 172.21.15.134:0/3193372251' entity='mgr.y' 2024-02-05T08:29:15.684 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-05T08:29:15.696 INFO:tasks.cephadm:Teardown begin 2024-02-05T08:29:15.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-05T08:29:15.735 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-05T08:29:15.790 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-05T08:29:15.791 DEBUG:teuthology.orchestra.run.smithi134:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-05T08:29:15.815 DEBUG:teuthology.orchestra.run.smithi196:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-05T08:29:15.841 INFO:tasks.cephadm:Stopping all daemons... 2024-02-05T08:29:15.841 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-05T08:29:15.841 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.a 2024-02-05T08:29:16.026 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:15 smithi134 systemd[1]: Stopping Ceph mon.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:29:16.501 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-mon[39278]: 4.4 scrub starts 2024-02-05T08:29:16.501 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-mon[39278]: 4.4 scrub ok 2024-02-05T08:29:16.501 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-mon[39278]: 3.1 scrub starts 2024-02-05T08:29:16.501 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-mon[39278]: 3.1 scrub ok 2024-02-05T08:29:16.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-mon[30086]: 4.4 scrub starts 2024-02-05T08:29:16.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-mon[30086]: 4.4 scrub ok 2024-02-05T08:29:16.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-mon[30086]: 3.1 scrub starts 2024-02-05T08:29:16.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-mon[30086]: 3.1 scrub ok 2024-02-05T08:29:16.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a[30063]: 2024-02-05T08:29:16.144+0000 7f45293fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:29:16.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a[30063]: 2024-02-05T08:29:16.144+0000 7f45293fe700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-05T08:29:16.502 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:16 smithi134 podman[240655]: 2024-02-05 08:29:16.186000023 +0000 UTC m=+0.140761717 container died 6397166ec857eebac547fa2f9d34f0ba0b61385d3127f224987e4a9fefa7ba1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T08:29:16.502 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:16] ENGINE Bus STOPPING 2024-02-05T08:29:16.502 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:16] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T08:29:16.503 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:16] ENGINE Bus STOPPED 2024-02-05T08:29:16.503 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:16] ENGINE Bus STARTING 2024-02-05T08:29:16.503 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:16] ENGINE Serving on http://:::9283 2024-02-05T08:29:16.503 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:16 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:16] ENGINE Bus STARTED 2024-02-05T08:29:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:16 smithi196 ceph-mon[33018]: 4.4 scrub starts 2024-02-05T08:29:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:16 smithi196 ceph-mon[33018]: 4.4 scrub ok 2024-02-05T08:29:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:16 smithi196 ceph-mon[33018]: 3.1 scrub starts 2024-02-05T08:29:16.564 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:16 smithi196 ceph-mon[33018]: 3.1 scrub ok 2024-02-05T08:29:16.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:16 smithi134 podman[240655]: 2024-02-05 08:29:16.709263382 +0000 UTC m=+0.664025124 container cleanup 6397166ec857eebac547fa2f9d34f0ba0b61385d3127f224987e4a9fefa7ba1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-05T08:29:16.777 INFO:journalctl@ceph.mon.a.smithi134.stdout:Feb 05 08:29:16 smithi134 bash[240655]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-a 2024-02-05T08:29:17.214 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.a.service' 2024-02-05T08:29:17.254 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:29:17.254 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-05T08:29:17.255 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-05T08:29:17.255 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.c 2024-02-05T08:29:17.527 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:17 smithi134 systemd[1]: Stopping Ceph mon.c for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:29:17.866 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-c[39255]: 2024-02-05T08:29:17.590+0000 7f352c524700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:29:17.867 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-c[39255]: 2024-02-05T08:29:17.590+0000 7f352c524700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-02-05T08:29:17.867 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:17] ENGINE Bus STOPPING 2024-02-05T08:29:17.867 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T08:29:17.867 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:17] ENGINE Bus STOPPED 2024-02-05T08:29:17.867 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:17] ENGINE Bus STARTING 2024-02-05T08:29:17.867 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:17] ENGINE Serving on http://:::9283 2024-02-05T08:29:17.867 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: [05/Feb/2024:08:29:17] ENGINE Bus STARTED 2024-02-05T08:29:17.867 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:17 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: ::ffff:172.21.15.196 - - [05/Feb/2024:08:29:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T08:29:18.164 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:17 smithi134 podman[240797]: 2024-02-05 08:29:17.867276487 +0000 UTC m=+0.389405628 container died 59ded051c19d862b7d30921ee79822829ec82c52091b6c8006f09a7cee7aff32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-c, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-05T08:29:18.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:18 smithi134 podman[240797]: 2024-02-05 08:29:18.322291914 +0000 UTC m=+0.844421100 container cleanup 59ded051c19d862b7d30921ee79822829ec82c52091b6c8006f09a7cee7aff32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-c, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T08:29:18.526 INFO:journalctl@ceph.mon.c.smithi134.stdout:Feb 05 08:29:18 smithi134 bash[240797]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-c 2024-02-05T08:29:18.764 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.c.service' 2024-02-05T08:29:18.801 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:29:18.801 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-05T08:29:18.801 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-05T08:29:18.801 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.b 2024-02-05T08:29:19.441 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:19 smithi196 systemd[1]: Stopping Ceph mon.b for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:29:19.696 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:19 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-b[32995]: 2024-02-05T08:29:19.441+0000 7f3073a8e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:29:19.696 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:19 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-b[32995]: 2024-02-05T08:29:19.441+0000 7f3073a8e700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-02-05T08:29:20.058 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:19 smithi196 podman[168587]: 2024-02-05 08:29:19.696569194 +0000 UTC m=+0.412281847 container died dd595281e22390b0715271bd408ec79e8a529e0f4f63da3b3079ad71b2042612 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T08:29:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:20 smithi196 podman[168587]: 2024-02-05 08:29:20.263487588 +0000 UTC m=+0.979200286 container cleanup dd595281e22390b0715271bd408ec79e8a529e0f4f63da3b3079ad71b2042612 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-b, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-05T08:29:20.314 INFO:journalctl@ceph.mon.b.smithi196.stdout:Feb 05 08:29:20 smithi196 bash[168587]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mon-b 2024-02-05T08:29:20.735 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mon.b.service' 2024-02-05T08:29:20.788 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:29:20.789 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-05T08:29:20.789 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-05T08:29:20.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mgr.y 2024-02-05T08:29:21.087 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:20 smithi134 systemd[1]: Stopping Ceph mgr.y for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:29:21.411 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:29:21.086+0000 7f216170c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:29:21.411 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:21 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y[30345]: 2024-02-05T08:29:21.086+0000 7f216170c700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-05T08:29:21.411 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:21 smithi134 podman[240939]: 2024-02-05 08:29:21.119408219 +0000 UTC m=+0.144209492 container died da2fb8fd1ea53c8d166129535579b17f315c1234ee36656af87803ed29134d76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y, ceph=True, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-05T08:29:21.776 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:21 smithi134 podman[240939]: 2024-02-05 08:29:21.568814049 +0000 UTC m=+0.593615337 container cleanup da2fb8fd1ea53c8d166129535579b17f315c1234ee36656af87803ed29134d76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T08:29:21.777 INFO:journalctl@ceph.mgr.y.smithi134.stdout:Feb 05 08:29:21 smithi134 bash[240939]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-y 2024-02-05T08:29:22.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mgr.y.service' 2024-02-05T08:29:22.041 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:29:22.041 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-05T08:29:22.041 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-05T08:29:22.041 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mgr.x 2024-02-05T08:29:22.410 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:29:22 smithi196 systemd[1]: Stopping Ceph mgr.x for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:29:22.711 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:29:22 smithi196 podman[168718]: 2024-02-05 08:29:22.410322038 +0000 UTC m=+0.176002336 container died 40e009f8fe59644dae71723ffcf98b5c22513d7b91d6622ec819eb88c46af66a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-05T08:29:23.019 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:29:22 smithi196 podman[168718]: 2024-02-05 08:29:22.801599815 +0000 UTC m=+0.567280153 container cleanup 40e009f8fe59644dae71723ffcf98b5c22513d7b91d6622ec819eb88c46af66a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-02-05T08:29:23.019 INFO:journalctl@ceph.mgr.x.smithi196.stdout:Feb 05 08:29:22 smithi196 bash[168718]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-mgr-x 2024-02-05T08:29:23.232 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@mgr.x.service' 2024-02-05T08:29:23.269 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:29:23.269 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-05T08:29:23.269 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-05T08:29:23.269 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.0 2024-02-05T08:29:23.609 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:23 smithi134 systemd[1]: Stopping Ceph osd.0 for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:29:24.026 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:23 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0[43745]: 2024-02-05T08:29:23.608+0000 7f74911e2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:29:24.027 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:23 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0[43745]: 2024-02-05T08:29:23.608+0000 7f74911e2700 -1 osd.0 469 *** Got signal Terminated *** 2024-02-05T08:29:24.027 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:23 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0[43745]: 2024-02-05T08:29:23.608+0000 7f74911e2700 -1 osd.0 469 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T08:29:29.527 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:29 smithi134 podman[241069]: 2024-02-05 08:29:29.271904254 +0000 UTC m=+5.816344585 container died 270e9c18c45bf523e528502e4fe202cc5eff9cdfca5be71fec1ccca8b4c11ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T08:29:30.026 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:29 smithi134 podman[241069]: 2024-02-05 08:29:29.730914742 +0000 UTC m=+6.275355064 container cleanup 270e9c18c45bf523e528502e4fe202cc5eff9cdfca5be71fec1ccca8b4c11ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-05T08:29:30.027 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:29 smithi134 bash[241069]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0 2024-02-05T08:29:30.383 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:30 smithi134 podman[241093]: 2024-02-05 08:29:30.062710874 +0000 UTC m=+0.787092061 container remove 270e9c18c45bf523e528502e4fe202cc5eff9cdfca5be71fec1ccca8b4c11ed9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-05T08:29:30.685 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:30 smithi134 podman[241191]: 2024-02-05 08:29:30.383561219 +0000 UTC m=+0.111555875 container create 4170c6842527e82828542df4fb214ce0e672c6127ed3b1d3637cc37cc8f05868 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-02-05T08:29:30.685 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:30 smithi134 podman[241191]: 2024-02-05 08:29:30.289849084 +0000 UTC m=+0.017843741 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:29:30.685 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:30 smithi134 podman[241191]: 2024-02-05 08:29:30.602562918 +0000 UTC m=+0.330557575 container init 4170c6842527e82828542df4fb214ce0e672c6127ed3b1d3637cc37cc8f05868 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-05T08:29:30.685 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:30 smithi134 podman[241191]: 2024-02-05 08:29:30.608429101 +0000 UTC m=+0.336423762 container start 4170c6842527e82828542df4fb214ce0e672c6127ed3b1d3637cc37cc8f05868 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2) 2024-02-05T08:29:30.685 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:30 smithi134 podman[241191]: 2024-02-05 08:29:30.685663731 +0000 UTC m=+0.413658399 container attach 4170c6842527e82828542df4fb214ce0e672c6127ed3b1d3637cc37cc8f05868 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0-deactivate, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-05T08:29:31.244 INFO:journalctl@ceph.osd.0.smithi134.stdout:Feb 05 08:29:30 smithi134 podman[241191]: 2024-02-05 08:29:30.970207791 +0000 UTC m=+0.698202454 container died 4170c6842527e82828542df4fb214ce0e672c6127ed3b1d3637cc37cc8f05868 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-05T08:29:31.744 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.0.service' 2024-02-05T08:29:31.784 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:29:31.785 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-05T08:29:31.785 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-05T08:29:31.785 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.1 2024-02-05T08:29:32.027 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:31 smithi134 systemd[1]: Stopping Ceph osd.1 for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:29:32.526 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:32 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1[202098]: 2024-02-05T08:29:32.137+0000 7f5319b89700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:29:32.527 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:32 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1[202098]: 2024-02-05T08:29:32.137+0000 7f5319b89700 -1 osd.1 469 *** Got signal Terminated *** 2024-02-05T08:29:32.527 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:32 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1[202098]: 2024-02-05T08:29:32.137+0000 7f5319b89700 -1 osd.1 469 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T08:29:37.810 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:37 smithi134 podman[241323]: 2024-02-05 08:29:37.543987604 +0000 UTC m=+5.535866514 container died c8f1a6e0ae3d94c172a1e94b09422e96042dbf575e083b86dc90918bbb540916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-05T08:29:38.144 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:37 smithi134 podman[241323]: 2024-02-05 08:29:37.984827524 +0000 UTC m=+5.976706435 container cleanup c8f1a6e0ae3d94c172a1e94b09422e96042dbf575e083b86dc90918bbb540916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T08:29:38.144 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:37 smithi134 bash[241323]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1 2024-02-05T08:29:38.526 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:38 smithi134 podman[241344]: 2024-02-05 08:29:38.144381799 +0000 UTC m=+0.599935210 container remove c8f1a6e0ae3d94c172a1e94b09422e96042dbf575e083b86dc90918bbb540916 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T08:29:38.795 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:38 smithi134 podman[241440]: 2024-02-05 08:29:38.445953988 +0000 UTC m=+0.017214106 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:29:38.795 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:38 smithi134 podman[241440]: 2024-02-05 08:29:38.553600208 +0000 UTC m=+0.124860324 container create 2ec7c68c5814d5fef8bb15578cfe9b6324eea041c3663a7e13690e968c7aca51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.2) 2024-02-05T08:29:38.795 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:38 smithi134 podman[241440]: 2024-02-05 08:29:38.789372923 +0000 UTC m=+0.360633051 container init 2ec7c68c5814d5fef8bb15578cfe9b6324eea041c3663a7e13690e968c7aca51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.2) 2024-02-05T08:29:39.100 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:38 smithi134 podman[241440]: 2024-02-05 08:29:38.795736905 +0000 UTC m=+0.366997019 container start 2ec7c68c5814d5fef8bb15578cfe9b6324eea041c3663a7e13690e968c7aca51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-05T08:29:39.101 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:38 smithi134 podman[241440]: 2024-02-05 08:29:38.897458896 +0000 UTC m=+0.468719026 container attach 2ec7c68c5814d5fef8bb15578cfe9b6324eea041c3663a7e13690e968c7aca51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-05T08:29:39.400 INFO:journalctl@ceph.osd.1.smithi134.stdout:Feb 05 08:29:39 smithi134 podman[241440]: 2024-02-05 08:29:39.101552483 +0000 UTC m=+0.672812609 container died 2ec7c68c5814d5fef8bb15578cfe9b6324eea041c3663a7e13690e968c7aca51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-1-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-05T08:29:39.729 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.1.service' 2024-02-05T08:29:39.772 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:29:39.773 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-05T08:29:39.773 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-05T08:29:39.773 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.2 2024-02-05T08:29:40.027 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:39 smithi134 systemd[1]: Stopping Ceph osd.2 for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:29:40.527 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:40 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2[56265]: 2024-02-05T08:29:40.132+0000 7f4ee93ad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:29:40.527 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:40 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2[56265]: 2024-02-05T08:29:40.132+0000 7f4ee93ad700 -1 osd.2 469 *** Got signal Terminated *** 2024-02-05T08:29:40.527 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:40 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2[56265]: 2024-02-05T08:29:40.132+0000 7f4ee93ad700 -1 osd.2 469 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T08:29:45.946 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:45 smithi134 podman[241571]: 2024-02-05 08:29:45.668627017 +0000 UTC m=+5.673140257 container died cac911e7529bde6601171ec64475d24accbbec1abc229950dd9549ae96a58706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-05T08:29:46.276 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:46 smithi134 podman[241571]: 2024-02-05 08:29:46.120415356 +0000 UTC m=+6.124928593 container cleanup cac911e7529bde6601171ec64475d24accbbec1abc229950dd9549ae96a58706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T08:29:46.277 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:46 smithi134 bash[241571]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2 2024-02-05T08:29:46.777 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:46 smithi134 podman[241593]: 2024-02-05 08:29:46.430300966 +0000 UTC m=+0.760664197 container remove cac911e7529bde6601171ec64475d24accbbec1abc229950dd9549ae96a58706 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-05T08:29:47.030 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:46 smithi134 podman[241689]: 2024-02-05 08:29:46.683329362 +0000 UTC m=+0.017854585 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:29:47.030 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:46 smithi134 podman[241689]: 2024-02-05 08:29:46.81469373 +0000 UTC m=+0.149218951 container create 9c3e2748ff314f363817ac8d67d1bb55a273dba13943f7c22b8c4cd1c0855ac6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T08:29:47.031 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:47 smithi134 podman[241689]: 2024-02-05 08:29:47.025015448 +0000 UTC m=+0.359540670 container init 9c3e2748ff314f363817ac8d67d1bb55a273dba13943f7c22b8c4cd1c0855ac6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2-deactivate, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131) 2024-02-05T08:29:47.031 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:47 smithi134 podman[241689]: 2024-02-05 08:29:47.031024671 +0000 UTC m=+0.365549901 container start 9c3e2748ff314f363817ac8d67d1bb55a273dba13943f7c22b8c4cd1c0855ac6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-05T08:29:47.326 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:47 smithi134 podman[241689]: 2024-02-05 08:29:47.10841298 +0000 UTC m=+0.442938237 container attach 9c3e2748ff314f363817ac8d67d1bb55a273dba13943f7c22b8c4cd1c0855ac6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-02-05T08:29:47.618 INFO:journalctl@ceph.osd.2.smithi134.stdout:Feb 05 08:29:47 smithi134 podman[241689]: 2024-02-05 08:29:47.327638637 +0000 UTC m=+0.662163874 container died 9c3e2748ff314f363817ac8d67d1bb55a273dba13943f7c22b8c4cd1c0855ac6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-2-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-05T08:29:48.025 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.2.service' 2024-02-05T08:29:48.066 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:29:48.066 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-05T08:29:48.066 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-05T08:29:48.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.3 2024-02-05T08:29:48.276 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:48 smithi134 systemd[1]: Stopping Ceph osd.3 for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:29:48.777 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:48 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3[62874]: 2024-02-05T08:29:48.383+0000 7fe2a5d84700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:29:48.777 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:48 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3[62874]: 2024-02-05T08:29:48.383+0000 7fe2a5d84700 -1 osd.3 469 *** Got signal Terminated *** 2024-02-05T08:29:48.777 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:48 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3[62874]: 2024-02-05T08:29:48.383+0000 7fe2a5d84700 -1 osd.3 469 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T08:29:51.259 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:29:50 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:29:50.876+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:29:51.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:29:51 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:29:51.259+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:29:52.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:29:51 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:29:51.857+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:29:52.064 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:51 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:51.791+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.861277+0000 front 2024-02-05T08:29:27.861399+0000 (oldest deadline 2024-02-05T08:29:51.361264+0000) 2024-02-05T08:29:52.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:29:52 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:29:52.276+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:29:52.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:29:52 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:29:52.281+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:29:53.233 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:52 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:52.837+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.861277+0000 front 2024-02-05T08:29:27.861399+0000 (oldest deadline 2024-02-05T08:29:51.361264+0000) 2024-02-05T08:29:53.234 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:29:52 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:29:52.886+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:29:53.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:29:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:29:53.236+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:29:53.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:29:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:29:53.233+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:29:54.064 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:53.813+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.861277+0000 front 2024-02-05T08:29:27.861399+0000 (oldest deadline 2024-02-05T08:29:51.361264+0000) 2024-02-05T08:29:54.065 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:29:53 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:29:53.894+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:29:54.158 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:53 smithi134 podman[241819]: 2024-02-05 08:29:53.88184119 +0000 UTC m=+5.598478790 container died d919eb90f264d87dd29cb967d55f532434ce1dce279b691eecafdcf41fd6f2ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T08:29:54.525 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:54 smithi134 podman[241819]: 2024-02-05 08:29:54.307756605 +0000 UTC m=+6.024394184 container cleanup d919eb90f264d87dd29cb967d55f532434ce1dce279b691eecafdcf41fd6f2ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-05T08:29:54.525 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:54 smithi134 bash[241819]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3 2024-02-05T08:29:54.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:29:54 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:29:54.225+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:29:54.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:29:54 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:29:54.260+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:29:54.776 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:54 smithi134 podman[241841]: 2024-02-05 08:29:54.525673985 +0000 UTC m=+0.637321758 container remove d919eb90f264d87dd29cb967d55f532434ce1dce279b691eecafdcf41fd6f2ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=) 2024-02-05T08:29:55.228 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:54 smithi134 podman[241936]: 2024-02-05 08:29:54.823312473 +0000 UTC m=+0.017770082 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:29:55.228 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:54 smithi134 podman[241936]: 2024-02-05 08:29:54.943243016 +0000 UTC m=+0.137700617 container create 9258f59bcf5451676531e8d6e45fcbc6ff1c641c6f1f0718610af4b41ccecace (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_BRANCH=HEAD) 2024-02-05T08:29:55.228 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:55 smithi134 podman[241936]: 2024-02-05 08:29:55.145312366 +0000 UTC m=+0.339769967 container init 9258f59bcf5451676531e8d6e45fcbc6ff1c641c6f1f0718610af4b41ccecace (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-05T08:29:55.228 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:55 smithi134 podman[241936]: 2024-02-05 08:29:55.150692763 +0000 UTC m=+0.345150364 container start 9258f59bcf5451676531e8d6e45fcbc6ff1c641c6f1f0718610af4b41ccecace (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3-deactivate, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-05T08:29:55.228 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:55 smithi134 podman[241936]: 2024-02-05 08:29:55.228419013 +0000 UTC m=+0.422876622 container attach 9258f59bcf5451676531e8d6e45fcbc6ff1c641c6f1f0718610af4b41ccecace (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-02-05T08:29:55.238 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:29:54 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:29:54.919+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:29:55.238 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:54 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:54.833+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.861277+0000 front 2024-02-05T08:29:27.861399+0000 (oldest deadline 2024-02-05T08:29:51.361264+0000) 2024-02-05T08:29:55.526 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:55 smithi134 podman[241936]: 2024-02-05 08:29:55.439014875 +0000 UTC m=+0.633472478 container died 9258f59bcf5451676531e8d6e45fcbc6ff1c641c6f1f0718610af4b41ccecace (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-05T08:29:55.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:29:55 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:29:55.238+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:29:55.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:29:55 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:29:55.271+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:29:55.973 INFO:journalctl@ceph.osd.3.smithi134.stdout:Feb 05 08:29:55 smithi134 podman[241983]: 2024-02-05 08:29:55.956632896 +0000 UTC m=+0.512513373 container remove 9258f59bcf5451676531e8d6e45fcbc6ff1c641c6f1f0718610af4b41ccecace (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-05T08:29:55.978 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.3.service' 2024-02-05T08:29:56.018 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:29:56.019 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-05T08:29:56.019 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-05T08:29:56.019 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.4 2024-02-05T08:29:56.233 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:29:55 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:29:55.901+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:29:56.233 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:55 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:55.854+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.861277+0000 front 2024-02-05T08:29:27.861399+0000 (oldest deadline 2024-02-05T08:29:51.361264+0000) 2024-02-05T08:29:56.233 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:56 smithi196 systemd[1]: Stopping Ceph osd.4 for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:29:56.541 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:29:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:29:56.232+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:29:56.542 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:29:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:29:56.235+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:29:56.542 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:56.420+0000 7efd220d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:29:56.542 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:56.420+0000 7efd220d0700 -1 osd.4 469 *** Got signal Terminated *** 2024-02-05T08:29:56.542 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:56.420+0000 7efd220d0700 -1 osd.4 469 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T08:29:56.814 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:29:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:29:56.541Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.134:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.134:8765: connect: connection refused" 2024-02-05T08:29:56.814 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:29:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:29:56.543Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.134:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.134:8765: connect: connection refused" 2024-02-05T08:29:56.814 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:29:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:29:56.543Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.134:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.134:8765: connect: connection refused" 2024-02-05T08:29:56.814 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:29:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:29:56.545Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.134:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.134:8765: connect: connection refused" 2024-02-05T08:29:57.210 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:56.884+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.861277+0000 front 2024-02-05T08:29:27.861399+0000 (oldest deadline 2024-02-05T08:29:51.361264+0000) 2024-02-05T08:29:57.210 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:29:56 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:29:56.889+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:29:57.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:29:57 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:29:57.256+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:29:57.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:29:57 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:29:57.210+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:29:58.168 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:29:57 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:29:57.869+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:29:58.168 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:57 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:57.873+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.861277+0000 front 2024-02-05T08:29:27.861399+0000 (oldest deadline 2024-02-05T08:29:51.361264+0000) 2024-02-05T08:29:58.168 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:57 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:57.873+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:33.662407+0000 front 2024-02-05T08:29:33.662345+0000 (oldest deadline 2024-02-05T08:29:57.762270+0000) 2024-02-05T08:29:58.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:29:58 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:29:58.279+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:29:58.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:29:58 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:29:58.167+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:29:58.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:29:58 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:29:58.167+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:29:59.198 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:29:58 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:29:58.915+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:29:59.198 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:58 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:58.909+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.861277+0000 front 2024-02-05T08:29:27.861399+0000 (oldest deadline 2024-02-05T08:29:51.361264+0000) 2024-02-05T08:29:59.198 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:58 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:58.909+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:33.662407+0000 front 2024-02-05T08:29:33.662345+0000 (oldest deadline 2024-02-05T08:29:57.762270+0000) 2024-02-05T08:29:59.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:29:59 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:29:59.229+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:29:59.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:29:59 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:29:59.197+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:29:59.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:29:59 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:29:59.197+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:00.231 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:29:59 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:29:59.917+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:00.231 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:59 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:59.923+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.861277+0000 front 2024-02-05T08:29:27.861399+0000 (oldest deadline 2024-02-05T08:29:51.361264+0000) 2024-02-05T08:30:00.231 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:29:59 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:29:59.923+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:33.662407+0000 front 2024-02-05T08:29:33.662345+0000 (oldest deadline 2024-02-05T08:29:57.762270+0000) 2024-02-05T08:30:00.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:00 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:00.253+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:30:00.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:00 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:00.253+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.914588+0000 front 2024-02-05T08:29:36.914762+0000 (oldest deadline 2024-02-05T08:29:59.814545+0000) 2024-02-05T08:30:00.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:00 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:00.231+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:00.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:00 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:00.231+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:01.263 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:30:00 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:30:00.912+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.861277+0000 front 2024-02-05T08:29:27.861399+0000 (oldest deadline 2024-02-05T08:29:51.361264+0000) 2024-02-05T08:30:01.263 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:30:00 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4[38587]: 2024-02-05T08:30:00.912+0000 7efd1e6cb700 -1 osd.4 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:33.662407+0000 front 2024-02-05T08:29:33.662345+0000 (oldest deadline 2024-02-05T08:29:57.762270+0000) 2024-02-05T08:30:01.263 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:00 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:00.930+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:01.263 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:00 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:00.930+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:01.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:01 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:01.263+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:30:01.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:01 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:01.263+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.914588+0000 front 2024-02-05T08:29:36.914762+0000 (oldest deadline 2024-02-05T08:29:59.814545+0000) 2024-02-05T08:30:01.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:01 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:01.266+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:01.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:01 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:01.266+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:02.243 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:01 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:01.897+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:02.243 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:01 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:01.897+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:02.244 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:30:01 smithi196 podman[168850]: 2024-02-05 08:30:01.956098908 +0000 UTC m=+5.657999067 container died f79fcd5ad49c6d1408528d5e19fe30bae2aa47e0a84fad3a17e8b727c618c1ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T08:30:02.564 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:30:02 smithi196 podman[168850]: 2024-02-05 08:30:02.342548364 +0000 UTC m=+6.044448527 container cleanup f79fcd5ad49c6d1408528d5e19fe30bae2aa47e0a84fad3a17e8b727c618c1ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131) 2024-02-05T08:30:02.564 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:30:02 smithi196 bash[168850]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4 2024-02-05T08:30:02.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:02 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:02.288+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:02.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:02 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:02.289+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:02.565 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:02 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:02.306+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:30:02.565 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:02 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:02.306+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.914588+0000 front 2024-02-05T08:29:36.914762+0000 (oldest deadline 2024-02-05T08:29:59.814545+0000) 2024-02-05T08:30:02.870 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:30:02 smithi196 podman[168873]: 2024-02-05 08:30:02.581641021 +0000 UTC m=+0.627304481 container remove f79fcd5ad49c6d1408528d5e19fe30bae2aa47e0a84fad3a17e8b727c618c1ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-05T08:30:03.205 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:02 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:02.870+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:03.205 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:02 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:02.870+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:03.205 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:30:02 smithi196 podman[168972]: 2024-02-05 08:30:02.793904047 +0000 UTC m=+0.017942463 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:30:03.205 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:30:02 smithi196 podman[168972]: 2024-02-05 08:30:02.894804341 +0000 UTC m=+0.118842751 container create c7be9b4016dd2ee9873af375b88be5d7b8a320513c3090146b2aa8d24830a08f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T08:30:03.205 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:30:03 smithi196 podman[168972]: 2024-02-05 08:30:03.088877701 +0000 UTC m=+0.312916116 container init c7be9b4016dd2ee9873af375b88be5d7b8a320513c3090146b2aa8d24830a08f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2) 2024-02-05T08:30:03.205 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:30:03 smithi196 podman[168972]: 2024-02-05 08:30:03.094552728 +0000 UTC m=+0.318591143 container start c7be9b4016dd2ee9873af375b88be5d7b8a320513c3090146b2aa8d24830a08f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD) 2024-02-05T08:30:03.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:03.312+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:30:03.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:03.312+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.914588+0000 front 2024-02-05T08:29:36.914762+0000 (oldest deadline 2024-02-05T08:29:59.814545+0000) 2024-02-05T08:30:03.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:03.323+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:03.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:03.323+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:03.565 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:30:03 smithi196 podman[168972]: 2024-02-05 08:30:03.205289106 +0000 UTC m=+0.429327523 container attach c7be9b4016dd2ee9873af375b88be5d7b8a320513c3090146b2aa8d24830a08f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T08:30:03.966 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:03.834+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:03.966 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:03 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:03.834+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:03.966 INFO:journalctl@ceph.osd.4.smithi196.stdout:Feb 05 08:30:03 smithi196 podman[168972]: 2024-02-05 08:30:03.672522123 +0000 UTC m=+0.896560535 container died c7be9b4016dd2ee9873af375b88be5d7b8a320513c3090146b2aa8d24830a08f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-4-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.build-date=20240131) 2024-02-05T08:30:04.303 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.4.service' 2024-02-05T08:30:04.349 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:30:04.349 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-05T08:30:04.349 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-05T08:30:04.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.5 2024-02-05T08:30:04.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:04.354+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:04.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:04.354+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:04.565 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:04.339+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:30:04.565 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:04.339+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.914588+0000 front 2024-02-05T08:29:36.914762+0000 (oldest deadline 2024-02-05T08:29:59.814545+0000) 2024-02-05T08:30:04.565 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:04 smithi196 systemd[1]: Stopping Ceph osd.5 for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:30:05.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:04.841+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:05.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:04.841+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:05.064 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:04.648+0000 7f84d4090700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:30:05.065 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:04.648+0000 7f84d4090700 -1 osd.5 469 *** Got signal Terminated *** 2024-02-05T08:30:05.065 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:04 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:04.648+0000 7f84d4090700 -1 osd.5 469 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T08:30:05.814 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:05 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:05.337+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:30:05.814 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:05 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:05.338+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.914588+0000 front 2024-02-05T08:29:36.914762+0000 (oldest deadline 2024-02-05T08:29:59.814545+0000) 2024-02-05T08:30:05.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:05 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:05.386+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:05.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:05 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:05.386+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:06.314 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:05 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:05.847+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:06.314 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:05 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:05.847+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:06.814 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:06 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:06.326+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:30:06.814 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:06 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:06.326+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.914588+0000 front 2024-02-05T08:29:36.914762+0000 (oldest deadline 2024-02-05T08:29:59.814545+0000) 2024-02-05T08:30:06.815 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:06 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:06.421+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:06.815 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:06 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:06.421+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:06.815 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:06 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:06.421+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:07.314 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:06 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:06.814+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:07.314 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:06 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:06.815+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:07.792 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:07 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:07.318+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:30:07.792 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:07 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:07.318+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.914588+0000 front 2024-02-05T08:29:36.914762+0000 (oldest deadline 2024-02-05T08:29:59.814545+0000) 2024-02-05T08:30:07.793 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:07 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:07.419+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:07.793 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:07 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:07.419+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:07.793 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:07 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:07.419+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:08.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:07 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:07.792+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:08.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:07 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:07.792+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:08.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:07 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:07.792+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:43.231265+0000 front 2024-02-05T08:29:43.231281+0000 (oldest deadline 2024-02-05T08:30:07.331228+0000) 2024-02-05T08:30:08.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:08 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:08.306+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:30:08.564 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:08 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:08.306+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.914588+0000 front 2024-02-05T08:29:36.914762+0000 (oldest deadline 2024-02-05T08:29:59.814545+0000) 2024-02-05T08:30:08.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:08 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:08.403+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:08.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:08 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:08.403+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:08.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:08 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:08.403+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:09.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:08 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:08.771+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:09.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:08 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:08.771+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:09.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:08 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:08.771+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:43.231265+0000 front 2024-02-05T08:29:43.231281+0000 (oldest deadline 2024-02-05T08:30:07.331228+0000) 2024-02-05T08:30:09.811 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:09 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:09.347+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:25.113832+0000 front 2024-02-05T08:29:25.113688+0000 (oldest deadline 2024-02-05T08:29:51.013651+0000) 2024-02-05T08:30:09.811 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:09 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:09.347+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.914588+0000 front 2024-02-05T08:29:36.914762+0000 (oldest deadline 2024-02-05T08:29:59.814545+0000) 2024-02-05T08:30:09.811 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:09 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5[44900]: 2024-02-05T08:30:09.347+0000 7f84cc683700 -1 osd.5 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:43.816206+0000 front 2024-02-05T08:29:43.816161+0000 (oldest deadline 2024-02-05T08:30:09.116043+0000) 2024-02-05T08:30:09.812 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:09 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:09.387+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:09.812 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:09 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:09.387+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:09.812 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:09 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:09.387+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:10.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:09 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:09.810+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:10.065 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:09 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:09.810+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:10.065 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:09 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:09.810+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:43.231265+0000 front 2024-02-05T08:29:43.231281+0000 (oldest deadline 2024-02-05T08:30:07.331228+0000) 2024-02-05T08:30:10.390 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:10 smithi196 podman[169102]: 2024-02-05 08:30:10.134483483 +0000 UTC m=+5.607518630 container died 43d47d9822650782ba4b1cc6885b5e81be00f6a41f3ef69b4779de49ba98a8f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-02-05T08:30:10.729 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:10 smithi196 podman[169102]: 2024-02-05 08:30:10.53654163 +0000 UTC m=+6.009576779 container cleanup 43d47d9822650782ba4b1cc6885b5e81be00f6a41f3ef69b4779de49ba98a8f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, ceph=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-05T08:30:10.729 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:10 smithi196 bash[169102]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5 2024-02-05T08:30:10.729 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:10 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:10.389+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:10.730 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:10 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:10.389+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:10.730 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:10 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:10.389+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:11.064 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:10 smithi196 podman[169124]: 2024-02-05 08:30:10.72936218 +0000 UTC m=+0.595353976 container remove 43d47d9822650782ba4b1cc6885b5e81be00f6a41f3ef69b4779de49ba98a8f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T08:30:11.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:10 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:10.824+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:11.065 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:10 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:10.824+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:11.065 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:10 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:10.824+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:43.231265+0000 front 2024-02-05T08:29:43.231281+0000 (oldest deadline 2024-02-05T08:30:07.331228+0000) 2024-02-05T08:30:11.390 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:11 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:11.370+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:11.390 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:11 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:11.370+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:11.390 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:11 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:11.370+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:11.391 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:11 smithi196 podman[169223]: 2024-02-05 08:30:11.041034035 +0000 UTC m=+0.018427851 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:30:11.391 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:11 smithi196 podman[169223]: 2024-02-05 08:30:11.180584649 +0000 UTC m=+0.157978466 container create 7fc25b559f7995f97935b30de0d53dba818b39b301184669bf270313e6f1fb4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-02-05T08:30:11.391 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:11 smithi196 podman[169223]: 2024-02-05 08:30:11.384633758 +0000 UTC m=+0.362027578 container init 7fc25b559f7995f97935b30de0d53dba818b39b301184669bf270313e6f1fb4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-05T08:30:11.391 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:11 smithi196 podman[169223]: 2024-02-05 08:30:11.390472259 +0000 UTC m=+0.367866076 container start 7fc25b559f7995f97935b30de0d53dba818b39b301184669bf270313e6f1fb4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T08:30:11.670 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:11 smithi196 podman[169223]: 2024-02-05 08:30:11.4911135 +0000 UTC m=+0.468507336 container attach 7fc25b559f7995f97935b30de0d53dba818b39b301184669bf270313e6f1fb4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T08:30:11.977 INFO:journalctl@ceph.osd.5.smithi196.stdout:Feb 05 08:30:11 smithi196 podman[169223]: 2024-02-05 08:30:11.671091471 +0000 UTC m=+0.648485298 container died 7fc25b559f7995f97935b30de0d53dba818b39b301184669bf270313e6f1fb4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-05T08:30:11.977 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:11 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:11.872+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:11.977 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:11 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:11.872+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:11.977 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:11 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:11.872+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:43.231265+0000 front 2024-02-05T08:29:43.231281+0000 (oldest deadline 2024-02-05T08:30:07.331228+0000) 2024-02-05T08:30:12.308 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.5.service' 2024-02-05T08:30:12.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:30:12.351 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-05T08:30:12.351 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-05T08:30:12.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.6 2024-02-05T08:30:12.564 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:12 smithi196 systemd[1]: Stopping Ceph osd.6 for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:30:12.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:12 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:12.408+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:12.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:12 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:12.408+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:12.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:12 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:12.408+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:12.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:12 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:12.408+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:13.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:12 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:12.690+0000 7fd818302700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:30:13.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:12 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:12.690+0000 7fd818302700 -1 osd.6 469 *** Got signal Terminated *** 2024-02-05T08:30:13.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:12 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:12.690+0000 7fd818302700 -1 osd.6 469 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T08:30:13.064 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:12 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:12.889+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:13.065 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:12 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:12.889+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:13.065 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:12 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:12.889+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:43.231265+0000 front 2024-02-05T08:29:43.231281+0000 (oldest deadline 2024-02-05T08:30:07.331228+0000) 2024-02-05T08:30:13.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:13.380+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:13.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:13.380+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:13.815 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:13.380+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:13.815 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:13.380+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:14.314 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:13.911+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:14.315 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:13.911+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:14.315 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:13 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:13.911+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:43.231265+0000 front 2024-02-05T08:29:43.231281+0000 (oldest deadline 2024-02-05T08:30:07.331228+0000) 2024-02-05T08:30:14.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:14.351+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:14.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:14.351+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:14.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:14.351+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:14.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:14.351+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:15.314 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:14.902+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:15.314 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:14.902+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:15.314 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:14 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:14.902+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:43.231265+0000 front 2024-02-05T08:29:43.231281+0000 (oldest deadline 2024-02-05T08:30:07.331228+0000) 2024-02-05T08:30:15.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:15 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:15.325+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:15.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:15 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:15.325+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:15.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:15 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:15.325+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:15.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:15 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:15.325+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:16.293 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:15 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:15.909+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:16.293 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:15 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:15.909+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:16.293 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:15 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:15.909+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:43.231265+0000 front 2024-02-05T08:29:43.231281+0000 (oldest deadline 2024-02-05T08:30:07.331228+0000) 2024-02-05T08:30:16.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:16.292+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:16.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:16.292+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:16.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:16.292+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:16.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:16.292+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6830 osd.3 since back 2024-02-05T08:29:50.902024+0000 front 2024-02-05T08:29:50.902107+0000 (oldest deadline 2024-02-05T08:30:16.201922+0000) 2024-02-05T08:30:16.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:16.292+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:17.314 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:16.910+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.627845+0000 front 2024-02-05T08:29:27.627990+0000 (oldest deadline 2024-02-05T08:29:50.527910+0000) 2024-02-05T08:30:17.314 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:16.910+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:34.430131+0000 front 2024-02-05T08:29:34.430041+0000 (oldest deadline 2024-02-05T08:30:00.330028+0000) 2024-02-05T08:30:17.314 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:16 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6[51129]: 2024-02-05T08:30:16.910+0000 7fd8148fd700 -1 osd.6 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:43.231265+0000 front 2024-02-05T08:29:43.231281+0000 (oldest deadline 2024-02-05T08:30:07.331228+0000) 2024-02-05T08:30:17.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:17 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:17.325+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:17.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:17 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:17.325+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:17.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:17 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:17.325+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:17.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:17 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:17.325+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6830 osd.3 since back 2024-02-05T08:29:50.902024+0000 front 2024-02-05T08:29:50.902107+0000 (oldest deadline 2024-02-05T08:30:16.201922+0000) 2024-02-05T08:30:17.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:17 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:17.325+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:18.564 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:18 smithi196 podman[169352]: 2024-02-05 08:30:18.292853266 +0000 UTC m=+5.716593998 container died c107877ad73619988b039bb51b2ee23ce6a9c8137f8067a34308fb7c0d13e15d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T08:30:18.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:18 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:18.363+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:18.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:18 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:18.363+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:18.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:18 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:18.363+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:18.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:18 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:18.363+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6830 osd.3 since back 2024-02-05T08:29:50.902024+0000 front 2024-02-05T08:29:50.902107+0000 (oldest deadline 2024-02-05T08:30:16.201922+0000) 2024-02-05T08:30:18.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:18 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:18.363+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:18.814 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:18 smithi196 podman[169352]: 2024-02-05 08:30:18.662900714 +0000 UTC m=+6.086641446 container cleanup c107877ad73619988b039bb51b2ee23ce6a9c8137f8067a34308fb7c0d13e15d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-05T08:30:18.814 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:18 smithi196 bash[169352]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6 2024-02-05T08:30:19.279 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:18 smithi196 podman[169374]: 2024-02-05 08:30:18.872945937 +0000 UTC m=+0.577640753 container remove c107877ad73619988b039bb51b2ee23ce6a9c8137f8067a34308fb7c0d13e15d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-05T08:30:19.531 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:19 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:19.382+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:19.532 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:19 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:19.382+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:19.532 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:19 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:19.382+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:19.532 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:19 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:19.382+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6830 osd.3 since back 2024-02-05T08:29:50.902024+0000 front 2024-02-05T08:29:50.902107+0000 (oldest deadline 2024-02-05T08:30:16.201922+0000) 2024-02-05T08:30:19.532 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:19 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:19.382+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:19.532 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:19 smithi196 podman[169471]: 2024-02-05 08:30:19.183150254 +0000 UTC m=+0.018634343 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:30:19.532 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:19 smithi196 podman[169471]: 2024-02-05 08:30:19.298554423 +0000 UTC m=+0.134038515 container create a0a82d6736a223013dab540fd53a818fca83ca0897ba8795cd71ce99bbec8485 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6-deactivate, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131) 2024-02-05T08:30:19.532 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:19 smithi196 podman[169471]: 2024-02-05 08:30:19.526042602 +0000 UTC m=+0.361526691 container init a0a82d6736a223013dab540fd53a818fca83ca0897ba8795cd71ce99bbec8485 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, ceph=True) 2024-02-05T08:30:19.532 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:19 smithi196 podman[169471]: 2024-02-05 08:30:19.53176194 +0000 UTC m=+0.367246034 container start a0a82d6736a223013dab540fd53a818fca83ca0897ba8795cd71ce99bbec8485 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-05T08:30:19.814 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:19 smithi196 podman[169471]: 2024-02-05 08:30:19.609370051 +0000 UTC m=+0.444854142 container attach a0a82d6736a223013dab540fd53a818fca83ca0897ba8795cd71ce99bbec8485 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-05T08:30:20.121 INFO:journalctl@ceph.osd.6.smithi196.stdout:Feb 05 08:30:19 smithi196 podman[169471]: 2024-02-05 08:30:19.817013743 +0000 UTC m=+0.652497836 container died a0a82d6736a223013dab540fd53a818fca83ca0897ba8795cd71ce99bbec8485 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-6-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2) 2024-02-05T08:30:20.379 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:20 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:20.346+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:20.379 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:20 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:20.346+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:20.379 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:20 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:20.346+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:20.379 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:20 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:20.346+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6830 osd.3 since back 2024-02-05T08:29:50.902024+0000 front 2024-02-05T08:29:50.902107+0000 (oldest deadline 2024-02-05T08:30:16.201922+0000) 2024-02-05T08:30:20.379 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:20 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:20.346+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:20.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.6.service' 2024-02-05T08:30:20.438 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:30:20.438 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-05T08:30:20.438 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-05T08:30:20.438 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.7 2024-02-05T08:30:20.776 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:20 smithi196 systemd[1]: Stopping Ceph osd.7 for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:30:21.064 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:20 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:20.775+0000 7f5ceebc3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:30:21.064 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:20 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:20.775+0000 7f5ceebc3700 -1 osd.7 469 *** Got signal Terminated *** 2024-02-05T08:30:21.064 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:20 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:20.775+0000 7f5ceebc3700 -1 osd.7 469 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T08:30:21.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:21 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:21.366+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:21.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:21 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:21.366+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:21.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:21 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:21.366+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:21.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:21 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:21.366+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6830 osd.3 since back 2024-02-05T08:29:50.902024+0000 front 2024-02-05T08:29:50.902107+0000 (oldest deadline 2024-02-05T08:30:16.201922+0000) 2024-02-05T08:30:21.814 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:21 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:21.367+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:22.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:22 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:22.323+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:22.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:22 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:22.323+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:22.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:22 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:22.323+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:22.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:22 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:22.323+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6830 osd.3 since back 2024-02-05T08:29:50.902024+0000 front 2024-02-05T08:29:50.902107+0000 (oldest deadline 2024-02-05T08:30:16.201922+0000) 2024-02-05T08:30:22.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:22 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:22.323+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:23.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:23.287+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:23.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:23.287+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:23.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:23.287+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:23.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:23.287+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6830 osd.3 since back 2024-02-05T08:29:50.902024+0000 front 2024-02-05T08:29:50.902107+0000 (oldest deadline 2024-02-05T08:30:16.201922+0000) 2024-02-05T08:30:23.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:23 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:23.287+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:24.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:24.250+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:24.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:24.250+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:24.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:24.250+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:24.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:24.250+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6830 osd.3 since back 2024-02-05T08:29:50.902024+0000 front 2024-02-05T08:29:50.902107+0000 (oldest deadline 2024-02-05T08:30:16.201922+0000) 2024-02-05T08:30:24.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:24.250+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-02-05T08:29:58.303699+0000 front 2024-02-05T08:29:58.303638+0000 (oldest deadline 2024-02-05T08:30:24.104230+0000) 2024-02-05T08:30:24.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:24 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:24.250+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:25.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:25 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:25.238+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6806 osd.0 since back 2024-02-05T08:29:27.297634+0000 front 2024-02-05T08:29:27.297751+0000 (oldest deadline 2024-02-05T08:29:51.997393+0000) 2024-02-05T08:30:25.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:25 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:25.238+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6814 osd.1 since back 2024-02-05T08:29:36.098346+0000 front 2024-02-05T08:29:36.098433+0000 (oldest deadline 2024-02-05T08:29:57.798285+0000) 2024-02-05T08:30:25.564 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:25 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:25.238+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6822 osd.2 since back 2024-02-05T08:29:44.000440+0000 front 2024-02-05T08:29:44.000493+0000 (oldest deadline 2024-02-05T08:30:05.700430+0000) 2024-02-05T08:30:25.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:25 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:25.238+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.134:6830 osd.3 since back 2024-02-05T08:29:50.902024+0000 front 2024-02-05T08:29:50.902107+0000 (oldest deadline 2024-02-05T08:30:16.201922+0000) 2024-02-05T08:30:25.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:25 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:25.238+0000 7f5ce71b6700 -1 osd.7 469 heartbeat_check: no reply from 172.21.15.196:6804 osd.4 since back 2024-02-05T08:29:58.303699+0000 front 2024-02-05T08:29:58.303638+0000 (oldest deadline 2024-02-05T08:30:24.104230+0000) 2024-02-05T08:30:25.565 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:25 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7[57343]: 2024-02-05T08:30:25.238+0000 7f5ce71b6700 -1 osd.7 469 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24388.0:1481 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e469) 2024-02-05T08:30:26.688 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:26 smithi196 podman[169601]: 2024-02-05 08:30:26.367522234 +0000 UTC m=+5.708227435 container died 4eef3f9b3253fb6e1e2b607029ebf18cfb5283e0498814eb43c524695e43baa3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T08:30:27.006 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:26 smithi196 podman[169601]: 2024-02-05 08:30:26.781179897 +0000 UTC m=+6.121885098 container cleanup 4eef3f9b3253fb6e1e2b607029ebf18cfb5283e0498814eb43c524695e43baa3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD) 2024-02-05T08:30:27.007 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:26 smithi196 bash[169601]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7 2024-02-05T08:30:27.314 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:27 smithi196 podman[169635]: 2024-02-05 08:30:27.007206732 +0000 UTC m=+0.636142432 container remove 4eef3f9b3253fb6e1e2b607029ebf18cfb5283e0498814eb43c524695e43baa3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-05T08:30:27.751 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:27 smithi196 podman[169729]: 2024-02-05 08:30:27.341895527 +0000 UTC m=+0.017712133 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T08:30:27.751 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:27 smithi196 podman[169729]: 2024-02-05 08:30:27.458090823 +0000 UTC m=+0.133907424 container create 68dba9ed30e3b2b1a7204e1f354c688d7257e7af3169ab49e7bc686787fec750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-05T08:30:27.751 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:27 smithi196 podman[169729]: 2024-02-05 08:30:27.660496129 +0000 UTC m=+0.336312740 container init 68dba9ed30e3b2b1a7204e1f354c688d7257e7af3169ab49e7bc686787fec750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T08:30:27.752 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:27 smithi196 podman[169729]: 2024-02-05 08:30:27.666568074 +0000 UTC m=+0.342384678 container start 68dba9ed30e3b2b1a7204e1f354c688d7257e7af3169ab49e7bc686787fec750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-05T08:30:27.752 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:27 smithi196 podman[169729]: 2024-02-05 08:30:27.751743791 +0000 UTC m=+0.427560405 container attach 68dba9ed30e3b2b1a7204e1f354c688d7257e7af3169ab49e7bc686787fec750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=) 2024-02-05T08:30:28.064 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:27 smithi196 podman[169729]: 2024-02-05 08:30:27.950941594 +0000 UTC m=+0.626758198 container died 68dba9ed30e3b2b1a7204e1f354c688d7257e7af3169ab49e7bc686787fec750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T08:30:28.532 INFO:journalctl@ceph.osd.7.smithi196.stdout:Feb 05 08:30:28 smithi196 podman[169775]: 2024-02-05 08:30:28.529907611 +0000 UTC m=+0.573990449 container remove 68dba9ed30e3b2b1a7204e1f354c688d7257e7af3169ab49e7bc686787fec750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-05T08:30:28.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@osd.7.service' 2024-02-05T08:30:28.586 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:30:28.587 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-05T08:30:28.587 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-05T08:30:28.587 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@rgw.foo.a 2024-02-05T08:30:28.891 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:30:28 smithi134 systemd[1]: Stopping Ceph rgw.foo.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:30:29.277 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:30:28 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-rgw-foo-a[68295]: 2024-02-05T08:30:28.891+0000 7f52748c0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T08:30:29.277 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:30:28 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-rgw-foo-a[68295]: 2024-02-05T08:30:28.891+0000 7f528312bac0 -1 shutting down 2024-02-05T08:30:39.179 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:30:38 smithi134 bash[242070]: time="2024-02-05T08:30:38Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-05T08:30:39.180 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:30:38 smithi134 podman[242070]: 2024-02-05 08:30:38.915521666 +0000 UTC m=+10.129599463 container died 784ffda1aee4a1abe76ad3e7e2b0e689e88b540e70ce18ffeb18b273a2c10ef9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-05T08:30:39.526 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:30:39 smithi134 podman[242070]: 2024-02-05 08:30:39.378626828 +0000 UTC m=+10.592704618 container cleanup 784ffda1aee4a1abe76ad3e7e2b0e689e88b540e70ce18ffeb18b273a2c10ef9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T08:30:39.527 INFO:journalctl@ceph.rgw.foo.a.smithi134.stdout:Feb 05 08:30:39 smithi134 bash[242070]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-rgw-foo-a 2024-02-05T08:30:39.793 DEBUG:teuthology.orchestra.run.smithi134:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@rgw.foo.a.service' 2024-02-05T08:30:39.830 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:30:39.830 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-05T08:30:39.831 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-05T08:30:39.831 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@prometheus.a 2024-02-05T08:30:40.158 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:39 smithi196 systemd[1]: Stopping Ceph prometheus.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:30:40.431 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:30:40.158Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-05T08:30:40.432 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:30:40.159Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-05T08:30:40.432 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:30:40.159Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-05T08:30:40.432 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:30:40.159Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-05T08:30:40.432 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:30:40.159Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-05T08:30:40.432 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:30:40.159Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-05T08:30:40.432 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:30:40.159Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-05T08:30:40.432 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:30:40.159Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-05T08:30:40.432 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:30:40.174Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-05T08:30:40.432 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:30:40.175Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-05T08:30:40.432 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:30:40.175Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-05T08:30:40.432 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a[73637]: ts=2024-02-05T08:30:40.175Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-05T08:30:40.433 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 podman[169862]: 2024-02-05 08:30:40.179740632 +0000 UTC m=+0.125112377 container died 07dc4e8cf9965ae5d21a0121af7cd6cd0379b9142a06f34106fb51913f97a6a1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T08:30:40.814 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 podman[169862]: 2024-02-05 08:30:40.622652181 +0000 UTC m=+0.568023971 container cleanup 07dc4e8cf9965ae5d21a0121af7cd6cd0379b9142a06f34106fb51913f97a6a1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T08:30:40.814 INFO:journalctl@ceph.prometheus.a.smithi196.stdout:Feb 05 08:30:40 smithi196 bash[169862]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-prometheus-a 2024-02-05T08:30:41.030 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@prometheus.a.service' 2024-02-05T08:30:41.066 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T08:30:41.066 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-05T08:30:41.066 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 --force --keep-logs 2024-02-05T08:30:41.257 INFO:teuthology.orchestra.run.smithi134.stdout:Deleting cluster with fsid: 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T08:30:43.711 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:30:43 smithi134 systemd[1]: Stopping Ceph alertmanager.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:30:44.026 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:30:43 smithi134 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a[89215]: ts=2024-02-05T08:30:43.711Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-05T08:30:44.027 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:30:43 smithi134 podman[242578]: 2024-02-05 08:30:43.752068348 +0000 UTC m=+0.153175213 container died 23561823c3c74d573473e7ac7c93e49aa2be690c18448be8ced26fde644935fc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T08:30:44.484 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:30:44 smithi134 podman[242578]: 2024-02-05 08:30:44.216794193 +0000 UTC m=+0.617901060 container cleanup 23561823c3c74d573473e7ac7c93e49aa2be690c18448be8ced26fde644935fc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T08:30:44.484 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:30:44 smithi134 bash[242578]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a 2024-02-05T08:30:44.777 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:30:44 smithi134 podman[242597]: 2024-02-05 08:30:44.48460649 +0000 UTC m=+0.729526187 container remove 23561823c3c74d573473e7ac7c93e49aa2be690c18448be8ced26fde644935fc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T08:30:44.777 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:30:44 smithi134 podman[242597]: 2024-02-05 08:30:44.560643709 +0000 UTC m=+0.805563406 volume remove c6d40b6d621dec42873ee7e6c937c3e19a943f4770b141a709174cc33b64e672 2024-02-05T08:30:44.777 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:30:44 smithi134 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-05T08:30:44.777 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:30:44 smithi134 systemd[1]: Stopped Ceph alertmanager.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:30:44.777 INFO:journalctl@ceph.alertmanager.a.smithi134.stdout:Feb 05 08:30:44 smithi134 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@alertmanager.a.service: Consumed 1.881s CPU time. 2024-02-05T08:30:45.142 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:30:44 smithi134 systemd[1]: Stopping Ceph node-exporter.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:30:45.455 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:30:45 smithi134 podman[242714]: 2024-02-05 08:30:45.142612853 +0000 UTC m=+0.115913947 container died 419ba33757dc8c37bd99ba56768944cca2f553b47fbbc91980fc388667072997 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T08:30:45.776 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:30:45 smithi134 podman[242714]: 2024-02-05 08:30:45.612570262 +0000 UTC m=+0.585871352 container cleanup 419ba33757dc8c37bd99ba56768944cca2f553b47fbbc91980fc388667072997 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T08:30:45.776 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:30:45 smithi134 bash[242714]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a 2024-02-05T08:30:46.157 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:30:45 smithi134 podman[242734]: 2024-02-05 08:30:45.905588209 +0000 UTC m=+0.763547878 container remove 419ba33757dc8c37bd99ba56768944cca2f553b47fbbc91980fc388667072997 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T08:30:46.157 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:30:45 smithi134 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-05T08:30:46.157 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:30:46 smithi134 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-05T08:30:46.157 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:30:46 smithi134 systemd[1]: Stopped Ceph node-exporter.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:30:46.157 INFO:journalctl@ceph.node-exporter.a.smithi134.stdout:Feb 05 08:30:46 smithi134 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@node-exporter.a.service: Consumed 4.810s CPU time. 2024-02-05T08:30:46.716 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 --force --keep-logs 2024-02-05T08:30:46.905 INFO:teuthology.orchestra.run.smithi196.stdout:Deleting cluster with fsid: 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T08:30:49.046 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:30:48 smithi196 systemd[1]: Stopping Ceph iscsi.iscsi.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:30:49.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:30:49 smithi196 ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a[62740]: debug Shutdown received 2024-02-05T08:30:59.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:30:59 smithi196 bash[170283]: time="2024-02-05T08:30:59Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-02-05T08:30:59.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:30:59 smithi196 podman[170283]: 2024-02-05 08:30:59.071995589 +0000 UTC m=+10.134636459 container died 3bf96afe72ea5e07c571425e06d627fec64ed7a70b5b13187139170e6e17306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=) 2024-02-05T08:30:59.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:30:59 smithi196 podman[170283]: 2024-02-05 08:30:59.457987727 +0000 UTC m=+10.520628596 container cleanup 3bf96afe72ea5e07c571425e06d627fec64ed7a70b5b13187139170e6e17306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-05T08:30:59.726 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:30:59 smithi196 bash[170283]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a 2024-02-05T08:31:00.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:30:59 smithi196 podman[170308]: 2024-02-05 08:30:59.725810283 +0000 UTC m=+0.654830859 container remove 3bf96afe72ea5e07c571425e06d627fec64ed7a70b5b13187139170e6e17306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-iscsi-iscsi-a, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-05T08:31:00.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:30:59 smithi196 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-02-05T08:31:00.065 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:30:59 smithi196 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-02-05T08:31:00.065 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:30:59 smithi196 systemd[1]: Stopped Ceph iscsi.iscsi.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:31:00.065 INFO:journalctl@ceph.iscsi.iscsi.a.smithi196.stdout:Feb 05 08:30:59 smithi196 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@iscsi.iscsi.a.service: Consumed 3.559s CPU time. 2024-02-05T08:31:01.567 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:31:01 smithi196 systemd[1]: Stopping Ceph grafana.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:31:01.950 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:31:01 smithi196 podman[170596]: 2024-02-05 08:31:01.67997031 +0000 UTC m=+0.110480092 container died d8ab63d57982625ccf5a94ae5003cf52815b91117fe62b7feaff9058531a98b9 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., architecture=x86_64, distribution-scope=public, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, name=ubi8, vcs-type=git) 2024-02-05T08:31:02.314 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:31:02 smithi196 podman[170596]: 2024-02-05 08:31:02.10761386 +0000 UTC m=+0.538123638 container cleanup d8ab63d57982625ccf5a94ae5003cf52815b91117fe62b7feaff9058531a98b9 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, release=239.1651231664, io.openshift.tags=base rhel8, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , description=Ceph Grafana Container, io.buildah.version=1.32.0, version=8.5, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, name=ubi8) 2024-02-05T08:31:02.314 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:31:02 smithi196 bash[170596]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a 2024-02-05T08:31:02.768 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:31:02 smithi196 podman[170616]: 2024-02-05 08:31:02.333777477 +0000 UTC m=+0.647137022 container remove d8ab63d57982625ccf5a94ae5003cf52815b91117fe62b7feaff9058531a98b9 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, release=239.1651231664, io.buildah.version=1.32.0, io.openshift.expose-services=, maintainer=Paul Cuzner , name=ubi8, description=Ceph Grafana Container, vcs-type=git, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, distribution-scope=public) 2024-02-05T08:31:02.768 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:31:02 smithi196 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-02-05T08:31:02.768 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:31:02 smithi196 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-05T08:31:02.768 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:31:02 smithi196 systemd[1]: Stopped Ceph grafana.a for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:31:02.768 INFO:journalctl@ceph.grafana.a.smithi196.stdout:Feb 05 08:31:02 smithi196 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@grafana.a.service: Consumed 13.303s CPU time. 2024-02-05T08:31:03.064 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:31:02 smithi196 systemd[1]: Stopping Ceph node-exporter.b for 43e13aa4-c3fc-11ee-95b6-87774f69a715... 2024-02-05T08:31:03.064 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:31:03 smithi196 podman[170736]: 2024-02-05 08:31:03.008459069 +0000 UTC m=+0.106257045 container died 779d88c4e760247af80687dc57457c29f3dab7ebc3a394ddc4c7ed1bc4b3c10f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T08:31:03.564 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:31:03 smithi196 podman[170736]: 2024-02-05 08:31:03.319776955 +0000 UTC m=+0.417574966 container cleanup 779d88c4e760247af80687dc57457c29f3dab7ebc3a394ddc4c7ed1bc4b3c10f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T08:31:03.564 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:31:03 smithi196 bash[170736]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b 2024-02-05T08:31:03.564 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:31:03 smithi196 podman[170755]: 2024-02-05 08:31:03.462210065 +0000 UTC m=+0.458283154 container remove 779d88c4e760247af80687dc57457c29f3dab7ebc3a394ddc4c7ed1bc4b3c10f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T08:31:03.564 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:31:03 smithi196 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-05T08:31:03.958 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:31:03 smithi196 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-05T08:31:03.958 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:31:03 smithi196 systemd[1]: Stopped Ceph node-exporter.b for 43e13aa4-c3fc-11ee-95b6-87774f69a715. 2024-02-05T08:31:03.958 INFO:journalctl@ceph.node-exporter.b.smithi196.stdout:Feb 05 08:31:03 smithi196 systemd[1]: ceph-43e13aa4-c3fc-11ee-95b6-87774f69a715@node-exporter.b.service: Consumed 4.565s CPU time. 2024-02-05T08:31:04.382 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-05T08:31:04.418 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-05T08:31:04.448 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-05T08:31:04.449 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546773/remote/smithi134/crash 2024-02-05T08:31:04.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/crash -- . 2024-02-05T08:31:04.514 INFO:teuthology.orchestra.run.smithi134.stderr:tar: /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/crash: Cannot open: No such file or directory 2024-02-05T08:31:04.514 INFO:teuthology.orchestra.run.smithi134.stderr:tar: Error is not recoverable: exiting now 2024-02-05T08:31:04.516 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546773/remote/smithi196/crash 2024-02-05T08:31:04.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/crash -- . 2024-02-05T08:31:04.569 INFO:teuthology.orchestra.run.smithi196.stderr:tar: /var/lib/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/crash: Cannot open: No such file or directory 2024-02-05T08:31:04.570 INFO:teuthology.orchestra.run.smithi196.stderr:tar: Error is not recoverable: exiting now 2024-02-05T08:31:04.570 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-05T08:31:04.571 DEBUG:teuthology.orchestra.run.smithi134:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-05T08:31:04.698 INFO:teuthology.orchestra.run.smithi134.stdout:2024-02-05T08:10:00.000144+0000 mon.a (mon.0) 1023 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T08:31:04.699 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-05T08:31:04.699 DEBUG:teuthology.orchestra.run.smithi134:> sudo egrep '\[SEC\]' /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-05T08:31:04.730 DEBUG:teuthology.orchestra.run.smithi134:> sudo egrep '\[ERR\]' /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-05T08:31:04.801 DEBUG:teuthology.orchestra.run.smithi134:> sudo egrep '\[WRN\]' /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-05T08:31:04.872 INFO:teuthology.orchestra.run.smithi134.stdout:2024-02-05T08:10:00.000144+0000 mon.a (mon.0) 1023 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T08:31:04.872 INFO:tasks.cephadm:Compressing logs... 2024-02-05T08:31:04.872 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-05T08:31:04.915 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-05T08:31:04.960 INFO:teuthology.orchestra.run.smithi196.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-05T08:31:04.963 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-05T08:31:04.969 INFO:teuthology.orchestra.run.smithi134.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-05T08:31:04.969 INFO:teuthology.orchestra.run.smithi134.stderr:: No such file or directory 2024-02-05T08:31:04.978 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mon.a.log 2024-02-05T08:31:04.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/objectstore_tool.6.log 2024-02-05T08:31:04.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.audit.log 2024-02-05T08:31:04.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/objectstore_tool.6.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-client.rgw.foo.a.log 2024-02-05T08:31:04.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.1.log 2024-02-05T08:31:04.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/objectstore_tool.7.log 2024-02-05T08:31:04.979 INFO:teuthology.orchestra.run.smithi134.stderr: 57.7% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-02-05T08:31:04.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.3.log 2024-02-05T08:31:04.979 INFO:teuthology.orchestra.run.smithi134.stderr:gzip/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/objectstore_tool.7.log: -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mgr.y.log 2024-02-05T08:31:04.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.0.log 2024-02-05T08:31:04.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.cephadm.log 2024-02-05T08:31:04.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.log 2024-02-05T08:31:04.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.2.log 2024-02-05T08:31:04.979 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-volume.log 2024-02-05T08:31:04.980 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/objectstore_tool.8.log 2024-02-05T08:31:04.980 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mon.c.log 2024-02-05T08:31:04.980 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.6.log 2024-02-05T08:31:04.982 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.audit.log 2024-02-05T08:31:04.982 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.4.log 2024-02-05T08:31:04.982 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.7.log 2024-02-05T08:31:04.982 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.audit.log: /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.5.log 2024-02-05T08:31:04.982 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mgr.x.log 2024-02-05T08:31:04.982 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/tcmu-runner.log 2024-02-05T08:31:04.982 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.cephadm.log 2024-02-05T08:31:04.983 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.log 2024-02-05T08:31:04.983 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/tcmu-runner.log: /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-volume.log 2024-02-05T08:31:04.983 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mon.b.log 2024-02-05T08:31:05.005 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/objectstore_tool.8.log: /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mon.c.log: 93.7% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/objectstore_tool.6.log.gz 2024-02-05T08:31:05.011 INFO:teuthology.orchestra.run.smithi134.stderr: 94.3% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/objectstore_tool.7.log.gz 2024-02-05T08:31:05.030 INFO:teuthology.orchestra.run.smithi134.stderr: 91.5% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.log.gz 2024-02-05T08:31:05.031 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.log: /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-volume.log: /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mon.b.log: 67.6% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/tcmu-runner.log.gz 2024-02-05T08:31:05.097 INFO:teuthology.orchestra.run.smithi196.stderr: 90.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-05T08:31:05.152 INFO:teuthology.orchestra.run.smithi134.stderr: 88.9% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.cephadm.log.gz 2024-02-05T08:31:05.176 INFO:teuthology.orchestra.run.smithi134.stderr: 94.0% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.audit.log.gz 2024-02-05T08:31:05.188 INFO:teuthology.orchestra.run.smithi134.stderr: 91.1% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-volume.log.gz 2024-02-05T08:31:05.200 INFO:teuthology.orchestra.run.smithi134.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-05T08:31:05.241 INFO:teuthology.orchestra.run.smithi196.stderr: 81.6% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.cephadm.log.gz 2024-02-05T08:31:05.291 INFO:teuthology.orchestra.run.smithi196.stderr: 90.0% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.audit.log.gz 91.1% 2024-02-05T08:31:05.291 INFO:teuthology.orchestra.run.smithi196.stderr: -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-volume.log.gz 2024-02-05T08:31:05.294 INFO:teuthology.orchestra.run.smithi196.stderr: 84.9% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph.log.gz 2024-02-05T08:31:05.480 INFO:teuthology.orchestra.run.smithi196.stderr: 92.3% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mgr.x.log.gz 2024-02-05T08:31:05.496 INFO:teuthology.orchestra.run.smithi134.stderr: 93.5% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/objectstore_tool.8.log.gz 2024-02-05T08:31:06.023 INFO:teuthology.orchestra.run.smithi134.stderr: 89.7% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mgr.y.log.gz 2024-02-05T08:31:09.609 INFO:teuthology.orchestra.run.smithi196.stderr: 91.2% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mon.b.log.gz 2024-02-05T08:31:23.324 INFO:teuthology.orchestra.run.smithi134.stderr: 90.4% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mon.a.log.gz 2024-02-05T08:31:25.722 INFO:teuthology.orchestra.run.smithi134.stderr: 91.2% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-mon.c.log.gz 2024-02-05T08:32:50.767 INFO:teuthology.orchestra.run.smithi134.stderr: 93.4% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.3.log.gz 2024-02-05T08:34:34.456 INFO:teuthology.orchestra.run.smithi196.stderr: 93.5% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.5.log.gz 2024-02-05T08:35:37.849 INFO:teuthology.orchestra.run.smithi134.stderr: 93.6% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.2.log.gz 2024-02-05T08:36:41.397 INFO:teuthology.orchestra.run.smithi196.stderr: 93.9% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.6.log.gz 2024-02-05T08:36:48.473 INFO:teuthology.orchestra.run.smithi196.stderr: 93.8% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.7.log.gz 2024-02-05T08:36:56.358 INFO:teuthology.orchestra.run.smithi196.stderr: 93.7% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.4.log.gz 2024-02-05T08:36:56.360 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-02-05T08:36:56.360 INFO:teuthology.orchestra.run.smithi196.stderr:real 5m51.430s 2024-02-05T08:36:56.360 INFO:teuthology.orchestra.run.smithi196.stderr:user 4m58.739s 2024-02-05T08:36:56.360 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m13.445s 2024-02-05T08:38:47.713 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.1.log.gz 2024-02-05T08:39:05.141 INFO:teuthology.orchestra.run.smithi134.stderr: 93.3% -- replaced with /var/log/ceph/43e13aa4-c3fc-11ee-95b6-87774f69a715/ceph-osd.0.log.gz 2024-02-05T08:39:05.144 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-02-05T08:39:05.144 INFO:teuthology.orchestra.run.smithi134.stderr:real 8m0.218s 2024-02-05T08:39:05.144 INFO:teuthology.orchestra.run.smithi134.stderr:user 5m33.579s 2024-02-05T08:39:05.144 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m15.165s 2024-02-05T08:39:05.144 INFO:tasks.cephadm:Archiving logs... 2024-02-05T08:39:05.144 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546773/remote/smithi134/log 2024-02-05T08:39:05.146 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-05T08:39:56.375 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546773/remote/smithi196/log 2024-02-05T08:39:56.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-05T08:40:41.465 INFO:tasks.cephadm:Removing cluster... 2024-02-05T08:40:41.466 DEBUG:teuthology.orchestra.run.smithi134:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 --force 2024-02-05T08:40:41.662 INFO:teuthology.orchestra.run.smithi134.stdout:Deleting cluster with fsid: 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T08:40:42.329 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 43e13aa4-c3fc-11ee-95b6-87774f69a715 --force 2024-02-05T08:40:42.567 INFO:teuthology.orchestra.run.smithi196.stdout:Deleting cluster with fsid: 43e13aa4-c3fc-11ee-95b6-87774f69a715 2024-02-05T08:40:43.182 INFO:tasks.cephadm:Removing cephadm ... 2024-02-05T08:40:43.183 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-05T08:40:43.208 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-05T08:40:43.223 INFO:tasks.cephadm:Teardown complete 2024-02-05T08:40:43.224 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-05T08:40:43.234 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-05T08:40:43.234 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-05T08:40:43.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-05T08:40:43.360 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-05T08:40:43.375 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-05T08:40:43.375 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T08:40:43.378 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T08:40:43.392 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-02-05T08:40:43.400 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-02-05T08:40:43.414 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T08:40:43.414 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-02-05T08:40:43.414 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 +1010us[+1010us] +/- 76ms 2024-02-05T08:40:43.415 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -475us[ -463us] +/- 56ms 2024-02-05T08:40:43.415 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 -446us[ -434us] +/- 91ms 2024-02-05T08:40:43.415 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T08:40:43.432 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T08:40:43.432 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-02-05T08:40:43.432 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 +1015us[+1027us] +/- 76ms 2024-02-05T08:40:43.432 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv02.front.sepia.ceph.com 3 6 377 14 -466us[ -455us] +/- 56ms 2024-02-05T08:40:43.432 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 -433us[ -433us] +/- 91ms 2024-02-05T08:40:43.433 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T08:40:43.433 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-05T08:40:43.442 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-05T08:40:43.442 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-05T08:40:43.488 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-05T08:40:43.615 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-05T08:40:43.702 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T08:40:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:40:43.735 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T08:40:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:40:43.764 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-05T08:40:43.775 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-05T08:40:43.808 INFO:teuthology.task.internal:Duration was 2982.226164 seconds 2024-02-05T08:40:43.809 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-05T08:40:43.858 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-05T08:40:43.859 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-05T08:40:43.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-05T08:40:43.953 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T08:40:43.968 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T08:40:44.374 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-05T08:40:44.374 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-02-05T08:40:44.375 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-05T08:40:44.396 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-02-05T08:40:44.396 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-05T08:40:44.420 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-05T08:40:44.420 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-05T08:40:44.439 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-05T08:40:44.490 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-05T08:40:44.490 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T08:40:44.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T08:40:46.099 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-05T08:40:46.114 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-05T08:40:46.115 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-05T08:40:46.155 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-05T08:40:46.196 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-05T08:40:46.207 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-05T08:40:46.212 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-05T08:40:46.240 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-02-05T08:40:46.279 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-02-05T08:40:46.304 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-05T08:40:46.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:40:46.322 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-05T08:40:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T08:40:46.361 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-05T08:40:46.371 INFO:teuthology.task.internal:Transferring archived files... 2024-02-05T08:40:46.372 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546773/remote/smithi134 2024-02-05T08:40:46.372 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-05T08:40:46.459 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546773/remote/smithi196 2024-02-05T08:40:46.459 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-05T08:40:46.504 INFO:teuthology.task.internal:Removing archive directory... 2024-02-05T08:40:46.504 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-05T08:40:46.507 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-05T08:40:46.554 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-05T08:40:46.567 INFO:teuthology.task.internal:Not uploading archives. 2024-02-05T08:40:46.567 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-05T08:40:46.635 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-05T08:40:46.635 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-05T08:40:46.638 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-05T08:40:46.650 INFO:teuthology.orchestra.run.smithi134.stdout: 265293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 08:40 /home/ubuntu/cephtest 2024-02-05T08:40:46.654 INFO:teuthology.orchestra.run.smithi196.stdout: 266030 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 08:40 /home/ubuntu/cephtest 2024-02-05T08:40:46.666 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-05T08:40:46.698 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-05T08:40:46.815 INFO:teuthology.nuke:Checking targets against current locks 2024-02-05T08:40:46.838 DEBUG:teuthology.nuke:shortname: smithi134 2024-02-05T08:40:46.838 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T08:40:46.861 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546773', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 07:42:18.792631', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKiYq7PnZVrWaRE/XchE/ec0BBUYiIqXs1TFpVXu3BjaqUFQu5fd8tuLhFhrEaQYnqpJrUYohctQwcmiVcJYk2U='} 2024-02-05T08:40:46.865 DEBUG:teuthology.nuke:shortname: smithi196 2024-02-05T08:40:46.865 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T08:40:46.890 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546773', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 07:42:18.794313', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPL54Xfgje0159iEnIFNlZXin0/8cKqk211ej0lfjf0mi2xcneOS7cDg16GCqeRq7/NmyD+sGJ44J8DkWehus3Q='} 2024-02-05T08:40:46.914 INFO:teuthology.orchestra.console.smithi134:Power off 2024-02-05T08:40:46.914 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-05T08:40:46.940 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-02-05T08:40:46.940 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:40:46.961 INFO:teuthology.orchestra.console.smithi196:Power off 2024-02-05T08:40:46.962 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-05T08:40:46.977 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-02-05T08:40:46.986 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off 2024-02-05T08:40:46.986 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:40:47.012 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-02-05T08:40:50.978 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:40:51.013 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:40:51.089 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-02-05T08:40:51.123 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-02-05T08:40:55.093 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:40:55.124 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:40:55.203 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-02-05T08:40:55.234 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off 2024-02-05T08:40:55.335 INFO:teuthology.orchestra.console.smithi196:Power off completed 2024-02-05T08:40:59.208 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T08:40:59.318 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-02-05T08:40:59.419 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-02-05T08:40:59.523 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} duration: 2982.2261641025543 failure_reason: '"2024-02-05T08:10:00.000144+0000 mon.a (mon.0) 1023 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log' flavor: default owner: scheduled_gabrioux@teuthology success: false 2024-02-05T08:40:59.523 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-05T08:40:59.600 INFO:teuthology.run:FAIL